Running Data Validation

1. Moving data files & monitoring HPSS
2. Monitoring jobs on CRS
3. Checking status of runs
4. Output: what & where
5. Checking log and error files
6. Running a re-pass of data
7. FAQ

Moving Data Files & Monitoring HPSS

• By default, the data taken by the DAQ are handled by the DataMover, and sunk to HPSS, from where Data Validation processes are run on the "ALLBEAM" type run files.

• To check the current sinking rate, follow the link: Data Sinking Rate, from the Counting House Portal, under Data Validation.

• For detailed documentation on how to check the status of HPSS, click here.

Monitoring Jobs on CRS

• From phucsia, on the Data Monitor Desktop, start the "CRS Farm Status for Phobos" window:

• Open the DATA VALIDATON MONITOR window (green), by typing:

                shift@phucsia:~> dvmonwin

• From the new window (green), login to RCF, by typing at the prompts:

                shift@phucsia:~> ssh phobreco@rsshgw.rhic.bnl.gov

• In the same window (green), now login to rcrsuser:

                rsshgw00:~ > ssh rcrsuser2

• Start the monitoring program by typing at the prompt:

              phobreco@rcrsuser2 > tk_CRS_status.pl &

• This will pop-up a new window entitled "CRS Farm Status for Phobos", which allows monitoring of the jobs on CRS.

• To view detailed documentation for this window, click here.

• Alternatively, from the Counting House Portal use the link: "CRS Job Status", under Data Validation. To view this page, Netscape is recommended, since IExplorer keeps the old pages on cache and doesn't show the most recent version.

Checking Status of Runs

• Go to the Sequence Processing Status webpage: http://www.bnl.gov/phobos/Detectors/ASP/Data_Processing/requestforsps.asp

• Enter the run range you want to look at.

• For every sequence, there are four colored boxes. The third box indicates the status of DV running: if it is yellow, the job is in progress; green indicates the processing is finished and this sequence is ready for validation; red means that the job has crashed.

Output: What & Where

• The output of Data Validation goes into $DV_OUT/pr01, on rcf. This disk is viewable from any of the rcas400x nodes. The PhoHit and PhoDST files are placed in HPSS.

• The output consists of the following files:

                 On $DV_OUT/pr01
        err/DateXXXXX_TimeZZZZ_Set1_[Run Number]s[Seq Number]_ONLV_X_X_X.err
        log/DateXXXXX_TimeZZZZ_Set1_[Run Number]s[Seq Number]_ONLV_X_X_X.log
        trgtree/TrgTree[Run Number]s[Seq Number]_ONLV_X_X_X.root
        objmgr/PhoOM[Run Number]s[Seq Number]_ONLV_X_X_X.root
        dvtree/DVTree[Run Number]s[Seq Number]_ONLV_X_X_X.root
                On HPSS:  /home/phobreco/pr01/recon
        PhoHit[Run Number]s[Seq Number]_ONLV_X_X_X.root
        PhoDST[Run Number]s[Seq Number]_ONLV_X_X_X.root

Checking Log and Error Files

• Log files:

              >  cd $DV_OUT/pr01/log

    The file names contain the run and sequence number: DateXXXXX_TimeZZZZ_Set1_[Run Number]s[Seq Number]_ONLV_X_X_X.log

• Error files:

            >  cd $DV_OUT/pr01/err

    The file names contain the run and sequence number: DateXXXXX_TimeZZZZ_Set1_[Run Number]s[Seq Number]_ONLV_X_X_X.err

• To check error files for errors and warnings, from $DV_OUT/pr01/err, type:

            > ./error.pl [Run #] [Seq #]

Running a Re-pass of Data

• Use database to get list of ALLBEAM runs in the range of the re-pass.

• Make a list of sequences with each status flag, and keep that list.

• Change the status of all sequences that have status =2 to status =0 and keep a record of these.

• Change the status of all sequences that ended in latchup and were marked as status =1 to status =0.

• Start submitting runs, not more than ~20 sequences at a time (if there is data taking at the time), since they take very long to stage and might affect running in the counting house. Make sure that drop queue is set, and that the queue used is 4. If there is no data taking in the counting house for a significant amount of time, queue 5 can be used, but only to about 1/2 capacity.

• Monitor the progress of running using the sequence status pages.

• For any runs that crash, check log files (and emails).

Frequently Asked Questions

• Something is wrong with RCF, how do I report it to them?

1. Submit a "trouble ticket", by going to the RCF webpage: http://www.rhic.bnl.gov/RCF, and following the link.
2. Page On-call RCF Staff, also by following the link on their webpage.
3. If the pagers don't work, call the "24h hot-line" number, 5480.
4. If after some time there is no response, and the problem is still present, page this number: 557-5268.

• I just got on shift. How do I know which runs should be validated?

    Look at the "Runs to be validated" window on phucsia, at the bottom of the Data Monitor Desktop.

• How can I manually sink data to from DAQ to HPSS?
• Something happened on phuscia, and I need to restart the data tracking windows, how do I do it?

1. Type  "ps -ef | grep phat dv_ready.C". If don't get any line returned, goto 2. If get a line with this process number, it means it is running. If you want to kill it, because it is in a funny state, then type "kill -9 {Process ID}".Then follow 2 to restart it. 
2. Type "cd"
3. Type "phat dv_ready.C", this will then start the windows, they take about 10 minutes to display.
4. If a problem occurs, then call a DV expert. 

    Click here for instructions. This is not the normal procedure, by default data should be handled by the DataMover only.