Last Updated on Thursday, June 28, 2001 at 00:51:02. by Author: Andrei Sukhanov

DAQ, General Information/Help

(Expert) (Checklist)

Contents

Restarting DAQ

  1. Locate 'vmesparc' window and enter:
    reset.mc; boot.mc; ROC;
    Note: You have only 3 minutes to execute steps 2 through 4. If you are late then ROC program will crash and you will need to repeat from step 1 again.
    If 'vmesparc' does not exist - ssh to phobos@phobosx, cd to $DAQ and execute 'source daq.start'.
  2. Locate 'phatdaq' window and enter:
    phatdaq 3
    It should say "Accepting...
  3. Locate 'runcontrol' window (lower right corner) change to ~phobos/DAQ directory and enter:
    root runcontrol.C
  4. When you get the root prompt, enter:
    rc_init_silicon()
    In the vmesparc window you should see a message saying that 42 FECs were loaded.
  5. You can check that all communications have been established by executing rc_stop_run()
    You should finally see "Ready to start" message, if don't - repeat the command again.
  6. The normal standby mode of running is
    rc_start(RUN_SIPED,999999)
    On Fastbus crate the green lights should blink with the rate of heartbeat and yellow lights - when event was accepted (defined by prescaling factor)
  7. The normal data taking mode:
    rc_start(RUN_ALLBEAM+WRITING)
    You should see flashing green and yellow lights on Fastbus
    The scaler 'EVENTS ACCEPTED' should increment each time for new event
    rc_status() should show growing number of Events and MB.
    No error messages in phatdaq window

Run Control Commands

Start Silicon Pedestal Run (SIPED)

In the runcontrol window, type:

rc_start(RUN_SIPED+WRITING)

to see the run status:

rc_status()

type this a couple of times, to check how many events have been collected
When phatdaq finishes the run you need still to stop it:

rc_stop_run()

Start Silicon Calibration Run (SICAL)

In general, SICAL (and SIPED) should be done during period when there is no beam in the machine. It takes about 7 minutes to complete a Silicon calibration run. If it so happens that RHIC starts beam activity while you are in the middle of an SICAL run, please stop the run and enter a comment into its runlog that the run was stopped due to beam activity.

in the run control window, type:

rc_start(RUN_SICAL+WRITING)
rc_status()

type this a couple of times, to check how many events have been collected

rc_stop_run()

Start Normal Run

rc_start(RUN_ALLBEAM+WRITING)

to run limited number of events (i.e. 3400):

rc_start(RUN_ALLBEAM+WRITING,3400)

Stop Any Run

rc_stop_run();

Wait 10 seconds before the next command.

How to change disk for writing

The DAQ will change file automatically after writing 1 GB of data. It will also change the disk if there is not enough space there. To change to disk 1:

rc_change_disk(1)

Reload Silicon Parameters

As of today to reload silicon parameters, you need to restart DAQ.

This is necessary if FECs was turned on or if  Cable and FEC Connection Worksheet had been changed.

 

Known Problems/Errors in EventBuilding

  1. If you experience any problem (beeps, running messages etc.) try first to stop the run, wait 5 sec and restart it.
  2. (Severe, Rare Error) Unexpected event on Silicon Crate.
            RINT Timeout
            ...
    Action: You need to restart DAQ:
  3. (Severe, Rare Error): In rare cases, the Silicon and Plastic ROC's will lose synchronisation with each other and the Event Manager leading to continuous error messages like:
            Error. FEC data from event b3 found in event c3, block [3]
            0000:000080e8 00000018 00000000 00000000 12345678 000001c3 ff000028 a5a5a5a5
            0020:90000382 000101c3 44a62534 00000000 01000000 00000000 00000000 00000000

            0000:03170000 0000aaaa 86000000 f01800b3 ab9ef510 00000000 000080f8 00000000
            Error. FEC data from event c3 found in event d3, block [3]
            0000:000080e8 00000018 00000000 00000000 12345678 000001d3 ff000028 a5a5a5a5
            0020:90000382 000101d3 44b63544 00000000 01000000 00000000 00000000 00000000

            0000:03170000 0000aaaa 86000000 f01800c3 ab9ef510 00000000 000080f8 00000000
            Error. FEC data from event d3 found in event e3, block [3]
            0000:000080e8 00000018 00000000 00000000 12345678 000001e3 ff000028 a5a5a5a5
            0020:90000382 000101e3 44c64554 00000000 01000000 00000000 00000000 00000000

            0000:03170000 0000aaaa 86000000 f01800d3 ab9ef510 00000000 000080f8 00000000
    Action: You need to stop the run, wait a few seconds,  start a new run without writing, if errors still present - repeat this sequence one more time. If after starting a new run, the error persists, you may need to reset the Mercury Crate and Fastbus Crates, and restart PhatDAQ (in that order). If that doesn't fix the problem, you need to call Andrei/Pradeep.
  4. (Severe Error): In some cases, the busy signals and handshakes in the various ROCs may be dropped (typically when switching from low trigger   rates to higher ones). This will lead to severe data errors in the Silicon data and continuous messages like:
            Error in FEC format, block [1]

            0000:51ab51ea 51165155 518351e8 515f51c8 f0003038 a5a5a5a5 51b65181 51c851a0
            Error in FEC format, block [1]

            0000:50fc5193 511a5161 516651a1 514750b7 f0003038 a5a5a5a5 516d50df 51155130
            Error in FEC format, block [1]

            0000:516651a3 5177510f 51915182 51505198 f0003038 a5a5a5a5 51c35134 516f50f0
            Error in FEC format, block [1]
    Action:  Same as (3) above.
  5. If vmesparc does not respond to the stop command try:
    .!daq_semaphore 5
    and issue rc_stop() several times.

How to restart parameter monitoring service

If Si Hybrids reading on PHOBOS portal gets old for 10 min, you need to restart rocdbgateway program. Locate the ROCDB window and press control C there.

To start this program from scratch:
    cd $DAQ
    rocdbgateway 1 0
Here the first argument enables connection to the database, second argument enables writing to the file rocdblogMMDD.csv
The program detects if phatdaq is running, in this case it requests the whole event from the phatdaq, extracts monitoring information from the FEC trailer blocks and sends it to the datadase.

To change reporting interval to 60 seconds:
    ~phobos/DAQ/rocdbgateway 1 0 60

How to sink data files to HPSS

All commands should be executed on phobosx terminal (in HPSS sinking panel).

You should avoid to occupy the same disk by writing and reading, to change the disk:
rc_change_disk(1);   // change writing disk to /data/1

After that you can start another data taking run in phatdaq and  start sinking data. To sink data, you use the write_and_recon command. This command will transfer the file to HPSS and afterwards spawn the automated data validation processing on the CRS. To use this, use:

~phobsink/bin/write_and_recon [<filename> ...]

You can ignore  messages like "Pseudo-terminal ..", "TPhDatabase ..." "Warning file iostream.h ..."

Any number of filenames can be specified. Files will be deleted from the local disk after being transferred. For example:

~phobsink/bin/write_and_recon /data/5/PhoRaw002345*.root

will copy all the sequences for run 2345 to HPSS and spawn data validation on them.

If for some reason you do not wish to start the data validation processing, you can use write_to_HPSS. With write_to_HPSS, you can either write every file in a particular directory, or a specific file, by entering (in the phobosx terminal window) the command:

~phobsink/bin/write_to_HPSS <filename or directory name>

So for example:

~phobsink/bin/write_to_HPSS /data/0

will write every data file in /data/0 to HPSS. If you wanted to sink only one file, say for run 2345, you would use:

~phobsink/bin/write_to_HPSS /data/0/PhoRaw002345s000.root

By default, write_to_HPSS will delete the file from the local directory after copying it to HPSS and verifying that the sizes match. If you do not want it to do this, specify the --nodelete flag:

~/phobsink/bin/write_to_HPSS --nodelete /data/0/PhoRaw002345s000.root

If during sinking a file reports that it exists on HPSS but with a different size, then write and email to George Heintlzeman with the run number and sequence number of the file sent, and note it in the run logbook.

If there is a data file of zero length, do not sink it, but call Andrei to explain what to do.

 

If asks you for passwords, then enter "phobos password on phobosx" until done. Before start new job's sinking , do the following to stop having to eneter the password each time

eval `ssh -agent` [ENTER]
ssh -add [ENTER]
(will ask for passphrase, i.e. phobos on phobosx one) enter it,

Then in that window should sink without asking for password.

How to backup tapes to DLT tape

The tape capacity is between 35GB (uncompressed) and 70GB(compressed). The full disk is about 15 GB.

To list all files on the tape:

Recovering from phobosx crash

If phobosx crashes (all DAQ windows disappeared, PHOBOSX monitor blank), it will reboot by itself after 5-10 minutes.

To get back DAQ windows on PHATDAQ X Term:

  1. open terminal window and start ssh session on phobosx
    ssh phobos@phobosx.phobos.bnl.gov
  2. open all windows:
    daq.start
  3. in phatdaq window:
    ~phobos/DAQ/phatdaq 3
  4. in runcontrol window:
    root runcontrol.C
    rc_init()
  5. in plastic_ROC, silicon_ROC and trigVME_ROC windows:
    telnet
    press ENTER, you should see the prompt '>'
  6. in rocdb window:
    cd ~phobos/DAQ/log
    ~phobos/DAQ/rocdbgateway 1 1     60
  7. Stop run and check if you get responses in all ROC windows. If don't, then you need to reboot corresponding ROC by pressing RST button on the ROC.

 

 

 

Please send all comments and suggestions regarding this page to Andrei.