How To Run Phobos Data Acquisition System

Last Updated on Tuesday, August 19, 2014 at 15:50:43.

Table of contents:

 

Login

Login to the phobosx as phobos

Locate terminal windows:

(if windows are not exist then start it using following commands:
dtterm -name phatdaq -sl 10000 &
dtterm -name runcontrol -geometry 80x12 &)

Start DAQ system

Start Readout Controllers

Start event builder phatdaq in the phatdaq window:

Start run control macro in the runcontrol window:

After that command, the event builder will be waiting until all ROCs are connected. You should wait until it say 'Ready to start'

To enter commands you can use menu buttons (or type them directly in the runcontrol window)

Run procedure

Note. Due to known start/stop event mismatch problem, to take the normal run you need to exit all ReadOut Controllers, and restart all of them (including phatdaq) again.

If online monitoring is required, make sure that the online computer is ready to analyze data.

To start run:

Enter one of the following start run commands:

Use following command to see if events are being recorded.

rc_status()

You also should see increased CPU, Disk and Packet activity in the performance monitor.
The phatdaq will report periodically (each 10 MB) updating of the data file. When the data file approaches 2GB limit, it will be automatically closed and a new file with incremented sequence number will be open. Make sure that there is enough free space on the disk, otherwise the program will halt.

To stop run:

rc_stop_run()

If necessary, write out run comments to database using WWW browser at https://www.bnl.gov/phobos/Secure/ASP/run_logbook_comment.asp

Around the clock command sequence

Suppose you are currently taking the normal run.

  1. Stop the run:
    rc_stop_run()
  2. Wait 5 sec. Check for message "Closing everything" in phatdaq window. If there is no such message, try to stop again.
  3. Start pedestal run for 7000 events and after that the calibration run for 6000 events
    rc_start_all_runs()
    The pedestal run takes 7000/300=23 sec, the calibration run - 6000/13.5=8 min. You will have time to calculate pedestals the run will stop automatically.
  4. Use rc_status()command to check if calibration run is finished
  5. Exit all ROCs:
    rc_exit()
    answer 'y' on the first prompt
  6. Restart the Silicon ROC(in Silicon_roc window):
    <ESC>k
    sp ph_roc
  7. Restart the Plastic ROC(in Plastic_roc window):
    <ESC>k
    >sp(rocsfi,"192.168.14.1",7000,1,3,0,0)
    (always choose the largest number if asked)
  8. Restart the phatdaq (in 'phatdaq window)
    ~phobos/DAQ/phatdaq 2
  9. Init:
    rc_init()
  10. Start the new normal run:
    rc_start_run()
  11. Wait 1 hour and repeat from 1:
  12. Go to the Online Machine and start the Monitor Grabber (unless development is in progress)

    Calculation of the pedestals

    Find the 'pedcalc' window icon on the left side. In that window:

    1. If you don't have root prompt then you need to start phat:
      cd ~phobos/DAQ
      phat
    2. having the root prompt, enter:
      .! cp -p /data/#/PhoRaw000###s00#.root /data/7/peddata
      (here # is the disk number, ### is the run number for pedestal run, 00# is the file sequence number. You can check correct file name using .! ls /data/#)
    3. .x pedcalc.C(###)
      if you get an error message "Error: no symbol ...", then you are apparenly in the wrong directory, enter .q command to exit phat and continue from step 1
    4. use default values <RETURN>

    Calculation of the gains

    Find the 'pedcalc' window icon on the left side. In that window:

    1. If not in root then:
      cd ~phobos/DAQ
      phat
    2. having the root prompt, enter:
      .! cp -p /data/0/PhoRaw000###s00#.root /data/7/gaindata
      (here # is the disk number, ### is the run number for pedestal run, 00# is the file sequence number. You can check correct file name using .! ls /data/#)
    3. .x calcalc.C(#)
      if you get an error message "Error: no symbol ...", then you are apparenly in the wrong directory, enter .q command to exit phat and continue from step 1
    4. use default values <RETURN>

Pedestal Run

The pedestal run should be taken and calculated regularly. During the calculation phase the pedestal data will be automatically sent to the online computer for up-to-date pedestal setting

Make sure that phatdaq and roc_silicon are running.
In runcontrol window enter:

rc_start_pedestal_run()
(pedestal run with writing)

Run will be stopped automatically after 1000 events (5 sec). To check if it stopped - use rc_status() command

You can start taking calibration data now (see Calibration Run), during that time you can calculate pedestals running PHAT macro in the 'pedcalc' terminal window:

Calibration Run

The calibration run should be taken regularly. During the calculation phase the gain calibration data will be automatically sent to the online computer for up-to-date calibration setting

Make sure that phatdaq and roc_silicon are running.
In runcontrol window enter:

rc_start_calibration_run()
(calibration  run with writing)

Run will be stopped automatically after 6000 events (8 min). To check if it stopped - use rc_status() command.

Trigger connection

 

To move data files to HPSS:

from default directory:
/export/home/phobsink/bin/write_to_RCF /data/0
(
and from other disks if required)
/export/home/phobsink/bin/write_to_RCF /data/1
/export/home/phobsink/bin/write_to_RCF /data/2
/export/home/phobsink/bin/write_to_RCF /data/3
/export/home/phobsink/bin/write_to_RCF /data/4
/export/home/phobsink/bin/write_to_RCF /data/5
/export/home/phobsink/bin/write_to_RCF /data/6
/export/home/phobsink/bin/write_to_RCF /data/7

Write down to the logbook.

Useful Commands:

general commands:

Notes:

  1. If phatdaq was killed or crashed when run was in progress then one need to restart all ROCs and init phatdaq.
  2. Closing the runcontrol will exit phatdaq.
  3. Do not enter init command twice!

Troubleshooting

To contact DAQ experts:

If phatdaq crashed and file wasn't properly closed

In that case the next run might have the same run number and therefore the previous file may be overwritten. To be safe, you need to rename current file and make a short run just to increment global run number.
to rename the file:
>mv PhoRaw000###s00#.root PhoRaw000###s00#crashed.root

In case of problems with phatdaq

Try to stop the run first:
rc_stop_run()

Exit phatdaq but keep readout controllers running:

rc_exit()
Do you want to exit all ReadOut Controllers? [y/n] >
n
Do you want to exit phatdaq? [y/n]>
y

How to kill phatdaq

If phatdaq did not exit, try to <Control C> the phatdaq. If this does not work, kill it from other terminal window. To do that, locate a process ID for the /export/home/phobos/phatdaq/phatdaq, then kill it:
>ps -ef | grep phatdaq -
>kill ###
- where ### is the process ID

Restart the phatdaq again:
>~phobos/DAQ/phatdaq 2

Restart, if necessary, runcontrol and init phatdaq:
.x runcontrol.C

If you get 'Ready to start' in phatdaq window - you are in business again, but for safety do following:
rc_start_run_without_writing()
rc_stop_run()

After succesful completion of above operation you can continue the regular work.

If you don't get 'Ready to start' in phatdaq window - then you need to restart or reboot ROC. When it is got up again, you will see this 'Ready to start'.

How to restart ReadOut Controllers

Exit all ReadOut Controllers using rc_exit() command and answer 'y' to first prompt.

Mercury VME crate

Locate Silicon_roc terminal window.

Software restart.

(Try it first, you may use <ESC>K to retrieve previous commands)
sp ph_roc
You should get "Waiting for Server command" message in Silicon_roc window.

Hardware restart

Press the 'RST' button MVME 2600-2 module (leftmost in the crate). The ph_roc will start automatically. The full reboot takes 3-5 minutes. After that time you should see a message in the phatdaq window saying that Silicon ROC was registered.

FASTBUS

Software restart
  1. Locate plastic_roc terminal window
  2. If rocsfi had crashed you will need to kill last rocsfi program. To do that, find task name using 'i' command. Look for rocsfi in the ENTRY column. If it exists then kill it using 'td' command, i.e:
    td t13
  3. restart rocsfi program (you can use <ESC>K command to retrieve previous command)
    sp (rocsfi,"192.168.14.1",7000,1,3,0,0)
    choose last version (highest number) if rocsfi overloaded
Hardware restart

The hardware reset is not recommended, contact Andrei first.
By the way turning on and off the FASTBUS power supply might help.The ReadOut Controller will start automatically after 2-3 minutes.

If you can't copy the pedestal/calibration file to /data/7

If you get the message 'error reading file' during the cp command, most likely case is that /data/7 is full.

If you don't get proper response from the runcontrol

How to run phatdaq without database connection

>~phobos/DAQ/phatdaq 2 0

How to backup disks to DLT tape

The tape capacity is between 35GB (uncompressed) and 70GB(compressed). The full disk is about 15 GB.

To list all files on the tape:

 

How to change global run number

Normally you don't need to change it. If phatdaq encounter s that the file is already exist it will increment current run number. If after 10 attempts it still cannot open file then you, probably need to change global number.

cd ~phobos/DAQ
mv config/phatdaq_runlog.root config/phatdaq_runlog.root.save
phat
.x runlog_write.C(runnum)
The next time you start phatdaq the global run number will be runnum+1;

How to start parameter monitoring service

On phobosx:
go to directory where you want the file to be written (~phobos/DAQ/log). Keep in mind that if you enable writing, then the new data will be appended to the existing file rocdblogMMDD.csv (where MMDD are month and day), if it does not exit then the new file will be created. To start the program::
    ~phobos/DAQ/rocdbgateway 1 1
Here the first argument enables connection to the database, second argument enables writing to the file rocdblogMMDD.csv
The program will be sending all monitoring information received at the UDP port 7010 to the database every 4 minutes. The data at this port comes from Silicon during off run time. The program detects if phatdaq is running, in this case it requests the whole event from the phatdaq, extracts monitoring information from the FEC trailer blocks and sends it to the datadase.

To change reporting interval to 123 seconds:
    ~phobos/DAQ/rocdbgateway 1 1 123

To terminate program one must kill the process rocdbgateway, to find the process ID use:
    ps -e -o pid -o fname | grep rocdbgat

Testing of the Programmable Trigger Logic

On phobosx - open the telnet session to the trigvme host:
telnet trigvme
If it said that the system is engaged that means that the session is already open somewhere, locate it.
The following is the list of the command which you can cut and paste for execution.

# use pcdp_init to initiate the globals
pcdp_init(0)
# if it said 'undefined symbol: pcdp_init' then you need to download the library uncommenting
# following line:
# ld </export/home/phobos/phatdaq/roc/src/phatdaqLib.o

# reset TML0 (board 4), TML1 (board 5) and EMM (board 6)
pcdp_set(4,0,0)
pcdp_set(4,1,0xc000)
pcdp_set(5,0,0)
pcdp_set(5,1,0xc000)
pcdp_set(6,0,0)
pcdp_set(6,1,0xc000)
# Use pcdp_init to look at the registers
pcdp_init(0)
# All scalers should be zero

# Connect trigger to the cosmic trigger input TML0.CI[8].
# Connect L0Pulse (TML0.CO[2]) to Veto (TML0.CI[15]) to simulate external reset
# Enable all cosmic trigger at CI[8:11] using pcdp_set:
pcdp_set(4,0,0x0f00)
# Should see the L0Pulse at CO[2] and L0 at C0[0]
# Enable all triggers if needed, normal CI[0:7] and cosmic
pcdp_set(4,0,0x0fff)
# Check propagation of all triggers

# Generate L0 pulse from TML0.CO[0] using NIM Gate Delay Unit 794, the width of the pulse
# defines the interval when L1 trigger being accepted. The L1 will be generated at the
# falling edge of L0
# Connect L0 from to TML1.CI[0]
# For L1 logic to be working one must provide reset of the L1 at TML1.CI[12].
# Use another Gate Delay Unit to generate delayed L1 from L1Pulse at TML1.CI[1]
# Connect the delayed L1 to TML1.CI[12].
# Enable and set prescaler0 to 0, this will set the prescaling factor equal to 1
pcdp_set(5,0,0x0001)
# The test signal L1OR at TML1.CO[3] is at 40 ns relative to main trigger
# For debugging you may look at  the prescaled signal at TML1.CO[7] (at 46.6 ns)
# The L1 signal should appear at RML1.CO[0], it starts at the end of L0 and propagates
# until external reset.
# The combined Busy at TML1.CO[15] starts at the beginning of the L0 and end at the end of L1
#
# Try different prescaling factors:
# Divide by 2:
# use pcdp_set(5,0,0x0003) or
pcdp_copy_bits(5,0,1,3,1)
# to check what was written in bits 1,2,3 of board 5, register 0 use: pcdp_get_mirror_bits(5,0,1,3)
#
# Divide by 4:
pcdp_copy_bits(5,0,1,3,2)
# Divide by 8:
pcdp_copy_bits(5,0,1,3,3)
# Divide by 16:
pcdp_copy_bits(5,0,1,3,4)
# Divide by 32:
pcdp_copy_bits(5,0,1,3,5)
# Divide by 64:
pcdp_copy_bits(5,0,1,3,6)
# Divide by 128:
pcdp_copy_bits(5,0,1,3,7)

# Test of the Fast Clear.
# If prescaling factor not equal to 1 then Fast Clear (TML1.CO[2]) will be generated instead of L1
# for missed events.
# If we disable all Prescalers:
pcdp_set(5,0,0x0000)
pcdp_set(5,2,0x0000)
# Then Fast Clear should appear for each event.
# In the case of Fast Clear, the combined Busy is extended by 1 us after end of L0 (for TDC/ADC)