Last Updated on Wednesday, December 15, 2004 at 11:03:22.
by Joel CorboThe Central Alarmer is a LabView program created to provide the shift crew with a single display that can be used to monitor the status of much of the software being run on the other computers in the counting house. It signals an alarm both for an alarm condition sent by one of the clients and for a broken or timed-out connection.
To start the Central Alarmer, double-click on the "PhAlarmCurrent.llb" icon sitting in the middle of the desktop on PhAlarm. This should immediately bring up the front panel of the Central Alarmer. If not, it will bring up a list of VI’s. Scroll through the list until you find "PhAlarmMain" and double-click on that. You should now have the Central Alarmer’s front panel on the screen.
To start the program, click on the little white arrow at the top left of the LabView window. There may be some error conditions reported from some of the clients immediately after the Central Alarmer starts. These should clear after a few seconds. To stop the program, click the STOP button below the arrow. There is text in the upper left corner of the front panel indicating where both of these buttons are.
The front panel of the Central Alarmer consists of a number of indicator windows whose color indicates the status of that client. Each indicator has a label that tells which client (MagMon, OnlLatchup1, DAQ, etc.) it is displaying, and some of them have a smaller label indicating on which machine on the console the client runs. The indicator windows are arranged into three groups, each of which is labeled as well. The first column of four windows shows the LabView-based clients. The second and third column of windows show the Online clients. The fourth and fifth columns of windows show the 'other' clients. Each window also contains two buttons and an LED indicator:
PAUSE: The PAUSE button is used to put the indicator window of a particular client into standby. This does not affect the client in the sense that the client continues to send status messages to the Central Alarmer. Instead, it turns the indicator window gray, preventing the Alarmer from displaying the information sent by the client. To unpause the indicator window, press the PAUSE button a second time. The PAUSE button is useful for preventing an alarm from going off when a client is deliberately taken off-line or for silencing a client which is sending an error condition which had already been acknowledged. By all means, NEVER quiet the Central Alarmer by turning down the speakers. Use the PAUSE button instead.
In addition to the individual PAUSE buttons, there is also a PAUSE ALL button on the left side of the Central Alarme front panel between the STOP button and the TimeStamp. The PAUSE ALL button, as its name suggests, pauses all of the active indicator windows on the screen. The idea behind this is to have one button which can be used to quickly pause a number of clients in the event that one root cause is making them all alarm atthe same time (i.e. online crashes or there is a silicon latchup).
INFO: The INFO button is used to find out why an alarm is going off. When a client is causing an alarm to sound, pressing the INFO button (sometimes more than one press is necessary) will cause a dialog box to appear listing the client having the problem, what the problem is, and what action to take to resolve the problem. Clicking on the OK button in the dialog box will close it and reset the alarm. However, in the case of an error condition, the alarm will go off again if the problem has not been resolved.
Alarm Indicator: The Alarm Indicator is a small LED in the lower right corner of each of the indicator windows. It turns red to indicate which client is setting off an alarm. It is necessary to check these indicators because very often it will be ambiguous as to which and how many clients are alarming.
Also on the front panel of the Central Alarmer is a box labeled "TimeStamp." This should update every ten seconds to show that the program is running. There is also a special indicator window labeled "Is There Beam?" This window receives its information from RadMon and indicates whether or not there is beam in the machine. Pressing the INFO button on this window at any time will indicate whether there is beam, as will the big green or blue box under the indicator window. This window also sounds a special alarm once when conditions change from no-beam to beam and a different alarm when conditions change from beam to no-beam.
If one were to scroll the LabView window so the that area to the right of the indicator windows were visible, one would find three arrays made up of strings, labeled client list, error message, and timeout message. The client list array is a list of the clients' names in the order of their port number. The error message array is a list of the different error messages associated with the clients. The timeout message array is a list of the timeout error associated with each client. The contents of these arrays is what makes up the messages in the INFO dialog boxes, and as such, should only be changed by an expert.
The colors of the indicator windows show at-a-glance information about the clients. The meaning of the colors is listed in brief on the Central Alarmer front panel.
RED: The client is sending an error message. This will cause an alarm to sound until both the error condition stops and the shift crew has acknowledged the error by pressing the INFO button. The alarm will continue so long as the indicator window is red.
ORANGE: The client has timed out or the connection to the client has been broken. This will cause an alarm to sound until the shift crew acknowledges the timeout by clicking the INFO button. After this, the alarm will stop but the indicator window will remain orange.
YELLOW: The Central Alarmer has initialized and is waiting for the client to connect. Note: It is NOT a bad thing for an indicator window to be yellow if it is known that there is no client trying to connect to it. In that case, yellow is as acceptable as green.
GREEN: The client has connected successfully and is reporting no error condition. Ideally, everything should always be green, except in the instance mentioned under the explanation of what "yellow" means.
BLUE/PURPLE: The Central Alarmer is reporting an internal error. More specifically this color indicates that a "strange" message is being read from the client. If this condition persists, contact the Alarmer expert.
GRAY: The indicator window is in standby mode because the PAUSE button has been pressed. To unpause, press the PAUSE button again.
When a client reports an error, the associated indicator window turns red and an alarm sounds. It will continue to sound until the client stops reporting an error AND the error has been acknowledged by the shift crew by pressing the INFO button. If INFO is not pressed, the alarm will continue even if the client stops reporting the error.
When a client times out or a connection to a client is broken, the associated indicator window turns orange and an alarm sounds. The alarm will continue to sound until the shift crew acknowledges the timeout by pressing the INFO button. After that, the alarm will stop, but the indicator window will remain orange until the client successfully reconnects. If this condition persists, check the client to see if it is sending to the Central Alarmer.
It is very likely that an alarm will sound when nothing is red or orange. The Central Alarmer traps errors in such a way that they will remain in the Alarmer even if the condition causing the error has ended and the indicator window has turned green again. The alarm will continue until the INFO button has been pressed and the error message acknowledged by the shift crew. To figure out which indicator window is causing an alarm, look at the little LED indicator in the lower right corner of the window. If it is red, that’s where the alarm is coming from.
First of all, NEVER NEVER NEVER TURN OFF THE SPEAKERS. This is an excellent way to make sure that any other error conditions that may arise are missed completely. If there is a persistent error that the shift crew knows about and is dealing with, the alarm may be silenced by clicking the PAUSE button. Note that the alarm will only stop after the INFO button is pressed, so just suspending the indicator window is not enough to make the alarm stop.
If you know that you are going to have to take one of the clients down for a period of time, press the PAUSE button for that client and wait for the window to turn gray. You can now do whatever you like with the client program without the Central Alarmer getting upset. Once you are done fiddling with the client and it is back up and sending to the Central Alarmer, click the PAUSE button on that client again to resume it.
NOTE: This information applies ONLY to clients running on LabView. For a LINUX based client, the shift crew is on its own.
First, go to the monitor indicated in the error dialog box. Check to see if the monitoring software and AlarmSender are running by bringing up their front panel displays and looking at the little arrow in the upper left of the display. If the arrow is black, then the program is running; if it is white, then the program is not running. Click on any white arrows to start the program; this should turn them black. If all of the arrows are black but the Central Alarmer still shows orange, something is wrong with the AlarmSender.
To restart the AlarmSender, click on the two buttons in the red area labeled "Push to Stop!" if the sender is still running. This should stop the sender, turning the arrow at the upper left to white. Once the AlarmSender is stopped, close it, leaving the monitoring software running. Go to the desktop, find the "AlarmSender_V1.2.llb" icon, and double-click on it. Eithe the AlarmSender front panel will open immidiately, or a file list will appear. If the latter happens, choose "AlarmSender.vi" to bring up the AlarmSender front panel. Click on the white arrow in the upper left of the LabView window to start the AlarmSender.
It might also be the case that LabView has stopped working altogether for whatever reason. If this is so, both the monitoring software and the AlarmSender will need to be restarted. First, try to stop all of the running software using the stop buttons on their front panels, NOT the button with the red stop sign at the upper left of the LabView window. Once the software is stopped, close all of the windows and exit from LabView. If LabView is unresponsive so that closing it the conventional way is not possible, right-click on a clear space in the Windows taskbar at the bottom of the screen. Select "Task Manager." Select the application you want to quit (i.e. LabView) and click on "End Task" at the bottom of the window. Select things like "Quit" and "End Task" in the dialog boxes that come up, and LabView should quit. Once this is done, restart the monitoring software by double-clicking on the appropriate icon on the desktop. Note that for PlasMon and RadMon there are two different VI’s that must be started. Run the software by clicking on the white arrow in the upper left of the LabView window. Only after the monitoring software is started, start the AlarmSender as indicated in the preceding paragraph.
If the AlarmSender still cannot connect to the Central Alarmer, repeat this process
once or twice and make a note of it in the logbook. Note that it might take a few minutes
for the AlarmSender to connect, especially if the "wait time" and
"heartbeat" times have been set high. If all else fails, call the AlarmSender
expert.
NOTE: This information applies ONLY to clients running on LabView. In principle, this error should not arrise on LINUX based clients.
This error indicates that the AlarmSender is sending just fine, but that there may be something wrong with the monitoring software. This may be a temporary problem as the AlarmSender may miss a heartbeat read; if this is the case then the error will clear itself. If it persists, something is wrong with the monitoring software. First, go to the monitor indicated in the dialog box. Check to see if the monitoring software is running by bringing up its front panel display and looking at the little arrow in the upper left of the display. If the arrow is black, then the program is running; if it is white, then the program is not running. Click on a white arrow to start the program; this should turn it black. If it starts correctly, the error message should clear.
If the monitoring software was running properly to begin with, the timing needs to be changed between heartbeat reads. Bring up the AlarmSender front panel. In the blue box, find the control labeled "heartbeat." Increase it by five seconds, noting that the display is in MILLISECONDS, so you will need to increase it by 5000. If this does not clear the error after a minute or so, increase it again. Note these changes in the logbook. If the error still does not go away, there is one last thing to check.
Select the "Windows" menu at the top of the AlarmSender’s LabView window, and choose "Show Diagram." Look for an icon with a picture of a little globe in it; any one of them will do. Right-click on it and select "Open Front Panel" from the menu; it should be the second-to-last option. This will bring up the AlarmSender Global front panel. Loot at the two indicators in the purple box; they should be labeled "Number of Iterations" and "Number of Iterations 2." For PlasMon and RadMon, both of these numbers should be increasing; for SiMon and MagMon only the first should be increasing. Note that for some of the monitoring software it will take 20 or 30 seconds for the number to increment, so wait a little while before deciding if it is or isn’t. If neither is increasing, but the monitoring software is definitely running, something strange is going on. Call the AlarmSender expert. If the number(s) is(are) increasing, pick the slowest if there are two of them running and time how long on average it takes to increment. Add ten seconds or so to this number and use this as the value for "heartbeat" on the AlarmSender front panel, remembering to convert to milliseconds. Be sure to click in the gray area of the front panel after changing this value to make sure the change takes effect. If after a minute or so this still does not clear the error, make a note in the logbook and call the AlarmSender expert.
There are cases in which the Central Alarmer gets stuck in some sort of a funny state in which it keeps on alarming even though there is nothing wrong with any of the clients at all. If this appears to be the case, the best solution is to stop the Alarmer and restart it. Stop the Central Alarmer by pressing the rectangular STOP button on the left side of the screen, just to the left of the SiMon indicator window. You will be able to tell that it has stopped by looking at the arrow in the upper left part of the LabView window. When this arrow turns from black to white the Alarmer has stopped. Click on the white arrow to start it again. This completely restarts the Alarmer side of the system, clearing out the funny state. All of the indicator windows should turn yellow to indicate that they are reinitializing, and then they should turn whatever color is appropriate (green, red, orange, or still yellow) given the state of the clients. After this is done, make a note of it in the logbook.
Currently, the only information worth documenting for future Central Alarmer experts is that several of the ports (specifically 15:SiCalib and 20:Is There Beam?) are special in that their Port Readers, Talk Modules, and Status Changers are not the same as those of the rest of the ports or each other. This means that if an update is made to one port that needs to be propagated to all of them, this MUST BE DONE MANUALLY on ports 15 and 20. DO NOT simply copy over what's already there, or you will loose the special changes. The special ports have also been noted with labels in the circuit diagram of the Central Alarmer front panel.
Please send all comments and suggestions regarding this page to Gerrit. |