Logging

Data acquisition systems consist of many connected parts, with many possible error-modes that have timing-sensitive behaviours, which are hard to reproduce. When something is not working, it is therefore important to have logfiles of what happens, to allow debugging issues even when they occur seemingly randomly.

The drasi readout and merger processes report both normal actions, as well as more or less severe errors through a log mechanism. Log messages are recorded to file over the network by a monitor process.

Each log messages has a severity level:

  • Bug — an internal condition which should not be possible.
  • Fatal — an irrecoverable error. After having produced this or a bug message, the generating thread will sleep indefinitely to allow the user to attach a debugger for further situation diagnosis.
  • Badcfg — a command-line configuration cannot be handled. The program will exit immediately. This is only used during start-up.
  • Error — a severe fault which requires some kind of (automatic) recovery action to be taken. This would either be due to a hardware fault, some external process disconnecting or malformed data. Even if recovery is handled automatically, the cause of the error should be investigated, since recovery often discards recorded event data and takes time.
  • Warning — a not-yet-critical situation.
  • Action — a more definite informational message about an action, e.g. acquisition start or file open/close.
  • Info — an informational message.
  • Log — a lesser informational message, that usually would not be interesting.
  • Debug — an even lesser message. These are enabled using the --debug=MASK command line option, or with lwrocctrl --debug=MASK.

Recording

In order to ensure that log messages are recorded, readout and merging processes will not start actual operations until a log consumer has been connected. (They will however not stop if the consumer is disconnected.) Multiple consumers can be connected—they will then all receive all messages (if reading fast enough). As long as at least one consumer is connected, each log message will be delivered to at least one consumer.

Messages are recorded by lwrocmon:

bin/lwrocmon --log <hostname_1> [<hostname_2> ...]

Note that each individual node has to be specified as a source. The logs are written to a text-file ‘lwlog.l’.

(It is intended to in the future have log messages forwarding capability in the merger processes, automatically set up to follow the data flow, such that recording only need to be pointed to a final event builder or time sorter.)

Viewing

While the log-files are readable text-files, they become easier to view when pretty-formatted and colourised by the lwroclog program:

tail -f lwlog.l | $DRASI/bin/lwroclog

The severity of messages to see and some formatting details can be controlled by command-line options.

lwroclog can also be used after selecting messages using e.g. grep. To keep the colour-coding when viewing with less, use the -R option, e.g.:

grep SOMETHING lwlog.l | bin/lwroclog | less -R

Format

The messages are written to text-files, as one line per message, with spaces separating the fields. Some less-important fields are at the beginning of the line, followed by a carrige-return character. This is such that those fields would not be seen when the file is simply printed on the terminal (using e.g. cat or grep).