Getting started¶

Drasi is a data transport pipeline for data acquired from readout electronics.

Get drasi¶

Pull the source code from the repository:

git clone https://git.chalmers.se/expsubphys/drasi.git

Suggested packages: git make gcc bison flex perl tmux libncurses-dev netcat-openbsd.

Compilation¶

And build it (on the machine / system where you want to run it):

cd drasi
make

At this point, the system can be easily seen in action by tested without hardware using a script.

Note: in case the target platform does not have a compiler natively, drasi can be cross-compiled.

Step-by-step without hardware¶

Drasi is a networked program, that in a typical configuration consists of several parts:

Readout program (on each readout node).
Event builder (optional, but recommended).
Log message writer.
Monitoring programs (optional, (several) as wanted by the user).
Monitor dump to time-series graphing (e.g. Grafana) (optional; not shown here).
Command line control program.

To maintain the overview while following this guide, it is suggested to create a new terminal window to start each of the following programs. These tests all run on the same machine, thus localhost will be used as hostname for all commands.

All programs can be aborted by pressing Ctrl-C.

In the following, some environment variables are used to specify the readout and event builder nodes. This is not necessary, but makes these examples more descriptive. To prepare the environment in any new sh-based shell:

export READOUTNODE=localhost:23000
export EBNODE=localhost

For a csh-based shell:

setenv READOUTNODE localhost:23000
setenv EBNODE localhost

For the guide, the drasi source and compiled binaries are assumed to be in a directory drasi/, i.e. after the compilation above, do:

cd ..

Message logging¶

In order to run any drasi node (readout or event-builder), an observing message logger process must be active on a system that can reach the respective nodes over the network. To start such a process, run:

drasi/bin/lwrocmon --log $READOUTNODE $EBNODE

This will start a logging process that monitors two drasi nodes. Note: it will print messages about not being able to connect when the programs to monitor are not (yet) running. The logs are written to a text-file lwlog.l.

Monitoring log messages¶

A helper program to format the log messages nicely exist; run in another terminal:

tail -f lwlog.l | drasi/bin/lwroclog

To see more of the messages, add --info, --log, or --debug.

Monitoring tree view¶

To monitor the readout and event-building processes that will be started in the next steps, run:

drasi/bin/lwrocmon --tree $READOUTNODE $EBNODE

While the tree view monitor is able to discover additional nodes through inspecting their reported data connections, for simplicity both host[:port] specifications are given here.

Readout process¶

To test without hardware, start a readout process which just generates data and does no actual readout or wait for any triggers:

drasi/bin/testdaq \
  --buf=size=100M \
  --port=23000 \
  --server=trans \
  --file-writer

(Since the guide later also includes an event-builder running on the same (local) machine, the readout process is here told to handle network connections on a non-default port (23000).)

When the readout started, some log messages should have been printed in the log message monitor, and the tree view should show some data rates and buffer fill factors. Check that!

The collected data can be picked up by connecting any system that reads the transport protocol, e.g. a UCESB instance (in directory $UCESBDIR):

cd $UCESBDIR
empty/empty trans://`echo $READOUTNODE | sed -e "s/:.*//"`

empty/empty above is the chosen generic unpacker, which just process and verify the event packaging, not the data itself. (The sed trickery is to remove the custom port number used for the readout node in this guide.) Add --print to print the event headers, and also --data to print the full data.

File writing¶

Any drasi instance can also write a data stream. This is usually enabled on the last event builder / time sorter in the chain, by adding --file-writer to the command line, as done above.

Opening and closing files is then done with the control program:

export FILEWRITERNODE=$READOUTNODE   # for sh shell
setenv FILEWRITERNODE $READOUTNODE   # for csh shell

drasi/bin/lwrocctrl $FILEWRITERNODE --file-open=auto=1G,FILENAME.LMD

drasi/bin/lwrocctrl $FILEWRITERNODE --file-close

Event builder¶

To send data to an event builder from a drasi node, the connection has to be specified once on the sending side, and once on the receiving side. The commands below expect the hostname of the drasi readout node to be $READOUTNODE and the the event builder to be $EBNODE.

Stop (Ctrl-C) the readout process, and restart it with these options:

drasi/bin/testdaq \
  --buf=size=100M \
  --port=23000 \
  --server=drasi,dest=$EBNODE

In a new terminal, start an event-builder:

drasi/bin/lwrocmerge \
  --buf=size=100M \
  --drasi=$READOUTNODE \
  --merge-mode=event \
  --merge-no-validate \
  --server=trans \
  --file-writer

Again, a UCESB or another online analysis tool can be used to read the collected data, now from the event builder, which also is responsible for file-writing (use lwrocctrl to communicate with the file writing system, but set FILEWRITER to $EBNODE).

(The option --merge-no-validate is used in this example as there is no trigger bus system to synchronise the startup.)

A setup with an event builder generally leads to less workload for the readout node compared to the setup without a dedicated event builder (even in the case with only a single readout node), and is thus suggested.

Readout rate monitoring¶

To monitor the readout or event-building rate of one or more nodes, run:

drasi/bin/lwrocmon --rate $EBNODE

drasi/bin/lwrocmon --rate $READOUTNODE

Add your user code¶

Drasi can be used as a drop-in replacement for MBS. In this case you can reuse your f_user.c file and compile with drasi instead of MBS binaries. On the other hand, you can also use your own native readout function (triggered or untriggered).

drasi as MBS replacement¶

Look in the f_user_example/ directory. This contains a makefile (makefile.drasi) and f_user_example.c file. You can simply adjust the makefile.drasi to compile your own f_user.c. This is the easiest way to implement your readout. Drasi also works nicely together with readout code written using the nurdlib library.

Here, the f_user_example.c is compiled to f_user_example/<architecture-triple>-f_user_example, which in the examples below is referred to as $READOUT.

If you have the drasi/bin/ directory in your $PATH (such that drasi-config.sh can be found), then it might work by just copying makefile.drasi to the directory with your own f_user.c and run:

make -f makefile.drasi f_user

drasi with readout from scratch¶

If you need more control over how subevents are created and how and when the output data is written, have a look in the testdaq/ example directory. This test program can also be used as an example showing how the readout functions of drasi can be defined without the structure imposed by the f_user file. This approach requires that you create and finalise your own subevents. If you do not want to care about such things, you should prefer the f_user style.

Here, the testdaq executable is testdaq/bin_<architecture-triple>/testdaq, but is automatically found by bin/testdaq as well.

Run drasi with your own readout¶

Drasi can be run either in a single standalone node (VME crate or PC), or as a set of tightly connected nodes to achieve common deadtime (via trigger bus / ratatrig) or time synchronised (via WR / ratatime). Tightly connected nodes may send their collected data to a merger designated as event builder, which takes care of merging the input subevent data from one or more sources into a single output stream. Time-stamped synchronised events can be sorted by a merger instance.

For the following examples, a readout node (hostname $READOUTNODE) is assumed to have a drasi executable with the name $READOUT.

Message logging¶

Start (or restart if continuing from the step-by-step guide above) the message logger process with the hosts ($READOUTNODE, $EBNODE) that will be used:

drasi/bin/lwrocmon --log $READOUTNODE $EBNODE

The logs will continue to be appended to the text-file lwlog.l.

Monitoring log messages¶

The message formatter does not need to be restarted. If none is running, or you want a second one, run:

tail -f lwlog.l | drasi/bin/lwroclog

Single node operation with trigger module¶

Single node operation with a configured trigger module (TRIVA) is started like this:

$READOUT \
  --buf=size=100M \
  --triva=master \
  --subev=subcrate=0,control=0 \
  --server=trans \
  --file-writer

If you instead of a TRIVA are using the TRIMI functionality in a VULOM or TRIDI with the TRLO II firmware, use --trimi= instead of --triva= above.

The data can now be picked up by e.g. UCESB:

cd $UCESBDIR
empty/empty trans://$READOUTNODE

For full readout performance, an event builder should however be used.

Single node operation with trigger module and event builder¶

To send data to an event builder from a drasi node, the connection is specified on both the sending and receiving sides. Below, it is expected that the hostname of the drasi readout node is in $READOUTNODE and the the event builder is in $EBNODE.

Start the readout:

$READOUT \
  --buf=size=100M \
  --triva=master \
  --subev=subcrate=0,control=0 \
  --server=drasi,dest=$EBNODE \
  --eb=$EBNODE

Start the event builder:

drasi/bin/lwrocmerge \
  --buf=size=100M \
  --drasi=$READOUTNODE \
  --eb-master=$READOUTNODE \
  --merge-mode=event \
  --server=trans \
  --file-writer

See TCP port verification for a table of connections pairs used above.

Then UCESB can be used to read the collected data from the event builder, which also is responsible for file-writing (use lwrocctrl to communicate with the file writing system to open and close files). This setup generally leads to less workload for the readout node compared to the setup without a dedicated event builder, in particular when data is also transmitted for online analysis, and is thus suggested.

Troubleshooting¶

Todo =)

Additional topics¶

Cross compilation¶

To cross-compile (i.e. to compile for a target platform on another system):

CROSS_COMPILE=<architecture-triple>- \
HOSTCC=cc \
CC=${CROSS_COMPILE}gcc \
LD=${CROSS_COMPILE}ld \
AR=${CROSS_COMPILE}ar \
make

(<architecture-triple> might be e.g. x86_64-linux-gnu or powerpc-linux-gnu.)