Getting started =============== Drasi is a data transport pipeline for data acquired from readout electronics. Get drasi --------- Pull the source code from the repository:: git clone https://git.chalmers.se/expsubphys/drasi.git Suggested packages: ``git make gcc bison flex perl tmux libncurses-dev``. Compilation ----------- And build it (on the machine / system where you want to run it):: cd drasi make At this point, the system can be easily seen in action by :ref:`tested without hardware using a script`. Note: in case the target platform does not have a compiler natively, drasi can be :ref:`cross-compiled `. .. _stepbystep: Step-by-step without hardware ----------------------------- Drasi is a networked program, that in a typical configuration consists of several parts: - Readout program (on each readout node). - Event builder (optional, but recommended). - Log message writer. - Monitoring programs (optional, (several) as wanted by the user). - Monitor dump to time-series graphing (e.g. `Grafana `_) (optional; not shown here). - Command line control program. To maintain the overview while following this guide, it is suggested to create a new terminal window to start each of the following programs. These tests all run on the same machine, thus ``localhost`` will be used as hostname for all commands. All programs can be aborted by pressing ``Ctrl-C``. In the following, some environment variables are used to specify the readout and event builder nodes. This is not necessary, but makes these examples more descriptive. To prepare the environment in any new sh-based shell:: export READOUTNODE=localhost:23000 export EBNODE=localhost For a csh-based shell:: setenv READOUTNODE localhost:23000 setenv EBNODE localhost For the guide, the drasi source and compiled binaries are assumed to be in a directory ``drasi/``, i.e. after the compilation above, do:: cd .. Message logging ............... In order to run any drasi node (readout or event-builder), an observing message logger process must be active on a system that can reach the respective nodes over the network. To start such a process, run:: drasi/bin/lwrocmon --log $READOUTNODE $EBNODE This will start a logging process that monitors two drasi nodes. Note: it will print messages about not being able to connect when the programs to monitor are not (yet) running. The logs are written to a text-file ``lwlog.l``. Monitoring log messages ....................... A helper program to format the log messages nicely exist; run in another terminal:: tail -f lwlog.l | drasi/bin/lwroclog To see more of the messages, add :option:`--info <--info|--log|--debug|--spam>`, :option:`--log <--info|--log|--debug|--spam>`, or :option:`--debug <--info|--log|--debug|--spam>`. Monitoring tree view .................... To monitor the readout and event-building processes that will be started in the next steps, run:: drasi/bin/lwrocmon --tree $READOUTNODE $EBNODE While the tree view monitor is able to discover additional nodes through inspecting their reported data connections, for simplicity both ``host[:port]`` specifications are given here. Readout process ............... To test without hardware, start a readout process which just generates data and does no actual readout or wait for any triggers:: drasi/bin/testdaq \ --buf=size=100M \ --port=23000 \ --server=trans \ --file-writer (Since the guide later also includes an event-builder running on the same (local) machine, the readout process is here told to handle network connections on a non-default port (23000).) When the readout started, some log messages should have been printed in the log message monitor, and the tree view should show some data rates and buffer fill factors. Check that! The collected data can be picked up by connecting any system that reads the transport protocol, e.g. a `UCESB `_ instance (in directory ``$UCESBDIR``):: cd $UCESBDIR empty/empty trans://`echo $READOUTNODE | sed -e "s/:.*//"` ``empty/empty`` above is the chosen generic unpacker, which just process and verify the event packaging, not the data itself. (The ``sed`` trickery is to remove the custom port number used for the readout node in this guide.) Add ``--print`` to print the event headers, and also ``--data`` to print the full data. .. _filewriting: File writing ............ Any drasi instance can also write a data stream. This is usually enabled on the last event builder / time sorter in the chain, by adding ``--file-writer`` to the command line, as done above. Opening and closing files is then done with the control program:: export FILEWRITERNODE=$READOUTNODE # for sh shell setenv FILEWRITERNODE $READOUTNODE # for csh shell drasi/bin/lwrocctrl $FILEWRITERNODE --file-open=auto=1G,FILENAME.LMD drasi/bin/lwrocctrl $FILEWRITERNODE --file-close Event builder ............. To send data to an event builder from a drasi node, the connection has to be specified once on the sending side, and once on the receiving side. The commands below expect the hostname of the drasi readout node to be ``$READOUTNODE`` and the the event builder to be ``$EBNODE``. Stop (``Ctrl-C``) the readout process, and restart it with these options:: drasi/bin/testdaq \ --buf=size=100M \ --port=23000 \ --server=drasi,dest=$EBNODE In a new terminal, start an event-builder:: drasi/bin/lwrocmerge \ --buf=size=100M \ --drasi=$READOUTNODE \ --merge-mode=event \ --merge-no-validate \ --server=trans \ --file-writer Again, a UCESB or another online analysis tool can be used to read the collected data, now from the event builder, which also is responsible for file-writing (use ``lwrocctrl`` to communicate with the file writing system, but set ``FILEWRITER`` to ``$EBNODE``). (The option ``--merge-no-validate`` is used in this example as there is no trigger bus system to synchronise the startup.) A setup with an event builder generally leads to less workload for the readout node compared to the setup without a dedicated event builder (even in the case with only a single readout node), and is thus suggested. Readout rate monitoring ....................... To monitor the readout or event-building rate of one or more nodes, run:: drasi/bin/lwrocmon --rate $EBNODE drasi/bin/lwrocmon --rate $READOUTNODE Add your user code ------------------ Drasi can be used as a drop-in replacement for `MBS `_. In this case you can reuse your ``f_user.c`` file and compile with drasi instead of MBS binaries. On the other hand, you can also use your own native readout function (triggered or untriggered). drasi as MBS replacement ........................ Look in the ``f_user_example/`` directory. This contains a ``makefile`` (``makefile.drasi``) and ``f_user_example.c`` file. You can simply adjust the makefile.drasi to compile your own ``f_user.c``. This is the easiest way to implement your readout. Drasi also works nicely together with readout code written using the `nurdlib `_ library. Here, the ``f_user_example.c`` is compiled to ``f_user_example/-f_user_example``, which in the examples below is referred to as ``$READOUT``. If you have the ``drasi/bin/`` directory in your ``$PATH`` (such that ``drasi-config.sh`` can be found), then it might work by just copying ``makefile.drasi`` to the directory with your own ``f_user.c`` and run:: make -f makefile.drasi f_user drasi with readout from scratch ............................... If you need more control over how subevents are created and how and when the output data is written, have a look in the ``testdaq/`` example directory. This test program can also be used as an example showing how the readout functions of drasi can be defined without the structure imposed by the ``f_user`` file. This approach requires that you create and finalise your own subevents. If you do not want to care about such things, you should prefer the ``f_user`` style. Here, the ``testdaq`` executable is ``testdaq/bin_/testdaq``, but is automatically found by ``bin/testdaq`` as well. Run drasi with your own readout ------------------------------- Drasi can be run either in a single standalone node (VME crate or PC), or as a set of tightly connected nodes to achieve common deadtime (via trigger bus / ratatrig) or time synchronised (via WR / ratatime). Tightly connected nodes may send their collected data to a merger designated as event builder, which takes care of merging the input subevent data from one or more sources into a single output stream. Time-stamped synchronised events can be sorted by a merger instance. For the following examples, a readout node (hostname ``$READOUTNODE``) is assumed to have a drasi executable with the name ``$READOUT``. Message logging ............... Start (or restart if continuing from the :ref:`step-by-step guide` above) the message logger process with the hosts (``$READOUTNODE``, ``$EBNODE``) that will be used:: drasi/bin/lwrocmon --log $READOUTNODE $EBNODE The logs will continue to be appended to the text-file ``lwlog.l``. Monitoring log messages ....................... The message formatter does not need to be restarted. If none is running, or you want a second one, run:: tail -f lwlog.l | drasi/bin/lwroclog Single node operation with trigger module ......................................... Single node operation with a configured trigger module (TRIVA) is started like this:: $READOUT \ --buf=size=100M \ --triva=master \ --subev=subcrate=0,control=0 \ --server=trans \ --file-writer If you instead of a TRIVA are using the TRIMI functionality in a VULOM or TRIDI with the `TRLO II `_ firmware, use ``--trimi=`` instead of ``--triva=`` above. The data can now be picked up by e.g. `UCESB `_:: cd $UCESBDIR empty/empty trans://$READOUTNODE For full readout performance, an event builder should however be used. Single node operation with trigger module and event builder ........................................................... To send data to an event builder from a drasi node, the connection is specified on both the sending and receiving sides. Below, it is expected that the hostname of the drasi readout node is in ``$READOUTNODE`` and the the event builder is in ``$EBNODE``. Start the readout:: $READOUT \ --buf=size=100M \ --triva=master \ --subev=subcrate=0,control=0 \ --server=drasi,dest=$EBNODE \ --eb=$EBNODE Start the event builder:: drasi/bin/lwrocmerge \ --buf=size=100M \ --drasi=$READOUTNODE \ --eb-master=$READOUTNODE \ --merge-mode=event \ --server=trans \ --file-writer See :ref:`TCP port verification ` for a table of connections pairs used above. Then UCESB can be used to read the collected data from the event builder, which also is responsible for file-writing (use ``lwrocctrl`` to communicate with the file writing system to :ref:`open and close` files). This setup generally leads to less workload for the readout node compared to the setup without a dedicated event builder, in particular when data is also transmitted for online analysis, and is thus suggested. Troubleshooting --------------- Todo =) Additional topics ----------------- .. _cross_compile: Cross compilation ................. To cross-compile (i.e. to compile for a target platform on another system):: CROSS_COMPILE=- \ HOSTCC=cc \ CC=${CROSS_COMPILE}gcc \ LD=${CROSS_COMPILE}ld \ AR=${CROSS_COMPILE}ar \ make (```` might be e.g. ``x86_64-linux-gnu`` or ``powerpc-linux-gnu``.)