Skip to content

lst_auto_rta

pipeline status Latest release coverage report documentation lstautorta conda channel

Description

Automation of RTA for the LST

Crontab example

SHELL=/bin/bash
BASH_ENV=~/.bashrc

#### Pixi based deployment (after pixi global install -c conda-forge -c https://prefix.dev/lapp-lst-rta lstautorta) ####

00 18 * * * cd /path/to/observation/launch && lst_auto_rta


#### Legacy conda deployment ####

00 18 * * * conda activate /dev/shm/latest_rta_LST_Prod_EVBv6_test_dev_shm; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA_EVB.py
#00 18 * * * conda activate latest_rta_LST_Prod_EVBv6; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA_EVB.py
#00 18 * * * conda activate latest_rta_LST_Prod; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA.py
00 18 * * * conda activate latest_rta_calib; cd /fefs/onsite/pipeline/rta/data; ./Auto_Check.py
00 18 * * * conda activate latest_rta_calib; cd /fefs/onsite/pipeline/rta/data; ./Auto_Calib.py
00 18 * * * conda activate gammapy-1.1; cd /fefs/onsite/pipeline/rta/data; ./High_Level_analysis.py
00 18 * * * conda activate gammapy-0.19; cd /fefs/onsite/pipeline/rta/data; ./High_Level_analysis_Maps.py

# Disabled new lst_auto_rta tests
#00 18 * * * conda activate LST_AUTO_RTA_Dev_CDB_2024_05_06; cd /fefs/onsite/pipeline/rta/sag_reco_auto_rta/configuration/; lst_auto_rta -c auto_rta_config_2024_05_06.json

Notes

RECO environment:

The RTA processing scripts are using a single conda environment. For start-up performances on fefs reasons, the environment must be available in the RTA (/dev/shm) of the slurm worker nodes. Therefore, this environment must be:

  • created in tcs06, using a prefix, eg: --prefix=/dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06. Full environment creation:
  • conda create -y --prefix /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 -c conda-forge -c nodefaults "python=3.8" "cython=0.29.28" "cmake=3.25.2" "make=4.3" "gcc_linux-64=12.2" "gxx_linux-64=12.2" "pytables=3.7.0" "blosc=1.21.1" "cppzmq=4.8.1" "sysroot_linux-64=2.17" "tbb==2021.11.0" "tbb-devel==2021.11.0"
  • conda activate /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06
  • ./install.sh in HiPeRTA cloned repository
  • ./install.sh in PhoenixDebugZMQ cloned repository
  • pip install git+https://github.com/cta-observatory/ctapipe@v0.8.0
  • pip install git+https://github.com/cta-observatory/ctapipe_io_lst@v0.5.3
  • pip install git+https://github.com/cta-observatory/ctapipe-extra@v0.3.1
  • pip install git+https://github.com/garciagenrique/cta-lstchain@v0.6.3+RTAv4.2.2
  • pip install https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeAltAzServer/-/archive/master/HiPeAltAzServer-master.zip
  • pip install -e https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/archive/master/HiPeRTA_Stream-master.zip
  • create archive of environment on fefs, so it is accessible by worker nodes for copy: tar -czf RTA_Dev_EVBv6_CDB_shm_2024_05_06.tar.gz -C /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06/ . Pay attention to the fact that tar keeps paths relative to the working directory, so the -C makes the paths of files in the archive relative to the conda environment folder. The extraction will expect the archive to be set this way.
  • lst_auto_rta will: untar the environment from fefs to a specified path: tar -xzf {} -C {} if copy_env is True in config. The paths are provided in the configuration as env_archive_extraction_path and should be set to /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 to match the initial environment place in the /dev/shm of tcs06.
  • During the night, lst_auto_rta will start hiperta_stream_start as a slurm job on the worker nodes with the command export PATH={}/bin/:$PATH ; {}/bin/hiperta_stream_start where {} is filled with /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 so that hiperta_stream_start will run from the conda environment in the worker nodes /dev/shm.
  • Then hiperta_stream_start will submit jobs to slurm for r0_dl1 daemons. These jobs inherit the environment of the submitter, therefore r0_dl1 daemons will try to use environment in /dev/shm as well. Since it was copied there on the worker nodes at the start of the night, it works...

Debugging

  • Start lst_auto_rta: In /fefs/onsite/pipeline/rta/sag_reco_auto_rta/configuration run conda activate LST_AUTO_RTA_Dev_CDB_2024_05_06, then lst_auto_rta -c auto_rta_config_2024_05_06.json
  • Start a fake stream with Phoenix Debug zmq: In /fefs/onsite/pipeline/rta/data/StreamEVBR0inR1: conda activate /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 then phoenix_debug_zmq_server -i dump_0.pzmq -p 25000 -f 10 -l
  • Watch for data, logs or config in night's directory, eg /fefs/onsite/pipeline/rta/data/2024/05/17