Welcome to lst_auto_rta documentation!¶

lst_auto_rta¶

Description¶

Automation of RTA for the LST

Crontab example

SHELL=/bin/bash
BASH_ENV=~/.bashrc

00 18 * * * conda activate /dev/shm/latest_rta_LST_Prod_EVBv6_test_dev_shm; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA_EVB.py
#00 18 * * * conda activate latest_rta_LST_Prod_EVBv6; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA_EVB.py
#00 18 * * * conda activate latest_rta_LST_Prod; cd /fefs/onsite/pipeline/rta/data; ./Auto_RTA.py
00 18 * * * conda activate latest_rta_calib; cd /fefs/onsite/pipeline/rta/data; ./Auto_Check.py
00 18 * * * conda activate latest_rta_calib; cd /fefs/onsite/pipeline/rta/data; ./Auto_Calib.py
00 18 * * * conda activate gammapy-1.1; cd /fefs/onsite/pipeline/rta/data; ./High_Level_analysis.py
00 18 * * * conda activate gammapy-0.19; cd /fefs/onsite/pipeline/rta/data; ./High_Level_analysis_Maps.py

# Disabled new lst_auto_rta tests
#00 18 * * * conda activate LST_AUTO_RTA_Dev_CDB_2024_05_06; cd /fefs/onsite/pipeline/rta/sag_reco_auto_rta/configuration/; lst_auto_rta -c auto_rta_config_2024_05_06.json

# Example VS code dev containers config:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/miniconda
{
  "initializeCommand": "docker pull gitlab-registry.in2p3.fr/cta-lapp/rta/lst_auto_rta/dev:latest",
     "image": "gitlab-registry.in2p3.fr/cta-lapp/rta/lst_auto_rta/dev:latest",
     "postAttachCommand": "micromamba run -n base pip install --no-deps -e /src",
     "customizations": {
             // Configure properties specific to VS Code.
             "vscode": {
                     // Add the IDs of extensions you want installed when the container is created.
                     "extensions": [
                             "charliermarsh.ruff",
                             "ms-python.isort",
                             "ms-python.python",
                             "ms-python.vscode-pylance",
                             "njpwerner.autodocstring",
                             "tamasfe.even-better-toml",
                             "wmaurer.change-case"
                     ]
             }
     },
     "workspaceMount": "source=${localWorkspaceFolder},target=/src,type=bind",
     "workspaceFolder": "/src"
}

Notes¶

LST AUTO RTA can run in a separate conda environment from RTA scripts (it just submits jobs to slurm). To create a production environment, you can use conda create -n lst_rta_prod_2024_06_05 -f docker/dependencies.yml. Do not forget the ``-n env_name`` argument otherwise it will be installed in ``base`` (don’t want that)

For now, there is only a development environment where it is installed in edit mode: For lstrta@tcs06: conda activate LST_AUTO_RTA_Dev_CDB_2024_05_06
lst_auto_rta is installed in edit mode from /fefs/onsite/pipeline/rta/sag_reco_auto_rta/lst_auto_rta

### RECO environment: The RTA processing scripts are using a single conda environment. For start-up performances on fefs reasons, the environment must be available in the RTA (/dev/shm) of the slurm worker nodes. Therefore, this environment must be:

created in tcs06, using a prefix, eg: --prefix=/dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06. Full environment creation:
- conda create -y --prefix /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 -c conda-forge -c nodefaults "python=3.8" "cython=0.29.28" "cmake=3.25.2" "make=4.3" "gcc_linux-64=12.2" "gxx_linux-64=12.2" "pytables=3.7.0" "blosc=1.21.1" "cppzmq=4.8.1" "sysroot_linux-64=2.17" "tbb==2021.11.0" "tbb-devel==2021.11.0"
- conda activate /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06
- ./install.sh in HiPeRTA cloned repository
- ./install.sh in PhoenixDebugZMQ cloned repository
- pip install git+https://github.com/cta-observatory/ctapipe@v0.8.0
- pip install git+https://github.com/cta-observatory/ctapipe_io_lst@v0.5.3
- pip install git+https://github.com/cta-observatory/ctapipe-extra@v0.3.1
- pip install git+https://github.com/garciagenrique/cta-lstchain@v0.6.3+RTAv4.2.2
- pip install https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeAltAzServer/-/archive/master/HiPeAltAzServer-master.zip
- pip install -e https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/archive/master/HiPeRTA_Stream-master.zip
create archive of environment on fefs, so it is accessible by worker nodes for copy: tar -czf RTA_Dev_EVBv6_CDB_shm_2024_05_06.tar.gz -C /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06/ . Pay attention to the fact that tar keeps paths relative to the working directory, so the -C makes the paths of files in the archive relative to the conda environment folder. The extraction will expect the archive to be set this way.
lst_auto_rta will: untar the environment from fefs to a specified path: tar -xzf {} -C {} if copy_env is True in config. The paths are provided in the configuration as env_archive_extraction_path and should be set to /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 to match the initial environment place in the /dev/shm of tcs06.
During the night, lst_auto_rta will start hiperta_stream_start as a slurm job on the worker nodes with the command export PATH={}/bin/:$PATH ; {}/bin/hiperta_stream_start where {} is filled with /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 so that hiperta_stream_start will run from the conda environment in the worker nodes /dev/shm.
- Then hiperta_stream_start will submit jobs to slurm for r0_dl1 daemons. These jobs inherit the environment of the submitter, therefore r0_dl1 daemons will try to use environment in /dev/shm as well. Since it was copied there on the worker nodes at the start of the night, it works…

Debugging¶

Start lst_auto_rta: In /fefs/onsite/pipeline/rta/sag_reco_auto_rta/configuration run conda activate LST_AUTO_RTA_Dev_CDB_2024_05_06, then lst_auto_rta -c auto_rta_config_2024_05_06.json
Start a fake stream with Phoenix Debug zmq: In /fefs/onsite/pipeline/rta/data/StreamEVBR0inR1: conda activate /dev/shm/RTA_Dev_EVBv6_CDB_shm_2024_05_06 then phoenix_debug_zmq_server -i dump_0.pzmq -p 25000 -f 10 -l
Watch for data, logs or config in night’s directory, eg /fefs/onsite/pipeline/rta/data/2024/05/17

lst_auto_rta