Repository navigation
Singularity as executable provider #863
Description
Activity
Hey @vsoch!
We actually have modifiers to make workflows around using containers easier. See the
apptainermodifier here: https://github.com/GoogleCloudPlatform/ramble/blob/develop/var/ramble/repos/builtin/modifiers/apptainer/modifier.pyAs an example, if you wanted to run NeMo with containers, you could use something like the following config:
ramble: variants: package_manager: spack modifiers: - name: apptainer env_vars: set: OMP_NUM_THREADS: '{n_threads}' variables: mpi_command: '' container_uri: docker://nvcr.io/nvidia/nemo:{nemo_version} container_name: nemo-{nemo_version} batch_submit: '{execute_experiment}' processes_per_node: 1 gpus_per_node: 1 apptainer_run_args: --nv --bind {container_mounts} --writable-tmpfs applications: py-nemo: workloads: pretraining: experiments: test: variables: env_name: apptainer nemo_stage: training nemo_model: gpt3 nemo_config_name: 5b n_nodes: 1 nemo_launcher_tag: 24.07 nemo_version: 24.07 exp_manager.checkpoint_callback_params.model_parallel_size: 1 max_steps: 5 model.optim.grad_sync_dtype: bf16 model.data.data_impl: mock model.data.data_prefix: [] trainer.max_steps: '{max_steps}' trainer.val_check_interval: '{max_steps}' trainer.limit_val_batches: '0.0' exp_manager.create_checkpoint_callback: False model.optim.name: distributed_fused_adam exp_manager.exp_dir: '{experiment_run_dir}' software: packages: apptainer: pkg_spec: apptainer environments: py-nemo: packages: - apptainerHere, spack is only used to manage the apptainer install. If apptainer is installed outside of spack, and always available in your path then you can remove the software definitions and the
varitantsblock.- locked and limited conversation to collaborators
on Feb 3, 2025
Hi @douglasjacobsen! It looks like Singularity is hiding in ramble as (what I'm guessing is) a spack containerize output? I'm wondering if we might consider having a container provider that knows how to provision a container. E.g., right now I'm going to provide a container binary directly to internals->custom executables, but I wonder if there would be an easy way to define a URI alongside the config, and then have ramble check for Singularity and do the pull of the binary (just to the Singularity cache would be fine) if it doesn't exist.
I'm good just pulling on my own for now and then defining as a custom executable, but wanted to get your thoughts. For some context, I'm working with Olga to see if we can add some apps from our performance study, all of which were run with containers (not spack). Thanks, and happy Monday!