Run Commands

Make sure have launched an instance and the instance is running before you run anything in that instance in the Server Machine.

singularity instance list

Precaution

WARNING: By default, tensorflow will try to access the total memory in as many GPUs as possible. With a precaution that this might lead to crashing existing scripts using the GPU, expose only the ones you want to use.

With the warning in mind, allot GPUs that can be used using a resource scheduler like SLURM.

TIP: If you do not use any managers, please set CUDA_VISIBLE_DEVICES to whichever GPU you want to expose to Tensorflow.

export CUDA_VISIBLE_DEVICES=0,1
# will expose only GPUs 0 and 1

If you want to automate this process specifying only number of GPUs and minimum memory, you can use a simple script or use apps like mask-gpu

Run Python Console

To run an interactive python console

singularity exec instance://my_instance python   

Run Shell

To run the shell

singularity shell instance://my_instance

Run Scripts

To run a long running script,

singularity shell instance://my_instance                      # Singularity Prompt: Singularity tf2.sif:~> 
tmux                                                          # Tmux Window: name@server:~$
export CUDA_VISIBLE_DEVICES=0,1
python some_folder/some_long_python_script.py
Ctrl+B Ctrl+D                                                 # Singularity Prompt: Singularity tf2.sif:~> 
exit                                                          # Server Prompt: name@server:~$

To check progress/output,

singularity shell instance://my_instance                      # Singularity Prompt: Singularity tf2.sif:~> 
tmux ls                                                       # Lists all tmux windows
tmux a -t <window_number>                                     # Tmux Window: name@server:~$
# do stuff
# Ctrl+B [ to start scrolling
# Page-Up/Down, Up/Down/Left/Right to navigate
# q to exit scrolling mode
Ctrl+B Ctrl+D to exit tmux                                    # Singularity Prompt: Singularity tf2.sif:~> 
exit                                                          # Server Prompt: name@server:~$