Chapter 7: Ordering compute power (the Slurm Workload Manager)

Now you have all the pieces of the puzzle: your data is uploaded (Chapter 4), your software Container is ready and you know how to use it (Chapter 5), and your AI code has been cloned from GitHub (Chapter 6).

But if you try to run your AI script on the Login Node right now, the system will stop it. To actually run your code on LUMI's powerful GPUs, you need to ask the system's supervisor for permission and resources. That supervisor is called Slurm.

What is Slurm and why do we need it?

LUMI is shared by thousands of users. If everyone clicked "Run" at the exact same time, the supercomputer would crash.

Slurm is a Workload Manager (often called a Job Scheduler). Think of Slurm like the manager of a massive, highly sought-after commercial kitchen.

1. You don't just walk into the kitchen and start cooking whenever you want.
2. Instead, you submit a booking form (a Batch Script) to the manager (Slurm). On this form, you specify exactly what equipment you need (e.g., 4 ovens for 2 hours) AND provide the step-by-step recipe (commands) you want executed.
3. The manager looks at the schedule. As soon as the exact equipment you requested becomes available, they execute your recipe and save the final dish (your output results) for you to pick up later.

The beauty of this system is that you do not need to sit at your computer waiting. Once you hand your ticket to Slurm, you can log out of LUMI, close your laptop, and go have dinner. LUMI will start running your Job as soon as enough free resources (GPUs, CPUs, etc.) are available for your Job. It will save all the outputs of the script into a text file for you to read later.

The anatomy of a Slurm ticket (Batch Script)

To talk to Slurm, we write a simple text file (ending in .sh). This file always starts with a standard script header (#!/bin/bash) and contains two main sections right after it:

1. Special instructions for Slurm: lines starting with #SBATCH that tell Slurm exactly what resources your Job is asking to book and use.
2. The actual Command Line commands: the specific instructions to load your environment, set necessary Environment Variables, and run e.g., a Python AI script from within a Container.

Here is what a simple AI script may look like:

#!/bin/bash#SBATCH --account=project_462xxxxxx      # Identifies your organisation's project#SBATCH --partition=small-g              # Choosing the Slurm Partition and the hardware Partition (-g for GPU)#SBATCH --nodes=1                        # Requesting 1 physical server node#SBATCH --ntasks-per-node=1              # Run one instance ("task") of the script at a time #SBATCH --cpus-per-task=14               # The number of CPU cores, in this case per task#SBATCH --gpus-per-node=2                # Requesting 2 AMD GCDs (1 full GPU) on that node#SBATCH --mem-per-gpu=60G                # 60GB of RAM per GPU#SBATCH --time=02:00:00                  # Asking for 2 hours of compute time​​# 1. Remove any previously loaded Modules and load the necessary onesmodule purgemodule use /appl/local/laifs/modulesmodule load lumi-aif-singularity-bindings​​# 2. Set necessary Environment VariablesMIOPEN_DIR=$(mktemp -d)export MIOPEN_CUSTOM_CACHE_DIR=$MIOPEN_DIR/cacheexport MIOPEN_USER_DB=$MIOPEN_DIR/config​​# 3. Run your Python AI script from inside the Containersrun singularity run /appl/local/laifs/containers/lumi-multitorch-latest.sif python3 run_ai.py

Handing the ticket to Slurm

LUMI AI Guide and examples of AI scripts usually contain this Slurm script and you only need to edit it with your actual project_ which will be "billed" for the Job. Once you've edited this file (let's call it run_ai.sh), you submit it to the queue using the sbatch command in your Terminal:

user@LUMI:~$ sbatch run_ai.sh

Exercise: Run your first AI Job on LUMI

First, let's create the Batch Script. Navigate to your project's /scratch directory and create a new file named my_first_slurm_script.sh (you can use nano like we did in Chapter 3). Copy and paste the following code, making sure to replace project_462xxxxxx with your actual project ID.

#!/bin/bash#SBATCH --account=project_462xxxxxx      # Identifies your organisation's project#SBATCH --partition=dev-g                # Choosing the Slurm Partition and the hardware Partition (-g for GPU)#SBATCH --nodes=1                        # Requesting 1 physical server node#SBATCH --ntasks-per-node=1              # Run one instance ("task") of the script at a time #SBATCH --cpus-per-task=14               # The number of CPU cores, in this case per task#SBATCH --gpus-per-node=2                # Requesting 2 AMD GCDs (1 full GPU) on that node#SBATCH --mem-per-gpu=60G                # 60GB of RAM per GPU#SBATCH --time=00:30:00                  # Asking for 30 minutes of compute time​​# 1. Remove any previously loaded Modules and load the necessary onesmodule purgemodule use /appl/local/laifs/modulesmodule load lumi-aif-singularity-bindings # Bindings give LUMI Containers access to the file system of the Working Directory​​# 2. Set necessary Environment VariablesMIOPEN_DIR=$(mktemp -d)export MIOPEN_CUSTOM_CACHE_DIR=$MIOPEN_DIR/cacheexport MIOPEN_USER_DB=$MIOPEN_DIR/config​# We use the PyTorch Container provided by the LUMI AI Factory Services, which contains vLLM - an engine library for running LLMs.# For this exercise we point to one exact dated Container, so the exercise works the same for every reader. For your own projects, using `lumi-multitorch-latest.sif` is a good default.export SIF=/appl/local/laifs/containers/lumi-multitorch-u24r70f21m50t210-20260415_130625/lumi-multitorch-full-u24r70f21m50t210-20260415_130625.sif​# Redirect all vLLM cache files from $HOME to scratch.export VLLM_CACHE_ROOT=/scratch/$SLURM_JOB_ACCOUNT/vllm-cache​# Where the models are downloaded (the "weights"). $SLURM_JOB_ACCOUNT automatically finds your project ID on LUMI. export HF_HOME=/scratch/$SLURM_JOB_ACCOUNT/hf-cache/​# Choose the LLM to run (from https://huggingface.co/ )export MODEL_NAME="Qwen/Qwen3.6-35B-A3B"​​# 3. Run the python script inside the Containersrun singularity run $SIF python3 generate_text.py

Like earlier, the Slurm script books resources, sets the Environment Variables necessary for the AI code, and starts a script called generate_text.py. For that to work, the Python script needs to exist in the exact same directory. Create a new file named generate_text.py and paste the following code into it:

import osfrom vllm import LLM, SamplingParams​def main():    # Read the variable we exported in the Slurm script    model_name = os.environ.get("MODEL_NAME")        print(f"Loading {model_name}...")    # Initialise the model (vLLM will automatically distribute it across our 2 requested GCDs)    # We use load_format="runai_streamer" to speed up weight loading using the Run:ai Model Streamer    llm = LLM(model=model_name, tensor_parallel_size=2, load_format="runai_streamer")        # Set up our prompt    prompts = ["What is the LUMI supercomputer?"]    sampling_params = SamplingParams(temperature=0.7, max_tokens=1000)        print("Generating text...")    outputs = llm.generate(prompts, sampling_params)        # Save the generated text to the project's scratch directory    with open("./my_first_ai_output.txt", "w") as f:        f.write(outputs[0].outputs[0].text)​if __name__ == "__main__":    main()

Step 1: Submit the Job

Finally, submit the script to the queue using sbatch:

user@LUMI:~$ sbatch my_first_slurm_script.sh

The Terminal will respond with something like: Submitted batch job 1234567. Congratulations, your Job is officially in the Slurm queue!

Step 2: Monitor the queue

You can check the status of your Job using the squeue command:

user@LUMI:~$ squeue --me

If your Job is running, you will see R under the ST (State) column. If it is waiting for resources, you will see PD (Pending).

Step 3: Check the live output

Unlike your laptop where text prints directly to your screen, Slurm captures everything your Python script "prints" and saves it to a log file in the same directory. This log file will be named slurm-<your_job_id>.out (e.g., slurm-1234567.out).

To watch the model weights load in real-time, you can "follow" the log file:

user@LUMI:~$ tail -f slurm-<your_job_id>.out

(Press Ctrl+C to stop watching the file. This doesn't stop the Job!)

Step 4: Review the final results

Once the Job finishes, it will disappear from squeue. If you are watching the live log with tail -f, you'll know it's done when the text stops flowing and you see a "Shutdown complete" message. You can then read the complete log file:

user@LUMI:~$ less slurm-<your_job_id>.out

More importantly, your Python script created a brand new file with the AI's answer! Let's read it:

user@LUMI:~$ less my_first_ai_output.txt

Interactive Jobs: Running code and AI models in real-time

Writing a script and waiting in a queue can be frustrating if you are just trying a new model or testing if it runs properly. For this, Slurm also offers Interactive Jobs.

Instead of submitting a Batch Script to run later, you ask Slurm to give you a live connection to a node right now.

To spin up a quick, 30-minute test environment on a GPU node, you use srun:

user@LUMI:~$ srun --partition=dev-g --nodes=1 --ntasks-per-node=1 --cpus-per-task=14 --gpus-per-node=2 --mem-per-gpu=60G --time=00:30:00 --account=project_462xxxxxx --pty bash

Once Slurm grants you the requested resources, your Command Line prompt will change from the Login Node (such as uan01) to the Compute Node nid002134 where all the code you run will use the GPUs and the rest of the hardware you requested. You can now run commands interactively.

Let's test it by checking our allocated GPUs. Run this command:

user@LUMI:~$ rocm-smi

This will print out the status of the 2 GCDs (GPUs) that Slurm just gave you! You can now run Python scripts, test code, or download models interactively.

When you are finished testing, always type exit or press Ctrl+D to close your session and release your allocated resources!

Learn more about Interactive Usage on LUMI (Optional)

How billing works on LUMI

Compute power on LUMI isn't billed in euros - it's billed in Billing Units (BUs). When your project was granted access to LUMI, it received a specific allocation of BUs. Every Job you run spends from that allocation.

BUs come in two currencies: GPU-hours (for LUMI-G) and CPU-hours (for LUMI-C). Since GPUs are the most valuable resource on LUMI, GPU-hours are significantly more expensive.

How to check your remaining Billing Units and storage quotas

• What you're billed for. You are billed for the resources that have been allocated to you, not the part of the resources you've actually used (such as 20% GPU utilisation). If your script reserves 4 GCDs but your code only utilises 1, you still pay for all 4 for the duration of your Job. However, if you request 2 hours of walltime but your Job finishes in 1 hour, the allocated resources are released and you are only billed for the 1 hour.

• Negative Billing Units. If your project runs out of Billing Units while a Job is running, the Job will be allowed to finish and your project may go into negative billing units. You will not be charged for this negative balance.

• GPU Billing Rates. In standard-g (full-node allocation): each LUMI-G node contains 4 MI250X GPUs (8 GCDs), so 1 node-hour equals 4 GPU-hours. GPU-hours are counted based on physical GPUs, 1 GCD = 0.5 GPU. For example, if you allocate 4 nodes and your Job runs for 24 hours:

In small-g and dev-g (per-GCD allocation, not full node): each GCD is billed at 0.5 GPU-hours per hour.

• CPU Billing Rates. For LUMI-C CPU nodes, the slice logic is different from GPU nodes. A CPU slice is 1 CPU core and 2 GB of RAM. You are billed based on whichever is higher: the number of cores requested or the amount of memory requested divided by 2 GB.

• Walltime is a hard limit: If you ask Slurm for 2 hours (#SBATCH --time=02:00:00), but your Job isn't complete when time runs out, the Job will be strictly terminated. Always give your Jobs a little bit of "buffer time" to ensure they complete cleanly!

To find out more about GPU, CPU and storage billing:

Read the official Breakdown of LUMI Billing Policies (Optional)

Job priority and practical advice

How priority works

It is important to reserve only what you need in terms of time and the size of the allocation (in slices). Slurm is not a simple first-come-first-serve queue. Priority is a combination of:

• The size of your Job and requested time (bigger or longer jobs typically queue for longer).
• How much you have run lately (your "fair share" - the more jobs you've submitted recently, the lower your current priority).
• How long the Job has been waiting in the queue (the longer a job waits, the higher its priority becomes).

Especially in the beginning, you should not have problems getting resources allocated reasonably fast.

Practical workflow advice

We highly suggest starting with partitions allocatable by resources (dev-g or small-g) and reserving multiples of slices (e.g., n × [1 GCD, 7 CPU cores, 60 GB of RAM]) to understand how your workload runs on LUMI-G.

While testing, you can check the GPU utilisation by running the rocm-smi command from within the Compute Node's shell (for example, during an Interactive Job). Once you are convinced that your workload efficiently utilises the GPUs and you actually require full nodes, you can confidently move your workloads to standard-g.

Summary checklist

• You understand how Slurm schedules Jobs.
• You can write a basic Slurm Batch Script.
• You know how to submit, monitor, and review Jobs.
• You understand how LUMI billing works.

Knowledge check