Skip to main content
This page shows you how to configure W&B to resume a run that was paused, stopped, or crashed, so you can continue logging from where you left off instead of starting over. To specify how W&B responds if a run stops or crashes, set the resume parameter in wandb.init(). When you initialize a run, W&B checks whether the run ID already exists and applies the behavior defined by the resume value. The following table outlines the behavior of W&B based on the argument passed to the resume parameter and whether the run ID exists or not.
When to use auto compared to allow:W&B recommends that you use resume="allow" and specify the specific run ID you want to resume.The resume="auto" option doesn’t require you to specify a run ID, but it can lead to unexpected behavior if you have multiple runs that fail in the same directory or if the file directory structure changes. You must also ensure that you restart the run from the same directory as the failed process when you use resume="auto".
For all the following examples, replace values enclosed within [] with your own.
View a live demo of a resumed run.

​
Resume a run that must use the same run ID

If a run is stopped, crashes, or fails, you can resume it using the same run ID. To do so, initialize a run and specify the following:
  • Set the resume parameter to "must" (resume="must").
  • Provide the run ID of the run that stopped or crashed.
The following code snippet shows how to accomplish this with the W&B Python SDK:
Unexpected results occur if multiple processes use the same id concurrently.For more information about how to manage multiple processes, see Log distributed training experiments.

​
Resume a run without overriding the existing run

Resume a run that stopped or crashed without overriding the existing run. This is helpful if your process doesn’t exit successfully, because W&B falls back to starting a new run when the specified run ID doesn’t already exist. The next time you start W&B, W&B starts logging from the last step. Set the resume parameter to "allow" (resume="allow") when you initialize a run with W&B. Provide the run ID of the run that stopped or crashed. The following code snippet shows how to accomplish this with the W&B Python SDK:

​
Enable runs to automatically resume

Automatic resuming lets W&B pick up a failed run without requiring you to specify a run ID, which is useful for long-running training jobs that might restart unattended. The following code snippet shows how to enable runs to automatically resume with the Python SDK or with environment variables.
Pass auto as an argument to the resume parameter when you initialize a run. Ensure that you restart the run from the same directory as the failed process.The following code snippet enables automatic resuming. Replace values enclosed within [] with your own:
Automatic resuming only works if the process is restarted on top of the same filesystem as the failed process.
To illustrate why the same filesystem matters, suppose you execute a Python script called train.py in a directory called Users/Employee/Desktop/ImageClassify/training/. Within train.py, the script creates a run that enables automatic resuming. Next, suppose the training script stops. To resume this run, you must restart your train.py script within Users/Employee/Desktop/ImageClassify/training/.
If you can’t share a filesystem, specify the WANDB_RUN_ID environment variable or pass the run ID with the W&B Python SDK. See Create a custom run ID for more information about run IDs.

​
Resume preemptible Sweeps runs

Handle preemption signals so W&B can automatically requeue interrupted sweep runs for another agent. This pattern is useful when the sweep agent runs on preemptible compute, such as a SLURM preemptible queue, an Amazon EC2 Spot Instance, or a Google Cloud preemptible VM. The instructions below apply when you start sweep agents with the wandb agent CLI. The CLI starts your training program as a subprocess. The instructions do not fully apply when you use only the Python API wandb.agent(). The Python API runs the training function in a thread, so OS signal delivery and forwarding differ from the CLI agent behavior.

​
Handle a preemption signal

Register a handler for the signal that your scheduler or platform uses to indicate preemption, such as SIGUSR1 or SIGTERM. In the handler:
  1. Call mark_preempting() when a run is active.
  2. Perform any required cleanup, such as saving a checkpoint.
  3. Exit with a nonzero status code. A common convention for signal termination is 128 + signum.
Do not call mark_preempting() unconditionally immediately after wandb.init(). Doing so can mark every failure, including code bugs, as preemption and requeue the run repeatedly. For runnable examples, --forward-signals on the CLI agent, and a full reference table for different uses of mark_preempting(), see Signal handling and sweep runs. When you follow that pattern, W&B records run state roughly as follows:
When a sweep agent fetches a preempted run, the training process must call wandb.init() within 60 minutes. If initialization does not occur, such as when the process fails after fetching the run but before calling wandb.init(), W&B does not make the run available to another agent until the 60-minute lease expires.
Sweep agents process requeued runs before requesting new hyperparameter combinations from the sweep search algorithm. After the queue is empty, the sweep resumes normal scheduling.