New Article

One API, Many Training Worlds

By Mahmoud Zalt

We are examining how Hugging Face Accelerate keeps distributed training behind one API. Accelerate lets PyTorch training code run across ordinary execution and distributed backends such as DDP, FSDP, DeepSpeed, Megatron-LM, tensor parallelism, context parallelism, mixed precision, and several hardware runtimes. Its roughly 2,200-line accelerator.py centers on Accelerator, the application-facing class that prepares training objects and coordinates training, synchronization, and checkpoints.

That convenience is not merely a wrapper around APIs. It is a lifecycle contract: objects must be transformed in the right order, references must follow replacements, and invalid backend combinations must fail early. I’m Mahmoud Zalt, an AI solutions architect, and we will trace that contract from prepare() to production operations. The primary lesson is simple: a stable distributed-training façade succeeds only when its hidden lifecycle invariants are explicit, protected, and observable.

One Control Surface

Accelerator is a façade: one interface in front of multiple subsystems. Callers pass models, optimizers, dataloaders, and schedulers to prepare(). The class validates their combination, selects a backend path, wraps or transforms the objects, records them, and later coordinates backward passes, accumulation, and state persistence.

Caller configuration
        |
        v
  Accelerator.__init__
        |
        +-- state and plugins
        |
        v
  Accelerator.prepare()
        |
        +-- generic adapters
        +-- DeepSpeed engine preparation
        +-- Megatron-LM preparation
        +-- FSDP2 sharding and parameter remapping
        |
        v
  backward() / accumulate() / collectives
        |
        v
  save_state() / load_state()
Accelerator owns the lifecycle while focused modules and optional runtimes perform backend-specific work.

The benefit is architectural containment. A training loop can call prepare(), accumulate(), backward(), and save_state() without scattering DeepSpeed or FSDP decisions across application code. But containment only works if the façade exposes invalid states rather than hiding them. DeepSpeed allows one model per Accelerator; FSDP2 requires an optimizer to be prepared with its model and also permits one model; dynamo_plugin and dynamo_backend are mutually exclusive.

Preparation Is an Ordered Graph

The key contract appears inside preparation. The generic path uses two passes: dataloaders, models, and optimizers are prepared first; learning-rate schedulers are prepared second because they may depend on the final optimizer list.

The two-pass preparation helper
def _prepare_one(self, obj, first_pass=False, device_placement=None):
    if first_pass:
        if isinstance(obj, torch.utils.data.DataLoader):
            return self.prepare_data_loader(obj, device_placement=device_placement)
        elif isinstance(obj, torch.nn.Module):
            return self.prepare_model(obj, device_placement=device_placement)
        elif isinstance(obj, torch.optim.Optimizer):
            return self.prepare_optimizer(obj, device_placement=device_placement)
    elif isinstance(obj, LRScheduler):
        return self.prepare_scheduler(obj)
    return obj

The important feature is not the type dispatch. It is the dependency order. Preparation can replace an optimizer with an AcceleratedOptimizer; preparing a scheduler before that replacement risks binding it to an obsolete optimizer. The same issue becomes more severe when FSDP2 or tensor parallelism replaces parameters. Optimizer parameter references must be remapped to the new parameter objects, or updates can target stale parameters rather than the prepared model. The local FSDP2 remapping cost is reported as O(P + a), where P is parameter count and a is argument count, but preserving identity relationships is the real correctness requirement.

InvariantWhat it protectsFailure if broken
Prepare schedulers after optimizersScheduler-to-optimizer associationA scheduler can target an obsolete optimizer
Remap replaced parametersOptimizer-to-model identityUpdates can miss prepared model parameters
Mark prepared objectsIdempotent preparationNested wrappers and duplicated behavior
Save registered state togetherResumable training lifecycleWeights return without the rest of training state

The _is_accelerate_prepared marker protects idempotency: repeating a preparation path should not keep changing the result. Gradient accumulation follows the same discipline. _do_sync() centralizes the decision to skip synchronization on ordinary accumulation steps, synchronize at the configured boundary, and force synchronization at the end of a dataloader. It is a state machine embedded in lifecycle code.

Concentrated Complexity Needs Boundaries

A small public API does not remove complexity; it relocates it. Accelerator.__init__ is reported at about 290 source lines, with cyclomatic complexity 35 and cognitive complexity 48. _prepare_deepspeed reaches cognitive complexity 57, and prepare_model reaches 51. Those figures do not imply careless code. They show the cost of keeping backend-specific rules out of every caller.

The class is cohesive around one lifecycle—configure, prepare, train, observe, checkpoint—but it risks becoming a god façade that knows too much about every subsystem. The useful refactor is therefore not to fragment the public API or split files for their own sake. Keep Accelerator.prepare(), but place generic, DeepSpeed, Megatron-LM, and FSDP2 preparation behind narrow internal strategies that own validation and ordering for their backend.

This reduces the chance that adding a backend requires synchronized edits across constructor validation, preparation, backward execution, clipping, state dictionaries, and checkpoints. It also makes the lifecycle policy inspectable: each strategy can declare what it accepts, what it replaces, and what state it registers.

Error handling needs the same boundary discipline. In gather_for_metrics(), a broad except Exception can return untrimmed gathered data after an unexpected programming error, potentially producing inaccurate evaluation metrics. A safer fallback catches only metadata incompatibilities such as AttributeError and TypeError, logs a diagnostic warning, and lets unrelated failures remain visible. Resilience is valuable only when we can explain the failure being recovered from.

Measure and Test the Lifecycle

Internal boundaries make the code easier to change; operational boundaries reveal where it actually costs time. Python conditionals rarely dominate distributed training. Startup can be dominated by device transfer, wrapper creation, sharding, or torch.compile; the step path by accumulation, backward execution, clipping, and collectives; and checkpoints by serialization and shared-filesystem behavior.

These signals matter more as scale rises. At 10× scale, dataloader throughput and ordinary DDP communication become prominent. Around 100×, collective stragglers, uneven inputs, checkpoint bandwidth, and state aggregation become central. At 1000×, topology-aware configuration, sharded checkpoints, rank-zero bottlenecks, timeout tuning, and avoiding object gathers become essential.

gather_object() is a clear example of a costly boundary: GPU tensors can move to CPU and be pickled, adding transfer, serialization, and result-size costs. Checkpointing has a similar profile. save_state() has O(S) bookkeeping plus O(B) I/O for B serialized bytes, while non-sharded paths can materialize state dictionaries in memory and shared directories add metadata contention, barriers, cleanup, and rank coordination. Safe serialization reduces pickle-related risk but not peak-memory pressure.

Accelerator.profile can surround preparation, representative steps, and checkpoint boundaries. Rank-specific Chrome traces can then distinguish compute, communication, waiting, and I/O. Startup logs should capture the effective distributed type, precision mode, device type, rank, world size, and selected backend; environment-driven behavior is otherwise hard to reproduce.

Tests should mirror the same lifecycle. Start with a CPU contract test that prepares a model, optimizer, dataloader, and scheduler; verify return order, registries, preparation markers, and a forward-backward-step cycle. Add constructor tests for mutually exclusive inputs. Then test accumulation transitions and checkpoint round trips, including model parameters, optimizer and scheduler state, internal step, and custom registered state. Finally, launch multi-process tests for collective gathering, duplicate trimming, uneven inputs, and concurrent checkpoint ownership—behavior mocks cannot prove.

What the Façade Must Guarantee

The primary lesson is that a distributed-training façade is trustworthy only when its hidden lifecycle invariants are explicit, protected, and observable. Accelerator earns its stable API by centralizing backend differences at preparation and lifecycle boundaries. Its two-pass preparation, parameter remapping, idempotency markers, state registration, and synchronization transitions show why ordering and object identity are part of correctness rather than implementation detail.

As training systems add backends and scale, callers should not need to learn every runtime’s rules. But the orchestration layer must become more deliberate, not less: complexity can be concentrated in one place only if that place has clear contracts, visible failures, and evidence from production.


View Permalink