BigMac: Breaking the Pareto Frontier in Multimodal LLM Training

Abstract

BigMac is a new pipeline-parallel training paradigm for native multimodal models. It addresses the tension between computational efficiency and GPU memory usage in multimodal large language model training through a dependency-safe nested pipeline. BigMac uses a mature LLM pipeline as its backbone and schedules encoder and generator computation at dependency-safe points without disrupting the LLM execution order. This enables efficient multimodal pipeline training without introducing additional bubbles into the LLM pipeline, while keeping activation memory bounded. Compared with conventional pipeline-parallel implementations, BigMac further decouples global scheduling from runtime execution and provides developer-friendly interfaces and tool chains that reduce the cost of model integration and system optimization. In our experiments, BigMac achieves 1.08–1.9× training speedups over baseline systems while maintaining stable memory usage as the global batch size increases. BigMac is now deployed in production as a core training component for Rednote’s Dots LLM family. Source code: https://github.com/Dots-Infra/BigMac/

Multimodal large language models (MLLMs) are becoming core infrastructure for the next generation of AI systems. Unlike text-only LLMs, multimodal models must understand different types of information—such as language, images, and audio—within a unified model. In many applications, they must also generate images, speech, and other modalities. As their capabilities expand, so does the complexity of the systems required to train them.

The challenge is that an MLLM is typically not a single homogeneous Transformer. Instead, it combines several modules with very different characteristics. A representative multimodal model contains three components: modality encoders, an LLM backbone, and modality generators. Encoders convert raw inputs such as images or audio into embeddings. The LLM reasons over the resulting multimodal token sequence. Generators map the LLM output back into a target modality—for example, an image diffusion model may generate pixels.

Figure 1: A representative MLLM architecture consisting of modality encoders, an LLM backbone, and modality generators.

This modular architecture provides tremendous expressive power, but it also pushes training systems into territory rarely encountered by standard LLM pipelines. Encoders, LLMs, and generators differ in model size, execution pattern, activation memory footprint, and preferred parallelization strategy. The data itself is irregular as well: one microbatch may contain only a few vision tokens, while another may contain many more. Even when the LLM sequence length is fixed, the amount of work performed by the encoder and generator can vary significantly.

The Compute–Memory Trade-off in Existing Systems

Existing MLLM training systems typically choose one of two designs. Both are reasonable, but both encounter significant bottlenecks at scale.

The first design is compute-efficient. It decouples encoder and generator computation from the LLM pipeline, similar to the modality-decoupled parallelism used in LongCat-Flash-Omni to handle heterogeneity in large-scale multimodal training. For example, a system may first run encoder forward passes for all microbatches, retain their activations, and then launch the LLM pipeline. This prevents variations in encoder and generator latency from disrupting the LLM pipeline: slower modality modules do not directly create bubbles inside it.

However, encoder activations must be retained until their corresponding gradients return, so activation memory grows with the number of microbatches. The situation becomes even worse when the model includes a generator: LLM activations may also need to remain live until generator computation finishes. This is especially expensive for production-scale models, where LLM activations are often much larger than encoder activations.

Figure 2: A compute-efficient pipeline. Decoupling modality computation from the LLM pipeline reduces bubbles but retains a large volume of activations.

The second design is memory-efficient. It incorporates the encoder and generator into the same pipeline as stages before and after the LLM. For example, in Megatron Core’s multimodal training example, the vision-language model is added to the LLM pipeline as a separate pipeline stage. This shortens activation lifetimes and therefore reduces memory usage. However, all modules are now coupled by pipeline dependencies. If an encoder or generator microbatch is slow, the LLM pipeline must wait. If generator latency varies significantly, bubbles emerge at the tail of the pipeline. The system saves memory at the expense of computational efficiency.

Figure 3: A memory-efficient pipeline. Integrating all modules into a single pipeline reduces memory usage but introduces cross-module bubbles.

This is precisely the Pareto frontier that BigMac aims to break. Compute-efficient systems run fast but consume substantial memory; memory-efficient systems save memory but are prone to bubbles. This trade-off may be acceptable for small-scale training, but it becomes a bottleneck in large-scale multimodal training, especially when modality generators are involved.

The Core Idea Behind BigMac

BigMac starts from a simple principle: the LLM pipeline should remain the scheduling backbone. Large-scale LLM training already relies on carefully optimized schedules such as 1F1B and interleaved 1F1B. These schedules are mature, efficient, and deeply integrated into production LLM training stacks. BigMac does not attempt to replace them; instead, it treats the LLM schedule as the base timeline.

BigMac first preserves this critical LLM pipeline, then inserts encoder and generator computation at appropriate points. A point is considered “appropriate” when its inputs are ready and inserting the work does not disturb the original LLM execution order. As a result, the LLM pipeline can continue to make progress while encoder and generator activations are released earlier. We call this design a dependency-safe nested pipeline.

Figure 4: BigMac preserves the original LLM pipeline schedule and embeds encoder and generator work at dependency-safe positions.

This design gives BigMac two important properties.

First, it preserves computational efficiency. Because the encoder and generator are not forced into the LLM pipeline as independent stages that advance in lockstep, variations in their execution time do not propagate through the entire LLM pipeline. The LLM continues to follow the same schedule it would use in a compute-efficient system.

Second, it bounds the activation memory of modality modules. BigMac runs encoder and generator backward passes as soon as their gradients become ready. The encoder no longer needs to retain activations for every microbatch until the LLM pipeline finishes, and the generator no longer leaves a long tail of retained activations. At the algorithmic level, BigMac reduces encoder and generator activation memory to O(1) while leaving LLM activation behavior unchanged.

This is the key to the entire pipeline: BigMac does not trade memory for bubbles, or bubbles for memory. It changes the schedule structure so that these two objectives no longer have to conflict.

BigMac: Making Multimodal Pipeline Training Practical

In real-world MLLM training, the challenge of pipeline parallelism extends well beyond arranging microbatches. When heterogeneous modules—such as an LLM, a vision or audio encoder, and a diffusion generator—must be trained together across multiple machines and GPUs, the system must solve an entire set of engineering problems. Where should heterogeneous modules be placed? How can a generated schedule integrate with existing training frameworks such as Megatron Core? How can model code remain free of pipeline-parallel details? And when bubbles or waits appear, how can engineers locate bottlenecks and iterate quickly?

BigMac focuses on these system-level requirements that bridge the gap between designing a schedule and running training in practice. It provides three key capabilities: a visible, inspectable, and executable global plan; a pipeline-parallelism-transparent interface that lets model developers continue writing modular model code; and a schedule-aware toolchain that incorporates profiling, simulation, and visualization into a closed-loop performance optimization workflow. We discuss each capability below.

1. Making the Global Schedule Explicit

At the system integration layer, BigMac extracts pipeline scheduling from rank-local runtime control flow and represents it as a global operator table. Encoder, LLM, and generator operations—and the communication between them—are no longer scattered across the execution logic of individual ranks. Instead, they are first expressed in a unified schedule that can be inspected and executed.

This differs sharply from pipeline schedules in mature LLM training frameworks such as Megatron Core. Conventional schedules are typically written from the perspective of a single pipeline rank. Each rank runs highly optimized runtime control flow: several forward microbatches first fill the pipeline during warmup, forward and backward passes then alternate in steady state, and cooldown completes the remaining backward passes. Communication, activation buffers, gradient accumulation, loss handling, and overlap logic are all intertwined within this control flow.

Consider Megatron Core 0.17.1. Its schedules.py contains 2,375 lines, and the interleaved pipeline schedule function forward_backward_pipelining_with_interleaving() alone spans 1,097 lines. This complexity is not accidental; it reflects the runtime detail required to make a standard LLM pipeline fast. It also explains why such code is difficult to modify when heterogeneous modules such as vision encoders, audio encoders, or diffusion generators are introduced. Adding one of these modules is not simply a matter of inserting an encoder_forward() call. It changes microbatch dependencies, activation lifetimes, backward-readiness conditions, and the matching of communication operations across ranks.

At a high level, a conventional rank-local pipeline runtime can be approximated as follows:

def mcore_like_schedule_for_one_rank(rank):
    # The schedule is embedded in rank-local execution logic.

    for i in warmup_microbatches(rank):
        x = recv_forward_if_needed(rank)
        y = llm_forward(x)
        send_forward_if_needed(rank, y)
        save_activation(y)

    for i in steady_state_microbatches(rank):
        x = recv_forward_if_needed(rank)
        y = llm_forward(x)
        send_forward_if_needed(rank, y)

        dy = recv_backward_if_needed(rank)
        dx = llm_backward(dy)
        send_backward_if_needed(rank, dx)

    for i in cooldown_microbatches(rank):
        dy = recv_backward_if_needed(rank)
        dx = llm_backward(dy)
        send_backward_if_needed(rank, dx)

This pseudocode is deliberately simplified. A real implementation must also handle virtual pipeline stages, overlapped point-to-point communication, rank boundary conditions, activation deallocation, loss reduction, and distributed gradient synchronization. The deeper problem is that scheduling policy and local runtime mechanisms are intertwined. Adding encoder or generator work requires updating the warmup, steady-state, and cooldown paths simultaneously while preserving the new dependency and communication order. With this style of code, global questions are difficult to answer directly: Where does the encoder forward pass for microbatch 7 run? When is its output consumed by the LLM? When can encoder backward safely release the saved activations?

BigMac introduces a different interface between scheduling and execution. The scheduler directly generates a global operator table spanning all pipeline ranks, microbatches, and module types. This table contains LLM forward and backward operators, encoder forward and backward operators, generator forward and backward operators, and the communication required at module boundaries. The executor then interprets the local operator sequence for each rank and dispatches every operator to the appropriate backend: LLM pipeline operators go to Megatron Core, modality operators go to the encoder or generator runtime, and data movement goes to the communication backend.

def bigmac_schedule_globally(config):
    # The scheduler owns the global plan.
    schedule = BigMacScheduler(
        pp_size=config.pp_size,
        vpp_size=config.vpp_size,
        num_microbatches=config.num_microbatches,
        encoder_policy=config.encoder_policy,
        generator_policy=config.generator_policy,
    ).generate_schedule()

    schedule = add_communication_ops(schedule)
    check_deadlock(schedule)
    visualize_timeline(schedule)
    return schedule


def bigmac_execute_one_rank(rank, schedule):
    # The executor owns fine-grained local execution.
    for op in schedule.local_ops(rank):
        if op.kind == "LLM_FORWARD":
            run_llm_forward(op)
        elif op.kind == "LLM_BACKWARD":
            run_llm_backward(op)
        elif op.kind == "ENCODER_FORWARD":
            run_encoder_forward(op)
        elif op.kind == "ENCODER_BACKWARD":
            run_encoder_backward(op)
        elif op.kind == "GENERATOR_FORWARD":
            run_generator_forward(op)
        elif op.kind == "GENERATOR_BACKWARD":
            run_generator_backward(op)
        elif op.kind == "COMM":
            run_communication(op)

The resulting schedule is no longer hidden inside runtime control flow. It becomes an execution table that can be printed, inspected, simulated, and visualized. The timeline below illustrates this representation for PP=4. It is a simplified example rather than a complete schedule; for the full details, see BigMac’s official visualization tool. Here, VF denotes vision-encoder forward, F denotes LLM forward, GF/GB denote generator forward/backward, B denotes LLM backward, and VB denotes vision-encoder backward:

Time    0       1       2       3       4       5       ...     later
Rank 0  VF0     VF4     F0      F1      F2      F3      ...     B* / GF* / GB* / VB*
Rank 1  VF1     VF5     IDLE    F0      F1      F2      ...     B* / GF* / GB* / VB*
Rank 2  VF2     VF6     IDLE    IDLE    F0      F1      ...     B* / GF* / GB* / VB*
Rank 3  VF3     VF7     IDLE    IDLE    IDLE    F0      ...     B* / GF* / GB* / VB*

This separation gives BigMac two practical advantages. First, the scheduling policy becomes visible and inspectable. We can directly see how encoder, LLM, and generator work is interleaved across all ranks without reverse-engineering global behavior from a long rank-local runtime function. Second, execution remains backend-friendly. LLM operators can continue to reuse optimized pipeline runtimes such as Megatron Core, while modality modules can retain their own data-parallel, FSDP, or optimizer implementations. Communication operators and deadlock checks can also be added to the schedule without rewriting the model runtime.

2. Integrating Model Code Naturally with Pipeline Parallelism

Multimodal training inherently requires close collaboration between system engineers and model researchers. Model researchers continually adjust how the encoder, LLM, and generator connect, as well as the loss design and data format. System engineers must place these modules within PP, TP, DP, FSDP, and other parallelization strategies while managing communication, activation lifetimes, and scheduling efficiency. When a conventional PP system is extended to multimodal training, however, this division of responsibility can quickly become blurred. A model architecture change may require stage partitioning and send/receive logic to be rewritten, while system optimizations may leak back into the model code.

Allowing system engineers to modify the schedule is not enough. If model researchers must understand the PP runtime every time they scale up an experiment, BigMac would still not be an easy-to-use training system. BigMac’s second key capability is therefore a pipeline-parallelism interface that is as transparent as possible to model researchers.

Multimodal experiments usually begin on a single GPU or under data parallelism. Model researchers care about how the encoder produces features, how the LLM consumes them, and how the generator computes a loss from the LLM output. When the experiment needs to scale to pipeline parallelism, however, conventional systems often require model authors to split this logic across pipeline stages and manually manage microbatches, send/receive operations, activation buffers, and gradient handoffs. Scaling up then becomes a distributed-runtime rewrite rather than a simple increase in training scale.

BigMac moves this complexity out of the model code. Users can continue describing module boundaries in a form close to ordinary training code:

Figure 5: BigMac’s pipeline-parallelism-transparent interface. Model researchers describe module boundaries, while BigMac handles scheduling, handoffs, and communication underneath.

With this interface, model researchers only need to specify what each module produces and consumes. BigMac integrates the modules into the global schedule and handles pipeline stages, activation handoffs, gradient handoffs, and cross-device communication underneath. In other words, BigMac does not remove pipeline parallelism from the system; it prevents pipeline-parallel details from intruding into the model researcher’s development loop. A multimodal experiment already validated on a single GPU or under data parallelism can scale more naturally to pipeline-parallel training. Meanwhile, system engineers can still integrate Megatron Core, DDP/FSDP, and high-performance communication implementations at the executor layer and continue optimizing scheduling, communication, and resource utilization without polluting the model code.

3. Diagnosing Bottlenecks with a Schedule-Aware Toolchain

BigMac’s final key capability is a profiler, simulator, and visualization toolchain that understands schedule structure. In large-scale multimodal pipeline training, performance problems can rarely be diagnosed from iteration time or a few log lines alone. A bubble may be caused by a slow microbatch, variations in encoder or generator runtime, an activation or gradient handoff, or cross-rank communication waits. Engineers need to break a training iteration down to the operator level and see exactly what each rank is executing at each point in time, where it is idle, and which dependency is blocking subsequent computation.

Figure 6: Conventional performance diagnosis often requires downloading and manually correlating heavyweight traces from multiple ranks. The BigMac PP profiler organizes this information into a schedule-aware pipeline diagnosis workflow.

The BigMac executor records the execution time of every operator and exports a per-rank timeline or trace, as shown in Figure 7. From this trace, engineers can directly observe how encoder forward/backward, LLM forward/backward, generator forward/backward, and communication operations are interleaved across ranks. They can then determine whether a pipeline bubble is caused by compute imbalance, communication waits, module handoffs, or inappropriate microbatch grouping.

Figure 7: An example pipeline trace generated from per-operator execution times.

We also provide an interactive PP profiler trace example. After downloading it, you can open it directly in the Perfetto UI to inspect timelines and dependencies across ranks and operators.

Beyond profiling, BigMac provides a simulator for faster iteration on parallelization strategies. The profiler explains why the current training run is slow; the simulator lets engineers try different PP/VPP configurations, microbatch counts, module placements, and scheduling policies before launching another expensive training job, and estimate their impact on bubbles and throughput. Parallelization strategies can therefore be explored quickly at the schedule level instead of relying entirely on trial and error with large-scale training runs.

Together, these three capabilities turn BigMac from a schedule into a training system. At the global-schedule layer, the scheduler makes the overall pipeline arrangement controllable, and the executor maps it onto existing high-performance backends. The PP-transparent interface keeps model code modular, while the profiler and simulator feed execution results back into strategy iteration. For complex workloads that combine multimodal models with pipeline parallelism, BigMac provides a complete loop spanning representation, execution, diagnosis, and iteration—not merely a local optimization.

Experimental Results

We evaluate BigMac on two representative training workloads. The first, MLLM-Understanding, represents multimodal understanding training: an encoder converts image inputs into tokens, which are then processed by the LLM for understanding and reasoning. This workload does not include a modality generator. It uses Qwen3-30B-A3B as the LLM backbone and a 1.3B-parameter ViT encoder. The second workload, MLLM-Generation, represents training with both understanding and generation paths. It uses the same LLM and encoder but adds a 20B-parameter MMDiT generator. Both workloads use an 8K LLM sequence length, and all modules are trained.

On MLLM-Understanding, BigMac achieves a 1.08–1.1× training speedup over the compute-efficient baseline, Optimus, and a 1.6–1.9× speedup over the memory-efficient baseline, Megatron-DistTrain. The gain over Megatron-DistTrain comes from eliminating cross-module pipeline interference. The advantage over Optimus is more subtle: BigMac avoids activation memory pressure, which makes compute-efficient designs increasingly costly as batch size grows.

Figure 8: End-to-end performance on the MLLM-Understanding workload.

The memory trend is just as important as the speedup. As the per-GPU batch size increases, BigMac’s peak memory remains stable, similar to the memory-efficient baseline. By contrast, Optimus retains encoder activations throughout the LLM pipeline. Its memory usage grows rapidly and eventually results in out-of-memory errors at larger batch sizes.

MLLM-Generation refers to multimodal training with a modality generator. Rather than producing only representations for understanding tasks, the LLM passes intermediate results to an MMDiT generator, which then computes the generation-side training objective. The difference between systems becomes even more pronounced on this workload. A generator makes compute-efficient designs harder to scale because LLM activations may need to be retained until generator computation finishes. In our paper’s experiments, Optimus runs out of memory at every tested batch size on the MLLM-Generation workload. BigMac avoids this problem by running generator backward promptly and releasing generator-side activations as soon as possible.

Figure 9: End-to-end performance on the MLLM-Generation workload.

Compared with Megatron-DistTrain, BigMac achieves a 1.5–1.9× training speedup on MLLM-Generation while maintaining stable memory usage. This is where the nested pipeline is most valuable: the system must handle dependencies involving both the encoder and generator without incurring either extensive activation retention or severe pipeline bubbles.

Conclusion

The challenge of training MLLMs is not simply that the models are larger or the data is more complex. Rather, different modality modules constrain pipeline training to an awkward Pareto frontier. BigMac breaks this frontier. It preserves the LLM pipeline as a stable backbone and embeds encoder and generator computation only where dependencies are satisfied and the LLM execution order remains undisturbed. This allows the system to retain the computational efficiency of the LLM pipeline while releasing modality-module activations as early as possible. More importantly, BigMac provides more than a schedule: it turns the schedule into a visible, inspectable, and executable global plan, complemented by a PP-transparent interface and a schedule-aware toolchain. Together, these capabilities make the design practical to develop, debug, and deploy as a training system.

Paper and Citation

Paper: BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training

If BigMac is useful to your research or systems work, please consider citing it:

@article{zhang2026bigmac,
  title={BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training},
  author={Zili Zhang and Chengxu Yang and Shenglong Zhang and Chenyu Wang and Yufan Zhang and Tuo Dai and Zhouyang Li and Yuhong Ge and Chao Jin and Xin Jin and Yuliang Liu},
  journal={arXiv preprint arXiv:2605.25451},
  year={2026},
  doi={10.48550/arXiv.2605.25451}
}