We recently released our paper Canzona: Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers.

The goal is straightforward: make matrix-based optimizers (e.g., Shampoo, Muon, SOAP) run efficiently in mainstream distributed training stacks such as Megatron.

What is the core challenge?

Matrix-based optimizers often require holistic updates, while distributed LLM training heavily relies on tensor fragmentation across devices.

This creates a structural mismatch:

• Synchronous solutions usually incur significant computation overhead.
• Layer-wise partitioning alone cannot resolve the mismatch while preserving communication-efficient geometry.

Key idea of Canzona

Canzona introduces two key designs:

1. Decoupling logical optimizer assignment from physical parameter distribution.
2. Parallelism-aware optimization.
   • For Data Parallelism: alpha-Balanced Static Partitioning to mitigate load imbalance.
   • For Tensor Parallelism: asynchronous compute pipeline with Micro-Group Scheduling to batch fragmented updates and hide reconstruction overhead.

Results (reported in the paper)

On Qwen3 models (up to 32B) with 256 GPUs, Canzona achieves:

• 1.57x speedup in end-to-end iteration time.
• 5.8x reduction in optimizer step latency.

One-sentence takeaway

Canzona is less about proposing yet another optimizer, and more about making existing matrix-based optimizers truly practical in modern distributed LLM training systems.

我们最近发布了论文 Canzona： Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers。

这篇工作的核心目标很直接： 让 Shampoo、Muon、SOAP 这类矩阵优化器，在 Megatron 等主流分布式训练框架下高效落地。

核心问题是什么？

矩阵优化器通常要求参数更新保持“整体性”（holistic updates），但大模型分布式训练会把参数切碎到不同设备（tensor fragmentation）。

这两者天然存在冲突：

• 同步式方案通常带来较高的计算冗余。
• 仅靠按层切分，难以同时满足通信效率和几何约束。

Canzona 的关键思路

Canzona 主要做了两件事：

1. 逻辑优化器分配与物理参数分布解耦。
2. 按并行模式定制优化策略。
   • 在 Data Parallelism 下：提出 alpha-Balanced Static Partitioning，缓解负载不均。
   • 在 Tensor Parallelism 下：提出异步计算管线与 Micro-Group Scheduling，将碎片化更新批处理并隐藏重构开销。

论文报告的效果

在 Qwen3 系列模型（最高 32B）和 256 GPU 上，Canzona 取得了：

• 端到端 iteration time 1.57x 加速。
• optimizer step latency 5.8x 降低。

一句话总结

Canzona 的重点不是再造一个新优化器，而是把现有矩阵优化器和现代分布式 LLM 训练系统更自然地对齐起来，让它们真正可用、可扩展。