Liangyu Wang

Liangyu Wang

LLM, MLsys

About Me

I am Liangyu Wang, a Researcher on the Alibaba Qwen Team, working on large language model (LLM) pretraining and efficient training systems. I received my Ph.D. in Computer Science from King Abdullah University of Science and Technology (KAUST). Before that, I earned my master's degree from The Chinese University of Hong Kong, where I focused on multimodal machine learning. My research interests include Model4Model training and recursive self-improvement (RSI) for model training, distributed training and inference of LLMs, and multi-threaded and multi-stream scheduling. I have interned as an LLM Pretraining Engineer at Aramco, working with large-scale GPU clusters to boost training throughput and model scalability. Currently, I am working on:
  • Model4Model training / RSI for model training
  • Efficient reinforcement learning (RL) for LLMs reasoning
  • Distributed training and inference of LLMs
  • Efficient algorithm and infrastructure design for LLMs

🔥 News

📝 Publications

Tech Report 2026 On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, and Dayiheng Liu
Paper
Preprint 2026 Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers
Liangyu Wang *, Siqi Zhang *, Junjie Wang, Yiming Dong, Bo Zheng, Zihan Qiu, Shengkun Tang, Di Wang, Rui Men, and Dayiheng Liu
Paper | Megatron Code | FSDP Code
Preprint 2026 History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Li Nanbo *, Firas Laakom *, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, and Jürgen Schmidhuber
Paper | Project Page
Preprint 2026 SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Shengkun Tang *, Zekun Wang *, Bo Zheng *, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, and Dayiheng Liu
Paper
COLM 2025 ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
Liangyu Wang, Jie Ren, Hang Xu, Junxiao Wang, Huanyi Xie, David E. Keyes, and Di Wang
Paper | Code
NeurIPS 2025 FlashDP: Memory-Efficient and High-Throughput DP-SGD Training for Large Language Models
Liangyu Wang, Junxiao Wang, Jie Ren, Zihang Xiang, David E. Keyes, and Di Wang
Paper | Code
Preprint 2025 Infinite-Sampling: Efficient and Stable Grouped RL Training for Large Language Models
Liangyu Wang, Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang
Paper
ACL 2026 findings DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
Liangyu Wang, Huanyi Xie, and Di Wang
Paper | Code

☃️ Projects

Canzona: Matrix-based Optimizers for Large-Scale Distributed Training
Implementations that make Muon and SOAP practical under distributed sharding by reconstructing full-matrix optimizer tasks, assigning logical host ranks, and pipelining gather, compute, scatter, and update.
Training StackCanzona ImplementationDescription
Megatron-LM Megatron-Canzona Megatron-Canzona GitHub starsMegatron integration with load-balanced DP partitioning, asynchronous TP micro-group scheduling, parameter splitting, and extensible matrix optimizer plugins.
PyTorch FSDP FSDP-Canzona FSDP-Canzona GitHub starsFSDP-oriented executor that assigns full-matrix optimizer tasks to host ranks, gathers gradient shards, computes full updates, and scatters local update shards back.
ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory GitHub stars
A framework that enables fine-tuning of extremely large language models (like OPT-175B) on limited GPU memory through zeroth-order optimization and CPU-GPU offloading.
Tiny-LLM-Libs: Minimalistic Re-Implementations of Popular LLM Libraries
Minimal replicas that map major distributed-training stacks to tiny, readable implementations.
Original StackTiny ReplicaDescription
FSDP Tiny-FSDP GitHub starsSide-by-side implementations of DDP, ZeRO-3, and FSDP, including their distinct communication patterns and memory trade-offs.
DeepSpeed Tiny-DeepSpeed GitHub starsA minimal DeepSpeed-style training stack with DDP + ZeRO-1/2/3, plus meta initialization, rank mapping, and compute-communication overlap.
Megatron-LM Tiny-Megatron GitHub starsEducational TP, DP, and 2D TP+DP hybrid training with custom modules and a runtime auto-tuner for kernel selection.
Tinytron: A Minimal Pre-Training Stack for GPT-Style Language Models GitHub stars
A hackable and developer-friendly framework featuring modular GPT architecture with FA/GQA/MoE support, distributed training (ZeRO-1, Sequence-Expert Joint Parallelism), mixed precision training, and comprehensive profiling utilities for efficient large model pre-training.

📋 Academic Service