CONTINUOUS DIFFUSION MODELSPARALLEL INFERENCENEXT-GENERATION FOUNDATION MODELS
Topics
Beyond one token at a time.
We currently focus on how continuous language representations and parallel computation for reasoning can improve machine intelligence more cost-effectively.
Continuous diffusion language models
Study model paradigms that generate language through continuous-state refinement in a consciousness space. Explore token representations, denoising architectures, noise conditioning, and scaling behavior.
MODELS + THEORY
Parallel inference & reasoning
Refine multiple token positions together, and investigate how sampling and verification affect generation. Evaluate quality, diversity, and end-to-end latency under explicit compute budgets.
Study how data, compute, and model design shape learning, generalization, and scaling in continuous diffusion language models.
FOUNDATIONS + SCALING
“Serial thinking limits LLM intelligence at a given parameter count. Parallel thinking will be essential to the next generation of efficient intelligence.”
Research
Recent research highlights
Read our latest research reports on key topics in continuous diffusion language models, including model architectures, loss functions, quantization, inference, and memory.
A beginner’s guide to continuous diffusion language models: continuous generative states, denoising, parallel token updates, and the dependencies that make tokens work together.
Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu
Abstract
A systematic study of normalization, noise conditioning, gradient routing, and self-conditioning. LangDiT improves continuous language generation within LangFlow and transfers compatible components to a Plaid-style formulation.
@misc{luo2026designing,
title={Designing Transformers for Continuous Diffusion Language Models},
author={Weijian Luo and Yujian Chen and Hao Jiang and Guang Cheng and Tianyang Hu},
year={2026},
eprint={2610.22330},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.22330}
}
Extended-training checkpoints: LangFlow 660k; Plaid-style 520k updates. Plaid-style sampling uses one additional final denoiser call. These scores use different training budgets and do not establish an isolated architecture gain or wall-clock speedup. See the paper for entropy, controls, and full settings.
NIPSD adapts score-based distribution matching to continuous diffusion language models through teacher-trajectory warmup, noise-interpolated perturbations, and continuous student-score estimation. It enables one-step and few-step generation with FLM and LangFlow while resolving token uncertainty and preserving variation across samples.
@unpublished{jiang2026understanding,
title={Understanding and Enabling Score Distillation in Continuous Diffusion Language Models},
author={Hao Jiang and Yujian Chen and Ruixi Chen and Yihong Luo and Guang Cheng and Weijian Luo and Tianyang Hu},
year={2026},
note={Preprint}
}
Table 2: one-step NIPSD models are initialized from already-distilled two-step generators and trained with Diff-Instruct (DI). GenPPL is evaluated by GPT-2 Large on 1,024 generated samples; entropy is mean per-sample unigram entropy. These scores do not measure wall-clock speedup. Table 1 reports standard-initialization results at 1, 2, and 4 NFE, including one-step GenPPL of 132.90 for FLM+NIPSD and 70.08 for LangFlow+NIPSD.
Yujian Chen, Hao Jiang, Guang Cheng, Weijian Luo, Tianyang Hu
Abstract
A standalone generation comparison of DSpark-style drafters and LangFlow without target features or verification. Recurrent head memory improves the drafter, but LangFlow leads on longer sequences under the evaluated budgets; additional full-sequence refinement improves generation at fixed weights.
@unpublished{chen2026generative,
title={The Generative Gap Between Speculative Drafter and Continuous Diffusion Language Model},
author={Yujian Chen and Hao Jiang and Guang Cheng and Weijian Luo and Tianyang Hu},
year={2026},
note={Preprint}
}
100k-update EMA checkpoints with common data preprocessing, backbone dimensions, and training-update budgets. LangFlow uses 1,024 denoiser evaluations; drafters use one backbone pass followed by 1,024 head sampling steps. These are standalone adaptations trained from scratch without target features or verification, and the comparison is not matched for sampling compute. LangFlow leads at lengths 128 and 1,024 under the default budgets; DSpark-RNN leads at length 16. See the paper for full settings and limitations.
ConvergeFlow parameterizes its data predictor as a convex combination of token embeddings, combining learned base weights with a Gaussian corruption kernel. Flow-matching MSE trains the predictor; under stated assumptions, the flow approaches token embeddings without a separately CE-trained decoding head.
@misc{li2026convergeflow,
title={ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings},
author={Na Li and Yuchen Jiao and Changxiao Cai and Gen Li},
year={2026},
eprint={2608.23551},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.23551}
}
Reported operating points from Table 1. GenPPL is scored by GPT-2 Large; entropy is mean within-sequence unigram entropy. ConvergeFlow uses LangFlow initialization, frozen pretrained embeddings, and 175k additional MSE updates, with sampling controls tuned toward dataset entropy 5.44. Model sizes and training/sampling budgets differ; these scores do not isolate architecture gains or speed. The convergence theorem requires regularity and time-grid conditions. Date: initial arXiv submission, August 24, 2026.
Haoyang Zheng, Xinyang Liu, Cindy Xiangrui Kong, Nan Jiang, Zheyuan Hu, Weijian Luo, Wei Deng, Guang Lin
Abstract
DiDi-Instruct distills masked diffusion language models through integral KL minimization with discriminator-estimated density-ratio rewards. Grouped reward normalization and intermediate-state matching stabilize training, while reward-guided ancestral sampling enables efficient few-step parallel generation.
@inproceedings{zheng2026didi,
title={Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct},
author={Haoyang Zheng and Xinyang Liu and Cindy Xiangrui Kong and Nan Jiang and Zheyuan Hu and Weijian Luo and Wei Deng and Guang Lin},
booktitle={International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=mtdyZsa47V}
}
arXiv v3, Section 4 and Table 1: the DiDi-Instruct rows use the complete configuration, including reward-guided ancestral sampling (RGAS). GenPPL is measured by GPT-2 Large; entropy is the reported sequence token entropy. RGAS uses a 131M discriminator, adding guidance and candidate-selection costs, so NFE ratios should not be interpreted as wall-clock speedups. The entry date follows the initial arXiv submission (September 29, 2025).
OrderGPT learns a task-level generation order from AO-GPT attention. A spectral teacher supervises OrderMLP priorities; training transitions to the learned order, which is frozen for autoregressive generation across language, human-pose, and tabular tasks.
@misc{yang2026ordergpt,
title={OrderGPT: Generation Order as a Learnable Modeling Choice},
author={Chenhe Yang and Yuhuan Lyu and Tianyang Hu},
year={2026},
eprint={2610.25949},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.25949}
}
Selected reported results & evaluation context
OpenWebText training · 400k updates · Selected zero-shot perplexity ↓
Model / inference order
WikiText-103
1BW
AO-GPT (random inference)
55.635
90.168
AO-GPT (L2R inference)
58.535
91.069
Fixed-random AR + curriculum
51.800
86.870
OrderGPT (Order MLP)
39.217
70.046
Internal comparisons from Table 1 use matched backbone capacity, preprocessing, and 400k-update budgets, with 1,024-token training contexts and no language-model EMA or prescribed L2R training mixture. The two AO-GPT rows evaluate the same checkpoint. OrderGPT freezes its learned task-level order for inference. These are reported perplexities, not generation perplexity or speed measurements; the paper also evaluates human-pose and tabular modeling.
Develops information-theoretic convergence bounds for masked diffusion sampling. KL error depends on token dependence, sampling iterations, and predictor accuracy; matching upper and lower bounds characterize settings where fewer rounds than the sequence length can suffice.
@inproceedings{breakingarbottleneck2025,
title={Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models},
author={Gen Li and Changxiao Cai},
booktitle={Advances in Neural Information Processing Systems},
year={2025},
url={https://proceedings.neurips.cc/paper_files/paper/2025/hash/111298628bfb153ee1f84b10fec3a8b9-Abstract-Conference.html}
}
Analyzes an adaptive decoding strategy that limits cumulative prediction entropy within each iteration. KL-accuracy guarantees connect the expected number of sampling rounds to target-distribution entropy, explaining how confidence can guide efficient parallel token generation under the paper’s assumptions.
@misc{confidencedecoding2026,
title={Confidence-Based Decoding is Provably Efficient for Diffusion Language Models},
author={Changxiao Cai and Gen Li},
year={2026},
eprint={2603.22248},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2603.22248}
}
Haonan Wang, Brian K. Chen, Siquan Li, Xinhe Liang, Hwee Kuan Lee, Kenji Kawaguchi, Tianyang Hu
Abstract
PrefixMemory-Tuning moves prefix information outside attention into a trainable, query-conditioned memory. Adding this memory output to frozen attention avoids the shared-softmax competition between input and prefix contributions, improving parameter-efficient adaptation across the evaluated classification, preference-alignment, and math-reasoning tasks.
@inproceedings{wang2026prefixmemory,
title={PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention},
author={Haonan Wang and Brian K. Chen and Siquan Li and Xinhe Liang and Hwee Kuan Lee and Kenji Kawaguchi and Tianyang Hu},
booktitle={International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=LvUMpZE44r}
}
Selected reported results & evaluation context
BigBench · One example per class · Accuracy (%) ↑
Adaptation method
LLaMA2-7B-Chat
Qwen2.5-3B-Instruct
Full fine-tuning
38.8
67.4
LoRA (r=64)
67.4
61.4
Prefix-Tuning (m=32)
21.3
52.0
PrefixMemory-Tuning
71.2
76.6
Table 1 of arXiv v3 / the ICLR 2026 paper. Mean test accuracy over five independent trials, each sampling one training example per class. LoRA uses rank 64; Prefix-Tuning uses 32 virtual tokens; PMT uses ELU features. These diagnostic few-shot results do not imply identical trainable-parameter counts or universal gains. Preference alignment and critique-based math fine-tuning are evaluated separately. The entry date follows the initial arXiv submission (June 16, 2025).
Studies masked diffusion and standard autoregressive modeling in a decoder-only architectural framework, separating generation-order choices from network design. Experiments investigate parallel decoding, temperature annealing, and the resulting quality–compute tradeoffs.
@misc{anyordergpt2025,
title={Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture},
author={Shuchen Xue and Tianyu Xie and Tianyang Hu and Zijin Feng and Jiacheng Sun and Kenji Kawaguchi and Zhenguo Li and Zhi-Ming Ma},
year={2025},
eprint={2506.19935},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2506.19935}
}
Traces attention sinks to variance differences created by value aggregation and amplified by FFN super neurons. Controlled interventions test the proposed causal chain; head-wise RMSNorm is studied as a way to stabilize aggregation statistics during pretraining.
@misc{attentionsink2026,
title={The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity},
author={Siquan Li and Kaiqi Jiang and Jiacheng Sun and Tianyang Hu},
year={2026},
eprint={2605.06611},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.06611}
}
Adapts few-step image generators to non-differentiable rewards by separating surrogate reward learning from generator optimization. Reward signals along deterministic TDM trajectories support post-training for text rendering, visual quality, and preference alignment.
@misc{tdmr12026,
title={TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward},
author={Yihong Luo and Tianyang Hu and Weijian Luo and Jing Tang},
year={2026},
eprint={2603.07700},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2603.07700}
}
Trains reward-enhanced few-step image generators from pretrained diffusion models using regularized reward maximization. Its reward-tilted parameter updates remove the need for a separate diffusion-distillation loss, studying the role of reward design in image quality and preference alignment.
@inproceedings{rewardinstruct2025,
title={Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation},
author={Yihong Luo and Tianyang Hu and Weijian Luo and Kenji Kawaguchi and Jing Tang},
booktitle={Advances in Neural Information Processing Systems},
year={2025},
url={https://proceedings.neurips.cc/paper_files/paper/2025/hash/f39cbd56877419229aef90dbe463c182-Abstract-Conference.html}
}
Transfers knowledge from pretrained diffusion models to differentiable generators without training data. Integral KL divergence compares distributions along a diffusion process, providing a shared framework for one-step diffusion distillation and the refinement of existing GAN generators.
@inproceedings{diffinstruct2023,
title={Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models},
author={Weijian Luo and Tianyang Hu and Shifeng Zhang and Jiacheng Sun and Zhenguo Li and Zhihua Zhang},
booktitle={Advances in Neural Information Processing Systems},
year={2023},
url={https://proceedings.neurips.cc/paper_files/paper/2023/hash/f115f619b62833aadc5acb058975b0e6-Abstract-Conference.html}
}
Bachelor’s degree in Mathematics, University of Oxford. PhD in Mathematics, National University of Singapore. Brazilian jiu-jitsu black belt and amateur MMA champion at the Hong Kong event.
Assistant Professor in the Department of Statistics at the Chinese University of Hong Kong.
HJ
Hao Jiang
Contributor
Visiting student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
YC
Yujian Chen
Contributor
Undergraduate, School of Computer Science, Peking University.
ZY
Zhiyuan Yang
Contributor
Undergraduate, Turing Class, School of Computer Science, Peking University.
MZ
Mingzhi Zhang
Contributor
Undergraduate, School of Mathematical Sciences, Peking University.
XC
Xinlang Cai
Contributor
Undergraduate, Turing Class, School of Computer Science, Peking University.
YS
Yihao Shen
Contributor
Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
YW
Yan Wang
Contributor
PhD student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
YL
Yuhuan Lyu
Contributor
Master’s student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
RC
Ruixi Chen
Contributor
Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
YS
Yulei Sun
Contributor
Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
JH
Jiayi Han
Contributor
Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.
Supported by CUHK-Shenzhen and UCLA, with members from PKU, CUHK, HKUST, Oxford, UCLA, and beyond.
Open Source
Open model code. Shared research progress.
We are building an open library of continuous diffusion language models. Our goal is to make strong models easy to discover, use, and study through shared tools, reproducible experiments, and community collaboration.
Collect and organize the best open-source continuous diffusion language models, with checkpoints, papers, and clear documentation.
Make models easy to use
Offer a unified API, one-click demos, and ready-to-run examples to load models, generate text, and compare methods.
Connect the community
Share reproducible experiments, compare results, and exchange ideas through community contributions and academic collaboration.
Our mission
Exploring new model paradigms.
We are a nonprofit academic organization dedicated to exploring next-generation model paradigms beyond Next-token Prediction.
Starting with continuous diffusion language models, we study continuous language representations and parallel reasoning to explore how machine intelligence can become more capable and computationally efficient.
Share models, code, and research findings openly.
Join & Support
Build with us.
Collaborate on research, contribute to the library, join the group, or support our nonprofit mission.