Polyadic Research Group

Intelligence is
not a sequence.

Starting with continuous diffusion large language models, we develop the next generation of highly efficient AI models.

0.85
CONTINUOUS DIFFUSION MODELSPARALLEL INFERENCENEXT-GENERATION FOUNDATION MODELS

Topics

Beyond one token at a time.

We currently focus on how continuous language representations and parallel computation for reasoning can improve machine intelligence more cost-effectively.

Continuous diffusion language models

Study model paradigms that generate language through continuous-state refinement in a consciousness space. Explore token representations, denoising architectures, noise conditioning, and scaling behavior.

MODELS + THEORY

Parallel inference & reasoning

Refine multiple token positions together, and investigate how sampling and verification affect generation. Evaluate quality, diversity, and end-to-end latency under explicit compute budgets.

SAMPLING + EVALUATION

Efficient, capable next-generation machine intelligence

Study how data, compute, and model design shape learning, generalization, and scaling in continuous diffusion language models.

FOUNDATIONS + SCALING

“Serial thinking limits LLM intelligence at a given parameter count. Parallel thinking will be essential to the next generation of efficient intelligence.”

Research

Recent research highlights

Read our latest research reports on key topics in continuous diffusion language models, including model architectures, loss functions, quantization, inference, and memory.

Blogs

Continuous Diffusion Language Model

Animated conceptual overview of NIPSD: a frozen CDLM teacher guides distillation through CD warmup, noise-interpolated perturbations, and FiTScore; at inference, only the distilled generator maps noise to joint token outputs in one or a few evaluations.
PREPRINT

Understanding and Enabling Score Distillation in Continuous Diffusion Language Models

Hao Jiang, Yujian Chen, Ruixi Chen, Yihong Luo, Guang Cheng, Weijian Luo, Tianyang Hu

Abstract

NIPSD adapts score-based distribution matching to continuous diffusion language models through teacher-trajectory warmup, noise-interpolated perturbations, and continuous student-score estimation. It enables one-step and few-step generation with FLM and LangFlow while resolving token uncertainty and preserving variation across samples.

PDF
Selected reported results & evaluation context
OpenWebText · 1,024-token generation · One-step models initialized from distilled two-step generators
ModelNFEGenPPL ↓Entropy
FLM teacher51270.915.41
FLM + NIPSD173.555.17
LangFlow teacher51241.525.31
LangFlow + NIPSD143.105.26

Table 2: one-step NIPSD models are initialized from already-distilled two-step generators and trained with Diff-Instruct (DI). GenPPL is evaluated by GPT-2 Large on 1,024 generated samples; entropy is mean per-sample unigram entropy. These scores do not measure wall-clock speedup. Table 1 reports standard-initialization results at 1, 2, and 4 NFE, including one-step GenPPL of 132.90 for FLM+NIPSD and 70.08 for LangFlow+NIPSD.

Animated comparison of standalone generation: DSpark-style drafters use one backbone pass and sequential Markov or recurrent head sampling, while LangFlow iteratively refines a continuous full-sequence state before joint token readout.
PREPRINT

The Generative Gap Between Speculative Drafter and Continuous Diffusion Language Model

Yujian Chen, Hao Jiang, Guang Cheng, Weijian Luo, Tianyang Hu

Abstract

A standalone generation comparison of DSpark-style drafters and LangFlow without target features or verification. Recurrent head memory improves the drafter, but LangFlow leads on longer sequences under the evaluated budgets; additional full-sequence refinement improves generation at fixed weights.

PDF
Selected reported results & evaluation context
OpenWebText · 1,024-token unconditional single-block generation
ModelGenPPL ↓MAUVE ↑
DSpark-Markov524.790.0104
DSpark-RNN216.930.0780
LangFlow50.160.8370

100k-update EMA checkpoints with common data preprocessing, backbone dimensions, and training-update budgets. LangFlow uses 1,024 denoiser evaluations; drafters use one backbone pass followed by 1,024 head sampling steps. These are standalone adaptations trained from scratch without target features or verification, and the comparison is not matched for sampling compute. LangFlow leads at lengths 128 and 1,024 under the default budgets; DSpark-RNN leads at length 16. See the paper for full settings and limitations.

Animated conceptual overview of ConvergeFlow: learned base weights and a Gaussian kernel define a convex embedding predictor; repeated flow updates approach vocabulary embeddings for terminal readout without a separately CE-trained decoder, under the stated convergence assumptions.
PREPRINT

ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings

Na Li, Yuchen Jiao, Changxiao Cai, Gen Li

Abstract

ConvergeFlow parameterizes its data predictor as a convex combination of token embeddings, combining learned base weights with a Gaussian corruption kernel. Flow-matching MSE trains the predictor; under stated assumptions, the flow approaches token embeddings without a separately CE-trained decoding head.

Selected reported results & evaluation context
OpenWebText · 1,024-token generation · Selected quality-diversity operating points
ModelGenPPL ↓Entropy
FLM (179M)62.235.33
LangFlow (130M)60.095.43
ELF (105M)65.305.40
ConvergeFlow (130M)33.175.44

Reported operating points from Table 1. GenPPL is scored by GPT-2 Large; entropy is mean within-sequence unigram entropy. ConvergeFlow uses LangFlow initialization, frozen pretrained embeddings, and 175k additional MSE updates, with sampling controls tuned toward dataset entropy 5.44. Model sizes and training/sampling budgets differ; these scores do not isolate architecture gains or speed. The convergence theorem requires regularity and time-grid conditions. Date: initial arXiv submission, August 24, 2026.

Masked Diffusion Language Model

Animated conceptual overview of OrderGPT: aggregated attention supports spectral supervision and OrderMLP priority learning, followed by autoregressive generation under one frozen task-level order. The displayed variable order is illustrative.
PREPRINT

OrderGPT: Generation Order as a Learnable Modeling Choice

Chenhe Yang, Yuhuan Lyu, Tianyang Hu

Abstract

OrderGPT learns a task-level generation order from AO-GPT attention. A spectral teacher supervises OrderMLP priorities; training transitions to the learned order, which is frozen for autoregressive generation across language, human-pose, and tabular tasks.

Selected reported results & evaluation context
OpenWebText training · 400k updates · Selected zero-shot perplexity ↓
Model / inference orderWikiText-1031BW
AO-GPT (random inference)55.63590.168
AO-GPT (L2R inference)58.53591.069
Fixed-random AR + curriculum51.80086.870
OrderGPT (Order MLP)39.21770.046

Internal comparisons from Table 1 use matched backbone capacity, preprocessing, and 400k-update budgets, with 1,024-token training contexts and no language-model EMA or prescribed L2R training mixture. The two AO-GPT rows evaluate the same checkpoint. OrderGPT freezes its learned task-level order for inference. These are reported perplexities, not generation perplexity or speed measurements; the paper also evaluates human-pose and tabular modeling.

Animated conceptual overview of Beyond Serial Sampling: Token dependence; KL error guarantees; Parallel sampling.
NeurIPS 2025

Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models

Gen Li, Changxiao Cai

Abstract

Develops information-theoretic convergence bounds for masked diffusion sampling. KL error depends on token dependence, sampling iterations, and predictor accuracy; matching upper and lower bounds characterize settings where fewer rounds than the sequence length can suffice.

Animated conceptual overview of Confidence-Based Decoding: Prediction entropy; Adaptive selection; Parallel unmasking.
PREPRINT

Confidence-Based Decoding is Provably Efficient for Diffusion Language Models

Changxiao Cai, Gen Li

Abstract

Analyzes an adaptive decoding strategy that limits cumulative prediction entropy within each iteration. KL-accuracy guarantees connect the expected number of sampling rounds to target-distribution entropy, explaining how confidence can guide efficient parallel token generation under the paper’s assumptions.

Traditional GPT

Animated conceptual overview of Any-Order GPT: Any-order modeling; Decoder-only backbone; Parallel decoding.
ICML 2026 · Oral

Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma

Abstract

Studies masked diffusion and standard autoregressive modeling in a decoder-only architectural framework, separating generation-order choices from network design. Experiments investigate parallel decoding, temperature annealing, and the resulting quality–compute tradeoffs.

Animated conceptual overview of Attention Sink: Value aggregation; Variance discrepancy; Attention sink.
ICML 2026

The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity

Siquan Li, Kaiqi Jiang, Jiacheng Sun, Tianyang Hu

Abstract

Traces attention sinks to variance differences created by value aggregation and amplified by FFN super neurons. Controlled interventions test the proposed causal chain; head-wise RMSNorm is studied as a way to stabilize aggregation statistics during pretraining.

Others

Animated conceptual overview of Reward-Instruct: Base diffusion model; Reward optimization; Few-step generator.
NeurIPS 2025

Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation

Yihong Luo, Tianyang Hu, Weijian Luo, Kenji Kawaguchi, Jing Tang

Abstract

Trains reward-enhanced few-step image generators from pretrained diffusion models using regularized reward maximization. Its reward-tilted parameter updates remove the need for a separate diffusion-distillation loss, studying the role of reward design in image quality and preference alignment.

Animated conceptual overview of Diff-Instruct: Pretrained diffusion; Integral KL matching; Differentiable generator.
NeurIPS 2023

Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models

Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, Zhihua Zhang

Abstract

Transfers knowledge from pretrained diffusion models to differentiable generators without training data. Integral KL divergence compares distributions along a diffusion process, providing a shared framework for one-step diffusion distillation and the refinement of existing GAN generators.

People

Different researchers. Shared curiosity.

Tianyang Hu

Co-initiator

Assistant Professor, The Chinese University of Hong Kong, Shenzhen. Director of SAGA Lab at CUHK-Shenzhen; independent principal investigator.

Brian K Chen

Co-initiator

Bachelor’s degree in Mathematics, University of Oxford. PhD in Mathematics, National University of Singapore. Brazilian jiu-jitsu black belt and amateur MMA champion at the Hong Kong event.

Guang Cheng

Academic advisor

Professor of Computer Science, UCLA. Director of UCLA Trustworth Lab. Fellow of the Institute of Mathematical Statistics.

Gen Li

Academic Advisor

Assistant Professor in the Department of Statistics at the Chinese University of Hong Kong.

Hao Jiang

Contributor

Visiting student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Yujian Chen

Contributor

Undergraduate, School of Computer Science, Peking University.

Zhiyuan Yang

Contributor

Undergraduate, Turing Class, School of Computer Science, Peking University.

Mingzhi Zhang

Contributor

Undergraduate, School of Mathematical Sciences, Peking University.

Xinlang Cai

Contributor

Undergraduate, Turing Class, School of Computer Science, Peking University.

Yihao Shen

Contributor

Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Yan Wang

Contributor

PhD student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Yuhuan Lyu

Contributor

Master’s student, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Ruixi Chen

Contributor

Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Yulei Sun

Contributor

Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Jiayi Han

Contributor

Undergraduate, SAGA Lab, The Chinese University of Hong Kong, Shenzhen.

Supported by CUHK-Shenzhen and UCLA, with members from PKU, CUHK, HKUST, Oxford, UCLA, and beyond.

Open Source

Open model code.
Shared research progress.

We are building an open library of continuous diffusion language models. Our goal is to make strong models easy to discover, use, and study through shared tools, reproducible experiments, and community collaboration.

Contribute to the library

Curate leading models

Collect and organize the best open-source continuous diffusion language models, with checkpoints, papers, and clear documentation.

Make models easy to use

Offer a unified API, one-click demos, and ready-to-run examples to load models, generate text, and compare methods.

Connect the community

Share reproducible experiments, compare results, and exchange ideas through community contributions and academic collaboration.

Our mission

Exploring new
model paradigms.

We are a nonprofit academic organization dedicated to exploring next-generation model paradigms beyond Next-token Prediction.

Starting with continuous diffusion language models, we study continuous language representations and parallel reasoning to explore how machine intelligence can become more capable and computationally efficient.

Share models, code, and research findings openly.

Join & Support

Build with us.

Collaborate on research, contribute to the library, join the group, or support our nonprofit mission.

Collaboration & support · William Weijian Luo

Research collaboration

Discuss open questions, share results, and build research projects together.

Open-source contributions

Contribute model integrations, examples, documentation, or reproducibility studies.

Join the group

Young researchers are welcome. Tell us about your interests and the questions you want to explore.

Support our work

Donations from institutes and individuals help advance open parallel intelligence. Contact us to discuss support.

Young researchers are warmly welcome to shape parallel intelligence with us. Connect via or .

We welcome donations from institutes and individuals to help Polyadic Intelligence advance the frontier of open parallel intelligence.