ByteDance Seed Open-Sources Cola2, a Continuous-Latent Language Model Trained on Just 346 Billion Tokens

ByteDance's Seed research team has open-sourced Cola2, a research language model that takes an unusual architectural bet: instead of predicting one token after another, it compresses text into continuous semantic latents, generates those latents with a diffusion transformer, and only then decodes them back into discrete text. The weights went live on Hugging Face on November 6 under the ByteDance-Seed organization, alongside a GitHub repository and an arXiv paper (2610.31305).

The architecture, described in the paper "Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation," chains three components: a dynamic-compression VAE that groups text into semantic spans, a block-causal Diffusion Transformer of roughly 4 billion parameters that learns the latent distribution via flow matching, and a span-causal token decoder with latent feedback that turns latents back into tokens. During inference the model repeatedly proposes a latent block, decodes tokens, and re-encodes the committed text so the continuous and discrete histories stay aligned.

The released checkpoint is deliberately modest: trained on just 346.03 billion source tokens, against 15 trillion or more for Llama-3-8B and 2.3 trillion for the diffusion baseline LLaDA-8B. The release tracker pegs the model at roughly 9 billion parameters. On the paper's eight-task evaluation suite it averages 60.10, and it beats Llama-3-8B on HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49) with over 43 times fewer pretraining tokens. Those are paper-reported results from the model's own authors and have not been independently verified; on most other tasks in the table, including MMLU and RACE, it trails the open-weight baselines.

The team is candid about the model's immaturity: the latent diffusion stage is still unconverged at the released checkpoint, and the paper's controlled scaling study shows Cola2 converging more slowly than alternatives like Cola-MDM, though it continues closing its gap to LLaDA past 150 billion tokens without saturating. The release includes eight sharded safetensors files, tokenizer and generation configs, remote-code modules for Hugging Face Transformers, and normalized evaluation data with SHA256 checksums — but no training code.

Cola2 matters less as a ChatGPT competitor than as a research signal. Diffusion and latent-generation approaches to language modeling are the most credible architectural alternative to autoregressive scaling, and the fact that ByteDance is shipping reproducible checkpoints — weights, evals and all — makes it one of the few such bets outsiders can actually inspect. At 388 downloads and 8 likes in its first days, the audience is so far specialist, but every serious diffusion-LM effort, from LLaDA to Mercury, started the same way.

For the broader open-weight race, the release fits a pattern: Chinese labs keep converting architectural research into inspectable artifacts while Western labs concentrate on closed frontier systems. Whether continuous-latent generation ever beats next-token prediction at scale is still an open question — Cola2's own paper shows the gap, not the win — but ByteDance has just made that question considerably easier for the research community to study.