CENO-80M-131k

CENO-80M-131k is the long-context (131k) checkpoint of the 80M CENO DNA foundation model — a causal language model over genomic sequence built on a Nemotron-H Mamba / Attention / Mixture-of-Experts hybrid backbone (no MSA inputs).

It is part of the CENO DNA foundation model family. Model code, the VEP pipeline, and a generation demo live in the companion CENO code repository. This checkpoint is standalone-loadable with trust_remote_code=True — the model code is bundled here.

Model details

Family	CENO (base)
Training stage	Long-context extension (stage 3)
Training context length	131,072 tokens
Parameters	79.3M
Precision	bfloat16
model_type	ceno
Architecture class	CENOForCausalLM
Auto-map (model)	modeling_ceno.CENOForCausalLM
Auto-map (tokenizer)	ceno_tokenizer.CENOCharLevelTokenizer

CENO's Mamba / Attention / MoE backbone has no fixed context window. The training context length above is the sequence length this checkpoint was trained at — not a hard limit. max_position_embeddings in config.json is nominal and non-restricting.

Architecture

Property	Value
Hidden layers	9
Hidden size	512
Attention heads	8
Intermediate size	2048
Experts (MoE)	8 (top-2 per token)
Vocabulary	512 (byte / character-level)

The backbone is a Mamba / Attention / Mixture-of-Experts hybrid (Nemotron-H architecture). The tokenizer is character-level, mapping DNA bases to their ASCII byte codes.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

ckpt = "CladeTeam/CENO-80M-131k"
model = AutoModelForCausalLM.from_pretrained(ckpt, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)

ids = tokenizer.encode("ATCGATCG", return_tensors="pt")
# out = model.generate(ids, max_new_tokens=128)   # needs a CUDA GPU (Mamba kernels)

The Mamba layers require CUDA kernels, so forward passes and generation need a GPU. Config, tokenizer, and weight loading are CPU-safe.

Intended use

• Base checkpoints (CENO-*) — genomic-sequence generation and embedding extraction; downstream adaptation (fine-tuning, probing) for genomics tasks.
• MSA checkpoints (CENO-P-*) — variant effect prediction (VEP) by scoring wild-type vs. variant sequences with delta log-likelihood. See the TraitGym VEP example in the CENO code repository.

License

Apache-2.0. The bundled model code is derived from NVIDIA's Nemotron-H Hugging Face implementation (Apache-2.0); the tokenizer is derived from the Arc Institute Evo2 CharLevelTokenizer (Apache-2.0). See the LICENSE and NOTICE files in this repository for full attribution.