arc-state-norman-gears-corrected

Fine-tuned Arc State checkpoint on the Norman 2019 K562 perturbation dataset, produced by VCBench v1.0.0 to reproduce Arc State's perturbation-prediction performance on the disjoint GEARS train/test split.

What this is

This checkpoint was fine-tuned on Norman 2019 K562 (GSE133344, via the GEARS API) using the GEARS simulation split (seed=1): 139 training perturbations and 107 held-out test perturbations with zero overlap — the same split used by every other foundation model evaluated in VCBench. The binding split is enumerated in data_split.toml.

Headline metric: PRR = 0.402 on the 107 held-out Norman test perturbations (vcbench evaluate_dim_a, real-control anchor — the canonical convention for VC Level decisions). Cross-validated by upstream cell-eval pearson_delta = 0.408 to 2e-6 absolute under matched anchor convention (see Cross-evaluator anchor convention below).

Results

Evaluated on the 107 GEARS test perturbations with both cell-eval (Arc Institute's official evaluator) and vcbench.dimensions.dim_a_perturbation.evaluate_dim_a:

Evaluator / convention	mean Pearson R on Δ-expression (PRR)	Direction score (top-20 DEG sign-agreement)
vcbench.evaluate_dim_a (real anchor — canonical)	0.4021	0.7514
vcbench.evaluate_dim_a (pred anchor)	0.4076	0.7846
cell-eval pearson_delta	0.4076	—

vcbench (pred-anchor) and cell-eval agree to 2e-6 absolute. Per-perturbation results are in eval_per_perturbation.csv; aggregate metrics in eval_aggregate.csv.

Cross-evaluator anchor convention (vcbench ↔ cell-eval)

VCBench's evaluate_dim_a and Arc Institute's upstream cell-eval pearson_delta differ in one design choice: the control-anchor convention used to form the Δ-expression vectors fed into per-perturbation Pearson R.

Convention	Definition	Role
Real anchor (vcbench default, control_anchor="real")	pred_delta = pert_pred − ctrl_real and real_delta = pert_real − ctrl_real (both anchored on the observed real control)	Canonical — used for VC Level decisions; the right convention for cross-model benchmarking (no per-model free baseline).
Pred anchor (cell-eval, control_anchor="pred")	pred_delta = pert_pred − ctrl_pred (model's own predicted control), real_delta = pert_real − ctrl_real	For cell-eval cross-validation. Reproduces upstream cell-eval pearson_delta to 1e-6 absolute.

Under matched conventions the two evaluators agree to numerical precision (locked by the Dim A evaluator tests in the source repository).

Loading

from vcbench.models import ArcState

arc = ArcState()
arc.load_pretrained("final.ckpt")                   # validates the disjoint split
result = arc.run_dim_a()                             # full pipeline → DimAResult
print(f"PRR: {result.mean_pearson_r_delta:.4f}")     # ≈ 0.4021 (real anchor, canonical)

Training recipe

Field	Value
Base model	arc-state==0.10.2 (state model variant)
Dataset	Norman 2019 K562 (GSE133344, via GEARS API)
Train perturbations	139 (per [fewshot."norman.A549"].train in data_split.toml)
Test perturbations	107 (GEARS simulation split, seed=1)
Train/test split	disjoint — 0 perturbations / 0 cells shared
Architecture	LLaMA bidirectional backbone, num_hidden_layers=8, hidden_dim=768, cell_set_len=512, n_attention_heads=12
Total params	110 M (86 M trainable)
Optimizer	AdamW
Learning rate	1×10⁻⁴
Batch size	8
Max steps	40,000
Loss	energy distance (samples loss)
Random seed	42
Hardware	NVIDIA A40 (46 GB), CUDA 12.4
Wall clock	~4h12m end-to-end (training; predict + eval ~10 min on top)

VC Level

Under the VCBench pre-registration, Arc State scores VC Level 1 on Norman: PRR 0.402 exceeds the no-change baseline (PRR 0.000) on Dim A but does not exceed the mean-prediction baseline (PRR 0.579).

Files

File	Size	Description
final.ckpt	1.13 GB	Final model state at step 40,000 (the canonical artefact)
best.ckpt	1.13 GB	Model state at lowest validation loss
training_config.yaml	2.6 KB	Resolved Hydra config used by arc-state v0.10.2 at runtime
data_split.toml	4.2 KB	The GEARS-split TOML — enumerates the 139 train / 107 test perturbations
eval_aggregate.csv	3.6 KB	Aggregate cell-eval metrics across all 107 test perturbations
eval_per_perturbation.csv	41 KB	Per-perturbation cell-eval metrics (107 rows)

Provenance

• Source repository: https://github.com/AppliedScientific/VCBench (tag v1.0.0)
• Pre-registration: configs/pre_registration.yaml in the source repository
• Manuscript: VCBench (2026)

Citation

@misc{vcbench-arc-state-norman-gears,
  author       = {{VCBench contributors}},
  title        = {Arc State Norman GEARS-split checkpoint},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/appliedscientific/arc-state-norman-gears-corrected}},
}

License

MIT — same as the upstream ArcInstitute/state codebase.