Introducing PTS Visualizer - an interactive tool for exploring how language models reason!
Visualize pivotal tokens, thought anchors, and reasoning circuits. See which tokens and sentences significantly impact success probability, explore embedding clusters, and trace reasoning step-by-step.
π One of the coolest parts about being an early Strawberry user has been the opportunity to build on the app at the ground floor.
The platform already has a ton of great integrations that let you interact with your external apps directly with tools, but I wanted to add the ability to do stuff in Slack as well.
πͺ So I took the base Anthropic Slack MCP server, added a whole bunch of new tools, and generalized it as an HTTP-based SSE-server and deployed it in like 2 minutes with Railway so that Strawberry could make use of it (as can Claude or any other MCP client).
Now, you can Chat with your Strawberry Companion (or Claude, or whatever) and do things like: β‘οΈ Get caught up across all of your Slack channels after a long weekend or noisy incident without having to read 20 threads in 10 different channels β‘οΈ Create, read, and edit Canvases, Messages, and Channels β‘οΈ Take any resources or content that you're using in your Chat and inject it directly into Slack without copy / paste
π I'm pretty pleased with the results, and I made a short demo video showing the results of the work (link in comments). The best part is, it's available on GitHub for anyone else to use too (link in the comments, instructions in the README). The setup takes about 5-10 minutes.
Recently, Essential AI released a new 8B base model EssentialAI/rnj-1 they highlighted the importance of data mix for pretraning -
"In the long run, we expect our methods to automatically represent, transform, and blend data to optimize measurable abilities in pre-training. Our work on modeling data taxonomies led to new approaches for jointly clustering and mixing data distributions under data repetition penalties. Many improvements in our STEM abilities can be traced back to this. "
What a trip. Just walked through @burtenshaw and @evalstate tutorial on adding Hugging Face Skills to your Claude Code agent so you can fine tune LLMs by chatting with AI.
These are the kinds of innovations that are going to help everyone benefit from the power of Artificial Intelligence. Well done gentlemen and thank you for sharing.
I just published Ellora - 6 production-ready LoRA recipes for enhancing LLMs with specific capabilities. Each recipe costs under $100 to run and includes complete training code, data generation, and evaluation.
The 6 Recipes: Recipe 1: Accuracy Recovery - Recover 75% of quantization losses with self-distillation Recipe 2: Reasoning LoRA - Add structured thinking with GRPO (0% to 60% adoption, 75% quality boost) Recipe 3: Tool Calling - Real execution on actual codebases Recipe 4: Context Extension - Scale from 32K to 2M tokens (61x increase) Recipe 5: Secure Code Generation - 97% vulnerability reduction using automated Semgrep analysis Recipe 6: Execution-Aware World Models - Teaching models runtime behavior
Why Recipes? Ellora provides methodologies, not frameworks. Use them with your existing tools (PEFT, LoRAX, vLLM, Unsloth, HuggingFace). Each recipe uses self-supervised data generation (Magpie approach) - no expensive human labeling required.
All recipes include Jupyter notebooks you can run immediately with clear success metrics.
π I keep seeing takes on LinkedIn from American business influencers melting down about Silicon Valley startup "dependence" on open-source Chinese models.
π€ Can anyone describe a credible scenario where these models can be leveraged by the Chinese government to endanger American security interests or am I right to believe that this is just Red Scare nonsense?
Introducing OpenEvolve Prompt Optimizer - a Space that automatically evolves and optimizes your prompts using OpenEvolve!
This tool uses OpenEvolve to iteratively improve prompts by testing them on real datasets and evolving better versions. No more manual prompt engineering guesswork - let OpenEvolve find the optimal prompts for you.
How it works: - Enter your initial prompt using {input} as a placeholder for dataset inputs - Input any HuggingFace dataset name you want to use for optimization - Specify the dataset split and field names for your use case - Click Optimize Prompt and the system will validate everything first - Compare your initial prompt vs the evolved best prompt side-by-side
π― Introducing Chayan: A Calibrated 4-Model LLM Router Achieving 69% Accuracy on RouterArena
We're excited to share Chayan, a cost-efficient LLM router that intelligently routes queries between 4 models to maximize accuracy while minimizing cost. Chayan just submitted to the RouterArena leaderboard and achieved 69.05% accuracy on the benchmark!
Chayan achieves impressive results on the RouterArena benchmark: β’ 69.05% accuracy (would rank #1 on current leaderboard) β’ $0.333 per 1K queries β’ +12.07pp improvement over all-mini baseline (56.98%) β’ 99% of perfect 2-model oracle performance at 57% lower cost
Compared to our previous 2-model router (61.43% accuracy), Chayan delivers +7.62pp improvement through smarter 4-model routing.
π§ How It Works
Chayan uses an Adaptive K-NN classifier with prototype memory to route between 4 models: β’ openai/gpt-4o-mini (fast & cheap) β’ google/gemini-2.5-flash-lite (balanced) β’ google/gemini-2.5-flash (capable) β’ openai/gpt-4o (most powerful)
π Getting Started
You can use Chayan directly from HuggingFace:
from adaptive_classifier import AdaptiveClassifier
These samples were created using reservoir sampling - an algorithm that guarantees statistically unbiased random samples from massive source datasets. This means results you get at the 1B token scale are representative of how these datasets behave at 100B+ token scales, letting you iterate quickly without the computational overhead.
The collection includes: - finePDFs-1B: High-quality textbook-style educational content - DCLM-baseline-1B: Filtered, diverse web content - FineWeb-Edu-1B: Curated educational web resources
We used these exact samples to run 50+ systematic experiments on dataset mixing strategies, ultimately discovering that a 50-30-20 mixture of finePDFs + DCLM-baseline + FineWeb-Edu achieves 90%+ of GPT-2's performance with just 1/10th the training data.
Whether you're researching optimal data mixtures, testing curriculum learning strategies, or just want to quickly prototype a pretraining run, these samples give you a solid foundation to start experimenting immediately.
A few months ago, I built a quick POC in Hugging Face that used a fine-tuned variant of OpenAI's OSS-20B model that I trained to convert the text from pre-reform Russian-language documents into modern Russian orthography.
β‘οΈ This morning, I launched novoyaz.io.
This is a production app, the frontend for which I built in like two hours with Lovable, that uses that same fine-tuned model for transliteration, but now has a bunch of extra features that make using it even easier (like taking and uploading pictures with your on-device camera for example π ).
π If you're a researcher, or know a researcher, for whom this app will improve their day-to-day workflows, please get in touch with me.
MARS Achieves Strong Results on Google DeepMind's IMO-Bench
We evaluated OptiLLM's MARS (Multi-Agent Reasoning System) approach on IMO-Bench, Google DeepMind's challenging mathematical reasoning benchmark with International Mathematical Olympiad-level problems.
What is MARS?
MARS is a multi-agent reasoning technique that works with any LLM. It uses 3 parallel reasoning agents that independently solve problems, then verifies their solutions through consensus and iterative refinement. The key advantage: it's model-agnostic and can be applied to any base model through OptiLLM's inference proxy.
Results on IMO-Bench:
AnswerBench (400 short-answer problems): MARS: 36.0% (144/400 correct) Baseline: 24.5% (98/400 correct) Improvement: +11.5pp across all domains
All results achieved using google/gemini-2.5-flash-lite-preview-09-2025 as the base model. The same MARS approach can enhance reasoning for any model through OptiLLM's OpenAI-compatible API.
Datasets available at: AnswerBench: huggingface.co/datasets/Hwilner/imo-answerbench ProofBench: huggingface.co/datasets/Hwilner/imo-proofbench
Try it yourself:
python optillm.py --approach mars --model google/gemini-2.5-flash-lite-preview-09-2025
On this day in 2019, OpenAI released the final GPT-2 model as part of their staged release. I still remember that November well - so much was happening, but GPT-2's release felt like a watershed moment for the field. It showed us what was possible with carefully trained language models.
To recreate some of that GPT-2 magic, I recently tackled an interesting challenge: can you pretrain a language model with just 1 billion tokens - roughly 1/10th of what GPT-2 used - and still get comparable performance? After 50+ systematic experiments testing different dataset mixtures, the answer is yes.
The result is codelion/gpt-2-70m, which achieves over 90% of GPT-2's benchmark performance despite being trained on 10x less data. The key was finding the optimal dataset composition: 50% high-quality textbook PDFs, 30% filtered web content, and 20% educational resources. It even beats GPT-2 on TruthfulQA (47.31% vs 40.69%).
The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix
We trained a GPT-2 model to 90%+ performance using just 1/10th the training data through 50+ systematic experiments on dataset mixing strategies.
Key Finding:
A static mix of 50% finePDFs + 30% DCLM-baseline + 20% FineWeb-Edu consistently outperforms complex curriculum learning approaches. Static mixing is simpler, faster, and avoids catastrophic failures from hard distribution shifts.
Results:
Our GPT-2-70M model (70M parameters, 1B tokens) scores 38.15% on benchmarks vs GPT-2's 39.13% - only 0.98 points behind despite 10x less data and 44% fewer parameters. It even beats GPT-2 on TruthfulQA (47.31% vs 40.69%).
The takeaway: careful dataset curation matters more than total data volume.
