Technical Report: Full-Stack Fine-Tuning for the Q Programming Language
Even though large language models are becoming increasingly capable, it is
still unreasonable to expect them to excel at tasks that are under-represented
on the Internet. Leveraging LLMs for specialized applications, particularly in
niche programming languages and private domains, remains challenging and
largely unsolved. In this work, we address this gap by presenting a
comprehensive, open-source approach for adapting LLMs to the Q programming
language, a popular tool in quantitative finance that is much less present on
the Internet compared to Python, C, Java, and other ``mainstream" languages and
is therefore not a strong suit of general-purpose AI models. We introduce a new
Leetcode style evaluation dataset for Q, benchmark major frontier models on the
dataset, then do pretraining, supervised fine tuning, and reinforcement
learning to train a suite of reasoning and non-reasoning models based on the
Qwen-2.5 series, spanning five parameter sizes (1.5B, 3B, 7B, 14B, 32B). Our
best model achieves a pass@1 accuracy of 59 percent on our Q benchmark,
surpassing the best-performing frontier model, Claude Opus-4 by 29.5 percent.
Additionally, all models, even our 1.5B model, outperform GPT-4.1 on this task.
In addition to releasing models, code, and data, we provide a detailed
blueprint for dataset construction, model pretraining, supervised fine-tuning,
and reinforcement learning. Our methodology is broadly applicable, and we
discuss how these techniques can be extended to other tasks, including those
where evaluation may rely on soft or subjective signals.
689b62c6fab6fdd2e52ac72f
Brendan R. Hogan
claimed_verified
2025-08-18T06:59:13.635Z
63445b570f69ad8aa6198f01
/avatars/a473341f8491ecf374e6c547b4858e5f.svg
Brendan Rappazzo
user
bhogan
689b62c6fab6fdd2e52ac730
Will Brown
689b62c6fab6fdd2e52ac731
Adel Boyarsky
689b62c6fab6fdd2e52ac732
Anderson Schneider
689b62c6fab6fdd2e52ac733
Yuriy Nevmyvaka
Technical Report: Full-Stack Fine-Tuning for the Q Programming Language
Even though large language models are becoming increasingly capable, it is
still unreasonable to expect them to excel at tasks that are under-represented
on the Internet. Leveraging LLMs for specialized applications, particularly in
niche programming languages and private domains, remains challenging and
largely unsolved. In this work, we address this gap by presenting a
comprehensive, open-source approach for adapting LLMs to the Q programming
language, a popular tool in quantitative finance that is much less present on
the Internet compared to Python, C, Java, and other ``mainstream" languages and
is therefore not a strong suit of general-purpose AI models. We introduce a new
Leetcode style evaluation dataset for Q, benchmark major frontier models on the
dataset, then do pretraining, supervised fine tuning, and reinforcement
learning to train a suite of reasoning and non-reasoning models based on the
Qwen-2.5 series, spanning five parameter sizes (1.5B, 3B, 7B, 14B, 32B). Our
best model achieves a pass@1 accuracy of 59 percent on our Q benchmark,
surpassing the best-performing frontier model, Claude Opus-4 by 29.5 percent.
Additionally, all models, even our 1.5B model, outperform GPT-4.1 on this task.
In addition to releasing models, code, and data, we provide a detailed
blueprint for dataset construction, model pretraining, supervised fine-tuning,
and reinforcement learning. Our methodology is broadly applicable, and we
discuss how these techniques can be extended to other tasks, including those
where evaluation may rely on soft or subjective signals.
A comprehensive approach is presented for adapting large language models to the Q programming language, achieving superior performance compared to existing models on a newly created benchmark dataset.
large language models Q programming language Leetcode style evaluation dataset pretraining supervised fine tuning reinforcement learning Qwen-2.5 series pass@1 accuracy Claude Opus-4 GPT-4.1 dataset construction model pretraining supervised fine-tuning reinforcement learning