Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations
  and Sentences
Conversion of spoken mathematical expressions is a challenging task that
involves transcribing speech into a strictly structured symbolic representation
while addressing the ambiguity inherent in the pronunciation of equations.
Although significant progress has been achieved in automatic speech recognition
(ASR) and language models (LM), the problem of converting spoken mathematics
into LaTeX remains underexplored. This task directly applies to educational and
research domains, such as lecture transcription or note creation. Based on ASR
post-correction, prior work requires 2 transcriptions, focuses only on isolated
equations, has a limited test set, and provides neither training data nor
multilingual coverage. To address these issues, we present the first fully
open-source large-scale dataset, comprising over 66,000 human-annotated audio
samples of mathematical equations and sentences in both English and Russian,
drawn from diverse scientific domains. In addition to the ASR post-correction
models and few-shot prompting, we apply audio language models, demonstrating
comparable character error rate (CER) results on the MathSpeech benchmark (28%
vs. 30%) for the equations conversion. In contrast, on the proposed
S2L-equations benchmark, our models outperform the MathSpeech model by a
substantial margin of more than 40 percentage points, even after accounting for
LaTeX formatting artifacts (27% vs. 64%). We establish the first benchmark for
mathematical sentence recognition (S2L-sentences) and achieve an equation CER
of 40%. This work lays the groundwork for future advances in multimodal AI,
with a particular focus on mathematical content recognition.
6899d9f6996e0a645a52a7e4
Dmitrii Korzh
6899d9f6996e0a645a52a7e5
Dmitrii Tarasov
6899d9f6996e0a645a52a7e6
Artyom Iudin
6899d9f6996e0a645a52a7e7
Elvir Karimov
6899d9f6996e0a645a52a7e8
Matvey Skripkin
claimed_verified
2025-08-21T10:34:22.780Z
6626c5d0a329de26e7eb16fa
/avatars/124f389f768fb666efd8b5a9b54c3b3c.svg
Matvey Skripkin
user
barracuda049
6899d9f6996e0a645a52a7e9
Nikita Kuzmin
6899d9f6996e0a645a52a7ea
Andrey Kuznetsov
6899d9f6996e0a645a52a7eb
Oleg Y. Rogov
claimed_verified
2025-08-28T08:56:57.892Z
6453df2b908e259483c12bdb
https://cdn-avatars.huggingface.co/v1/production/uploads/6453df2b908e259483c12bdb/j0QZDDWffeZNiTMEgQTtS.jpeg
Oleg Y. Rogov
user
qubitter
6899d9f6996e0a645a52a7ec
Ivan Oseledets
Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations
  and Sentences
Conversion of spoken mathematical expressions is a challenging task that
involves transcribing speech into a strictly structured symbolic representation
while addressing the ambiguity inherent in the pronunciation of equations.
Although significant progress has been achieved in automatic speech recognition
(ASR) and language models (LM), the problem of converting spoken mathematics
into LaTeX remains underexplored. This task directly applies to educational and
research domains, such as lecture transcription or note creation. Based on ASR
post-correction, prior work requires 2 transcriptions, focuses only on isolated
equations, has a limited test set, and provides neither training data nor
multilingual coverage. To address these issues, we present the first fully
open-source large-scale dataset, comprising over 66,000 human-annotated audio
samples of mathematical equations and sentences in both English and Russian,
drawn from diverse scientific domains. In addition to the ASR post-correction
models and few-shot prompting, we apply audio language models, demonstrating
comparable character error rate (CER) results on the MathSpeech benchmark (28%
vs. 30%) for the equations conversion. In contrast, on the proposed
S2L-equations benchmark, our models outperform the MathSpeech model by a
substantial margin of more than 40 percentage points, even after accounting for
LaTeX formatting artifacts (27% vs. 64%). We establish the first benchmark for
mathematical sentence recognition (S2L-sentences) and achieve an equation CER
of 40%. This work lays the groundwork for future advances in multimodal AI,
with a particular focus on mathematical content recognition.
A new open-source dataset and models for converting spoken mathematical expressions into LaTeX improve accuracy over existing benchmarks, supporting both equations and sentences in multiple languages.
automatic speech recognition language models ASR post-correction audio language models character error rate MathSpeech benchmark S2L-equations benchmark S2L-sentences benchmark multimodal AI mathematical content recognition