UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via
  HUMAIN Chat
Large language models (LLMs) trained primarily on English corpora often
struggle to capture the linguistic and cultural nuances of Arabic. To address
this gap, the Saudi Data and AI Authority (SDAIA) introduced the ALLaM family
of Arabic-focused models. The most capable of these available to the public,
ALLaM-34B, was subsequently adopted by HUMAIN, who developed and deployed
HUMAIN Chat, a closed conversational web service built on this model. This
paper presents an expanded and refined UI-level evaluation of ALLaM-34B.
Using a prompt pack spanning modern standard Arabic, five regional dialects,
code-switching, factual knowledge, arithmetic and temporal reasoning, creative
generation, and adversarial safety, we collected 115 outputs (23 prompts times
5 runs) and scored each with three frontier LLM judges (GPT-5, Gemini 2.5 Pro,
Claude Sonnet-4). We compute category-level means with 95\% confidence
intervals, analyze score distributions, and visualize dialect-wise metric heat
maps. The updated analysis reveals consistently high performance on generation
and code-switching tasks (both averaging 4.92/5), alongside strong results in
MSA handling (4.74/5), solid reasoning ability (4.64/5), and improved dialect
fidelity (4.21/5). Safety-related prompts show stable, reliable performance of
(4.54/5). Taken together, these results position ALLaM-34B as a robust and
culturally grounded Arabic LLM, demonstrating both technical strength and
practical readiness for real-world deployment.
68b69c0626d3ff3f69d5fdc8
Omer Nacar
claimed_verified
2025-09-02T07:54:04.897Z
628f7a71dd993507cfcbe587
https://cdn-avatars.huggingface.co/v1/production/uploads/628f7a71dd993507cfcbe587/3frAcMgfBCx-OJsrdAhnb.png
Omartificial Intelligence Space
user
Omartificial-Intelligence-Space
UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via
  HUMAIN Chat
Large language models (LLMs) trained primarily on English corpora often
struggle to capture the linguistic and cultural nuances of Arabic. To address
this gap, the Saudi Data and AI Authority (SDAIA) introduced the ALLaM family
of Arabic-focused models. The most capable of these available to the public,
ALLaM-34B, was subsequently adopted by HUMAIN, who developed and deployed
HUMAIN Chat, a closed conversational web service built on this model. This
paper presents an expanded and refined UI-level evaluation of ALLaM-34B.
Using a prompt pack spanning modern standard Arabic, five regional dialects,
code-switching, factual knowledge, arithmetic and temporal reasoning, creative
generation, and adversarial safety, we collected 115 outputs (23 prompts times
5 runs) and scored each with three frontier LLM judges (GPT-5, Gemini 2.5 Pro,
Claude Sonnet-4). We compute category-level means with 95\% confidence
intervals, analyze score distributions, and visualize dialect-wise metric heat
maps. The updated analysis reveals consistently high performance on generation
and code-switching tasks (both averaging 4.92/5), alongside strong results in
MSA handling (4.74/5), solid reasoning ability (4.64/5), and improved dialect
fidelity (4.21/5). Safety-related prompts show stable, reliable performance of
(4.54/5). Taken together, these results position ALLaM-34B as a robust and
culturally grounded Arabic LLM, demonstrating both technical strength and
practical readiness for real-world deployment.
The evaluation of ALLaM-34B, an Arabic-focused LLM, demonstrates high performance across various tasks including generation, code-switching, MSA handling, reasoning, dialect fidelity, and safety, positioning it as a robust and culturally grounded model.
LLMs Arabic-focused models ALLaM-34B HUMAIN Chat UI-level evaluation prompt pack regional dialects code-switching factual knowledge arithmetic temporal reasoning creative generation adversarial safety LLM judges category-level means confidence intervals score distributions dialect-wise metric heat maps MSA handling reasoning ability dialect fidelity safety-related prompts