AI Translates Science for Arabic Speakers

A new benchmark for evaluating how well artificial intelligence translates scientific texts into Arabic could help bridge a critical language gap for over 400 million Arabic speakers. The ASCAT corpus, developed by researchers, targets full scientific abstracts across domains like physics, mathematics, computer science, quantum mechanics, and artificial intelligence, averaging 141.7 words in English and 111.78 words in Arabic. Unlike existing resources that rely on short sentences or single-domain content, this dataset prioritizes depth and validation, making it a tool for assessing translation quality in complex, technical contexts where accuracy is paramount.

The researchers found that current AI models struggle with scientific Arabic translation, with performance gaps highlighting the need for specialized resources. In evaluations, three state-of-the-art large language models were tested: GPT-4o-mini scored a BLEU of 37.07, Gemini-3.0-Flash-Preview scored 30.44, and Qwen3-235B-A22B scored 23.68. This spread of up to 13.4 BLEU points demonstrates the benchmark's ability to distinguish between systems, revealing that even the best model has room for improvement in handling long-form scientific content. underscore the inherent difficulty of translating nuanced scientific discourse, where terminological precision and structural fidelity are essential.

To build the corpus, the team followed a systematic three-stage pipeline: data collection, multi-engine translation, and human validation. They gathered 500 scientific abstracts from diverse domains, ensuring representational diversity by random sampling. Each abstract was translated using three complementary approaches: generative AI via Gemini for contextual nuance, transformer-based models from Hugging Face for domain-adapted neural translations, and commercial APIs like Google Translate and DeepL as high-fluency baselines. This multi-engine strategy allowed for comparative analysis and leveraged the strengths of different translation architectures.

Human validation was a critical component, involving seven domain experts with graduate-level degrees in Arabic linguistics or relevant scientific fields. These validators used a structured checklist to assess translations at lexical, syntactic, and semantic levels, correcting errors in terminology, grammar, and meaning. The process ensured that judgments were made by subject-matter specialists, with disagreements resolved through consensus. The resulting dataset contains 67,293 English tokens and 60,026 Arabic tokens, with an Arabic vocabulary of 17,604 unique words, reflecting the language's morphological richness and higher lexical diversity compared to English.

Analysis of the corpus revealed key linguistic characteristics that pose s for machine translation. Arabic abstracts averaged 111.78 words with a standard deviation of 58.87, showing substantial length variability across domains. The Arabic side exhibited a Type-Token Ratio of 0.29, higher than English's 0.19, indicating greater lexical diversity due to Arabic's agglutinative properties. This complexity means translation models must handle a larger vocabulary space and more information per word, which can impact performance if not adequately addressed through techniques like subword tokenization.

Of this work are significant for Arabic-speaking researchers and professionals who face accessibility barriers in scientific discourse. By providing a high-quality benchmark, ASCAT enables rigorous evaluation of translation systems, potentially leading to improved tools for accessing global scientific literature. This could enhance education, research, and innovation in Arabic-speaking regions, where language gaps have historically limited participation in international scientific communities. The corpus's focus on full abstracts, rather than isolated sentences, also supports training models to produce coherent, multi-sentence discourse.

However, the study acknowledges limitations that future research must address. The dataset size is limited to 500 abstracts due to the intensive human validation required, prioritizing quality over scale. Domain distribution is uneven, which may affect the generalizability of models to underrepresented scientific fields. Evaluation relied on automatic metrics like BLEU and ROUGE, which do not fully capture qualitative aspects such as semantic adequacy or terminological felicity. Additionally, scientific translation presents inherent s like interdisciplinary terminological ambiguity and handling of non-standardized terms, which current systems still struggle with.

Future work should focus on expanding the corpus for better domain balance, incorporating large-scale human evaluation to complement automatic metrics, and fine-tuning domain-adapted models to assess translation improvements. These steps could advance machine translation systems toward more reliable cross-disciplinary scientific communication. The researchers believe ASCAT represents a meaningful step toward closing the language gap, serving as a foundation for future advances in Arabic machine translation research.