A new evaluation benchmark called Alyah ⭐️ has been introduced to assess how well large language models (LLMs) can understand and generate the Emirati dialect of Arabic. The benchmark aims to fill a gap in current Arabic NLP evaluation, which often focuses on Modern Standard Arabic or only major dialects like Egyptian or Levantine.
Developed by a team of researchers, Alyah provides a set of standardized tests covering tasks such as dialect identification, translation, and natural language understanding in Emirati Arabic. Early results show that leading Arabic LLMs still struggle with the dialect, highlighting the need for more dialect-specific training data and fine-tuning.
The benchmark is named after a traditional Emirati folk song and is available open-source for researchers and developers. The creators hope Alyah will drive progress toward more inclusive Arabic language AI.