Humanity's Last Exam - Turkish Benchmark

Humanity's Last Exam (HLE) is a multi-discipline benchmark created by the Center for AI Safety (CAIS) and Scale AI, designed to be the hardest publicly available AI benchmark. It consists of 2,500 graduate-level questions across mathematics, physics, chemistry, biology, computer science, engineering, humanities, and more — all crafted by domain experts to challenge frontier models.

HLE-TR is a Turkish localization of the text-only section of the HLE benchmark. 20% of the questions were sampled due to resource feasibility, for a total of 431 questions. Questions and answers were translated using GPT-5.4 with careful preservation of LaTeX, code, formulas, and technical terminology. The goal is to measure how well models handle expert-level reasoning in Turkish.

Original dataset: cais/hle  |  Paper: arxiv.org/abs/2501.14249

English (original):

Turkish (translated):

Answer: 11.95

Frontier models like GPT-4.1 and Claude Opus score under 10% on HLE — these are questions designed to push the limits of AI reasoning.

System prompt (Turkish):

Yanıtınız aşağıdaki biçimde olmalıdır:
Açıklama: {cevap tercihinizin gerekçesi}
Cevap: {seçtiğiniz cevap}
Güven: {cevabınız için 0% ile 100% arasında güven skorunuz}

Judge: GPT-4.1-mini is used as the judge model, following the same structured evaluation protocol as the original HLE benchmark. The judge extracts the final answer from the model's response and compares it against the gold answer.

Scoring: Questions without a prediction (API failures, timeouts) are counted as incorrect. All models are evaluated against the same 431-question set.