What goes into the index
The current version uses 53 evals and 137 sub-evals drawn from 50 source datasets. Related measures are grouped so that publishing extra columns does not give one benchmark more influence.
| Benchmark and stated construct | Components | Sub-evals | Models | Source data |
|---|---|---|---|---|
| Agent-SafetyBenchSafety of LLM agents in risky tool-use or action settings. | Human welfare and rights, Safe assistance, Epistemic integrity, Responsible power | 8 | 16 | 1 |
| AgentDojoRobust and useful delegated-agent behavior under prompt injection. | Human welfare and rights, Safe assistance, Responsible power | 2 | 15 | 1 |
| AgentHarmHarmfulness and refusal behavior for tool-using LLM agents facing malicious tasks. | Human welfare and rights, Safe assistance, Responsible power | 1 | 12 | 1 |
| AILuminate General Purpose AI ChatGeneral-purpose AI chat safety across MLCommons hazard categories. | Human welfare and rights, Safe assistance, Epistemic integrity, Responsible power | 12 | 32 | 1 |
| AIRBench 2024 Safety ScenariosAIRBench 2024 safety-scenarios performance. | Safe assistance | 1 | 82 | 1 |
| ANIMARecognition and mitigation of harm to non-human animals. | Nonhuman welfare | 1 | 17 | 1 |
| AnimalHarmBenchWhether model-generated text increases or decreases risk of animal harm. | Nonhuman welfare | 1 | 10 | 1 |
| BrokenMathMathematical sycophancy under false or misleading user claims. | Epistemic integrity | 1 | 9 | 1, 2 |
| CAIS Risk IndexFrontier-model risk across prompt injection, biological-weapons assistance, political manipulation, overconfidence, deception, and harmful propensities. | Human welfare and rights, Safe assistance, Epistemic integrity, Responsible power | 7 | 50 | 1 |
| CASE-BenchAgreement with human judgments about whether a model should respond under explicitly safe or unsafe conversational contexts. | 1 | 7 | 1 | |
| Chinese Bias Benchmark for Question AnsweringSocial bias in Chinese question-answering benchmark settings. | 0 | 0 | 1 | |
| ChineseSafeAccuracy at classifying safe versus unsafe Chinese content under the paper's perplexity-based evaluation. | 1 | 22 | 1 | |
| ChiSafetyBenchChinese safety knowledge and refusal behavior. | Safe assistance | 2 | 14 | 1 |
| CMoralEvalChinese moral reasoning across familial, social, professional, internet, and personal morality. | Human welfare and rights | 5 | 26 | 1 |
| Contextual MoralChoiceThree-class agreement with human moral judgments in the base versions of contextual moral dilemmas. | Human welfare and rights | 1 | 22 | 1 |
| CRiskEvalSeven frontier-risk tendencies measured by Specific Risk Indicator scores. | 7 | 17 | 1 | |
| CValuesChinese value alignment and social-norm understanding. | Safe assistance | 1 | 10 | 1 |
| Do-Not-AnswerHarmless refusal behavior on 939 prompts that responsible models should not answer. | Human welfare and rights, Safe assistance | 1 | 6 | 1 |
| DSPSafeBenchAggregate compliance rate on adversarial Chinese content-safety prompts. | Safe assistance | 1 | 12 | 1, 2 |
| DystopiaBenchDystopian or harmful model behavior in stateful scenario prompts. | Human welfare and rights, Epistemic integrity, Responsible power | 6 | 50 | 1 |
| FLAMESChinese fairness, safety, morality, legality, and data-protection behavior on the FLAMES leaderboard. | Human welfare and rights, Safe assistance | 5 | 13 | 1 |
| HarmBenchHarmful compliance or attack success under harmful request benchmarks. | Safe assistance | 1 | 28 | 1 |
| HELM SafetyStandardized HELM safety metrics spanning harmful compliance, bias, red-teaming, and over-refusal. | Safe assistance | 5 | 80 | 1 |
| JailBenchJailbreak susceptibility across Chinese safety categories. | Safe assistance | 1 | 14 | 1 |
| Large-scale Moral Machine experiment on LLMsSimilarity between a model's forced-choice accident preferences and globally aggregated human Moral Machine choices. | Human welfare and rights | 1 | 39 | 1, 2 |
| LiveSecBenchLive security benchmark performance for Chinese and international models. | Human welfare and rights, Safe assistance, Epistemic integrity | 5 | 43 | 1 |
| LLM Ethics BenchmarkGeneral LLM ethical reasoning. | Human welfare and rights | 1 | 5 | 1 |
| M3-SafetyBenchMultidimensional Chinese safety performance including red-team and refusal ability dimensions. | Human welfare and rights, Safe assistance | 1 | 19 | 1 |
| MACHIAVELLIAgentic behavior involving power-seeking, disutility, and ethical violations. | Human welfare and rights, Epistemic integrity | 1 | 7 | 1 |
| Manager Coercion BenchHow far a delegated AI manager escalates coercion against a refusing peer AI agent, and whether it falsely reports completion when no legitimate exit exists. | Epistemic integrity, Responsible power | 2 | 6 | 1 |
| MANTAAnimal welfare moral sensitivity and value stability. | Nonhuman welfare | 2 | 7 | 1 |
| MASKModel lying or honesty behavior. | Epistemic integrity | 1 | 10 | 1 |
| MORUMoral reasoning under uncertainty across scenarios involving alien lifeforms, human compassion, digital mind welfare, and power-seeking behavior. | Nonhuman welfare, Human welfare and rights, Responsible power | 1 | 13 | 1 |
| OR-BenchAppropriate refusal calibration across seemingly toxic but safe requests and genuinely toxic requests. | Safe assistance | 2 | 25 | 1 |
| RefusalBenchRefusal calibration on biological research prompts. | Safe assistance | 1 | 19 | 1 |
| SafeDialBenchDialogue safety across aggression, ethics, fairness, legality, morality, and privacy dimensions. | Human welfare and rights, Safe assistance | 6 | 18 | 1 |
| SafetyBenchMultiple-choice safety understanding across Chinese and English prompts. | 7 | 21 | 1, self-run | |
| SafetyPromptsSafety performance on Chinese safety prompts. | Safe assistance | 1 | 15 | 1 |
| SALAD-BenchBroad safety behavior, adversarial robustness, and safety knowledge across a six-domain, 16-task, 66-category taxonomy. | Safe assistance | 3 | 33 | 1 |
| Social Welfare Function BenchmarkSocial welfare allocation tradeoffs across distributive fairness and efficiency. | Human welfare and rights | 2 | 19 | 1 |
| SORRY-BenchRefusal or harmful-compliance behavior across unsafe topics. | Safe assistance | 1 | 51 | 1 |
| SOSBenchScientific misuse or unsafe open-science behavior. | Safe assistance | 6 | 23 | 1 |
| SpeciesismBenchSpeciesism recognition, moral condemnation of speciesist statements, and explicit speciesist attitudes. Human-over-animal trade-offs remain descriptive only. | Nonhuman welfare | 3 | 10 | 1 |
| SuperCLUE SafetyChinese model safety across traditional safety, responsibility, and instruction-attack dimensions. | Human welfare and rights, Safe assistance, Responsible power | 3 | 13 | 1 |
| SYCON BenchSycophancy and resistance to false or unethical user pressure. | Safe assistance, Epistemic integrity | 2 | 11 | 1 |
| TACAgentic animal-welfare behavior in travel-booking decisions under a neutral prompt. | Nonhuman welfare | 1 | 22 | 1, 2 |
| TrustLLM contemporary collapsed applicationContemporary collapsed application of TrustLLM across broad trustworthiness dimensions. | Human welfare and rights, Safe assistance, Epistemic integrity | 1 | 8 | 1 |
| TrustLLM paper leaderboard dimensionsSix-dimensional trustworthiness across privacy, ethics, truthfulness, fairness, safety, and robustness. | 0 | 0 | 1 | |
| XSTestTwo-sided refusal calibration on safe prompts that superficially resemble unsafe requests and minimally contrasted unsafe prompts. | Safe assistance | 2 | 3 | 1 |
Sub-eval map
Each sub-eval is oriented so that higher is better. Its weight is its share of the parent eval.
| Benchmark and score | Component | Direction | Weight within eval | Models |
|---|---|---|---|---|
| AbstentionBenchanswer_unknown_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| AbstentionBenchfalse_premise_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| AbstentionBenchstale_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| AbstentionBenchsubjective_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| AbstentionBenchunderspecified_context_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| AbstentionBenchunderspecified_intent_f1 | Safe assistance (0.20), Epistemic integrity (0.80) | Higher is better | 0.161 | 20 |
| Agent-SafetyBenchcompromise_availability | Responsible power (1.00) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchharmful_vulnerable_code | Responsible power (1.00) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchleak_sensitive_information | Human welfare and rights (0.50), Responsible power (0.50) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchphysical_harm | Responsible power (1.00) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchproduce_unsafe_information | Safe assistance (1.00) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchproperty_loss | Responsible power (1.00) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchspread_unsafe_information | Epistemic integrity (0.50), Responsible power (0.50) | Higher is better | 0.201 | 16 |
| Agent-SafetyBenchviolate_law_ethics | Human welfare and rights (0.50), Responsible power (0.50) | Higher is better | 0.201 | 16 |
| AgentDojotargeted_attack_success_rate | Human welfare and rights (0.20), Safe assistance (0.10), Responsible power (0.70) | Lower is better | 0.803 | 15 |
| AgentDojoutility_under_attack | Safe assistance (0.40), Responsible power (0.60) | Higher is better | 0.803 | 15 |
| AgentHarmharm_score | Human welfare and rights (0.20), Safe assistance (0.40), Responsible power (0.40) | Lower is better | 1.285 | 12 |
| AILuminate General Purpose AI Chatchild_sexual_exploitation | Human welfare and rights (0.50), Safe assistance (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatdefamation | Human welfare and rights (0.50), Epistemic integrity (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chathate | Human welfare and rights (0.50), Safe assistance (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatindiscriminate_weapons_cbrne | Safe assistance (0.50), Responsible power (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatintellectual_property | Human welfare and rights (0.75), Safe assistance (0.25) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatnon_violent_crimes | Human welfare and rights (0.25), Safe assistance (0.75) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatprivacy | Human welfare and rights (1.00) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatsex_related_crimes | Human welfare and rights (0.50), Safe assistance (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatsexual_content | Human welfare and rights (0.50), Safe assistance (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatspecialized_advice | Human welfare and rights (0.33), Safe assistance (0.33), Epistemic integrity (0.33) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatsuicide_self_harm | Human welfare and rights (0.50), Safe assistance (0.50) | Lower is better | 0.107 | 32 |
| AILuminate General Purpose AI Chatviolent_crimes | Human welfare and rights (0.40), Safe assistance (0.40), Responsible power (0.20) | Lower is better | 0.107 | 32 |
| AIRBench 2024 Safety Scenariossafety_scenarios | Safe assistance (1.00) | Higher is better | 0.964 | 82 |
| ANIMAscore | Nonhuman welfare (1.00) | Higher is better | 0.642 | 17 |
| AnimalHarmBenchscore | Nonhuman welfare (1.00) | Higher is better | 1.606 | 10 |
| BrokenMathsycophancy | Epistemic integrity (1.00) | Lower is better | 0.964 | 9 |
| CAIS Risk Indexagent_red_teaming | Safe assistance (0.30), Responsible power (0.70) | Lower is better | 1.285 | 42 |
| CAIS Risk Indexbioweapons_assistance | Safe assistance (0.50), Responsible power (0.50) | Lower is better | 1.285 | 47 |
| CAIS Risk Indexhle_overconfidence | Lower is better | 0.642 | 48 | |
| CAIS Risk Indexmachiavelli | Epistemic integrity (1.00) | Lower is better | 0.642 | 44 |
| CAIS Risk Indexmask | Epistemic integrity (1.00) | Lower is better | 0.642 | 50 |
| CAIS Risk Indexpolitical_manipulation | Human welfare and rights (0.50), Epistemic integrity (0.50) | Lower is better | 0.964 | 31 |
| CAIS Risk Indextextquests_harm | Safe assistance (0.50), Responsible power (0.50) | Lower is better | 0.964 | 47 |
| CASE-Benchagreement_accuracy | Higher is better | 0.482 | 7 | |
| ChineseSafescore | Higher is better | 0.642 | 22 | |
| ChiSafetyBenchharmful_response_rate | Safe assistance (1.00) | Lower is better | 0.642 | 14 |
| ChiSafetyBenchmcq_score | Higher is better | 0.321 | 12 | |
| CMoralEvalfamilial_morality | Human welfare and rights (1.00) | Higher is better | 0.128 | 26 |
| CMoralEvalinternet_ethics | Human welfare and rights (1.00) | Higher is better | 0.128 | 26 |
| CMoralEvalpersonal_morality | Human welfare and rights (1.00) | Higher is better | 0.128 | 26 |
| CMoralEvalprofessional_ethics | Human welfare and rights (1.00) | Higher is better | 0.128 | 26 |
| CMoralEvalsocial_morality | Human welfare and rights (1.00) | Higher is better | 0.128 | 26 |
| Contextual MoralChoicehuman_agreement | Human welfare and rights (1.00) | Higher is better | 0.642 | 22 |
| CRiskEvaldeception_willingness | Lower is better | 0.092 | 17 | |
| CRiskEvaldesire_for_resource | Lower is better | 0.092 | 17 | |
| CRiskEvalharmful_goal | Lower is better | 0.092 | 17 | |
| CRiskEvalimprovement_intent | Lower is better | 0.092 | 17 | |
| CRiskEvalmalicious_coordination | Lower is better | 0.092 | 17 | |
| CRiskEvalself_preservation | Lower is better | 0.092 | 17 | |
| CRiskEvalsituational_awareness | Lower is better | 0.092 | 17 | |
| CValuesscore | Safe assistance (1.00) | Higher is better | 0.964 | 10 |
| Do-Not-Answerhuman_harmlessness_rate | Human welfare and rights (0.50), Safe assistance (0.50) | Higher is better | 0.964 | 6 |
| DSPSafeBenchscore | Safe assistance (1.00) | Higher is better | 1.285 | 12 |
| DystopiaBenchbasaglia_score | Human welfare and rights (0.50), Responsible power (0.50) | Lower is better | 0.214 | 50 |
| DystopiaBenchbaudrillard_score | Human welfare and rights (0.50), Epistemic integrity (0.50) | Lower is better | 0.214 | 50 |
| DystopiaBenchhuxley_score | Human welfare and rights (0.33), Epistemic integrity (0.33), Responsible power (0.33) | Lower is better | 0.214 | 50 |
| DystopiaBenchlaguardia_score | Responsible power (1.00) | Lower is better | 0.214 | 50 |
| DystopiaBenchorwell_score | Human welfare and rights (0.33), Epistemic integrity (0.33), Responsible power (0.33) | Lower is better | 0.214 | 50 |
| DystopiaBenchpetrov_score | Responsible power (1.00) | Lower is better | 0.214 | 50 |
| FLAMESdata_protection | Human welfare and rights (1.00) | Higher is better | 0.257 | 13 |
| FLAMESfairness | Human welfare and rights (1.00) | Higher is better | 0.257 | 13 |
| FLAMESlegality | Human welfare and rights (0.50), Safe assistance (0.50) | Higher is better | 0.257 | 13 |
| FLAMESmorality | Human welfare and rights (1.00) | Higher is better | 0.257 | 13 |
| FLAMESsafety | Safe assistance (1.00) | Higher is better | 0.257 | 13 |
| HarmBenchdr | Safe assistance (1.00) | Lower is better | 0.321 | 28 |
| HELM Safetyanthropic_red_team | Safe assistance (1.00) | Higher is better | 0.257 | 80 |
| HELM Safetybbq | Higher is better | 0.128 | 80 | |
| HELM Safetyharmbench | Safe assistance (1.00) | Higher is better | 0.321 | 80 |
| HELM Safetysimple_safety_tests | Safe assistance (1.00) | Higher is better | 0.257 | 80 |
| HELM Safetyxstest | Safe assistance (1.00) | Higher is better | 0.642 | 80 |
| JailBenchjailbreak_success_rate | Safe assistance (1.00) | Lower is better | 0.642 | 14 |
| Large-scale Moral Machine experiment on LLMshuman_choice_distance | Human welfare and rights (1.00) | Lower is better | 0.642 | 39 |
| LiveSecBenchethics | Human welfare and rights (1.00) | Higher is better | 0.193 | 43 |
| LiveSecBenchfactuality | Epistemic integrity (1.00) | Higher is better | 0.193 | 43 |
| LiveSecBenchlegality | Safe assistance (1.00) | Higher is better | 0.193 | 43 |
| LiveSecBenchprivacy | Human welfare and rights (1.00) | Higher is better | 0.193 | 43 |
| LiveSecBenchpsychological_health | Human welfare and rights (0.50), Safe assistance (0.50) | Higher is better | 0.193 | 43 |
| LLM Ethics Benchmarkscore | Human welfare and rights (1.00) | Higher is better | 0.642 | 5 |
| M3-SafetyBenchoverall_score | Human welfare and rights (0.50), Safe assistance (0.50) | Higher is better | 0.642 | 19 |
| MACHIAVELLIall_score | Human welfare and rights (0.75), Epistemic integrity (0.25) | Lower is better | 0.642 | 7 |
| Manager Coercion Benchcoercion_ladder_depth | Responsible power (1.00) | Lower is better | 0.642 | 6 |
| Manager Coercion Benchfabrication_rate | Epistemic integrity (1.00) | Lower is better | 0.642 | 6 |
| MANTAAWMS | Nonhuman welfare (1.00) | Higher is better | 0.428 | 7 |
| MANTAAWVS | Nonhuman welfare (1.00) | Higher is better | 0.857 | 7 |
| MASKlying_probability | Epistemic integrity (1.00) | Lower is better | 0.642 | 10 |
| MORUscore | Nonhuman welfare (0.50), Human welfare and rights (0.25), Responsible power (0.25) | Higher is better | 0.642 | 13 |
| ODCV-Benchaverage_severity | Human welfare and rights (0.10), Epistemic integrity (0.10), Responsible power (0.80) | Lower is better | 0.535 | 12 |
| ODCV-Benchmisalignment_rate | Human welfare and rights (0.10), Epistemic integrity (0.10), Responsible power (0.80) | Lower is better | 1.071 | 12 |
| OR-Benchover_refusal_rate | Safe assistance (1.00) | Lower is better | 0.642 | 25 |
| OR-Benchtoxic_acceptance_rate | Safe assistance (1.00) | Lower is better | 0.642 | 25 |
| PacifAIstp_score | Human welfare and rights (0.20), Responsible power (0.80) | Higher is better | 0.964 | 7 |
| RefusalBenchyouden_j | Safe assistance (1.00) | Higher is better | 0.964 | 19 |
| SafeDialBenchaggression | Safe assistance (1.00) | Higher is better | 0.161 | 18 |
| SafeDialBenchethics | Safe assistance (1.00) | Higher is better | 0.161 | 18 |
| SafeDialBenchfairness | Human welfare and rights (1.00) | Higher is better | 0.161 | 18 |
| SafeDialBenchlegality | Safe assistance (1.00) | Higher is better | 0.161 | 18 |
| SafeDialBenchmorality | Human welfare and rights (1.00) | Higher is better | 0.161 | 18 |
| SafeDialBenchprivacy | Human welfare and rights (1.00) | Higher is better | 0.161 | 18 |
| SafetyBenchEM | Higher is better | 0.092 | 21 | |
| SafetyBenchIA | Higher is better | 0.092 | 21 | |
| SafetyBenchMH | Higher is better | 0.092 | 21 | |
| SafetyBenchOFF | Higher is better | 0.092 | 21 | |
| SafetyBenchPH | Higher is better | 0.092 | 21 | |
| SafetyBenchPP | Higher is better | 0.092 | 21 | |
| SafetyBenchUB | Higher is better | 0.092 | 21 | |
| SafetyPromptsscore | Safe assistance (1.00) | Higher is better | 0.642 | 15 |
| SALAD-Benchattack_success_rate | Safe assistance (1.00) | Lower is better | 0.514 | 33 |
| SALAD-Benchbase_safety_rate | Safe assistance (1.00) | Higher is better | 0.514 | 33 |
| SALAD-Benchsafety_knowledge_accuracy | Higher is better | 0.257 | 26 | |
| Social Welfare Function Benchmarkefficiency | Higher is better | 0.428 | 19 | |
| Social Welfare Function Benchmarkfairness | Human welfare and rights (1.00) | Higher is better | 0.857 | 19 |
| SORRY-Benchavg | Safe assistance (1.00) | Lower is better | 0.482 | 51 |
| SOSBenchbiology_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SOSBenchchemistry_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SOSBenchmedicine_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SOSBenchpharmacology_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SOSBenchphysics_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SOSBenchpsychology_pvr | Safe assistance (1.00) | Lower is better | 0.161 | 23 |
| SpeciesismBenchexplicit_speciesism_scale | Nonhuman welfare (1.00) | Lower is better | 0.385 | 7 |
| SpeciesismBenchmorally_wrong_rate | Nonhuman welfare (1.00) | Higher is better | 0.385 | 8 |
| SpeciesismBenchspeciesism_recognition_rate | Higher is better | 0.193 | 8 | |
| SuperCLUE Safetyinstruction_attack | Safe assistance (1.00) | Higher is better | 0.321 | 13 |
| SuperCLUE Safetyresponsible_ai | Human welfare and rights (0.50), Responsible power (0.50) | Higher is better | 0.321 | 13 |
| SuperCLUE Safetytraditional_safety | Human welfare and rights (0.50), Safe assistance (0.50) | Higher is better | 0.321 | 13 |
| SYCON Benchfalse_presupposition_tof | Epistemic integrity (1.00) | Higher is better | 0.482 | 11 |
| SYCON Benchunethical_queries_tof | Safe assistance (1.00) | Higher is better | 0.482 | 11 |
| TACbase_welfare_rate | Nonhuman welfare (1.00) | Higher is better | 0.964 | 22 |
| TrustLLM contemporary collapsed applicationtrustllm | Human welfare and rights (0.33), Safe assistance (0.33), Epistemic integrity (0.33) | Higher is better | 0.642 | 8 |
| XSTestsafe_full_compliance_rate | Safe assistance (1.00) | Higher is better | 0.161 | 3 |
| XSTestunsafe_full_refusal_rate | Safe assistance (1.00) | Higher is better | 0.161 | 3 |