Score composition
Model behavior supplies 90% of the score. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 6 | 8 | 47 |
| Human welfare and rights | 22.5% | Human welfare, rights, fairness, privacy, autonomy, and moral judgment. | 22 | 49 | 259 |
| Safe assistance | 18% | Refusal of genuinely harmful requests without indiscriminate refusal of benign requests. | 30 | 61 | 304 |
| Epistemic integrity | 13.5% | Truthfulness, honesty, resistance to deception and sycophancy, and non-manipulation. | 13 | 24 | 180 |
| Responsible power | 13.5% | Responsible behavior when exercising delegated, tool-mediated, or potentially systemic power. | 13 | 26 | 149 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 124 |
Scores by component
Behavior-component cells show 90% uncertainty intervals and the number of direct evals. A component is left blank when a model has no direct evidence for it. Governance is shown as a point value because the source index does not publish an uncertainty model. In the current model, component score are largely anchored by the global score (note the very close component scores within each row).
| Model | Nonhuman welfare | Human welfare and rights | Safe assistance | Epistemic integrity | Responsible power | Governance |
|---|---|---|---|---|---|---|
 claude-opus-4.7 | 61.890%: 50.7–68.4 3 direct evals | 60.690%: 50.7–66.9 4 direct evals | 60.390%: 50.7–66.6 4 direct evals | 60.790%: 50.9–66.9 5 direct evals | 61.890%: 51.0–68.8 6 direct evals | 87.4Direct |
| claude-3.7-sonnet | Not shown0 direct evals | 59.290%: 54.1–62.9 2 direct evals | 59.590%: 54.4–63.3 7 direct evals | 59.490%: 53.9–63.2 2 direct evals | 59.290%: 53.9–63.0 1 direct eval | 87.3Direct |
| claude-opus-4.8 | 59.090%: 53.0–63.5 1 direct eval | 58.390%: 52.9–62.3 2 direct evals | 58.590%: 53.0–62.5 3 direct evals | 58.690%: 52.9–62.6 5 direct evals | 59.390%: 53.2–63.9 5 direct evals | 87.4Direct |
| claude-3.5-sonnet | 56.190%: 50.3–61.8 2 direct evals | 57.890%: 52.1–62.5 5 direct evals | 58.190%: 52.3–62.7 10 direct evals | 57.790%: 52.4–62.4 3 direct evals | 58.690%: 52.6–63.7 4 direct evals | 87.3Direct |
 gpt-5.5 | 57.890%: 52.0–61.8 4 direct evals | 57.390%: 52.5–60.5 4 direct evals | 57.090%: 52.2–60.2 4 direct evals | 57.390%: 52.5–60.6 5 direct evals | 57.690%: 52.7–60.9 6 direct evals | 82.4Direct |
| gpt-4 | Not shown0 direct evals | 57.290%: 53.3–60.6 4 direct evals | 57.590%: 53.0–61.3 8 direct evals | Not shown0 direct evals | 56.990%: 52.9–60.2 1 direct eval | 62.0Direct |
| claude-sonnet-4.6 | 55.690%: 50.2–59.8 3 direct evals | 56.190%: 50.8–60.5 3 direct evals | 56.490%: 51.0–60.7 5 direct evals | 56.590%: 50.9–60.9 5 direct evals | 57.190%: 50.8–62.4 6 direct evals | 87.6Direct |
| claude-3-opus | 54.390%: 49.4–58.1 3 direct evals | 54.590%: 49.9–58.5 5 direct evals | 54.790%: 50.2–58.4 9 direct evals | 54.590%: 50.0–58.3 1 direct eval | 55.390%: 50.1–59.6 4 direct evals | 87.3Direct |
 gemini-1.5-pro | 55.990%: 50.4–61.3 2 direct evals | 54.590%: 50.4–58.5 4 direct evals | 54.590%: 50.4–58.4 10 direct evals | 54.690%: 50.5–58.6 3 direct evals | 54.490%: 50.4–58.2 3 direct evals | 68.3Direct |
 muse-spark-1.1 | Not shown0 direct evals | 54.890%: 51.8–58.1 1 direct eval | 54.990%: 51.8–58.1 3 direct evals | 54.990%: 51.9–58.2 3 direct evals | 55.090%: 51.9–58.2 3 direct evals | 59.6Direct |
| claude-fable-5 | Not shown0 direct evals | 54.290%: 50.8–57.6 1 direct eval | 54.490%: 50.7–58.1 3 direct evals | 54.490%: 50.9–57.6 3 direct evals | 54.590%: 50.7–58.2 3 direct evals | 87.4Direct |
 mistral-large | 55.490%: 49.2–62.4 1 direct eval | 54.190%: 49.1–59.7 1 direct eval | 54.090%: 49.0–59.3 2 direct evals | 54.190%: 49.2–59.7 1 direct eval | 54.190%: 49.2–59.7 1 direct eval | 26.0Direct |
| claude-opus-4.6 | 53.390%: 50.7–55.6 2 direct evals | 53.290%: 50.6–55.5 3 direct evals | 53.490%: 50.7–56.1 4 direct evals | 53.290%: 50.6–55.6 4 direct evals | 53.490%: 50.6–56.1 5 direct evals | 87.6Direct |
| claude-opus-4.5 | Not shown0 direct evals | 52.890%: 50.6–54.8 1 direct eval | 53.290%: 50.6–55.4 4 direct evals | 53.090%: 50.6–55.1 3 direct evals | 53.190%: 50.6–55.4 3 direct evals | 87.6Direct |
| claude-sonnet-4.5 | 52.290%: 46.3–57.7 2 direct evals | 53.190%: 47.5–58.6 3 direct evals | 53.490%: 47.9–58.8 8 direct evals | 53.390%: 47.8–58.7 3 direct evals | 53.090%: 47.5–58.1 4 direct evals | 87.6Direct |
| gemini-1.5-flash | 53.790%: 50.7–56.9 1 direct eval | 53.390%: 50.8–55.6 3 direct evals | 53.090%: 50.6–55.3 8 direct evals | 53.190%: 50.7–55.5 1 direct eval | 53.390%: 50.8–55.6 2 direct evals | 68.3Direct |
| gpt-4-turbo | Not shown0 direct evals | 53.090%: 50.6–55.4 3 direct evals | 53.190%: 50.7–55.8 9 direct evals | 52.990%: 50.7–55.3 1 direct eval | 52.890%: 50.5–55.3 2 direct evals | 62.0Direct |
| internlm-chat-7b | Not shown0 direct evals | 53.290%: 49.9–57.4 1 direct eval | 52.890%: 50.0–56.4 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| gpt-5.2 | 52.390%: 48.9–55.0 3 direct evals | 52.390%: 49.0–54.8 3 direct evals | 52.390%: 49.1–54.6 4 direct evals | 52.690%: 49.1–55.0 5 direct evals | 52.190%: 49.1–54.3 5 direct evals | 81.7Direct |
| llama-2-7b-chat | Not shown0 direct evals | 53.190%: 47.7–58.0 3 direct evals | 53.790%: 48.5–58.7 6 direct evals | Not shown0 direct evals | Not shown0 direct evals | 25.3Direct |
| llama-3.1-405b-instruct | Not shown0 direct evals | 53.190%: 49.1–56.9 3 direct evals | 53.290%: 49.1–57.0 9 direct evals | 53.590%: 49.1–57.6 3 direct evals | 53.090%: 48.9–56.8 3 direct evals | 25.3Direct |
| claude-3-haiku | Not shown0 direct evals | 52.290%: 49.9–54.8 4 direct evals | 52.190%: 49.6–54.8 8 direct evals | Not shown0 direct evals | 52.290%: 49.8–54.8 2 direct evals | 87.3Direct |
| gpt-5-nano | 52.190%: 49.8–54.8 1 direct eval | Not shown0 direct evals | 52.290%: 49.8–55.3 7 direct evals | 52.190%: 49.7–54.9 2 direct evals | 52.090%: 49.5–55.2 3 direct evals | 73.8Direct |
 qwen-14b-chat | Not shown0 direct evals | 52.990%: 51.0–54.6 3 direct evals | 52.790%: 51.0–54.2 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | 29.5Direct |
| yi-34b-chat | Not shown0 direct evals | 52.890%: 49.3–56.4 2 direct evals | 52.290%: 49.3–55.4 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| gpt-5-mini | 52.190%: 49.9–54.4 2 direct evals | 51.990%: 49.8–54.0 1 direct eval | 51.890%: 49.8–53.9 7 direct evals | 51.890%: 49.9–53.8 2 direct evals | 51.790%: 49.7–53.9 4 direct evals | 73.8Direct |
| claude-3.5-haiku | 51.190%: 47.8–54.7 1 direct eval | 51.790%: 48.7–55.3 3 direct evals | 51.790%: 48.7–55.3 2 direct evals | 51.690%: 48.7–55.2 2 direct evals | 51.890%: 48.7–55.6 2 direct evals | 87.3Direct |
| gemini-2.5-flash | 52.090%: 47.7–56.4 1 direct eval | 51.790%: 47.1–56.4 3 direct evals | 51.990%: 47.6–56.3 9 direct evals | 51.890%: 47.4–56.2 3 direct evals | 52.590%: 47.4–57.9 4 direct evals | 64.9Direct |
 glm-5 | Not shown0 direct evals | 52.390%: 50.0–54.4 2 direct evals | 52.590%: 50.0–54.7 2 direct evals | 52.390%: 50.0–54.4 2 direct evals | 52.390%: 50.0–54.5 1 direct eval | 38.3Direct |
| o3 | Not shown0 direct evals | 51.390%: 48.0–54.2 2 direct evals | 52.090%: 48.9–54.9 7 direct evals | 51.690%: 49.0–54.3 2 direct evals | 51.690%: 48.7–54.2 2 direct evals | 73.8Direct |
| gpt-5.6-sol | Not shown0 direct evals | 51.490%: 48.2–54.0 1 direct eval | 51.290%: 47.8–54.1 3 direct evals | 51.590%: 48.3–54.1 3 direct evals | 51.490%: 47.9–54.3 3 direct evals | 82.4Direct |
| qwen3-235b-a22b | 52.090%: 48.4–55.6 2 direct evals | 52.190%: 48.2–55.9 3 direct evals | 51.990%: 47.9–55.6 6 direct evals | 52.190%: 48.3–55.6 1 direct eval | 52.790%: 48.3–57.1 1 direct eval | 29.5Direct |
| gpt-oss-120b | 51.490%: 48.4–54.5 1 direct eval | 51.490%: 48.4–54.6 3 direct evals | 51.690%: 48.5–54.8 5 direct evals | 52.090%: 48.4–55.7 4 direct evals | 51.190%: 48.0–54.5 2 direct evals | 73.8Direct |
| gpt-5.6-terra | Not shown0 direct evals | 51.290%: 49.6–52.6 1 direct eval | 51.290%: 49.5–52.7 3 direct evals | 51.490%: 49.7–52.9 3 direct evals | 51.390%: 49.5–52.8 3 direct evals | 82.4Direct |
| claude-sonnet-4 | Not shown0 direct evals | 50.990%: 48.6–53.5 2 direct evals | 51.690%: 48.9–54.7 7 direct evals | 51.190%: 48.7–53.7 1 direct eval | 50.790%: 48.0–53.6 3 direct evals | 88.2Direct |
| gpt-4.1 | 51.490%: 50.5–52.2 1 direct eval | 51.490%: 50.4–52.4 2 direct evals | 51.490%: 50.5–52.3 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 73.8Direct |
| claude-opus-4 | Not shown0 direct evals | 51.190%: 50.2–51.7 1 direct eval | 51.090%: 50.2–51.7 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | 88.2Direct |
| gpt-3.5-turbo | 51.390%: 46.7–55.6 1 direct eval | 52.290%: 46.9–57.0 6 direct evals | 50.990%: 45.9–55.7 13 direct evals | 51.590%: 46.7–56.0 1 direct eval | 51.190%: 46.3–55.6 3 direct evals | 62.0Direct |
| glm-5.1 | Not shown0 direct evals | 51.790%: 48.9–53.9 2 direct evals | 51.490%: 48.7–53.7 3 direct evals | 51.790%: 48.9–54.0 4 direct evals | 51.890%: 48.7–54.4 5 direct evals | 37.8Direct |
| gpt-5.1 | Not shown0 direct evals | 50.990%: 50.0–51.6 2 direct evals | 51.090%: 50.0–51.8 7 direct evals | 50.990%: 49.9–51.6 3 direct evals | 50.890%: 49.8–51.5 3 direct evals | 81.7Direct |
| claude-haiku-4.5 | 50.490%: 46.7–53.1 2 direct evals | 50.890%: 47.1–53.5 5 direct evals | 50.990%: 47.2–53.5 8 direct evals | 50.890%: 47.2–53.6 5 direct evals | 50.890%: 47.3–53.4 5 direct evals | 87.6Direct |
 grok-4-2 | Not shown0 direct evals | 51.490%: 49.8–53.1 1 direct eval | 51.490%: 49.8–53.2 3 direct evals | 51.490%: 49.8–53.1 3 direct evals | 51.490%: 49.7–53.2 3 direct evals | 37.4Direct |
| internlm2-chat-7b | Not shown0 direct evals | 51.290%: 49.3–52.7 1 direct eval | 51.390%: 49.4–52.7 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| internlm2-chat-20b | Not shown0 direct evals | 51.390%: 49.8–52.8 1 direct eval | 51.290%: 49.8–52.5 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| baichuan-2-13b-chat | Not shown0 direct evals | 50.890%: 48.8–52.8 5 direct evals | 51.390%: 49.1–53.3 6 direct evals | Not shown0 direct evals | 51.090%: 49.0–53.0 1 direct eval | 49.8Median-imputed |
| o4-mini | Not shown0 direct evals | 50.490%: 48.6–52.0 1 direct eval | 51.090%: 49.4–52.8 7 direct evals | 50.690%: 49.3–52.2 2 direct evals | 50.690%: 49.3–52.1 2 direct evals | 73.8Direct |
| o1 | Not shown0 direct evals | Not shown0 direct evals | 50.990%: 49.3–52.2 6 direct evals | 50.890%: 49.2–52.3 2 direct evals | 50.790%: 49.3–51.9 1 direct eval | 62.0Direct |
| llama-3.3-70b-instruct | 51.690%: 47.6–56.6 3 direct evals | 51.390%: 47.8–55.6 2 direct evals | 51.190%: 47.5–55.3 4 direct evals | 51.090%: 47.2–55.3 3 direct evals | Not shown0 direct evals | 25.3Direct |
 kimi-k2 | Not shown0 direct evals | 51.490%: 48.0–55.1 2 direct evals | 50.890%: 47.9–53.7 8 direct evals | 50.890%: 48.0–53.6 3 direct evals | 50.690%: 47.6–53.5 3 direct evals | 49.8Median-imputed |
| gpt-5.6-luna | Not shown0 direct evals | 50.490%: 49.0–51.3 1 direct eval | 50.390%: 48.8–51.3 3 direct evals | 50.490%: 49.0–51.5 3 direct evals | 50.390%: 48.8–51.4 3 direct evals | 82.4Direct |
| gemini-1.0-pro | Not shown0 direct evals | 50.790%: 50.0–51.2 1 direct eval | 50.690%: 50.0–51.1 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 68.3Direct |
| o1-mini | Not shown0 direct evals | 50.790%: 49.7–51.8 1 direct eval | 50.590%: 49.5–51.5 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 62.0Direct |
 olmo-2-0325-32b-instruct | Not shown0 direct evals | 50.890%: 50.0–51.8 1 direct eval | 50.890%: 50.1–51.8 4 direct evals | 50.790%: 50.0–51.6 1 direct eval | 50.790%: 50.0–51.6 1 direct eval | 49.8Median-imputed |
| gemma-7b-it | Not shown0 direct evals | 50.490%: 49.5–51.0 1 direct eval | 50.690%: 49.7–51.3 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 68.3Direct |
 deepseek-v3 | 51.390%: 47.5–54.4 3 direct evals | 51.790%: 47.9–55.1 4 direct evals | 50.690%: 46.7–53.8 9 direct evals | 50.890%: 47.0–54.3 3 direct evals | Not shown0 direct evals | 21.9Direct |
| gemini-2-flash | Not shown0 direct evals | 50.490%: 50.0–50.8 1 direct eval | 50.490%: 50.0–50.8 5 direct evals | 50.490%: 50.0–50.7 1 direct eval | 50.490%: 50.0–50.7 1 direct eval | 68.3Direct |
| gemma-2-27b-it | Not shown0 direct evals | 50.490%: 49.8–50.7 1 direct eval | 50.590%: 49.9–50.9 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 68.3Direct |
| gpt-5.4 | Not shown0 direct evals | 50.190%: 47.6–51.9 2 direct evals | 50.190%: 47.7–51.9 3 direct evals | 50.090%: 47.6–51.9 4 direct evals | 50.390%: 47.7–52.1 4 direct evals | 82.4Direct |
 minimax-m2.7 | Not shown0 direct evals | 50.690%: 49.2–52.2 2 direct evals | 50.890%: 49.3–52.6 1 direct eval | 50.690%: 49.2–52.2 2 direct evals | 50.590%: 48.9–52.0 2 direct evals | 49.8Median-imputed |
| gpt-4.5-preview | Not shown0 direct evals | Not shown0 direct evals | 50.590%: 50.1–50.8 4 direct evals | 50.490%: 50.0–50.7 1 direct eval | Not shown0 direct evals | 62.0Direct |
| llama-3-8b-instruct | Not shown0 direct evals | 50.790%: 48.1–53.1 2 direct evals | 51.290%: 48.7–53.8 7 direct evals | Not shown0 direct evals | Not shown0 direct evals | 25.3Direct |
| grok-4-fast | 50.490%: 47.3–53.0 1 direct eval | 50.790%: 47.6–53.6 1 direct eval | 50.590%: 47.6–53.1 2 direct evals | 50.790%: 47.4–53.8 4 direct evals | 50.590%: 47.6–53.1 2 direct evals | 44.0Direct |
| gpt-oss-20b | Not shown0 direct evals | 50.190%: 49.0–50.9 2 direct evals | 50.290%: 49.0–51.1 4 direct evals | 50.190%: 49.0–50.9 2 direct evals | 50.190%: 48.9–50.9 2 direct evals | 73.8Direct |
| yi-1.5-34b-chat | Not shown0 direct evals | 50.490%: 49.0–52.0 1 direct eval | 50.490%: 49.0–52.0 2 direct evals | 50.490%: 49.1–52.0 1 direct eval | 50.490%: 49.1–52.0 1 direct eval | 49.8Median-imputed |
| deepseek-r1 | 51.290%: 47.9–55.1 1 direct eval | 50.790%: 47.8–54.5 4 direct evals | 50.690%: 47.0–55.1 13 direct evals | 51.090%: 47.8–55.5 5 direct evals | 50.590%: 47.4–54.4 3 direct evals | 21.9Direct |
| claude-3-sonnet | Not shown0 direct evals | 49.590%: 46.7–52.1 3 direct evals | 49.690%: 46.7–52.4 8 direct evals | Not shown0 direct evals | 49.490%: 46.3–52.2 2 direct evals | 87.3Direct |
| qwen-7b-chat | Not shown0 direct evals | 50.490%: 48.8–51.6 4 direct evals | 50.590%: 48.9–51.8 5 direct evals | Not shown0 direct evals | 50.490%: 49.0–51.6 1 direct eval | 29.5Direct |
| gemma-2b-it | Not shown0 direct evals | 49.990%: 49.2–50.3 1 direct eval | 49.990%: 49.1–50.4 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 68.3Direct |
| gemma-2-9b-it | Not shown0 direct evals | 49.790%: 47.2–51.4 2 direct evals | 50.290%: 48.3–51.6 5 direct evals | 49.590%: 47.6–51.0 2 direct evals | 49.890%: 47.9–51.2 1 direct eval | 68.3Direct |
| olmo-2-1124-13b-instruct | Not shown0 direct evals | 50.190%: 49.0–51.1 1 direct eval | 49.990%: 48.7–51.0 4 direct evals | 50.090%: 48.9–51.0 1 direct eval | 50.090%: 48.9–50.9 1 direct eval | 49.8Median-imputed |
| gemini-3.1-pro-preview | Not shown0 direct evals | 49.790%: 46.0–53.1 3 direct evals | 50.190%: 46.5–53.9 5 direct evals | 49.590%: 46.0–52.9 5 direct evals | 49.190%: 45.2–53.0 4 direct evals | 72.3Direct |
| grok-4.5 | Not shown0 direct evals | 50.190%: 48.6–51.5 1 direct eval | 50.190%: 48.5–51.7 3 direct evals | 50.090%: 48.6–51.5 3 direct evals | 50.090%: 48.4–51.7 3 direct evals | 37.4Direct |
| qwen1.5-7b-chat | Not shown0 direct evals | 50.290%: 48.5–51.6 1 direct eval | 50.190%: 48.1–51.9 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 29.5Direct |
| qwen-2.5-72b-instruct | 50.590%: 47.9–53.4 1 direct eval | 50.190%: 47.9–52.6 1 direct eval | 49.990%: 47.4–52.5 7 direct evals | 50.090%: 47.8–52.6 2 direct evals | 50.090%: 47.9–52.6 1 direct eval | 29.5Direct |
| grok-4.1-fast | 49.890%: 47.3–52.7 1 direct eval | 49.990%: 47.4–52.8 3 direct evals | 49.990%: 47.4–52.8 4 direct evals | 50.090%: 47.5–53.1 4 direct evals | 49.890%: 47.2–52.8 4 direct evals | 44.0Direct |
| gpt-4.1-mini | Not shown0 direct evals | 49.490%: 48.1–50.3 2 direct evals | 49.590%: 48.3–50.4 5 direct evals | 49.490%: 48.3–50.3 1 direct eval | Not shown0 direct evals | 73.8Direct |
| yi-6b-chat | Not shown0 direct evals | 49.990%: 48.5–51.2 2 direct evals | 49.790%: 48.4–50.8 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| vicuna-7b-v1.5 | Not shown0 direct evals | 50.090%: 49.6–50.1 1 direct eval | 50.090%: 49.7–50.1 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 25.3Base-model inherited |
 command-r | Not shown0 direct evals | 49.790%: 48.5–50.5 1 direct eval | 49.390%: 48.0–50.1 5 direct evals | Not shown0 direct evals | 49.790%: 48.5–50.5 1 direct eval | 49.8Median-imputed |
| deepseek-v4-flash | 49.590%: 47.1–51.6 3 direct evals | 49.590%: 47.6–51.3 1 direct eval | Not shown0 direct evals | 49.590%: 47.6–51.3 1 direct eval | 49.590%: 47.6–51.3 1 direct eval | 49.8Median-imputed |
| gemini-3-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 49.290%: 46.6–51.3 7 direct evals | 48.990%: 46.6–50.8 2 direct evals | 49.290%: 46.7–51.4 3 direct evals | 72.3Direct |
| deepseek-v4-pro | 49.590%: 47.0–52.3 2 direct evals | 49.590%: 47.4–51.8 1 direct eval | Not shown0 direct evals | 49.990%: 47.7–51.8 2 direct evals | 48.890%: 46.5–51.3 2 direct evals | 49.8Median-imputed |
| gpt-5 | Not shown0 direct evals | 48.690%: 42.4–54.9 4 direct evals | 49.590%: 43.4–55.8 8 direct evals | 50.190%: 43.6–56.7 5 direct evals | 48.390%: 41.5–55.2 3 direct evals | 73.8Direct |
| grok-4 | Not shown0 direct evals | 49.990%: 47.7–51.9 4 direct evals | 49.290%: 47.2–50.6 8 direct evals | 49.490%: 47.4–50.9 6 direct evals | 49.290%: 47.1–50.7 4 direct evals | 50.0Direct |
| gpt-5.4-nano | Not shown0 direct evals | 48.690%: 46.0–50.4 1 direct eval | 48.790%: 46.4–50.4 3 direct evals | 48.690%: 46.0–50.4 3 direct evals | 48.790%: 46.3–50.4 3 direct evals | 82.4Direct |
| glm-5.2 | 49.290%: 46.5–51.2 2 direct evals | 49.490%: 47.2–50.9 1 direct eval | 49.490%: 47.2–51.0 3 direct evals | 49.390%: 47.1–50.9 3 direct evals | 49.490%: 47.2–51.0 3 direct evals | 37.8Direct |
| llama-4-scout | Not shown0 direct evals | 49.390%: 48.4–49.9 1 direct eval | 49.290%: 48.3–49.9 5 direct evals | 49.390%: 48.5–49.9 1 direct eval | 49.390%: 48.4–49.9 1 direct eval | 41.7Direct |
| glm-4-9b-chat | Not shown0 direct evals | 49.390%: 46.9–52.3 3 direct evals | 48.890%: 46.1–52.0 5 direct evals | 49.190%: 46.9–51.9 1 direct eval | 49.390%: 47.0–52.0 1 direct eval | 49.5Direct |
| o3-mini | Not shown0 direct evals | 48.990%: 45.2–52.6 1 direct eval | 49.190%: 45.2–53.2 8 direct evals | 48.890%: 45.2–53.0 3 direct evals | 48.990%: 45.6–52.7 2 direct evals | 62.0Direct |
| command-r-plus | Not shown0 direct evals | 49.290%: 48.1–49.9 1 direct eval | 48.890%: 47.4–49.9 5 direct evals | Not shown0 direct evals | 49.190%: 48.1–49.9 1 direct eval | 49.8Median-imputed |
| vicuna-13b-v1.5 | Not shown0 direct evals | 49.490%: 48.5–50.1 1 direct eval | 49.390%: 48.3–50.1 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 25.3Base-model inherited |
| gpt-5.4-mini | Not shown0 direct evals | 48.590%: 45.6–50.9 2 direct evals | 48.390%: 45.3–50.9 4 direct evals | 48.490%: 45.6–50.9 4 direct evals | 48.790%: 46.0–51.0 4 direct evals | 82.4Direct |
| llama-2-13b-chat | Not shown0 direct evals | 49.290%: 47.6–50.7 1 direct eval | 49.390%: 47.6–51.0 6 direct evals | Not shown0 direct evals | 49.390%: 47.8–50.7 1 direct eval | 25.3Direct |
| kimi-k2.6-64k | Not shown0 direct evals | 48.990%: 47.4–50.2 1 direct eval | 48.790%: 47.0–50.1 3 direct evals | 48.990%: 47.4–50.2 3 direct evals | 48.690%: 46.9–50.1 3 direct evals | 49.8Median-imputed |
| mixtral-8x7b-instruct-v0.1 | Not shown0 direct evals | 49.290%: 48.0–50.0 1 direct eval | 48.890%: 47.4–49.8 6 direct evals | Not shown0 direct evals | Not shown0 direct evals | 26.0Direct |
| grok-3 | 48.790%: 47.0–49.9 1 direct eval | Not shown0 direct evals | 48.490%: 46.1–50.0 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 50.0Direct |
| kimi-k2.6 | Not shown0 direct evals | 48.790%: 46.8–50.0 2 direct evals | 48.490%: 45.9–50.0 1 direct eval | 48.790%: 46.8–50.0 2 direct evals | 48.690%: 46.8–50.0 2 direct evals | 49.8Median-imputed |
| kimi-k2.5 | Not shown0 direct evals | 48.790%: 46.8–50.5 2 direct evals | 48.590%: 46.5–50.4 4 direct evals | 48.790%: 46.7–50.5 4 direct evals | 48.490%: 46.3–50.2 4 direct evals | 49.8Median-imputed |
| chatglm-6b | Not shown0 direct evals | 48.290%: 43.0–53.1 1 direct eval | 48.390%: 41.8–54.2 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.5Direct |
| gpt-4o | 47.690%: 43.4–51.7 2 direct evals | 48.790%: 43.9–52.7 9 direct evals | 48.490%: 44.4–52.2 17 direct evals | 48.390%: 44.1–52.2 7 direct evals | 47.890%: 43.2–52.0 7 direct evals | 62.0Direct |
| zephyr-7b-beta | Not shown0 direct evals | 48.790%: 47.3–49.7 1 direct eval | 48.490%: 46.9–49.8 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 26.0Base-model inherited |
| gemini-3.5-flash | 47.890%: 45.0–50.2 3 direct evals | 47.990%: 45.6–50.0 3 direct evals | 47.990%: 45.6–49.9 3 direct evals | 47.790%: 45.4–49.9 4 direct evals | 47.990%: 45.6–50.0 5 direct evals | 75.8Direct |
| gemini-3-flash-preview | Not shown0 direct evals | 47.990%: 46.1–49.4 1 direct eval | 47.890%: 46.0–49.5 3 direct evals | 47.690%: 45.9–49.4 3 direct evals | 47.790%: 45.7–49.4 4 direct evals | 72.3Direct |
| qwen-2.5-14b-instruct | Not shown0 direct evals | 48.390%: 46.0–51.0 3 direct evals | 48.190%: 45.6–51.0 4 direct evals | 48.290%: 46.1–50.9 2 direct evals | 48.190%: 45.8–50.9 1 direct eval | 29.5Direct |
| deepseek-v3.2 | 48.690%: 45.7–51.2 3 direct evals | 48.390%: 45.6–50.8 3 direct evals | 47.790%: 44.5–50.5 5 direct evals | 48.190%: 45.4–50.7 4 direct evals | 47.990%: 45.2–50.7 5 direct evals | 29.5Direct |
| llama-3-70b-instruct | Not shown0 direct evals | 48.490%: 44.8–51.6 2 direct evals | 48.090%: 44.2–51.3 7 direct evals | Not shown0 direct evals | 47.890%: 43.7–51.4 1 direct eval | 25.3Direct |
| gemini-2.5-pro | Not shown0 direct evals | 47.790%: 42.5–53.6 4 direct evals | 47.890%: 42.6–53.6 9 direct evals | 47.390%: 41.1–54.2 6 direct evals | 46.890%: 41.0–53.1 4 direct evals | 64.9Direct |
| llama-4-maverick | 47.590%: 42.9–51.7 1 direct eval | 47.790%: 42.8–51.7 4 direct evals | 48.090%: 43.1–52.1 7 direct evals | 47.790%: 42.8–51.8 4 direct evals | 48.390%: 43.3–52.3 2 direct evals | 41.7Direct |
| chatglm3-6b | Not shown0 direct evals | 47.690%: 44.4–50.2 3 direct evals | 47.590%: 43.8–50.2 5 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.5Direct |
| mistral-large-3 | Not shown0 direct evals | 47.690%: 45.2–50.0 2 direct evals | 47.490%: 44.3–50.0 2 direct evals | 47.790%: 45.3–50.0 2 direct evals | 47.790%: 45.2–50.0 1 direct eval | 26.0Direct |
| mistral-7b-instruct | Not shown0 direct evals | 47.490%: 45.3–49.2 3 direct evals | 46.890%: 44.2–49.0 10 direct evals | 47.590%: 45.5–49.2 1 direct eval | Not shown0 direct evals | 26.0Direct |
| baichuan-2-7b-chat | Not shown0 direct evals | 46.990%: 42.4–50.9 4 direct evals | 46.290%: 40.7–51.1 6 direct evals | Not shown0 direct evals | Not shown0 direct evals | 49.8Median-imputed |
| gemini-3.1-flash-lite | 45.390%: 40.7–50.3 1 direct eval | 46.190%: 42.6–50.2 2 direct evals | 46.390%: 42.6–50.5 4 direct evals | 46.090%: 42.3–50.1 4 direct evals | 46.390%: 42.5–50.6 4 direct evals | 75.8Direct |
| llama-3.1-70b-instruct | Not shown0 direct evals | 45.690%: 37.8–52.6 5 direct evals | 46.890%: 40.7–52.5 8 direct evals | 46.790%: 40.6–52.3 2 direct evals | 46.690%: 40.5–52.4 2 direct evals | 25.3Direct |
| qwen-2.5-7b-instruct | 46.690%: 41.9–51.9 1 direct eval | 46.290%: 41.5–51.5 3 direct evals | 45.990%: 41.0–51.5 9 direct evals | 46.190%: 41.3–51.5 2 direct evals | 45.890%: 40.8–51.5 1 direct eval | 29.5Direct |
| deepseek-v3.1 | 46.390%: 39.7–51.2 1 direct eval | 46.490%: 39.7–51.3 2 direct evals | Not shown0 direct evals | 45.090%: 36.7–51.2 1 direct eval | Not shown0 direct evals | 21.9Direct |
| grok-4.20 | Not shown0 direct evals | 45.790%: 37.7–53.2 2 direct evals | 46.490%: 38.6–54.6 1 direct eval | 45.790%: 37.7–53.2 2 direct evals | 44.690%: 35.3–53.1 2 direct evals | 37.4Direct |
| gemini-2.5-flash-lite | 45.090%: 42.4–47.4 2 direct evals | 45.390%: 42.8–47.5 1 direct eval | 45.290%: 42.6–47.5 7 direct evals | 44.990%: 42.3–47.3 2 direct evals | 45.090%: 42.4–47.2 4 direct evals | 64.9Direct |
| grok-3-mini | Not shown0 direct evals | 45.390%: 40.4–49.6 2 direct evals | 45.390%: 40.8–49.6 6 direct evals | 45.590%: 40.9–49.6 1 direct eval | 44.790%: 39.2–49.6 1 direct eval | 50.0Direct |
| gpt-4o-mini | 44.290%: 40.2–50.4 1 direct eval | 45.590%: 41.4–51.1 6 direct evals | 45.190%: 41.3–50.3 9 direct evals | 45.090%: 41.4–50.3 2 direct evals | 44.690%: 40.9–49.9 4 direct evals | 62.0Direct |
| grok-4.3 | 43.690%: 39.2–50.9 1 direct eval | 44.390%: 40.1–50.8 2 direct evals | 44.690%: 40.3–51.1 3 direct evals | 43.690%: 38.6–50.8 5 direct evals | 44.290%: 39.9–51.1 5 direct evals | 37.4Direct |
| mistral-large-2 | Not shown0 direct evals | 44.090%: 38.8–49.0 3 direct evals | 43.790%: 38.2–49.0 5 direct evals | 44.390%: 39.3–49.1 2 direct evals | 43.990%: 38.3–49.1 2 direct evals | 26.0Direct |
| chatglm2-6b | Not shown0 direct evals | 42.590%: 34.8–49.2 3 direct evals | 42.390%: 34.7–49.0 4 direct evals | Not shown0 direct evals | 43.290%: 36.4–49.2 1 direct eval | 49.5Direct |
| llama-3.1-8b-instruct | Not shown0 direct evals | 43.290%: 36.6–49.8 4 direct evals | 42.490%: 34.6–50.3 10 direct evals | 43.190%: 36.4–50.0 3 direct evals | 43.290%: 36.4–50.6 2 direct evals | 25.3Direct |