mcode on FrontierHarness Eval

76.7% pass rate (23/30 attempts). Third-party result; numerical comparison is not an official leaderboard placement.

本轮 30 次尝试已结束：23 题通过、4 题未通过、3 题因此前外层超时未获得 verifier 评分。3 题未评分不当作能力失败；官方汇总的通过率分母仍为 30。

本轮 Kimi 模型费用已知约 ¥279.85，为原始 usage 估算；未返回用量仍待统计。美元成本字段未采集，不能把 n/a 当作零费用。

采用官方第三方 skill 的共享 checkpoint，每题 fresh restore；公开基线使用逐题 checkpoint。本轮使用 Kimi 官方中国 API，而非 Fireworks。DeepSWE 本轮任务文件时限为 5400 秒，同题公开记录中有 3600 秒结束的尝试，尚不能确认时限完全一致。图表只作数字比较，不表示官榜认证排名。

在第 9 个 case 开始前遇到 Runta 基础设施问题，按官方建议重建 checkpoint 后续跑；评测代码、任务镜像、Python 依赖和模型配置保持一致，底层 Runta 镜像的系统 runc 从 1.4.3 变为 1.5.1。

前 8 题 checkpoint：01a079eb-fde2-7553-8d91-5c47fc642de5；后 22 题 checkpoint：01a07c81-d5e9-7dce-8b67-e03c2bae8bcb。原始 run.json 和逐题结果未修改。

完整中文说明 · 人民币费用 CSV

mcode versus FrontierHarness Eval v1.0 Pass rate against effective cost per pass · model openai/kimi-k3 · 30 tasks $1$2$5$10$2045.0%50.0%55.0%60.0%65.0%70.0%75.0%80.0%85.0% Effective cost per pass (log scale) Pass rate Exo Harness · 53.3% · $1.05Pi · 60.0% · $2.43Hermes · 50.0% · $2.90OpenCode · 50.0% · $3.24DSH Creator · 63.3% · $3.28DSH Standard · 60.0% · $3.46Codex · 66.7% · $3.47Kimi Code · 56.7% · $3.65DSH PTC · 60.0% · $4.58DSH Minimal · 56.7% · $4.72Oh My Pi · 56.7% · $4.75Claude Code · 63.3% · $18.34 1 harness omitted from the scatter: cost unavailable Pass rate 01mcode76.7%02Codex66.7%03Claude Code63.3%04DSH Creator63.3%05DSH PTC60.0%06DSH Standard60.0%07Pi60.0%08DSH Minimal56.7%09Kimi Code56.7%10Oh My Pi56.7%11Exo Harness53.3%12Hermes50.0%13OpenCode50.0% Numerical ordering only; provider, checkpoint and time-limit differences apply. See RESULTS.md.

Comparison

#HarnessPass rateEffective cost per passCache, medianMedian time
1mcode76.7%n/an/a4m 33s
2Codex66.7%$3.4788.0%6m 43s
3Claude Code63.3%$18.3467.8%9m 38s
4DSH Creator63.3%$3.2884.3%6m 44s
5DSH PTC60.0%$4.5887.2%7m 44s
6DSH Standard60.0%$3.4686.5%6m 17s
7Pi60.0%$2.4379.4%7m 33s
8DSH Minimal56.7%$4.7284.6%5m 41s
9Kimi Code56.7%$3.6588.0%7m 56s
10Oh My Pi56.7%$4.7582.2%6m 46s
11Exo Harness53.3%$1.0570.3%6m 17s
12Hermes50.0%$2.9085.9%6m 58s
13OpenCode50.0%$3.2478.4%6m 27s

Task results

TaskResultCostTimeTurns
datacurve/anko-typed-variable-bindingssuccessn/a32m 15sn/a
datacurve/arktype-json-schema-refs-dependenciestimeoutn/a90m 16sn/a
datacurve/expr-try-catch-errorsfailuren/a78m 39sn/a
datacurve/fastapi-deprecation-response-headerssuccessn/a51m 27sn/a
datacurve/httpx-multipart-response-parsingsuccessn/a32m 28sn/a
datacurve/katex-multicolumn-array-spanssuccessn/a69m 8sn/a
datacurve/meriyah-explicit-resource-declarationstimeoutn/a90m 16sn/a
datacurve/python-statemachine-state-data-scopingtimeoutn/a90m 16sn/a
datacurve/scc-bounded-memory-spillingsuccessn/a69m 50sn/a
terminal-bench/build-cython-extfailuren/a15m 29sn/a
terminal-bench/chess-best-movesuccessn/a4m 33sn/a
terminal-bench/code-from-imagesuccessn/a1m 57sn/a
terminal-bench/constraints-schedulingsuccessn/a4m 3sn/a
terminal-bench/db-wal-recoverysuccessn/a3m 28sn/a
terminal-bench/dna-insertfailuren/a22m 41sn/a
terminal-bench/extract-elfsuccessn/a7m 12sn/a
terminal-bench/gcode-to-textsuccessn/a7m 39sn/a
terminal-bench/git-leak-recoverysuccessn/a3m 33sn/a
terminal-bench/kv-store-grpcsuccessn/a2m 47sn/a
terminal-bench/largest-eigenvalsuccessn/a13m 42sn/a
terminal-bench/log-summary-date-rangessuccessn/a2m 1sn/a
terminal-bench/merge-diff-arc-agi-tasksuccessn/a5m 26sn/a
terminal-bench/modernize-scientific-stacksuccessn/a2m 18sn/a
terminal-bench/multi-source-data-mergersuccessn/a3m 18sn/a
terminal-bench/openssl-selfsigned-certsuccessn/a3m 47sn/a
terminal-bench/polyglot-c-pyfailuren/a10m 23sn/a
terminal-bench/regex-logsuccessn/a9m 3sn/a
terminal-bench/sanitize-git-reposuccessn/a4m 40sn/a
terminal-bench/sqlite-db-truncatesuccessn/a3m 11sn/a
terminal-bench/vulnerable-secretsuccessn/a3m 29sn/a