AlphaEvolve Makes Benchmarks Harder Through Evolution · 1109-811

2026-11-09 08:00

Author: RadarAI Editorial Editor: RadarAI Editorial Last updated: 2026-11-09 Review status: Editorial review pending Brief 速报官方 AI动态开源

Editorial standards and source policy: Editorial standards, Team. Content links to primary sources; see Methodology.

## 🔍 Core Insights **Stanford** and **Google DeepMind** have teamed up to use **AlphaEvolve** to "evolve" static tool-calling benchmarks to be harder, directly targeting the industry pain point that evaluations are too easy [1]; meanwhile, an **Amazon** audit found that the reference answers and scoring code of 9 policy decision benchmarks contain errors themselves [17], and a mysterious new **Anthropic** model doubled its RSI speed in a **METR** investigation [21]—evaluation credibility and model capabilities are being interrogated in tandem. ## 🚀 Key Developments - **Stanford and DeepMind use AlphaEvolve to evolve tool-calling benchmarks** [1]: The TURTLE method mutates benchmarks such as AppWorld, adding filtering conditions and hidden queries to transform overly easy tasks into tests that remain solvable but are harder. - **Amazon paper audits 9 policy decision benchmarks** [17]: It found that benchmarks in tax, legal, and business rule domains often mark correct answers as wrong, with the root cause being errors in the reference answers or scoring code themselves. - **Anthropic's strongest Model 4 exposed for the first time** [21]: A METR investigation shows its AECI score at 178.6, with R&D speed doubling after October 2025, and 80% of the acceleration coming from AI self-directed R&D. - **Simon Willison uses Claude to compare C and Rust implementations of a SQLite extension** [9]: With Claude's assistance, he wrote a parse_number() extension, comparing the performance and size differences between the two language implementations. - **swyx interviews Mihail Eric: How Stanford CS146S teaches AI-native engineering** [16]: As Latent Space reaches 1 million subscribers on the AI Engineer channel, they discuss what AI-native software engineering should teach [18]. - **A wave of individuals "hand-building" AI stock analysis assistants** [11]: Models such as DeepSeek and OpenAI, combined with Ollama, OpenClaw, and low-code platforms, allow ordinary investors to build their own investment research tools without programming. - **Tech stock valuations "cheaper" than the broad market for the first time in 9 years** [23]: The forward P/E of the S&P 500 tech sector fell below the overall market level for the first time, with multiple Wall Street institutions simultaneously issuing buy recommendations [24]. - **Jensen Huang admits to an extramarital affair; Nvidia establishes an independent investigation committee** [19]: The board is reviewing whether he became a target of an intelligence operation, involving a relationship with an executive of Singapore's Megaspeed. ## 🔗 Sources [1] Stanford and DeepMind use AlphaEvolve to evolve tool-calling benchmarks — https://aihot.news/items/rialbh5b2eqfs3dk649pr5ak7 [9] Simon Willison uses Claude to compare the performance and size of C and Rust implementations of the parse_number() SQLite extension — https://aihot.news/items/ppdjty0ow3ytzemiylt9jb0xa [11] A wave of individuals "hand-building" AI stock analysis assistants—is an era of investment research for all approaching? — https://www.bestblogs.dev/article/391a0473a9?utm_source=rss&utm_medium=feed&utm_campaign=resources&entry=rss_article_item [16] swyx interviews Mihail Eric: How Stanford CS146S teaches AI-native software engineering — https://aihot.news/items/qjymzxvzgpiulyjjkcjaqt10u [17] Amazon paper audits 9 policy decision benchmarks, finding that the scorer and reference answers themselves contain errors — https://aihot.news/items/gp0q36wvaswnf2rnc5gz9lhrb [18] How Stanford CS146S teaches AI

Stanford and Google DeepMind have teamed up to use AlphaEvolve to "evolve" static tool-calling benchmarks to be harder, directly targeting the industry pain point that evaluations are too easy [1]; meanwhile, an Amazon audit found that the reference answers and scoring code of 9 policy decision benchmarks contain errors themselves [17], and a mysterious new Anthropic model doubled its RSI speed in a METR investigation [21]—evaluation credibility and model capabilities are being interrogated in tandem.

🚀 Key Developments

Stanford and DeepMind use AlphaEvolve to evolve tool-calling benchmarks [1]: The TURTLE method mutates benchmarks such as AppWorld, adding filtering conditions and hidden queries to transform overly easy tasks into tests that remain solvable but are harder.
Amazon paper audits 9 policy decision benchmarks [17]: It found that benchmarks in tax, legal, and business rule domains often mark correct answers as wrong, with the root cause being errors in the reference answers or scoring code themselves.
Anthropic's strongest Model 4 exposed for the first time [21]: A METR investigation shows its AECI score at 178.6, with R&D speed doubling after October 2025, and 80% of the acceleration coming from AI self-directed R&D.
Simon Willison uses Claude to compare C and Rust implementations of a SQLite extension [9]: With Claude's assistance, he wrote a parse_number() extension, comparing the performance and size differences between the two language implementations.
swyx interviews Mihail Eric: How Stanford CS146S teaches AI-native engineering [16]: As Latent Space reaches 1 million subscribers on the AI Engineer channel, they discuss what AI-native software engineering should teach [18].
A wave of individuals "hand-building" AI stock analysis assistants [11]: Models such as DeepSeek and OpenAI, combined with Ollama, OpenClaw, and low-code platforms, allow ordinary investors to build their own investment research tools without programming.
Tech stock valuations "cheaper" than the broad market for the first time in 9 years [23]: The forward P/E of the S&P 500 tech sector fell below the overall market level for the first time, with multiple Wall Street institutions simultaneously issuing buy recommendations [24].
Jensen Huang admits to an extramarital affair; Nvidia establishes an independent investigation committee [19]: The board is reviewing whether he became a target of an intelligence operation, involving a relationship with an executive of Singapore's Megaspeed.

🔗 Sources

[1] Stanford and DeepMind use AlphaEvolve to evolve tool-calling benchmarks — https://aihot.news/items/rialbh5b2eqfs3dk649pr5ak7 [9] Simon Willison uses Claude to compare the performance and size of C and Rust implementations of the parse_number() SQLite extension — https://aihot.news/items/ppdjty0ow3ytzemiylt9jb0xa [11] A wave of individuals "hand-building" AI stock analysis assistants—is an era of investment research for all approaching? — https://www.bestblogs.dev/article/391a0473a9?utm_source=rss&utm_medium=feed&utm_campaign=resources&entry=rss_article_item [16] swyx interviews Mihail Eric: How Stanford CS146S teaches AI-native software engineering — https://aihot.news/items/qjymzxvzgpiulyjjkcjaqt10u [17] Amazon paper audits 9 policy decision benchmarks, finding that the scorer and reference answers themselves contain errors — https://aihot.news/items/gp0q36wvaswnf2rnc5gz9lhrb [18] How Stanford CS146S teaches AI

← Back to Updates