the study measured what the vibes wouldn't admit

A glowing upward arrow whose shadow points downward, above dark machinery of gears, a sandbox cube, and a queue of light packets.

Week ending July 11, 2025; archive coverage: July 10, July 11.

Every week the benchmarks go up and every week someone tells me agents are about to replace us. So it's refreshing when someone actually straps a camera to the problem and hits record. This week they did, and the number came back sideways.

The short version: the tools might be great, but our ability to tell whether they're helping us is genuinely broken. People felt faster and were slower. That's not a rounding error, that's a perception gap you could drive a roadmap through. The rest of the week's news is, conveniently, a tour of the actual engineering that determines whether these things help at all — sandboxes, transports, expert modules — the unglamorous plumbing nobody puts on a landing page.

So let's take the reality check seriously and then look at where the real work is.

the number nobody wanted

METR recruited 16 experienced maintainers from big, familiar repos, gave them 246 real issues, and randomly allowed or forbade AI per task. Screen recordings, real work, frontier tools (mostly Cursor with Claude 3.5/3.7). Result: 19% slower with AI. The kicker is the self-report. Devs forecast a 24% speedup going in, and even after eating the slowdown they still believed AI had sped them up 20%.

Read the caveats, because METR wrote them for you. This is experienced people on codebases they know cold, where implicit requirements — tests, linting, docs, high quality bars — are exactly where the model burns your time. It is not a claim that AI is useless, or that juniors or unfamiliar codebases look the same. But it does land one durable blow: anecdotal speedup estimates are worthless. If you're deciding tooling by how fast it feels, you're flying blind. And unlike a benchmark, this methodology is hard to game — which is precisely why I trust it more.

open models that don't blow the budget

Meanwhile the models keep getting genuinely better and cheaper. Devstral Small 1.1 shipped Apache 2.0 at 53.6% on SWE-Bench Verified — state-of-the-art among open models, 24B params, runnable locally, with the nice touch that it generalizes across scaffolds and supports both function calling and XML. Devstral Medium hits 61.6% and, per Mistral, undercuts Gemini 2.5 Pro and GPT-4.1 at roughly a quarter of the price.

Hold this next to METR and the point sharpens: raw capability isn't the bottleneck. A cheap, open, fine-tunable model you control is worth more than a leaderboard king if the leaderboard doesn't predict your actual throughput.

the magic is a for-loop and a sandbox

The best antidote to model worship this week was Beam's teardown of how Lovable and Bolt actually work. They built a clone to find out, and the answer is deflating in the best way: plan, edit files, run the app in a sandbox, feed errors back, repeat, then cross a deployment boundary. The visible magic is a preview iframe over a very ordinary loop.

Their conclusion matches mine — you don't need a better model, you need better system design: typed prompts, tests, retries, context management, isolation. "It's not magic. It's just software engineering." Frame that.

agents that don't have to answer right now

Which brings me to plumbing I actually got excited about. asyncmcp adds async transport layers to MCP over SQS, SNS, and webhooks. The idea: an MCP server doesn't have to respond immediately. It can drop the request on a queue and the client can walk away. That sounds boring until you've tried to bolt batch APIs, long jobs, or webhooks onto a protocol that assumes synchronous request-response. Durable, event-sourced agent runtimes want exactly this. It's early, but it's the right shape.

training without surrendering the data

Finally, Ai2's FlexOlmo tackles the other bottleneck: data. Owners train an expert module locally against a frozen public "anchor" model, then contribute the module into a mixture-of-experts — no raw data shared, opt in or out at inference, attribution when your data is used. They report performance near a model trained on all the pooled data, and a 0.7% extraction rate (with DP available if you want formal guarantees). Whether the coordination holds at scale is the open question, but the politics are the point: control and credit for data owners in healthcare, government, finance. That's a different distribution of power than "scrape everything."

what I'd do Monday

Measure, don't vibe. Before you standardize on an agent, run a small paired trial on real tickets and time them — your intuition is, per METR, actively lying. Invest in the loop, not the model: sandbox, error feedback, tests around prompts. Try Devstral Small locally to see how far cheap and open gets you. And if any of your tools involve batch or long-running work, look hard at async MCP transports before you hand-roll polling.