Microsoft open-sources ThinkingBox to test whether agents finish the job

Microsoft published ThinkingBox on August 19, 2026, an open-source sandbox that grades an agent on the state it actually leaves behind in a backend rather than on what it claims to have done. Agents run in isolated, stateful tool environments with a simulated user that answers follow-up questions. The paired ThinkingBox-Bench covers 507 policy-conditioned tasks across retail and ecommerce, travel and hospitality, auto insurance, neobank internal IT, and consulting IT and HR.

Why it matters

Microsoft ran 12 models, 6 proprietary and 6 open-weight, over 20 trials per task, and the reliability gap is the finding: the strongest model passed 65.36% of tasks on the first attempt but only 25.25% across all 20. A benchmark reporting single-attempt scores tells you very little about whether an agent is safe to run unattended.

Who should care

Anyone shipping agents that write to a real system, and teams picking a model for automation

What you can do

The runtime is at github.com/microsoft/thinkingbox and the task set at microsoft/thinkingbox-data. Both are open source, so the harness can be pointed at your own workflows.