Researchers from Microsoft and City University of Hong Kong have introduced Taste-Bench to evaluate how well artificial intelligence agents make decisions during long-running tasks. Current benchmarks only measure whether an agent completes a task, rather than assessing the quality of its steps along the way. The new method uses hindsight from thousands of scored agent runs on coding and research tasks to identify decision forks, where alternative paths diverge. By freezing runs at these forks, the researchers created 502 questions to test if models can choose the superior long-term direction.
The team tested 14 models from leading families including GPT, Claude, Grok, DeepSeek, GLM, MiniMax, and Mistral. The best-performing model achieved an average accuracy of only 59.7 percent, with all other models scoring significantly lower on the binary choice evaluation. The results demonstrated that models struggle severely when the consequences of a decision only become apparent after extensive later work, with accuracy dropping to near random guessing when foresight is required. Furthermore, increasing reasoning settings did not significantly improve model performance on these complex, multi-step decisions.
This research matters because as AI agents increasingly handle complex, multi-step projects like software development and research, their ability to make sound long-term strategic choices remains a major bottleneck.