Finance benchmarks are making great AI agents look like bad analysts

Finance benchmarks are making great AI agents look like bad analysts. On Mercor's investment banking benchmarks, frontier models like GPT 6 Astra and Fable 5 only score 55–65%!

Abhyuday Roychowdhury and I have been digging into the tests used to measure whether AI agents can actually do analyst work.

Take this LBO modelling question from Mercor's APEX-Agents benchmark:

• The task asked the agent to find the 2025 revenue growth rate needed to deliver a 20% IRR in 2029.
• The agent answered 7.5%. It correctly changed 2025, which also updated the linked growth rates for 2026–2029. Growth went from 4.5%–4.1% to 7.5%–7.1%.
• The rubric marked this wrong. It only accepts 19.25%, with growth falling from 19.25% in 2025 to 4.1% in 2029.

The task was intentionally vague: it never said to change only 2025's growth rate and keep later years fixed. When I made that requirement explicit, the agent passed the test!

Most good analysts would interpret the task as the agent did and model a smooth growth curve.

These tests are being used to tell us how good AI agents are at finance. If they get marked down for exercising the judgement we'd expect from an analyst, we are systematically underestimating what the models can already do. And introducing bad judgement into the training data.

There's another issue here. Mercor (and other benchmark providers) sells specialised training data to AI labs. The worse AI looks at specialised work, the more valuable specialised training data looks.

I'll be writing more about this in the coming weeks and want to hear from analysts currently working on these tasks.

Links: Mercor's sample task and the APEX-Agents benchmark tasks on Hugging Face.