A fair arena for AI research assistants in biomedicine
BioMedArena is an open toolkit that makes it easy to build biomedical AI research agents and compare them fairly on the same footing.
The question
AI research agents can search, read and reason across many sources. But comparing them is surprisingly hard. The same underlying model can report different scores in different papers, because each team builds its own setup and tools. Plugging in a new model can cost weeks of one-off engineering. We call this the per-paper engineering tax.
What we did
We built BioMedArena, an open-source toolkit that separates the parts of an evaluation, such as loading the test, offering tools, choosing tools, running, handling context and scoring. It includes 147 biomedical benchmarks and 75 tools, and adding a new model, test or tool takes only a few lines.
What we found
Using the toolkit, we put together agents that achieved state-of-the-art results on 8 representative biomedical benchmarks, on average 15 percentage points above the previous best.
Why it matters
When everyone is tested the same way, differences in results mean something. An open arena makes AI research agents easier to build, compare and trust.