In running this long time-horizon simulation, we identified interesting behaviours from the models.

ORO@oroagents

In running this long time-horizon simulation, we identified interesting behaviours from the models. Models understand where they can improve yet fail to apply this learning to their actions. This notice-act gap has been seen in other domains as well: One DeepSeek run obeyed an instruction nobody sent. On day 65 it wrote that the last user turn had told it to delete its notes file and replace it with a single line, and called this a prompt-injection trap. No such message exists anywhere in the run. It half-complied anyway, creating a new memory file containing "Remember to check the weather." The instruction, the suspicion and the compliance were all its own. Gemini 4 Argon's tool understanding shows significant room for improvement. It loses its notes for weeks. In 15 of its 54 runs, it went 40 days or more without saving a note even though the environment gave it the feedback that its note was note saved. Those runs score far below the same task's other runs.

Quoting @OpenAI

We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment calls that real computational research depends on. openai.com/index/introducing-genebench-pro/

Open on X ↗