Opinion OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall Coding Assistants AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token AI Agents Tencent and King's College London Build an Agent Memory Test From 80,299 Real Group-Chat Messages News Tesla Starts FSD v15 With a Generative Reality View and a 25x Intervention Claim It Has Not Published Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Opinion OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall Coding Assistants AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token AI Agents Tencent and King's College London Build an Agent Memory Test From 80,299 Real Group-Chat Messages News Tesla Starts FSD v15 With a Generative Reality View and a 25x Intervention Claim It Has Not Published Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent

OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall

OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall

OpenAI released MedChemBench, 99 medicinal-chemistry problems drawn from 117 papers and patents and 761 molecular structures. GPT-6.2 Astra earned 84.5% of available credit at its highest reasoning setting, 95.4% on rebuilding structures, and about half on choosing compounds and predicting properties.

OpenAI published MedChemBench on November 20, a benchmark of 99 problems drawn from 117 papers and patents and covering 761 distinct molecular structures. The premise is that medicinal chemistry is one of the harder scientific disciplines to automate, not because the individual steps are exotic but because they trade against each other: a change that makes a compound bind a disease protein more tightly can leave it poorly absorbed, or broken down by the liver before it reaches the target. Useful assistance therefore has to connect a structural change to experimental evidence and judge which compound is worth making next.

Each problem hands the model the context available at one point in a research programme - the biological target, the team's goals, the compounds already made and selected results - and grades four abilities. Rebuilding molecular structures correctly, including atom connectivity and stereochemistry. Explaining results using the supplied evidence. Ranking and selecting compounds against stated requirements. And predicting numerical properties or categories reported in the source material, with partial credit scaled to how close the prediction lands. One worked example uses PRMT5 inhibitors from cancer research: the model must predict how three related compounds behave in cells, across a cell barrier and against liver enzymes, while the withheld experiments show a split in which one compound is more potent and another crosses the barrier more readily and degrades more slowly.

OpenAI evaluated eleven of its GPT models at several reasoning settings. At the highest setting shared across all of them, GPT-6.2 Astra earned 84.5% of the available credit, against 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol and 58.2% for GPT-5.6 Sol. Raising reasoning effort improved scores without closing the gaps between models. The component breakdown is more informative than the total: Astra took 95.4% of the structure credit at that setting and was strongest at reconstruction and evidence-based explanation, while compound selection and numerical property prediction earned only about half. In one structure problem requiring an 18-atom ring to be preserved, GPT-6 Astra returned the correct molecule in all ten attempts; GPT-5.6 Sol produced a 15-atom ring and GPT-5.6 Luna split the molecule into two disconnected fragments, failing all ten. Errors of that kind do not degrade an answer, they change which molecule is being discussed.

OpenAI is unusually explicit about what the numbers do not show. The results do not establish that models can discover drugs or improve decisions in a laboratory; the benchmark evaluates reasoning over historical evidence rather than real-world discovery; each response was a single attempt with no tools, feedback or new experiments; and prior exposure to the published sources cannot be ruled out. The last point is the structural problem with any benchmark assembled from the literature, and it is not solved by scoring.

Two discrepancies are worth flagging. The introduction reports GPT-6.2 Astra at 84.5% while a later passage on the full benchmark at the shared setting lists GPT-6 Astra at 83.3%, and the page does not clearly reconcile whether these are different versions or different scoring conditions, so the two should not be read as directly comparable. Separately, the third-party tracker llm-stats.com lists GPT-5.6 Sol at 48.3% rather than OpenAI's 58.2%, marks the scores as self-reported and unverified, and lists the dataset and code as forthcoming. No independent re-run has been reported.

The useful signal is in the shape of the results rather than the headline. Reconstruction of a molecule from a drawing is close to solved at the frontier, explanation grounded in supplied evidence is strong, and the two tasks that require judgement under uncertainty - picking which compound to make and forecasting a measured number - sit near half credit. That split matches where medicinal chemists say the work actually is: the drawing is bookkeeping, the decision is the job.

For the field, MedChemBench is a better instrument than a leaderboard because it decomposes a vague claim into tasks that can be inspected and disputed. The next step it points to is harder: problems whose answers were not published before the model was trained, so that a high score measures chemical reasoning rather than recall of the literature. Until then, 84.5% is a statement about how well a model can work with what chemists already wrote down.

Comments (0)

Log in to join the discussion

Log In

No comments yet