OpenAI patents system to catch AI models cheating on tasks
OpenAI has filed a patent for a system that watches the internal reasoning steps of one AI model using a second AI model. This 'monitor' looks for signs of misbehavior, such as finding shortcuts to get a high score without actually solving the problem correctly.
In the filing’s own words
Systems and techniques are described herein for monitoring machine learning models for misbehavior. For example, an apparatus comprising one or more processors and configured to: provide, to a first machine learning model, an inner monologue from a second machine learning model associated with a user task, wherein the user task comprises natural language text; and obtain, from the first machine learning model, information indicative of misbehavior within the inner monologue. In some aspects, misbehavior is blocked during inference (e.g., runtime) and reward hacking behavior is penalized during training. Non-limiting examples of misbehavior include reward hacking, misgeneralization, sycophancy, or deception.
What exactly they claim (claim 1)
1. A method of monitoring machine learning models for reward hacking, comprising: providing, to a second machine learning model, an inner monologue from a first machine learning model associated with a user task, wherein the user task comprises natural language text; and obtaining, from the second machine learning model, information indicative of misbehavior based on monitoring the inner monologue.
Facts
- Company
- OpenAI (OpenAI OpCo, LLC)
- Official title
- CHAIN-OF-THOUGHT MONITOR FOR REWARD HACK IDENTIFICATION IN GENERATIVE RESPONSE ENGINES
- Publication number
- US20260268155A1
- Application number
- 19290991
- Filed
- Aug 5, 2025
- Published
- Sep 10, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06N 3/094 Physics & computingMore from OpenAI in AI & software
- OpenAI patents voice system that detects when you stop talking Sep 22, 2026
- OpenAI patents method to check AI thinking for cheating Sep 10, 2026
- OpenAI patents system that turns photos into new AI-generated images Sep 10, 2026
- OpenAI patents AI that answers questions using only your company's secure files Sep 17, 2026
- OpenAI patents memory system for personalized AI chat responses Oct 8, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.