OpenAI patents method to check AI thinking for cheating
OpenAI has filed a patent for a system that uses one AI model to watch the internal reasoning steps of another AI model. This monitoring model looks for signs of 'reward hacking,' where an AI might take shortcuts or act deceptively to get a high score instead of solving the task correctly.
In the filing’s own words
Systems and techniques are described herein for monitoring machine learning models for misbehavior. For example, an apparatus comprising one or more processors and configured to: provide, to a first machine learning model, an inner monologue from a second machine learning model associated with a user task, wherein the user task comprises natural language text; and obtain, from the first machine learning model, information indicative of misbehavior within the inner monologue. In some aspects, misbehavior is blocked during inference (e.g., runtime) and reward hacking behavior is penalized during training. Non-limiting examples of misbehavior include reward hacking, misgeneralization, sycophancy, or deception.
What exactly they claim (claim 1)
1. A method of monitoring machine learning models for reward hacking, comprising: providing, to a second machine learning model, an inner monologue from a first machine learning model associated with a user task, wherein the user task comprises natural language text; and obtaining, from the second machine learning model, information indicative of misbehavior based on monitoring the inner monologue.
Facts
- Company
- OpenAI (OpenAI OpCo, LLC)
- Official title
- MONITORING CHAIN-OF-THOUGHT IN REASONING MODELS TO MITIGATE REWARD HACKING
- Publication number
- US20260268214A1
- Application number
- 19289994
- Filed
- Aug 4, 2025
- Published
- Sep 10, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06N 20/00 Physics & computingMore from OpenAI in AI & software
- OpenAI patents voice system that detects when you stop talking Sep 22, 2026
- OpenAI patents system to catch AI models cheating on tasks Sep 10, 2026
- OpenAI patents system that turns photos into new AI-generated images Sep 10, 2026
- OpenAI patents AI that answers questions using only your company's secure files Sep 17, 2026
- OpenAI patents memory system for personalized AI chat responses Oct 8, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.