Google patents method for training language models to spot fake words
Google has patented a technique for training language models by mixing real words with fake ones generated by a separate system. The model learns to identify which words are original and which are replacements, improving its ability to understand language structure.
In the filing’s own words
Systems and methods are provided that train a machine-learned language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task where the encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example the proposed method masks out some subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a “generator” (e.g., which may be a small masked language model), and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator.
What exactly they claim (claim 1)
1. One or more non-transitory computer-readable media that store a machine-learned model comprising: an encoder model, the encoder model having been pre-trained by performance of pre-training operations, the pre-training operations comprising: obtaining an original input that comprises a plurality of original input tokens; respectively replacing one or more masked tokens in the original input with one or more replacement input tokens to form a noised input that comprises a plurality of updated input tokens, the plurality of updated input tokens comprising a mixture of the one or more replacement input tokens and the original input tokens that were not selected to serve as masked tokens; processing the noised input with the encoder model to produce a respective prediction for each updated input token included in the plurality of updated input tokens, wherein the prediction produced by the encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens; and training the encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the encoder model; and one or more neural network layers configured to process an output of the encoder model to generate an output for performing a task.
Facts
- Company
- Google (Google LLC)
- Official title
- Contrastive pre-training for language tasks
- Patent number
- US12801866B2
- Application number
- 18990884
- Filed
- Dec 20, 2024
- Granted
- Nov 17, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06F 40/40 Physics & computingG06N 5/04 Physics & computingG06N 20/00 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents method for LLMs to self-correct and learn from mistakes Oct 27, 2026
- Google patents search results that mix real clips with AI-made video Nov 10, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
