GoogleGranted patentAI & softwareInterest 4/10

Google patents method for training language models to spot fake words

Patent drawing
Patent drawing from the official filing.

Google has patented a technique for training language models by mixing real words with fake ones generated by a separate system. The model learns to identify which words are original and which are replacements, improving its ability to understand language structure.

Why it mattersThis method offers a new way to improve how software understands text, potentially making language tools more accurate without needing massive amounts of labeled data.

In the filing’s own words

Systems and methods are provided that train a machine-learned language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task where the encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example the proposed method masks out some subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a “generator” (e.g., which may be a small masked language model), and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator.

What exactly they claim (claim 1)

1. One or more non-transitory computer-readable media that store a machine-learned model comprising: an encoder model, the encoder model having been pre-trained by performance of pre-training operations, the pre-training operations comprising: obtaining an original input that comprises a plurality of original input tokens; respectively replacing one or more masked tokens in the original input with one or more replacement input tokens to form a noised input that comprises a plurality of updated input tokens, the plurality of updated input tokens comprising a mixture of the one or more replacement input tokens and the original input tokens that were not selected to serve as masked tokens; processing the noised input with the encoder model to produce a respective prediction for each updated input token included in the plurality of updated input tokens, wherein the prediction produced by the encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens; and training the encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the encoder model; and one or more neural network layers configured to process an output of the encoder model to generate an output for performing a task.

Facts

Company
Google (Google LLC)
Official title
Contrastive pre-training for language tasks
Patent number
US12801866B2
Application number
18990884
Filed
Dec 20, 2024
Granted
Nov 17, 2026
Inventors
3
Claims
20

Classified under

G06F 40/40 Physics & computingG06N 5/04 Physics & computingG06N 20/00 Physics & computing

More from Google in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.