/Patent Tracker
GooglePatent applicationAI & softwareInterest 4/10

Google patents method to speed up AI text generation using virtual batches

Patent drawing
Patent drawing from the official filing.

Google describes a technique for large language models that creates multiple 'virtual batches' during text generation. The system builds a map of how different parts of the text relate to each other, then processes them in specific groups to handle non-causal data more efficiently.

Why it mattersThis could make AI models run faster or cheaper by optimizing how they process information, though it is a technical improvement rather than a new feature.

In the filing’s own words

This document describes systems and techniques directed at virtual batches in non-causal large language model (LLM) inferences. An LLM generates a plurality of tokens. The plurality of tokens includes a prefix sequence. The LLM generates a dependency map. The dependency map includes one or more index markers for each of the plurality of tokens and a correlation marker for each of the one or more index markers. The LLM generates a plurality of virtual batches based on the dependency map. Generating the plurality of virtual batches may include applying an attention mask having a non-causal mask portion corresponding to the prefix sequence. Each of the plurality of virtual batches includes a discrete inference. The LLM selects one or more of the plurality of virtual batches as a final inference.

What exactly they claim (claim 1)

1. A method comprising: generating, with a large language model (LLM), a plurality of tokens comprising a prefix sequence; generating, by the LLM, a dependency map comprising: one or more index markers for each of the plurality of tokens; and a correlation marker for each of the one or more index markers; generating, based on the dependency map, a plurality of virtual batches, each of the plurality of virtual batches comprising a discrete inference; and selecting, by the LLM, one or more of the plurality of virtual batches as a final inference.

Facts

Company
Google (Google LLC)
Official title
Virtual Batches in Non-Causal Large Language Model Inferences
Publication number
US20260320364A1
Application number
19696173
Filed
Jun 2, 2026
Published
Oct 15, 2026
Inventors
2
Claims
20

Classified under

G06N 3/0499 Physics & computingG06F 16/33295 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.