Microsoft patents language model that splits text for faster processing
Microsoft describes a method for training a single language model to handle both understanding and generating text. The system breaks text into smaller chunks and processes them separately before combining the results, which the filing says improves efficiency.
In the filing’s own words
Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.
What exactly they claim (claim 1)
1. A computer system that trains a natural language model to perform multiple different types of tasks, the computer system comprising: a processor; and a hardware storage device that stores instructions that are executable by the processor to cause the computer system to: access a natural language model; tokenize a set of training data to generate a plurality of tokens; mask one or more tokens of the plurality of tokens to generate masked training data; divide the set of training data into a plurality of portions; generate encoded representations for the plurality of portions, wherein at least some of the portions are encoded independently; apply an attention mechanism across the encoded representations of the plurality of portions; combine the encoded representations into a unified representation; and train the natural language model using the unified representation and the masked training data to enable the natural language model to perform multiple different types of tasks.
Facts
- Company
- Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
- Official title
- UNIFIED NATURAL LANGUAGE MODEL WITH SEGMENTED AND AGGREGATE ATTENTION
- Publication number
- US20260278306A1
- Application number
- 19683586
- Filed
- May 20, 2026
- Published
- Sep 17, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06F 40/56 Physics & computingG06F 40/149 Physics & computingG06F 40/284 Physics & computingG06F 40/40 Physics & computingG06F 40/51 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.