Microsoft patents faster AI text generation via expanded token matching
Microsoft has filed a patent for a technique that speeds up how AI models generate text by letting a smaller "drafting" model suggest words that a larger "verifying" model checks. The system uses a specific matching rule to accept some suggested words even if they aren't perfect matches, which reduces the time and computing power needed to produce a response.
In the filing’s own words
A speculative decoding technique generates tokens of a response using a verifying model and drafting model. In operation, the verifying model receives a set of draft tokens produced by the drafting model. The verifying model then verifies whether the draft tokens are correct by comparing probability information generated by the verifying model with probability information generated by the drafting model. The verifying model then determines whether any draft token that is rejected by the verifying is otherwise accepted based on a specified matching criterion. The above process is repeated one or more times to generate the response. The technique has the overall effect of expanding a number of draft tokens that are accepted, thereby reducing use of resources and deceasing latency.
What exactly they claim (claim 1)
1. A method for executing operations by a first machine-trained model, comprising: receiving a set of draft tokens that have been produced by a second machine-trained model; verifying correctness of the draft tokens in the set of draft tokens by comparing probability information generated by the first machine-trained model with probability information generated by the second machine-trained model; determining whether any draft token in the set of draft tokens that is rejected by the verifying is accepted based on a specified matching criterion; and providing draft tokens of the set of draft tokens that have been accepted by the verifying or the determining to the second machine-trained model for use by the second machine-trained model in generating another set of draft tokens.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- Speculative Decoding using Expanded Token Matching
- Publication number
- US20260260065A1
- Application number
- 19067886
- Filed
- Mar 1, 2025
- Published
- Sep 3, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06F 40/284 Physics & computingG06F 16/33295 Physics & computingG06F 16/345 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.