Microsoft patents prompt refinement loop for multimodal models
Microsoft describes a method to improve how large multimodal models handle instructions without retraining them. The system uses a cycle where a prompt is converted to another format, converted back, and compared to the original to generate hints for improvement. This iterative process continues until the prompt and its translation match, effectively refining the input using the model's own output as a guide.
In the filing’s own words
This disclosure presents a technique for enhancing the performance of models such as large multimodal models (LMMs) without retraining or fine-tuning. This technique includes an iterative refinement process implemented by three main components: a forward generator, a backward generator, and a discriminator. The forward generator translates a prompt into an output in a different modality, the backward generator translates this output back into the original modality, and the discriminator compares the prompt and the translated prompt to generate a hint for refining the prompt to reduce differences. This cycle continues until the original prompt and the translated prompt match, achieving cycle consistency. The solution offers several advantages, including improving model performance without the need for costly fine-tuning, training data, or expertise. It simplifies system complexity by not relying on external environments like compilers and APIs and uses cycle consistency as a supervisory signal to iteratively refine a prompt.
What exactly they claim (claim 1)
1. A method for improving modality translation by a model comprising: (a) receiving a prompt in a first modality; (b) translating the prompt by a forward generator to an output in a second modality; (c) translating the output by a backward generator to a translated prompt in the first modality; (d) generating, by a discriminator, a revised prompt based on differences between the prompt and the translated prompt both in the first modality; and repeating steps (a)-(d) until a stopping criterion is reached.
Facts
- Company
- Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
- Official title
- Cycle-consistent refinement of prompts provided to models
- Patent number
- US12737559B2
- Application number
- 18731126
- Filed
- May 31, 2024
- Granted
- Sep 15, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06F 40/40 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.