Google patents speech-to-text system that uses language models to find natural pauses

1. A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to implement a joint segmenting and automated speech recognition (ASR) model comprising: an encoder configured to: receive, as input, a sequence of acoustic frames characterizing one or more spoken utterances; and generate, at each output step of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and a decoder configured to: receive, as input, the higher order feature representation generated by the encoder at each output step of the plurality of output steps; and generate, at each output step of the plurality of output steps: a probability distribution over possible speech recognition hypotheses; and an indication of whether the output step corresponds to an end of segment, wherein the joint segmenting and ASR model is trained on a set of training samples, each training sample in the set of training samples comprising: audio data characterizing multiple segments of long-form speech; and a corresponding transcription of the long-form speech, the corresponding transcription annotated with ground-truth end of segment labels obtained via distillation from a language model teacher that receives the corresponding transcription as input and injects the ground-truth end of segment labels into the corresponding transcription between semantically complete segments, wherein: the language model teacher is trained on a corpus of written text containing punctuation to teach the language model teacher to learn how to semantically predict ground-truth end of segment labels based on positions of punctuation in the written text; the language model teacher processes the corresponding transcription to generate predicted punctuation; and an augmentor inserts a ground-truth end of segment label into the corresponding transcription for each comma, period, question mark, and exclamation point predicted by the language model teacher.