Google patents speech-to-text system that blends sound and language clues

1. A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving a sequence of acoustic frames characterizing an utterance; encoding, using a shared encoder, each acoustic frame in the sequence of acoustic frames to generate a corresponding encoded acoustic frame; generating, using a first-pass decoder, based on the encoded acoustic frames, streamed hypotheses; at each output step of a plurality of output steps for a second-pass decoder: determining, using a same attention mechanism, both: an acoustic context vector that summarizes the encoded acoustic frames; and a linguistic context vector based on a sequence of decoded labels previously output by the second-pass decoder; determining, using the acoustic context vector, as output from the second-pass decoder, an acoustic-based probability distribution over possible output labels; determining, using the linguistic context vector, as output from the second-pass decoder, a text-based probability distribution over possible output labels; and interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels, wherein the possible output labels in the acoustic-based probability distribution and the possible output labels in the text-based probability distribution comprise a same set subword units; and determining a transcription of the utterance based on the interpolating of the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels at each of the plurality of output steps.