Connectionist Temporal Classification (CTC), introduced by Alex Graves and colleagues in 2006, solved a fundamental problem in sequence-to-sequence learning: how to train a neural network to produce a label sequence when the alignment between input frames and output labels is unknown. CTC introduces a blank symbol that represents "no output" and defines the probability of an output sequence as the sum over all possible frame-level alignments that collapse to that sequence after removing blanks and repeated labels.
The CTC Loss Function
B: collapsing function (remove blanks, merge repeats)
B^{-1}(Y): set of all valid alignments for label sequence Y
π_t ∈ L ∪ {blank}: label at time t
Loss: L_CTC = −log P(Y|X)
The key insight of CTC is the collapsing function B, which maps a frame-level alignment path to an output label sequence by first removing all blank tokens and then merging consecutive identical labels. For example, the alignment "—aab—bc—" (where — denotes blank) collapses to "abc." The CTC loss is computed as the negative log-probability of the target sequence, which requires summing over the exponentially many valid alignment paths. This summation is performed efficiently using a forward-backward dynamic programming algorithm analogous to the HMM forward algorithm.
Conditional Independence and Decoding
CTC makes a strong conditional independence assumption: given the input, the output at each time step is independent of the outputs at all other time steps. This means CTC relies entirely on the encoder to capture temporal dependencies and cannot model output label dependencies directly. Despite this limitation, CTC works remarkably well when paired with powerful encoders (deep bidirectional LSTMs or Conformers) that provide rich contextual representations at each frame.
Greedy CTC decoding simply takes the most probable label at each frame and applies the collapsing function, but this ignores the fact that multiple alignment paths can contribute to the same output sequence. CTC prefix beam search maintains a set of active output prefixes and accumulates their probabilities across alignment paths, tracking separate probabilities for prefixes ending in blank versus non-blank. This algorithm yields significantly better results than greedy decoding, especially when combined with an external language model that scores partial hypotheses.
CTC is often combined with attention-based decoding in hybrid CTC/attention architectures, where a CTC loss applied to the encoder output provides a monotonic alignment constraint that stabilizes attention training, while the attention decoder models output dependencies. This joint training, implemented in frameworks such as ESPnet, has become a standard recipe for end-to-end ASR and achieves competitive results on benchmarks from LibriSpeech to multilingual tasks.
Beyond speech recognition, CTC has found applications in handwriting recognition, action detection in video, and any sequence labeling task where the alignment between input and output is unknown. Its elegant solution to the alignment problem remains one of the most influential contributions to sequence-to-sequence learning.