Computational Linguistics

A measure of how well a probability model predicts a sample, defined as the exponentiated average negative log-likelihood per token. Lower perplexity indicates that the model assigns higher probability to the test data, reflecting better language modeling performance.

Term Frequency-Inverse Document Frequency is a numerical statistic reflecting how important a word is to a document within a corpus. It increases proportionally with the number of times a word appears in a document but is offset by the frequency of the word across all documents, down-weighting common terms.

Bilingual Evaluation Understudy is an automatic metric for evaluating the quality of machine-translated text by measuring modified n-gram precision against one or more reference translations. A brevity penalty prevents degenerate short translations from scoring artificially high.

A neural network component that computes a weighted sum of value vectors, where the weights are derived from the compatibility between a query and a set of keys. Scaled dot-product attention divides by the square root of the key dimension to prevent vanishing gradients from softmax saturation in high-dimensional spaces.

A measure of association between two events that quantifies how much more (or less) likely they are to co-occur than expected under independence. Positive PMI indicates that the co-occurrence is more frequent than chance, making it a foundational tool for discovering collocations and semantic associations in text.

The average number of bits needed to encode data drawn from distribution p using a coding scheme optimized for distribution q. In language modeling, cross-entropy measures how well the model distribution q approximates the true distribution p of the language, serving as the standard training objective.

Dense, low-dimensional vector representations of words learned from co-occurrence patterns in large corpora. Words with similar distributional contexts receive nearby vectors in the embedding space, capturing semantic and syntactic regularities as geometric relationships.

The minimum number of single-character operations (insertions, deletions, and substitutions) required to transform one string into another, also known as Levenshtein distance. It provides a fundamental metric for measuring string similarity used in spelling correction, DNA sequence alignment, and approximate string matching.

The conditional probability of a word given its preceding context of n-1 words, estimated from corpus frequency counts. N-gram models apply the Markov assumption to make language modeling tractable, approximating the full joint probability of a word sequence as a product of local conditional probabilities.

Precision measures the fraction of predicted positive instances that are truly positive, while recall measures the fraction of actual positives that are correctly predicted. The F1 score is their harmonic mean, providing a single metric that balances both concerns and is especially useful when class distributions are imbalanced.

A discriminative undirected graphical model that defines a conditional probability distribution over label sequences given an observation sequence. Unlike generative models such as HMMs, CRFs can incorporate arbitrary overlapping features of the input without making independence assumptions about the observations.

A scalar parameter that controls the sharpness of the probability distribution produced by the softmax function. Lower temperatures sharpen the distribution toward a one-hot encoding (greedy selection), while higher temperatures flatten it toward uniform (more random sampling), providing a tunable exploration-exploitation tradeoff in text generation.

The information content of a single event, measured in bits when using log base 2. It quantifies the 'surprise' of observing an outcome: rare events carry more information than common ones. Self-information is the building block from which entropy and cross-entropy are derived.

A measure of similarity between two non-zero vectors defined as the cosine of the angle between them, ranging from -1 (opposite) through 0 (orthogonal) to 1 (identical direction). In NLP, it is the standard metric for comparing word embeddings and document vectors because it is invariant to vector magnitude, focusing purely on directional similarity.