perplexity (NLP)

Syllable breakdown

Dictionary meaning

the standard metric for evaluating language models, measuring how well a probability model predicts a sample of text; formally, perplexity is the exponentiated cross-entropy of the model on the test set — equivalently, the geometric mean of the inverse probabilities assigned to each token, raised to the power of the number of tokens; a perplexity of N means the model is on average as uncertain as if it had to choose uniformly among N equally likely possibilities at each token; lower perplexity indicates a better-fitting model that is less surprised by the test text; perplexity is used to compare language models trained on the same data and to track the effect of architectural improvements, training data expansion, and scale on model quality; limitations of perplexity as an evaluation metric include its dependence on tokenisation — perplexity scores are not comparable across models with different tokenisers — and its poor correlation with downstream task performance for large language models, which has led to the development of task-specific benchmarks

Examples

• GPT-3 achieved a perplexity of 20.5 on the Penn Treebank, compared to 35 for the best LSTM language model.
• Researchers found that perplexity improvements from scaling did not always translate to improved performance on reasoning benchmarks.