language model

Syllable breakdown

Dictionary meaning

a probabilistic model that assigns probabilities to sequences of words or tokens, capturing the statistical structure of language by learning from large text corpora; language models are foundational to NLP because they can be used for text generation, scoring candidate outputs in machine translation and speech recognition, and as the basis for transfer learning in downstream tasks; early n-gram language models estimated the probability of a word given its n-1 preceding words from corpus counts with smoothing to handle unseen sequences; neural language models using LSTMs and then transformers dramatically improved perplexity on language modelling benchmarks by learning longer-range dependencies; large pretrained language models — GPT-3, GPT-4, LLaMA, PaLM, Claude — trained on hundreds of billions of tokens from the internet and books have demonstrated emergent capabilities including few-shot learning, instruction following, reasoning, and code generation that have transformed the field; the scaling hypothesis that model capability improves predictably with model size and training data has driven the rapid increase in language model scale

Examples

• A language model assigns a higher probability to 'the cat sat on the mat' than to 'the mat sat on the cat' because the first is more consistent with English usage patterns.
• GPT-4 is a large language model trained on a diverse internet corpus that can generate coherent essays, write code, and answer questions across many domains.