word
tokenization
TOH-ken-ih-ZAY-shun
ˌtoʊkənɪˈzeɪʃənListen
Hear it in a clear American voice. Slow it down to catch each syllable.
Syllable breakdown
tokenization
Dictionary meaning
the process of splitting text into smaller units called tokens — words, subwords, or characters — that can be processed by a language model; subword tokenization methods such as byte-pair encoding allow models to handle rare and out-of-vocabulary words by breaking them into known subword units
Learn one new pronunciation a day
Get a word, name or idiom like “tokenization” in your inbox — daily or weekly, free. Unsubscribe any time.
Something wrong? Suggest a correction