Contextualizebeta

tokenization

noun

Etymology

From token + -ization.

  1. inherited from tācnian
  2. inherited from toknen
  3. derived from *deyḱ-✻ — “to show, instruct, teach”
  4. inherited from *taikną✻ — “indicator, symbol, sign”
  5. inherited from *taikn✻
  6. inherited from tācn — “sign, symbol”
  7. inherited from token
  8. suffixed as tokenization — “token + ization”

Definitions

  1. The act or process of tokenizing.

  2. Something tokenized.

    • This was an unlikely tokenization of the input string.

The neighborhood

Vish — recursive loop

No curated loop yet for tokenization. Loops are being traced one word at a time while the ingestion pipeline matures.

sense glosses and etymology drawn from English Wiktionary · source · CC-BY-SA