Contextualizebeta

tokenize

verb
/ˈtoʊ.kən.aɪz/US

Etymology

From token + -ize.

  1. inherited from tācnian
  2. inherited from toknen
  3. derived from *deyḱ-✻ — “to show, instruct, teach”
  4. inherited from *taikną✻ — “indicator, symbol, sign”
  5. inherited from *taikn✻
  6. inherited from tācn — “sign, symbol”
  7. inherited from token
  8. suffixed as tokenize — “token + ize”

Definitions

  1. To reduce to a token or set of tokens by lexical analysis.

  2. To substitute sensitive data with meaningless placeholders.

  3. To treat as a token minority.

The neighborhood

Derived

tokenizable, tokenization, tokenizer

Vish — recursive loop

No curated loop yet for tokenize. Loops are being traced one word at a time while the ingestion pipeline matures.

sense glosses and etymology drawn from English Wiktionary · source · CC-BY-SA