How does tokenization, stemming, and lowercasing affect the index?
Ingestion splits text into tokens, normalizes them to lowercase, and stems them to root forms (e.g., 'testing' → 'test'). This deduplicates terms across documents—'tests' and 'testing' both map to 'test'—reducing index size and improving recall so searches find more relevant results.
Answered in
The Inverted IndexA sorted dictionary mapping terms to document IDs. Transform search from O(corpus size) to O(1) lookup, enabling full-text search at scale.
Read the full analysisOther questions this article answers
More system design questions
- Why does an index's internal data structure matter if it all ends up 'faster than a scan'?
- Why is disk I/O the thing index structures are actually optimizing for?
- Why can't a hash index handle range queries?
- What makes a bitmap index different from a B-tree, and when is it better?
- Why do B-trees stay balanced automatically as data is inserted?
- Why does a database need an index at all — why can't it just scan the table?
- When is a hash index better than a B-tree index?
- What is a composite index and why does column order matter?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design beat.