What is dictionary encoding and how much does it compress?
Dictionary encoding replaces repetitive string values with small integer codes. If a country column has 100,000 rows but only 50 distinct countries, each row stores a code (1–2 bytes) instead of the full string (8–20 bytes). Combined with bit-packing, compression ratios of 50–100× are common for repetitive columns. DuckDB and Parquet both use it.
Answered in
Columnar Storage: Why Column Stores Beat Row Stores for AnalyticsColumnar storage reads only needed columns, skipping the rest. Dictionary encoding shrinks data 50–100×. Analytics queries go from minutes to milliseconds.
Read the full analysisOther questions this article answers
More system design questions
- Why doesn't Google just run Dijkstra faster?
- What is a shortcut edge and when is it precomputed?
- How much space do shortcut edges take compared to the original graph?
- Can Contraction Hierarchies handle dynamic graphs like traffic or road closure?
- Why contract low-degree nodes first instead of high-degree ones?
- What is a CRDT and why does it matter for real-time collaboration?
- How do CRDTs handle concurrent edits without a central server referee?
- Why did Figma move from operational transforms to CRDTs?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design beat.