Making Git Fast in a Huge Monorepo: Sparse Checkout, Partial Clone and Friends
At 200,000 files, clone, checkout and status take minutes. The Git features that fix it, the exact commands, and what a tool like Sparo adds on top.
On this page
Nobody decides a monorepo is a mistake on day one. Code sharing is easy, dependency updates are atomic, and one change can touch ten projects. Years later git status takes seven seconds and a fresh clone is a lunch break. The fix is rarely “split the repo”. It is “stop asking Git to look at everything”.
Why monorepos exist, and what they cost
| Benefit | Why teams want it |
|---|---|
| Code sharing and reuse | One copy of a library, no version drift |
| Simple dependency management | Projects reference each other directly |
| One view of the codebase | Cross-cutting refactors and uniform tooling |
The cost is that Git’s work grows with repository size: files to scan for status, objects to download for clone, entries to update for checkout. TikTok’s front-end repository grew past 1,000 projects and 200,000 source files, with clones taking over 20 minutes on a fast connection and over 40 on a slow one, a checkout around a minute and a half, status seven seconds and a commit fifteen.
Fix 1: sparse checkout, fewer files on disk
A developer working on one app needs that app and its dependencies, a small, slowly growing slice of the whole.
git clone --filter=blob:none --no-checkout git@host:org/monorepo.git
cd monorepo
git sparse-checkout init --cone
git sparse-checkout set apps/web packages/ui packages/utils
git checkout mainCone mode matches whole directories, which is both simpler and much faster than arbitrary patterns. Now status, checkout and file-watching touch a fraction of the tree.
Fix 2: partial clone, fewer objects downloaded
Even with sparse checkout, a normal clone downloads every file’s content for all history. A partial clone, using --filter=blob:none, fetches commits and trees up front and file contents only when a checkout needs them.
| Approach | Keeps history | Downloads content up front | Works for all files |
|---|---|---|---|
| Full clone | Yes | Everything | Yes |
Shallow clone (--depth) | No, truncated | Recent only | Yes, but history missing |
| Git LFS | Yes | Pointers only | Only files you configure |
Partial clone (blob:none) | Yes | Commits and trees | Yes, automatically |
Operations that need old file contents, such as blame over a long history or a wide git log -p, trigger on-demand fetches and are slower. That is the correct trade for most developers, but go offline and you can only read what you already fetched.
Fix 3: stop re-scanning the disk
git config core.fsmonitor true lets Git ask the operating system what changed instead of walking the whole tree. Combined with the untracked cache, status stops scaling with repository size.git maintenance start schedules prefetch, commit-graph updates and repacking so performance does not decay between your manual cleanups.scalar clone, that applies partial clone, sparse checkout, fsmonitor and maintenance with sensible defaults in one step.Fix 4: share the slice, not the incantation
Commands are easy. Knowing which directories your team needs is the hard part for a new hire. TikTok’s Sparo tool adds two ideas on top of the Git features:
- Checkout profiles
A JSON file in the repository names the projects and folders for a team, for example “web app, its dependencies, docs”. New developers run a single command with the profile and get exactly their slice. Because profiles are versioned with the code, they stay accurate.
- Mirrored commands
A CLI that mirrors Git’s commands (
clone,checkout,status,commit) and configures sparse and partial behaviour automatically. Teams adopt it gradually, falling back to plain Git for unusual cases.
The reported results, on the same repository: clone from about 23 minutes to 2, checkout from 90 seconds to 30, status from 7 seconds to 1, commit from 15 seconds to 11.
What to do before reaching for a tool
| Symptom | First move |
|---|---|
| Clone takes forever | --filter=blob:none, and consider --depth only for CI |
| Checkout and switch are slow | Sparse checkout in cone mode |
git status is slow | core.fsmonitor and untracked cache |
| History commands are slow | Commit-graph, background maintenance |
| New hires cannot find their projects | Profiles or documented sparse sets |
| CI clones the whole thing every run | Shallow, filtered, sparse clones scoped to changed projects |
CI deserves attention of its own: build only what changed and its dependents, and clone only that slice. Build-graph tools that understand project dependencies make “affected projects” cheap to compute.
Do
Measure clone, checkout, status and commit times as team metrics, set defaults in onboarding docs or a bootstrap script, keep profiles in the repo, and review them when projects move.
Don't
Split a healthy monorepo into dozens of repositories to fix slow Git, and then rebuild its benefits with version-pinning and release coordination. Fix the tooling first.
Takeaway
Monorepo slowness is a symptom of doing whole-repository work for a slice-sized job. Check out only what you need, fetch contents only when you need them, let the filesystem tell Git what changed, and publish team slices as profiles. Do those and a 200,000-file repository feels like a small one.
Frequently asked questions
What is sparse checkout in Git?
A setting that makes Git materialise only the directories you ask for in your working tree, instead of every file in the repository. Git operations then scan far fewer files, so checkout, status and commit get faster. The full repository data can still be present or fetched later.
How is a partial clone different from a shallow clone?
A shallow clone truncates history to the last few commits. A partial clone keeps the full history and omits object contents, typically file blobs, downloading them lazily when needed. With the blob:none filter you still have every commit and tree, just not every file's content until you check it out.
Does a monorepo make Git unusable at scale?
No, but default Git settings assume repositories of moderate size. Sparse checkout, partial clone, a filesystem monitor, commit-graph and background maintenance together bring operations back to seconds. Google-scale monorepos also use custom tooling, which is why helper CLIs exist.
What are checkout profiles?
Named lists of directories or projects, stored in the repository, that define a team's slice of a monorepo. A new developer applies the profile and gets exactly their team's projects and dependencies without learning the repository layout. The idea builds on sparse checkout.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.