While building Ekit, a collaborative SaaS tool relying on highly dynamic schemas, we ran into a recurring MongoDB challenge that is common in developer-focused platforms.
When users can freely define their own custom fields, creating new indexes for every schema change quickly becomes unmanageable.
This article shares a production-tested approach that proved reliable on a dataset of roughly 15 million documents.
The challenge
The core difficulty lies in allowing user-defined fields while keeping query performance predictable.
On small collections, wildcard indexes can work reasonably well, but their behavior becomes far less reliable as data volume grows.
The setup
To keep indexing behavior stable and predictable, we implemented an index slotting approach.
User-defined fields are mapped to a small, fixed set of indexed slots.
To validate the design, approximately 15 million documents were injected into the collection.
Initially, we assumed performance would mostly depend on having the right indexes. In practice, query shape and pagination strategy turned out to be just as critical.
- 4 string slots
- 1 integer slot
- 1 date slot
What made the biggest impact
Anchored prefix search
By structuring queries around anchored prefix searches, some lookups dropped from several seconds down to roughly 2–3 milliseconds.
Using explain() made the reason clear: once the index was properly used, the number of scanned documents collapsed.
Keyset pagination instead of offsets
Offset-based pagination using skip() and limit() does not scale at this data volume.
Switching to a keyset-based pagination strategy allows each query to scan a small, predictable range.
As a result, latency remains stable around 2 milliseconds, even when navigating deep into the dataset.
Outcome
Once these changes were in place, the system stabilized very quickly.
This is not meant as a silver bullet, but it was reassuring to see MongoDB handle this workload cleanly while we continued working on Ekit’s collaborative UX layer.
This article was originally shared on r/mongodb and is part of my ongoing technical notes while building Ekit.
Discussion
- How are you handling fast prefix searches at scale in MongoDB?
- Have you experimented with index slotting versus wildcard indexes for dynamic schemas?