Distributed, cloud-native bitmaps · roaring shipped · v0.10.0
Big sets, the same six calls, and no bill while idle.
CloudBitmaps keeps large id-sets as immutable objects in storage you already own, and reads only the chunks a query can match — 100 of 2,000. The reads are the calls you know — has, count, iterate, intersect, union, andNot — and the bill is the part that changes: 1.2 GiB at rest is $0.03 a month, against $346 for a node standing by.
npm i @cloudbitmaps/roaring @cloudbitmaps/s3
A codec and a storage · memory drivers need no setup
Many use cases, one shape.
Many ids, computed in batches, mostly idle, and it has to survive a restart. That’s the shape — these five are just where it turns up most often. If it describes a list you maintain, it fits, whatever you call it.
| The job | What’s in the set | You call | How it’s used |
|---|---|---|---|
| Audience segmentation | user ids who bought in the last 90 days AND opened the app this week | intersect | rebuilt on a schedule, read on a fraction of requests |
| Feature-flag & experiment cohorts | the ids assigned to each variant | has | read on nearly every request, written rarely |
| Permission & entitlement sets | ids allowed to see a given resource | has | read constantly, and must survive a restart |
| Recommendation candidate pools | ids eligible before ranking narrows them | intersect | narrowed per request instead of scanned |
| Suppression lists | never-contact, opted-out, already-seen | andNot | reloaded whole, then subtracted inside the read |
Also fits Fraud and abuse blocklists · batch deduplication, the “which of these have I already processed?” set · follower and social-graph edges · consent and regional-eligibility registries · and distinct-count reporting, where count() is exact rather than a sketch with an error bar.
Ids must be u32 An id is a 32-bit unsigned integer: it splits into a 16-bit chunk key and a 16-bit remainder, and anything else raises a ValidationError. UUIDs and string keys need a dense-integer mapping that you own and operate — worth knowing before you install, not after.
intersect and andNot are marked because they are the operations the tiering exists for — the ones that would otherwise pull whole sets into memory. Large means the measured run behind this page: two 2,000,000-id segments, of which an intersection fetches 100 chunks of 2,000. Step through it.
The set is large, mostly read, and computed in batches.
Audience lists, feature cohorts, permission sets — bigger than you want to keep resident, or heading that way, and produced by something upstream: a warehouse query, a nightly job, a combine of two other segments. You load the result.
It has to survive a restart, and cost nothing while idle.
The corpus outlives any one process, your compute is stateless, and you would rather not run a node at all just so the set is there when something asks.
The data has to stay yours.
Your own storage account, a documented format, and a way out that does not involve us — and someone may have to be erased on request, which here rewrites the generation without them and deletes the one that held the bit.
You need a sub-millisecond p99.
Not just on hits. Our cache tier is in-process RAM, but a miss is a ranged GET against object storage, where a resident set is a memory read. A sub-millisecond tail needs the whole set resident, which is a different machine — and we publish no in-region latency figure until an in-region run measures one.
The set changes one id at a time.
There is no add and no remove here: a segment changes only by publishing a whole new generation. Accumulate per-id writes where they arrive — Redis is genuinely good at that — and load the set on a cadence.
For the first one that usually means Redis — Redis as your set store, which is a different thing from Redis in front of one. And past 329.15 reads a second, every one of them a cache miss, metered GETs cost more than a node standing by: the cache hit rate is what moves that line.
Two tiers, one pointer, one API
A segment is immutable objects in your bucket plus one row saying which of them is current. You call has(); the store decides whether that costs a read at all.
Immutable, generation-keyed objects in your own bucket. Documented format, one-command export.
One small row per segment, saying which generation is current. A publish moves it forward by compare-and-swap, and never back.
A bounded LRU of decoded chunks inside your process. Nothing is pinned, and a new generation misses rather than serving stale bytes.
A ∩ B without downloading A or B
Intersecting two 2,000,000-id segments does not download two segments. Chunk keys are compared first, and only chunks whose keys appear in both are ever fetched — 100 of 2,000, in 24.6 ms. The other 1,900 are never fetched, and never billed.
2 tiers 4 drivers 0 dependencies in @cloudbitmaps/core
import { CloudRoaring, MemoryStorage }
from '@cloudbitmaps/roaring';
const store = new CloudRoaring({ storage: new MemoryStorage() });
// one write-once object, and then the pointer moves to it
await store.load({ segment: 'bought-last-90d' }, [75, 99_999]);
const bought = store.segment('bought-last-90d');
const opened = store.segment('opened-this-week');
// fetches only the chunks that can contribute
for await (const id of bought.intersect([opened])) { … }The hard part is the protocol, not the compression.
Roaring already solves the compression. Putting it in object storage creates four problems that have nothing to do with bit math: a format something other than your own code can read, a refresh that cannot be half-visible, a read that cannot straddle two versions of the same set, and a delete that actually deletes.
The problem
What CloudBitmaps does
- !A private chunk layout leaves the data only as portable as the code that wrote it, and unreadable by anything else you own
- ✓A documented .crbm format, generation-keyed and immutable, with a one-command export
- !A refresh that writes over the live bytes leaves a reader a set that is neither the old one nor the new one
- ✓Nothing overwrites a stored object: a load writes a new generation and then moves the pointer forward, so a crash before the publish leaves the previous set live
- !A read assembled from more than one source is torn by a write that lands mid-flight
- ✓One generation per read, resolved once before any fan-out — and every chunk is CRC32C-verified before it reaches the deserializer
- !Immutable objects make one person's erasure the hard case, and masking a bit is not deleting it
- ✓eraseSubject() rewrites the generation without that id and deletes the one that held it — gone from the bucket when the call returns
All four rest on a protocol written down as 7 hard correctness invariants — write-once generations published forward-only, one generation per read, a GC that never touches the current one, every tier byte untrusted. Each has named tests, and they run on every commit.
A codec, a storage, one bucket.
Install the codec you want and the storage you have — each storage package brings its own cloud SDK, so nothing is an optional peer and no install carries an SDK you never use. The memory drivers need nothing at all — the segments.ts sample above runs as-is, so you can see the shape before you point it at a bucket.
npm i @cloudbitmaps/roaring @cloudbitmaps/s3
# the codec you want, and the storage you have
# also: @cloudbitmaps/gcs, @cloudbitmaps/azure-blob