Distributed, cloud-native bitmaps · roaring shipped · v0.4.1
Big sets, the same five calls, and no bill while idle.
CloudBitmaps keeps large id-sets as bitmaps in object storage you already own, and reads only the chunks a query can match — 100 of 2,000. The calls don't change — add, has, remove, count, intersect — but the bill does: 1.2 GiB at rest is $0.03 a month, against $346 for a node standing by.
npm i @cloudbitmaps/roaring
One flavor · memory drivers need no setup
Many use cases, one shape.
Many ids, mostly idle, and it has to survive a restart. That’s the shape — these five are just where it turns up most often. If it describes a list you maintain, it fits, whatever you call it.
| The job | What’s in the set | You call | How it’s used |
|---|---|---|---|
| Audience segmentation | user ids who bought in the last 90 days AND opened the app this week | intersect | rebuilt on a schedule, read on a fraction of requests |
| Feature-flag & experiment cohorts | the ids assigned to each variant | has | read on nearly every request, written rarely |
| Permission & entitlement sets | ids allowed to see a given resource | has | read constantly, and must survive a restart |
| Recommendation candidate pools | ids eligible before ranking narrows them | intersect | narrowed per request instead of scanned |
| Suppression lists | never-contact, opted-out, already-seen | add andNot | append-only — it only ever grows |
Also fits Fraud and abuse blocklists · event deduplication, the “have I already processed this?” set · follower and social-graph edges · consent and regional-eligibility registries · and distinct-count reporting, where count() is exact rather than a sketch with an error bar.
Ids must be u32 An id is a 32-bit unsigned integer: it splits into a 16-bit chunk key and a 16-bit remainder, and anything else raises a ValidationError. UUIDs and string keys need a dense-integer mapping that you own and operate — worth knowing before you install, not after.
intersect and andNot are marked because they are the operations the tiering exists for — the ones that would otherwise pull whole sets into memory. Large means the measured run behind this page: two 2,000,000-id segments, of which an intersection fetches 100 chunks of 2,000. Step through it.
The set is large, and mostly read.
Audience lists, feature cohorts, permission sets — bigger than you want to keep resident, or heading that way. Sizing a node for that growth is the part you would rather not do.
It has to survive a restart, and cost nothing while idle.
The corpus outlives any one process, your compute is stateless, and you would rather not run a node at all just so the set is there when something asks.
The data has to stay yours.
Your own storage account, a documented format, and a way out that does not involve us — and someone may have to be erased on request without rewriting immutable objects to do it.
You need a sub-millisecond p99.
Not just on hits. Our hot tier is in-process RAM, but a miss is a network hop — warm has() is 5.27 ms at p50 and 12.71 ms at p99. A sub-millisecond tail needs the whole set resident, which is a different machine.
You write to it constantly.
Writes are metered per request, so past 26.33 sustained writes a second — 8 KiB items, on-demand DynamoDB as the warm tier — a flat node is simply cheaper. Four inputs move that line, and two remove it.
For the first one that usually means Redis — Redis as your set store, which is a different thing from Redis as one of our warm drivers.
Three tiers, one API
A segment lives in all three places at once. You call has(); the store decides where the answer comes from.
A bounded LRU of decoded chunks inside your process. Nothing is pinned; nothing is required.
Per-chunk delta rows — adds and removes since the last publish. This is what makes writes cheap.
Immutable, generation-keyed objects in your own bucket. Documented format, one-command export.
A ∩ B without downloading A or B
Intersecting two 2,000,000-id segments does not download two segments. Chunk keys are compared first, and only chunks whose keys appear in both are ever fetched — 100 of 2,000, in 25.3 ms. The other 1,900 stay cold and unbilled.
3 tiers 10 drivers 1 third-party dependency
import {
CloudRoaring, MemoryWarmDriver, MemoryColdChunkSource,
} from '@cloudbitmaps/roaring';
const store = new CloudRoaring({
warm: new MemoryWarmDriver(),
cold: new MemoryColdChunkSource(),
});
const bought = store.segment('bought-last-90d');
const opened = store.segment('opened-this-week');
await bought.addMany([75, 99_999, 1_234_567_890]);
// fetches only the chunks that can contribute
for await (const id of bought.intersect([opened])) { … }The hard part is concurrency, not compression.
Roaring already solves the compression. Putting it on tiered storage creates four problems that have nothing to do with bit math: a format something other than your own code can read, writes that cost the change instead of the whole segment, deletes that actually delete, and a compaction that cannot lose a write which landed while it was running.
The problem
What CloudBitmaps does
- !A private chunk layout leaves the data only as portable as the code that wrote it, and unreadable by anything else you own
- ✓A documented .crbm format, generation-keyed and immutable, with a one-command export
- !Whole-blob GET, decode, modify, PUT makes a write cost the size of the segment rather than the size of the change
- ✓Per-chunk delta rows: $0.75 per 1M incremental writes, and nothing at all when idle
- !An OR-merge across tiers cannot express a delete, so one removal means rebuilding the segment
- ✓Tombstones are first-class — a real remove(), merged as (cold ∪ adds) \ removes
- !Two compactions running at once drop whichever writes landed mid-scan
- ✓Compaction purges conditionally on the version it archived, so post-scan writes survive
These are four of the project's 7 hard correctness invariants — each one has named tests, and they run on every commit.
Six lines, one bucket.
Install one flavor and the backend SDKs you already use. The memory drivers need nothing at all — the segments.ts sample above runs as-is, so you can see the shape before you point it at a bucket.
npm i @cloudbitmaps/roaring
# plus only the backends you actually use
npm i @aws-sdk/client-s3 @aws-sdk/client-dynamodb