CloudBitmaps

Distributed, cloud-native bitmaps · roaring shipped · v0.27.0

Big sets, the same six calls, and no node to pay for while idle.

CloudBitmaps keeps large id-sets as immutable objects in storage you already own, and reads only the chunks a query can match — 100 of 2,000. The reads are the calls you know — has, count, iterate, intersect, union, andNot — and the bill is the part that changes: 1.2 GiB at rest is $0.03 a month, against $346 for a three-node Redis cluster standing by.

npm i @cloudbitmaps/roaring @cloudbitmaps/s3 A codec and a storage · memory drivers need no setup

See the quickstart → How it works Choose a flavor

Apache-2.0 · zero-dependency core · 4 storage drivers · published tokenlessly via OIDC

Reaching for SETBIT on a big key? This is a durable home for set-shaped work, computed in batches upstream — not a drop-in replacement. What carries over, and what doesn't →

Many use cases, one shape.

Many ids, computed in batches, mostly idle, and it has to survive a restart. That’s the shape — these five are just where it turns up most often. If it describes a list you maintain, it fits, whatever you call it.

Where it fitsThe five most common
The job What’s in the set You call How it’s used
Audience segmentation user ids who bought in the last 90 days AND opened the app this week intersect rebuilt on a schedule, read on a fraction of requests
Feature-flag & experiment cohorts the ids assigned to each variant has read on nearly every request, written rarely
Permission & entitlement sets ids allowed to see a given resource has read constantly, and must survive a restart
Recommendation candidate pools ids eligible before ranking narrows them intersect narrowed per request instead of scanned
Suppression lists never-contact, opted-out, already-seen andNot reloaded whole, then subtracted inside the read

Also fits Fraud and abuse blocklists · batch deduplication, the “which of these have I already processed?” set · follower and social-graph edges · consent and regional-eligibility registries · and distinct-count reporting, where count() is exact rather than a sketch with an error bar.

Ids must be u32 An id is a 32-bit unsigned integer: it splits into a 16-bit chunk key and a 16-bit remainder, and anything else raises a ValidationError. UUIDs and string keys need a dense-integer mapping that you own and operate — worth knowing before you install, not after.

intersect and andNot are marked because they are the operations the tiering exists for — the ones that would otherwise pull whole sets into memory. Large means the measured run behind this page: two 2,000,000-id segments, of which an intersection fetches 100 chunks of 2,000. Step through it.

Use this when

The set is large, mostly read, and computed in batches.

Audience lists, feature cohorts, permission sets — bigger than you want to keep resident, or heading that way, and produced by something upstream: a warehouse query, a nightly job, a combine of two other segments. You load the result.

It has to survive a restart, and cost next to nothing while idle.

The corpus outlives any one process, your compute is stateless, and you would rather not run a node at all just so the set is there when something asks.

The data has to stay yours.

Your own storage account, a documented format, and a way out that does not involve us — and someone may have to be erased on request, which here rewrites the generation without them and deletes the one that held the bit.

Use something else when

You need a sub-millisecond p99.

Not just on hits. Our cache tier is in-process RAM, but a miss is a ranged GET against object storage, where a resident set is a memory read. A sub-millisecond tail needs the whole set resident, which is a different machine — and a ranged chunk GET from inside the region took about 29 ms at the median in the most recent run that timed it.

The set changes one id at a time.

There is no add and no remove here: a segment changes only by publishing a whole new generation. Accumulate per-id writes where they arrive — Redis is genuinely good at that — and load the set on a cadence.

For the first one that usually means Redis — Redis as your set store, which is a different thing from Redis in front of one. And past 329.15 reads a second, every one of them a cache miss, metered GETs cost more than a cluster standing by: the cache hit rate is what moves that line.

Two tiers, one pointer, one API

A segment is immutable objects in your bucket plus one row saying which of them is current. You call has(); the store decides whether that costs a read at all.

STORAGE · .crbm

Immutable, generation-keyed objects in your own bucket. Documented format, one-command export.

REGISTRY · POINTER

One small row per segment, saying which generation is current. A publish moves it forward by compare-and-swap; the one call that moves it back is an operator's rollback.

CACHE · RAM

A bounded LRU of decoded chunks inside your process. Every entry is evictable, and a new generation misses rather than serving stale bytes.

A ∩ B without downloading A or B

Intersecting two 2,000,000-id segments never requests by key, or decodes, the 1,900 chunks per segment outside the shared 100. Chunk keys are compared first, and only chunks whose keys appear in both are ever requested by key — 100 of 2,000, in 27.8 ms on the in-memory drivers. The other 1,900 are never requested by key: each object’s one opening read takes its last 256 KiB (the default), where its index of keys sits.

Chunk keys · shared only
1,900 never requested100 / 2,000 · 27.8 ms in memory
See the full walk-through →

2 tiers 4 drivers 0 dependencies in @cloudbitmaps/core

segments.tsNo setup
import { CloudRoaring, MemoryStorage }
  from '@cloudbitmaps/roaring';

const store = new CloudRoaring({ storage: new MemoryStorage() });

// each load: one write-once object, then the pointer moves to it
await store.load({ segment: 'bought-last-90d' }, [75, 99_999]);
await store.load({ segment: 'opened-this-week' }, [75, 1_234]);

const bought = store.segment('bought-last-90d');
const opened = store.segment('opened-this-week');
// fetches only the chunks that can contribute
for await (const id of bought.intersect([opened])) {
  console.log(id); // 75
}

The hard part is the protocol, not the compression.

Roaring already solves the compression. Putting it in object storage creates four problems that have nothing to do with bit math: a format something other than your own code can read, a refresh that cannot be half-visible, a read that a write landing mid-flight cannot tear, and a delete that actually deletes.

The problem

What CloudBitmaps does

!A private chunk layout leaves the data only as portable as the code that wrote it, and unreadable by anything else you own
✓A documented .crbm format, generation-keyed and immutable, with a one-command export
!A refresh that writes over the live bytes leaves a reader a set that is neither the old one nor the new one
✓Nothing overwrites a stored object: a load writes a new generation and then moves the pointer forward, so a crash before the publish leaves the previous set live
!A read assembled from more than one source is torn by a write that lands mid-flight
✓Each read resolves its generation once, before any fan-out, and every chunk is one whole chunk of one generation, CRC32C-verified before it reaches the deserializer. A long read can move to a newer generation part-way; seg.pin() holds one, and fails rather than read another once its own is collected
!Immutable objects make one person's erasure the hard case, and masking a bit is not deleting it
✓eraseSubject() rewrites the generation without that id and deletes the one that held it — for each segment its ledger reports erased, gone from the bucket when the call returns. A backup or a noncurrent version keeps its copy until its own lifecycle removes it

All four rest on a protocol written down as 7 hard correctness invariants — write-once generations behind a compare-and-swap pointer, a read that is never torn, a GC that never touches the current generation, every tier byte untrusted. Each has named tests, and they run on every commit.

A codec, a storage, one bucket.

Install the codec you want and the storage you have — each storage package brings its own cloud SDK, so nothing is an optional peer and no install carries an SDK you never use. The memory drivers need nothing at all — the segments.ts sample above runs as-is, so you can see the shape before you point it at a bucket.

Apache-2.0 · v0.27.0

InstallA codec and a storage
npm i @cloudbitmaps/roaring @cloudbitmaps/s3

# the codec you want, and the storage you have
# also: @cloudbitmaps/gcs, @cloudbitmaps/azure-blob
# to price a store: @cloudbitmaps/tools