CloudBitmaps v0.1.2 · pre-1.0Why it exists
Stop paying for RAM, 24/7, to hold sets that are mostly read
If you want big bitmap-backed audience segments or membership / feature-flag sets that are shared across services and survive restarts, today you reach for an always-on Redis cluster. It's fast — but expensive to keep running, and it forgets everything on restart unless you bolt on persistence.
CloudBitmaps keeps the bitmap engine and the developer experience, but makes the data cheap and durable at rest (cents/month in object storage), fast where it needs to be, and able to answer set queries over enormous bitmaps from small, stateless functions. A leaked bucket can reveal nothing; a GDPR erasure can be provable; a crash mid-compaction can't lose a write.
The design target wasn't hypothetical: a bitmap / membership service for audience segmentation, which has to answer — fast and cheaply — how many users are in a segment, who they are, and what's the overlap with other segments. Every trade-off here is aimed at that shape of workload. See the use cases and flows →
The crown jewel
Intersect two billion-ID segments in a serverless function
A Roaring Bitmap is already a set of 16-bit chunks. To AND two segments you only need the chunks whose key appears in both — a key missing from either side contributes nothing and is never downloaded. So two huge segments that barely overlap transfer almost nothing, and the whole join streams under a bounded in-flight window: it runs in a 128 MB Lambda regardless of segment size.
const shoppers = store.segment('high-value-shoppers'); const active = store.segment('active-this-week'); // only chunks present in BOTH ever leave cold storage for await (const id of shoppers.intersect([active])) { /* … */ } // …or materialize the result straight into a new segment await shoppers.intersectInto(store.segment('campaign-targets'), [active]);
How it works
One bitmap, three temperatures
Every 32-bit ID splits into a 16-bit chunk key (which chunk) and a 16-bit remainder (which bit inside it). CloudBitmaps maps each chunk onto whichever tier its access pattern deserves — and a read merges them.
id ∈ [0, 2³²) ──split──▶ chunkKey (16-bit) ‖ remainder (16-bit) │ ┌────────────────────────────────▼─────────────────────────────────────┐ │ HOT — bounded LRU of decoded chunks, in process RAM │ ├──────────────────────────────────────────────────────────────────────┤ │ WARM — mutable per-chunk delta rows { adds, removes } · OCC │ ├──────────────────────────────────────────────────────────────────────┤ │ COLD — immutable, generation-keyed .crbm · CRC + (optional) AES-GCM │ └──────────────────────────────────────────────────────────────────────┘ │ a read merges the tiers: ▼ effective(chunk) = ( COLD ∪ WARM.adds ) \ WARM.removes
Everything is drivers behind interfaces — in-memory, local-filesystem, and
cloud implementations that all pass the same conformance suite — and the
core/ is a pure function of its inputs (no clock, no randomness, no cloud SDK),
so a deterministic simulator can replay any concurrency or crash bug from a single seed.
See the full architecture, phase by phase →
Your data stays yours
A library over your storage — not a service that holds it hostage
Betting a billion IDs on a young library is a fair thing to hesitate over — so here's the straight answer. CloudBitmaps is a library, not a service: your data lives in your S3 bucket, your DynamoDB table, your filesystem — we never see or hold it. If the project vanished tomorrow, nothing is deleted or held hostage; the objects are still sitting in your bucket.
An open, standard format
Cold .crbm objects are a documented container around standard portable Roaring serialization — the exact bytes every roaring library (Java, Go, Python, Rust…) already reads. Getting your data out is reading the format.
Immutable + checksummed
Write-once, generation-numbered objects; a pointer swap makes one "current." A bug can only write a new generation — the old one stays intact and you roll back. Every byte is CRC/SHA-verified before use.
A one-command exit
store.exportSegments() (or the export-segments CLI) dumps every segment to a portable file — roaring (loadable by any roaring library) or ndjson (zero-dependency). Leaving is a command, not a research project.
The honest trade-off: the warm/registry tiers use CloudBitmaps's own (documented) schema, and it's a young, single-maintainer, pre-1.0 project — so the real risks are operational and maturity, not losing the bytes (which are immutable and yours). Pure roaring libraries have zero lock-in only because they manage no storage; bitmap databases manage it but keep your data in their engine. CloudBitmaps is the middle — your storage, your format, our orchestration.
What you get
A distributed system, not just a data structure
Tiered by temperature
Hot RAM LRU → warm NoSQL deltas → cold immutable object storage. Cheap at rest, fast where it counts.
Chunk-skipping intersection
AND across segments while downloading only the chunks that can possibly overlap. Streams in bounded memory.
Tombstone-correct deletes
A real remove() over an immutable cold base — one chunk write, no scan, no rebuild.
Crash-safe compaction
A background 2-phase commit folds warm deltas into fresh cold generations — version-fenced, flat on the cold side, recovers from a crash at any step. (The warm delta set is still buffered; the un-buffered external-merge path is planned.)
Encryption + crypto-shred
Opt-in AES-256-GCM (IDs and cardinality hidden), bring-your-own-key with no forced cloud dependency, and provable erasure — even on backups.
Resilience, on by default
Every warm/cold call retries transient faults with jittered backoff. An outage costs latency, not correctness.
Quickstart
Zero setup: in-memory in six lines
A store is wired to a warm driver (live mutations) and a cold source (the immutable archive). The in-memory drivers need no setup — swap in the S3 / DynamoDB drivers later and the code above them doesn't change.
import { CloudRoaring, MemoryWarmDriver, MemoryColdChunkSource } from '@cloudbitmaps/roaring'; const store = new CloudRoaring({ warm: new MemoryWarmDriver(), cold: new MemoryColdChunkSource(), }); const vips = store.segment('high-value-shoppers'); await vips.addMany([5, 99_999, 1_234_567_890]); // grouped by chunk → one write per chunk await vips.has(1_234_567_890); // → true await vips.remove(1_234_567_890); // tombstone — no rebuild await vips.count(); // exact cardinality
From here: persist on the local filesystem, move cold
to S3 (@cloudbitmaps/roaring/s3) and warm to DynamoDB
(@cloudbitmaps/roaring/dynamodb) — the cloud SDKs are optional peer dependencies, so the
core package never pulls them in.
Where it is
Status & roadmap
0.1.0, and deliberately pre-1.0. The engine, nine cloud backends across three tiers,
the compaction daemon, encryption, and the compliance surface are built and tested;
1.0 is earned by real-cloud cost calibration, adoption, and freezing the
.crbm on-disk format.
Tiered core ✓
Core engine, .crbm format + deterministic simulator, chunk-skipping intersection, warm OCC, resilience, registry, crash-safe compaction (flat on the cold side), encryption & crypto-shred.
Run it for real ✓
Observability metrics sink, honest cost estimator + published benchmarks, audit sink, and the compliance & governance layer — PRIVACY.md, subject access & erasure, portable export.
Nine cloud backends ✓
Cold on S3, GCS, and Azure Blob; warm on DynamoDB, Postgres, MySQL, Redis, Mongo, and Cassandra; a registry on DynamoDB or S3 alone — each green against the shared conformance suite on a real backend.
Toward 1.0
Real-AWS cost & latency calibration, the .crbm format freeze, native registries beyond AWS, and the 64-bit / billions-of-IDs axis. Language ports follow the freeze.
The roadmap has the full picture — including the validated envelope: what we've proven, at what scale, and what we haven't. The architecture deep-dive walks the design with diagrams and the invariant each layer secures.