CloudBitmaps — usage, flows & use cases

The mental model

Three nouns: a store, a segment, an ID

A store (new CloudRoaring({ warm, cold })) is wired to a warm driver (live mutations) and a cold driver (the immutable archive — a raw driver the store wraps, or a pre-built cold source). Retries and the hot cache are on by default.
A segment (store.segment('name')) is one named bitmap — an audience, a membership set, a feature cohort.
An ID is a dense integer in [0, 2³²). It splits into a 16-bit chunk key + a 16-bit remainder; a non-integer / out-of-range ID throws ValidationError.

Every read merges the tiers, so a delete is honored the instant you make it:

   store.segment('high-value-shoppers').has(id)
                        │
                        ▼   merge the tiers for that chunk
        effective = ( COLD ∪ WARM.adds ) \ WARM.removes

Wire it up

Start in memory, graduate to the cloud — same code above the drivers

The engine only ever sees the driver interfaces, so you swap storage backends without changing a line of application code. Start with zero setup, then move each tier independently. Only cold and warm are required — a registry unlocks eject/compaction/encryption, and everything else is optional tuning.

1 · in memory — zero setup (tests, a first look)

import { CloudRoaring, MemoryWarmDriver, MemoryColdChunkSource } from '@cloudbitmaps/roaring';

const store = new CloudRoaring({
  warm: new MemoryWarmDriver(),
  cold: new MemoryColdChunkSource(),
});

2 · local filesystem — durable, survives a restart

import { CloudRoaring, LocalFsWarmDriver, LocalFsColdDriver } from '@cloudbitmaps/roaring';

const store = new CloudRoaring({
  warm: new LocalFsWarmDriver('./.cloudbitmaps/warm'),
  cold: new LocalFsColdDriver('./.cloudbitmaps/cold'), // raw driver — wrapped for you
});

3 · cloud — S3 cold + DynamoDB warm + registry (the SDKs are optional peer deps)

import { CloudRoaring } from '@cloudbitmaps/roaring';
import { S3ColdDriver } from '@cloudbitmaps/roaring/s3';
import { DynamoDbWarmDriver, DynamoDbRegistryDriver } from '@cloudbitmaps/roaring/dynamodb';

const cold = new S3ColdDriver({ client: s3, bucket: 'my-bitmaps' });
const registry = new DynamoDbRegistryDriver({ client: dynamo, tableName: 'cloudbitmaps' });

const store = new CloudRoaring({
  warm: new DynamoDbWarmDriver({ client: dynamo, tableName: 'cloudbitmaps' }),
  cold, // raw S3 driver
  registry, // resolves currentGen with no list-scan
});

The operations

Everything a segment can do

Method	Returns	Notes
`add(id)` / `remove(id)`	`Promise<void>`	single ID; `ValidationError` if `id ∉ [0, 2³²)`
`addMany(ids)` / `removeMany(ids)`	`Promise<void>`	grouped by chunk (one write per chunk); not atomic across chunks
`has(id)`	`Promise<boolean>`	membership test
`count()`	`Promise<number>`	exact cardinality
`iterate()`	`AsyncIterable<number>`	ascending IDs
`intersect(others)`	`AsyncIterable<number>`	ascending; chunk-skipping; `concurrency` option
`intersectInto(dest, others)`	`Promise<void>`	materialize the result into `dest`; not atomic

Failures are typed errors (ValidationError, WriteConflictError, IntegrityError, KeyUnavailableError, …), never thrown strings — so callers can branch on why something failed. addMany groups IDs by chunk first, so 10,000 IDs spanning 12 chunks become 12 writes, not 10,000.

The real flows

The workflows you'll actually run

Membership changes two ways: a bulk re-derivation (rebuild the set from a query) and an incremental delta (flip one bit on an event). CloudBitmaps has a path for each — plus the background compactor and optional encryption.

1 · Batch seed / sweep — build a generation from a query

The batch path: fold a (possibly huge, unsorted) ID stream into a fresh immutable cold generation with bulkLoadCrbmGeneration. Input is consumed lazily and deduped — pipe a DB cursor, a file, or a query result straight through it, and publish it to the registry in the same call.

rebuild a segment from the source of truth (the "sweep")

import { bulkLoadCrbmGeneration } from '@cloudbitmaps/roaring';

// ids: any sync/async iterable — an Athena cursor, a file stream, an array…
const ids = queryEntityStore(segmentSql);        // dense integer _ids that match the rule
await bulkLoadCrbmGeneration(
  cold, { segment: 'active-us', generation: 2 }, ids, { registry },
); // → new gen published; readers pick it up, warm deltas merge on top

2 · Real-time matcher — flip one bit per event

The live path: on each change event, evaluate the rule for that one user and set or clear a single bit. Each is a small, conflict-safe warm write.

per-event membership update (the "matcher")

const seg = store.segment('opened-push-7d');

// on an attribute-change event for one user:
if (matchesRule(user)) await seg.add(user.id);
else                    await seg.remove(user.id); // tombstone — no rebuild

3 · Targeting — combine segments with chunk-skipping intersection

The crown jewel. Stream the IDs present in every operand — downloading only the cold chunks whose key appears in all of them, under a bounded memory window. Runs over enormous segments in a small / serverless process.

who is in the audience AND eligible

const audience = store.segment('high-value-shoppers');
const eligible = store.segment('notifications-enabled');

for await (const id of audience.intersect([eligible])) { /* send to id */ }

// or materialize the result into a reusable segment
await audience.intersectInto(store.segment('campaign-targets'), [eligible]);

Intersection is the shipped set operation (commutative, streaming). Whole-segment union and difference operators are on the roadmap; today you compose with intersect and apply suppression at the ID level via removeMany / intersectInto.

4 · Keep the warm tier small — the compaction daemon

Every write lands as a small warm delta; left alone, reads slow down. The compaction daemon folds warm deltas into a fresh cold generation and clears the warm rows — a separate background process (it never slows your app path), crash-safe, and constant-memory. Run it as a Lambda/cron (once) or a long-running service (loop).

one compaction cycle (bundled CLI shown below)

import { runCompactionCycle } from '@cloudbitmaps/roaring';

const deps = { cold, warm, registry, clock: { now: () => Date.now() } };
await runCompactionCycle(deps, { owner: 'worker-1', keep: 1 });

// or the bundled CLI over the local filesystem:
//   CR_COMPACT_ROOT=./.cloudroaring CR_COMPACT_MODE=loop npx compact-segments

5 · Encryption at rest + crypto-shred (opt-in)

Pass a keystore and the cold objects are AES-256-GCM-encrypted — a leaked bucket reveals neither IDs nor cardinality. You bring your own 32-byte key (no forced cloud dependency); deleting the wrapped key crypto-shreds the segment (unreadable everywhere, including backups). Full detail — key management, rotation, recovery keys — is in the getting-started guide, §9.

encrypt a segment; erase it irreversibly

import { CloudRoaring, InProcessKeystore, destroySegment } from '@cloudbitmaps/roaring';

const keystore = new InProcessKeystore({
  keys: { '2026-06': loadKekFromSecrets() }, activeKeyId: '2026-06',
});
// Pass the raw cold driver + registry + keystore straight to the store — encrypted at rest:
const store = new CloudRoaring({ warm, cold: s3Cold, registry, keystore });

// GDPR erasure — you must name the segment as confirmation:
await destroySegment({ segment: 'pii' }, { registry, warm }, { confirmSegment: 'pii' });

6 · Observe it — the metrics sink

Optionally hand the store a metrics sink and it emits typed events — cold GETs and bytes, warm reads/writes, cache hit rate, retries, intersection efficiency, op latency. Off by default (a no-op); the events are vendor-neutral, so a ~12-line adapter maps them to OpenTelemetry / Datadog / a log — no telemetry dependency in the library.

tally events, or forward them to your telemetry

import { CountingMetricsSink } from '@cloudbitmaps/roaring';

const metrics = new CountingMetricsSink();
const store = new CloudRoaring({ warm, cold, metrics });
await store.segment('users').has(42);
console.log(metrics.snapshot()); // { cold, warm, cache, retries, intersect, ops }

// or forward each event to your stack (OpenTelemetry shown — ~12 lines, no lib dependency):
new CloudRoaring({ warm, cold, metrics: { onEvent(e) {
  if (e.kind === 'cold.get') coldBytes.add(e.bytes);
  if (e.kind === 'cache' && e.hit) cacheHits.add(1);
} } });

7 · Take your data with you — export / eject

No lock-in, by construction. store.exportSegments(sink, options) dumps every segment's current effective set (tier-merged, warm deltas folded in) through an injected sink, using only public read APIs — so the dump is readable without CloudBitmaps. Pick roaring (a portable RoaringBitmap32 per segment, loadable by any roaring library in Java/Go/Python/Rust/…) or ndjson (newline-delimited ids, zero dependencies to read). A segment it can't read is recorded in the manifest's failed[] and the run continues (one bad segment never blocks the rest); all-warm segments not yet in the registry can be named via candidates / CR_EXPORT_SEGMENTS. Full detail — custom sinks, consistency, encrypted segments — is in the getting-started guide, §14.

dump every segment to a portable file (bundled CLI shown below)

import type { ExportSink } from '@cloudbitmaps/roaring';

// your sink: open(ref, ext) => { write, close, abort? } — an fs writer, an S3 upload, a buffer…
const manifest = await store.exportSegments(mySink, { format: 'roaring' /* or 'ndjson' */ });
// manifest: { version, format, totalSegments, totalIds, segments: [...] }

// or the bundled CLI over the local filesystem (writes manifest.json last = complete export):
//   CR_EXPORT_ROOT=./.cloudroaring CR_EXPORT_OUT=./dump npx export-segments

Where it fits

Use cases

Anywhere you keep large integer-keyed sets that are shared across services, mostly read, and must survive restarts — and where an always-on cache is overkill. The workload it was designed against is an audience-segmentation / engagement platform: the service that answers, for any segment, how many users are in it, who they are, and what's the overlap with other segments.

SEGMENTS

Audience segmentation

One segment per audience. count() drives the campaign UI; iterate() pages members to send; intersect() does targeting (audience ∩ eligible).

FLAGS

Feature flags & entitlements

"Who has feature X / plan Y" as a durable membership set, shared across services and read from stateless functions — no always-on cache to babysit.

MEMBERS

Any big membership set

Fraud lists, cohorts, dedup/seen-sets, access sets — billions of IDs, cheap at rest, exact counts and set operations on demand.

The two write modes → the two topologies

Most segments are refreshed in batch ("good-enough fresh"); a high-value subset also needs real-time updates. That maps one-to-one onto CloudBitmaps's two write paths — which is why the cold/read path (Topology-A) shipped first and the live-write path (Topology-B) followed.

Mode	How membership changes	CloudBitmaps path
Sweep (batch)	re-derive the full set from a query, rebuild the bitmap	bulk-load → new cold generation (Topology-A)
Matcher (real-time)	evaluate one user on an event, flip one bit	add / remove → warm + compaction (Topology-B)

End to end: a campaign send

  save segment ─▶ compile rule → SQL
       │
   SEED (batch · Topology-A):   bulkLoadCrbmGeneration ─▶ COLD .crbm (gen N) on S3
       │
   LIVE (real-time · Topology-B): per-event add()/remove() ─▶ WARM ─▶ daemon ─▶ gen N+1
       │
   launch ─▶ count()                  ─▶ audience size in the UI
          ─▶ intersect(audience, eligible) ─▶ who to send to
          ─▶ iterate() paged members     ─▶ push / email delivery
          ─▶ opens / clicks feed back      ─▶ new attributes ─▶ next sweep

No lock-in. Apache-2.0, an open, documented .crbm format, and portable CRoaring serialization — your membership data can always be read or migrated out. One command dumps every segment to a portable file (export-segments / §7 above), and coupling it behind your own interface makes swapping the implementation one adapter, not a rewrite. See how it's built →

Use it like an in-memory Set — that happens to live in the cloud.