CloudBitmaps

Usage · @cloudbitmaps/roaring v0.10.0

Zero setup, then the same code in production.

The in-memory drivers need nothing — no bucket, no credentials. Swap them for S3 later and every line that loads or reads stays as it was.

The eight lines → The whole API Four drivers Retention

Apache-2.0 · zero-dependency core · one package per storage service, each with its own SDK

Eight lines, then the same eight lines

The only difference between these two is which backend the first line constructs. Nothing below them knows, or can know, whether the bytes are in a Map or in a bucket.

quickstart.tsNo setup
import { CloudRoaring, MemoryStorage }
  from '@cloudbitmaps/roaring';

const store = new CloudRoaring({ storage: new MemoryStorage() });

// the whole set, as one immutable generation
await store.load({ segment: 'high-value-shoppers' },
  [75, 99_999, 1_234_567_890]);
const vips = store.segment('high-value-shoppers');
await vips.has(1_234_567_890);  // true
await vips.count();            // exact, 0 payloads
production.tsOnly the drivers
import { CloudRoaring } from '@cloudbitmaps/roaring';
import { S3Storage } from '@cloudbitmaps/s3';

// one bucket holds both the data and the pointer, stated once
const backend = new S3Storage({
  bucket: 'segments', region: 'us-east-1',
});

const store = new CloudRoaring({
  storage: backend,
  cache: { maxChunks: 512 },   // optional bounded LRU
});

// …every line below here is unchanged

The registry is what resolves a generation in one read instead of a list-scan, what makes an encrypted segment readable, and what every write and every lifecycle call publishes through. You do not wire it separately. A backend — S3Storage, GcsStorage and the rest — carries it, which is why storage is the only location key the store takes. Hand the store a bare IStorageDriver instead and there is no pointer at all: it finds the highest generation in the bucket by list-scan, cleartext and read-only. Fine for a first look, not past it.

The whole API

The read verbs are deliberately the shape of an in-memory Set. The write is not: a segment changes only by getting a whole new generation. So the third column — what a call costs you in backend requests — is the one worth reading.

Load, segment and storeScrolls sideways on narrow screens
Call Does Touches
store.load(ref, ids) The only way data gets in. Streams an unsorted id source into one immutable .crbm object, then advances the segment's pointer forward-only. Grouped by chunk first, so it bills per object, not per id. On S3, a segment's first load is expected at 4 PUT-class and 7 GET requests, listings included — S3 bills a listing at the PUT rate. 4 PUT-class + 7 GET
has(id) Resolves one chunk of the current generation: the cache if it is there, otherwise one ranged GET of that chunk. The first read of a segment also reads its pointer and its index. Never a merge of two sources, so it can never be torn. cache, else 1 GET
count() Sums per-chunk cardinality from the .crbm index. Exact, not estimated — no HyperLogLog — and it never opens a payload. A segment's first read also reads its pointer. the index
intersect(others, { exclude }) Async-iterates the intersection. Compares chunk keys first and requests only shared chunks; the storage payload footprint is bounded by concurrency × operands × chunk, and that is the whole term — there is no second tier to snapshot. exclude subtracts suppression segments in the same pass. the shared chunks
intersectInto(dest, others)
unionInto · andNotInto
Publishes the result as a new generation of dest rather than streaming it to you — superseded, not added to — and hands back { generation, cardinality, chunkCount, size }. storage + registry
store.eraseSubject(id) Rewrites every segment holding the id without it and deletes the generation that held the bit, so it is physically gone from the bucket when the call returns. A compliance operation, not a hot-path verb — there is no per-id write. storage + registry
store.retireExpired() The retention sweep: everything whose expiresAt has passed, each through dropSegment. A call, not a daemon — bounded, previewable, and it returns a per-segment ledger instead of throwing. registry, then storage
store.exportSegments(sink) Writes every registered segment out as cross-language roaring or newline-delimited ndjson, using only public read APIs. The one-command exit, and there is a CLI that wraps it. storage
estimateCost(input) Your region and rates in, projected bill out. The benchmarks page is generated from this same function. nothing — pure

No per-id TTL, by design A segment can expire — record setRetention(ref, { expiresAt }) and a retireExpired() sweep retires it, on a schedule you own. An individual id cannot, because a bitmap stores ids and not (id, timestamp) pairs — an expiry per id costs 4–8 bytes each and takes the compression with it. Retention is a segment that expires, not a bit that ages.

Both writes are the same protocol: build one immutable object, then move the pointer.

load.tsPer object, not per id
// `store` from above — the backend is already wired.

const ref = { namespace: 'audience', segment: 'high-value' };

// The warehouse decides the set; hand over the cursor,
// not an array. Re-run on a cadence — each run
// publishes the next generation and supersedes the last.
const r = await store.load(ref, athenaCursor());

r.cardinality;  // distinct ids now published
audience.tsThe bytes never come to you
const seg = (name) =>
  store.segment(name, { namespace: 'audience' });

const vips = seg('high-value');
const eu = seg('eu-residents');
const optedOut = seg('opted-out');
const target = seg('eu-vips');

// Suppression folds into the same pass, and the result
// becomes a new generation of `target` — unless it comes
// out empty, which is refused rather than wiping it.
const res = await vips.intersectInto(
  target, [eu], { exclude: [optedOut] },
);
if (!res.published) console.warn(res.reason); // 'empty'

Retention: a segment that expires

Put the date in the name and the window becomes set algebra. A user is re-eligible the moment the wave moves to tomorrow’s bucket, because tomorrow’s bucket is a different, empty set. The old bucket retires itself once you record when it should — see setRetention below.

dedup.tsNo per-user timers
// The bucket IS the re-eligibility rule.
const ref = { namespace: 'sent-daily', segment: localDay };
const sent = store.segment(localDay, { namespace: 'sent-daily' });

if (await sent.has(userId)) return;  // already sent today
await send(userId);                  // then queue the id

// On a cadence — every few minutes, or once at the end
// of the wave — publish the day so far as one object.
await store.load(ref, sentToday());
retire.tsOne call, ordered correctly
// The namespace is part of the identity — omit it
// and you silently address a different segment.
const ref = { namespace: 'sent-daily', segment: oldDay };

// Preview first — dryRun changes nothing.
await store.dropSegment(ref, {
  confirmSegment: ref.segment, dryRun: true,
});

// Tombstone the pointer, then sweep every generation.
const r = await store.dropSegment(ref, {
  confirmSegment: ref.segment,
});
// r.reason === 'absent' means you named it wrong.

The check is exactly as fresh as the cadence. A read sees the published generation, so an id queued since the last load is not in sent yet — this is at-most-once per published set, not against the last millisecond. Where the window is a day and the cadence is minutes, that is the right trade; where the answer has to be true the instant an id is claimed, claim it where the ids are queued and keep the segment as the durable record.

Names — any non-empty string, segment and namespace alike. sent:2026-08-01, orders/2026 and user@example.com are all legal; each storage layer escapes what it cannot take literally. The only limits are non-empty and 256 characters once encoded for a storage key. The namespace split is still the better shape for a family: listing the namespace enumerates exactly that family’s buckets, which is the list a retention sweep wants.

Do not reach for an object-store lifecycle rule on its own. It deletes the bytes while the registry still points at them — the torn state a DR runbook says not to serve traffic on — and it surfaces intermittently, because a read checks the in-process cache before storage. It passes in a process whose cache is already populated and starts failing after a restart. dropSegment exists so the ordering cannot be got wrong: the pointer is tombstoned first, then the objects are swept — and it hands back generationsRemaining, which is non-empty exactly when bytes survived and the drop should be re-run. A lifecycle rule is a fine backstop for orphans — just give it a longer window than your retention. The guide has the rest.

The API reference is checked against the real exports by CI, so it cannot drift from the package.

Four drivers, each carrying both halves

A backend carries the immutable generation objects and the pointer row, from one bucket and one prefix — so a deployment names one service, not two. Anything with a cheap ranged read and a write-once put can serve as storage; anything with a cheap compare-and-swap can serve as the pointer; each of the four below does both. Need them apart? createBackend({ storage, registry }) still composes a storage driver and a registry driver by hand. Each backend's SDK is a real dependency of its own package — install only the storage you use.

Redis is not in this table. It is the line the benchmarks page prices against, not a backend this library writes to — the storage here is your own bucket plus one small pointer row per segment. It is still a good neighbour: when your set is defined by events arriving one at a time, accumulate them where they arrive — Redis is genuinely good at that — and load the set on a cadence.

DriversStorage · registry
Backend Package Roles Cloud SDK (a real dependency)
S3 @cloudbitmaps/s3 storage · registry @aws-sdk/client-s3 >=3.645.0 <4
Google Cloud Storage @cloudbitmaps/gcs storage · registry @google-cloud/storage ^7 || ^8
Azure Blob @cloudbitmaps/azure-blob storage · registry @azure/storage-blob ^12
Filesystem main entry storage · registry none

Memory drivers ship in the main entry too and need nothing — they are what the quickstart uses. An S3-compatible endpoint (R2, MinIO) works through the S3 driver by pointing the client's endpoint at it.

Estimate your own bill

Our published numbers are us-east-1 on-demand. Yours are not. Feed your own rates in and get your own crossover — the site's modelled figures call this same function, so they cannot drift ahead of the library. It does not count the pointer yet — the object beside your data that says which generation is current — so until it does, pass requestsPerLoad: 4.56 for a segment's first store.load(), 4.72 from its third, or 2.24 for a bare write and publish, and chunksPerIntersect at every GET your intersect makes, pointer and tail reads included.

estimate.tsPure — touches nothing
import { estimateCost, AWS_US_EAST_1_ONDEMAND }
  from '@cloudbitmaps/roaring';

const bill = estimateCost({
  segments: [{ sizeBytes: 60_000_000, count: 20 }],
  workload: { readsPerSec: 800, loadsPerMonth: 900 },
  pricing: myPricingProfile,  // or the AWS default
});
bill.redisCrossover.readsPerSec;
  • 01 Schedule the retention sweep retireExpired() The only scheduled work there is — no daemon, no compaction pass, no lifecycle worker. An EventBridge rule, a Kubernetes CronJob or a queue job all do; nothing here starts a timer.
  • 02 Bound the cache to what your runtime can hold cache.maxChunks An LRU with a hard ceiling — 1,024 decoded chunks by default — and the ceiling is yours to choose. Correctness does not depend on it either way.
  • 03 Collect superseded generations, keeping one $0.03/mo · 1.2 GiB Every load leaves its predecessor in the bucket, still billed, and nothing deletes it for you. gcOrphanGenerations does — keeping the newest superseded one by default as a grace window, so a read still fetching from it saves a round trip. Storage is the only cost of that window. It is not a snapshot facility: no setting here holds a long job on one generation.

Your data is not held hostage

The reason to be comfortable adopting a pre-1.0 library is not a promise about the library. It is that leaving costs you one command, and the bytes were never anywhere you could not reach.

The benchmarks page carries the rate past which a flat tier is cheaper, the methodology, and what each measurement does and does not establish.

Start with the memory drivers.

They need no account and no configuration, and the code you write against them is the code you keep. Point it at a bucket when you are ready.

Apache-2.0 · v0.10.0 · zero-dependency core · 4 storage drivers

InstallA codec and a storage
npm i @cloudbitmaps/roaring @cloudbitmaps/s3

# the codec you want, and the storage you have.
# also: @cloudbitmaps/gcs, @cloudbitmaps/azure-blob