Usage · @cloudbitmaps/roaring v0.10.0
Zero setup, then the same code in production.
The in-memory drivers need nothing — no bucket, no credentials. Swap them for S3 later and every line that loads or reads stays as it was.
Eight lines, then the same eight lines
The only difference between these two is which backend the first line constructs. Nothing below them knows, or can know, whether the bytes are in a Map or in a bucket.
import { CloudRoaring, MemoryStorage }
from '@cloudbitmaps/roaring';
const store = new CloudRoaring({ storage: new MemoryStorage() });
// the whole set, as one immutable generation
await store.load({ segment: 'high-value-shoppers' },
[75, 99_999, 1_234_567_890]);
const vips = store.segment('high-value-shoppers');
await vips.has(1_234_567_890); // true
await vips.count(); // exact, 0 payloads
import { CloudRoaring } from '@cloudbitmaps/roaring';
import { S3Storage } from '@cloudbitmaps/s3';
// one bucket holds both the data and the pointer, stated once
const backend = new S3Storage({
bucket: 'segments', region: 'us-east-1',
});
const store = new CloudRoaring({
storage: backend,
cache: { maxChunks: 512 }, // optional bounded LRU
});
// …every line below here is unchanged
The registry is what resolves a generation in one read instead of a list-scan, what makes an encrypted segment readable, and what every write and every lifecycle call publishes through. You do not wire it separately. A backend — S3Storage, GcsStorage and the rest — carries it, which is why storage is the only location key the store takes. Hand the store a bare IStorageDriver instead and there is no pointer at all: it finds the highest generation in the bucket by list-scan, cleartext and read-only. Fine for a first look, not past it.
The whole API
The read verbs are deliberately the shape of an in-memory Set. The write is not: a segment changes only by getting a whole new generation. So the third column — what a call costs you in backend requests — is the one worth reading.
| Call | Does | Touches |
|---|---|---|
| store.load(ref, ids) | The only way data gets in. Streams an unsorted id source into one immutable .crbm object, then advances the segment's pointer forward-only. Grouped by chunk first, so it bills per object, not per id. On S3, a segment's first load is expected at 4 PUT-class and 7 GET requests, listings included — S3 bills a listing at the PUT rate. | 4 PUT-class + 7 GET |
| has(id) | Resolves one chunk of the current generation: the cache if it is there, otherwise one ranged GET of that chunk. The first read of a segment also reads its pointer and its index. Never a merge of two sources, so it can never be torn. | cache, else 1 GET |
| count() | Sums per-chunk cardinality from the .crbm index. Exact, not estimated — no HyperLogLog — and it never opens a payload. A segment's first read also reads its pointer. | the index |
| intersect(others, { exclude }) | Async-iterates the intersection. Compares chunk keys first and requests only shared chunks; the storage payload footprint is bounded by concurrency × operands × chunk, and that is the whole term — there is no second tier to snapshot. exclude subtracts suppression segments in the same pass. | the shared chunks |
|
intersectInto(dest, others) unionInto · andNotInto |
Publishes the result as a new generation of dest rather than streaming it to you — superseded, not added to — and hands back { generation, cardinality, chunkCount, size }. | storage + registry |
| store.eraseSubject(id) | Rewrites every segment holding the id without it and deletes the generation that held the bit, so it is physically gone from the bucket when the call returns. A compliance operation, not a hot-path verb — there is no per-id write. | storage + registry |
| store.retireExpired() | The retention sweep: everything whose expiresAt has passed, each through dropSegment. A call, not a daemon — bounded, previewable, and it returns a per-segment ledger instead of throwing. | registry, then storage |
| store.exportSegments(sink) | Writes every registered segment out as cross-language roaring or newline-delimited ndjson, using only public read APIs. The one-command exit, and there is a CLI that wraps it. | storage |
| estimateCost(input) | Your region and rates in, projected bill out. The benchmarks page is generated from this same function. | nothing — pure |
No per-id TTL, by design A segment can expire — record setRetention(ref, { expiresAt }) and a retireExpired() sweep retires it, on a schedule you own. An individual id cannot, because a bitmap stores ids and not (id, timestamp) pairs — an expiry per id costs 4–8 bytes each and takes the compression with it. Retention is a segment that expires, not a bit that ages.
Both writes are the same protocol: build one immutable object, then move the pointer.
// `store` from above — the backend is already wired.
const ref = { namespace: 'audience', segment: 'high-value' };
// The warehouse decides the set; hand over the cursor,
// not an array. Re-run on a cadence — each run
// publishes the next generation and supersedes the last.
const r = await store.load(ref, athenaCursor());
r.cardinality; // distinct ids now published
const seg = (name) =>
store.segment(name, { namespace: 'audience' });
const vips = seg('high-value');
const eu = seg('eu-residents');
const optedOut = seg('opted-out');
const target = seg('eu-vips');
// Suppression folds into the same pass, and the result
// becomes a new generation of `target` — unless it comes
// out empty, which is refused rather than wiping it.
const res = await vips.intersectInto(
target, [eu], { exclude: [optedOut] },
);
if (!res.published) console.warn(res.reason); // 'empty'
Retention: a segment that expires
Put the date in the name and the window becomes set algebra. A user is re-eligible the moment the wave moves to tomorrow’s bucket, because tomorrow’s bucket is a different, empty set. The old bucket retires itself once you record when it should — see setRetention below.
// The bucket IS the re-eligibility rule.
const ref = { namespace: 'sent-daily', segment: localDay };
const sent = store.segment(localDay, { namespace: 'sent-daily' });
if (await sent.has(userId)) return; // already sent today
await send(userId); // then queue the id
// On a cadence — every few minutes, or once at the end
// of the wave — publish the day so far as one object.
await store.load(ref, sentToday());
// The namespace is part of the identity — omit it
// and you silently address a different segment.
const ref = { namespace: 'sent-daily', segment: oldDay };
// Preview first — dryRun changes nothing.
await store.dropSegment(ref, {
confirmSegment: ref.segment, dryRun: true,
});
// Tombstone the pointer, then sweep every generation.
const r = await store.dropSegment(ref, {
confirmSegment: ref.segment,
});
// r.reason === 'absent' means you named it wrong.
The check is exactly as fresh as the cadence. A read sees the published generation, so an id queued since the last load is not in sent yet — this is at-most-once per published set, not against the last millisecond. Where the window is a day and the cadence is minutes, that is the right trade; where the answer has to be true the instant an id is claimed, claim it where the ids are queued and keep the segment as the durable record.
Names — any non-empty string, segment and namespace alike. sent:2026-08-01, orders/2026 and user@example.com are all legal; each storage layer escapes what it cannot take literally. The only limits are non-empty and 256 characters once encoded for a storage key. The namespace split is still the better shape for a family: listing the namespace enumerates exactly that family’s buckets, which is the list a retention sweep wants.
Do not reach for an object-store lifecycle rule on its own. It deletes the bytes while the registry still points at them — the torn state a DR runbook says not to serve traffic on — and it surfaces intermittently, because a read checks the in-process cache before storage. It passes in a process whose cache is already populated and starts failing after a restart. dropSegment exists so the ordering cannot be got wrong: the pointer is tombstoned first, then the objects are swept — and it hands back generationsRemaining, which is non-empty exactly when bytes survived and the drop should be re-run. A lifecycle rule is a fine backstop for orphans — just give it a longer window than your retention. The guide has the rest.
The API reference is checked against the real exports by CI, so it cannot drift from the package.
Four drivers, each carrying both halves
A backend carries the immutable generation objects and the pointer row, from one bucket and one prefix — so a deployment names one service, not two. Anything with a cheap ranged read and a write-once put can serve as storage; anything with a cheap compare-and-swap can serve as the pointer; each of the four below does both. Need them apart? createBackend({ storage, registry }) still composes a storage driver and a registry driver by hand. Each backend's SDK is a real dependency of its own package — install only the storage you use.
Redis is not in this table. It is the line the benchmarks page prices against, not a backend this library writes to — the storage here is your own bucket plus one small pointer row per segment. It is still a good neighbour: when your set is defined by events arriving one at a time, accumulate them where they arrive — Redis is genuinely good at that — and load the set on a cadence.
| Backend | Package | Roles | Cloud SDK (a real dependency) |
|---|---|---|---|
| S3 | @cloudbitmaps/s3 | storage · registry | @aws-sdk/client-s3 >=3.645.0 <4 |
| Google Cloud Storage | @cloudbitmaps/gcs | storage · registry | @google-cloud/storage ^7 || ^8 |
| Azure Blob | @cloudbitmaps/azure-blob | storage · registry | @azure/storage-blob ^12 |
| Filesystem | main entry | storage · registry | none |
Memory drivers ship in the main entry too and need nothing — they are what the quickstart uses. An S3-compatible endpoint (R2, MinIO) works through the S3 driver by pointing the client's endpoint at it.
Estimate your own bill
Our published numbers are us-east-1 on-demand. Yours are not. Feed your own rates in and get your own crossover — the site's modelled figures call this same function, so they cannot drift ahead of the library. It does not count the pointer yet — the object beside your data that says which generation is current — so until it does, pass requestsPerLoad: 4.56 for a segment's first store.load(), 4.72 from its third, or 2.24 for a bare write and publish, and chunksPerIntersect at every GET your intersect makes, pointer and tail reads included.
import { estimateCost, AWS_US_EAST_1_ONDEMAND }
from '@cloudbitmaps/roaring';
const bill = estimateCost({
segments: [{ sizeBytes: 60_000_000, count: 20 }],
workload: { readsPerSec: 800, loadsPerMonth: 900 },
pricing: myPricingProfile, // or the AWS default
});
bill.redisCrossover.readsPerSec;
- 01 Schedule the retention sweep retireExpired() The only scheduled work there is — no daemon, no compaction pass, no lifecycle worker. An EventBridge rule, a Kubernetes CronJob or a queue job all do; nothing here starts a timer.
- 02 Bound the cache to what your runtime can hold cache.maxChunks An LRU with a hard ceiling — 1,024 decoded chunks by default — and the ceiling is yours to choose. Correctness does not depend on it either way.
- 03 Collect superseded generations, keeping one $0.03/mo · 1.2 GiB Every load leaves its predecessor in the bucket, still billed, and nothing deletes it for you. gcOrphanGenerations does — keeping the newest superseded one by default as a grace window, so a read still fetching from it saves a round trip. Storage is the only cost of that window. It is not a snapshot facility: no setting here holds a long job on one generation.
Your data is not held hostage
The reason to be comfortable adopting a pre-1.0 library is not a promise about the library. It is that leaving costs you one command, and the bytes were never anywhere you could not reach.
- ✓ The bytes are in your bucket, immutable your account Storage objects are generation-keyed and never mutated in place. Nothing is stored anywhere we control, and there is no service to be shut off.
- ✓ The format is documented .crbm Chunked Remote BitMap — chunked because 16-bit chunks are the data model, remote because every choice in it assumes the bytes are far away and billed per request. A sequence of per-chunk payloads behind a footer index; every chunk carries a CRC32C in its index entry, verified before any deserialize, and the index and footer carry their own. The container is codec-independent — only the payload bytes belong to a flavor. Specified rather than reverse-engineered.
- ✓ One command writes everything out exportSegments() Cross-language roaring or newline-delimited ndjson, via public read APIs only, with an export-segments CLI that wraps it. The exit is a shipped feature, not a migration project.
The benchmarks page carries the rate past which a flat tier is cheaper, the methodology, and what each measurement does and does not establish.
Start with the memory drivers.
They need no account and no configuration, and the code you write against them is the code you keep. Point it at a bucket when you are ready.
npm i @cloudbitmaps/roaring @cloudbitmaps/s3
# the codec you want, and the storage you have.
# also: @cloudbitmaps/gcs, @cloudbitmaps/azure-blob