Usage · @cloudbitmaps/roaring v0.27.0
Zero setup, then the same code in production.
The in-memory drivers need nothing — no bucket, no credentials. Swap them for S3 later and every line that loads or reads stays as it was.
Eight lines, then the same eight lines
The difference that matters between these two is the backend the store is built on; the cache bound beside it is optional tuning. Nothing below them knows, or can know, whether the bytes are in a Map or in a bucket.
import { CloudRoaring, MemoryStorage }
from '@cloudbitmaps/roaring';
const store = new CloudRoaring({ storage: new MemoryStorage() });
// the whole set, as one immutable generation
await store.load({ segment: 'high-value-shoppers' },
[75, 99_999, 1_234_567_890]);
const vips = store.segment('high-value-shoppers');
await vips.has(1_234_567_890); // true
await vips.count(); // exact, 0 payloads
import { CloudRoaring } from '@cloudbitmaps/roaring';
import { S3Storage } from '@cloudbitmaps/s3';
// one bucket holds both the data and the pointer, stated once
const backend = new S3Storage({
bucket: 'segments', region: 'us-east-1',
});
const store = new CloudRoaring({
storage: backend,
cache: { maxChunks: 512 }, // optional: the LRU's bound
});
// …every line below here is unchanged
The registry is what resolves a generation in one read instead of a list-scan, what makes an encrypted segment readable, and what every write and every lifecycle call publishes through. You do not wire it separately. A backend — S3Storage, GcsStorage and the rest — carries it, which is why storage is the only location key the store takes. Hand the store a bare IStorageDriver instead and there is no pointer at all: it finds the highest generation in the bucket by list-scan, cleartext and read-only. Fine for a first look, not past it.
Everything else the store takes is optional. cache bounds memory and read staleness. encryption: { keystore } encrypts a segment at rest from its first generation — InProcessKeystore ships in the package, or bring your own IKeystore, the port @cloudbitmaps/core/driver-kit exports — and needs a backend, because the wrapped key lives in the pointer's row. budget: { maxRequests } caps the chunk fetches one count, iterate or combine may fan out into, and the segments one subject scan may reach — 1,000,000 by default — and refuses the call before it fans out; false lifts it. retry tunes how a read retries a transient fault, metrics takes the sink for the events the engine emits, and seams injects a clock and randomness for tests.
The whole API
The read verbs are deliberately the shape of an in-memory Set. The write is not: a segment changes only by getting a whole new generation. So the third column — what a call costs you in backend requests — is the one worth reading.
| Call | Does | Touches |
|---|---|---|
| store.load(ref, ids) | The only way data gets in. Streams an unsorted id source into one immutable .crbm object, then moves the segment's pointer to it by compare-and-swap, fenced on the row it read, and collects what that superseded, keeping one. An empty result over a non-empty segment is refused rather than published, and so is one outside the bounds you set in guard: { minCardinality, minRetained, maxGrowth }, so branch on published. Grouped by chunk first, so it bills per object, not per id. On S3, a segment's first single-part load makes 2 PUT-class and 3 GET requests, measured, and a multipart load made 5 PUT-class and 3 GET; it lists the segment only every 16th generation (at any keep up to 64), and S3 bills a listing at the PUT rate. | 2 PUT + 3 GET |
| has(id) | Resolves one chunk of the current generation: the cache if it is there, otherwise one ranged GET of that chunk. The first read of a segment also reads its pointer and its index. Never a merge of two sources, so it can never be torn. | cache, else 1 GET |
| count() | Reads the id count the registry row records for the current generation, so a cold count is the one pointer read and opens no object; a row with no summary it can use sums per-chunk cardinality from the .crbm index instead. Exact, not estimated — no HyperLogLog — and it never opens a payload. | cache, else 1 GET |
|
iterate() iterate({ after, through }) |
Every id, ascending, reading ahead as ranges: one request at first, widening to up to 32 in flight. With a range, only the ids in (after, through], fetching only the chunks it overlaps — keyset paging — and every combine takes the same two options. Each of these streams has .batches(), which yields the same ids one Uint32Array per chunk. | each chunk read |
| intersect(others, { exclude }) | Async-iterates the intersection. Compares chunk keys first and requests only shared chunks, neighbouring chunks together in one range request; the payload in flight is bounded by concurrency × operands × the 1 MiB a range may be. exclude subtracts suppression segments in the same pass. union takes the same options and reads every chunk of every operand; andNot takes all of them but exclude, since its argument is the subtraction. An operand that names no segment is refused with ValidationError rather than read as empty — a misspelled suppression list would otherwise suppress nothing — unless you pass allowAbsentOperands: true. | the shared chunks |
| seg.stat() | The generation, its count, its metadata and its size (sizeBytes, null when the segment has no generation or the source cannot report a size), all from one resolution. A cold call is one registry read when the row's summary describes the generation, and a tail read of the object more when it does not; a warm one sends nothing. Feed the size to groundedReport from @cloudbitmaps/tools to price a running store. | the row, else the tail |
| seg.pin() | A handle held at the generation that is current now, for as long as you keep it — for an export, a reconciliation, a send that has to match the count you reported. An ordinary handle moves on to a newer generation within cache.genTtlMs of one being published. A hold, not a lease: a pinned read whose generation has been collected fails rather than serve another, so size keep to your longest pinned job. | the row and the index |
|
intersectInto(dest, others) unionInto · andNotInto |
Publishes the result as a new generation of dest rather than streaming it to you — superseded, not added to — and hands back { generation, published, reason, cardinality, … }. Branch on published: an empty result over a non-empty dest is refused and reported, as a load's is, and a lost race throws WriteConflictError. It collects nothing unless you pass keep. | storage + registry |
| store.materializeMany({ operands, outputs, keep }) | Many *Into outputs from one chunk-ordered pass: each operand is read once for the batch, not once per call. Each output is a dest and an and / or / andNot expression over named operands, and run.outputs[i] is what that output's *Into would have returned, or { published: false, error }. Its counts are in run.stats; it reports one op event and a storage.get per range request to the metrics sink, and with dryRun: true it writes nothing and reports what each publish would do. | storage + registry |
| store.eraseSubject(id, { namespace }) | Rewrites every segment in the namespace that holds the id without it, and deletes the generation that held the bit. An id found only outside the current generation (a retained older generation, one above the pointer, a first load's object on a segment with no generation yet) is deleted with those generations, and the ledger reports erased: true all the same. So for each entry its ledger reports erased: true, the bit is physically gone from the bucket when the call returns; a noncurrent version, a replica or a backup keeps the old object until its own lifecycle removes it. It needs a scope: a namespace, or { allNamespaces: true } to sweep every tenant. Read the ledger: an entry noted superseded was not erased by this call, so re-run. A compliance operation, not a hot-path verb — there is no per-id write. | storage + registry |
| store.rollback(ref, generation) | Moves the pointer to a generation still in the bucket that its row knows it published — back, or forward again onto one an earlier rollback left (store.generations(ref) lists them). The one pointer move that is not forward-only, and only an operator makes it: it deletes nothing, refuses a generation that is gone, refuses one its row does not know it published (such as a load that wrote its object and died before the publish), outside the two cases the pointer alone decides, with a ValidationError unless you pass allowUnknown: true, and records segment.rollback on the audit sink you pass it. | registry + two listings |
| store.retireExpired() | The retention sweep: everything whose expiresAt has passed, each through dropSegment. A call, not a daemon — bounded, previewable, and it returns a per-segment ledger instead of throwing. | registry, then storage |
| store.exportSegments(sink) | Writes every registered segment that is not destroyed out as cross-language roaring or newline-delimited ndjson, using only public read APIs; a destroyed segment (crypto-shredded, destroyed in the clear or dropped) is listed in manifest.skipped, not read. The one-command exit, and there is a CLI that wraps it. | storage |
| estimateCost(input) | Your region and rates in, projected bill out, from @cloudbitmaps/tools. The benchmarks page is generated from this same function. | nothing — pure |
No per-id TTL, by design A segment can expire — record setRetention(ref, { expiresAt }) and a retireExpired() sweep retires it, on a schedule you own. An individual id cannot, because a bitmap stores ids and not (id, timestamp) pairs — an expiry per id costs 4–8 bytes each and takes the compression with it. Retention is a segment that expires, not a bit that ages. To stop serving a set on time even when the sweep is late, check the deadline where you read it.
Both writes are the same protocol: build one immutable object, then move the pointer.
// `store` from above — the backend is already wired.
const ref = { namespace: 'audience', segment: 'high-value' };
// The warehouse decides the set; hand over its cursor
// (any AsyncIterable of ids), not an array. Re-run on a
// cadence — each run publishes the next generation.
const r = await store.load(ref, athenaCursor());
if (r.published) console.log(r.cardinality); // now current
else console.warn(r.reason); // 'empty', 'superseded' or a guard bound you set
const seg = (name: string) =>
store.segment(name, { namespace: 'audience' });
// Each operand is loaded as in load.ts: a combine refuses
// one that names no segment rather than read it as empty.
const vips = seg('high-value');
const eu = seg('eu-residents');
const optedOut = seg('opted-out');
const target = seg('eu-vips');
// Suppression folds into the same pass, and the result
// becomes a new generation of `target` — unless it comes
// out empty over a non-empty one, which is refused.
const res = await vips.intersectInto(
target, [eu], { exclude: [optedOut] },
);
if (!res.published) console.warn(res.reason); // 'empty'
Retention: a segment that expires
Put the date in the name and the window becomes set algebra. A user is re-eligible the moment the wave moves to tomorrow’s bucket, because tomorrow’s bucket is a different, empty set. Nothing retires itself: drop the old bucket as below, or record when it expires with setRetention and let the retireExpired sweep you schedule drop it.
// The bucket IS the re-eligibility rule.
const ref = { namespace: 'sent-daily', segment: localDay };
const sent = store.segment(localDay, { namespace: 'sent-daily' });
if (!(await sent.has(userId))) { // not sent today
await send(userId); // then queue the id
}
// On a cadence — every few minutes, or once at the end
// of the wave — publish the day so far as one object.
await store.load(ref, sentToday());
// The namespace is part of the identity — omit it
// and you silently address a different segment.
const ref = { namespace: 'sent-daily', segment: oldDay };
// Preview first — dryRun changes nothing.
await store.dropSegment(ref, {
confirmSegment: ref.segment, dryRun: true,
});
// Tombstone the pointer, then sweep every generation.
const r = await store.dropSegment(ref, {
confirmSegment: ref.segment,
});
// r.reason === 'absent' means you named it wrong.
The check is exactly as fresh as the cadence. A read sees the published generation, so an id queued since the last load is not in sent yet — this is at-most-once per published set, not against the last millisecond. Where the window is a day and the cadence is minutes, that is the right trade; where the answer has to be true the instant an id is claimed, claim it where the ids are queued and keep the segment as the durable record.
Names — any non-empty string, segment and namespace alike. sent:2026-08-01, orders/2026 and user@example.com are all legal; each storage layer escapes what it cannot take literally. The limits are three: non-empty, encodable as UTF-8 (no unpaired surrogate), and 256 characters once encoded for storage. Encoding expands anything outside A–Z a–z 0–9 . _ -, and a leading _, so a name full of colons or non-ASCII text reaches that sooner than its length suggests. The namespace split is still the better shape for a family: listing the namespace enumerates exactly that family’s buckets, which is the list a retention sweep wants.
Do not reach for an object-store lifecycle rule on its own. It deletes the bytes while the registry still points at them — the torn state a DR runbook says not to serve traffic on — and it surfaces intermittently, because a read checks the in-process cache before storage. It passes in a process whose cache is already populated and starts failing after a restart. dropSegment exists so the ordering cannot be got wrong: the pointer is tombstoned first, then the objects are swept — and it hands back generationsRemaining, which is non-empty exactly when bytes survived and the drop should be re-run. A lifecycle rule is a fine backstop for orphans — just give it a longer window than your retention. The guide has the rest.
The API reference is checked against the real exports by CI, so it cannot drift from the package.
Four drivers, each carrying both halves
A backend carries the immutable generation objects and the pointer row, from one bucket and one prefix — so a deployment names one service, not two. Anything with a cheap ranged read and a write-once put can serve as storage; anything with a cheap compare-and-swap can serve as the pointer; each of the four below does both. A driver author who needs halves of their own pairs them with brandAsBackend({ storage, registry }) from @cloudbitmaps/core/driver-kit. Each backend's SDK is a real dependency of its own package — install only the storage you use.
Redis is not in this table. It is the line the benchmarks page prices against, not a backend this library writes to — the storage here is your own bucket plus one small pointer row per segment. It is still a good neighbour: when your set is defined by events arriving one at a time, accumulate them where they arrive — Redis is genuinely good at that — and load the set on a cadence.
| Backend | Package | Roles | Cloud SDK (a real dependency) |
|---|---|---|---|
| S3 | @cloudbitmaps/s3 | storage · registry | @aws-sdk/client-s3 >=3.700.0 <4 |
| Google Cloud Storage | @cloudbitmaps/gcs | storage · registry | @google-cloud/storage ^8.1.0 |
| Azure Blob | @cloudbitmaps/azure-blob | storage · registry | @azure/storage-blob ^12.23.0 |
| Filesystem | main entry | storage · registry | none |
Memory drivers ship in the main entry too and need nothing — they are what the quickstart uses. An S3-compatible store (R2, MinIO) works through the S3 driver, given its endpoint — most also need pathStyle and credentials — if it honours S3's conditional writes and is strongly consistent: the write-once put and the pointer's compare-and-swap are built on the first, and the erasure, drop and rollback checks on the second. MinIO is the one the integration suite runs against. Deletes decided from a read are conditional (If-Match or ifGenerationMatch) where the backend reports conditionalDelete: on by default for S3 on an AWS host and for Azure Blob, off by default for GCS and for an S3-compatible endpoint, where the conditionalDelete option turns it on.
Estimate your own bill
Our published numbers are us-east-1 on-demand. Yours are not. Feed your own rates in and get your own crossover — the site's modelled figures call this same function, so they cannot drift ahead of it. It ships in @cloudbitmaps/tools, a package of offline tools beside the library. A segment's stat() reports the size that groundedReport prices a running store from: groundedReport({ storageBytes: (await seg.stat()).sizeBytes }), where a null size prices storage at nothing and sets assumptions.grounded to false. The estimate counts the pointer too — the object beside your data that says which generation is current — and each operand's index, so pass chunksPerIntersect as the chunk range requests your intersects make, summed over their operands, and hotSegments for the segments each long-lived reader keeps reading, with readerProcesses for how many readers: each re-reads a segment's pointer at most once per genTtlMs, which the estimate takes as the store's default of 2 s unless the workload names your cache.genTtlMs. By default it compares with the Redis that would hold your data; the site's chart compares with one $346 cluster, ONE_REDIS_HA_CLUSTER, passed as the profile's redis.
import { estimateCost, AWS_US_EAST_1_ONDEMAND }
from '@cloudbitmaps/tools';
const bill = estimateCost({
segments: [{ sizeBytes: 60_000_000, count: 20 }],
workload: { readsPerSec: 800, loadsPerMonth: 900, hotSegments: 20 },
pricing: AWS_US_EAST_1_ONDEMAND, // or your own profile
});
bill.redisCrossover.readsPerSec; // against a Redis sized to the data
- 01 Schedule the retention sweep retireExpired() The only scheduled work there is — no daemon, no lifecycle worker. An EventBridge rule, a Kubernetes CronJob or a queue job all do; nothing here starts a timer.
- 02 Bound the cache to what your runtime can hold cache.maxChunks An LRU with a hard ceiling — 1,024 decoded chunks by default — and the ceiling is yours to choose. Correctness does not depend on it either way.
- 03 Collect superseded generations, keeping one $0.03/mo · 1.2 GiB A load collects what it supersedes, keeping the newest superseded generation by default as a grace window, so a read still fetching from it saves a round trip. An *Into write collects nothing unless given keep, and the next load of its destination lists twice when its number was free (once to seed the ledger, once to collect, with no separate write), collects everything below its pointer beyond its keep, and records the window it kept. Storage is the only cost of that window. It is not a snapshot facility: no setting here holds a long job on one generation — seg.pin() does.
Your data is not held hostage
The reason to be comfortable adopting a pre-1.0 library is not a promise about the library. It is that leaving costs you one command, and the bytes were never anywhere you could not reach.
- ✓ The bytes are in your bucket, immutable your account Storage objects are generation-keyed and never mutated in place. Nothing is stored anywhere we control, and there is no service to be shut off.
- ✓ The format is documented .crbm Chunked Remote BitMap — chunked because 16-bit chunks are the data model, remote because every choice in it assumes the bytes are far away and billed per request. A sequence of per-chunk payloads behind a footer index; every chunk carries a CRC32C in its index entry, verified before any deserialize, and the index and footer carry their own. The container is codec-independent — only the payload bytes belong to a flavor. Specified rather than reverse-engineered.
- ✓ One command writes everything out exportSegments() Cross-language roaring or newline-delimited ndjson, via public read APIs only, with an export-segments CLI that wraps it. The exit is a shipped feature, not a migration project.
The benchmarks page carries the rate past which a standing cluster is cheaper, the methodology, and what each measurement does and does not establish.
Start with the memory drivers.
They need no account and no configuration, and the code you write against them is the code you keep. Point it at a bucket when you are ready.
npm i @cloudbitmaps/roaring @cloudbitmaps/s3
# the codec you want, and the storage you have.
# also: @cloudbitmaps/gcs, @cloudbitmaps/azure-blob
# to price a store: @cloudbitmaps/tools