Benchmarks · aws us-east-1 on-demand · gated
What it costs, and where it stops paying.
Measured against real S3 — no emulator — with the request counts read off the AWS SDK rather than off our own arithmetic. Every money figure and rate below is checked against its source by a build gate, so a regression or an overclaim fails the build.
The soak and the hard RSS ceiling were measured on the TypeScript codec, the one the store uses, on 2026-10-21 (the ceiling's record is the file CI's gate wrote that day). The in-region run and the large-operand run were measured on the TypeScript codec, on 2026-11-03, through the published packages. The at-scale figures were measured on the TypeScript codec on 2026-11-10. The read and fleet-memory figures were measured on the native codec, in the release each section names, and are to be re-measured. The cost model and the crossover do not depend on the codec. The comparison is the TypeScript codec against native.
Cost anchors
Three kinds of figure, and each row says which it is: request counts metered on real S3 and priced at published rates, a run against the in-memory drivers, and what the shipped estimator models. The last row is the crossover, in GETs and in cold intersects — past it the cluster is simply cheaper.
| Scenario | Value | Basis | Verdict |
|---|---|---|---|
| 1.2 GiB at rest, no traffic | $0.03 /mo | Modelled by estimateCost(): 0.008% of the $346 cluster, and storage is the only line billed while nobody is asking. | Win big |
| A ∩ B over two 2M-id segments | 100 / 2,000 | Measured against the in-memory drivers: chunks fetched in 27.8 ms, 1,900 per segment never requested. Step through it. | Win big |
| 1M cold A ∩ B | $2.40 | Measured requests at list prices: two 500,000-id segments of about 2,000 chunks sharing 100 that lie together made 6 GETs, a pointer read and a tail read per operand and one range each, in 103.85 ms at the median. A combine reads each operand's chunks as ranges, so the requests follow the bytes the shared chunks span, not how many there are; spread over each segment they are 6 GETs too, and most of each object is read. An andNot of a segment of about 2,000 chunks against ten excludes made 33 GETs, $13.20 per million, in 230.61 ms. An iterate of it is expected to make 3 GETs, $1.20: the run did not time it. | Win |
| 1M first single-part loads with store.load() | $11.20 | A generation written once, then the pointer moved to it: a segment's first single-part store.load() made 2 PUT + 3 GET on real S3, with the pointer in the same bucket (the requests measured, the price the list price); a single-part load ran at 3.03 million ids a second. Each kind of steady load at keep: 12 was measured on the same run. The meter counts objects, not ids — so batch a refresh. | Batch it |
| Sustained read throughput above | 329.15 /s 54.9 A ∩ B/s |
Modelled by estimateCost(): object GETs with the cache off — the pessimal posture, chosen deliberately — against a standing three-node Redis-HA cluster at $346 /mo. The same line, for cold intersects of the run's shape, is 54.9 a second at the 6 GETs the run measured. What moves this line. | Use a flat tier |
Assumptions: cache hit rate 0, the stored-bytes term at zero, us-east-1 on-demand request rates, and the reserved-RAM baseline is an ElastiCache HA cluster — 1 primary + 2 replicas on cache.m7g.large, about $115/mo single-node. And one crossover rather than two: a segment changes only by publishing a whole new generation, so the write side is priced per object PUT and a pointer — the loads row — and a writes-per-second threshold would put a rate on an operation the library cannot perform. Change an assumption and the read line moves; the five below are how far.
Past the crossover
That row is a threshold on one axis — sustained reads — and that axis is the only one where a flat tier competes, because the write side is not a rate at all. Two of the five inputs behind it are set to their worst plausible value on purpose; the other three are set in our favour, and this table says which is which. Two of them can remove the crossover entirely.
| Input | Published as | Change it and |
|---|---|---|
| Cache hit rate | 0 | Every read billed — the pessimal value, chosen so the figure cannot flatter us. A cache that actually works moves the line out by the reciprocal of the miss rate, and at a 100% hit rate the crossover disappears: the estimator returns Infinity, because nothing reaches the meter. |
| Per-GET request rate | on-demand | The profile's storage.getPerMillion, at the published us-east-1 list rate, with no volume or committed discount applied. The line is inversely proportional to it, and a store with no per-request charge at all removes it outright: no meter, no crossing. |
| Stored bytes | zero | The first of the three set in our favour, so it is stated: storage is subtracted from the baseline before the division, and publishing it at zero hands the whole node price to the read axis as headroom. The reference corpus — $0.03 /mo, or 0.008% of the cluster — is how much that flatters us. |
| Hot segments | none | The second set in our favour. A long-lived reader re-reads the pointer of each segment it keeps reading once every cache.genTtlMs, and that is taken out of the baseline before the division, like storage; publishing none hands it to the read axis too. Pass hotSegments and readerProcesses to take it back. |
| The baseline itself | $346 /mo | The third: three nodes rather than one, and more than the reference 1.2 GiB set needs. The line scales with whatever price you compare against — a single-node baseline at about $115/mo lowers it in the same ratio — and the estimator's own default compares with the cheapest cluster in its catalogue that holds the data, which for that set is $142.35 /mo, 3 × cache.t4g.medium: against it the line would sit at 135.42 reads a second. Feed estimateCost() the Redis you would actually run. |
The estimator will also tell you this itself: every report carries back the cache posture and pricing profile it used, and names the term that dominates the bill — dominated by point reads (object GETs). Leave loadsPerMonth or hotSegments unset and it says so in its notes, rather than quietly reporting a month with no write side or no pointer refresh at all.
- 01 What you give up going flat the idle price One property, and it is the headline one: paying for storage alone while nobody is asking. $0.03 /mo for 1.2 GiB at rest is 0.008% of the $346 cluster. A flat tier bills you the same whether the traffic arrives or not.
- 02 What no pricing choice can take away 100 / 2,000 Chunk-skipping happens above the driver and above the codec — it is key arithmetic, not a storage feature. Crossing a throughput threshold does not affect it. Step through it.
- 03 And a flat tier still is not RAM for everything two tiers The corpus stays in object storage; only the working set ever occupies RAM, under a cap you set. A node sized for your working set is a much smaller node than one sized for your whole history — which is the comparison a single flat number hides.
- 04 And the write side crosses nothing at all $11.20 / 1M first single-part store.load(), measured requests There is no writes-per-second line here because there is no per-id write: a set changes by loading a whole new generation and advancing the registry pointer, so ingest bills per object PUT — one, or a handful when the object goes multipart — plus the pointer's own PUT and reads, however many ids that generation carries. That figure is the write and the publish; store.load()'s check and collection come on top. Batching is the entire optimisation, and you make it at load time rather than at the meter. How a generation is loaded.
The one crossover, as a shape
The dashed line does not move — that is what paying for a node instead of for usage looks like. The rising curve is what you pay for the same reads. Where they meet is the rate in the table above, and the shaded halves are which side of it you are on.
What is still owed
The in-region run timed a cold intersect, a load, point reads and andNot on S3, so those are published above, as one shape on one client. Seven things are not measured. Wall-clock throughput on shared runners is too noisy to gate. The first seven rows below are what is still owed; the eighth is a measured ceiling, marked as such.
- 01 iterate on S3 owed Its request count is expected and not measured, and the run did not time it.
- 02 Ranges of other sizes owed The cold intersect's and the andNot's times are measured for ranges of up to about 1 MiB; the latency of the larger chunks of a medium or large deployment, and the cost of the bytes a range reads outside the region, are not.
- 03 A Lambda figure owed A function's cold start and initialisation against a real store, measured from inside one.
- 04 materializeMany onto loaded destinations owed The refresh call, whose every destination already holds a generation, has its request counts counted in memory and not measured on S3; the batch runs measured a first materialization onto new destinations, which does not confirm them. The *Into verbs and combines on operands of about a million, five million and ten million ids are measured, and operands that are run-encoded, or with other overlaps, are not.
- 05 Other combine shapes owed More than two operands, and other overlaps than the sweep to 2,000 shared chunks, against a real object store.
- 06 A steady store.load() at another keep owed The run measured keep: 12; the requests at a keep of 1, or of 64, are counted by the engine and not measured on S3.
- 07 GCS and Azure Blob, in-region owed The in-region run measured S3 only.
- 08 RSS ceiling 384m A sustained read + combine + re-load workload over 400 segments completes inside a hard 384m cgroup ceiling with swap disabled — so the limit is a true RSS bound and covers memory a heap sample cannot see; the gate runs the built packages as they are, with no native addon. No OOM-kill, and both creep verdicts (heap and external memory) inside their bands. The ceiling is a property of the workload, so it is what is published; the RSS one machine reads under it is a property of that machine, and is not.
A hundredfold more segments, the same memory
The fear with a library that holds bitmaps is that a long-running server accumulates them until it dies. So the fleet was grown from a thousand segments to a hundred thousand, reading every one, under a fixed reader-cache cap. Retained heap is flat. That flatness is the claim — memory is a function of the cap you set, not of how much data you own.
| Fleet | Retained heap (cap 1024) | Peak RSS | Discovery scan |
|---|---|---|---|
| 1,000 segments | 13.0 MiB | 83.7 MiB | 102.7 ms |
| 10,000 segments | 12.3 MiB | 181.5 MiB | 1,035.2 ms |
| 100,000 segments | 12.4 MiB | 185.3 MiB | 21,825.6 ms |
Across a 100× larger fleet, retained heap stayed inside a 0.7 MiB band. Intersection of two 2,000,000-id segments (2,000 chunks each, 100 shared) fetched only 100 of the 2,000 chunks per segment, in 27.8 ms against the in-memory drivers. The run's range requests carried 403,200 bytes, against the 8,104,225 bytes of the two stored objects (4,052,112 and 4,052,113): 5.0% (derived), a floor, because the counter leaves out each reader's first read, the object's last 262,144 bytes. With those two reads the run read about 927,488 bytes, about 11.4% (derived). Fleet seeded through store.load() at ~45–63 durable segments/s (fsync-bound). Measured on Apple M3 Pro (arm64, node v24.22.1, commit df765485, 2026-11-18, load 2.1 at the start and 2.4 at the end) — discovery is filesystem-bound here, and its scan took 10.1x for 10x the segments, then 21.1x for 10x the segments; read the milliseconds as this run's.
- 01 What is bounded retained heap Sampled after a forced GC, so it is live data rather than garbage awaiting collection. This is the column the invariant is about.
- 02 What is not process RSS Peak RSS is not the bound, and the table shows it. The harness seeds a hundred thousand segments inside the same process, and peak RSS is a high-water mark: it keeps the most the process ever held, which may include the seeding. It is measuring the benchmark, not the bound.
- 03 What gets worse discovery time Only the admin and discovery passes pay this — checkConsistency, retireExpired, eraseSubject, subjectReport and exportSegments enumerate the registry before they do any work, and segments() is that enumeration. No read verb enumerates: has, count, iterate and intersect each address one segment. The retirement sweep can already skip the walk entirely (scan: 'index' reads only the buckets that are due) or be sharded across workers — but a shard filter is applied after the enumeration, so each worker still walks every row and that walk stays O(total). An indexed cursor for the general case would bound it, and has not been built.
The TypeScript codec against native
Measured on one Apple M3 Pro under Node v24.22.1 on 2026-10-22, through the store with a warm cache, 31 rounds a
figure, each figure against an A/A control (the native codec twice), one codec per process for reads. Negative
means the TypeScript codec was faster. The record is
bench/ts-codec-comparison-results.json and the per-operation tables are on the
benchmarks page in the repository.
Wall-clock, one machine, recorded and not asserted.
- 01 Reads +3.4% worst has, count, iterate, intersect and andNot on five data sets are all inside the 5% bar: dense four-operand intersect is the worst cell at +3.4%, and sparse intersect is 37% to 40% faster. No union was timed through the store.
- 02 Bulk load, warm −35.6% 183.1 ms to 117.2 ms for a million ids; an erasure's rewrite −33.8%, an export −14.1%.
- 03 First load of two small segments, fresh process +13.1% Slower, not faster: 135.02 ms to 153.0 ms, and the first has is +6.6%. Module load is 19.5 ms against 1.1 ms (−94.2%).
- 04 One container pair up to 3.4× slower In the per-operation record (2026-11-10, Apple M3 Pro, Node v24.22.1, commit 1bee11b9, load 1.57 before and 2.41 after), a union of a dense pair of containers took 900 ns native and 3,066 ns TypeScript; of a skewed pair, 4,393 ns native and 14,133 ns TypeScript. Through the store the combines above are inside the bar anyway.
Method
Real services
Real S3 in us-east-1 at on-demand rates. No emulator and no simulated latency. The harness rehearses its workload against MinIO for free, and a rehearsal's results file is kept out of git, so what gets published is a real-account run or nothing.
Metered at the wire
Request counts come from the AWS SDK layer, not from the library's metrics sink — which cannot see a PUT at all (it emits no storage.put event, a known observability gap), and reports only chunk reads as storage.get, not pointer or index reads. A PUT bills at 12.5× a GET, so an ingest-heavy workload priced without them is understated.
Gated, both directions
Every money figure on this page, the crossover rate and the calibration run's figures are checked by a build gate against the source that produced them — bench/results.json, docs/benchmarks.md, a calibration run's own results file, the soak's own results, the pricing profile itself — and the check also fails on a figure those sources cannot account for, so a number cannot improve without evidence either. The at-scale table is held to the benchmark's own results file too.
And the estimator itself
The modelled figures here come from estimateCost(), so its accuracy is the claim they rest on. For point reads, priced against the chunk GETs a metrics sink observed, its prediction must land on or above what the engine incurred: it never quotes fewer chunk reads than the engine makes. For the rest it counts what the engine is tested to send: each operand's pointer and index before an intersect's chunks, what store.load() adds to a load, and the pointer refresh for the segments you keep reading, on S3's request shape, in as many reader processes as you pass it. And it still quotes low where it cannot see: more hot segments than a reader keeps open, the index every reader opens again after each load, an intersect that outlives the pointer TTL, an index larger than the tail read on an encrypted segment or a row with no usable summary, a load that loses a publish race.
| Operation | Per million | Requests |
|---|---|---|
| Cold A ∩ B, 100 of 1,999 chunks shared, the median | $2.40 | 6 GETs |
| The same, each pointer read once — expected | $2.40 | 6 GETs |
| A segment's first single-part store.load() in the calibration run | $11.20 | 2 PUT + 3 GET |
| A segment's first multipart store.load() in the calibration run | $26.20 | 5 PUT-class + 3 GET |
Measured request counts at list prices, and expected where marked: the second row is the code's prediction, which the run met exactly. The two load rows are a segment's first load: the requests are measured and the price is the list price; a steady load at keep: 12 is measured too, on the same run, with each kind in the benchmarks doc. All 40 cold intersects — each on a fresh store, with nothing cached — returned the planned ids, checked by count and by sum, and each fetched 100 of 1,999 chunks per segment, the chunks the two share, and no other, as one range. Each also made one tail read per segment: its last 256 KiB, which holds the footer and the index. The median took 103.85 ms from AWS CloudShell inside the region with a client of 128 sockets, a cold count() 25.43 ms, and a warm intersect 3.73 ms. PUT-class requests are the ones S3 bills at the PUT rate, listings included. The run's requests, teardown included, cost $0.0027964; there is no data-transfer charge inside the region. The run's report explains every figure.
What these numbers do not establish
Each of these is a real limit on what you should conclude.
- 01 Published rates × wire-metered ops — not a cloud bill not the invoice Request tiers and storage-class transitions are unmeasured, and AWS billing lags hours with no per-run granularity, so the figure quoted is arithmetic over real counts: stronger than a model, and weaker than an invoice.
- 02 One shape, one client, one day a single run The latency on this page is one in-region run from AWS CloudShell: two operands at one overlap, plus a sweep, all of whose shared chunks lie together. It is not a guarantee for other shapes or other clients — what is still owed.
- 03 Your region and workload move the crossover estimateCost() These are us-east-1 on-demand rates with the cache off and the stored-bytes term at zero. Feed your own pricing profile in and get your own crossover — the same function generates this page. How.
- 04 The intersection window is bounded on local drivers, not on a cloud one 2 × 32 payloads, any segment size The window is bounded two ways: a per-commit test counts the chunk payloads actually in flight and holds them at concurrency × operands range requests — 8× the chunks gives the same peak, which is the claim that matters, since it is residency not scaling with segment size that makes a 2,000-chunk intersection fit in a small runtime. And the long-running soak issues combines: 11632 of them over 90 s, alongside 4656 re-loads — each re-load publishes a new generation, which is what makes the reader cache's generation refresh and the cache's generation-keyed entries do real work instead of sitting idle — after which the store's external memory (the typed arrays its bitmaps live in) had moved by -0.36 MiB — where anything above 1.49 MiB would have failed the run. What stays local is the driver: both run against local storage, so this bounds our own bookkeeping and says nothing about a real S3 round trip. The measured run remains a single operation, not a duration.
Run it against your own rates.
The cost model is a shipped, pure function. Ours is one pricing profile; yours will move every number on this page, including which side of the crossover you are on.
git clone https://github.com/cloudbitmaps/cloudbitmaps
pnpm install
# regenerates bench/results.json
pnpm bench