Benchmarks · aws us-east-1 on-demand · gated
What it costs, and where it stops paying.
Measured against real S3 — no emulator — with the request counts read off the AWS SDK rather than off our own arithmetic. Every money figure and rate below is checked against its source by a build gate, so a regression or an overclaim fails the build.
Cost anchors
Published rates applied to wire-metered operation counts. The last row is the crossover, in GETs and in cold intersects — past it a flat node is simply cheaper.
| Scenario | Value | Basis | Verdict |
|---|---|---|---|
| 1M count() calls | $0.14 | Zero payload reads — summed straight from the .crbm index, and exact rather than estimated. This is the object-store half of the July run: the pointer read that resolves the segment's current generation was billed to a NoSQL registry that no longer ships, so it is not in this figure. | Win |
| 1.2 GiB at rest, no traffic | $0.03 /mo | 0.008% of a Redis-HA node. Nothing is billed while nobody is asking. | Win big |
| A ∩ B over two 2M-id segments | 100 / 2,000 | Chunks fetched in 24.6 ms — 1,900 per segment never requested. Step through it. | Win big |
| 1M cold A ∩ B on real S3 | $82.40 | Two 500,000-id segments sharing 100 of 1,999 chunks: 206 GETs at the median — a tail read per operand, the shared chunks, and each pointer read twice, because from a laptop an intersect outlasted the 2 s pointer refresh. Priced by the chunks the two share, not by their size, while each index fits the 256 KiB tail read. Every one cold, and every one exact. Its crossover is the last row. The single-bucket run. | Win |
| 1M loads, written and published | $11.20 | A generation written once, then the pointer moved to it: 2 PUT + 3 GET, measured on real S3 with the pointer in the same bucket. store.load() also lists and collects, which about doubles it. The meter counts objects, not ids — so batch a refresh. | Batch it |
| Sustained read throughput above | 329.15 /s 1.6 A ∩ B/s |
Object GETs with the cache off — the pessimal posture, chosen deliberately — against a standing Redis-HA node at $346 /mo. The same line, for cold intersects of the run's shape at 206 GETs each, is 1.6 a second. What moves this line. | Use a flat tier |
Assumptions: cache hit rate 0, the stored-bytes term at zero, us-east-1 on-demand request rates, and the reserved-RAM baseline is an ElastiCache HA cluster — 1 primary + 2 replicas on cache.m7g.large, about $115/mo single-node. And one crossover rather than two: a segment changes only by publishing a whole new generation, so the write side is priced per object PUT and a pointer — the loads row — and a writes-per-second threshold would put a rate on an operation the library cannot perform. Change an assumption and the read line moves; four of them are how far.
Past the crossover
That row is a threshold on one axis — sustained reads — and that axis is the only one where a flat node competes, because the write side is not a rate at all. Two of the four inputs behind it are set to their worst plausible value on purpose; the other two are set in our favour, and this table says which is which. Two of them can remove the crossover entirely.
| Input | Published as | Change it and |
|---|---|---|
| Cache hit rate | 0 | Every read billed — the pessimal value, chosen so the figure cannot flatter us. A cache that actually works moves the line out by the reciprocal of the miss rate, and at a 100% hit rate the crossover disappears: the estimator returns Infinity, because nothing reaches the meter. |
| Per-GET request rate | on-demand | The profile's storage.getPerMillion, at the published us-east-1 list rate, with no volume or committed discount applied. The line is inversely proportional to it, and a store with no per-request charge at all removes it outright: no meter, no crossing. |
| Stored bytes | zero | The first of the two set in our favour, so it is stated: storage is subtracted from the baseline before the division, and publishing it at zero hands the whole node price to the read axis as headroom. The reference corpus — $0.03 /mo, or 0.008% of the node — is how much that flatters us. |
| The baseline itself | $346 /mo | The second: three nodes rather than one, and the line scales with whatever price you compare against — a single-node baseline at about $115/mo lowers it in the same ratio. Feed estimateCost() the node you would actually run. |
The estimator will also tell you this itself: every report carries back the cache posture and pricing profile it used, and names the term that dominates the bill — dominated by point reads (object GETs). Leave loadsPerMonth unset and it says so in its notes rather than quietly reporting a month with no write side at all.
- 01 What you give up going flat the idle price One property, and it is the headline one: paying nothing while nobody is asking. $0.03 /mo for 1.2 GiB at rest is 0.008% of a Redis node. A flat tier bills you the same whether the traffic arrives or not.
- 02 What no pricing choice can take away 100 / 2,000 Chunk-skipping happens above the driver and above the codec — it is key arithmetic, not a storage feature. Crossing a throughput threshold does not affect it. Step through it.
- 03 And a flat node still is not RAM for everything two tiers The corpus stays in object storage; only the working set ever occupies RAM, under a cap you set. A node sized for your working set is a much smaller node than one sized for your whole history — which is the comparison a single flat number hides.
- 04 And the write side crosses nothing at all $11.20 / 1M loads There is no writes-per-second line here because there is no per-id write: a set changes by loading a whole new generation and advancing the registry pointer, so ingest bills per object PUT — one, or a handful when the object goes multipart — plus the pointer's own PUT and reads, however many ids that generation carries. That figure is the write and the publish; store.load()'s listings and collection pass come on top. Batching is the entire optimisation, and you make it at load time rather than at the meter. How a generation is loaded.
The one crossover, as a shape
The dashed line does not move — that is what paying for a node instead of for usage looks like. The rising curve is what you pay for the same reads. Where they meet is the rate in the table above, and the shaded halves are which side of it you are on.
What is still owed
The loaded read path has never been timed inside a region, so no latency is published here as the library's. The bill of a cold intersect and of a load is measured; their stopwatch is not, and neither is everything a load or a read can cost. Wall-clock throughput on shared runners is too noisy to gate. The first six rows below are what is still owed, latency and throughput first; the seventh is a measured ceiling, marked as such.
- 01 In-region read latency owed Both calibration runs were driven from a laptop outside us-east-1, so their wall-clock was dominated by internet transit and they calibrated the cost claim only. No percentile for has, count or intersect is published as the library's, and the figures gate fails the build if such a row reappears in the benchmark record — so publishing one takes a run and a deliberate change to the gate that forbids it.
- 02 Load throughput owed Sustained bulkLoadCrbmGeneration rate against a real object store, from inside the region, at the segment sizes a real refresh produces. Its cost is measured — the loads row above — but the September run timed whole loads from a laptop 83 ms from the region, and the at-scale table's ~38–51 durable segments/s is local disk and fsync-bound. Neither is that number.
- 03 Intersect latency owed In-region wall-clock for a chunk-skipping A ∩ B, and for andNot with a large exclude, against a real object store rather than local disk; then the *Into verbs, and the sweep over operand count and chunk overlap. The request count and the bill of an A ∩ B are measured on real S3, and an andNot's are not; neither's wall-clock is. The 24.6 ms below is one operation against the in-memory drivers.
- 04 store.load() on S3 owed The September run measured a load's write and publish. store.load() also lists the segment to choose a generation number and runs a collection pass after the publish. A test counts what that adds, about twice the bill; it has not been measured on S3.
- 05 An estimator that counts the pointer owed estimateCost() has no term for the pointer's requests in a load or an intersect, none for an intersect's tail reads, and none for the pointer refresh, so it under-quotes both operations on a single-bucket store. Until it counts them, the measured figures on this page are the ones to use.
- 06 A Lambda figure owed A function's cold start and initialisation against a real store, measured from inside one.
- 07 RSS ceiling 384m A sustained read + combine + re-load workload over 400 segments completes inside a hard 384m cgroup ceiling with swap disabled — so the limit is a true RSS bound and covers the addon's off-heap memory. No OOM-kill, and both creep verdicts inside their bands. Two very different machines agree: a Linux CI runner and an Apple M3 Pro under Docker landed 0.4 MiB apart on reader RSS while throughput differed 3.7× — which is why the ceiling is published here and the latency in 01 still is not.
A hundredfold more segments, the same memory
The fear with a library that holds bitmaps is that a long-running server accumulates them until it dies. So the fleet was grown from a thousand segments to a hundred thousand, reading every one, under a fixed reader-cache cap. Retained heap is flat. That flatness is the claim — memory is a function of the cap you set, not of how much data you own.
| Fleet | Retained heap (cap 1024) | Peak RSS | Discovery scan |
|---|---|---|---|
| 1,000 segments | 8.2 MiB | 68.1 MiB | 87.6 ms |
| 10,000 segments | 8.3 MiB | 86.8 MiB | 1,067.9 ms |
| 100,000 segments | 7.4 MiB | 162.2 MiB | 11,606.4 ms |
A 100× larger fleet moved retained heap by 0.9 MiB. Intersection of two 2,000,000-id segments (2,000 chunks each, 100 shared) fetched only 100 of the 2,000 chunks per segment, in 24.6 ms. Fleet seeded at ~38–51 durable segments/s (fsync-bound). Measured on Apple M3 Pro (arm64, node v24.18.1) — discovery is filesystem-bound here, so the shape is the claim, not the absolute milliseconds.
- 01 What is bounded retained heap Sampled after a forced GC, so it is live data rather than garbage awaiting collection. This is the column the invariant is about.
- 02 What is not process RSS RSS grows, and we are not going to pretend otherwise. The harness seeds a hundred thousand segments inside the same process, and RSS is monotonic — it never returns pages. It is measuring the benchmark, not the bound.
- 03 What gets worse discovery time Only the admin passes pay this — checkConsistency, retireExpired, eraseSubject and subjectReport enumerate the registry before they do any work. No read verb enumerates: has, count, iterate and intersect each address one segment. The retirement sweep can already skip the walk entirely (scan: 'index' reads only the buckets that are due) or be sharded across workers — but a shard filter is applied after the enumeration, so each worker still walks every row and that walk stays O(total). An indexed cursor for the general case would bound it, and has not been built.
Method
Real services
Real S3 in us-east-1 at on-demand rates. No emulator and no simulated latency. The harness rehearses its workload against MinIO for free, and a rehearsal's results file is kept out of git, so what gets published is a real-account run or nothing.
Metered at the wire
Request counts come from the AWS SDK layer, not from the library's metrics sink — which cannot see a PUT at all (it emits no storage.put event, a known observability gap). A PUT bills at 12.5× a GET, so an ingest-heavy workload priced without them is understated.
Gated, both directions
Every money figure on this page, the crossover rate and the calibration run's figures are checked by a build gate against the source that produced them — bench/results.json, docs/benchmarks.md, a calibration run's own results file, the soak's own results, the pricing profile itself — and the check also fails on a figure those sources cannot account for, so a number cannot improve without evidence either. The at-scale table is the exception: nothing checks it yet.
And the estimator itself
The modelled figures here come from estimateCost(), so its accuracy is the claim they rest on. For point reads, priced against the chunk GETs a metrics sink observed, its prediction must land on or above what the engine incurred: it never quotes fewer chunk reads than the engine makes. That is all it covers. For an intersect it prices the chunksPerIntersect you give it, 1 by default, and it has no term yet for a single-bucket store's pointer and tail reads, so for a load or an intersect there it under-quotes, and the measured figures below are the ones to use.
| Operation | Per million | Requests |
|---|---|---|
| Cold A ∩ B, 100 of 1,999 chunks shared, the median | $82.40 | 206 GETs |
| The same, each pointer read once — expected | $81.60 | 204 GETs |
| Write and publish a segment | $11.20 | 2 PUT + 3 GET |
| Write and publish, multipart | $26.20 | 5 PUT-class + 3 GET |
| A segment's first store.load() — expected | $22.80 | 4 PUT-class + 7 GET |
Measured request counts at list prices, and expected where the run did not measure: the same intersect with each pointer read once, as it would be inside the region, and store.load()'s listings and collection pass, which a test counts. All 40 cold intersects — each on a fresh store, with nothing cached — returned the planned ids, checked by count and by sum, and each requested 100 of 1,999 chunks per segment, the chunks the two share, and no other. Each also made one tail read per segment: its last 256 KiB, which holds the footer and the index. PUT-class requests are the ones S3 bills at the PUT rate, listings included. The run's requests, teardown included, cost $0.0034816; data transfer is not in that. The client was a laptop outside the region, so no latency or upload rate from this run is published as the library's: those numbers measured its connection. The run's report explains every figure, in six sections, each with a diagram.
| Term | Billed quantity | Cost |
|---|---|---|
| S3 PUT / LIST | 22 | $0.000110 |
| S3 GET | 23 | $0.000009 |
This is half of a run. Its other half metered a NoSQL delta tier the library no longer has, so those line items and the rates behind them are not restated here — republishing them would put a price on a code path you cannot take. Nor is a total: the rows above are two terms out of four, and a “total” over a subset is a number no run produced. The harness that produced the run, and its raw artifact, were removed along with the tier they were built to meter, so these two rows are the record. What is left is the object-store half — and the object store is now the whole write path and the whole read path. Its two unit figures, $0.14 per million count() calls and $5.88 per million publishes, left the pointer out. The count figure is still the one in the cost anchors; the publish figure is superseded by the single-bucket run above, which measured the pointer too.
What these numbers do not establish
Each of these is a real limit on what you should conclude.
- 00 The July figures predate the single-bucket design pointer measured since Generation resolution ran against a NoSQL table in the July run, so its figures are the object-store half of that shape. The library now keeps the pointer in the same bucket as the data, and the September run measured what that adds: a load's write and publish is 2 PUT + 3 GET, $11.20 per million, where the July figure, which could not see the pointer, was $5.88. Its count() figure has not been re-measured; read it as the object-store half, without the pointer.
- 01 Published rates × wire-metered ops — not a cloud bill not the invoice Request tiers and storage-class transitions are unmeasured, and AWS billing lags hours with no per-run granularity — so the July run tagged its resources and the Cost Explorer comparison followed a day later, rather than being the figure quoted. This is arithmetic over real counts, which is stronger than a model and weaker than an invoice.
- 02 Cost, not latency no run behind it No in-region latency for the loaded read path is on this page, because no in-region run has measured one yet. Everything here is a bill, not a stopwatch — what that owes you.
- 03 Your region and workload move the crossover estimateCost() These are us-east-1 on-demand rates with the cache off and the stored-bytes term at zero. Feed your own pricing profile in and get your own crossover — the same function generates this page. How.
- 04 The intersection window is bounded on local drivers, not on a cloud one 2 × 8 payloads, any segment size This row used to say the window was not soak-tested, because it was not. It is now bounded two ways: a per-commit test counts the chunk payloads actually in flight and holds them at concurrency × operands — 8× the chunks gives the same peak, which is the claim that matters, since it is residency not scaling with segment size that makes a 2,000-chunk intersection fit in a small runtime. And the long-running soak issues combines: 7262 of them over 90 s, alongside 2908 re-loads — each re-load publishes a new generation, which is what makes the reader cache's generation refresh and the cache's generation-keyed entries do real work instead of sitting idle — after which the roaring engine's live off-heap bytes had moved by 0.00 MiB — where 0.28 MiB would have failed the run. What is still true is the driver: both run against local storage, so this bounds our own bookkeeping and says nothing about a real S3 round trip. The measured run remains a single operation, not a duration.
Run it against your own rates.
The cost model is a shipped, pure function. Ours is one pricing profile; yours will move every number on this page, including which side of the crossover you are on.
git clone https://github.com/cloudbitmaps/cloudbitmaps
pnpm install
# regenerates bench/results.json
pnpm bench