CloudBitmaps

Benchmarks · aws us-east-1 on-demand · gated

What it costs, and where it stops paying.

Measured against real S3 — no emulator — with the request counts read off the AWS SDK rather than off our own arithmetic. Every money figure and rate below is checked against its source by a build gate, so a regression or an overclaim fails the build.

The soak and the hard RSS ceiling were measured on the TypeScript codec, the one the store uses, on 2026-10-21 (the ceiling's record is the file CI's gate wrote that day). The in-region run and the large-operand run were measured on the TypeScript codec, on 2026-11-03, through the published packages. The at-scale figures were measured on the TypeScript codec on 2026-11-10. The read and fleet-memory figures were measured on the native codec, in the release each section names, and are to be re-measured. The cost model and the crossover do not depend on the codec. The comparison is the TypeScript codec against native.

The cost anchors → What they do not establish

Modelled figures come from the shipped estimateCost(), measured ones from each run's committed record — neither can drift ahead of its source.

Cost anchors

Three kinds of figure, and each row says which it is: request counts metered on real S3 and priced at published rates, a run against the in-memory drivers, and what the shipped estimator models. The last row is the crossover, in GETs and in cold intersects — past it the cluster is simply cheaper.

Cost anchorsestimateCost() + the AWS run
Scenario Value Basis Verdict
1.2 GiB at rest, no traffic $0.03 /mo Modelled by estimateCost(): 0.008% of the $346 cluster, and storage is the only line billed while nobody is asking. Win big
A ∩ B over two 2M-id segments 100 / 2,000 Measured against the in-memory drivers: chunks fetched in 27.8 ms, 1,900 per segment never requested. Step through it. Win big
1M cold A ∩ B $2.40 Measured requests at list prices: two 500,000-id segments of about 2,000 chunks sharing 100 that lie together made 6 GETs, a pointer read and a tail read per operand and one range each, in 103.85 ms at the median. A combine reads each operand's chunks as ranges, so the requests follow the bytes the shared chunks span, not how many there are; spread over each segment they are 6 GETs too, and most of each object is read. An andNot of a segment of about 2,000 chunks against ten excludes made 33 GETs, $13.20 per million, in 230.61 ms. An iterate of it is expected to make 3 GETs, $1.20: the run did not time it. Win
1M first single-part loads with store.load() $11.20 A generation written once, then the pointer moved to it: a segment's first single-part store.load() made 2 PUT + 3 GET on real S3, with the pointer in the same bucket (the requests measured, the price the list price); a single-part load ran at 3.03 million ids a second. Each kind of steady load at keep: 12 was measured on the same run. The meter counts objects, not ids — so batch a refresh. Batch it
Sustained read throughput above 329.15 /s
54.9 A ∩ B/s
Modelled by estimateCost(): object GETs with the cache off — the pessimal posture, chosen deliberately — against a standing three-node Redis-HA cluster at $346 /mo. The same line, for cold intersects of the run's shape, is 54.9 a second at the 6 GETs the run measured. What moves this line. Use a flat tier

Assumptions: cache hit rate 0, the stored-bytes term at zero, us-east-1 on-demand request rates, and the reserved-RAM baseline is an ElastiCache HA cluster — 1 primary + 2 replicas on cache.m7g.large, about $115/mo single-node. And one crossover rather than two: a segment changes only by publishing a whole new generation, so the write side is priced per object PUT and a pointer — the loads row — and a writes-per-second threshold would put a rate on an operation the library cannot perform. Change an assumption and the read line moves; the five below are how far.

Past the crossover

That row is a threshold on one axis — sustained reads — and that axis is the only one where a flat tier competes, because the write side is not a rate at all. Two of the five inputs behind it are set to their worst plausible value on purpose; the other three are set in our favour, and this table says which is which. Two of them can remove the crossover entirely.

What moves this line All five are inputs, not constants
Input Published as Change it and
Cache hit rate 0 Every read billed — the pessimal value, chosen so the figure cannot flatter us. A cache that actually works moves the line out by the reciprocal of the miss rate, and at a 100% hit rate the crossover disappears: the estimator returns Infinity, because nothing reaches the meter.
Per-GET request rate on-demand The profile's storage.getPerMillion, at the published us-east-1 list rate, with no volume or committed discount applied. The line is inversely proportional to it, and a store with no per-request charge at all removes it outright: no meter, no crossing.
Stored bytes zero The first of the three set in our favour, so it is stated: storage is subtracted from the baseline before the division, and publishing it at zero hands the whole node price to the read axis as headroom. The reference corpus — $0.03 /mo, or 0.008% of the cluster — is how much that flatters us.
Hot segments none The second set in our favour. A long-lived reader re-reads the pointer of each segment it keeps reading once every cache.genTtlMs, and that is taken out of the baseline before the division, like storage; publishing none hands it to the read axis too. Pass hotSegments and readerProcesses to take it back.
The baseline itself $346 /mo The third: three nodes rather than one, and more than the reference 1.2 GiB set needs. The line scales with whatever price you compare against — a single-node baseline at about $115/mo lowers it in the same ratio — and the estimator's own default compares with the cheapest cluster in its catalogue that holds the data, which for that set is $142.35 /mo, 3 × cache.t4g.medium: against it the line would sit at 135.42 reads a second. Feed estimateCost() the Redis you would actually run.

The estimator will also tell you this itself: every report carries back the cache posture and pricing profile it used, and names the term that dominates the bill — dominated by point reads (object GETs). Leave loadsPerMonth or hotSegments unset and it says so in its notes, rather than quietly reporting a month with no write side or no pointer refresh at all.

The one crossover, as a shape

The dashed line does not move — that is what paying for a node instead of for usage looks like. The rising curve is what you pay for the same reads. Where they meet is the rate in the table above, and the shaded halves are which side of it you are on.

Where pay-per-use crosses one Redis-HA cluster aws-us-east-1-ondemand · cache off · one Redis-HA cluster = $346/mo, whatever the data size CloudBitmaps (pay-per-use)Redis HA cluster (flat) Sustained point reads (object GETs, cache off) cheaper than Redis Redis is cheaper $0 $200 $400 $600 0 100 200 300 400 500 reads / sec crossover ≈ 329 /s
Generated by pnpm bench from estimateCost() — the same function @cloudbitmaps/tools ships, so the chart cannot disagree with the estimator a caller gets.

What is still owed

The in-region run timed a cold intersect, a load, point reads and andNot on S3, so those are published above, as one shape on one client. Seven things are not measured. Wall-clock throughput on shared runners is too noisy to gate. The first seven rows below are what is still owed; the eighth is a measured ceiling, marked as such.

A hundredfold more segments, the same memory

The fear with a library that holds bitmaps is that a long-running server accumulates them until it dies. So the fleet was grown from a thousand segments to a hundred thousand, reading every one, under a fixed reader-cache cap. Retained heap is flat. That flatness is the claim — memory is a function of the cap you set, not of how much data you own.

Memory at fleet scaleMeasured · 1,000 → 100,000 segments
FleetRetained heap (cap 1024)Peak RSSDiscovery scan
1,000 segments13.0 MiB83.7 MiB102.7 ms
10,000 segments12.3 MiB181.5 MiB1,035.2 ms
100,000 segments12.4 MiB185.3 MiB21,825.6 ms

Across a 100× larger fleet, retained heap stayed inside a 0.7 MiB band. Intersection of two 2,000,000-id segments (2,000 chunks each, 100 shared) fetched only 100 of the 2,000 chunks per segment, in 27.8 ms against the in-memory drivers. The run's range requests carried 403,200 bytes, against the 8,104,225 bytes of the two stored objects (4,052,112 and 4,052,113): 5.0% (derived), a floor, because the counter leaves out each reader's first read, the object's last 262,144 bytes. With those two reads the run read about 927,488 bytes, about 11.4% (derived). Fleet seeded through store.load() at ~45–63 durable segments/s (fsync-bound). Measured on Apple M3 Pro (arm64, node v24.22.1, commit df765485, 2026-11-18, load 2.1 at the start and 2.4 at the end) — discovery is filesystem-bound here, and its scan took 10.1x for 10x the segments, then 21.1x for 10x the segments; read the milliseconds as this run's.

The TypeScript codec against native

Measured on one Apple M3 Pro under Node v24.22.1 on 2026-10-22, through the store with a warm cache, 31 rounds a figure, each figure against an A/A control (the native codec twice), one codec per process for reads. Negative means the TypeScript codec was faster. The record is bench/ts-codec-comparison-results.json and the per-operation tables are on the benchmarks page in the repository. Wall-clock, one machine, recorded and not asserted.

Method

Real services

Real S3 in us-east-1 at on-demand rates. No emulator and no simulated latency. The harness rehearses its workload against MinIO for free, and a rehearsal's results file is kept out of git, so what gets published is a real-account run or nothing.

Metered at the wire

Request counts come from the AWS SDK layer, not from the library's metrics sink — which cannot see a PUT at all (it emits no storage.put event, a known observability gap), and reports only chunk reads as storage.get, not pointer or index reads. A PUT bills at 12.5× a GET, so an ingest-heavy workload priced without them is understated.

Gated, both directions

Every money figure on this page, the crossover rate and the calibration run's figures are checked by a build gate against the source that produced them — bench/results.json, docs/benchmarks.md, a calibration run's own results file, the soak's own results, the pricing profile itself — and the check also fails on a figure those sources cannot account for, so a number cannot improve without evidence either. The at-scale table is held to the benchmark's own results file too.

And the estimator itself

The modelled figures here come from estimateCost(), so its accuracy is the claim they rest on. For point reads, priced against the chunk GETs a metrics sink observed, its prediction must land on or above what the engine incurred: it never quotes fewer chunk reads than the engine makes. For the rest it counts what the engine is tested to send: each operand's pointer and index before an intersect's chunks, what store.load() adds to a load, and the pointer refresh for the segments you keep reading, on S3's request shape, in as many reader processes as you pass it. And it still quotes low where it cannot see: more hot segments than a reader keeps open, the index every reader opens again after each load, an intersect that outlives the pointer TTL, an index larger than the tail read on an encrypted segment or a row with no usable summary, a load that loses a publish race.

The in-region run's bill · measured, and expected where marked 2026-11-03 · us-east-1 · run 2026-11-03-56b81
Operation Per million Requests
Cold A ∩ B, 100 of 1,999 chunks shared, the median $2.40 6 GETs
The same, each pointer read once — expected $2.40 6 GETs
A segment's first single-part store.load() in the calibration run $11.20 2 PUT + 3 GET
A segment's first multipart store.load() in the calibration run $26.20 5 PUT-class + 3 GET

Measured request counts at list prices, and expected where marked: the second row is the code's prediction, which the run met exactly. The two load rows are a segment's first load: the requests are measured and the price is the list price; a steady load at keep: 12 is measured too, on the same run, with each kind in the benchmarks doc. All 40 cold intersects — each on a fresh store, with nothing cached — returned the planned ids, checked by count and by sum, and each fetched 100 of 1,999 chunks per segment, the chunks the two share, and no other, as one range. Each also made one tail read per segment: its last 256 KiB, which holds the footer and the index. The median took 103.85 ms from AWS CloudShell inside the region with a client of 128 sockets, a cold count() 25.43 ms, and a warm intersect 3.73 ms. PUT-class requests are the ones S3 bills at the PUT rate, listings included. The run's requests, teardown included, cost $0.0027964; there is no data-transfer charge inside the region. The run's report explains every figure.

What these numbers do not establish

Each of these is a real limit on what you should conclude.

Run it against your own rates.

The cost model is a shipped, pure function. Ours is one pricing profile; yours will move every number on this page, including which side of the crossover you are on.

Apache-2.0 · v0.27.0 · pre-1.0 · single maintainer

ReproduceNo cloud account
git clone https://github.com/cloudbitmaps/cloudbitmaps
pnpm install

# regenerates bench/results.json
pnpm bench