Module 4 · Lecture 17

Redis & the Key-Value Model

The speed layer — why a second database earns its place, and how rich values turn a cache into an analytics engine.

⏱ 90 minutes 🎯 CLO3 🐳 Redis 7 in Docker

Where We Are

Your Atlas dashboard works. Now scale it.

Module 3 gave you 15,000 bookings in MongoDB, an aggregation pipeline for revenue by vehicle and failure rate by hour, and an Atlas Charts dashboard on top.

At 15,000 documents

Every dashboard tile fires a pipeline. It scans, groups, sorts, returns — in a few milliseconds. Nobody notices.

At 15,000,000

The same pipeline takes seconds. And you re-run it for every analyst who hits refresh.

Today's question: when is the right answer not "tune the query" but "add a different kind of database"?

The Argument Is Arithmetic

The latency budget

Operation	Latency	Relative	Meaning
RAM read	~100 ns	1×	Effectively instant
SSD read	~100 µs	~1,000×	Fast — but 1,000 RAM reads could have happened
DB round trip	~1–10 ms	~10,000–100,000×	Planning + disk + serialise + network
"Instant" to a human	~100 ms	—	Your entire page-load budget

Twelve tiles × 10 ms = 120 ms on data alone. Budget gone before a pixel is drawn.

Business translation: latency is conversion, analyst productivity, and whether anyone opens the dashboard twice.

Polyglot Persistence

Two databases, two jobs

🗄️ MongoDB — system of record

All 15,000 bookings, durably. Answers questions nobody anticipated. Optimised for completeness and flexibility; costs milliseconds.

⚡ Redis — speed layer

A few dozen pre-computed aggregates. Optimised for read latency; costs you the need to decide in advance what to track.

The property that makes this safe

Everything in Redis is derived. If the process dies, you lose nothing that cannot be rebuilt by replaying bdma.rides. That is why a volatile store is acceptable — and why it must never be the only copy.

You met "polyglot persistence" in Lecture 11 as a myth-buster. In Module 4 it becomes an architecture.

What It Is

RE·mote DI·ctionary S·erver

Released 2009 by Salvatore Sanfilippo — after an analytics workload proved too slow on MySQL. Three design decisions define it:

1. Lives in RAM

No buffer pool, no cache-miss path. Every read is a memory read. Ceiling = your RAM.

2. Single-threaded

Commands execute one at a time, in order, to completion. Deliberate, not a defect.

3. Values are structures

The server understands and manipulates the value in place. This is the whole point.

Why single-threaded is a feature

No locks, no contention, no cache-line ping-pong. And every command is atomic: INCR is one indivisible step, not read-add-write. 200 concurrent booking events can hit the same counter and the count is exactly right.

Modern Redis threads I/O and background saves — but command execution stays serial, and that is what you reason about.

The Crucial Difference

Not a cache — a data-structure server

	Memcached (plain cache)	Redis
Value model	Opaque blob of bytes	A typed structure the server understands
Add 1 to a counter	Fetch, parse, add, serialise, write back — and race	`INCR key` — one atomic server-side op
Keep a top-10	Fetch all, sort in the app, write back	`ZINCRBY` → `ZREVRANGE key 0 9`
Network cost	Whole value crosses the wire every change	Only the delta crosses the wire

Say it precisely: a key-value store whose values are rich structures manipulated in place on the server. That is why a "cache" can power a live dashboard.

Structure 1 of 5

String — counters and cache slots

SET    rides:last_loaded "2024-12-31T23:59:59Z"
INCR   rides:2024-03-14:total           # atomic, O(1), no race
INCRBYFLOAT revenue:2024-03-14 412.50   # booking_value is decimal
SETEX  dash:summary:cache 60 '{"total":15000}'   # self-invalidating

Business uses

Bookings today · running revenue · feature flags · cached report payloads.

Why it wins

INCR is atomic regardless of writer count, and SETEX makes staleness impossible past the TTL.

Up to 512 MB per value — but if you are near that, you have modelled something wrong.

Structures 2 & 3 of 5

Hash — an object · List — a feed

Hash

HSET booking:CNR3000142
     status "Completed"
     vehicle_type "Go Sedan"
HINCRBY dash:payment "UPI" 1
HGETALL dash:payment

One key, one round trip, one TTL — and a compact encoding for small hashes.

List

LPUSH  feed:recent "CNR3000142 …"
LTRIM  feed:recent 0 99
LRANGE feed:recent 0 9
RPOP   queue:driver_payouts

O(1) at the ends, not the middle. A queue or a recent-activity strip — never an array.

Memorise: LPUSH + LTRIM = a feed that can never grow without bound.

Structure 4 of 5

Set — uniqueness is free

SADD    customers:2024-03-14 "CID482910"
SADD    customers:2024-03-14 "CID482910"   # -> 0, already there
SCARD   customers:2024-03-14               # unique customers that day

SINTER  customers:2024-03-13 customers:2024-03-14   # returned both days
SUNION  customers:2024-03-13 customers:2024-03-14   # reach
SDIFF   customers:2024-03-13 customers:2024-03-14   # churn candidates

Set algebra executes inside the server — no data crosses the wire to be intersected in your application.

⚠️ The cost to watch

A Set stores every member in full. One million CID… IDs ≈ 60–80 MB per day, per key. Fine for two days. Ruinous for a year.

If you only need the count, there is something far cheaper — two slides away.

Structure 5 of 5

Sorted Set — the one SQL cannot cheaply copy

ZINCRBY   dash:vehicle:revenue 412.50 "Go Sedan"
ZREVRANGE dash:vehicle:revenue 0 2 WITHSCORES
# Premier Sedan 536855 · Go Sedan 491473 · Uber XL 431927

ZINCRBY   dash:pickup:top 1 "Noida Sector 62"
ZREVRANGE dash:pickup:top 0 4 WITHSCORES
# Noida Sector 62 1209 · IGI Airport 1183 · NDLS 1179
# Gurgaon Cyber City 1178 · Connaught Place 1139

ZRANK dash:pickup:top "Connaught Place"    # rank of one member, O(log N)

Why this matters most

"Top 10 by revenue" in SQL or a Mongo pipeline means sorting at query time, on every read. A ZSET maintains the order incrementally on write — so the read is a slice of an already-sorted structure.

Leaderboards · trending · top-N by any accumulating measure. But never rank by a rate without gating on volume — by failure rate 03:00 "wins" at 43.48%, on 46 bookings all year.

Cheat Sheet

Pick the structure from the question

Business question	Structure	Commands
How many bookings today?	String	`INCR` / `GET`
Full state of booking CNR3000142?	Hash	`HSET` / `HGETALL`
Last 10 bookings?	List	`LPUSH` / `LTRIM` / `LRANGE`
Which customers booked? Who returned?	Set	`SADD` / `SCARD` / `SINTER`
Top vehicle by revenue? Top 5 pickups?	Sorted Set	`ZINCRBY` / `ZREVRANGE`
How many uniques — at millions?	Bitmap / HLL	`SETBIT` · `PFADD`

The skill is not memorising commands. It is mapping a business question to a structure before you write a line of code.

The Headline

One bit per user. 1,000,000 users = 125 KB.

SETBIT   dash:actives:2024-03-14 482910 1    # CID482910 was active
GETBIT   dash:actives:2024-03-14 482910      # -> 1
BITCOUNT dash:actives:2024-03-14             # exact daily actives

125 KB

1,000,000 bits ÷ 8 = 125,000 bytes

Scale it out

365 daily keys ≈ 45 MB for a year. Ten million users ≈ 1.25 MB/day, ~456 MB/year. Numbers you can put in a budget.

The same million customers as a Set of CID… strings: 60–80 MB per day — hundreds of times larger.

A bitmap stores a yes/no fact about a known population, not the identifiers. CID482910 → offset 482910, free of charge.

The Key Idea for Lecture 18

Retention in two commands

# Customers active on BOTH 13 and 14 March
BITOP AND dash:retained:0313-0314 dash:actives:2024-03-13 dash:actives:2024-03-14
BITCOUNT  dash:retained:0313-0314

# Reach across either day
BITOP OR  dash:reach:0313-0314 dash:actives:2024-03-13 dash:actives:2024-03-14

What just happened

A bitwise AND over 125 KB of memory — the CPU handles 64 users per machine word. In MongoDB the same cohort is a $lookup and $group over millions of documents.

⚠️ The precondition, honestly

You need a dense, stable integer identity per user. UUID systems must keep an ID→offset map. Bitmap size is set by the largest offset. Never hash a UUID into an offset — collisions silently corrupt the count.

This is the idea the entire Lecture 18 dashboard is built on.

When You Cannot Assign Offsets

HyperLogLog — 12 KB, always

PFADD   dash:hll:2024-03-14 "CID482910" "CID991044"
PFCOUNT dash:hll:2024-03-14                      # ≈ distinct count
PFMERGE dash:hll:week:11 dash:hll:2024-03-11 dash:hll:2024-03-12   # union

	Set	Bitmap	HyperLogLog
Memory @ 1M	~60–80 MB	~125 KB	12 KB fixed
Accuracy	Exact	Exact	~0.81% std error
List members?	Yes	Yes	Never
Needs int IDs?	No	Yes	No
Combine	`SINTER`/`SUNION`	`BITOP` AND/OR/XOR	`PFMERGE` (union only)

The trade in business terms: "1,000,000 visitors" really means ~991,900–1,008,100. Fine for a board-pack chart. Unusable for billing.

Governance

No schema means the key names ARE the schema

<namespace>:<entity>:<identifier>:<attribute>

rides:2024-03-14:completed      dash:actives:2024-03-14
dash:vehicle:revenue            booking:CNR3000142

One separator. ISO dates, so keys sort chronologically. A prefix that names the owning feature. And the schema written down in a table — nobody can derive it from the server.

EXPIRE dash:actives:2024-03-14 7776000   # 90 days — set it at write time
TTL    dash:actives:2024-03-14           # -1 = no expiry, -2 = no key
CONFIG SET maxmemory-policy volatile-lru # evict LRU only among TTL'd keys

⚠️ Never `KEYS *` in production

O(N) over the whole keyspace, and single-threaded means it blocks every other client. A total outage caused by someone looking around. Use SCAN 0 MATCH dash:* COUNT 100.

Eviction policies

noeviction (default — writes fail) · allkeys-lru (pure cache) · volatile-lru (our pattern — aggregates protected, cached payloads expendable) · volatile-ttl (sessions).

A cache without an eviction policy is not a cache — it is an unbounded memory leak. The policy states which data you are willing to lose first.

Durability & Limits

RDB vs AOF — and three wrong answers

	RDB snapshot	AOF append-only log
Worst-case loss	Everything since last snapshot — minutes	~1 s (`everysec`); ~0 (`always`)
Write cost	~0, plus a fork spike	Continuous; `always` waits on `fsync`
Restart	Fast binary load	Slower log replay

❌ Wrong answer when…

It is the primary store for data you cannot lose; the dataset is larger than RAM (memory costs ~10× SSD); or you need complex ad-hoc queries.

✅ Right answer when…

The access pattern is known in advance, read volume is high, the latency budget is tight, and the data is derived or reconstructible.

Redis can lose the last few seconds of writes. Irrelevant for a speed layer. Disqualifying for a ledger. Same property — opposite verdict.

Lab · 12 minutes

Run it. Touch every structure.

docker run -d --name bdma-redis -p 6379:6379 redis:7-alpine
docker exec -it bdma-redis redis-cli
PING                                   # -> PONG

HINCRBY   dash:payment "UPI" 6755
ZINCRBY   dash:vehicle:revenue 536855 "Premier Sedan"
ZREVRANGE dash:vehicle:revenue 0 2 WITHSCORES

SETBIT    dash:actives:2024-03-13 482910 1
SETBIT    dash:actives:2024-03-14 482910 1
SETBIT    dash:actives:2024-03-14 991044 1
BITOP AND dash:retained dash:actives:2024-03-13 dash:actives:2024-03-14
BITCOUNT  dash:retained                # -> 1
MEMORY USAGE dash:actives:2024-03-14   # ≈ 124 KB — and it would be the
                                       # same at a million customers

Leave the container running. Lecture 18 writes into this exact server. No Docker? Redis Cloud free tier, 30 MB — credentials in .env, never committed.

Wrap-Up

Six things to carry forward

Polyglot persistence is an architecture. Mongo is the system of record; Redis holds derived answers — which is why losing it is survivable.
The case is arithmetic. RAM ~100 ns · SSD ~100 µs · DB round trip ~1–10 ms · human "instant" ~100 ms.
Single-threaded is a feature. No locks, no contention, every command atomic.
The values are data structures. Strings, Hashes, Lists, Sets — and Sorted Sets, which SQL cannot copy cheaply.
One bit per user. 1,000,000 ÷ 8 = 125 KB. BITOP AND + BITCOUNT = retention in two commands.
The key naming convention IS the schema. Colon namespaces, ISO dates, TTL at write time, explicit eviction policy, SCAN not KEYS.

Next — Lecture 18: build the live dashboard. Node writer, Express API, bitmaps for retention — and one reversal: compute on write, not on read.