Building a Real-Time Analytics Dashboard with Redis

A build session — compute on write, read in O(1), and count a million customers with a bitwise AND

Learning Objectives

• ✓ Describe the four-stage architecture of a real-time dashboard — event source, writer, read API, browser — and the responsibility of each stage
• ✓ Contrast compute-on-read (MongoDB aggregation) with compute-on-write (Redis counters) and state when each is the correct choice
• ✓ Design and document a complete key schema mapping key pattern → type → purpose for a stated set of business metrics
• ✓ Write a Node.js ingest function that updates eight Redis structures atomically with MULTI/EXEC, and defend every guard against absent fields
• ✓ Build an Express endpoint that assembles a full dashboard payload with MGET, ZREVRANGE … WITHSCORES, BITCOUNT and HGETALL
• ✓ Compute day-over-day retention with BITOP AND plus BITCOUNT and justify the memory cost at a million customers
• ✓ Evaluate the Redis dashboard against the Atlas Charts dashboard on freshness, query cost, flexibility, memory, durability and fit

Session Timeline

Section 1: The Architecture of a Live Dashboard

We are building the thing a ride-hailing operations room actually stares at: total bookings, revenue so far today, the status mix, revenue by vehicle type, the busiest pickup points, the hourly curve, the payment split, and how many distinct customers have booked today — all refreshing every two seconds.

The model is the Redis Labs reference project basic-analytics-dashboard-redis-bitmaps-nodejs, rebuilt against our own ride-booking domain so it connects to everything you have already modelled.

Four Stages

Section 2: Compute on Write, Not on Read

This is the central idea of the lecture, and it is a direct reversal of what you did in Module 3.

What You Did in Lectures 13–16

// MongoDB — the work happens when somebody asks
db.rides.aggregate([
  { $match: { status: "Completed" } },
  { $group: { _id: "$vehicle_type", revenue: { $sum: "$booking_value" } } },
  { $sort:  { revenue: -1 } },
  { $limit: 5 }
])

Beautiful, flexible, and re-executed in full every single time the tile renders. The cost is proportional to the number of matching documents, so it grows as the business grows — precisely backwards from what you want.

What You Will Do Today

// Redis — the work happens when the booking arrives
ZINCRBY dash:vehicle:revenue 412.50 "Go Sedan"     // once, on write

// ... and the dashboard read, forever after:
ZREVRANGE dash:vehicle:revenue 0 4 WITHSCORES      // O(log N + 5)

The aggregate is maintained incrementally. A read is a slice of an already-sorted structure, and its cost does not depend on whether you have processed 15,000 bookings or 15,000,000.

Section 3: The Redis Key Schema

Redis has no catalogue, so this table is the schema. Write it before you write code; keep it in the repository next to the writer.

Three Decisions Embedded in That Table

1. Status values are slugged, not raw

"Cancelled by Driver" becomes dash:status:cancelled-by-driver. Spaces in key names are legal but a constant nuisance in redis-cli, in logs, and in shell scripts. Normalise once, in one helper, and never think about it again.

2. Customer offsets come from the ID

Our IDs are CID482910. Strip the three-character prefix and the remainder is a dense integer offset, which is exactly what a bitmap needs. This is a gift of the dataset design — a real system with UUID customer keys would need a separate customer:offset Hash and an INCR-based allocator.

3. Which keys get a TTL

Date-scoped keys (dash:customers:*, dash:revenue:*) get 90 days. Cumulative keys (dash:total:bookings, the two ZSETs, dash:payment) get none — they are the dashboard's permanent state. Combined with maxmemory-policy volatile-lru from Lecture 17, that means pressure evicts old daily keys first and never touches the live aggregates.

Section 4: The Writer — Ingesting Bookings

Project Setup

mkdir bdma-redis-dashboard && cd bdma-redis-dashboard
npm init -y
npm pkg set type=module          # enables import syntax and top-level await
npm install redis express
docker start bdma-redis          # the container from Lecture 17

keys.js — The Schema as Code

Never build a key string inline. One module owns every key pattern, so a rename is one edit and a typo is impossible.

// keys.js
export const slug = (s) => s.toLowerCase().replace(/\s+/g, '-');

// "CID482910" -> 482910. Dense, stable, and exactly what SETBIT needs.
export const customerOffset = (id) => Number.parseInt(id.slice(3), 10);

export const K = {
  totalBookings:          'dash:total:bookings',
  vehicleRevenue:         'dash:vehicle:revenue',
  pickupTop:              'dash:pickup:top',
  payment:                'dash:payment',
  status:    (s)    => `dash:status:${slug(s)}`,
  revenue:   (date) => `dash:revenue:${date}`,
  hour:      (h)    => `dash:hour:${h}`,
  customers: (date) => `dash:customers:${date}`,
};

writer.js — One Booking, One Transaction

// writer.js
import { createClient } from 'redis';
import { K, customerOffset } from './keys.js';

export const redis = createClient({
  url: process.env.REDIS_URL ?? 'redis://localhost:6379'
});
redis.on('error', (err) => console.error('[redis]', err.message));
await redis.connect();

const NINETY_DAYS = 60 * 60 * 24 * 90;

// Queues one booking's updates onto an existing MULTI. Pure: no awaits,
// so the same function serves a single write and a batched pipeline.
function queueBooking(tx, b) {
  const day  = b.date;                              // "2024-03-14"
  const hour = new Date(b.booked_at).getUTCHours(); // 18

  // --- always present on every document ---
  tx.incr(K.totalBookings);
  tx.incr(K.status(b.status));
  tx.incr(K.hour(hour));
  tx.zIncrBy(K.pickupTop, 1, b.route.pickup);
  tx.setBit(K.customers(day), customerOffset(b.customer_id), 1);

  // NX = only set the TTL if the key has none, so a day's window is not
  // pushed forward by every booking that lands in it.
  tx.expire(K.customers(day), NINETY_DAYS, 'NX');
  tx.expire(K.revenue(day),  NINETY_DAYS, 'NX');

  // --- absent on Cancelled / No Driver Found (5,657 of 15,000 docs) ---
  if (typeof b.booking_value === 'number') {
    tx.incrByFloat(K.revenue(day), b.booking_value);
    tx.zIncrBy(K.vehicleRevenue, b.booking_value, b.vehicle_type);
  }

  // --- absent unless money actually changed hands ---
  if (b.payment_method) {
    tx.hIncrBy(K.payment, b.payment_method, 1);
  }
}

export async function recordBooking(booking) {
  const tx = redis.multi();
  queueBooking(tx, booking);
  return tx.exec();          // all ~8 commands, atomically
}

export async function recordBatch(bookings) {
  const tx = redis.multi();
  for (const b of bookings) queueBooking(tx, b);
  return tx.exec();          // one round trip for the whole batch
}

Why MULTI/EXEC

replay.js — Backfilling the Course Dataset

Load the 15,000 generated bookings to populate the dashboard, then stream the tail slowly so the page visibly moves during class.

// replay.js
import { readFile } from 'node:fs/promises';
import { recordBatch, recordBooking, redis } from './writer.js';

const bookings = JSON.parse(
  await readFile(new URL('../data/uber_bookings.json', import.meta.url), 'utf8')
);
bookings.sort((a, b) => a.booked_at.localeCompare(b.booked_at));

// 1. Backfill everything except the last 200, in batches of 500.
const history = bookings.slice(0, -200);
for (let i = 0; i < history.length; i += 500) {
  await recordBatch(history.slice(i, i + 500));
}
console.log(`backfilled ${history.length} bookings`);

// 2. Dribble the remainder in so the dashboard moves while you talk.
for (const booking of bookings.slice(-200)) {
  await recordBooking(booking);
  await new Promise((r) => setTimeout(r, 400));
}
await redis.quit();

import { MongoClient } from 'mongodb';
const mongo = await MongoClient.connect(process.env.MONGO_URL);
const cursor = mongo.db('bdma').collection('rides')
                    .find({}, { sort: { booked_at: 1 } });

let batch = [];
for await (const booking of cursor) {
  batch.push(booking);
  if (batch.length === 500) { await recordBatch(batch); batch = []; }
}
if (batch.length) await recordBatch(batch);

After the backfill, check your work against the verified figures from Module 3:

GET dash:total:bookings                     # -> 15000
GET dash:status:completed                   # -> 9343   (62.29%)
GET dash:status:cancelled-by-driver         # -> 2698   (17.99%)
GET dash:status:no-driver-found             # -> 1036    (6.91%)
GET dash:status:cancelled-by-customer       # -> 1022    (6.81%)
GET dash:status:incomplete                  # ->  901    (6.01%)
ZREVRANGE dash:vehicle:revenue 0 2 WITHSCORES
# Premier Sedan 536855 · Go Sedan 491473 · Uber XL 431927
HGETALL dash:payment                        # UPI 45.03%, Cash 25.31% ...
GET dash:hour:18                            # -> 1373, the busiest hour

Section 5: The Reader — An API over the Aggregates

Note what this handler does not contain: no grouping, no sorting, no filtering, no scanning. Every number it returns already exists in memory. Its only job is to fetch and shape.

// api.js
import express from 'express';
import { redis } from './writer.js';
import { K } from './keys.js';

const app = express();
const STATUSES = [
  'Completed', 'Cancelled by Driver', 'No Driver Found',
  'Cancelled by Customer', 'Incomplete'
];
const HOURS = Array.from({ length: 24 }, (_, h) => h);
const num = (v) => Number(v ?? 0);

app.get('/api/dashboard', async (req, res, next) => {
  try {
    const day = req.query.date ?? '2024-03-14';

    const [headline, statusCounts, hourCounts, vehicles, pickups, payment, actives] =
      await Promise.all([
        redis.mGet([K.totalBookings, K.revenue(day)]),
        redis.mGet(STATUSES.map((s) => K.status(s))),
        redis.mGet(HOURS.map((h) => K.hour(h))),
        redis.zRangeWithScores(K.vehicleRevenue, 0, 4, { REV: true }),
        redis.zRangeWithScores(K.pickupTop,      0, 4, { REV: true }),
        redis.hGetAll(K.payment),
        redis.bitCount(K.customers(day)),
      ]);

    res.json({
      date: day,
      total_bookings:    num(headline[0]),
      revenue_today:     Number(num(headline[1]).toFixed(2)),
      active_customers:  actives,
      status: Object.fromEntries(
        STATUSES.map((s, i) => [s, num(statusCounts[i])])
      ),
      hourly: HOURS.map((h) => ({ hour: h, bookings: num(hourCounts[h]) })),
      vehicle_revenue: vehicles.map(({ value, score }) =>
        ({ vehicle_type: value, revenue: score })),
      top_pickups: pickups.map(({ value, score }) =>
        ({ pickup: value, bookings: score })),
      payment_mix: payment,
      generated_at: new Date().toISOString(),
    });
  } catch (err) { next(err); }
});

app.use(express.static('public'));
app.listen(3000, () => console.log('dashboard on http://localhost:3000'));

Deriving a Rate — and the Gate It Needs

Every number in that payload is a count. The moment the business asks for a rate — "failure rate by hour", the chart you built in Module 3 — you hit the first real limit of compute-on-write: a ratio needs a numerator and a denominator, and Redis will only ever hand you the counters you decided to keep. dash:hour:{H} is the denominator. The numerator does not exist yet, so you add one counter and one line to the writer:

// keys.js — one addition
hourFail:  (h)    => `dash:hourfail:${h}`,

// writer.js — inside queueBooking, alongside tx.incr(K.hour(h))
if (b.status !== 'Completed') tx.incr(K.hourFail(h));

Then re-run replay.js, because a counter you add today starts at zero and knows nothing about the 15,000 bookings already in Mongo. That is the compute-on-write bargain in miniature, and Section 7 returns to it.

// api.js — the rate is computed on read, from two O(1) counters
const [hourTotals, hourFails] = await Promise.all([
  redis.mGet(HOURS.map((h) => K.hour(h))),
  redis.mGet(HOURS.map((h) => K.hourFail(h))),
]);

const MIN_BOOKINGS = 1000;                  // the gate — see below
const hourly = HOURS.map((h) => {
  const bookings = num(hourTotals[h]);
  const failed   = num(hourFails[h]);
  return {
    hour: h,
    bookings,
    failed,
    failure_rate: bookings ? Number((100 * failed / bookings).toFixed(2)) : null,
    ranked: bookings >= MIN_BOOKINGS,       // only these are comparable
  };
});

// Worst hour — but only among hours that carry volume
const worst = hourly
  .filter((x) => x.ranked)
  .sort((a, b) => b.failure_rate - a.failure_rate)[0];

That filter is the whole lesson. Without it, the dashboard ranks by rate alone and reports the wrong hour with total confidence:

The Browser View

A static page in public/index.html. Poll, repaint, nothing clever.

<div id="total"></div>
<div id="revenue"></div>
<div id="actives"></div>
<ol id="vehicles"></ol>
<ol id="pickups"></ol>

<script>
const inr = (n) => '₹' + Number(n).toLocaleString('en-IN',
                     { maximumFractionDigits: 0 });

async function refresh() {
  const res = await fetch('/api/dashboard?date=2024-03-14');
  if (!res.ok) return;                       // keep the last good paint
  const d = await res.json();

  total.textContent    = d.total_bookings.toLocaleString('en-IN');
  revenue.textContent  = inr(d.revenue_today);
  actives.textContent  = d.active_customers.toLocaleString('en-IN');

  vehicles.innerHTML = d.vehicle_revenue
    .map((v) => `<li>${v.vehicle_type} — ${inr(v.revenue)}</li>`).join('');
  pickups.innerHTML = d.top_pickups
    .map((p) => `<li>${p.pickup} — ${p.bookings}</li>`).join('');
}

refresh();
setInterval(refresh, 2000);
</script>

# Two terminals
node api.js                 # terminal 1 — http://localhost:3000
node replay.js              # terminal 2 — watch the numbers climb

curl -s localhost:3000/api/dashboard?date=2024-03-14 | head -c 400

Section 6: Retention with Bitmaps

Everything so far could have been done with ordinary counters. This is the part that could not.

The business question: of the customers who booked on 13 March, how many came back on the 14th? Day-over-day retention is one of the few metrics that genuinely predicts the health of a marketplace, and it is expensive to compute the obvious way — a set of yesterday's customers, a set of today's, and an intersection over millions of identifiers.

# Two commands. That is the entire computation.
BITOP AND dash:retained:2024-03-13:2024-03-14 \
          dash:customers:2024-03-13 dash:customers:2024-03-14
BITCOUNT  dash:retained:2024-03-13:2024-03-14

// api.js — exposed as an endpoint
app.get('/api/retention', async (req, res, next) => {
  try {
    const { from, to } = req.query;            // ?from=2024-03-13&to=2024-03-14
    const dest = `dash:retained:${from}:${to}`;

    await redis.bitOp('AND', dest, [K.customers(from), K.customers(to)]);
    await redis.expire(dest, 300);             // a derived scratch key

    const [returning, dayOne, dayTwo] = await Promise.all([
      redis.bitCount(dest),
      redis.bitCount(K.customers(from)),
      redis.bitCount(K.customers(to)),
    ]);

    res.json({
      from, to,
      day_one_actives: dayOne,
      day_two_actives: dayTwo,
      returning,
      retention_pct: dayOne ? Number((returning / dayOne * 100).toFixed(2)) : 0,
      new_customers: dayTwo - returning,
    });
  } catch (err) { next(err); }
});

The Memory Argument, Concretely

Measure the same thing the other ways and the comparison is stark:

Section 7: Redis vs Atlas Charts — and the Limits

You have now built the same dashboard twice. Same business questions, opposite engineering trade. Being able to articulate this table is the examinable skill of Module 4.

About the Reference Project

The Redis Labs project this session is modelled on — redis-developer/basic-analytics-dashboard-redis-bitmaps-nodejs — is worth reading in full, and worth reading honestly.

Its web-analytics framing is the canonical use case — it is, after all, what Google Analytics does. Our substitution of bookings for page views changes nothing structural, which is the point: the pattern transfers across domains unchanged.

The Limit You Just Bought

Key Takeaways