Building a Privacy-First, Serverless Analytics Engine with Next.js and Redis

When it comes to website analytics, the default choice is often Google Analytics or a paid privacy-focused alternative. I decided to build a custom solution to better understand the infrastructure requirements of real-time data processing and to maintain full control over the data lifecycle.

Architecture Overview

The system followed a decoupled architecture where the tracking logic resided at the edge, and the dashboard functioned as a consumer of aggregated Redis data.

graph TD
    User([User Visit]) --> Buffer[Client-side Event Buffer]
    Buffer --> Edge[Next.js API /api/track - Batched]
    Edge --> Aggregator[Server-side Aggregator]
    Aggregator --> Redis{Upstash Redis}
    
    subgraph "Data Structures"
    Redis --> Events[List: visits:events - Rolling Buffer]
    Redis --> HLL[HyperLogLog: unique:visits - Cardinality]
    Redis --> Daily[Hash: visits:daily:date - Aggregates]
    end
    
    Dashboard[Next.js Dashboard] --> Redis

Architecture Decision Record (ADR)

To maintain clarity on design choices, I documented the following key decisions:

Feature	Choice	Rationale
Hot Path Storage	Redis	Required sub-millisecond writes for tracking and low-latency reads for the dashboard.
Counting Unique IDs	HyperLogLog	Traditional sets grow linearly with traffic; HLL maintains a constant 12KB footprint.
Data Retention	Rolling Buffer	Storing every raw event indefinitely is cost-prohibitive. A fixed-size List preserves recent context.
Privacy	Salted Hashing	Ensures PII (IP addresses) is never stored, making the system GDPR-compliant by design.
Write Strategy	Buffer & Flush	Batched events from the client (10s intervals) to minimize Redis request counts and avoid rate limits.
List Trimming	Daily Cron	Running LTRIM per request wastes 2 commands per batch. A Vercel Cron job trims the event list once per day instead.

Technical Challenges & Evolutions

Building this system from scratch revealed several non-obvious hurdles that required significant architectural pivots:

1. The "Request Price" Challenge:

   • Problem: Each visitor interaction (scroll, click, pageview) originally triggered individual Redis commands. In a serverless/Upstash environment, this consumed the free tier quota rapidly.
   • Fix: Transitioned to a Client-side Event Buffer. By grouping events and sending them in "micro-batches" every 10 seconds, Redis command volume was reduced by ~80% without losing data fidelity.

2. Type-Conflict & Migration Errors:

   • Problem: Moving from standard Sets (for unique visitor IDs) to HyperLogLog caused WRONGTYPE errors on existing keys, leading to persistent 500 API failures.
   • Fix: I cleared the legacy keys and implemented a Log-based Backfill. Since the system maintains a rolling buffer of raw events, I was able to re-process historical traffic and reconstruct the new HLL structures safely.

3. The Pipeline Limit:

   • Problem: When fetching data for the dashboard, a single massive Redis Pipeline would sometimes exceed REST API payload limits or timeout.
   • Fix: Implemented Chunked Pipeline Execution, splitting large data fetches into smaller batches of 500 commands to ensure stability.

4. The Free-Tier Command Budget (416K / 500K per month):

   • Problem: After adding session tracking, heatmaps, and geo rollups, I hit 83% of Upstash's 500K/month free-tier command limit — and the month wasn't over. The write-to-read ratio was ~5:1, meaning the tracking endpoint was burning commands, not the dashboard reads. Profiling the pipeline revealed four culprits:
     1. Two out-of-pipeline await redis.set(...{nx:true}) calls per request (live round-trips that blocked the pipeline from batching them).
     2. Two separate SET commands per event for last_page and last_activity.
     3. Three separate commands per request for visitor hash updates (HSET, HSETNX, EXPIRE).
     4. An LTRIM on the visits:events list fired on every single tracking batch.
   • Fix — In-memory session dedup: Replaced the two live awaited SET NX calls with a module-level Map cache. Warm Lambda instances reuse the cache; cold starts fall back to a pipelined SET NX that is batched with all other writes. Net saving: ~2 commands per request.
   • Fix — HSET collapse: Merged SET last_page + SET last_activity into a single HSET on the existing session hash. Net saving: ~1 command per event in each batch.
   • Fix — Visitor hash merge: Collapsed HSET + HSETNX + EXPIRE for the visitor record into two commands by handling firstSeen through the same session-new guard that was already present.
   • Fix — Cron-based LTRIM: Moved LTRIM visits:events from the hot path to a Vercel Cron job that fires once daily at 3am UTC. Net saving: ~1 command per batch.
   • Total estimated saving: ~50K commands/month, bringing the budget back to a comfortable ~360K.

---

Step-by-Step Implementation Guide

If you're looking to build something similar, here's how to structure the core components of the "Buffer & Flush" analytics engine.

Step 1: Client-side Event Buffering

Most tracking scripts send an HTTP request immediately for every click or scroll. This is expensive and slow. Instead, use a memory buffer in your _app.js or a custom React Hook.

let eventBuffer = [];
let flushTimeout = null;

function trackEvent(pathname, data = {}) {
  eventBuffer.push({ pathname, timestamp: new Date().toISOString(), ...data });

  // Flush immediately for pageviews, buffer others for 10s
  if (data.type === 'pageview') {
    flushEvents();
  } else if (!flushTimeout) {
    flushTimeout = setTimeout(flushEvents, 10000);
  }
}

function flushEvents() {
  if (eventBuffer.length === 0) return;
  const payload = JSON.stringify(eventBuffer);
  eventBuffer = []; // Clear buffer BEFORE sending to prevent duplicates

  // Use sendBeacon for more reliability on page exit
  if (navigator.sendBeacon) {
    navigator.sendBeacon('/api/track', new Blob([payload], { type: 'application/json' }));
  } else {
    fetch('/api/track', { method: 'POST', body: payload, keepalive: true });
  }
}

Step 2: Server-side Command Aggregation

On the server, your /api/track endpoint shouldn't just dump events into a database. It should aggregate them first. If one user clicks three links in a 10-second window, you should send one command to Redis, not three.

// pages/api/track.js
const sessionCache = new Map(); // Simple in-memory cache for warm Lambdas

export default async function handler(req, res) {
  const events = Array.isArray(req.body) ? req.body : [req.body];
  const pipe = redis.pipeline();
  
  // Use a map to aggregate counters locally before pipelining
  const localPageCounters = {};
  
  for (const event of events) {
    const { pathname, sessionId } = event;
    const isNewSession = !sessionCache.has(sessionId);
    
    if (isNewSession) {
      sessionCache.set(sessionId, true);
      pipe.set(`sessions:${sessionId}`, 'active', { nx: true, ex: 86400 });
    }

    localPageCounters[pathname] = (localPageCounters[pathname] || 0) + 1;
  }

  // Push all aggregated increments into the pipeline in one go
  Object.entries(localPageCounters).forEach(([path, count]) => {
    pipe.hincrby('visits:pages', path, count);
  });

  await pipe.exec();
}

Step 3: Fast Rollups with Redis

To keep the dashboard fast, I use Redis Daily Hashes for geo and referrer data. Instead of counting raw logs every time the dashboard loads, I "roll up" the data instantly during the write phase using HINCRBY.

Step 4: The Dashboard

The final dashboard consumes these aggregated keys using a single Redis pipeline, rendering metrics in milliseconds without any complex SQL joins or heavy query engines.

---

Dashboard Metrics

The dashboard provides a breakdown of traffic across several dimensions:

• Geography: Visit distribution by country and city.
• Referrer: Traffic sources (e.g., social media, GitHub, or search).
• Device: Distribution between mobile and desktop users.

Inspirations & Similar Approaches

This architecture drew inspiration from several established patterns in the web analytics community:

• Plausible Analytics: Their focus on simplicity and privacy over deep behavioral tracking served as a primary model for this project.
• Tinybird (Real-time Analytics): Their approach to using ingestion pipelines informed how I structured the Redis rollups.
• Redis "Fast Counter" Pattern: Using HINCRBY for atomic daily aggregates is a well-documented strategy for high-performance dashboards.

Privacy Considerations

IP addresses were hashed with a rotating salt, meaning the original data cannot be recovered from the database. This approach ensures that privacy is built into the architecture from the start.

References & Further Reading

• Redis HyperLogLog Documentation - Technical details on cardinality estimation.
• Upstash Redis Docs - Getting started with serverless Redis on Upstash.
• GDPR and IP Anonymization - Best practices for privacy-first tracking.
• Next.js Middleware - Exploring edge-side processing for lower latency.