Skip to content

📊 User Analytics

📋 Overview

This document outlines the architectural and design decisions behind our User Analytics (a.k.a. Signals) capture system. The goal of this system is to accurately measure user intent, confidence lift, and goal completion rates without compromising product performance, engineering velocity, or user privacy.

🎯 Core Philosophy

  • Privacy by Design: Analytics are tied to anonymous session IDs and canonical User IDs. No PII is logged in the analytics payload.
  • Storage over Compute: We favor dumping raw data into cheap, infinitely scalable storage over writing to a transactional database.
  • Asynchronous Processing: Aggregation happens out-of-band to decouple the read (dashboard) and write (survey) pathways.
  • Stateless Dashboards: We avoid heavy BI tools in favor of simple, static, self-sufficient HTML reports.

🏗️ Architectural Decisions

1. 📤 Fire-and-Forget Frontend Implementation

Decision: The frontend client (s3Client.ts) wraps all signal API calls in a fire-and-forget try/catch block that explicitly swallows errors.

Why: Analytics are secondary to the core product experience. If the backend is down, if a network request times out, or if the payload is malformed, the user's session must continue uninterrupted. We trade absolute data completeness for absolute application stability.

2. 🪣 S3 Data Lake over Transactional DB

Decision: The backend (POST /api/signals) writes survey payloads directly to S3 (visualli-feedback/${ENV}/signals/raw/) as individual JSON files rather than inserting rows into MongoDB.

Why: - Scale: S3 handles virtually infinite concurrent writes without connection pooling issues or database locking. - Simplicity: No schema migrations are required when survey questions change. - Isolation: Analytics traffic cannot accidentally spike database CPU and affect core application latency.

3. ⏱️ Asynchronous Nightly Aggregation

Decision: Instead of calculating statistics on-the-fly, an out-of-band Lambda function (cron/aggregateSignals.js) runs on an EventBridge schedule to process the raw JSON files into a single summary.json.

Why: Calculating average confidence lift, experience scores, and completion rates requires merging pre-session and post-session records. Doing this on the fly for every dashboard view would be computationally expensive and slow. Pre-calculating this nightly ensures the dashboard loads instantly.

4. 🌐 Serverless & Static Dashboarding

Decision: The analytics dashboard (user-analytics/signals/index.html) is a static HTML file that fetches summary.json via relative paths. It is deployed to Cloudflare R2.

Why: - Zero Maintenance: No Metabase, Tableau, or custom React dashboard application to maintain. - Cost: Cloudflare R2 serves the HTML and JSON for fractions of a cent with zero compute overhead. - Self-Sufficiency: By using relative paths (./summary.json), the exact same HTML file works locally (file:/// with a local server) and in production without complex environment variable injection.

5. 🔗 Stage-Isolated Environments

Decision: All signals and aggregations are strictly separated by deployment stage (e.g., dev, alpha, beta) using S3 prefixes (${ENV}/signals/...).

Why: We must prevent local developer testing or alpha user feedback from polluting production business metrics. This aligns with our GTM tracking strategy outlined in Behavioral Insights.

🔍 Data Model & Correlation

To successfully calculate "Confidence Lift", we must link a user's intent before using Visualli with their outcome after using it.

The Correlation Key

We generate a deterministic Signal ID format: <visualli-user-id>-<session-id> - visualli-user-id: The canonical usr_ULID (stripped of prefix) to ensure cross-stack consistency. - session-id: The Clerk session ID to group multiple workflows in a single sitting.

The Merge Logic

The nightly aggregator performs a left-join of pre-signals and post-signals on this correlation key. - Pre + Post: Calculates true confidence lift and goal completion. - Pre + No Post: Applies a "Penalty" score (user abandoned the workflow). - Post + No Pre: Still captures the final experience score and free-text feedback.

✨ Key Non-Negotiables For Future Changes

  1. Never block the main thread or await a signal response before allowing the user to proceed.
  2. Never store PII (emails, names) in the signals payload. Rely strictly on the canonical usr_ULID.
  3. Never add a database dependency to the POST /api/signals route. S3 PutObject is the only acceptable destination.
  4. Never mix environments. Always ensure the ENV prefix is respected in the S3 bucket path.