Run Privacy-First Product Analytics Without Losing Insight

25 years running a digital marketing agency means I've had to solve this tension constantly -- clients want actionable data, but burning user trust to get it kills the long-term relationship you're trying to build.

The single choice that changed how we approach this: we shifted clients from passive tracking to CRM-led lead scoring. Instead of vacuuming up behavioral data across every touchpoint, we define a handful of meaningful engagement milestones -- did they open the email, did they request more info, did they convert -- and build the picture from those intentional signals only.

A practical example: one client wanted to understand which service pages were driving real interest. Rather than pixel-tracking every scroll, we used CRM categorization tied to form submissions and direct inquiries. That gave us clean audience segmentation without needing to shadow anyone's browsing session.

The mindset shift is this -- you don't need more data, you need better-defined data. Decide what a meaningful engagement looks like before you build your collection system, and you'll almost always find you need far fewer data points than you thought.

The shift for us was moving from collecting "everything we might need" to collecting only what supports a specific decision.

One choice that made a real difference was designing analytics around events and patterns instead of user-level tracking. Instead of storing detailed, persistent profiles of individual behavior, we focused on anonymized event streams and short-lived aggregation windows—basically understanding what's happening in the system without tying it deeply to who a specific person is over time.

For example, rather than tracking a full user journey indefinitely, we look at how cohorts behave within a defined context (a feature, a timeframe, a flow) and then discard or heavily aggregate the underlying data. That still gives us enough signal to answer questions like "where are people dropping off" or "what improves engagement," without building a long-term behavioral record on individuals.

What made this effective is that it forces clarity. If you can't justify why a piece of data is needed for a decision, you don't collect it. That reduces risk, simplifies compliance, and actually makes analytics easier to work with because there's less noise.

The key takeaway is that good product insight doesn't require maximum data—it requires intentional data.

The most valuable thing you can do to honor visitor privacy is to keep all data collection on-premise. While most data can be anonymized, there are times when you will want to collect personal data with consent as well. While you may be a good steward of this data, if it's shared across other platforms, it opens avenues for abuse and makes compliance with regional privacy laws, such as the GDPR, much more difficult.

As a command-line interface platform designed to work inside of Claude Code, we have minimal visibility. Everything analytics-wise is opt-in. We only have visibility into what network calls happened on our servers..

Agentic products make this harder. Outcomes are non-deterministic, they break constantly, and your quality depends on how the agent uses you. Your APIs can be fine and it still doesn't matter if the full experience doesn't work.

So we focus on errors and bad outcomes first. Users opt in, we detect failures, and we only collect the context on those. Less data means worse UX and slower iteration, but you have to pick. Agent experiences are frustrating enough that we get a sizable amount of feedback explicitly. What's hard to get this way is the near-misses, or "good enough" outcomes.

The trend I'm predicting: the sampling bias in agentic products will be more noticeable, and producs will end up working better for the types of people who opt into telemetry - typically less technical and less privacy-conscious.