AI observability (beta)

What it is

Every call your product makes to a language model becomes an event in OpenPanel: which model, how many tokens, what it cost, how long it took, whether it failed, and, if you want, the messages in and out. Calls are grouped into traces (one request) and optionally conversations (one thread).

What sets it apart from a standalone LLM tool is that the event lands next to everything else you track. An AI call belongs to a profile, happens inside a session and can carry groups, so you can answer questions like:

• What does our AI feature cost per user, per plan or per customer account?
• Which model or feature burns the most tokens, and where does latency spike?
• Do users who hit the assistant convert more often? That is a funnel with an ai_generation step.
• What did this user see on the page right before and after the model call? Open the session.

What you get

• AI section in the dashboard with an overview (cost, generations, users, cost per user, tokens, p95 latency, error rate, all with previous-period comparison), cost by model, feature, provider, user and group, a traces list with a timeline per trace, and a generations list with the conversation of each call.
• Reports: AI events are ordinary events named ai_generation, ai_embedding, ai_span and ai_trace with flat ai_* properties, so the report builder, funnels, retention, cohorts and dashboards work on them. Sum ai_cost by ai_model, or take the p95 of ai_latency_ms by ai_name.
• Profiles, sessions and groups show an AI usage widget with lifetime cost, tokens, latency and errors.
• Cost is computed for you from a built-in price catalog covering the major providers. Send your own cost when a gateway already knows it.

How to send events

Privacy

Prompts, completions and tool definitions are stored for 30 days, then dropped; tokens, cost and latency are kept with your other events. Set privacyMode: true on a call to never store its content, or turn content storage off for the whole project under Settings → AI.