Monitoring Dashboard

Prerequisites: You’ve completed the Quickstart and understand the Context & Event Schema.

​

What You Get

• Evaluator-defined Recall on unlabeled traffic (flag queries with recall < 1.0)
• Precision (proxy): Ratio of supporting passages to total context (detects context bloat)
• F1 & nDCG derived from recall + precision
• P95 Latency tracking for evaluation time
• Trend charts to catch drift over time
• Environment filtering to compare prod vs staging vs dev
• Evaluator Accuracy (when ground truth is provided)

​

Enable Monitoring

from seer import SeerClient
import os

client = SeerClient()

def retrieve(query: str) -> list[dict]:
    # your real retriever here
    return [{"text": "...", "score": 0.9}]

def handle_request(query: str):
    context = retrieve(query)

    client.log(
        task=query,
        context=context,
        metadata={
            "env": os.getenv("APP_ENV", "prod"),
            "feature_flag": "retrieval-v1",
        },
        sample_rate=0.10,  # 10% of calls are evaluated
    )

import { SeerClient } from '@seer/sdk';

const client = new SeerClient();

function retrieve(query: string) {
    // your real retriever here
    return [{ text: "...", score: 0.9 }];
}

function handleRequest(query: string) {
    const context = retrieve(query);

    client.log({
        task: query,
        context: context,
        metadata: {
            env: process.env.APP_ENV ?? "prod",
            feature_flag: "retrieval-v1",
        },
        sample_rate: 0.10,  // 10% of calls are evaluated
    });
}

​

Sampling Guidance

​

Smart Sampling with Decorator

from seer import seer_trace
import os

@seer_trace(
    task_arg="query",
    context_from_return=True,
    metadata={"env": os.getenv("APP_ENV", "prod")},
    sample_rate=0.10,  # 10% sampling
)
def retrieve(query: str) -> list[dict]:
    return [{"text": "...", "score": 0.9}]

# Logs automatically when called
results = retrieve("user question...")

​

The Monitoring Dashboard

​

KPI Cards

​

Trend Charts

​

Latency Tab

​

Structure Tab

​

Filtering

• Environment: Select a specific env (prod, staging, dev)
• Period: 24 hours, 7 days, or 30 days

​

Cost Management

• Sampling keeps evaluation cost predictable. Start at 0.05 and tune up if needed.
• SDK batching minimizes request overhead (events are queued and sent in batches).
• Async mode (default) — logging never blocks your request path.

​

Estimated Costs

​

Privacy Considerations

• You control what you send. If passages are sensitive, include only what’s needed for evaluation.
• Use metadata to tag records with access boundaries (e.g., collection, tenant_id) for future filtering.
• Consider truncating or summarizing very long passages.

​

Example: Tier-Based Sampling

from seer import SeerClient

client = SeerClient()

def handle_request(query: str, user_tier: str):
    context = retrieve(query)

    # Higher sampling for enterprise customers
    rate = 0.25 if user_tier == "enterprise" else 0.05

    client.log(
        task=query,
        context=context,
        metadata={
            "env": "prod",
            "user_tier": user_tier,
            "feature_flag": "retrieval-v1",
        },
        sample_rate=rate,
    )

​

See Also

• Quickstart
• Context & Event Schema
• Metrics
• Change Testing (A/B)
• Alerting