Evaluation

#Evaluation Is a Pure Helper Layer

daita.evaluation aggregates caller-retained observer events and caller-supplied human labels. It does not run an evaluation suite, call a judge model, persist datasets, or own report storage.

This separation keeps benchmark prompts, expected answers, labels, and artifacts under application control.

#Measure One Run

from daita import Agent, AgentEvent
from daita.evaluation import measure_observer_events
 
events: list[AgentEvent] = []
 
agent = await Agent.open("atlas", observer=events.append)
try:
    result = await agent.run("Summarize revenue by region")
finally:
    await agent.close()
 
measurement = measure_observer_events(events)

RunMeasurement contains content-free operational counts such as model calls, tool calls, catalog discovery, failed and corrected SQL calls, duration, token usage, approval outcomes, learning proposals, and estimated cost when cost information is complete.

Keep each run's events separate before measurement. The helper validates a bounded event collection and does not infer answer correctness from lifecycle events.

#Add Human Judgments

from daita.evaluation import (
    BenchmarkJudgment,
    BenchmarkOutcome,
    BenchmarkVariant,
)
 
baseline = BenchmarkOutcome(
    case_id="regional-revenue",
    variant=BenchmarkVariant.BASELINE,
    judgment=BenchmarkJudgment(
        answer_correct=True,
        source_selection_correct=True,
        resource_selection_correct=True,
    ),
    measurement=baseline_measurement,
)
 
learned = BenchmarkOutcome(
    case_id="regional-revenue",
    variant=BenchmarkVariant.LEARNED,
    judgment=BenchmarkJudgment(
        answer_correct=True,
        business_definition_correct=True,
        stale_activation_count=0,
        conflicting_claim_selection_count=0,
        cross_source_leakage_count=0,
    ),
    measurement=learned_measurement,
)

Human judgments can cover answer correctness, business definitions, source/resource/field selection, semantic constraints, recalled meaning, skill relevance, and explicit safety counts.

#Compare Baseline and Learned Runs

from daita.evaluation import build_learning_effectiveness_report
 
report = build_learning_effectiveness_report([baseline, learned])
 
print(report.to_markdown())
machine_readable = report.to_mapping()

Each case must have exactly one baseline and one learned outcome. The deterministic verdict can report improved correctness, improved efficiency, regression, no measured change, or unsafe learned behavior.

Hard safety requires zero stale activations, conflicting claims selected, and cross-source leakage. Stored memory or skill counts are not themselves evidence of improvement.

#Candidate Review Measurements

CandidateReviewMeasurement and CandidateReviewReport summarize the separate learning-review process: proposals, decisions, false positives, duplicates suppressed, reviewer calls, token and cost totals, and safety counts.

These records are also caller-owned. Populate them from your review workflow and retain the rendered or structured report wherever your application stores benchmark results.