Skip to main content

Automatic Tracing

Daita provides zero-configuration automatic tracing for all operations. Every agent execution, LLM call, plugin usage, and decision is automatically tracked without any setup required.

What's Traced​

All operations are automatically traced:

  • Agent Execution - Task processing, timing, status, input/output
  • LLM Calls - Tokens used, cost, latency, model details
  • Plugin Operations - Database queries, API calls, file operations
  • Decisions - Reasoning chains, confidence scores, alternatives
  • Workflow Communication - Messages between agents
  • Lifecycle Events - Agent start/stop, errors, retries

Basic Usage​

Tracing happens automatically - no configuration needed:

from daita import SubstrateAgent

agent = SubstrateAgent(name="My Agent", llm_provider="openai", model="gpt-4")

# This is automatically traced
result = await agent.process("analyze", {"data": [1, 2, 3]})

# Get trace statistics
stats = agent.get_trace_stats()
print(stats)
# {
#     'total_operations': 1,
#     'success_rate': 1.0,
#     'avg_latency': 234.5,
#     'total_tokens': 150,
#     'total_cost': 0.0023
# }

Get Recent Operations​

View recent agent operations:

# Get last 10 operations
operations = agent.get_recent_operations(limit=10)

for op in operations:
    print(f"Operation: {op['operation_name']}")
    print(f"Status: {op['status']}")
    print(f"Duration: {op['duration']}ms")
    print(f"Timestamp: {op['timestamp']}")

Decision Tracing​

Agents automatically track decision-making with reasoning:

# Get recent decisions
decisions = agent.get_recent_decisions(limit=10)

for decision in decisions:
    print(f"Decision Type: {decision['decision_type']}")
    print(f"Confidence: {decision['confidence']}")
    print(f"Reasoning: {decision['reasoning']}")
    print(f"Chosen Option: {decision['chosen_option']}")

# Get decision statistics
decision_stats = agent.get_decision_stats()
print(f"Total decisions: {decision_stats['total_decisions']}")
print(f"Avg confidence: {decision_stats['avg_confidence']}")

LLM Call Tracking​

Every LLM call is automatically tracked:

from daita import SubstrateAgent

agent = SubstrateAgent(name="My Agent", llm_provider="openai", model="gpt-4")

# Process some tasks (LLM calls tracked automatically)
await agent.process("analyze", data1)
await agent.process("analyze", data2)

# Get token usage
usage = agent.get_token_usage()
print(f"Total tokens: {usage['total_tokens']}")
print(f"Prompt tokens: {usage['prompt_tokens']}")
print(f"Completion tokens: {usage['completion_tokens']}")
print(f"Total requests: {usage['requests']}")
print(f"Estimated cost: ${usage['estimated_cost']:.4f}")

Plugin Tracing​

Plugin operations are automatically traced:

from daita import SubstrateAgent
from daita.plugins import PostgreSQLPlugin
from daita.core.tools import tool

# Create plugin
db_plugin = PostgreSQLPlugin(host="localhost", database="mydb")

agent = SubstrateAgent(
    name="DB Agent",
    llm_provider="openai",
    model="gpt-4",
    tools=[db_plugin]  # Plugin tools automatically traced
)

await agent.start()

# This traces both the agent execution AND the database query
result = await agent.run("Get all users from the database")

# View all operations including plugin calls
operations = agent.get_recent_operations(limit=10)

Workflow Tracing​

Workflows automatically trace communication between agents:

from daita import SubstrateAgent
from daita.core.workflow import Workflow

# Create agents
agent1 = SubstrateAgent(name="Agent 1", llm_provider="openai", relay="channel1")
agent2 = SubstrateAgent(name="Agent 2", llm_provider="openai")

# Create workflow
workflow = Workflow("My Pipeline")
workflow.add_agent("agent1", agent1)
workflow.add_agent("agent2", agent2)
workflow.connect("agent1", "channel1", "agent2")

await workflow.start()

# Communication is automatically traced
await workflow.inject_data("agent1", {"data": "test"}, task="process")

# Get communication log
comm_log = workflow.get_communication_log()
for message in comm_log:
    print(f"From: {message['from_agent']}")
    print(f"To: {message['to_agent']}")
    print(f"Channel: {message['channel']}")
    print(f"Timestamp: {message['timestamp']}")

Trace Types​

Different operation types are tracked:

from daita.core.tracing import TraceType

# Available trace types:
# - TraceType.AGENT_EXECUTION: Agent task processing
# - TraceType.LLM_CALL: Language model calls
# - TraceType.PLUGIN_EXECUTION: Plugin/tool usage
# - TraceType.DECISION: Decision-making processes
# - TraceType.WORKFLOW_COMMUNICATION: Agent-to-agent messages
# - TraceType.AGENT_LIFECYCLE: Start/stop events

Manual Decision Recording​

Record custom decisions with reasoning:

from daita.core.decision_tracing import DecisionRecorder
from daita.core.tools import tool

@tool
async def classify_data(data: dict) -> dict:
    """Classify data with decision recording."""
    async with DecisionRecorder("classification") as decision:
        # Add reasoning steps
        decision.add_reasoning("Checked field X")
        decision.add_reasoning("Validated constraint Y")

        # Set confidence
        decision.set_confidence(0.85)

        # Record alternatives considered
        decision.add_alternative("option_a", 0.85)
        decision.add_alternative("option_b", 0.65)

        # Make decision
        chosen = "option_a"
        decision.set_chosen_option(chosen)

        return {"decision": chosen}

Trace Statistics​

Get comprehensive statistics:

stats = agent.get_trace_stats()

# Available statistics:
print(f"Total operations: {stats['total_operations']}")
print(f"Success rate: {stats['success_rate']:.2%}")
print(f"Average latency: {stats['avg_latency']}ms")
print(f"Total tokens: {stats['total_tokens']}")
print(f"Total cost: ${stats['total_cost']:.4f}")
print(f"LLM calls: {stats['llm_calls']}")
print(f"Plugin calls: {stats['plugin_calls']}")
print(f"Decisions made: {stats['decisions']}")

Performance Impact​

Tracing is designed to be lightweight:

  • In-memory storage - Recent 500 spans kept
  • Async reporting - Non-blocking dashboard uploads
  • Minimal overhead - Sub 1ms per operation
  • No external dependencies - Works offline

Privacy Considerations​

Traces may contain sensitive data:

  • Input/output data from operations
  • LLM prompts and responses
  • Database query results
  • Decision reasoning

Best practices:

  • Review what data is being traced
  • Use focus parameters to filter sensitive fields
  • Consider data retention policies
  • Sanitize data before processing if needed

Disable Tracing​

Tracing is always enabled, but you can limit data collection:

# Don't send to dashboard (local only)
# Simply don't set DAITA_API_KEY

# Limit trace history
from daita.core.tracing import trace_manager
# Internal API - traces auto-rotate after 500 operations