Evaluation

​

Overview

• dazense validate for config consistency
• dazense eval for governance test cases
• dazense eval --engine for policy-engine based checks

​

dazense test

​

Create unit tests

your-project/
├── dazense_config.yaml
├── RULES.md
├── tests/                          # Test folder
│   ├── total_revenue.yml          # Test file 1
│   ├── customer_metrics.yml       # Test file 2
│   └── outputs/                   # Test results (auto-generated)
│       └── results_20250209_143022.json

name: total_revenue
prompt: What is the total revenue from all orders?
sql: |
  SELECT SUM(amount) as total_revenue
  FROM orders

• name: A descriptive name for the test
• prompt: The question or prompt to test
• sql: SQL query which produces the right data

​

Launch dazense test command

• Start the dazense chat server (for example with dazense chat or your usual local setup) so that the backend API is available.
• On the first dazense test run, the CLI will prompt you to log in in your browser — use the same account you use in the local dazense chat interface, so tests run under the same project and permissions.

dazense test

• Discover all .yml and .yaml files in the tests/ folder
• Run each test against the default model (openai:gpt-4.1)
• Display results in a summary table
• Save detailed results to tests/outputs/results_TIMESTAMP.json

# Test with GPT 4.1
dazense test -m openai:gpt-4.1

# Run with 4 parallel threads
dazense test -t 4

​

How It Works

1. Test Discovery: Scans the tests/ folder for .yml and .yaml files
2. Test Execution: For each test:
   • Sends the prompt to your agent and runs it normally (the agent may execute SQL queries, search context, etc.)
   • Captures the agent’s full conversation history, tool calls, and response text
3. Data Verification:
   • Extract actual data: a prompt is sent to the LLM with the full conversation history, asking it to extract the final answer as structured data. The extraction prompt is:
     Based on your previous analysis, provide the final answer to the original question.
     Format the data with these columns: [expected columns]
     Return the data as an array of rows, where each row is an object with the column names as keys.
     If you cannot answer, set data to null.
     
   • The LLM responds with structured data formatted as a table matching the expected columns.
   • Execute expected SQL: the sql query from your test file is executed against your database to get the expected results
   • Compare data: the agent answer’s data and expected data (from SQL execution) are compared
4. Data Comparison Process:
   • Normalize datasets: Both datasets are converted to DataFrames and normalized (resets index, infers types, and sorts columns alphabetically)
   • Row count match: If row count doesn’t match, they are not compared
   • Exact match: First attempts exact equality comparison
   • Approximate match: For numeric columns, uses numpy’s allclose with tolerance (relative tolerance: 1e-5, absolute tolerance: 1e-8) to handle floating-point differences
   • Diff generation: If both comparisons fail, generates a detailed diff showing where values differ
5. Result Collection: Collects metrics including:
   • Pass/fail status of the data diff
   • Token usage and costs (inputs and outputs of the LLM)
   • Execution duration
   • Tool call count

​

Test Outputs

• Test name
• Model used
• Pass/fail status
• Message (e.g., “match”, “values differ”)
• Token usage
• Cost
• Execution time
• Tool call count
• A final summary with total passed/failed counts

{
  "timestamp": "2025-02-09T14:30:22.123456",
  "results": [
    {
      "name": "total_revenue",
      "model": "openai:gpt-4.1",
      "passed": true,
      "message": "match",
      "tokens": 1250,
      "cost": 0.0125,
      "duration_ms": 234,
      "tool_call_count": 1,
      "details": {
        "response_text": "...",
        "actual_data": ["..."],
        "expected_data": ["..."],
        "tool_calls": ["..."]
      }
    }
  ],
  "summary": {
    "total": 3,
    "passed": 3,
    "failed": 0,
    "total_tokens": 3750,
    "total_cost": 0.0375,
    "total_duration_ms": 702,
    "total_duration_s": 0.7,
    "total_tool_calls": 3,
    "avg_duration_ms": 234,
    "avg_tool_calls": 1.0
  }
}

​

dazense test server

• Summary Dashboard: Overview cards showing pass rate, total tests, tokens, costs, and duration
• Results Table: Interactive table of all test runs with status, metrics, and details
• Detailed View: Click any test to see:
  • Full response text
  • Tool calls with arguments and results
  • Data comparison (actual vs expected)
  • Diff view for failed tests
  • Performance metrics

dazense test server

​

Best Practices

​

Creating Effective Tests

1. Start with critical queries: Test the most important questions your users ask
2. Cover edge cases: Include tests for boundary conditions and complex scenarios
3. Keep tests focused: Each test should verify one specific behavior
4. Avoid overfitting and leakage: Avoid including exact answers or overly specific details in your context that would allow the agent to “cheat” by pattern matching rather than actually understanding the context.

​

Integrating into Workflow

1. Version control: Commit your tests/ folder to git
2. CI/CD integration: Run tests automatically on context changes
3. Regular evaluation: Run tests weekly or after major context updates
4. Track trends: Monitor pass rates and costs over time

​

Governance validation workflow (V1/V2)

dazense validate
dazense eval
dazense eval --engine
dazense test

• validate catches config drift
• eval checks declared governance expectations
• eval --engine checks enforcement behavior
• test checks answer quality/correctness