Agenta Evaluators

Agenta offers several evaluators to assess model performance in classification and entity extraction tasks.

Exact Match

The Exact Match evaluator determines if the model's output precisely matches the expected answer.

How It Works

This evaluator compares the generated output with the correct answer stored in the test set. It returns a boolean value: true for an exact match, and false otherwise.

Configuration

Contains JSON

The Contains JSON evaluator checks if the model's output contains a valid JSON structure.

How It Works

This evaluator attempts to parse the output as JSON. It returns true if a valid JSON structure is found within the output, and false otherwise.

JSON Field Match (Deprecated)

The JSON Field Match evaluator has been replaced by JSON Multi-Field Match. The new evaluator supports multiple fields, nested paths, and provides per-field scoring. Existing configurations will continue to work, but we recommend migrating to the new evaluator.

JSON Multi-Field Match

The JSON Multi-Field Match evaluator compares multiple fields between two JSON objects and reports a score for each field. This evaluator is ideal for entity extraction tasks where you need to validate that specific fields (like name, email, or address) match the expected values.

How It Works

The evaluator parses both the model output and the ground truth as JSON. It then compares each configured field path and produces:

1. A score for each field (1 if matched, 0 if not matched)
2. An aggregate score showing the percentage of fields that matched

Path Formats

You can specify field paths in three formats:

Configuration

Example

Ground truth (in the correct_answer column):

{
  "name": "John Doe",
  "email": "john@example.com",
  "address": {
    "city": "New York",
    "zip": "10001"
  }
}

Model output:

{
  "name": "John Doe",
  "email": "jane@example.com",
  "address": {
    "city": "New York",
    "zip": "10002"
  }
}

Configured fields: ['name', 'email', 'address.city', 'address.zip']

Results:

JSON Diff Match

How It Works

This evaluator compares the output JSON with the correct answer JSON. Here's a detailed breakdown of the process:

1. JSON Flattening: Both the output and correct answer JSONs are flattened into single-level dictionaries.

2. Comparison: The evaluator compares these flattened structures, checking each key-value pair while considering configuration options (like case sensitivity and schema-only comparison).

3. Scoring: For each matching key-value pair, a score of 1 is assigned. The final score is the average of all comparisons, resulting in a value between 0 and 1.

Configuration