agent_inspect.metrics.validator package

Submodules

agent_inspect.metrics.validator.exact_match module

agent_inspect.metrics.validator.exact_match.exact_match(candidate, ground_truth, config=None)[source]
Return type:

bool

Parameters:

  • candidate (str)

  • ground_truth (str)

  • config (Dict[str, Any] | None)

agent_inspect.metrics.validator.llm_check module

async agent_inspect.metrics.validator.llm_check.llm_check(client, variables, template, post_process)[source]
Return type:

bool

Parameters:

agent_inspect.metrics.validator.regex_match module

agent_inspect.metrics.validator.regex_match.regex_match(candidate, pattern, config=None)[source]
Return type:

bool

Parameters:

  • candidate (str)

  • pattern (str)

  • config (Dict[str, Any] | None)

agent_inspect.metrics.validator.subgoal_completion module

class agent_inspect.metrics.validator.subgoal_completion.SubGoalCompletionValidator(llm_client, config=None)[source]

Bases: Validator

Validator based on LLM-as-a-judge to assess whether the agent has completed the specified subgoal.

\[LLM_{judge}(i, g_{i, j}, \tau_i ) = 1 \ \mathrm{if \ agent \ accomplish} \ g_{i,j}, \mathrm{and} \ 0 \ \mathrm{otherwise}, \]

where \(g_{i,j}\) is the \(j\)-th grading note for the task sample \(i\) and \(\tau_i\) is the agent trajectory consisting of tool calls, agent responses, and user inputs.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the LLM-as-a-judge model for evaluation.

  • config (Optional[Dict[str, Any]]) –

    Default to None. Configuration options:

    • templates_subgoal: a user provided LLM-as-a-judge template which will be sent to the judge with user inputs (static setting), subgoal, trajectory, agent responses, user task (dynamic setting), and user utterances (dynamic setting). If this is not provided, the default template for either static or dynamic conversation setting will be used.

    • num_judge_trials: the number of LLM-as-a-judge runs. Default to 5. A majority vote is used when the number of LLM-as-a-judge runs is set to a value larger than 1.

    • include_judge_explanation: a bool flag to indicate whether the output should also return judge explanations. Default to False.

    • optimize_judge_trials: a bool flag to indicate whether to use optimized judge runs when doing a majority vote. Default to False.

    • max_retry_judge_trials: an int value indicating the maximum number of retry attempts for each judge trial in case of errors related to LLM as a judge. Default to 5. This will be ignored if optimize_judge_trials is set to True.

    • include_entire_prompt_in_validation_result: a bool flag to indicate whether to include the entire prompt sent to LLM-as-a-judge in the SubGoalValidationResult. Use this in the debugging tool UI for display. Default to False.

async validate(turn_traces, sub_goal)[source]

Returns the LLM-as-a-judge binary score given the subgoal and the agent’s trace up to the current turn in a static conversation setting.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • sub_goal (SubGoal) – a SubGoal object representing a grading note in the form of a natural language text.

Return type:

SubGoalValidationResult

Returns:

a SubGoalValidationResult object containing the judge binary score, the subgoal details, and the judge explanations.

Example:

>>> from agent_inspect.metrics.validator import SubGoalCompletionValidator
>>> from agent_inspect.metrics.constants import INCLUDE_JUDGE_EXPLANATION, OPTIMIZE_JUDGE_TRIALS
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> data_subgoal = load_subgoal(sample_path) # Load subgoal
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = SubGoalCompletionValidator(
...     llm_client=client,
...     config={INCLUDE_JUDGE_EXPLANATION: True, OPTIMIZE_JUDGE_TRIALS: False}
... )
>>> validator_result = asyncio.run(
...     validator.validate(
...         turn_traces=agent_turn_traces,
...         sub_goal=data_subgoal
...     )
... )
>>> print(validator_result.is_completed)
async validate_dynamic(turn_traces, sub_goal, user_instruction)[source]

Returns the LLM-as-a-judge binary score given the subgoal, user task instructions (optional), and the agent’s trace up to the current turn in a dynamic conversation setting.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • sub_goal (SubGoal) – a SubGoal object representing a grading note in the form of a natural language text.

  • user_instruction (str) – a str object representing the user task instructions. Provide user task instruction through this variable to use judge template with user summary instruction. Otherwise, set it as empty string to use judge template without any user summary instruction.

Return type:

SubGoalValidationResult

Returns:

a SubGoalValidationResult object containing the judge binary score, the subgoal details, and the judge explanations.

Example:

>>> from agent_inspect.metrics.validator import SubGoalCompletionValidator
>>> from agent_inspect.metrics.constants import INCLUDE_JUDGE_EXPLANATION, OPTIMIZE_JUDGE_TRIALS
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> data_subgoal, user_instruct = load_subgoal_user_instruct(sample_path) # Load subgoal and user instructions
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = SubGoalCompletionValidator(
...     llm_client=client,
...     config={INCLUDE_JUDGE_EXPLANATION: True, OPTIMIZE_JUDGE_TRIALS: False}
... )
>>> validator_result = asyncio.run(
...     validator.validate_dynamic(
...         turn_traces=agent_turn_traces,
...         sub_goal=data_subgoal,
...         user_instruction=user_instruct
...     )
... )
>>> print(validator_result.is_completed)

agent_inspect.metrics.validator.tool_call_completion module

class agent_inspect.metrics.validator.tool_call_completion.ToolCallCompletionValidator(llm_client, config=None)[source]

Bases: Validator

Tool call completion validator using both exact match and LLM-as-a-judge approaches to assess whether the tool call is correctly completed.

This validator performs a three-dimensional evaluation of tool calls:

  1. Tool name validation: Checks if the tool name in the agent’s trajectory matches the expected tool name exactly.

  2. Tool input arguments validation: Validates each input parameter using either exact match (when the expected parameter specifies a value) or LLM-as-a-judge (when the expected parameter specifies a check).

  3. Tool output validation: Validates the tool’s output using either exact match (when the expected output specifies a value) or LLM-as-a-judge (when the expected output specifies a check).

A tool call is considered correctly completed only if all the above three dimensions pass validation. The validator iterates through all tool calls in the latest turn of the agent trace to find a matching tool call that satisfies all validation criteria set in ExpectedToolCall.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the LLM-as-a-judge model for evaluation.

  • config (Optional[Dict[str, Any]]) –

    Default to None. Configuration options:

    • num_judge_trials: the number of LLM-as-a-judge runs. Default to 5. A majority vote is used when the number of LLM-as-a-judge runs is set to a value larger than 1.

    • include_judge_explanation: a bool flag to indicate whether the output should also return judge explanations. Default to False.

async validate(agent_trace_turns, expected_tool_call)[source]

Returns a ToolCallValidationResult indicating whether the tool call is correctly completed by the agent in the latest turn of the agent trace.

Parameters:

  • agent_trace_turns (List[TurnTrace]) – A list of TurnTrace representing the entire agent trace up to the current turn.

  • expected_tool_call (ExpectedToolCall) – An ExpectedToolCall representing the expected tool call checklist to validate against.

Return type:

ToolCallValidationResult

Returns:

a ToolCallValidationResult indicating whether the tool call is correctly completed by the agent.

Example:

>>> from agent_inspect.metrics import ToolCallCompletionValidator
>>> from agent_inspect.metrics.constants import NUM_JUDGE_TRIALS, INCLUDE_JUDGE_EXPLANATION
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> expected_tool_call = load_expected_tool_call(expected_tool_call_file_path) # Load expected tool call checklist
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = ToolCallCompletionValidator(
...    llm_client=client, 
...    config={NUM_JUDGE_TRIALS: 5, INCLUDE_JUDGE_EXPLANATION: True}
... )
>>> validation_result = asyncio.run(
...   validator.validate(
...        agent_trace_turns = agent_turn_traces, 
...        expected_tool_call = expected_tool_call
...    )
... )
>>> print(validation_result.is_completed)  # True or False

agent_inspect.metrics.validator.validator module

class agent_inspect.metrics.validator.validator.Validator(llm_client, config=None)[source]

Bases: object

Abstract class which should be extended for actual implementation of validators.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the llm-as-a-judge model for evaluations.

  • config (Optional[Dict[str, Any]]) – configuration for validator initialization. Default to None.

async get_majority_voted_score_from_judge_responses(prompt)[source]
Return type:

(int, List[str])

Parameters:

prompt (str)

abstract async validate(turn_traces, kwargs)[source]

This is an abstract method and should be implemented in a concrete class.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • kwargs – Additional keyword arguments that may be required for specific validation logic. These arguments can be used to pass optional parameters or configuration settings to the validator.

Return type:

ValidationResult

Returns:

a ValidationResult object containing the validation output.

Module contents

class agent_inspect.metrics.validator.SubGoalCompletionValidator(llm_client, config=None)[source]

Bases: Validator

Validator based on LLM-as-a-judge to assess whether the agent has completed the specified subgoal.

\[LLM_{judge}(i, g_{i, j}, \tau_i ) = 1 \ \mathrm{if \ agent \ accomplish} \ g_{i,j}, \mathrm{and} \ 0 \ \mathrm{otherwise}, \]

where \(g_{i,j}\) is the \(j\)-th grading note for the task sample \(i\) and \(\tau_i\) is the agent trajectory consisting of tool calls, agent responses, and user inputs.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the LLM-as-a-judge model for evaluation.

  • config (Optional[Dict[str, Any]]) –

    Default to None. Configuration options:

    • templates_subgoal: a user provided LLM-as-a-judge template which will be sent to the judge with user inputs (static setting), subgoal, trajectory, agent responses, user task (dynamic setting), and user utterances (dynamic setting). If this is not provided, the default template for either static or dynamic conversation setting will be used.

    • num_judge_trials: the number of LLM-as-a-judge runs. Default to 5. A majority vote is used when the number of LLM-as-a-judge runs is set to a value larger than 1.

    • include_judge_explanation: a bool flag to indicate whether the output should also return judge explanations. Default to False.

    • optimize_judge_trials: a bool flag to indicate whether to use optimized judge runs when doing a majority vote. Default to False.

    • max_retry_judge_trials: an int value indicating the maximum number of retry attempts for each judge trial in case of errors related to LLM as a judge. Default to 5. This will be ignored if optimize_judge_trials is set to True.

    • include_entire_prompt_in_validation_result: a bool flag to indicate whether to include the entire prompt sent to LLM-as-a-judge in the SubGoalValidationResult. Use this in the debugging tool UI for display. Default to False.

async validate(turn_traces, sub_goal)[source]

Returns the LLM-as-a-judge binary score given the subgoal and the agent’s trace up to the current turn in a static conversation setting.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • sub_goal (SubGoal) – a SubGoal object representing a grading note in the form of a natural language text.

Return type:

SubGoalValidationResult

Returns:

a SubGoalValidationResult object containing the judge binary score, the subgoal details, and the judge explanations.

Example:

>>> from agent_inspect.metrics.validator import SubGoalCompletionValidator
>>> from agent_inspect.metrics.constants import INCLUDE_JUDGE_EXPLANATION, OPTIMIZE_JUDGE_TRIALS
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> data_subgoal = load_subgoal(sample_path) # Load subgoal
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = SubGoalCompletionValidator(
...     llm_client=client,
...     config={INCLUDE_JUDGE_EXPLANATION: True, OPTIMIZE_JUDGE_TRIALS: False}
... )
>>> validator_result = asyncio.run(
...     validator.validate(
...         turn_traces=agent_turn_traces,
...         sub_goal=data_subgoal
...     )
... )
>>> print(validator_result.is_completed)
async validate_dynamic(turn_traces, sub_goal, user_instruction)[source]

Returns the LLM-as-a-judge binary score given the subgoal, user task instructions (optional), and the agent’s trace up to the current turn in a dynamic conversation setting.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • sub_goal (SubGoal) – a SubGoal object representing a grading note in the form of a natural language text.

  • user_instruction (str) – a str object representing the user task instructions. Provide user task instruction through this variable to use judge template with user summary instruction. Otherwise, set it as empty string to use judge template without any user summary instruction.

Return type:

SubGoalValidationResult

Returns:

a SubGoalValidationResult object containing the judge binary score, the subgoal details, and the judge explanations.

Example:

>>> from agent_inspect.metrics.validator import SubGoalCompletionValidator
>>> from agent_inspect.metrics.constants import INCLUDE_JUDGE_EXPLANATION, OPTIMIZE_JUDGE_TRIALS
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> data_subgoal, user_instruct = load_subgoal_user_instruct(sample_path) # Load subgoal and user instructions
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = SubGoalCompletionValidator(
...     llm_client=client,
...     config={INCLUDE_JUDGE_EXPLANATION: True, OPTIMIZE_JUDGE_TRIALS: False}
... )
>>> validator_result = asyncio.run(
...     validator.validate_dynamic(
...         turn_traces=agent_turn_traces,
...         sub_goal=data_subgoal,
...         user_instruction=user_instruct
...     )
... )
>>> print(validator_result.is_completed)
class agent_inspect.metrics.validator.ToolCallCompletionValidator(llm_client, config=None)[source]

Bases: Validator

Tool call completion validator using both exact match and LLM-as-a-judge approaches to assess whether the tool call is correctly completed.

This validator performs a three-dimensional evaluation of tool calls:

  1. Tool name validation: Checks if the tool name in the agent’s trajectory matches the expected tool name exactly.

  2. Tool input arguments validation: Validates each input parameter using either exact match (when the expected parameter specifies a value) or LLM-as-a-judge (when the expected parameter specifies a check).

  3. Tool output validation: Validates the tool’s output using either exact match (when the expected output specifies a value) or LLM-as-a-judge (when the expected output specifies a check).

A tool call is considered correctly completed only if all the above three dimensions pass validation. The validator iterates through all tool calls in the latest turn of the agent trace to find a matching tool call that satisfies all validation criteria set in ExpectedToolCall.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the LLM-as-a-judge model for evaluation.

  • config (Optional[Dict[str, Any]]) –

    Default to None. Configuration options:

    • num_judge_trials: the number of LLM-as-a-judge runs. Default to 5. A majority vote is used when the number of LLM-as-a-judge runs is set to a value larger than 1.

    • include_judge_explanation: a bool flag to indicate whether the output should also return judge explanations. Default to False.

async validate(agent_trace_turns, expected_tool_call)[source]

Returns a ToolCallValidationResult indicating whether the tool call is correctly completed by the agent in the latest turn of the agent trace.

Parameters:

  • agent_trace_turns (List[TurnTrace]) – A list of TurnTrace representing the entire agent trace up to the current turn.

  • expected_tool_call (ExpectedToolCall) – An ExpectedToolCall representing the expected tool call checklist to validate against.

Return type:

ToolCallValidationResult

Returns:

a ToolCallValidationResult indicating whether the tool call is correctly completed by the agent.

Example:

>>> from agent_inspect.metrics import ToolCallCompletionValidator
>>> from agent_inspect.metrics.constants import NUM_JUDGE_TRIALS, INCLUDE_JUDGE_EXPLANATION
>>> from agent_inspect.clients import AzureOpenAIClient
>>> import asyncio
>>>
>>> expected_tool_call = load_expected_tool_call(expected_tool_call_file_path) # Load expected tool call checklist
>>> agent_turn_traces = load_agent_turn_traces(trace_file_path) # Load agent trajectory information
>>> client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096) # create client needed for LLM-based metric
>>> validator = ToolCallCompletionValidator(
...    llm_client=client, 
...    config={NUM_JUDGE_TRIALS: 5, INCLUDE_JUDGE_EXPLANATION: True}
... )
>>> validation_result = asyncio.run(
...   validator.validate(
...        agent_trace_turns = agent_turn_traces, 
...        expected_tool_call = expected_tool_call
...    )
... )
>>> print(validation_result.is_completed)  # True or False
class agent_inspect.metrics.validator.Validator(llm_client, config=None)[source]

Bases: object

Abstract class which should be extended for actual implementation of validators.

Parameters:

  • llm_client (LLMClient) – the client which allows connection to the llm-as-a-judge model for evaluations.

  • config (Optional[Dict[str, Any]]) – configuration for validator initialization. Default to None.

async get_majority_voted_score_from_judge_responses(prompt)[source]
Return type:

(int, List[str])

Parameters:

prompt (str)

abstract async validate(turn_traces, kwargs)[source]

This is an abstract method and should be implemented in a concrete class.

Parameters:

  • turn_traces (List[TurnTrace]) – a List [TurnTrace] object constructed with the agent trajectory information from the first turn up to the current turn.

  • kwargs – Additional keyword arguments that may be required for specific validation logic. These arguments can be used to pass optional parameters or configuration settings to the validator.

Return type:

ValidationResult

Returns:

a ValidationResult object containing the validation output.

agent_inspect.metrics.validator.exact_match(candidate, ground_truth, config=None)[source]
Return type:

bool

Parameters:

  • candidate (str)

  • ground_truth (str)

  • config (Dict[str, Any] | None)

async agent_inspect.metrics.validator.llm_check(client, variables, template, post_process)[source]
Return type:

bool

Parameters:
agent_inspect.metrics.validator.regex_match(candidate, pattern, config=None)[source]
Return type:

bool

Parameters:

  • candidate (str)

  • pattern (str)

  • config (Dict[str, Any] | None)