agent_inspect.metrics.utils package

Submodules

agent_inspect.metrics.utils.expected_tool_validators module

class agent_inspect.metrics.utils.expected_tool_validators.ExpectedToolCallValidator[source]

Bases: object

A utility class for validating expected tool call in eval data samples.

agent_inspect.metrics.utils.metrics_utils module

agent_inspect.metrics.utils.metrics_utils.get_config_or_default(config, config_key, default)[source]
Parameters:

  • config (Dict[str, Any])

  • config_key (str)

  • default (Any)

agent_inspect.metrics.utils.metrics_utils.get_majority_voted_score(score_to_vote_count)[source]
Parameters:

score_to_vote_count (Dict[Any, int])

agent_inspect.metrics.utils.metrics_utils.map_subgoal_validations_to_binary_matrix(completions)[source]
Return type:

List[int]

Parameters:

completions (List[str])

agent_inspect.metrics.utils.metrics_utils.match_to_int(completion)[source]
agent_inspect.metrics.utils.metrics_utils.tally_judge_voting(complete_cnt, incomplete_cnt, invalid_cnt, judge_responses)[source]
agent_inspect.metrics.utils.metrics_utils.tally_votes(complete_cnt, incomplete_cnt, invalid_cnt, completions)[source]
agent_inspect.metrics.utils.metrics_utils.validate_inputs_for_pass_k_initialisation(k_value, num_trials)[source]
Parameters:

  • k_value (int)

  • num_trials (int)

agent_inspect.metrics.utils.subgoal_validators module

class agent_inspect.metrics.utils.subgoal_validators.SubGoalValidator[source]

Bases: object

A utility class for validating subgoals in agent turn traces.

agent_inspect.metrics.utils.trace_validators module

class agent_inspect.metrics.utils.trace_validators.TraceValidator[source]

Bases: object

A utility class for validating agent turn traces.

agent_inspect.metrics.utils.user_proxy_validators module

class agent_inspect.metrics.utils.user_proxy_validators.UserProxyInputValidator[source]

Bases: object

Module contents