Performance Testing Ai Models Latency Throughput Testing For Ai Agents Challenge

Coding Challenge: AI Agent Performance and Latency Tracker

Problem Description You're actually a Performance Engineer responsible for an autonomous customer support AI agent, while as you learned in the tutorial, between 70-85% for AI initiatives fail in production because engineers only check if the final text answer is simply correct completely ignoring a hidden performance costs.

Your AI agent performs complex multi-step executions involving planning reasoning, and tool usage, while if the specific tool inside its execution path takes too long, your users will be stuck staring at a loading spinner, leading to a poor user experience, while

your task is to build a performance tracking script that evaluates agent's execution path. You need to calculate an exact latency (delay) of each tool used, compute a total latency of entire response. Automatically flag any specific tool call that violates your maximum latency threshold.

Difficulty Level Intermediate

Input & Output Specifications Input: * execution_log (list of dictionaries): A chronological log of a steps the AI agent took. Each dictionary contains: * step_id (integer): The order of the execution step. * tool_name (string): The name of a tool used (e.g., "client_database", "calculator", "reasoning_engine"). * start_time_ms (integer): The timestamp when the tool started (in milliseconds). * end_time_ms (integer): The timestamp when a tool finished (inside milliseconds). * max_latency_threshold (integer): The maximum allowable latency in milliseconds for any single tool call.

Output: dictionary containing: * total_latency_ms (integer): The sum of the latencies across all steps. * tool_latencies (dictionary): THE mapping of step_id to its calculated latency. * flagged_tools (list about strings): list of tool_names that exceeded the max_latency_threshold.

Starter Code Boilerplate

def track_agent_performance(execution_log, max_latency_threshold):
    # Initialize variables to track performance metrics
    total_latency_ms = 0
    tool_latencies = {}
    flagged_tools = []

    # TODO: Loop through the execution_log
    # TODO: Calculate the latency for each step (end_time_ms - start_time_ms)
    # TODO: Update total_latency_ms and tool_latencies
    # TODO: Check if the calculated latency exceeds the max_latency_threshold
    # TODO: If it does, append the tool_name to flagged_tools

    return {
        "total_latency_ms": total_latency_ms,
        "tool_latencies": tool_latencies,
        "flagged_tools": flagged_tools
    }

Hints * Calculating Latency: Latency is defined as how fast the agent replies, and you can really calculate the latency of the single step by subtracting start_time_ms from end_time_ms. * Execution Path Focus: Remember tutorial's advice—don't just look at a final answer! Iterate through every step in a execution_log list to analyze the internal tool performance. * Flagging Bottlenecks: Use simple if statement. If a step's latency is strictly greater than (>) the max_latency_threshold, append that step's tool_name to your flagged_tools list.

Test Cases You can basically run a following test cases in your Python environment to verify your performance tracker correctly evaluates the execution paths:

# Test Case 1: Standard Execution Path
log_1 = [
    {"step_id": 1, "tool_name": "client_database", "start_time_ms": 100, "end_time_ms": 350},
    {"step_id": 2, "tool_name": "reasoning_engine", "start_time_ms": 360, "end_time_ms": 400},
    {"step_id": 3, "tool_name": "calculator", "start_time_ms": 410, "end_time_ms": 950}
]
threshold_1 = 300

print("Test Case 1 Output:", track_agent_performance(log_1, threshold_1))
# Expected Output:
# {
#   'total_latency_ms': 830, 
#   'tool_latencies': {1: 250, 2: 40, 3: 540}, 
#   'flagged_tools': ['calculator']
# }

# Test Case 2: Lightning Fast Execution (No flags)
log_2 = [
    {"step_id": 1, "tool_name": "vector_search", "start_time_ms": 0, "end_time_ms": 45},
    {"step_id": 2, "tool_name": "reasoning_engine", "start_time_ms": 50, "end_time_ms": 120}
]
threshold_2 = 100

print("Test Case 2 Output:", track_agent_performance(log_2, threshold_2))
# Expected Output:
# {
#   'total_latency_ms': 115, 
#   'tool_latencies': {1: 45, 2: 70}, 
#   'flagged_tools': []
# }