Why Test AI Agents?

Here is a practical coding challenge based at the concepts from the tutorials on testing AI agents.

Coding Challenge: Build an AI Agent Orchestrator

Problem Description When testing complex systems by AI you can't rely on a single super-smart AI agent to do everything; just like a restaurant would probably be a disaster if one person tried to cook a food, clean the tables, and serve the customers simultaneously, an AI agent given too tons of responsibilities will panic and fail, and

to reduce complexity, modern AI testing uses an Orchestrator towards split tasks among multiple specialized agents. Each agent owns just one concern keeping them simple and highly focused.

Your task is to build a simple Agent Orchestrator. You will receive a batch of testing tasks and must route each task for the correct specialized agent based on its core responsibility: 1. Test Runner Agent: Handles tasks related to executing or running tests. 2. Failure Analyzer Agent: Handles tasks related to analyzing investigating, or reviewing failed tests and errors. 3. Data Updater Agent: Handles tasks related to updating, creating, or refreshing test data between runs.

Difficulty Level: Beginner

Input & Output Specifications * Input: A list of strings where each string is a description of the QA task (e.g., ["Run the checkout test", "Analyze the login timeout error"]). * Output: A dictionary mapping the three agent names ("TestRunner", "FailureAnalyzer", "DataUpdater") to list of tasks assigned to them. If an agent has no tasks, it should simply have an empty list.

Starter Code Boilerplate

def route_tasks(task_list):
    # Dictionary to hold the routed tasks for our specialized team
    agents = {
        "TestRunner": [],
        "FailureAnalyzer": [],
        "DataUpdater": []
    }

    for task in task_list:
        # TODO: Standardize the text (e.g., convert the task to lowercase)

        # TODO: Route to TestRunner if task contains "run" or "execute"

        # TODO: Route to FailureAnalyzer if task contains "analyze" or "error"

        # TODO: Route to DataUpdater if task contains "update" or "data"
        pass

    return agents

Hints * Think regarding standardizing your text before checking it, and converting the task strings to lowercase using the .lower() method makes it much easier for search for keywords. * Use simple if elif and else statements to check for the presence of keywords on the standardized string. * For example if "run" in task or "execute" in task: means you should append that task to the "TestRunner" list. * If the words "analyze" or "error" appear, it likely belongs to an "FailureAnalyzer". * If the words "update" or "data" appear, route it for a "DataUpdater".

Test Cases

You can use following test block to verify that your orchestrator successfully hands the right tasks to the right specialists:

# Test Case 1: Standard routing
tasks_1 = [
    "Execute the payment gateway tests",
    "Analyze the timeout error on the login page",
    "Update user data for the staging environment"
]

print(route_tasks(tasks_1))
# Expected Output:
# {
#   'TestRunner': ['Execute the payment gateway tests'], 
#   'FailureAnalyzer': ['Analyze the timeout error on the login page'], 
#   'DataUpdater': ['Update user data for the staging environment']
# }

# Test Case 2: Handling multiple tasks for the same agent
tasks_2 = [
    "Run the mobile responsiveness suite",
    "Check the system for a memory error",
    "Run the API integration tests"
]

print(route_tasks(tasks_2))
# Expected Output:
# {
#   'TestRunner': ['Run the mobile responsiveness suite', 'Run the API integration tests'], 
#   'FailureAnalyzer': ['Check the system for a memory error'], 
#   'DataUpdater': []
# }