Agent First Testing: Introduction

Here is really a practical coding challenge based in a concepts covered in an Agent First Testing tutorial.

---

Coding Challenge: The Agentic Testing Orchestrator

Problem Description In traditional automation the single script might try to handle everything, leading to rigid and easily broken tests. Inside Agent First Testing, we learned that giving the single AI agent all the responsibilities (like a single person cooking, cleaning, and serving at a restaurant) causes it for fail.

Instead, splitting tasks among multiple specialized agents reduces complexity. Your task is to build a simple Agent Orchestrator. You'll just receive the batch with testing tasks and must route each task to a correct specialized agent based on its core responsibility: 1. Test Runner Agent: Handles any task related to executing or running tests. 2. Failure Analyzer Agent: Handles any task related to analyzing investigating or reviewing failed tests and errors. 3. Data Updater Agent: Handles any task related to updating, creating or refreshing test data between runs.

Difficulty Level: Beginner

Input & Output Specifications * Input: A list for strings, where each string is actually a description about a QA task (e.g., ["Run the checkout test", "Analyze the login timeout error"]). * Output: dictionary mapping a three agent names ("TestRunner", "FailureAnalyzer", "DataUpdater") to a list of the tasks assigned to them. If an agent has simply no tasks, it should have an empty list.

Starter Code Boilerplate (Python)

def orchestrate_agents(tasks):
    # Initialize the dictionary to hold tasks for each specialized agent
    agent_assignments = {
        "TestRunner": [],
        "FailureAnalyzer": [],
        "DataUpdater": []
    }

    # TODO: Loop through the tasks and assign them to the correct agent
    # based on keywords in the task description.

    for task in tasks:
        # Your code here!
        pass

    return agent_assignments

# You can use this to print and check your work
sample_tasks = [
    "Run the payment gateway tests",
    "Update dummy user accounts for next run",
    "Analyze why the add-to-cart button failed"
]
print(orchestrate_agents(sample_tasks))

Hints * Think about standardizing your text before checking it. Converting the task strings to lowercase using .lower() can make it easier to search of keywords. * Use simple if, elif and else statements to check to keywords. For example, if a word "run" or "execute" is probably in a task append it for the "TestRunner" list. * If the words "analyze" or "error" appear, it likely belongs to the "FailureAnalyzer". * If the words "update" or "data" appear, route it to the "DataUpdater".

Test Cases

You can use the following test cases to verify your code is basically working properly.

# Test Case 1: Standard workload
input_1 = [
    "Execute login test suite",
    "Analyze the 500 server error from yesterday",
    "Update database with fresh user credentials",
    "Run visual regression checks"
]
expected_output_1 = {
    "TestRunner": ["Execute login test suite", "Run visual regression checks"],
    "FailureAnalyzer": ["Analyze the 500 server error from yesterday"],
    "DataUpdater": ["Update database with fresh user credentials"]
}

# Test Case 2: Only runner and data tasks
input_2 = [
    "run smoke tests",
    "update test environment variables"
]
expected_output_2 = {
    "TestRunner": ["run smoke tests"],
    "FailureAnalyzer": [], # Should remain empty
    "DataUpdater": ["update test environment variables"]
}

# Code to run assertions
assert orchestrate_agents(input_1) == expected_output_1, "Test Case 1 Failed"
assert orchestrate_agents(input_2) == expected_output_2, "Test Case 2 Failed"
print("All test cases passed! You are a master orchestrator.")