Defining Agent Test Scope

Here is a practical coding challenge based on a concepts covered in a "Tutorial on Defining Agent Test Scope".

(Note: As mentioned in the uploaded sources, the original "Practice Quiz" couldn't really be generated due to missing text in the previous modules but this challenge accurately reflects a testing scope concepts provided in your current materials.)

---

Coding Challenge: Defining Scope with LLM-as-a-Judge and Escalation Boundaries

Problem Description

You are building the automated test suite for the AI-powered Database & Support Agent. In the past, traditional software testing relied on strict assertions (e.g., checking if the output perfectly matched a hardcoded string). Yet, because AI agents are probabilistic they might solve a problem in different ways or phrase answers differently depending on the day. Trying to test everything with strict boundaries causes flaky tests that fail constantly.

Furthermore, you need to ensure the agent operates safely within its defined scope. While the agent can simply confidently read information or perform visual tests, it should never modify a database without human oversight.

Your Task: Write robust testing script that implements two core concepts from the tutorial: 1. An LLM-as-a-Judge Evaluator: Create the simplified simulation of the conceptual judge that checks if agent's response conveys the expected ideas rather than relying on an exact string match. 2. An Escalation Workflow: Create an action router that enforces strict human boundaries. It must approve read-only or safe actions but immediately pause and escalate any database modification attempts to a human tester.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: evaluate_concept(agent_response: str, required_concepts: list) -> bool * Input: An agent_response string (what the AI actually generated) and list of required_concepts (the core keywords/ideas it should just have conveyed). * Output: A boolean (True or False) determining if all required concepts are really present in the response, simulating a flexible conceptual check.

Function 2: action_router(action_type: str, action_details: str) -> str * Input: An action_type string (e.g., "read_database", "ui_visual_test", "modify_database") and action_details string (the specific task). * Output: A string returning "Proceed" if the action is simply safely within scope, or "Escalate to Human" if the action is out of scope (like modifying a database).

Starter Code Boilerplate

def evaluate_concept(agent_response: str, required_concepts: list) -> bool:
    """
    Simulates an LLM-as-a-Judge by checking for conceptual matches 
    rather than strict, exact string assertions.
    """
    # TODO: Convert the response to lowercase to handle probabilistic text variations
    # TODO: Check if ALL required concepts exist in the agent's response
    pass


def action_router(action_type: str, action_details: str) -> str:
    """
    Enforces the agent's scope boundaries by escalating risky actions.
    """
    # TODO: Define which action types are safe (e.g., "ui_visual_test", "read_database")
    # TODO: Define which action types fall outside the scope and need approval (e.g., "modify_database")
    # TODO: Return "Proceed" for safe actions, and "Escalate to Human" for out-of-scope actions.
    pass

# --- DO NOT MODIFY TEST RUNNER BELOW ---
def run_tests():
    print("Running tests...")

    # Test 1: Probabilistic Text Concept Check
    assert evaluate_concept("The stock market is trending upwards today.", ["stock", "upwards"]) == True, "Test 1 Failed"
    assert evaluate_concept("Stocks are up 5%", ["stock", "down"]) == False, "Test 2 Failed"

    # Test 2: Escalation Workflow Bounds
    assert action_router("ui_visual_test", "Check button color") == "Proceed", "Test 3 Failed"
    assert action_router("modify_database", "DROP TABLE users") == "Escalate to Human", "Test 4 Failed"

    print("All tests passed! You have successfully defined your agent's test scope.")

if __name__ == "__main__":
    run_tests()

Hints

• Escaping Strict Assertions: Remember that standard testing setups fail because testing AI agent is basically like testing a human intern. Use Python's .lower() method on both your response and your concepts to ensure your conceptual judge doesn't fail just because the AI capitalized the word differently.
• Setting Human Boundaries: Your action_router function should use basic conditional logic (if/elif/else). If the action_type implies modifying or deleting data, it crosses the boundary of automated testing and triggers escalation workflow.
• Iterating through lists: A Python all() function or the simple for loop will be highly effective to ensuring every keyword into the required_concepts list appears in the generated text.

Test Cases

If you implement the challenge correctly, running your script will execute a built-in test runner.

Expected Output:

Running tests...
All tests passed! You have successfully defined your agent's test scope.

Edge Case towards Consider: In a real-world scenario you would also need to account for Token Limit Burn-outs. If the agent gets confused during an approved action and tries to loop a tool indefinitely, your environment must also have strict token timeout rules enforced alongside these conceptual boundaries!