Basic Agent Test Cases

(Note: The uploaded sources don't just include a specific text to a "Practice Quiz on Basic Agent Test Cases." However, this coding challenge is comprehensively built upon the core concepts taught in the provided "Tutorial on Basic Agent Test Cases" to test your practical understanding.)

---

Coding Challenge: Mastering Basic Agent Test Cases

Problem Description

You're pretty much an intermediate QA engineer building an automated test suite for the new AI-powered Customer Support Agent. You have really recently run into three major roadblocks that make traditional software testing impossible: 1. Strict Assertions are Failing: An AI is probabilistic, meaning its phrasing changes every time, while checking for exact string matches causes tests to fail constantly. 2. Flaky Tests: The agent uses a live web search tool to fetch documentation. Live web pages change daily causing your tests to pass on Tuesday and fail on Wednesday. 3. Disastrous Edge Cases: You have noticed agent sometimes gets confused and enters an infinite loop or attempts to execute destructive actions (like modifying a database) without permission.

Your Task: Write a robust testing script that implements the three core code patterns needed for tame this AI agent: 1. A Conceptual Judge: Evaluate a properties of the AI's output rather than demanding an exact word match. 2. A Mocked Tool: Replace the live web search with the controlled function to prioritize consistency over realism. 3. The Escalation Router: Build a safety net that enforces strict token limits and requires human approval for dangerous tasks.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: conceptual_judge(agent_response: str, required_concepts: list) -> bool * Input: agent_response (a string the AI generated) and required_concepts (a list of core ideas/words it must convey). * Output: A boolean (True or False) that simulates testing properties over exact words, while returns True if all required concepts are present inside the response regardless of capitalization.

Function 2: mock_web_search(query: str) -> str * Input: query string (e.g., "latest support docs"). * Output: A hardcoded predictable string (e.g., "Mocked support data: Issue resolved") to prevent flaky tests.

Function 3: action_router(action_type: str, current_tokens: int, token_limit: int) -> str * Input: action_type (e.g., "read_database", "modify_database"), current_tokens (integer), and token_limit (integer). * Output: * Returns "Halt: Token Limit Burn-out" if current tokens exceed the token limit. * Returns "Escalate to Human" if the action type implies a destructive action (like modifying or deleting). * Returns "Proceed" if an action is read-only and within token limits.

Starter Code Boilerplate

def conceptual_judge(agent_response: str, required_concepts: list) -> bool:
    """
    Evaluates the properties of the output instead of strict assertions.
    """
    # TODO: Implement logic to check if all concepts exist in the response
    # Hint: Watch out for case sensitivity!
    pass


def mock_web_search(query: str) -> str:
    """
    Intercepts a live web search to prioritize consistency over realism.
    """
    # TODO: Return a predictable, hardcoded string regardless of the query
    pass


def action_router(action_type: str, current_tokens: int, token_limit: int) -> str:
    """
    Catches token burn-outs and escalates destructive actions.
    """
    # TODO: Implement the escalation workflow and token timeout rules
    pass


# --- TEST RUNNER ---
if __name__ == "__main__":
    print("Testing Conceptual Judge:")
    # Should print True
    print(conceptual_judge("The stock market is trending upwards today.", ["stock", "upwards"]))

    print("\nTesting Mocked Tool:")
    # Should print your hardcoded string
    print(mock_web_search("Why is my screen blank?"))

    print("\nTesting Action Router:")
    # Should print "Halt: Token Limit Burn-out"
    print(action_router("read_database", 5500, 5000))
    # Should print "Escalate to Human"
    print(action_router("modify_database", 1200, 5000))
    # Should print "Proceed"
    print(action_router("ui_visual_test", 1200, 5000))

Hints

• Testing Properties Not Words: Testing an AI agent is like testing a human intern. To avoid flaky logic inside your conceptual_judge, convert both the agent's response and a concepts to lowercase using Python's .lower() method. A for loop or the all() function works perfectly here.
• Consistency Over Realism: Your mock_web_search doesn't need any complex logic. It literally just needs to return a static string, while the goal is to guarantee your automated tests are predictable.
• Order of Operations: On your action_router, catching budget-destroying infinite loops should be your top priority. Check for token burn-outs before evaluating what type of action the agent is trying towards perform. Use simple if/elif/else logic to categorize "modify" as a destructive action that requires escalation.

Test Cases

If you implement the challenge correctly, running your script will execute the built-in test runner and output exactly:

Testing Conceptual Judge:
True

Testing Mocked Tool:
Mocked support data: Issue resolved

Testing Action Router:
Halt: Token Limit Burn-out
Escalate to Human
Proceed