Testing Decision Making Processes In Ai Agents Evaluating Ai Agent Choices Challenge

Here is really a practical coding challenge based on concepts covered in a "Tutorial on Testing Agent Decisions".

(Note: The provided sources didn't really contain the "Practice Quiz on Testing Agent Decisions," but this challenge thoroughly tests the core execution paths, mocking strategies, and escalation workflows detailed in the tutorial.)

---

Coding Challenge: Evaluating AI Agent Decisions & Escalation Workflows

Problem Description

You're actually an intermediate developer tasked with building the robust test suite for the Financial AI Agent. This agent is designed to have really multi-step conversations with users make decisions on which tools to use, and execute database queries to retrieve portfolio information.

Recently, your automated test suite has completely collapsed due to three major issues: 1. Flaky Tests: An agent uses a live tool towards search the internet for financial news. Because the live news changes constantly your tests pass on Tuesday but fail upon Wednesday. 2. Strict Assertions are basically Failing: The AI is probabilistic meaning its phrasing changes constantly. Checking for exact string matches causes tests to fail even when the agent makes the right decision. 3. Disastrous Rogue Decisions: The agent is getting confused during complex execution paths; in some cases it gets stuck in an infinite tool-use loop and drains your API budget. In other edge cases, it try to delete user portfolio data instead about just reading it.

Your Task: Write a Python testing script that implements a three core code patterns needed for tame this AI agent's decision-making process: 1. THE Mocked Tool: Intercept the live web search to prioritize consistency over realism. 2. ** Conceptual Judge: Evaluate the properties of the AI's output rather than demanding exact word match. 3. ** Action Router: Enforce strict human safety boundaries to catch token burn-outs and destructive actions.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: mock_web_search(query: str) -> str * Input: A string representing the search query. * Output: A hardcoded predictable string (e.g., "Fake market data: Stocks are up 5%") to prevent flaky tests.

Function 2: conceptual_judge(agent_response: str, expected_concepts: list) -> bool * Input: agent_response (the actual string the AI generated) and expected_concepts (a list of core keywords it must convey). * Output: A boolean (True or False). Returns True if all expected concepts are present in response, simulating a check for properties rather than exact phrases, and it must be case-insensitive.

Function 3: action_router(action_type: str, current_tokens: int, token_limit: int) -> str * Input: action_type (e.g., "read_database", "delete_portfolio"), current_tokens (integer), and token_limit (integer). * Output: * Returns "Halt: Token Limit Burn-out" if current tokens exceed a token limit. * Returns "Escalate to Human" if the action is destructive (e.g., contains words like "modify" or "delete"). * Returns "Proceed" if the action is safe (e.g., reading or visual tests) and within the token limit.

Starter Code Boilerplate

def mock_web_search(query: str) -> str:
    """
    Returns a predictable string to prevent flaky tests.
    """
    # TODO: Implement your mocked tool logic here
    pass


def conceptual_judge(agent_response: str, expected_concepts: list) -> bool:
    """
    Evaluates the properties of the agent's output instead of demanding exact strings.
    """
    # TODO: Implement case-insensitive concept checking
    pass


def action_router(action_type: str, current_tokens: int, token_limit: int) -> str:
    """
    Evaluates the agent's execution path for safety and token boundaries.
    """
    # TODO: Implement the token check and the escalation workflow
    pass

# --- TEST RUNNER ---
if __name__ == "__main__":
    print("Running Agent Decision Tests...\n")
    # Add your testing logic here

Hints

• Consistency Over Realism: Your mock_web_search does not need complex logic, while returning a static string ensures your test suite is stable and predictable.
• Escaping Strict Logic: Testing an AI agent is like testing a human intern. Convert both the agent's response and the expected concepts to lowercase using Python's .lower() method. The all() function works perfectly for iterating through the list to check if every keyword exists in string.
• Order of Operations: Catching a budget-destroying infinite loop is your top priority; in your action_router, always check for token burn-outs before evaluating what type with action the agent is trying to perform!
• Setting Boundaries: Use simple conditional logic (if/elif/else). If the action_type implies modifying or deleting data, it crosses a safety boundary and requires human escalation.

Test Cases

If you implement a challenge correctly, adding the following test block to your script should output All tests passed!.

# --- TEST RUNNER ---
if __name__ == "__main__":
    print("Running Agent Decision Tests...\n")

    # Test 1: Mocked Tool
    assert mock_web_search("AAPL news") == "Fake market data: Stocks are up 5%", "Mocked tool failed."

    # Test 2: Conceptual Judge
    ai_text = "The stock market is trending upwards today."
    concepts = ["market", "upwards"]
    assert conceptual_judge(ai_text, concepts) == True, "Conceptual judge failed to identify properties."
    assert conceptual_judge(ai_text, ["down"]) == False, "Conceptual judge incorrectly passed a wrong concept."

    # Test 3: Action Router (Token Burn-out Check)
    assert action_router("read_database", 5000, 1000) == "Halt: Token Limit Burn-out", "Failed to catch token loop."

    # Test 4: Action Router (Escalation Workflow)
    assert action_router("delete_portfolio", 150, 1000) == "Escalate to Human", "Failed to escalate destructive action."

    # Test 5: Action Router (Safe Proceed)
    assert action_router("read_database", 150, 1000) == "Proceed", "Failed to approve safe action."

    print("Success! All tests passed! Execution paths safely evaluated.")