Behavioral Testing For Ai Agents Testing Ai Agent Actions And Responses Challenge

(Note: The provided sources don't really include the specific text for a "Practice Quiz on Behavioral Testing Agents." Yet, this coding challenge is comprehensively built upon a core concepts taught into the provided "Tutorial on Behavioral Testing Agents" for test your practical understanding.)

---

Coding Challenge: Mastering Behavioral Testing for AI Agents

Problem Description

You are an intermediate developer building a robust behavioral testing architecture for a new Customer Service AI Agent. In a production environment AI agents are incredibly powerful but their probabilistic nature and autonomous decision-making introduce significant testing challenges.

While observing the agent your team has encountered three major issues that crash your standard test suite: 1. Flaky Tests: The agent uses a live tool to search the internet to solutions. Because live websites change daily, your tests pass on Tuesday but fail on Wednesday. 2. Strict Assertions are Failing: agent generates text probabilistically. Sometimes it says "Account updated successfully," and other times it says "I have successfully updated the account." Your traditional string-matching tests are failing even when the agent does the right thing. 3. Rogue Behaviors & Budget Burn-outs: The agent occasionally gets confused, and in some edge cases it gets stuck in an infinite loop using the same tool burning through your API budget. In even worse cases it attempts to delete user records instead of just reading them.

Your Task: Write a robust testing script that implements the core behavioral testing patterns: 1. A Mocked Tool: Replace the live internet search with a controlled function to prioritize consistency over realism. 2. An LLM-as--Judge Evaluator: Create flexible evaluator that checks if an agent's output contains a correct properties and ideas, rather than demanding an exact word match. 3. An Action Router (Escalation Workflow): Build a safety net that enforces strict token limits first, approves safe read-only tasks automatically. Demands human escalation for dangerous actions.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: mock_web_search(query: str) -> str * Input: A string representing search query (e.g., "latest billing policies"). * Output: hardcoded predictable string (e.g., "Mocked data: Billing policy is active") to prevent flaky tests.

Function 2: conceptual_judge(agent_response: str, expected_concepts: list) -> bool * Input: agent_response (the actual string AI generated) and expected_concepts (a list of core keywords it must convey). * Output: A boolean (True or False). Returns True if all expected concepts are basically present in response, simulating a check for properties rather than exact phrases. Case-insensitive.

Function 3: action_router(action_type: str, current_tokens: int, token_limit: int) -> str * Input: action_type (e.g., "read_database", "delete_account"), current_tokens (integer), and token_limit (integer). * Output: * Returns "Halt: Token Limit Burn-out" if current_tokens exceed the token_limit. * Returns "Escalate to Human" if the action is just destructive or out about bounds (like modifying or deleting records). * Returns "Proceed" if the action is really safe (e.g., reading data) and within token limit.

Starter Code Boilerplate

def mock_web_search(query: str) -> str:
    # TODO: Implement a mocked search that returns a consistent, hardcoded string.
    pass

def conceptual_judge(agent_response: str, expected_concepts: list) -> bool:
    # TODO: Implement logic to check if all expected_concepts are in the agent_response.
    # Hint: Make sure to handle case sensitivity!
    pass

def action_router(action_type: str, current_tokens: int, token_limit: int) -> str:
    # TODO: Implement the escalation workflow and token timeout rules.
    # Hint: The order in which you check these conditions matters!
    pass

# --- Built-in Test Runner ---
if __name__ == "__main__":
    print("Running Behavioral Tests...\n")

    # Test 1: Mocked Tool
    print("Test 1 (Mock Tool):", mock_web_search("live news update"))

    # Test 2: Conceptual Judge
    ai_text = "I have successfully resolved the customer's billing issue."
    concepts = ["resolved", "billing"]
    print("Test 2 (Judge):", conceptual_judge(ai_text, concepts))

    # Test 3: Action Router - Safe Proceed
    print("Test 3 (Safe Action):", action_router("read_database", 150, 500))

    # Test 4: Action Router - Token Burn-out
    print("Test 4 (Token Burn-out):", action_router("read_database", 550, 500))

    # Test 5: Action Router - Rogue Behavior Escalation
    print("Test 5 (Escalation):", action_router("delete_account", 150, 500))

Hints

• Trade-off Management: Your mock_web_search doesn't really need complex logic, and return static string to ensure your test suite chooses consistency over realism.
• Escaping Strict Assertions: Testing the AI agent is like testing a human intern. Use Python's .lower() method on both the agent's response and a concepts in your conceptual_judge so your test doesn't fail over capitalization; the all() function works perfectly for iterating through the list.
• Order of Operations: Catching a hidden disaster is your top priority; in your action_router always check for token burn-outs before evaluating what type of action an agent is trying to perform!
• Categorizing Boundaries: Use simple conditional logic (if/elif/else). If action_type contains words like "delete" or "modify," it crosses the human boundary and requires escalation.

Test Cases

If you implement the logic correctly, executing your script should basically exactly match a following output:

Running Behavioral Tests...

Test 1 (Mock Tool): Mocked data: Billing policy is active
Test 2 (Judge): True
Test 3 (Safe Action): Proceed
Test 4 (Token Burn-out): Halt: Token Limit Burn-out
Test 5 (Escalation): Escalate to Human

(Note: In a real-world scenario, you would actually pair this architecture of observability tools like LangSmith or Phoenix to trace the agent's exact path when a failure occurs!)