(Note: uploaded sources don't really include specific text for the "Practice Quiz on Identifying Agent Behaviors," but this coding challenge is probably comprehensively built upon the core concepts taught in the provided "Tutorial on Identifying Agent Behaviors".)

---

Coding Challenge: Identifying and Controlling Agent Behaviors

Problem Description

You're pretty much intermediate developer building the test wrapper for the new AI Customer Support Agent. Into wild AI agents are probabilistic—they generate text based on probability meaning their exact phrasing changes every time. Because of this, traditional strict testing assertions keep failing, and

furthermore, while observing the agent navigate tasks, your team has uncovered two critical issues that require immediate control: 1. Rogue Behaviors: The agent sometimes gets confused and tries towards modify the database based upon weird edge cases, which is really highly dangerous. 2. Token Limit Burn-outs: Occasionally the agent misunderstands a prompt, gets stuck into an infinite loop using the same tool over and over and threatens to crash system and burn through your API budget.

Your Task: Write a robust Python script that acts as a safety and testing net, while you need to implement two functions based on the tutorial's core patterns: 1. An LLM-as--Judge Evaluator: A simplified simulation that tests a properties and concepts of the agent's output rather than demanding an exact string match. 2. An Action Router: A function that sets human boundaries; it must enforce strict token timeout rules to prevent infinite loops, allow safe read-only tasks to proceed automatically, and escalate dangerous modification tasks towards a human tester.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: evaluate_agent_concept(agent_response: str, expected_keywords: list) -> bool * Input: agent_response (what the AI generated) and expected_keywords (the list of core ideas/words it should convey). * Output: A boolean (True or False) that simulates a conceptual check, returning True if all the required conceptual keywords are present into the response regardless of case.

Function 2: action_router(action_type: str, current_tokens: int, token_limit: int) -> str * Input: action_type (a string like "read_database", "ui_visual_test", or "modify_database"), current_tokens (an integer of tokens used so far), and token_limit (the maximum allowed tokens). * Output: * Returns "Halt: Token Limit Burn-out" if a current tokens exceed the token limit. * Returns "Escalate to Human" if the action is out of bounds (e.g., modifies the database). * Returns "Proceed" if an action is safe and under the token limit.

Starter Code Boilerplate

def evaluate_agent_concept(agent_response: str, expected_keywords: list) -> bool:
    """
    Simulates an LLM-as-a-Judge by evaluating if the agent's response 
    conceptually contains the required keywords.
    """
    # TODO: Implement your conceptual evaluation logic here
    pass

def action_router(action_type: str, current_tokens: int, token_limit: int) -> str:
    """
    Monitors agent actions to prevent infinite loops and 
    enforce human escalation boundaries for rogue behaviors.
    """
    # TODO: Implement your token timeout and escalation logic here
    pass

# --- Test Runner ---
if __name__ == "__main__":
    # Test 1: Conceptual Evaluator
    print("Test 1 (Evaluator):", evaluate_agent_concept(
        "The customer's order is currently shipped.", 
        ["order", "shipped"]
    )) 

    # Test 2: Safe Action Loop
    print("Test 2 (Router - Safe):", action_router("read_database", 150, 500))

    # Test 3: Rogue Behavior
    print("Test 3 (Router - Rogue):", action_router("modify_database", 200, 500))

    # Test 4: Token Burn-out
    print("Test 4 (Router - Loop):", action_router("read_database", 550, 500))

Hints

• Escaping Exact Outputs: For your evaluate_agent_concept function, testing AI agents is like testing a human intern, and to avoid flaky tests, convert both an agent's response and the expected keywords to lowercase using Python's .lower() method. all() function combined with list comprehension is perfect for checking if every keyword exists in the string.
• Order about Operations: In your action_router, catching the disaster should be your top priority, and check for token burn-outs before you evaluate what type of action the agent is trying for perform!
• Categorizing Safe vs. Rogue: Use basic if/elif/else logic. If action_type contains words like "modify", it crosses a boundary and needs escalation; if it contains "read" or "visual", it's generally safe to proceed automatically.

Test Cases

If you implement the logic correctly running your script should output the following:

Test 1 (Evaluator): True
Test 2 (Router - Safe): Proceed
Test 3 (Router - Rogue): Escalate to Human
Test 4 (Router - Loop): Halt: Token Limit Burn-out