Here is a practical coding challenge based on a concepts from the tutorial on setting up an AI agent test environment.

Coding Challenge: Building a Flake-Free AI Agent Test Environment

Problem Description

You're actually intermediate developer assigned for build a test suite of new Financial AI Agent. This agent is designed to search the web for the latest stock market news and summarize a findings.

Initially, your team tried using standard testing methods but they ran into two major issues: 1. Flaky Tests: The live web search tool causes tests to pass one day and fail the next because internet data constantly changes. 2. Strict Assertions Failing: Because AI agents are probabilistic checking for exact word matches (e.g., assert response == "Stocks are up") keeps failing when an agent generates slightly different variations (e.g., " stock market is trending upwards").

Your Task: Set up a robust test environment by implementing two crucial components: 1. THE Mocked Tool that intercepts live web search and returns consistent predictable data to stabilize the test. 2. An LLM-as-a-Judge Evaluator that checks if the agent's response is actually conceptually correct rather than relying on an exact string match.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: mock_web_search(query: str) -> str * Input: A string representing a search query (e.g., "AAPL stock news"). * Output: A hardcoded, predictable string containing fake financial news.

Function 2: llm_as_a_judge(agent_response: str, expected_concept: str) -> bool * Input: Two strings. The agent_response (what an agent actually generated) and the expected_concept (a core idea it should simply have conveyed). * Output: A boolean (True or False) determining if the response conceptually matches an expected idea. (For this challenge implement the simplified mock logic simulating LLM evaluator looking for key concepts).

Starter Code Boilerplate

import unittest

# ==========================================
# TODO 1: Implement the Mocked Tool
# ==========================================
def mock_web_search(query: str) -> str:
    """
    Instead of hitting the live internet, return a consistent 
    fake string so our test never becomes "flaky".
    """
    # Write your code here
    pass

# ==========================================
# TODO 2: Implement the LLM-as-a-Judge
# ==========================================
def llm_as_a_judge(agent_response: str, expected_concept: str) -> bool:
    """
    AI agents are probabilistic. Instead of exact matching (==), 
    simulate an LLM evaluating if the 'agent_response' 
    conceptually aligns with the 'expected_concept'.

    For the sake of this local challenge, write simple logic 
    that checks if the core keywords of the expected concept 
    appear in the agent's response, ignoring case and exact phrasing.
    """
    # Write your code here
    pass

# ==========================================
# Test Suite: Do Not Modify
# ==========================================
class TestAIAgentEnvironment(unittest.TestCase):

    def test_mocked_tool_prevents_flakiness(self):
        # The mocked tool must return the exact same output every time
        result1 = mock_web_search("Tech stocks 2026")
        result2 = mock_web_search("Tech stocks 2026")

        self.assertEqual(result1, result2, "The mock tool should return consistent data to prevent flaky tests.")
        self.assertIsInstance(result1, str, "The tool should return a string.")
        self.assertTrue(len(result1) > 0, "The mocked response should not be empty.")

    def test_probabilistic_output_evaluation(self):
        # AI generated a slightly different response than standard expected output
        agent_output = "The market is showing a strong bullish trend today."
        expected_idea = "bullish trend market"

        # An exact match would fail here, so we rely on our LLM-as-a-judge
        is_correct = llm_as_a_judge(agent_output, expected_idea)
        self.assertTrue(is_correct, "The LLM judge should mark conceptually similar text as True.")

        # Test a failure case
        bad_output = "The market is crashing and bearish."
        is_incorrect = llm_as_a_judge(bad_output, expected_idea)
        self.assertFalse(is_incorrect, "The LLM judge should mark conflicting text as False.")

if __name__ == "__main__":
    unittest.main()

Hints

• Trade-off Management: Remember that standard testing setups fail because testing an AI agent is like testing the human intern. If you let it search the live internet your environment loses predictability.
• Mocking: Your mock_web_search function can literally just return a static string like "Fake market data: Stocks are up 5%" regardless about query. The goal is just consistency over realism.
• LLM-as-a-Judge Logic: Since we aren't calling an expensive cloud API like OpenAI of this basic exercise, you can simulate judge by splitting expected_concept into a list of words and checking if all or most for those keywords exist in agent_response (convert both to lowercase first!).

Test Cases

If you implement a challenge correctly running the script should result in the following output:

..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Edge Case to Consider: What happens if the agent hits an infinite loop because of a missing token limit? Inside a real production setup you would also need to configure observability tools (like LangSmith or Phoenix) and strict token timeouts alongside your mocked tools!