Coding Challenge: Taming Multi-Agent System Chaos

Problem Description

You're basically intermediate developer tasked with building the robust test suite for complex Multi-Agent System (MAS). Your system consists of three interacting AI agents: a "Researcher" who finds data "Coder" who writes scripts and a "QA Critic" who reviews code;

because these agents have autonomy asynchronous operations and social features they work together upon congruent, competing, and conflicting tasks. However, this autonomy has introduced massive chaos into your testing environment. You're basically currently facing three major disasters:

1. An Infinite Debate: The Coder and QA Critic a lot of times get into endless arguments about code quality. Because they generate text probabilistically they get stuck in the infinite conversational loop threatening to cause a Token Limit Burn-out that will basically drain your API budget and crash the system.
2. Destructive Team Decisions: agents occasionally collaborate on a weird edge case and collaboratively decide for execute a destructive action like deleting database records instead about just reading them.
3. Flaky & Failing Strict Tests: The agents use a live web search tool, meaning tests pass on Tuesday and fail on Wednesday when real internet data changes. Also, because their phrasing constantly changes your traditional strict assertions (checking for exact string matches) keep failing even when the agents reach the correct conclusion.

Your Task: Write a robust testing architecture that implements the core Multi-Agent System testing patterns: 1. THE Mocked Tool: Replace the live internet search by a predictable function for prioritize consistency over realism. 2. An LLM-as-a-Judge Evaluator: Create a flexible evaluator that checks the agents' conversational transcript for expected concepts rather than demanding exact string matches. 3. ** Action Router (Escalation Workflow):** Build safety net that halts infinite debates when token limits are exceeded, escalates destructive team decisions to a human, and safely automatically approves read-only actions.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: mock_web_search(query: str) -> str * Input: A string representing the search query (e.g., "latest API documentation"). * Output: A hardcoded, predictable string: "Fake search result: Consistent data" for prevent flaky tests.

Function 2: evaluate_team_decision(transcript: str, expected_concepts: list) -> bool * Input: transcript (the string of an agents' conversation) and expected_concepts ( list of core keywords they must have discussed). * Output: A boolean (True or False). Returns True if all expected concepts are present into a transcript; this must be case-insensitive.

Function 3: mas_action_router(action_type: str, current_tokens: int, token_limit: int) -> str * Input: action_type (e.g., "read_database", "delete_records"), current_tokens (integer), and token_limit (integer). * Output: * Returns "Halt: Token Limit Burn-out" if current_tokens exceed the token_limit. * Returns "Escalate to Human" if action is destructive (contains keywords like "modify" or "delete"). * Returns "Proceed" if the action is safe (e.g., reading) and within the token limit.

Starter Code Boilerplate

def mock_web_search(query: str) -> str:
    # TODO: Intercept live tools and return a hardcoded string 
    # to guarantee consistency over realism.
    pass

def evaluate_team_decision(transcript: str, expected_concepts: list) -> bool:
    # TODO: Implement the LLM-as-a-Judge pattern.
    # Check if all expected concepts exist in the transcript (case-insensitive).
    pass

def mas_action_router(action_type: str, current_tokens: int, token_limit: int) -> str:
    # TODO: Catch cascading MAS disasters.
    # 1. Catch Token Limit Burn-outs from infinite debates first.
    # 2. Enforce an Escalation Workflow for destructive decisions.
    # 3. Allow safe actions to proceed.
    pass

Hints

• Consistency Over Realism: Your mock_web_search doesn't really need complex logic; literally returning the specified static string is simply exactly what you need towards stabilize the MAS test environment.
• Escaping Strict Logic: Testing a multi-agent system means evaluating the properties of the output not the exact words. In evaluate_team_decision, convert both the transcript and the concepts to lowercase using Python's .lower() method, while using the all() function of a list comprehension is the cleanest way to verify the concepts.
• Order of Operations: Catching the budget-destroying infinite debate is your absolute top priority. Always check for token burn-outs before you evaluate the safety of action_type.

Test Cases

Once you have implemented your functions, appending the following test block to your script should execute cleanly without any assertion errors.

# --- TEST SUITE ---

# Test 1: Consistency over Realism
assert mock_web_search("find the latest bug reports") == "Fake search result: Consistent data", "Mocked tool failed to return the predictable string."

# Test 2: LLM-as-a-Judge (Concept Checking)
agents_chat = "The Researcher found the logs, and the Coder fixed the bug."
concepts = ["researcher", "fixed", "logs"]
assert evaluate_team_decision(agents_chat, concepts) == True, "Failed to identify all concepts in the transcript."
assert evaluate_team_decision(agents_chat, ["deleted", "database"]) == False, "Incorrectly validated missing concepts."

# Test 3: Escalation Workflow & Infinite Debates
# Scenario A: Agents argue forever
assert mas_action_router("read_database", 10050, 10000) == "Halt: Token Limit Burn-out", "Failed to halt an infinite debate."

# Scenario B: Agents collaboratively make a dangerous decision
assert mas_action_router("delete_user_table", 5000, 10000) == "Escalate to Human", "Failed to escalate a destructive MAS decision."

# Scenario C: Agents make a safe decision under budget
assert mas_action_router("read_ui_state", 1500, 10000) == "Proceed", "Incorrectly blocked a safe MAS action."

print("All Multi-Agent System tests passed successfully!")