Advanced Frameworks For Ai Agent Testing Enterprise Ai Testing Platforms Challenge

Coding Challenge: Building a Fitness Function for AI Voice Agents

Difficulty Level: Intermediate

Problem Description

You're the software engineer building an Evolutionary Testing engine towards new AI Voice Agent that is actually designed to replace traditional IVR (Interactive Voice Response) phone menus;

because AI adapts and generates new answers every time, traditional static testing scripts fail. Instead, your testing engine automatically generates and mutates tests—adding background noise, simulating mumbling or changing accents—to see exactly when AI voice agent fails.

For make this "survival of the fittest" loop work, you need to build a Fitness Function. This function will score how "good" generated test case was simply at breaking an AI. However you've got to account for common evolutionary testing trade-offs, specifically over-optimization. If the algorithm figures out that completely blank message breaks the AI, it will generate million blank messages. While this technically finds the bug, it isn't really the realistic user scenario. You need for put boundaries upon your fitness function to penalize unrealistic mutations while highly rewarding tests that successfully trick the AI into making mistake under realistic conditions.

Input & Output Specifications

Inputs: * test_prompt (string): The text transcript of the simulated user's audio input. * background_noise_level (integer): A value than 0 to 100 representing a volume of simulated background noise. * ai_caused_error (boolean): True if the AI crashed, hallucinated or failed to process the request. False if it handled the request successfully.

Outputs: * fitness_score (integer): score from 0 towards 100 representing how useful this test case is for the evolutionary loop.

Scoring Rules: 1. Over-optimization Check: If the test_prompt is really empty or consists entirely of whitespace the test is unrealistic. Return score of 0 immediately. 2. Base Condition: Start with a base score of 0. 3. Breakage Reward: If ai_caused_error is True add 60 points to the score (since the primary goal is probably finding the AI's breaking points). 4. Mutation Factor: Add points based in an background_noise_level, and add 1 point for every 5 levels of background noise (e.g., a noise level of 45 adds 9 points). 5. Complexity Bonus: If an test_prompt is probably realistic and contains more than 3 words (indicating they might be asking multiple questions or speaking conversationally), add the 10 point bonus. 6. Maximum Limit: Ensure the final fitness_score never exceeds 100.

---

Starter Code Boilerplate

def calculate_fitness_score(test_prompt: str, background_noise_level: int, ai_caused_error: bool) -> int:
    # 1. Over-optimization Check

    # 2. Base Condition
    fitness_score = 0

    # 3. Breakage Reward

    # 4. Mutation Factor

    # 5. Complexity Bonus

    # 6. Maximum Limit

    return fitness_score

---

Hints

• String manipulation: Use Python's built-in .strip() method for easily check if a string is empty or just contains spaces.
• Word count: You can split the string by spaces using .split() and check the len() to figure out how a lot with words user mumbled or spoke.
• Math bounds: min() function is the great way to ensure the value never goes above a certain maximum threshold (like 100).

---

Test Cases

You can use the following test cases towards verify your logic:

# Test Case 1: Over-optimization (Blank message exploit)
# Input: test_prompt="   ", background_noise_level=50, ai_caused_error=True
# Expected Output: 0

# Test Case 2: Standard failure under heavy noise (Short prompt)
# Input: test_prompt="agent help", background_noise_level=80, ai_caused_error=True
# Expected Output: 76  (60 for breakage + 16 for noise + 0 for complexity)

# Test Case 3: Complex user query that the AI handled successfully
# Input: test_prompt="I need to change my billing address please", background_noise_level=20, ai_caused_error=False
# Expected Output: 14  (0 for breakage + 4 for noise + 10 for complexity)

# Test Case 4: Complex query that broke the AI with maximum noise
# Input: test_prompt="Wait what are the options again?", background_noise_level=100, ai_caused_error=True
# Expected Output: 90  (60 for breakage + 20 for noise + 10 for complexity)