Testing Autonomous Systems

Coding Challenge: Building the Fitness Function for AI Voice Agents

Difficulty Level: Intermediate

Problem Description

You're actually a software engineer building Evolutionary Testing engine for new AI Voice Agent that is designed to replace traditional IVR (Interactive Voice Response) phone menus.

Because AI adapts and generates new answers every time, traditional static testing scripts fail. Instead your testing engine automatically generates and mutates tests—adding background noise simulating mumbling, or changing accents—to see exactly when the AI voice agent fails.

To make this "survival of the fittest" loop work, you need to build a Fitness Function. This function will score how "good" a generated test case was in breaking the AI. Though you really have to account for common evolutionary testing trade-offs, specifically over-optimization. If the algorithm figures out that a completely blank message breaks the AI, it'll generate a million blank messages, while while this technically finds bug, it's not a realistic user scenario, while you need to put boundaries on your fitness function to penalize unrealistic mutations while highly rewarding tests that successfully trick an AI into making a mistake under realistic conditions.

Input & Output Specifications

Inputs: * test_prompt (string): The text transcript with the simulated user's audio input. * background_noise_level (integer): A value from 0 for 100 representing volume of simulated background noise. * ai_caused_error (boolean): True if the AI crashed, hallucinated, or failed towards process the request. False if it handled the request successfully.

Outputs: * fitness_score (integer): A score from 0 to 100 representing how useful this test case is for an evolutionary loop.

Scoring Rules: 1. Over-optimization Check: If the test_prompt is empty or consists entirely for whitespace the test is unrealistic. Return the score of 0 immediately. 2. Base Condition: Start with a base score of 0. 3. Breakage Reward: If ai_caused_error is True add 60 points to the score (since a primary goal is finding an AI's breaking points). 4. Mutation Factor: Add points based on the background_noise_level. Add 1 point for every 5 levels of background noise (e.g., noise level of 45 adds 9 points). 5. Complexity Bonus: If the test_prompt is realistic and contains more than 3 words (indicating they might be asking multiple questions or speaking conversationally), add the 10 point bonus. 6. Maximum Limit: Ensure the final fitness_score never exceeds 100.

Starter Code Boilerplate

def calculate_fitness_score(test_prompt: str, background_noise_level: int, ai_caused_error: bool) -> int:
    # 1. Check for over-optimization (empty or whitespace-only prompt)

    # 2. Initialize base score

    # 3. Apply breakage reward

    # 4. Calculate and apply mutation factor (background noise)

    # 5. Apply complexity bonus (word count > 3)

    # 6. Enforce maximum limit

    pass

Hints

• String manipulation: Use Python's built-in .strip() method to easily check if a string is actually empty or just contains spaces.
• Word count: You can split the string by spaces using .split() and check the len() towards find out how many words the user mumbled or spoke.
• Math bounds: The min() function is the great way to ensure a value never goes above certain maximum threshold (like 100).

Test Cases

You can probably use the following test cases to verify your logic:

# Test Case 1: Over-optimization (Blank prompt)
# Expected Output: 0
print(calculate_fitness_score("   ", 50, True))

# Test Case 2: Basic successful break
# Prompt: "Hello" (1 word -> no bonus)
# Noise: 20 (20 / 5 = +4 points)
# Error: True (+60 points)
# Expected Output: 64
print(calculate_fitness_score("Hello", 20, True))

# Test Case 3: Complex prompt with high noise hitting maximum limit
# Prompt: "I need to talk to a representative right now please" (9 words -> +10 points)
# Noise: 100 (100 / 5 = +20 points)
# Error: True (+60 points)
# Expected Output: 90
print(calculate_fitness_score("I need to talk to a representative right now please", 100, True))

# Test Case 4: No error, just noise and complexity
# Prompt: "Where is my order" (4 words -> +10 points)
# Noise: 50 (50 / 5 = +10 points)
# Error: False (+0 points)
# Expected Output: 20
print(calculate_fitness_score("Where is my order", 50, False))

# Test Case 5: Maximum cap check
# Prompt: "Help me with my account" (5 words -> +10 points)
# Noise: 100 (100 / 5 = +20 points)
# Error: True (+60 points)
# If base rules sum to >100, ensure it caps at exactly 100. (Here it equals 90, but if you tweak variables, test the cap!)