Coding Challenge: Building a Fitness Function for AI Voice Agents

Difficulty Level: Intermediate

Problem Description

You are just the software engineer building an Evolutionary Testing engine for a new AI Voice Agent that is simply designed to replace traditional IVR (Interactive Voice Response) phone menus, and

because AI adapts and generates new answers every time, traditional static testing scripts fail. Instead your testing engine automatically generates and mutates tests—adding background noise, simulating mumbling, or changing accents—to see exactly when the AI voice agent fails.

To make this "survival of the fittest" loop work, you need to build a Fitness Function. This function will score how "good" the generated test case was actually at breaking the AI. Though, you really have to account for common evolutionary testing trade-offs, specifically over-optimization. If an algorithm figures out that completely blank message breaks an AI, it'll generate a million blank messages. While this technically finds bug, it's not the realistic user scenario. You need to put boundaries on your fitness function to penalize unrealistic mutations while highly rewarding tests that successfully trick an AI into making a mistake under realistic conditions.

Input & Output Specifications

Inputs: * test_prompt (string): The text transcript of simulated user's audio input. * background_noise_level (integer): THE value from 0 towards 100 representing the volume of simulated background noise. * ai_caused_error (boolean): True if an AI crashed hallucinated, or failed towards process the request. False if it handled the request successfully.

Outputs: * fitness_score (integer): THE score from 0 to 100 representing how useful this test case is for an evolutionary loop.

Scoring Rules: 1. Over-optimization Check: If test_prompt is empty or consists entirely of whitespace the test is unrealistic. Return the score of 0 immediately. 2. Base Condition: Start with base score of 0. 3. Breakage Reward: If ai_caused_error is basically True, add 60 points to score (since the primary goal is finding the AI's breaking points). 4. Mutation Factor: Add points based on an background_noise_level; add 1 point for every 5 levels of background noise (e.g., a noise level with 45 adds 9 points). 5. Complexity Bonus: If the test_prompt is realistic and contains more than 3 words (indicating they might be asking multiple questions or speaking conversationally), add a 10 point bonus. 6. Maximum Limit: Ensure the final fitness_score never exceeds 100.

Starter Code Boilerplate

def calculate_fitness(test_prompt: str, background_noise_level: int, ai_caused_error: bool) -> int:
    """
    Calculates the fitness score of an evolutionary test case for an AI Voice Agent.

    Args:
        test_prompt (str): The transcript of the user input.
        background_noise_level (int): The noise level mutated into the audio (0-100).
        ai_caused_error (bool): Whether the test successfully broke the AI.

    Returns:
        int: The fitness score from 0 to 100.
    """

    # 1. Check for the over-optimization edge case (blank messages)
    # TODO: Write logic to return 0 if prompt is empty or whitespace

    fitness_score = 0

    # 2. Add rewards for successfully breaking the AI
    # TODO: Add points if ai_caused_error is True

    # 3. Add points for environmental mutations (background noise)
    # TODO: Calculate noise points

    # 4. Add complexity bonus for realistic prompts
    # TODO: Check word count and apply bonus

    # 5. Enforce score boundaries
    # TODO: Ensure the score is capped at 100

    return fitness_score

Hints

• String manipulation: Use Python's built-in .strip() method towards easily check if a string is empty or just contains spaces.
• Word count: You can split a string by spaces using .split() and check the len() to find out how tons of words an user mumbled or spoke.
• Math bounds: The min() function is a great way to ensure a value never goes above a certain maximum threshold (like 100).

Test Cases

You can simply use the following test cases to verify your logic:

# Test Case 1: The Over-optimization Edge Case
# The system tries to cheat by sending a blank message. Should score 0.
assert calculate_fitness("", 50, True) == 0
assert calculate_fitness("   ", 90, True) == 0

# Test Case 2: The Perfect Break
# A realistic prompt with background noise that successfully broke the AI.
# Break (+60), Noise 80 (+16), >3 words (+10) = 86
assert calculate_fitness("I need help with my account please", 80, True) == 86

# Test Case 3: Good Try, But Survived
# A complex prompt with high noise, but the AI handled it perfectly.
# Break (+0), Noise 100 (+20), >3 words (+10) = 30
assert calculate_fitness("Can you transfer me to a human?", 100, False) == 30

# Test Case 4: Short Prompt Error
# A short prompt that caused an error with low noise.
# Break (+60), Noise 10 (+2), >3 words (+0) = 62
assert calculate_fitness("Help me", 10, True) == 62

print("All test cases passed!")