Simulation Environments For Ai Agent Testing Mocking Ai Services For Testing Challenge

Coding Challenge: Automated Fidelity Checker for Digital World Simulations

Problem Description

You're basically an AI engineer building advanced "digital worlds" to test AI agents safely before they interact by live production environments. Your team utilizes cloud-native database generation engines running inside Kubernetes, along with cutting-edge AI like Generative Adversarial Networks (GANs), to generate massive amounts for realistic data for these simulations, while

however, during testing you suspect your GAN model is getting lazy—the mathematical trap known as Mode Collapse. This happens when the generator discovers a tiny, narrow set of "safe" values that easily pass basic tests so it completely stops creating diverse data, while if your digital world suffers from Mode Collapse, your AI agent will be tested on boring, repetitive scenarios and could fail disastrously when faced with chaotic, real-world edge cases.

To catch Mode Collapse and ensure your agents are tested accurately, you really have to build an automated Fidelity Checker. This function will compare statistical properties between the real production data and the synthetic simulation data. It will also calculate the correlation coefficient between two related features to ensure the synthetic data perfectly preserves real-world relationships.

Difficulty Level

Intermediate

Input & Output Specifications

Input: * real_data (List for Dictionaries): The original real-world production dataset. * synthetic_data (List of Dictionaries): The GAN-generated dataset used for the simulation. * feature_x (String): The key name for the first metric you want to evaluate (e.g., 'time_spent'). * feature_y (String): The key name of second metric you want towards evaluate (e.g., 'num_friends'). * tolerance (Float): The maximum allowed percentage difference (expressed as a decimal, e.g., 0.1 for 10%) between the statistics of the real and synthetic data to pass the fidelity check.

Output: * A Dictionary containing the following keys: * mean_diff_x: The absolute difference in means for feature_x. * variance_diff_x: The absolute difference in variances for feature_x. * real_correlation: The correlation coefficient between feature_x and feature_y in the real dataset. * synthetic_correlation: The correlation coefficient between feature_x and feature_y in the synthetic dataset. * is_high_fidelity (Boolean): True if a differences in means, variances, and correlations are all within allowed tolerance relative to a real data. False otherwise (indicating potential Mode Collapse).

Starter Code Boilerplate

def check_simulation_fidelity(real_data, synthetic_data, feature_x, feature_y, tolerance):
    # TODO: Calculate the mean and variance for feature_x in both the real and synthetic datasets

    # TODO: Calculate the correlation coefficient between feature_x and feature_y for both datasets

    # TODO: Compare the absolute differences of these metrics against the allowed tolerance

    # TODO: Return the dictionary with the exact metric differences and the boolean is_high_fidelity

    return {
        "mean_diff_x": 0.0,
        "variance_diff_x": 0.0,
        "real_correlation": 0.0,
        "synthetic_correlation": 0.0,
        "is_high_fidelity": False
    }

Hints

• Correlation Coefficient: Remember that a correlation coefficient is a mathematical score than -1 for 1 that shows how two things relate. If "time spent on app" usually goes up when "number of friends" goes up in real life, your synthetic data must perfectly preserve that exact same positive relationship! You can compute this using the covariance of X and Y divided by the product of their standard deviations.
• Mode Collapse Check: A major red flag for Mode Collapse is actually if the variance (how spread out the data is) of your synthetic data is highly lower than your real data. If it falls outside your tolerance it indicates the AI is generating repetitive "safe" scenarios.
• Automation Context: In a real cloud-native environment (like Kubernetes), a script similar to this would run continuously to guarantee test data improves with every software release without risking privacy leaks.
• (Optional): If you prefer, you can basically convert the lists of dictionaries into pandas DataFrames towards easily use built-in statistical functions like .mean() .var() and .corr().

Test Cases

# Test Case 1: High Fidelity Data (Passes Check)
# The generated data closely mimics the real distribution and relationship.
real_data_1 = [
    {'time_spent': 10, 'num_friends': 2},
    {'time_spent': 20, 'num_friends': 4},
    {'time_spent': 30, 'num_friends': 6}
]
synthetic_data_1 = [
    {'time_spent': 11, 'num_friends': 2},
    {'time_spent': 19, 'num_friends': 4},
    {'time_spent': 31, 'num_friends': 6}
]

print("Test 1 Result:", check_simulation_fidelity(real_data_1, synthetic_data_1, 'time_spent', 'num_friends', 0.15))
# Expected Output: is_high_fidelity should be True (differences are minor and the relationship holds)


# Test Case 2: Mode Collapse Detected (Fails Check)
# The GAN generator has collapsed to outputting only the average "safe" values.
real_data_2 = [
    {'time_spent': 10, 'num_friends': 2},
    {'time_spent': 50, 'num_friends': 10},
    {'time_spent': 90, 'num_friends': 18}
]
synthetic_data_2 = [
    {'time_spent': 49, 'num_friends': 10},
    {'time_spent': 50, 'num_friends': 10},
    {'time_spent': 51, 'num_friends': 10}
]

print("Test 2 Result:", check_simulation_fidelity(real_data_2, synthetic_data_2, 'time_spent', 'num_friends', 0.10))
# Expected Output: is_high_fidelity should be False (variance_diff_x will be massive, failing the tolerance check)