Automating Agent Tests

Coding Challenge: Automated Fidelity Checker for Digital World Simulations

Problem Description

You're an AI engineer building advanced "digital worlds" to test AI agents safely before they interact with live production environments, and your team utilizes cloud-native database generation engines running in Kubernetes along with cutting-edge AI like Generative Adversarial Networks (GANs), to generate massive amounts of realistic data for these simulations.

Though during testing, you suspect your GAN model is getting lazy—a mathematical trap known as Mode Collapse. This happens when the generator discovers a tiny, narrow set of "safe" values that easily pass basic tests. It completely stops creating diverse data, while if your digital world suffers out of Mode Collapse your AI agent will be tested at boring, repetitive scenarios and could fail disastrously when faced with chaotic real-world edge cases.

For catch Mode Collapse and ensure your agents are tested accurately you must build an automated Fidelity Checker. This function will basically compare statistical properties between the real production data and the synthetic simulation data. It will also calculate a correlation coefficient between two related features to ensure synthetic data perfectly preserves real-world relationships.

Difficulty Level

Intermediate

Input & Output Specifications

Input: * real_data (List of Dictionaries): The original, real-world production dataset. * synthetic_data (List of Dictionaries): GAN-generated dataset used for the simulation. * feature_x (String): The key name of first metric you want towards evaluate (e.g., 'time_spent'). * feature_y (String): The key name for the second metric you want to evaluate (e.g., 'num_friends'). * tolerance (Float): The maximum allowed percentage difference (expressed as a decimal e.g., 0.1 for 10%) between the statistics of the real and synthetic data towards pass the fidelity check.

Output: * THE Dictionary containing following keys: * mean_diff_x: The absolute difference into means for feature_x. * variance_diff_x: The absolute difference in variances for feature_x. * real_correlation: The correlation coefficient between feature_x and feature_y in the real dataset. * synthetic_correlation: The correlation coefficient between feature_x and feature_y in the synthetic dataset. * is_high_fidelity (Boolean): True if percentage differences in means, variances and correlations are all within the allowed tolerance relative to the real data, while false otherwise (indicating potential Mode Collapse).

---

Starter Code Boilerplate

import pandas as pd

def check_fidelity(real_data, synthetic_data, feature_x, feature_y, tolerance):
    # TODO: Convert the lists of dictionaries to a format suitable for analysis (e.g., pandas DataFrames)

    # TODO: Calculate the mean and variance for feature_x in both datasets

    # TODO: Calculate the correlation coefficient between feature_x and feature_y for both datasets

    # TODO: Compute the absolute differences

    # TODO: Determine if the differences are within the acceptable tolerance percentage

    # TODO: Return the results in the specified dictionary format

    return {
        "mean_diff_x": 0.0,
        "variance_diff_x": 0.0,
        "real_correlation": 0.0,
        "synthetic_correlation": 0.0,
        "is_high_fidelity": False
    }

---

Hints

• Correlation Coefficient: Remember that the correlation coefficient is a mathematical score from -1 to 1 that shows how two things relate. If "time spent on app" usually goes up when "number of friends" goes up in real life your synthetic data must perfectly preserve that exact same positive relationship! You can compute this using the covariance of X and Y divided by the product of their standard deviations.
• Mode Collapse Check: A major red flag for Mode Collapse is if the variance (how spread out the data is) of your synthetic data is highly lower than your real data. If it falls outside your tolerance, it indicates the AI is generating repetitive "safe" scenarios.
• Automation Context: In a real cloud-native environment (like Kubernetes), the script similar to this would run continuously to guarantee test data improves with every software release without risking privacy leaks.
• (Optional): If you prefer you can convert the lists of dictionaries into pandas DataFrames to easily use built-in statistical functions like .mean(), .var() and .corr().

---

Test Cases

Test Case 1: High Fidelity Data (Pass)

real_data = [
    {"time_spent": 10, "num_friends": 2},
    {"time_spent": 20, "num_friends": 4},
    {"time_spent": 30, "num_friends": 6},
    {"time_spent": 40, "num_friends": 8}
]

synthetic_data = [
    {"time_spent": 11, "num_friends": 2},
    {"time_spent": 19, "num_friends": 4},
    {"time_spent": 31, "num_friends": 6},
    {"time_spent": 39, "num_friends": 8}
]

# feature_x = "time_spent", feature_y = "num_friends", tolerance = 0.15
# Expected Output: 'is_high_fidelity' should be True

Test Case 2: Mode Collapse Detected (Fail - Low Variance)

real_data = [
    {"time_spent": 10, "num_friends": 2},
    {"time_spent": 50, "num_friends": 10},
    {"time_spent": 100, "num_friends": 20}
]

# The GAN got lazy and only outputs middle-ground numbers (variance will be extremely low)
synthetic_data = [
    {"time_spent": 50, "num_friends": 10},
    {"time_spent": 51, "num_friends": 10},
    {"time_spent": 49, "num_friends": 10}
]

# feature_x = "time_spent", feature_y = "num_friends", tolerance = 0.1
# Expected Output: 'is_high_fidelity' should be False