Coding Challenge: Synthetic Data Fidelity Checker

Problem Description

You're pretty much an AI engineer working on cloud-native database provisioning system; your team is utilizing Generative Adversarial Networks (GANs) and transformer models to generate massive amounts of production-like test data.

Though during testing you suspect the GAN model is getting lazy—a phenomenon known as Mode Collapse, where the generator stops creating diverse data and only outputs narrow set of "safe" values. To prevent your AI from learning completely wrong behaviors you really have to rigorously measure a "fidelity" (realness and accuracy) of the generated data;

your task is to write an automated fidelity checker, while this function will probably compare statistical properties (means and variances) between the real data and synthetic data. It'll also calculate a correlation coefficient between two related features (e.g., "time spent on app" and "number of friends") to ensure the synthetic data accurately preserves real-world relationships.

Difficulty Level

Intermediate

Input & Output Specifications

Input: * real_data (List of Dictionaries): The original production dataset. * synthetic_data (List of Dictionaries): GAN-generated dataset. * feature_x (String): A key name of the first metric (e.g., 'time_spent'). * feature_y (String): The key name for the second metric (e.g., 'num_friends'). * tolerance (Float): The maximum allowed percentage difference (expressed as a decimal, e.g., 0.1 for 10%) between a statistics of real and synthetic data to pass the fidelity check.

Output: * A Dictionary containing: * mean_diff_x: The absolute difference in means for feature_x. * variance_diff_x: An absolute difference on variances for feature_x. * real_correlation: A correlation coefficient between feature_x and feature_y into a real dataset. * synthetic_correlation: correlation coefficient between feature_x and feature_y in the synthetic dataset. * is_high_fidelity (Boolean): True if the mean differences, variance differences, and correlation differences are all within an allowed tolerance relative to the real data's statistics. False otherwise (indicating potential Mode Collapse).

Starter Code Boilerplate

import math

def calculate_mean(data, feature):
    # TODO: Calculate the mean of a given feature
    pass

def calculate_variance(data, feature, mean):
    # TODO: Calculate the variance of a given feature
    pass

def calculate_correlation(data, feature_x, feature_y):
    # TODO: Calculate the Pearson correlation coefficient between two features
    pass

def check_data_fidelity(real_data, synthetic_data, feature_x, feature_y, tolerance=0.1):
    """
    Evaluates the fidelity of synthetic data against real production data 
    by comparing means, variances, and correlation coefficients.
    """

    # 1. Calculate statistics for real_data
    # 2. Calculate statistics for synthetic_data
    # 3. Compare statistics and check against the tolerance threshold
    # 4. Return the results dictionary

    return {
        "mean_diff_x": 0.0,
        "variance_diff_x": 0.0,
        "real_correlation": 0.0,
        "synthetic_correlation": 0.0,
        "is_high_fidelity": False
    }

Hints

• Correlation Coefficient: Remember that the correlation coefficient is a score from -1 to 1 that shows how two things relate. If "time spent on app" goes up when "number of friends" goes up in real life, your synthetic data needs to reflect that same positive relationship! You can probably compute this using the covariance of X and Y divided by a product with their standard deviations.
• Mode Collapse Check: If synthetic data's variance is significantly lower than the real data's variance (falling outside your tolerance), this is strong mathematical indicator of Mode Collapse.
• Automation: In a real cloud-native environment (like Kubernetes), a script similar to this would run continuously towards guarantee test data improves with every software release without risking privacy leaks.
• (Optional): If you prefer, you can convert the lists of dictionaries into pandas DataFrames to easily use built-on statistical functions like .mean() .var(), and .corr().

Test Cases

# Test Case 1: High Fidelity Data
real_data_1 = [
    {"time_spent": 10, "num_friends": 2},
    {"time_spent": 20, "num_friends": 4},
    {"time_spent": 30, "num_friends": 6},
    {"time_spent": 40, "num_friends": 8}
]

synthetic_data_1 = [
    {"time_spent": 11, "num_friends": 2},
    {"time_spent": 19, "num_friends": 4},
    {"time_spent": 31, "num_friends": 6},
    {"time_spent": 39, "num_friends": 8}
]

# Expected Output: 
# Mean differences and variance differences should be minimal. 
# Both datasets should show a perfect or near-perfect positive correlation (approx 1.0).
# is_high_fidelity should be True.

# Test Case 2: Mode Collapse (Low Fidelity)
real_data_2 = [
    {"time_spent": 10, "num_friends": 2},
    {"time_spent": 50, "num_friends": 10},
    {"time_spent": 20, "num_friends": 4},
    {"time_spent": 80, "num_friends": 16}
]

synthetic_data_2 = [
    {"time_spent": 40, "num_friends": 8},
    {"time_spent": 41, "num_friends": 8},
    {"time_spent": 40, "num_friends": 9},
    {"time_spent": 42, "num_friends": 8}
]

# Expected Output: 
# The synthetic data lacks diversity (variance is far too low compared to real data).
# The relationship between time spent and friends is distorted.
# is_high_fidelity should be False.