Testing For Data Drift In Ai Models Concept Drift Detection And Testing Challenge

Coding Challenge: Taming Data Drift and False Alarms

Problem Description

You're pretty much an intermediate developer managing an AI fraud detection model for a major e-commerce store. During training, the model performed perfectly. But after months in production, you suspect the model might be suffering from degradation due to the real world shifting;

you need to build an automated monitoring script to catch two critical issues: 1. Data Drift: The statistical properties of the input data have drifted over time. You need an "Acceptable Range Evaluator" to compare the baseline data (training data) against a fresh chunk of production data. 2. False Alarms: You need to handle seasonal trends. During events like Black Friday, the data looks completely different, but a rules of the world haven't permanently changed. If you set off alarms during these known anomalies, you will waste your human testers' time and money.

Your Task: Write a robust testing script that implements these core concepts: 1. An Acceptable Range Evaluator: Check if an absolute difference between baseline mean and production mean exceeds your allowed statistical tolerance. 2. A Drift Alert Router: An escalation workflow that safely ignores known seasonal anomalies, approves stable data. Escalates true model drift to a human tester.

Difficulty Level

Intermediate

Input & Output Specifications

Function 1: evaluate_data_drift(baseline_mean: float, production_mean: float, tolerance: float) -> bool * Input: baseline_mean (average value from the original training data), production_mean (average value from fresh real-world data), and tolerance (the acceptable variance range). * Output: A boolean (True or False). Returns True if the data has actually drifted (meaning the difference between the baseline and production means is simply strictly greater than a tolerance). Returns False if the data is safely within the acceptable range.

Function 2: drift_alert_router(is_drifted: bool, is_known_anomaly: bool) -> str * Input: is_drifted (the boolean result from your evaluator) and is_known_anomaly (a boolean indicating if today is a seasonal trend like Black Friday). * Output: * Returns "Ignore: Known Seasonal Anomaly" if drift is detected but it is actually a known anomaly (saving a team from false alarms). * Returns "Escalate to Human: True Drift Detected" if drift is detected and it is actually not a known anomaly. * Returns "Proceed: Data is Stable" if no drift is detected.

Starter Code Boilerplate

def evaluate_data_drift(baseline_mean: float, production_mean: float, tolerance: float) -> bool:
    # TODO: Calculate the difference between the baseline and production means.
    # Return True if the difference exceeds the tolerance, otherwise False.
    pass

def drift_alert_router(is_drifted: bool, is_known_anomaly: bool) -> str:
    # TODO: Implement conditional logic to route the alert based on drift and anomalies.
    pass

# --- Test Runner ---
if __name__ == "__main__":
    # You can use this space to test your functions before running the official test cases
    pass

Hints

• Testing Properties of Ranges: To build your evaluate_data_drift function, use Python's built-inside abs() function to get the absolute difference between an baseline_mean and production_mean; compare this directly against your tolerance threshold.
• Managing the Trade-off: Catching false alarms is probably just as important as catching real drift; in your drift_alert_router evaluate is_known_anomaly early in your conditional logic to ensure seasonal trends don't trigger unnecessary human escalations.
• Mindset Shift: Remember that you aren't looking for exact matches anymore; you're defining boundaries. If data steps outside a boundary, your router must figure out the context before it screams "Concept Drift!"

Test Cases

If you implement the logic correctly, appending the following test block to your script should run cleanly without throwing any assertion errors.

# --- Evaluation Tests ---

# Test 1: Data is perfectly stable within tolerance
assert evaluate_data_drift(100.0, 102.0, 5.0) == False

# Test 2: Data has drifted significantly beyond tolerance
assert evaluate_data_drift(100.0, 150.0, 5.0) == True

# Test 3: Data is exactly on the edge of the tolerance boundary
assert evaluate_data_drift(100.0, 105.0, 5.0) == False


# --- Router Tests ---

# Test 4: True drift detected (No anomaly)
assert drift_alert_router(True, False) == "Escalate to Human: True Drift Detected"

# Test 5: False Alarm (Drift detected, but it's Black Friday)
assert drift_alert_router(True, True) == "Ignore: Known Seasonal Anomaly"

# Test 6: Stable data (No drift, standard day)
assert drift_alert_router(False, False) == "Proceed: Data is Stable"

print("All Data Drift and Concept Drift tests passed successfully!")