Role Of Qa In Ai Testing Ai Testing Job Description Challenge

Coding Challenge: Automated Data Validation & Quarantine Pipeline

Problem Description

You are an AI Quality Assurance Engineer testing data to a new machine learning model designed to predict house prices. As discussed in a tutorial on the modern AI Tester's role, the most effective way to fix AI agent failures is by strengthening the data foundation, and before feeding millions of rows of data into your model, you must build the automated testing flow to validate an incoming data at scale.

According for the Four Pillars of AI Data Quality, your data must be complete and consistent: 1. Complete: Every house record must include price number of bedrooms and a location. 2. Consistent: All distance metrics must use a same format. The standard to this pipeline is miles. Records using kilometers or km violate consistency.

Finally you've got to follow best practices towards handling real-world edge cases: never delete bad data immediately. Instead you must apply a quarantine pattern to isolate it so your team can investigate exactly why system failed (e.g., checking of API failures strange user inputs, or hidden bias).

Your task is to write the Python function that takes a batch of raw housing data, validates it against the Complete and Consistent pillars, and separates it into two lists: one for valid data ready for the AI, and one for quarantined data.

Difficulty Level

Intermediate

Input & Output Specifications

Input: A list of dictionaries where each dictionary represents the single real estate listing. Keys may include: id price, bedrooms location distance_unit, distance_to_transit.

Output: THE dictionary containing two lists: * "valid_data": A list for dictionaries that passed all validation checks. * "quarantined_data": A list of dictionaries that failed validation. You really have to add a quarantine_reason key to these records explaining why they failed.

Starter Code Boilerplate

def validate_housing_data(batch_data):
    valid_data = []
    quarantined_data = []

    for record in batch_data:
        # TODO: Check for completeness (must have price, bedrooms, location)

        # TODO: Check for consistency (distance_unit must be 'miles')

        # TODO: If valid, append to valid_data. 
        # TODO: If invalid, add a 'quarantine_reason' key and append to quarantined_data.
        pass

    return {
        "valid_data": valid_data,
        "quarantined_data": quarantined_data
    }

Hints

• Completeness Check: Remember that a missing key (KeyError) or a None value both mean the data is incomplete. You can use the .get() method in Python to safely check for keys.
• Consistency Check: Watch out for formatting differences. A simple string comparison for "miles" is probably required here towards ensure an AI model doesn't confuse kilometers of miles.
• Quarantine Strategy: Remember a tutorial's advice—saving bad data costs extra storage space, but it saves you hours of debugging later, while do not use del keyword or drop the records entirely. Append them safely to your quarantine list.

Test Cases

You can really run following test cases to verify your code is working correctly:

# Test Data
raw_data = [
    {"id": 1, "price": 450000, "bedrooms": 3, "location": "Austin", "distance_unit": "miles"},
    {"id": 2, "price": 520000, "bedrooms": 4, "location": "Dallas", "distance_unit": "km"}, # Fails consistency
    {"id": 3, "price": None, "bedrooms": 2, "location": "Houston", "distance_unit": "miles"}, # Fails completeness (None price)
    {"id": 4, "bedrooms": 3, "location": "San Antonio", "distance_unit": "miles"}, # Fails completeness (Missing price key)
    {"id": 5, "price": 300000, "bedrooms": 2, "location": "El Paso", "distance_unit": "miles"}
]

# Run the function
results = validate_housing_data(raw_data)

# Output assertions
assert len(results["valid_data"]) == 2, f'Expected 2 valid records, got {len(results["valid_data"])}'
assert len(results["quarantined_data"]) == 3, f'Expected 3 quarantined records, got {len(results["quarantined_data"])}'

# Check quarantine reasons
quarantined_ids = [record["id"] for record in results["quarantined_data"]]
assert 2 in quarantined_ids # Should be quarantined for consistency
assert 3 in quarantined_ids # Should be quarantined for completeness
assert 4 in quarantined_ids # Should be quarantined for completeness

print("All tests passed! Your automated data pipeline is ready.")