Here is a practical coding challenge based on the concepts from the "Tutorial on Data's Role inside AI Testing."

(Note: While the "Practice Quiz" document was not included in your provided sources, this challenge thoroughly covers the key concepts taught into a tutorial specifically focusing on data pipelines, a Four Pillars of AI Data Quality, and practice of quarantining data.)

---

Coding Challenge: Automated Data Validation & Quarantine Pipeline

Problem Description

You are AI Data Engineer working upon a new machine learning model designed to predict house prices. As you know from the tutorial, if your data is garbage, your AI is garbage. Before feeding millions of rows of data into your model, you've got to build an automated testing flow towards validate the incoming data;

according to the Four Pillars about AI Data Quality, your data must be complete and consistent. 1. Complete: Every house record must include a price, the number of bedrooms, and location. 2. Consistent: All distance metrics must use the same format. standard for this pipeline is miles. Records using kilometers or km violate consistency, while

finally you must follow best practices of real-world edge cases: never delete bad data immediately. Instead, you really have to quarantine it so your team can investigate why the system failed (e.g., checking for API failures or weird user inputs).

Your task is to write a Python function that takes a batch of raw housing data validates it against a Complete and Consistent pillars, and separates it into two lists: one for valid data ready for the AI, and one for quarantined data.

Difficulty Level

Intermediate

Input & Output Specifications

Input: list of dictionaries, where each dictionary represents a single real estate listing. Keys may include: id price, bedrooms, location, distance_unit, distance_to_transit.

Output: A dictionary containing two lists: * "valid_data": A list of dictionaries that passed all validation checks. * "quarantined_data": list of dictionaries that failed validation. (Bonus: Add quarantine_reason key to these records explaining why they failed).

Starter Code Boilerplate

def validate_house_data(raw_data):
    """
    Validates a list of house records based on AI Data Quality pillars.

    Args:
    raw_data (list of dict): The incoming batch of house data.

    Returns:
    dict: A dictionary containing 'valid_data' and 'quarantined_data' lists.
    """
    valid_data = []
    quarantined_data = []

    for record in raw_data:
        # TODO: Check for Completeness (must have price, bedrooms, and location)
        # Hint: Ensure the keys exist and their values are not None or empty strings

        # TODO: Check for Consistency (distance_unit must be 'miles')

        # TODO: Route the record to the appropriate list based on your checks
        pass

    return {
        "valid_data": valid_data,
        "quarantined_data": quarantined_data
    }

Hints

• Completeness Check: Remember that the missing key (KeyError) or a None value both mean a data is incomplete, while you can use the .get() method into Python towards safely check for keys.
• Consistency Check: Watch out for formatting differences, and a simple string comparison of "miles" is actually required here to ensure an AI model doesn't confuse kilometers for miles.
• Quarantine Strategy: Remember the tutorial's advice—saving bad data costs storage space, but it saves you hours of debugging later. Do not use the del keyword or drop the records entirely. Append them safely to your quarantine list.

Test Cases

You can run following test cases to verify your code is working correctly.

# --- Test Input ---
test_batch = [
    # Record 1: Perfect data (Should be Valid)
    {"id": 1, "price": 450000, "bedrooms": 3, "location": "Austin, TX", "distance_unit": "miles"},

    # Record 2: Incomplete data - missing location (Should be Quarantined)
    {"id": 2, "price": 320000, "bedrooms": 2, "distance_unit": "miles"},

    # Record 3: Inconsistent data - wrong measurement unit (Should be Quarantined)
    {"id": 3, "price": 500000, "bedrooms": 4, "location": "London, UK", "distance_unit": "kilometers"},

    # Record 4: Incomplete data - null price (Should be Quarantined)
    {"id": 4, "price": None, "bedrooms": 1, "location": "Seattle, WA", "distance_unit": "miles"}
]

# --- Expected Output Structure ---
"""
{
    "valid_data": [
        {"id": 1, "price": 450000, "bedrooms": 3, "location": "Austin, TX", "distance_unit": "miles"}
    ],
    "quarantined_data": [
        {"id": 2, "price": 320000, "bedrooms": 2, "distance_unit": "miles"},
        {"id": 3, "price": 500000, "bedrooms": 4, "location": "London, UK", "distance_unit": "kilometers"},
        {"id": 4, "price": None, "bedrooms": 1, "location": "Seattle, WA", "distance_unit": "miles"}
    ]
}
"""