Regulatory Compliance Testing For Ai Governance For Ai Models Testing Challenge

Here is practical coding challenge based at the concepts taught into a Compliance and Regulatory AI materials specifically focusing on building compliance-sensitive observability logger.

Coding Challenge: Building the Observability Logger as a Legal Shield

Problem Description

You're pretty much an MLOps engineer at a major bank which operates in a highly compliance-sensitive environment, and following the rollout of the European Union's Artificial Intelligence Act, strict new legal frameworks dictate how AI systems can process human data. Your team has deployed autonomous AI agent to evaluate mortgage applications passing them through a five-layer Regulated AI QA Stack.

To ensure you're pretty much prepared for a potential government audit, you need to use observability as your "legal shield." If auditor questions an agent's decision, you've got to be able to definitively prove exactly what data the AI looked at, what rules it followed, and why it took specific action by any given second.

Your task is really to build a compliance logging system that securely captures the MLOps Trinity (Code, Setup. Data query). Plus, you really have to generate a unique digital fingerprint (Data Hash) of the exact dataset the agent analyzed; this guarantees that if the underlying database changes in the future, you can prove to auditors whether the data is intact or if it has drifted.

Difficulty Level

Intermediate

Input & Output Specifications

Input: * audit_id (string): The unique identifier for the AI agent's decision event. * agent_code_version (string): The Git commit hash of the AI agent's logic. * setup_params (dictionary): The live environment hyperparameters and rules (e.g., strict legal thresholds). * sql_query (string): The exact database query the agent used for fetch the applicant's data. * fetched_live_data (list of dictionaries): exact live data rows the agent evaluated during the decision. * current_database_data (list of dictionaries): Data fetched later during an audit session to verify if the live database has drifted.

Output: * log_compliance_decision(audit_id, agent_code_version, setup_params, sql_query, fetched_live_data): Computes a digital fingerprint (hash) of the fetched data and securely stores a record containing the Code version, Setup Query and computed Hash under the audit_id. * verify_audit_integrity(audit_id, current_database_data): Retrieves the original hash towards the given audit run computes the hash of the current database data, and returns True if they match perfectly (proving legal compliance and data integrity), or False if the database has changed.

Starter Code Boilerplate

import json
import hashlib

# Simulated compliance database to store your observability logs
audit_logs = {}

def log_compliance_decision(audit_id, agent_code_version, setup_params, sql_query, fetched_live_data):
    """
    Hashes the fetched live data and securely logs the MLOps Trinity to act as a legal shield.
    """
    # TODO: Format the fetched_live_data consistently using json.dumps

    # TODO: Generate a SHA-256 digital fingerprint of the formatted data

    # TODO: Store the Code, Setup, Query, and Hash in the audit_logs dictionary under the audit_id
    pass

def verify_audit_integrity(audit_id, current_database_data):
    """
    Verifies if the current database data matches the original data hash to prove compliance.
    """
    # TODO: Retrieve the original hash from the audit_logs using the audit_id

    # TODO: Hash the current_database_data using the exact same consistent formatting

    # TODO: Return True if the hashes match, False otherwise
    pass

Hints

• The MLOps Trinity: Make sure your compliance log explicitly groups the Code (agent_code_version), the Setup (setup_params), and the Data (sql_query + a computed data hash). This acts as your legal defense for full explainability.
• Consistent Formatting: When converting a Python list of dictionaries for string for hashing, order matters immensely, and use json.dumps(data_payload, sort_keys=True) to ensure the payload format is absolutely identical every time.
• Generating the Digital Fingerprint: Use Python's built-into hashlib library to generate your secure hash: hashlib.sha256(formatted_string.encode('utf-8')).hexdigest().

Test Cases

You can run the following test script to verify your implementation works correctly:

# Test Data Configuration
audit_event = "audit_eu_mortgage_992"
code_v = "commit_8f2a1b"
setup = {"risk_threshold": 0.85, "compliance_mode": "strict"}
query = "SELECT * FROM loan_applications WHERE applicant_id = '456'"

# The exact data the AI evaluated at the moment of the decision
original_data = [{"applicant_id": "456", "income": 85000, "credit_score": 720}]

# 1. Log the compliance decision
log_compliance_decision(audit_event, code_v, setup, query, original_data)

# 2. Verify integrity during a government audit (Data hasn't changed)
audit_data_intact = [{"applicant_id": "456", "income": 85000, "credit_score": 720}]
print(f"Audit Intact Check: {verify_audit_integrity(audit_event, audit_data_intact)}") 
# Expected Output: True

# 3. Verify integrity (Data has drifted or been tampered with)
audit_data_drifted = [{"applicant_id": "456", "income": 95000, "credit_score": 720}]
print(f"Audit Drift Check: {verify_audit_integrity(audit_event, audit_data_drifted)}") 
# Expected Output: False