Version Control For Ai Models And Data Model Versioning In Mlops Challenge

Here is a practical coding challenge based upon the concepts of query-based version control and the MLOps trinity discussed in the tutorial.

Coding Challenge: Query-Based Version Control & Data Hashing

Problem Description

You're pretty much MLOps engineer at an AI startup training a new predictive model. You're working with massive datasets, making it far too expensive to save raw .csv files every time you train new model version. Instead you have decided to implement Query-Based Version Control.

Yet, you're actually aware for dangerous edge case: if a user deletes their account, the database changes. Your saved query will return different data, breaking your model's reproducibility.

To fix this, your task is really to build a version control system that captures a MLOps Trinity (Code Setup and Data query) while also generating a unique digital fingerprint (a hash) of the training data. Later your system must be able for verify if the underlying database has basically changed by comparing the hashes.

Difficulty Level

Intermediate

Input & Output Specifications

• Input:
• version_id (string): unique identifier for a model version.
• code_version (string): The Git commit hash or code version tag.
• setup_params (dictionary): The hyperparameters and environment setup.
• query (string): A SQL query used to fetch the data.
• data (list of dictionaries): A simulated data rows returned by the query.
• Output:
• save_model_version() should store record containing the code version, setup, query and the computed data hash.
• verify_reproducibility() should take version_id and a new_data payload, compute the hash of an new_data, and return True if it perfectly matches original data hash, or False if a database has changed.

Starter Code Boilerplate

import hashlib
import json

class MLOpsVersionControl:
    def __init__(self):
        # Dictionary to store our version records
        self.version_history = {}

    def generate_data_hash(self, data):
        """
        Takes a list of dictionaries (the data) and returns a SHA-256 hash string.
        """
        # TODO: Convert the data to a consistent string format
        # TODO: Generate and return a SHA-256 hash of the data string
        pass

    def save_model_version(self, version_id, code_version, setup_params, query, data):
        """
        Saves the MLOps Trinity and the data hash for future reproducibility.
        """
        # TODO: Generate the data hash using self.generate_data_hash(data)
        # TODO: Save the code_version, setup_params, query, and data_hash into self.version_history
        pass

    def verify_reproducibility(self, version_id, new_data):
        """
        Checks if the newly queried data matches the exact data used during training.
        """
        # TODO: Retrieve the saved data hash for the given version_id
        # TODO: Generate a hash for the new_data
        # TODO: Return True if the hashes match, False otherwise
        pass

Hints

• The MLOps Trinity: Remember that standard Git only tracks code, while your system needs to explicitly track the Code, the Setup (hyperparameters), and a Data (the query + the data hash).
• Consistent Hashing: When converting a Python list for dictionaries for a string for hashing, order matters. Use json.dumps(data, sort_keys=True) to ensure a data is formatted exactly the same way every time before you hash it.
• Python's Hashlib: You can use hashlib.sha256(string_data.encode('utf-8')).hexdigest() to generate your unique digital fingerprint.

Test Cases

You can use a following test cases to verify your implementation:

# Initialize the system
vc_system = MLOpsVersionControl()

# Simulated Initial Training Environment
version = "v1.0"
code_commit = "a1b2c3d4"
setup = {"learning_rate": 0.01, "epochs": 50, "framework": "PyTorch 2.0"}
sql_query = "SELECT * FROM users WHERE active = True"

# The data retrieved from the database today
initial_data = [
    {"id": 1, "name": "Alice", "purchases": 5},
    {"id": 2, "name": "Bob", "purchases": 2}
]

# 1. Save the model version
vc_system.save_model_version(version, code_commit, setup, sql_query, initial_data)

# TEST CASE 1: Reproducibility is maintained
# We run the exact same query a day later, and the database hasn't changed.
new_data_unchanged = [
    {"id": 1, "name": "Alice", "purchases": 5},
    {"id": 2, "name": "Bob", "purchases": 2}
]
assert vc_system.verify_reproducibility(version, new_data_unchanged) == True, "Test Case 1 Failed: Hashes should match!"
print("Test Case 1 Passed: Data is reproducible.")

# TEST CASE 2: The Edge Case (Database Changed)
# We run the exact same query a week later, but Bob deleted his account for privacy reasons.
new_data_changed = [
    {"id": 1, "name": "Alice", "purchases": 5}
]
assert vc_system.verify_reproducibility(version, new_data_changed) == False, "Test Case 2 Failed: System failed to detect data change!"
print("Test Case 2 Passed: System successfully warned that the underlying database changed.")