Human-Agent Interaction Testing

Here is a practical coding challenge based at a concepts taught in the Human-Agent Interaction Testing tutorial.

Coding Challenge: Logging Human-Agent Interactions & Data Drift

Problem Description

You're basically MLOps engineer responsible for a transparent AI customer support agent. Because human conversations are unpredictable and chaotic, manual QA isn't actually enough. To evaluate your system you deploy Tester Agents with various personas (e.g., confused beginner, angry customer) for chat with your main AI. Relying on Deep Observability to trace the AI's internal "thoughts" (Model Calls Tool Selections, Decision Chains and Token Consumption).

On Tuesday Tester Agent uncovers terrible interaction. However, by Wednesday, the simulated human "deleted" their profile from the database; if you use standard Query-Based Version Control running exact same database query on Wednesday will probably return different data than it did on Tuesday, completely breaking your ability for reproduce and debug the error;

to solve this edge case, your task is for build a robust logging pipeline that captures the MLOps Trinity (Code Setup Data) a moment the interaction occurs. You've got to also implement Data Hashing to generate the unique digital fingerprint of the exact live data the AI looked at allowing you to instantly detect if the underlying human database has changed when you go to debug the interaction.

Difficulty Level

Intermediate

Input & Output Specifications

Input: * interaction_id (string): An unique identifier of the live chat session. * agent_code_version (string): The Git commit hash of the AI agent's logic. * setup_params (dictionary): live environment setup (e.g., observability flags, transparency rules). * sql_query (string): The query used to fetch an user's profile and chat history. * fetched_live_data (list of dictionaries): The exact live data the agent analyzed during a chat. * current_database_data (list about dictionaries): Data fetched later during debugging to verify if the profile still exists.

Output: * log_interaction(): Computes the secure data hash for fetched_live_data and stores a record containing the Code version, Setup Query, and computed Hash in a mock database ( dictionary). * verify_reproducibility(): Retrieves the original hash for a given interaction_id, computes the hash for current_database_data and returns True if they match (data is intact) or False if the database has drifted.

Starter Code Boilerplate

import hashlib
import json

# Global dictionary to act as our observability log database
interaction_logs = {}

def generate_data_hash(data):
    """
    Generate a consistent SHA-256 hash for a list of dictionaries.
    """
    # TODO: Convert the data to a consistent string format
    # TODO: Generate and return the SHA-256 hash
    pass

def log_interaction(interaction_id, agent_code_version, setup_params, sql_query, fetched_live_data):
    """
    Securely log the MLOps Trinity and the digital data fingerprint.
    """
    # TODO: Generate the data hash
    # TODO: Store the Code, Setup, and Data (query + hash) in the interaction_logs dictionary
    pass

def verify_reproducibility(interaction_id, current_database_data):
    """
    Verify if the live database has changed since the interaction was logged.
    """
    # TODO: Retrieve the original hash from interaction_logs
    # TODO: Hash the current_database_data
    # TODO: Compare the hashes and return True if they match, False otherwise
    pass

Hints

• The MLOps Trinity: Standard Git isn't enough for AI; your log needs to explicitly group the Code (agent_code_version), the Setup (setup_params), and the Data (sql_query + computed hash) together under a interaction_id.
• Consistent Formatting: When converting the Python list for dictionaries to the string for hashing order matters immensely. Use json.dumps(data_payload, sort_keys=True) to ensure the payload format is absolutely identical every time.
• Generating the Digital Fingerprint: Use Python's built-in hashlib library. You can generate your secure hash by encoding the consistent string: hashlib.sha256(formatted_string.encode('utf-8')).hexdigest().

Test Cases

You can just use a following test cases to verify your implementation works correctly:

# --- Test Setup ---
interaction_id = "chat_9942_tuesday"
code_version = "git_commit_773ab"
setup = {"model": "gpt-4", "transparency_mode": True, "token_limit": 1500}
query = "SELECT * FROM users WHERE user_id = '8821'"

# The data the AI saw on Tuesday
data_tuesday = [
    {"user_id": "8821", "name": "Alice", "status": "active", "complaints": 2}
]

# The data in the database on Wednesday (Alice deleted her account)
data_wednesday = []

# --- Run Tests ---

# 1. Log the interaction on Tuesday
log_interaction(interaction_id, code_version, setup, query, data_tuesday)

# 2. Verify that the log contains the MLOps Trinity
assert interaction_id in interaction_logs, "Log entry missing."
assert "code_version" in interaction_logs[interaction_id], "Code version missing from Trinity."
assert "setup" in interaction_logs[interaction_id], "Setup missing from Trinity."
assert "data_hash" in interaction_logs[interaction_id], "Data Hash missing from Trinity."

# 3. Check reproducibility on Tuesday (Data hasn't changed yet)
is_reproducible_tuesday = verify_reproducibility(interaction_id, data_tuesday)
print(f"Reproducible on Tuesday? {is_reproducible_tuesday}")
assert is_reproducible_tuesday == True, "Failed: Hash should match identical data."

# 4. Check reproducibility on Wednesday (User deleted profile)
is_reproducible_wednesday = verify_reproducibility(interaction_id, data_wednesday)
print(f"Reproducible on Wednesday? {is_reproducible_wednesday}")
assert is_reproducible_wednesday == False, "Failed: Hash should NOT match drifted data."

print("All tests passed! You have successfully secured the interaction logs.")