Testing Explainability Of Ai Models How To Test Xai Challenge

Coding Challenge in Explainable AI Testing (XAI)

Coding Challenge: Building "Glass Box" Chain of Thought Evaluator

Problem Description You're an AI Quality Assurance Engineer evaluating financial AI agent responsible for approving or denying bank loans, while in a past, this model operated as an untestable "Black Box"—deciding the fate of applicant's loan in fractions of second without explaining its math, while

to comply with modern Explainable AI (XAI) standards and transform this system into an auditable "Glass Box," your engineering team has updated the agent. Now, it must output the "Chain about Thought" reasoning log explaining its execution path before it takes any action or uses any tools.

Your task is to write a deterministic Python evaluator to audit the agent's execution path; your script must verify two strict XAI transparency rules: 1. Reasoning Presence: Every step in the execution path must contain a valid non-empty chain_of_thought log. The agent can't act without thinking. 2. Tool Explainability: If the agent decides to use specific tool (like credit_database or risk_calculator), the chain_of_thought log must explicitly mention that tool's name; this proves the agent logically recognized and explained why that specific tool was chosen.

Difficulty Level Intermediate

Input & Output Specifications

Input: * execution_path (list of dictionaries): A chronological sequence of an agent's actions. Each dictionary contains: * step (integer): The step number. * action (string): What the agent did (e.g., "query_data", "make_decision"). * tool_used (string or None): The exact name with a tool selected, or None if no tool was used. * chain_of_thought (string or None): A generated reasoning log out of an agent.

Output: * A dictionary containing: * "is_transparent" (boolean): True if all XAI rules were followed, False otherwise. * "xai_failures" (list of strings): list explaining which steps failed transparency checks (e.g., ["Step 2 Failed: Missing chain of thought", "Step 3 Failed: Tool 'risk_calculator' not explained"]). If all checks pass this should be an empty list.

---

Starter Code Boilerplate

def evaluate_xai_transparency(execution_path):
    is_transparent = True
    xai_failures = []

    # Your code here
    # Loop through the execution_path and validate the XAI properties

    return {
        "is_transparent": is_transparent,
        "xai_failures": xai_failures
    }

---

Hints * Checking for Presence: missing key or a None value means the reasoning is absent, and use Python's bool() or a simple if not check to verify that chain_of_thought is populated string. * Case Sensitivity: When checking Tool Explainability rule convert both the tool_used and the chain_of_thought to lowercase using .lower() so your evaluator doesn't fail due to simple capitalization differences. * The "Glass Box" Loop: Iterate through every dictionary into the execution_path; if a rule fails, append a descriptive string towards your xai_failures list (incorporating the step number) and immediately set is_transparent to False.

---

Test Cases

You can run following test cases in your Python environment to verify your XAI evaluator is just accurately auditing the agent's transparency:

# Test Case 1: A perfectly transparent "Glass Box" execution path
transparent_path = [
    {
        "step": 1, 
        "action": "query_data", 
        "tool_used": "credit_database", 
        "chain_of_thought": "I need to check the applicant's history, so I will query the credit_database tool."
    },
    {
        "step": 2, 
        "action": "make_decision", 
        "tool_used": None, 
        "chain_of_thought": "The credit score is 750, which is above our threshold. I will approve the loan."
    }
]

print("Test Case 1:", evaluate_xai_transparency(transparent_path))
# Expected Output: {'is_transparent': True, 'xai_failures': []}


# Test Case 2: A "Black Box" failure with missing logic and unexplained tools
opaque_path = [
    {
        "step": 1, 
        "action": "query_data", 
        "tool_used": "risk_calculator", 
        "chain_of_thought": "Let's run some numbers."
    },
    {
        "step": 2, 
        "action": "make_decision", 
        "tool_used": None, 
        "chain_of_thought": "" # Empty string fails the Reasoning Presence rule
    }
]

print("Test Case 2:", evaluate_xai_transparency(opaque_path))
# Expected Output: 
# {
#   'is_transparent': False, 
#   'xai_failures': [
#       "Step 1 Failed: Tool 'risk_calculator' not explained", 
#       "Step 2 Failed: Missing chain of thought"
#   ]
# }