How to Audit AI Systems: Logging, Verification, and Compliance
Auditing an AI system is not the same as auditing software. It requires verifiable records of what data was used, what decisions were made, and whether those decisions can be independently verified.
This guide covers the six steps required to audit any AI system — from establishing logging infrastructure to performing cryptographic verification of decision records.
The six steps of AI system auditing
Identify what must be audited
Define the audit scope: which decisions matter, which data sources were used, which model versions are in scope, and what compliance framework applies (EU AI Act, ISO 42001, HIPAA, etc.). High-risk AI systems under the EU AI Act require auditability of decisions, data provenance, and model behavior.
Verify dataset provenance
Before auditing decisions, verify the datasets. A certified dataset carries a SHA-256 fingerprint and an Ed25519 signature — compute the hash of the dataset file and compare it to the certificate's artifact_hash. If they match, the dataset is byte-for-byte identical to the certified version. Mismatches prove the dataset was altered after certification.
Synthetic Data Certification →Establish decision logging
Every significant AI decision must be recorded as a structured log entry — not a plain-text log, but a cryptographically signed record. CertifiedData decision logs include: decision ID, timestamp, outcome, model version, certified dataset reference, SHA-256 chain hash, and Ed25519 signature. Without this structure, decisions cannot be independently verified.
AI Decision Logging →Verify log integrity
Verify that the decision log has not been tampered with by recomputing the chain hash sequence. Each entry's chain_hash should equal the SHA-256 of the prior entry. If any chain hash fails, an entry was modified, deleted, or inserted between those positions. CertifiedData's append-only structure enforces this at the storage level.
AI Audit Logs →Trace decisions to their source data
Each decision log entry references a dataset_certificate_id — the certificate of the dataset that informed the model producing that decision. Resolve this ID against the certificate registry to confirm which dataset was used, when it was certified, and that its fingerprint still matches the file. This creates end-to-end lineage from data to decision.
AI Decision Lineage →Produce the audit report
An AI audit report should include: the dataset certificate IDs referenced, the decision log hash chain verification result, the range of decisions audited (by timestamp or decision ID), any chain integrity failures, and the Ed25519 signature verification results. This satisfies EU AI Act Article 12 (record-keeping) and Article 19 (technical documentation) requirements.
Verify a dataset locally
Run these commands to verify a certified dataset file matches its certificate — no account required:
# Linux / macOS
sha256sum dataset.zip
# Windows PowerShell
Get-FileHash dataset.zip -Algorithm SHA256
# Then compare output against the artifact_hash in the certificate:
curl https://certifieddata.io/api/cert/{certificate_id}/manifestRegulatory requirements for AI auditing
EU AI Act — Article 10
Training and validation data must be documented, with provenance traceable. Dataset certificates satisfy this requirement.
EU AI Act — Article 12
High-risk AI systems must automatically log inputs, outputs, and behavior in a way that enables post-hoc reconstruction.
EU AI Act — Article 19
Technical documentation must be maintained and made available to competent authorities. Audit trails and certificate records constitute this documentation.
ISO 42001
AI management systems must implement processes for auditability and traceability. Decision logging and dataset certification provide the technical foundation.