Why This Matters Before We Talk About Technology
When an LLM call goes through a gateway, the log of that call is the only record that the call ever happened — what was sent, what came back, and what policy verdict was applied. In regulated industries that record is treated like a transaction journal, not like an application log.
Regulators don't ask "do you have logs." They ask whether you can prove the log you are reading is the log that was written. This document explains how Smartflow answers that question, what it deliberately does not claim, and how the same question is answered by LiteLLM, Azure API Management's AI Gateway, and OpenRouter today.
The five questions an examiner actually asks:
- Can you prove no one has modified a log entry after it was written?
- Can you prove no one has deleted a log entry?
- Can you prove no one has inserted a log entry retroactively to cover a gap?
- If a log is amended (e.g. a delayed compliance scan finds a violation), can you prove the original was preserved?
- Who has the keys, and what happens when those keys are rotated or compromised?
A gateway that only writes JSON rows into a database — even an encrypted one — cannot answer any of those questions on its own. The database admin, a compromised service account, or a misconfigured backup restore can change history with no detectable trace.
The industry term for what is needed is tamper-evident logging. It does not prevent tampering — that requires a true WORM substrate, see the Roadmap. It guarantees that if tampering happens, the next verification walk will detect it.
SOC 2 CC7.2 HIPAA §164.312(b) FINRA Rule 4511 SEC 17a-4 EU AI Act Art. 12 ISO 27001 A.12.4How Smartflow Records a VAS Log Entry
Every request that flows through smartflow-proxy produces a VASLog entry covering the request body, response body, model, provider, latency, cost, user/session identity, and policy verdict.
Before the entry is written to its lookup-by-ID location, it goes through audit_chain::seal(). This step is shared between the proxy hot path and the API server's audit handlers, so every writer chains into the same head.
Design decisions worth calling out for compliance reviewers
prev_hash chains every entry to its predecessorseq, and the chain cannot fork under concurrent load.key_id is stamped on every entrysecretKeyRef (VAS_HMAC_KEY). It never appears in the database, in the chain entries, or in logs.vas_log:{request_id} is mutable convenience data for dashboards. The chain entry at vas:chain:entry:{seq} is the audit record. Verifiers always read the chain entry.How "amendments" work
Asynchronous workflows — for example, a slow compliance scan that finishes after the response was already returned — cannot be allowed to silently rewrite the original entry, because that would be indistinguishable from tampering.
Smartflow handles this by appending a patch entry: a new sequenced + HMAC'd entry whose is_patch_of field references the original request_id and whose patch_reason records why the patch was written (compliance_violations, stream_finalize, etc.). The original entry's bytes are never touched. A verifier walking the chain sees both the original and the patch, in order, with their cryptographic linkage intact.
How log reads are recorded
GET /api/vas/logs is itself a regulated event in many environments — auditors want to know who looked at the journal. Smartflow appends a chained audit_log_read entry every time a log-read endpoint is hit, including the requesting admin's identity and the filter parameters. This means even read access to the audit trail is itself on the audit trail.
Today the chain entries live in the same Redis instance as operational data, with a 30-day TTL aligned to the snapshot retention. This is appropriate for "detective" tamper-evidence: a verifier walk will catch tampering. It is not a true WORM substrate — see What This Does NOT Claim and Roadmap.
Verifying the Chain
Smartflow exposes an admin-only endpoint:
The handler walks vas:chain:entry:{from..=to}, recomputes the HMAC of every entry against its canonical JSON, and re-checks prev_hash continuity end-to-end. The response is a structured report:
A worked tamper-detect example
That round-trip is the proof an examiner is asking for: not "we trust our database," but "here is the cryptographic verifier and here is its output." Run it on a schedule, alert on "ok": false, and the chain becomes an active control rather than a passive log.
We recommend a Kubernetes CronJob that calls verify_chain over a sliding window every 15 minutes and ships any non-ok result to your SIEM (Splunk, Sentinel, Sumo, etc.) as a P1 alert. The job itself is also logged on the chain — so disabling the verifier is itself an auditable event.
How This Compares to Other LLM Gateways
This section is a snapshot of what each of the major LLM gateway products documents publicly, as of early 2026. Capabilities change quickly; treat this as a starting point for your own due-diligence questions, not as a final scorecard.
LiteLLM
LiteLLM's open-source proxy stores call logs in PostgreSQL/Prisma and supports forwarding to observability backends (Langfuse, Helicone, S3, GCS, custom callbacks). The Enterprise tier adds an entity-level audit log that tracks administrative actions on keys, teams, users, and models — who created/updated/deleted what, and when. This is a real and useful control for the admin-action surface.
What is not part of the shipping LiteLLM product, as of early 2026, is per-call cryptographic chaining of the actual LLM request/response logs. The community is actively discussing it (e.g. the BerriAI/litellm discussion #25237 "Signed audit trails for every LLM call through the proxy" proposes ML-DSA-65 signatures chained per request), and third-party integrations such as Capsule Protocol and ASQAV layer hash chains on top of LiteLLM externally. If you're evaluating LiteLLM for a regulated workload, the right question to ask is: "for the per-request call log itself — not the entity-change log — what protects it from in-place modification by anyone with database access?"
Azure API Management AI Gateway
Azure's GenAI gateway logs LLM requests through the standard APIM telemetry pipeline into Azure Monitor / Log Analytics. The captured fields are well-documented in the ApiManagementGatewayLlmLog table reference — token counts, prompt and completion text, model deployment, correlation IDs.
Tamper-evidence in this stack is not a property of the gateway itself; it is delegated to whatever backend the customer chooses to export logs into. Azure does support immutable Blob Storage (time-based retention or legal-hold WORM policies) as an export destination, and that is the standard pattern Microsoft recommends for compliance-bound retention. The trade-off is operational: the gateway hands raw log records to Azure Monitor, and protection only begins once those records land in the immutable destination. There is no per-entry HMAC or hash chain emitted by the gateway, so detecting in-flight tampering between APIM and the WORM tier requires additional plumbing the customer has to build (e.g. signing a digest before export and verifying after).
If you're standardising on APIM, the model is "trust Azure's storage immutability primitives, and configure them deliberately." That is a defensible model for many use cases. It is a different model from a chain that the gateway itself produces and a verifier endpoint that any auditor can run on demand.
OpenRouter
OpenRouter offers an Input & Output Logging feature (currently in beta) that stores prompts and completions in an isolated Google Cloud Storage project, encrypted at rest with AES-256, retained for at least three months, accessible only to organisation admins. This is a strong baseline for "we keep a copy of what happened."
What OpenRouter does not currently document is per-entry cryptographic chaining, an HMAC on each record, or a customer-facing verify_chain-style endpoint. The integrity model is "we control the bucket and the IAM." For consumer and prosumer use cases that is reasonable; for regulated industries where the auditor will ask "and what if OpenRouter's own staff modify a record" the answer needs more structure.
Scorecard
| Capability | Smartflow | LiteLLM (OSS + Enterprise) | Azure APIM AI Gateway | OpenRouter |
|---|---|---|---|---|
| Per-call log capture | Yes | Yes | Yes | Beta, opt-in |
| Admin/entity-action audit log | Yes | Yes (Enterprise) | Yes (Azure Activity Log) | Limited |
| Per-entry HMAC over canonical JSON | Native | No (RFC / 3rd-party) | No (delegated to sink) | No |
Hash-chained prev_hash linkage |
Native | No (3rd-party only) | No | No |
| Atomic, gap-free sequence numbers | Redis Lua | N/A | N/A | N/A |
Customer-runnable verify_chain endpoint |
Native | No | No | No |
| Patch-as-append for delayed scan results | is_patch_of |
In-place updates | N/A | N/A |
| Read-of-audit-log itself recorded on chain | Yes | No | Via Activity Log | No |
| Key rotation without losing past verifiability | per-entry key_id |
N/A | N/A | N/A |
The takeaway is not that the other gateways are insecure. It is that, today, Smartflow is the only one of the four that ships a per-call tamper-evident chain and a customer-runnable verifier as part of the default product surface. The rest treat that as either a roadmap item, a third-party integration, or a problem to solve at the storage tier — all valid choices, but ones the buyer has to plan for.
What This Does Not Claim
Being precise about limits is part of being credible to a regulator.
The chain entries currently live in Redis with a 30-day TTL alongside the operational data. An attacker with full Redis write access can still modify or delete an entry. What they cannot do is modify or delete an entry without the next verify_chain walk flagging the affected seq as hmac_mismatches or missing. To make tampering prevented, not just detected, the chain entries must live on a true WORM substrate — see Roadmap.
The same key signs and verifies. That is appropriate for first-party verification by the platform operator. It is not appropriate as evidence to an external party who does not trust the holder of the key. For external-evidence use cases, a public-key signature scheme (Ed25519 or ML-DSA-65) over the chain head is the right next step; today we recommend mirroring the chain head to a customer-controlled location on a regular interval as a compensating control.
Today timestamps are clock-of-the-host. Adding RFC 3161 trusted timestamping to the chain head, on a regular cadence, is on the roadmap; until then, the chain proves order but not absolute time of creation.
The convenience snapshot at vas_log:{request_id} is mutable. It is a lookup index, not the source of truth. The chain entry at vas:chain:entry:{seq} is the record an auditor should read; the snapshot is for operational dashboards. We label this clearly in the storage layer so that nobody confuses the two.
Roadmap
These items are tracked internally and are explicitly not part of the shipping product today.
- WORM backend for chain entries. Move
vas:chain:entry:*from Redis-with-TTL to an append-only substrate. Candidate technologies: immudb, AWS QLDB, or PostgreSQL withpgauditplus filesystem WORM (e.g.chattr +awith offsite replication). The chain protocol does not change; only the substrate does. - RFC 3161 trusted timestamping. Periodically (every minute, every N entries, or on demand) submit the current
vas:chain:head_hmacto a public timestamp authority (FreeTSA, AWS Timestamp, DigiCert). Store the returnedTimeStampTokennext to the chain head. This binds the chain to wall-clock time in a way no in-cluster compromise can backdate. - External pen-test against the post-Phase-1 cluster with the audit-chain code path explicitly in scope. Findings and remediation will be published.
Practical Guidance for Evaluating a Gateway for Regulated Use
If you are running a vendor selection for a regulated environment and audit-trail integrity is on your scorecard, these are the questions worth asking every vendor in the running, including us:
- Is per-call tamper-evidence a property of the gateway, or is it delegated to a downstream storage tier? If delegated, what protects the records in flight between the gateway and that tier?
- Is there an endpoint a third-party auditor can call to verify a range of log entries, without needing your engineers to write a one-off script?
- When a delayed compliance scan finds a violation hours after the original request, is the original log entry modified in place, or is the amendment appended as a separate, linked record?
- When the signing key is rotated, do older entries remain verifiable against the older key? How is the key version recorded on each entry?
- Is reading the audit log itself an audited event?
- What happens to the chain if the underlying datastore is restored from backup?
- Has the audit-log code path been included in a third-party penetration test, and may we see the report?
Email [email protected]. The goal of this brief is not to claim a finished product — it is to be transparent about a control we treat as load-bearing, and to make the comparison concrete instead of marketing.