> ## Documentation Index
> Fetch the complete documentation index at: https://docs.uselayerup.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Observability, Logging & Audit Trails
> Telemetry architecture across CloudWatch and Azure Monitor, step-level reasoning capture, deterministic audit trails for regulatory examination, log retention and immutability, and long-term archival.
# 7 — Observability, logging & audit trails — telemetry architecture, deterministic reasoning capture & regulatory compliance.
This section addresses the requirements of your internal audit, compliance, and IT operations teams. The Layerup agent is designed to be a fully transparent workload — every decision, reasoning step, document extraction, and model inference is logged in structured, queryable format within your own observability infrastructure. No telemetry is emitted to Layerup's systems.
***
## 7.1 Telemetry architecture — structured, queryable, yours
The Layerup agent emits structured telemetry to your native cloud monitoring systems throughout the lifecycle of each agent session. The telemetry architecture is designed to be immediately consumable by your existing operations and audit tooling without bespoke integration.
### 7.1.1 AWS: Amazon CloudWatch
The agent emits the following data streams to CloudWatch:
| log stream | level | contents |
| -------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Application Logs** | `INFO` | High-level session events: session start, document ingestion complete, LLM reasoning initiated, reasoning complete, output written, session end. Useful for operational dashboards and SLA monitoring. |
| **Audit Logs** | `AUDIT` | Immutable, append-only structured log entries for each agent decision, source citation, and flag raised. Includes the full reasoning chain that produced each recommendation, enabling complete reconstruction of the agent's decision process for any case. |
| **Error Logs** | `ERROR` / `WARN` | Structured error events for failed document reads, OCR failures, model timeout events, and unexpected exceptions. Each event carries a correlation ID linking it back to the originating case. |
| **Metrics** | CloudWatch Metrics | Custom metrics: `cases_processed_count`, `cases_flagged_for_escalation_count`, `average_processing_time_seconds`, `document_ocr_confidence_score_average`, `guardrail_interventions_count`. Powers operational dashboards and drives CloudWatch Alarms. |
### 7.1.2 Azure: Azure Monitor and Application Insights
On Azure, equivalent telemetry flows to Azure Monitor and Application Insights:
* Structured logs are emitted to a **Log Analytics Workspace**, enabling KQL queries across all agent sessions and cross-session analytics.
* **Azure Application Insights** provides distributed tracing across the agent's processing pipeline, enabling your team to visualise the end-to-end processing timeline for any case.
* **Azure Monitor Alerts** are configured to notify your on-call operations team of any agent health degradation or SLA breach.
***
## 7.2 Deterministic audit trails — regulatory-grade reasoning capture
A critical requirement for any regulated carrier deploying AI in underwriting decisions is the ability to produce a complete, immutable audit trail for any given case — including the AI's reasoning — to satisfy state insurance department examination requirements, internal audit standards, and potential legal discovery.
### 7.2.1 Step-level reasoning capture
For every underwriting case, the Layerup agent writes a structured audit record that contains:
The exact prompt sent to the foundation model for each reasoning step (with applicant PII redacted according to your Bedrock Guardrail configuration). No reasoning occurs in a black box.
The model's complete response for each reasoning step, including the specific guardrail policy evaluations applied and their outcomes.
The agent's decision logic at each step: which rules from the AOP were triggered, what evidence was evaluated, and what conclusion was reached — mapped to specific AOP rule identifiers.
Source citations for every extracted data point, including the document filename, page number, section, and verbatim extracted text excerpt.
Timestamps with millisecond precision for each reasoning step — enabling point-in-time reconstruction of the agent's processing timeline for any case.
The AOP version and model version active at the time of the case — enabling exact point-in-time reconstruction of the agent's configuration and behavior for historical review or re-examination.
### 7.2.2 CloudWatch Log Insights — example audit queries
Your audit team uses CloudWatch Log Insights to query the structured audit trail. The structured log format supports a wide range of compliance queries without requiring custom tooling:
```sql theme={null}
-- Retrieve the full reasoning chain for a specific case
fields @timestamp, reasoning_step, aop_rule_triggered, evidence_cited, conclusion
| filter case_id = "case_20260601_00123"
| sort @timestamp asc
-- Find all cases where a Bedrock Guardrail was triggered in the last 30 days
fields @timestamp, case_id, guardrail_policy, guardrail_outcome
| filter guardrail_intervention = true
| filter @timestamp > dateadd(day, -30, now())
-- Identify low-confidence cases where recommendation was Defer
fields @timestamp, case_id, confidence_score, ai_recommendation
| filter confidence_score < 85
| filter ai_recommendation = "Defer Pending Requirements"
| stats count(*) by bin(1d)
-- Count escalations to senior underwriter by occupation class and month
fields @timestamp, case_id, occupation_class, escalation_reason
| filter escalation_flag = true
| stats count(*) by occupation_class, datefloor(@timestamp, 1mo)
```
### 7.2.3 Log retention and immutability
CloudWatch Log Groups containing agent audit logs are configured to meet insurance regulatory record-keeping requirements:
| control | configuration |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Retention Policy** | Minimum 7-year retention, aligned with insurance regulatory record-keeping requirements. Configurable to match your organization's own retention schedule. |
| **Log Group Lock (Data Protection)** | Your team applies a CloudWatch Logs data protection policy to prevent log tampering or deletion. Audit logs are append-only from the moment of creation. |
| **Export to S3 and Glacier** | Long-term audit logs are exported from CloudWatch to S3 and transitioned to S3 Glacier Deep Archive for cost-effective long-term retention, encrypted with your KMS CMK. |
| **Azure Equivalent** | Log Analytics Workspace with 7-year retention, immutability policy enabled, and Azure Monitor export to Azure Storage with lifecycle management to archive tier. |
```mermaid theme={null}
flowchart LR
AGT[Agent — Each Reasoning Step] -->|structured JSON| CWL
subgraph cloudwatch ["CloudWatch Log Architecture"]
CWL[CloudWatch Log Group
AUDIT level · append-only]
LGP[Data Protection Policy
tamper-prevention lock]
RET[7-year retention rule]
CWL --> LGP & RET
end
CWL -->|Log Insights| QRY[Audit Queries
by case · by guardrail · by date]
CWL -->|export pipeline| S3ARC[S3 Archive Bucket
CMK-encrypted]
S3ARC -->|lifecycle rule| GLA[S3 Glacier Deep Archive
long-term regulatory retention]
classDef arch fill:#fafafa,stroke:#111,color:#111;
class cloudwatch arch;
```
*Fig. A7.1 — Audit log lifecycle. From step-level emission to Glacier archival, every audit record remains within your account, under your encryption keys, with tamper prevention applied from the moment of creation.*