Observability

For operators and security teams who need to know what CoSec decided, why, and who changed the rules. This page covers the decision listeners that observe every request, the audit events, spans and metrics they produce, how to see whether instances are current with the documents, the control plane's change log, and the logs. To see every statement a decision checked, not only the one that decided, explain it through the control plane (Explaining a decision).

Decision listeners

Every request through the pipeline ends in one decision event, whatever its outcome: a permit, a deny, a 429, a 400, a 503, a defect (500), or a caller that went away. Decision listeners receive these events; the audit log, Kafka, metrics, tracing and shadow windows are all listeners. A listener observes only: it cannot change a decision, except that a REQUIRED one that fails turns a permit into 503.

Delivery	When the listener runs	What a failure does
REQUIRED	Before the response, together with the other REQUIRED listeners, each within cosec.observability.listener.required-timeout (1 second) and within what is left of cosec.pipeline.timeout. It sees the outcome as decided so far.	A permit becomes 503 unavailable; the request does not proceed. Any other outcome keeps its status; the failure is logged at ERROR and counted.
BEST_EFFORT (the default)	After the response is decided, off the request path. Each listener has its own queue of cosec.observability.listener.queue-capacity events (1,024) and its own consumer, so a slow listener never costs another its events. It sees the final outcome.	Nothing for the request. A full queue drops the event; a listener that throws is logged at WARN at most every 10 seconds, and both are counted.

The listeners the starter registers:

Listener (name)	Delivery	Present when
LoggingAuditListener (logging-audit)	BEST_EFFORT	cosec.audit.enabled (default true)
KafkaAuditListener (kafka-audit)	cosec.audit.kafka.delivery, BEST_EFFORT by default	cosec-kafka, a KafkaOperations bean, cosec.audit.enabled and cosec.audit.kafka.enabled (both default true)
MetricsListener (metrics)	BEST_EFFORT	cosec-micrometer, a MeterRegistry bean (Spring Boot's own auto-configured one included) and cosec.metrics.enabled (default true)
TracingListener (tracing)	BEST_EFFORT	cosec-opentelemetry
ShadowListener (shadow)	BEST_EFFORT	cosec-cocache and cosec.authorization.shadow.enabled (default true)

Any DecisionListener bean of yours joins them; names must be unique, or startup fails (Extending). An AuditEventSink bean joins them too: the starter wraps it as a BEST_EFFORT listener named after the bean (AuditListener.of(sink, beanName)), in its @Order among the listeners. It is always BEST_EFFORT, because AuditEventSink.publish cannot acknowledge a write. A bean that is both is registered once, as the DecisionListener it is.

• Break-glass needs a REQUIRED listener. A request permitted through a break-glass binding is 503 unless at least one REQUIRED listener is registered and every one of them records it. Out of the box no listener is REQUIRED, so the starter logs at startup No REQUIRED decision listener is registered: break-glass bindings cannot permit (RFC 0006 §9.4.3). Set cosec.audit.kafka.delivery=required, or register a REQUIRED listener of your own, before you rely on break-glass.
• A REQUIRED listener should be idempotent by the event's id: a write that timed out may still have succeeded.
• On shutdown, the listeners stop taking events, and the events already queued are still delivered for up to cosec.observability.listener.drain-timeout (5 seconds). What is left then is dropped, with Decision listeners did not deliver every queued event within <timeout>; the rest is dropped. at WARN. The Kafka audit listener then gets up to the same timeout for the sends it still holds and the broker's answers to them, before the producer is closed. At that bound its queued sends are dropped and a send in progress is interrupted, with The Kafka audit listener of [<topic>] did not finish within <timeout>: ... at WARN.
• The evaluation trace. A DecisionEvent carries evaluationTrace, what the engine looked at, to the depth of cosec.pipeline.trace-level: off (the default, no trace and no cost), full (every rule of each layer the decision reaches, including those after the deciding one) or exhaustive (also the layers it does not reach). It is null at off and when the request did not reach the engine. Only your own listeners read it: the audit events, spans and metrics below do not include it.
• Services in gateway mode (cosec.pipeline.mode=gateway) emit no decision events: they decide nothing and register no listeners. The gateway in front records the decision (Deploying the gateway server).

Audit events

The audit listeners map each decision event to an AuditEvent. It keeps what the response body leaves out: explicit and implicit denies apart, why a credential was rejected, and why a request was 503.

Field	Contents
eventId, timestamp, tenantId	the event; tenantId is the principal's tenant
kind	DECIDED, BAD_REQUEST (400), AUTHENTICATION_UNAVAILABLE, CANCELLED or ERROR (500)
principal	id, authenticated, roles (the roles of the resolved scope)
request	id (the CoSec-Request-Id), appId, spaceId, remoteIp, method, path (normalized), device.id
authorization.decision, authorization.reasonCode	see the table below
authorization.reason	the outcome, and for a 503 its cause, such as INDETERMINATE: SourceUnavailable
authorization.elapsedNanos	the whole pipeline, from the start of the request to its outcome
authorization.source	what decided: via, document, policy (its id) and statement (index, name); see the second table below
binding	how the deciding statement applied: a binding id (a role's derived binding is role:<roleId>@<app>:<permission>), attached:<policyId>, builtin (the blacklist) or boundary:<tenantId>
snapshotVersion	the revisions the decision read: platform, and app and tenant when it read them
breakGlass	whether the permit came through a break-glass binding
trace	traceId and spanId of the request's span, when tracing is active

Outcome	decision	reasonCode
permit	ALLOW	ALLOW
429	TOO_MANY_REQUESTS	TOO_MANY_REQUESTS
a DENY matched	EXPLICIT_DENY	EXPLICIT_DENY
scope rejected	EXPLICIT_DENY	SCOPE_REJECTED
a permit outside the tenant's boundary	EXPLICIT_DENY	BOUNDARY
nothing allowed	IMPLICIT_DENY	IMPLICIT_DENY
any deny of a request whose credential was rejected	as above	TOKEN_EXPIRED, TOKEN_REVOKED or TOKEN_INVALID (also for an API key); DELEGATION_UNSUPPORTED for a token with an act claim, which the response reports as token-invalid. The principal stays anonymous; authorization.reason names whose credential was refused when that is known (a key that does not work, a principal the membership provider does not know or knows as deactivated), as the decision event's subject.refused does
400	EXPLICIT_DENY	EXPLICIT_DENY (with kind BAD_REQUEST)
503, a strict rate-limit rule that could not be counted	INDETERMINATE	RATE_LIMITER_UNAVAILABLE
any other 503, and a cancelled request	INDETERMINATE	INDETERMINATE
500	ERROR	ERROR

authorization.source.via says how what decided came to apply, and document which snapshot document holds it:

via	document	Meaning
BINDING	PLATFORM, APP or TENANT	a binding of that document, a role's derived binding included; binding is its id. A break-glass permit is a binding of cosec:root, with breakGlass: true
ATTACHMENT	null	a policy attached through the principal's memberships
BUILTIN	null	the built-in blacklist
BOUNDARY	PLATFORM	the tenant's platform boundary refused a permit. policy is the boundary and statement is null, since none of its statements matched
NONE	null	no statement decided: nothing allowed the request, its scope was rejected, or no decision was made. policy and statement are null

Audit log and Kafka

The audit log listener writes each event as one JSON line to the logger me.ahoo.cosec.audit: ALLOW at DEBUG, denials, TOO_MANY_REQUESTS and INDETERMINATE at WARN, ERROR at ERROR. At the default INFO level, only denials, 503s and errors appear.

The Kafka audit listener writes the same JSON to cosec.audit.kafka.topic (cosec-audit), keyed by [tenantId, principalId] as a JSON array. It runs alongside the audit log, not instead of it; set cosec.audit.enabled=false and register only what you want to change that. It sends from a thread of its own with at most 1,024 sends waiting:

• best-effort: a failed send is logged as Failed to publish audit event to Kafka topic [<topic>].
• required: the request waits for the broker's acknowledgement. A failed or late send, or a full send queue (The Kafka audit queue of [<topic>] is full.), makes a permit 503. Configure the producer with acks=all.

Tracing

With cosec-opentelemetry, each decision event becomes a span named cosec.authorize (instrumentation me.ahoo.cosec, kind INTERNAL). The span is made after the decision, from the request's start time and the pipeline's duration, under the span that was current when the adapter received the request; without one, it is a root span. Tracing therefore never delays or fails a request. Calls the pipeline makes for the request (Redis, a membership provider) are siblings of cosec.authorize, under the same parent. The tracer comes from GlobalOpenTelemetry.

Attribute	Value
cosec.tenant_id, user.id, cosec.request_id	the principal's tenant and id, and the request id
cosec.app_id, cosec.space_id, device.id	the resolved scope and the device, when not empty
cosec.authorize.result	PERMIT, TOO_MANY_REQUESTS, EXPLICIT_DENY, IMPLICIT_DENY, SCOPE_REJECTED, BOUNDARY, INDETERMINATE, AUTHENTICATOR_UNAVAILABLE, DEADLINE_EXCEEDED, INVALID_REQUEST, CANCELLED or ERROR
cosec.authorize.policy.id, cosec.authorize.statement.index, cosec.authorize.statement.name	the deciding statement
cosec.binding.id	the binding it applied through, when it was a binding
cosec.snapshot.version	the revisions read, as <platform>/<app>/<tenant>, - for a document not read

A 500 sets the span's status to ERROR. Audit events and problem bodies carry the trace id of the request's span, which ties a response to its trace.

Metrics

With cosec-micrometer and a MeterRegistry (for example from Spring Boot Actuator), every decision event is recorded by the timer cosec.authorization, over the whole pipeline. Every tag has a small fixed set of values, so the number of time series stays bounded however many policies, apps and tenants there are:

Tag	Values
result	allow, too_many_requests, explicit_deny (scope rejections and boundary denies included), implicit_deny, indeterminate (every 503), bad_request, cancelled, error
source	binding, attachment, builtin, boundary, scope (a scope rejection), none
cause	on indeterminate only: source_unavailable, missing_policies, membership, attribute, matcher, authenticator, required_listener, boundary, rate_limiter, deadline, unknown

The listeners themselves are counted:

Meter	Tag	Counts
cosec.listener.dropped	listener	events a full queue dropped
cosec.listener.failures	listener	listener calls that threw or, for REQUIRED ones, timed out
cosec.audit.required.failures	none	REQUIRED failures on decisions they could not change (not permits)

Alert on the rate of result=indeterminate by cause, on explicit_deny, and on cosec.listener.dropped. A rise of indeterminate with cause=source_unavailable means documents cannot be read (Troubleshooting). CoCache records its own meters for CoSec's caches in the same registry (CoCache's documentation).

Snapshot health

CoSec registers no meters for the documents an instance holds. To see whether instances are current:

• Health. With Spring Boot Actuator, the cosecSnapshot health indicator is UP once the platform document and every prewarmed document are installed, DOWN before, with the details platform (installed, failed, absent or loading) and prewarmed. Readiness follows the same condition (Readiness).
• Eviction. On Redis, the cosecSnapshotEviction health indicator is DOWN while Redis may evict the snapshot, with an eviction detail that says why. It reports what the instance read last: every node's maxmemory-policy, read every revalidate-interval on a thread of its own, never by a probe. A policy it cannot read is DOWN under eviction-check: fail, and UP with eviction: unknown under warn. It is an alert, not a readiness signal: keep it out of the readiness group, where one change to Redis would take every instance out of rotation while requests are still decided (Redis requirements). It also turns the aggregate /actuator/health DOWN, as every indicator does: point load balancers and platform health checks at the readiness group, or leave it out of the group they poll (management.endpoint.health.group.<group>.exclude=cosecSnapshotEviction).
• Lag logs. An instance logs a document not confirmed by Redis for longer than lag-warn at WARN, and longer than lag-escalate at ERROR (How instances read documents). Other lines to alert on: Snapshot document [<doc>] disappeared from the store; failing closed., Snapshot document [<doc>] regressed from revision <a> to <b>; ..., and Snapshot prewarm failed for [<docs>]; retrying in <backoff>.
• Propagation. The control plane's GET /documents/<doc>/revisions/<n>/propagation answers how many of the instances that reported in the last minute have revision <n> or later installed. An instance counts a revision as installed only while it can evaluate it: one that read the revision but could not compile it still reports, so it is among those counted, but it has not installed it. Instances report only with shadow windows on (Revisions, history and rollback).
• Per decision. snapshotVersion in the audit event and cosec.snapshot.version on the span name the revisions a decision read.

Control-plane change log

The control plane emits a change event for every step of a change: PROPOSED, CHECKED, SHADOW_STARTED, SHADOWED, APPROVED, REJECTED, PUBLISHING, PUBLISHED, FAILED, BREAK_GLASS_REQUESTED, BREAK_GLASS_APPROVED, REVOKED, HEADS_MOVED, REVIEW_NEEDED, ABANDONED. An event names the change, the document, the operator, the base and candidate revisions, what needed approval, the confirmations, the shadow summary and the note; it never carries a document's content. Change listeners receive them.

• The default listener logs each event at INFO, as Change <changeId> <kind> on <doc> by <principal>@<tenant>: base=..., candidate=..., escalations=<n>, confirmations=<n>. Change listeners add up: every ChangeListener bean is registered, and the log listener is left out only when the application declares a LoggingChangeListener bean of its own.
• PUBLISHING, BREAK_GLASS_REQUESTED and BREAK_GLASS_APPROVED are gated: every REQUIRED change listener must record the event within cosec.control.audit.required-timeout (5 seconds, for all of them together) before the step takes effect, or the step does not happen and the control API answers 503. Every other event, and every event to a BEST_EFFORT listener, is delivered afterwards; a failure is logged as Change listener [<name>] could not record <event>. The revision record is the durable record of what was published. A gated event says the step was about to happen, not that it did: a check after it can still refuse the step (a 409 when the change moved meanwhile), as after a BREAK_GLASS_APPROVED or an ABANDONED. The change's state, and for a publish the revision history, say what happened.
• A note is free text, part of it the change's author's (the reason of a proposal, a request or a revoke): read who did a step from the event's operator, never from its note.
• Break-glass needs a REQUIRED change listener: without one, creating or changing a break-glass binding answers 503 (break-glass needs a REQUIRED change listener, and none is configured).
• ABANDONED names who abandoned a change and, in its note, its author (author <tenant>/<principal>: <note>). Abandoning a break-glass change is gated like PUBLISHING: it needs a REQUIRED change listener (503 without one), which records it before the change is stored as abandoned. Abandoning any other change is recorded afterwards.
• REVOKED of a stalled revoke published again names the revoker who revoked again, and its note adds their reason (<first reason>; published again by <tenant>/<principal>: <reason>), as does its PUBLISHING.
• HEADS_MOVED and REVIEW_NEEDED follow a PUBLISHED when another document the change was checked with moved before its write landed (What a publish compares). Each names the heads that moved (moved: the document, or list:apps for the app list, the digest it was checked with, the digest it has now, and whether it could be read), carries the escalations the grant analysis finds on the current heads, and says in its note what that means. HEADS_MOVED needs nothing; REVIEW_NEEDED means the analysis found an escalation the publish did not have approved, could not run, or the comparison did not finish in time (moved is then empty, and the one escalation has the reason DETECTION_TIMED_OUT): a person should review the published change, and the control plane also logs Change <changeId> published on [<doc>]: <note>; moved [...]. at WARN. They are recorded even when the client that published disconnects. Like every event that is not gated, they are best effort: the publish has already happened.
• Escalations in an event are subject, reason and detail (Testing changes). A TENANT_BINDING one names only the binding and its tenant document (binding <id> of [tenant:<t>]), with the policy and the statements in detail; whom the binding binds, and in which space, is the tenant's and is left out. It is informational: it never counts as what needed approval.

With cosec-kafka and a KafkaOperations bean, the starter also registers a KafkaChangeListener, unless cosec.control.audit.kafka.enabled is false. It writes each event to cosec.control.audit.kafka.topic (cosec-control-audit), keyed by the document (platform, app:order, tenant:acme), so a document's events stay in order. Its delivery is cosec.control.audit.kafka.delivery: best-effort by default, required to gate publishing and break-glass on the broker's acknowledgement (configure the producer with acks=all). With delivery=required, startup fails when the Kafka change listener cannot be created (no cosec-kafka, no KafkaOperations bean, or enabled=false), rather than publishing without the record: cosec.control.audit.kafka.delivery=required, but no Kafka change listener was created: ....

Logs

Logger	What it logs
me.ahoo.cosec.audit	the audit log (Audit log and Kafka)
me.ahoo.cosec.runtime	Request [<requestId>] failed with an unmodelled exception. at ERROR with the stack trace, for every 500; REQUIRED listener failures; dropped events; attribute providers that fail; rate limiters that cannot count
me.ahoo.cosec.runtime.snapshot	loading, lag, lost and regressed documents (Snapshot health)
me.ahoo.cosec.spring.boot.starter	the startup checks, such as the !!! eviction warning; No BlacklistSource bean: requests are not checked against a blacklist (ext.blacklist.*). at INFO when the application declares no BlacklistSource (Blacklist); the WARN of a service in gateway mode (Services behind the gateway)
me.ahoo.cosec.control	the change log, and Control API <method> <path> failed (request <requestId>). for every control-plane 500

Correlate a response with its log lines, audit event and span through the CoSec-Request-Id response header, which is also the requestId of every problem body.

Next: go from a symptom to its cause, in Troubleshooting.