Monitoring & Observability
Proxy Smart includes a comprehensive monitoring system covering OAuth flows, FHIR server health, consent decisions, and admin audit events. Each subsystem provides real-time SSE streams, REST query endpoints, and JSONL export.
Overview
| Subsystem | Prefix | SSE Streams | REST Endpoints | Log Files |
|---|---|---|---|---|
| OAuth | /admin/monitoring/oauth | 2 (events, analytics) | 5 | logs/oauth-metrics/ |
| FHIR Health | /admin/monitoring/fhir | 2 (checks, summaries) | 2 | logs/fhir-health/ |
| FHIR Proxy | /admin/monitoring/fhir-proxy | 2 (events, analytics) | 2 | -- |
| Auth Events | /admin/monitoring/auth | 2 (events, analytics) | 2 | -- |
| Email Events | /admin/monitoring/email | 2 (events, analytics) | 2 | -- |
| Consent | /admin/monitoring/consent | 2 (events, analytics) | 3 | logs/consent-metrics/ |
| Admin Audit | /admin/monitoring/admin-audit | 2 (events, analytics) | 3 | logs/admin-audit/ |
| System Status | /health, /status | -- | 2 | logs/keycloak/ |
All monitoring endpoints live under /admin and need an Authorization: Bearer token with an admin role, or the read-only proxy-smart-auditor role. SSE streams take the token in the header too, so read them with fetch rather than EventSource, which cannot send one. The REST reads are also available as MCP resources; the streams are not.
Monitoring a deployment from outside
Everything above watches one deployment from the inside, for whoever operates it. An external uptime checker or public status page should probe the unauthenticated GET /health instead. It always answers 200 while the process runs, so read the body: status is healthy when the FHIR servers and Keycloak are reachable and degraded when one of them is not. Treat degraded as a degraded service, and a 5xx, a timeout or a body without status as down. Usage analytics are a separate concern and are not part of this monitoring.
Admin UI
The Monitoring tab in the admin dashboard is a tabbed container with:
| Tab | Content |
|---|---|
| Dashboard | OAuth flow summary stats + FHIR server uptime widget |
| FHIR Proxy | Proxied FHIR request metrics -- throughput, response times, error rates by server and resource type |
| OAuth | Live OAuth event table and analytics charts with predictive insights |
| Auth | Keycloak authentication events (logins, failures, session activity) |
| Email delivery events from Keycloak (verification, password reset) | |
| Consent | Consent decision monitoring dashboard |
| Audit Log | Admin audit event log with filtering and JSONL export |
FHIR server health is also shown on the main Dashboard page as a status card.
OAuth Monitoring
Tracks all OAuth 2.0 authorization flows: token requests, refresh grants, authorization codes, errors.
Endpoints -- /admin/monitoring/oauth
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time OAuth flow events |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query events (filter: limit, type, status, clientId, since) |
GET | /analytics | REST | Analytics snapshot (totals, rates, top clients, hourly stats) |
GET | /health | REST | System health metrics + alerts |
GET | /analytics/export | REST | Download the /analytics snapshot as JSON |
GET | /events/export | REST | Download event log as JSONL |
Analytics Fields
totalRequests,successfulRequests,failedRequests,successRate-- counted over the last 24 hoursaverageResponseTimeactiveTokens-- access tokens issued in the window that have not expired yet; revocations are not subtractedtopClients-- ranked by request count with per-client success rateflowsByType-- breakdown by grant type (authorization_code, refresh_token, etc.)errorsByType-- breakdown by error categoryhourlyStats-- time series (hour, success, error, total)predictiveInsights-- trend direction, next-hour forecast, anomaly risk assessmentweekdayInsights-- per-weekday averages with projections
Health
/health derives each status from the analytics window, once there are at least 10 OAuth requests to judge:
oauthServer.status--unhealthyat 50% failed requests,degradedat 10% or when requests average over 1 s. A refresh refused withinvalid_grantis not a failure here: the session behind it ended (idle timeout, sign-out, withdrawn access). It still shows inerrorsByTypenetwork.status-- the same failure thresholds, without the latency ruletokenStore.status--degradedwhen the in-memory event buffer is full and analytics cover less than 24 hours
Each non-healthy status comes with an alert that names the numbers behind it.
WebSocket Transport
An alternative WebSocket transport is available at /oauth/monitoring/websocket supporting bidirectional messaging (subscribe, unsubscribe, filter, ping/pong). Its auth message takes the same admin or auditor token. Connection info at GET /oauth/monitoring/websocket/info.
Log Files
logs/oauth-metrics/oauth-events.jsonl-- event log (one JSON object per line)logs/oauth-metrics/oauth-analytics.json-- latest analytics snapshot
FHIR Server Health Monitoring
Background poller that checks every registered FHIR server (configured or added in the admin UI) every 30 seconds.
How It Works
- On startup,
FhirHealthLoggerinitializes and replays recent checks from JSONL start()begins polling -- asks each server's/metadatalive, never from the metadata cache, over the same upstream connection (and client certificate) the proxy uses- Each check produces a
FhirHealthCheckunder the server's registered identifier, with its response time and status:healthyfor a CapabilityStatement of a supported FHIR version,degradedfor an answer the proxy cannot use,unhealthyfor an HTTP error or no answer - Results are persisted to JSONL and broadcast to SSE subscribers
- After each full sweep, uptime summaries are computed and broadcast
Endpoints -- /admin/monitoring/fhir
| Method | Path | Type | Description |
|---|---|---|---|
GET | /checks/stream | SSE | Real-time per-check results |
GET | /summaries/stream | SSE | Uptime summaries after each sweep |
GET | /checks | REST | Query checks (filter: serverUrl, limit, since) |
GET | /summaries | REST | Per-server uptime summaries |
Summary Fields (per server)
serverName,serverUrl,currentStatusuptimePercent-- percentage of healthy checksavgResponseTimeMs-- average response time across all checkschecksTotal,checksHealthylastChecked,lastErrorrecentChecks-- last ~1 hour of individual check results
Log Files
logs/fhir-health/health-checks.jsonl-- check history
Memory Limits
Ring buffer holds 2,880 entries per server (~24 hours at 30-second intervals).
FHIR Proxy Monitoring
Tracks all proxied FHIR requests with per-request metrics.
Endpoints -- /admin/monitoring/fhir-proxy
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time proxied request events |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query proxied request events |
GET | /analytics | REST | Analytics snapshot (throughput, response times, errors) |
Tracked Fields
Each proxied request logs: server name, HTTP method, resource path, resource type, status code, response time (ms), client ID, and error details (if any).
Auth Event Monitoring
Polls Keycloak authentication events (login, logout, session expiry, failed attempts) every 60 seconds.
Endpoints -- /admin/monitoring/auth
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time auth events |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query auth events |
GET | /analytics | REST | Analytics snapshot |
Email Event Monitoring
Polls Keycloak email delivery events (verification emails, password resets, account notifications) every 60 seconds.
Endpoints -- /admin/monitoring/email
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time email events |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query email events |
GET | /analytics | REST | Analytics snapshot |
Consent Monitoring
Tracks patient consent decisions during authorization flows.
Endpoints -- /admin/monitoring/consent
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time consent decisions |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query events (filter: limit, decision, clientId, patientId, resourceType, since) |
GET | /analytics | REST | Analytics snapshot (last 24h) |
GET | /events/export | REST | Download events as JSONL |
WebSocket Transport
Available at /consent/monitoring/websocket with the same message protocol as the OAuth WebSocket.
Analytics Fields
totalDecisions,permitRate,denyRateaverageCheckDuration,cacheHitRatedecisionsByMode-- enforce, audit-only, disableddecisionsByResourceType-- e.g., Patient, Observation, MedicationRequesttopDeniedClients,topDeniedPatientshourlyStats
Event Fields
Each consent event includes:
decision(permit/deny),enforced,modeconsentId,patientId,clientIdresourceType,method,reasoncached,checkDurationMs- IAL fields (Identity Assurance Level):
actualLevel,requiredLevel,sensitiveResource
Log Files
logs/consent-metrics/consent-events.jsonl-- event loglogs/consent-metrics/consent-analytics.json-- latest analytics
Admin Audit Monitoring
Tracks all admin mutations (create, update, delete actions) with actor identification.
Automatic Capture
The adminAuditPlugin middleware automatically captures all admin route mutations. No manual instrumentation needed -- it hooks into the Elysia request lifecycle.
Endpoints -- /admin/monitoring/admin-audit
| Method | Path | Type | Description |
|---|---|---|---|
GET | /events/stream | SSE | Real-time audit events (mutations only) |
GET | /analytics/stream | SSE | Real-time analytics updates |
GET | /events | REST | Query events (filter: limit, action, resource, actor, success, since) |
GET | /analytics | REST | Analytics snapshot |
GET | /events/export | REST | Download full audit JSONL (streamed) |
Event Fields
- Actor:
sub. The stored event and the JSONL export keep the user id only;usernameis looked up in Keycloak when an event is read or streamed, so theactorfilter matches names too. - Request:
method,path,action(create/update/delete/action/read) - Target:
resourcedomain,resourceId - Result:
statusCode,success,durationMs ipAddress,detail
Analytics Fields
totalActions,successRateactionsByType,actionsByResourcetopActors-- most active admin usershourlyStatsrecentFailures-- last failed admin operations
Log Files
logs/admin-audit/admin-audit.jsonl-- full audit log (replays last 7 days on startup)
System Status & Health
GET /health
Lightweight liveness probe (30-second TTL cache). Returns:
{
"status": "ok",
"uptime": 86400
}GET /status
Full system status check including:
- FHIR Servers -- per-server reachability and response time
- Keycloak -- connection status, realm accessibility, last connected time
- Memory -- heap usage, RSS, external
- Overall -- aggregate health status
Keycloak connection events are logged to logs/keycloak/connections.jsonl and logs/keycloak/last-connection.json.
SSE Stream Protocol
All SSE streams follow the same pattern:
- Connection event -- sent immediately on connect
data: {"type":"connection","message":"Connected to ...","timestamp":"..."} - Data events -- sent as they occur
data: {"id":"...","type":"...","timestamp":"..."} - Keepalive -- every 30 seconds
data: {"type":"keepalive","timestamp":"..."}
Streams automatically clean up when the client disconnects (detected via controller.desiredSize === null).
Log Directory Structure
logs/
├── oauth-metrics/
│ ├── oauth-events.jsonl
│ ├── oauth-analytics.json
│ └── system-health.json
├── fhir-health/
│ └── health-checks.jsonl
├── consent-metrics/
│ ├── consent-events.jsonl
│ └── consent-analytics.json
├── admin-audit/
│ └── admin-audit.jsonl
└── keycloak/
├── connections.jsonl
└── last-connection.jsonAll JSONL files use newline-delimited JSON (one object per line) and can be downloaded via the export endpoints.