Skip to content

Monitoring & Observability ​

Proxy Smart includes a comprehensive monitoring system covering OAuth flows, FHIR server health, consent decisions, and admin audit events. Each subsystem provides real-time SSE streams, REST query endpoints, and JSONL export.

Overview ​

SubsystemPrefixSSE StreamsREST EndpointsLog Files
OAuth/admin/monitoring/oauth2 (events, analytics)5logs/oauth-metrics/
FHIR Health/admin/monitoring/fhir2 (checks, summaries)2logs/fhir-health/
FHIR Proxy/admin/monitoring/fhir-proxy2 (events, analytics)2--
Auth Events/admin/monitoring/auth2 (events, analytics)2--
Email Events/admin/monitoring/email2 (events, analytics)2--
Consent/admin/monitoring/consent2 (events, analytics)3logs/consent-metrics/
Admin Audit/admin/monitoring/admin-audit2 (events, analytics)3logs/admin-audit/
System Status/health, /status--2logs/keycloak/

All monitoring endpoints live under /admin and need an Authorization: Bearer token with an admin role, or the read-only proxy-smart-auditor role. SSE streams take the token in the header too, so read them with fetch rather than EventSource, which cannot send one. The REST reads are also available as MCP resources; the streams are not.

Monitoring a deployment from outside ​

Everything above watches one deployment from the inside, for whoever operates it. An external uptime checker or public status page should probe the unauthenticated GET /health instead. It always answers 200 while the process runs, so read the body: status is healthy when the FHIR servers and Keycloak are reachable and degraded when one of them is not. Treat degraded as a degraded service, and a 5xx, a timeout or a body without status as down. Usage analytics are a separate concern and are not part of this monitoring.

Admin UI ​

The Monitoring tab in the admin dashboard is a tabbed container with:

TabContent
DashboardOAuth flow summary stats + FHIR server uptime widget
FHIR ProxyProxied FHIR request metrics -- throughput, response times, error rates by server and resource type
OAuthLive OAuth event table and analytics charts with predictive insights
AuthKeycloak authentication events (logins, failures, session activity)
EmailEmail delivery events from Keycloak (verification, password reset)
ConsentConsent decision monitoring dashboard
Audit LogAdmin audit event log with filtering and JSONL export

FHIR server health is also shown on the main Dashboard page as a status card.

OAuth Monitoring ​

Tracks all OAuth 2.0 authorization flows: token requests, refresh grants, authorization codes, errors.

Endpoints -- /admin/monitoring/oauth ​

MethodPathTypeDescription
GET/events/streamSSEReal-time OAuth flow events
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery events (filter: limit, type, status, clientId, since)
GET/analyticsRESTAnalytics snapshot (totals, rates, top clients, hourly stats)
GET/healthRESTSystem health metrics + alerts
GET/analytics/exportRESTDownload the /analytics snapshot as JSON
GET/events/exportRESTDownload event log as JSONL

Analytics Fields ​

  • totalRequests, successfulRequests, failedRequests, successRate -- counted over the last 24 hours
  • averageResponseTime
  • activeTokens -- access tokens issued in the window that have not expired yet; revocations are not subtracted
  • topClients -- ranked by request count with per-client success rate
  • flowsByType -- breakdown by grant type (authorization_code, refresh_token, etc.)
  • errorsByType -- breakdown by error category
  • hourlyStats -- time series (hour, success, error, total)
  • predictiveInsights -- trend direction, next-hour forecast, anomaly risk assessment
  • weekdayInsights -- per-weekday averages with projections

Health ​

/health derives each status from the analytics window, once there are at least 10 OAuth requests to judge:

  • oauthServer.status -- unhealthy at 50% failed requests, degraded at 10% or when requests average over 1 s. A refresh refused with invalid_grant is not a failure here: the session behind it ended (idle timeout, sign-out, withdrawn access). It still shows in errorsByType
  • network.status -- the same failure thresholds, without the latency rule
  • tokenStore.status -- degraded when the in-memory event buffer is full and analytics cover less than 24 hours

Each non-healthy status comes with an alert that names the numbers behind it.

WebSocket Transport ​

An alternative WebSocket transport is available at /oauth/monitoring/websocket supporting bidirectional messaging (subscribe, unsubscribe, filter, ping/pong). Its auth message takes the same admin or auditor token. Connection info at GET /oauth/monitoring/websocket/info.

Log Files ​

  • logs/oauth-metrics/oauth-events.jsonl -- event log (one JSON object per line)
  • logs/oauth-metrics/oauth-analytics.json -- latest analytics snapshot

FHIR Server Health Monitoring ​

Background poller that checks every registered FHIR server (configured or added in the admin UI) every 30 seconds.

How It Works ​

  1. On startup, FhirHealthLogger initializes and replays recent checks from JSONL
  2. start() begins polling -- asks each server's /metadata live, never from the metadata cache, over the same upstream connection (and client certificate) the proxy uses
  3. Each check produces a FhirHealthCheck under the server's registered identifier, with its response time and status: healthy for a CapabilityStatement of a supported FHIR version, degraded for an answer the proxy cannot use, unhealthy for an HTTP error or no answer
  4. Results are persisted to JSONL and broadcast to SSE subscribers
  5. After each full sweep, uptime summaries are computed and broadcast

Endpoints -- /admin/monitoring/fhir ​

MethodPathTypeDescription
GET/checks/streamSSEReal-time per-check results
GET/summaries/streamSSEUptime summaries after each sweep
GET/checksRESTQuery checks (filter: serverUrl, limit, since)
GET/summariesRESTPer-server uptime summaries

Summary Fields (per server) ​

  • serverName, serverUrl, currentStatus
  • uptimePercent -- percentage of healthy checks
  • avgResponseTimeMs -- average response time across all checks
  • checksTotal, checksHealthy
  • lastChecked, lastError
  • recentChecks -- last ~1 hour of individual check results

Log Files ​

  • logs/fhir-health/health-checks.jsonl -- check history

Memory Limits ​

Ring buffer holds 2,880 entries per server (~24 hours at 30-second intervals).

FHIR Proxy Monitoring ​

Tracks all proxied FHIR requests with per-request metrics.

Endpoints -- /admin/monitoring/fhir-proxy ​

MethodPathTypeDescription
GET/events/streamSSEReal-time proxied request events
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery proxied request events
GET/analyticsRESTAnalytics snapshot (throughput, response times, errors)

Tracked Fields ​

Each proxied request logs: server name, HTTP method, resource path, resource type, status code, response time (ms), client ID, and error details (if any).

Auth Event Monitoring ​

Polls Keycloak authentication events (login, logout, session expiry, failed attempts) every 60 seconds.

Endpoints -- /admin/monitoring/auth ​

MethodPathTypeDescription
GET/events/streamSSEReal-time auth events
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery auth events
GET/analyticsRESTAnalytics snapshot

Email Event Monitoring ​

Polls Keycloak email delivery events (verification emails, password resets, account notifications) every 60 seconds.

Endpoints -- /admin/monitoring/email ​

MethodPathTypeDescription
GET/events/streamSSEReal-time email events
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery email events
GET/analyticsRESTAnalytics snapshot

Tracks patient consent decisions during authorization flows.

MethodPathTypeDescription
GET/events/streamSSEReal-time consent decisions
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery events (filter: limit, decision, clientId, patientId, resourceType, since)
GET/analyticsRESTAnalytics snapshot (last 24h)
GET/events/exportRESTDownload events as JSONL

WebSocket Transport ​

Available at /consent/monitoring/websocket with the same message protocol as the OAuth WebSocket.

Analytics Fields ​

  • totalDecisions, permitRate, denyRate
  • averageCheckDuration, cacheHitRate
  • decisionsByMode -- enforce, audit-only, disabled
  • decisionsByResourceType -- e.g., Patient, Observation, MedicationRequest
  • topDeniedClients, topDeniedPatients
  • hourlyStats

Event Fields ​

Each consent event includes:

  • decision (permit/deny), enforced, mode
  • consentId, patientId, clientId
  • resourceType, method, reason
  • cached, checkDurationMs
  • IAL fields (Identity Assurance Level): actualLevel, requiredLevel, sensitiveResource

Log Files ​

  • logs/consent-metrics/consent-events.jsonl -- event log
  • logs/consent-metrics/consent-analytics.json -- latest analytics

Admin Audit Monitoring ​

Tracks all admin mutations (create, update, delete actions) with actor identification.

Automatic Capture ​

The adminAuditPlugin middleware automatically captures all admin route mutations. No manual instrumentation needed -- it hooks into the Elysia request lifecycle.

Endpoints -- /admin/monitoring/admin-audit ​

MethodPathTypeDescription
GET/events/streamSSEReal-time audit events (mutations only)
GET/analytics/streamSSEReal-time analytics updates
GET/eventsRESTQuery events (filter: limit, action, resource, actor, success, since)
GET/analyticsRESTAnalytics snapshot
GET/events/exportRESTDownload full audit JSONL (streamed)

Event Fields ​

  • Actor: sub. The stored event and the JSONL export keep the user id only; username is looked up in Keycloak when an event is read or streamed, so the actor filter matches names too.
  • Request: method, path, action (create/update/delete/action/read)
  • Target: resource domain, resourceId
  • Result: statusCode, success, durationMs
  • ipAddress, detail

Analytics Fields ​

  • totalActions, successRate
  • actionsByType, actionsByResource
  • topActors -- most active admin users
  • hourlyStats
  • recentFailures -- last failed admin operations

Log Files ​

  • logs/admin-audit/admin-audit.jsonl -- full audit log (replays last 7 days on startup)

System Status & Health ​

GET /health ​

Lightweight liveness probe (30-second TTL cache). Returns:

json
{
  "status": "ok",
  "uptime": 86400
}

GET /status ​

Full system status check including:

  • FHIR Servers -- per-server reachability and response time
  • Keycloak -- connection status, realm accessibility, last connected time
  • Memory -- heap usage, RSS, external
  • Overall -- aggregate health status

Keycloak connection events are logged to logs/keycloak/connections.jsonl and logs/keycloak/last-connection.json.

SSE Stream Protocol ​

All SSE streams follow the same pattern:

  1. Connection event -- sent immediately on connect
    data: {"type":"connection","message":"Connected to ...","timestamp":"..."}
  2. Data events -- sent as they occur
    data: {"id":"...","type":"...","timestamp":"..."}
  3. Keepalive -- every 30 seconds
    data: {"type":"keepalive","timestamp":"..."}

Streams automatically clean up when the client disconnects (detected via controller.desiredSize === null).

Log Directory Structure ​

logs/
├── oauth-metrics/
│   ├── oauth-events.jsonl
│   ├── oauth-analytics.json
│   └── system-health.json
├── fhir-health/
│   └── health-checks.jsonl
├── consent-metrics/
│   ├── consent-events.jsonl
│   └── consent-analytics.json
├── admin-audit/
│   └── admin-audit.jsonl
└── keycloak/
    ├── connections.jsonl
    └── last-connection.json

All JSONL files use newline-delimited JSON (one object per line) and can be downloaded via the export endpoints.

Proxy Smart — Healthcare Interoperability Platform