Dara DocsMonitoring
Self-hosted, no SaaS: Prometheus, Alertmanager, Grafana, Loki and Alloy (docker-compose.observability.yml,
deploy/observability/). Locally: make demo then make observability-up — Grafana http://localhost:3300
(dashboards Dara — overview, Dara — Stellar, Dara — logs), Prometheus :9090, Alertmanager :9093.
make observability-check validates the rules and runs their unit tests.
Every backend process serves /metrics on METRICS_ADDR (:9464), never on a public listener. The worker also
answers for database state (open intents, proposals, channel pool, clawbacks, incidents) at scrape time. Gauges set
by a worker loop are read only from the replica that leads the loop (worker_loop_leader == 1).
Alert receivers (who is paged, through which channel) are still to be chosen (owner decision 2026-10-31: later).
Until then alerts are visible in Alertmanager and Grafana only. deploy/observability/alertmanager/production.yml.example
is ready for Telegram, email or a webhook (secrets in files); fill it in and deploy it as alertmanager.yml. Critical = act now (money, supply or trading at risk);
warning = act within business hours.
Series
| Area | Series |
|---|---|
| Processes | up, process_start_time_seconds, worker_loop_leader{loop}, worker_loop_last_success_timestamp_seconds{loop} (loops: txmanager, gate, indexer, reconciliation, sweep), db_up, db_pool_* |
| Money | token_supply{asset}, reserve_capacity{asset}, treasury_inventory{asset} (HOT), reconciliation_last_run_passed, reconciliation_mismatches_total{check}, ledger_imbalance_detected_total, incidents_open, incidents_unresolved{severity}, clawbacks{status} |
| Business | orders_total{kind,status}, order_settlement_seconds, supply_operations_total{kind,status}, ledger_postings_total{kind}, http_requests_total, http_request_duration_seconds |
| Stellar | stellar_rpc_node_{healthy,latest_ledger,oldest_ledger,ledger_age_seconds,latency_seconds}{node}, stellar_rpc_node_sample_failures_total, stellar_intents{status}, stellar_tx_submissions_total{result}, stellar_intents_failed_total{purpose}, stellar_fee_charged_stroops_total{purpose}, stellar_channels{pool,state}, stellar_xlm_balance_stroops{role,account}, signing_proposals_awaiting, signing_proposal_oldest_age_seconds |
| Gate, indexer | trading_gate_open, trading_gate_condition{condition}, indexer_last_ledger, indexer_lag_ledgers, token_unexplained_movements_total |
Alerts
DaraWorkerDown
No worker process is up. Nothing is submitted, indexed or reconciled; the gate state ages out and trading is refused.
Check the worker container/unit logs for the startup error — the most common is the startup verification of the
account manifest against the chain (app: the chain does not match the manifest), which must never be bypassed.
DaraAPIDown
No customer API is reachable. Check the process, its database connectivity (DatabaseDown) and /readyz.
DaraInstanceDown
One instance is not scraped; others still serve. Restore redundancy.
DaraWorkerLoopStalled
A loop has not finished an error-free pass for 5 minutes. Its errors are in the worker log with loop=<name>.
txmanager: RPC or signer problems; indexer: a gap or an unreadable ledger (INDEXER_GAP incident); gate: database.
DaraWorkerLoopWithoutLeader
Workers run but none holds the loop's advisory lock: a stuck session holds it. Find it in pg_locks
(locktype = 'advisory') and terminate the stale backend; the loop recovers by itself.
DaraReconciliationFailed
The last run found a mismatch or could not complete. Admin → Reconciliation shows the node, ledger and failing checks;
each failure opened an incident. Follow RECONCILIATION_RUNBOOK.md. Never correct balances by hand before the cause
is known.
DaraReconciliationStale
No run completed for an hour. See DaraWorkerLoopStalled for loop reconciliation.
DaraLedgerImbalance
A ledger transaction does not balance. Stop manual postings; escalate to engineering with the run ID.
DaraUnexplainedTokenMovement
An asset moved on-chain with no intent: possible key compromise. Trading is already stopped by a CRITICAL incident.
Follow INCIDENT_RESPONSE.md (key compromise): identify the source account and key, rotate it through the human
signers, and use freeze/clawback to contain.
DaraSupplyAboveBacking
Supply exceeds effective backing (shortfall, D-024): issuance and BUY are blocked for the asset. Usually an expired attestation; record a new approved reserve snapshot or burn from cold.
DaraClawbackApplied
A clawback was applied. Confirm it in Admin → Clawbacks against an approved request with a legal reference
(CLAWBACK_POLICY.md); a clawback without one is an unexplained movement and pages separately.
DaraIncidentUnresolved
An incident has been open for an hour without resolution. Assign an owner; resolve with a note.
DaraTradingGateClosed
Trading is refused while the named condition holds; it reopens by itself after the hold-down.
no_healthy_node/ledger_stale → Stellar alerts below; signer_unavailable → the signer backend;
funder_balance_low/channels_unfunded → top up the funder (it funds channels); unknown_submissions → submissions
whose outcome is unknown are being proven by the engine, check node health; blocking_incident → resolve it;
treasury_not_active → the hot Treasury account is not registered.
DaraFunderXLMLow
The fee funder holds less than 3× its gate minimum. Send XLM to the FUNDER account (its address is on the System page and in the manifest) from the operator's XLM treasury.
DaraSponsorXLMLow
The reserve sponsor is low: new customers cannot be onboarded when it runs out (each sponsored account and trustline locks base reserve). Top it up.
DaraAPIErrors
More than 2% of requests fail with 500. The log line carries the request_id.
DaraAPISlow
p95 latency above 1 s for 15 minutes. Check database load and RPC latency.
StellarNoHealthyNode
None of our RPC nodes is healthy: nothing can be read or submitted and the gate is closed. Check each node
(System page shows the reason: wrong passphrase, stale ledger, unreachable). See STELLAR_RPC_RUNBOOK.md.
StellarNodeUnhealthy
One node is out of rotation; the others serve. Restore it before the redundancy is needed.
StellarLedgerStale
The newest ledger any of our nodes has seen is over a minute old. Compare with an independent view of the network (stellar.expert or a second, separately hosted node); if the network itself is halted, wait — the platform refuses to trade on stale data by design.
StellarIndexerLagging
The indexer trails by more than 60 ledgers (~5 minutes). Dara Scan is stale and reconciliation fails
indexer_current. If the node's history window has passed the cursor, the indexer reads our Galexie lake (deploy/datalake/): check that Galexie is running and the lake is mounted.
StellarIntentsReconciliationRequired
An intent's outcome could not be proven (a late ledger answer after the platform gave up on it, or conflicting
evidence). Its holds stay in place. Investigate the attempts' hashes on-chain; resolve per TRANSACTION_FINALITY.md.
StellarIntentsStuck
Platform-signed intents in flight for 15 minutes. Normal is seconds: check nodes, channel balances and UNKNOWN submissions.
SigningProposalWaiting
A human-signed proposal has waited 12 hours. It expires at 24 hours and is rebuilt; remind the issuer or Treasury signers (Admin → Signing shows which account still needs weight).
SigningChannelsExhausted
Every GOV channel is held by a proposal awaiting human signatures, so new issuance, refill, burn or clawback proposals wait. Have the pending proposals signed (Admin → Signing) or let them expire; add GOV channels if this recurs (one per proposal that may be pending at the same time; the ceremony creates three).
StellarTransactionFailures
More than three intents failed in an hour. Their holds were released; look for a common result code (Admin → Chain operations) — usually an account or configuration problem.
DatabaseDown
A process cannot reach PostgreSQL. Everything else follows from it (inhibited while this fires).
DatabasePoolSaturated
More than 90% of a process's pool is in use for 10 minutes. Look for long transactions before raising the pool size.