Skip to content

ProductLogs

Know when your logs say something is wrong.

Write the query once. Observer turns it into one number, judges it against your thresholds and puts the result on your status page. The lines never leave your log store.

Start freeLoki guide

Loki: the query returns one number, 3 failures. Healthy under 5, unhealthy over 25. Verdict: Healthy.

In your log store

The query

LogQL
sum(count_over_time(
  {app="checkout"} |= "payment_failed" [5m]))

What Loki answers

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      { "metric": {}, "value": [1791381895, "3"] }
    ]
  }
}

One number

Payment failures, last 5 minutes

3failures

The verdict

Healthy
< 5
Unhealthy
> 25
Otherwise
degraded

Healthy

3 is under 5: the healthy rule matches.

Your status page

Shown as each reading lands. A real page waits out the confirmation time first, 3 minutes by default.

status.example.com
All systems operational

Services

  • Checkout
    100%Operational
15 OctNow
The status page is the product's own components, drawing sample data. Switch the log store or step through the readings.

Boundary

Only the number leaves your log store.

The agent asks your log store for one aggregate and sends that number out. Observer never stores a log line, and never sees your token.

Stored in Observer

Cloud
key: payment-failures
source_type: loki
source_config:
  base_url: http://loki.monitoring:3100
  query: |
    sum(count_over_time(
      {app="checkout"} |= "payment_failed" [5m]))
  auth_mode: bearer
  token_ref: LOKI_TOKEN

The token is a name. Its value is read on the agent host when the query runs.

Only on the agent host

Your network
# agent environment
LOKI_TOKEN=glc_••••••••••••••••

Loki answers the agent with one number. The lines it counted stay in Loki.

Sent to Observer

Each reading
{
  "metric_id": "5c0e…",
  "value": 31,
  "timestamp": "2026-10-06T12:05:00Z",
  "status": "unhealthy"
}

The reading and the agent's verdict. A failed read adds a reason and no value.

Elasticsearch

No documents come back.

The agent forces size to 0 on every search and reads one named aggregation: a count, an average, a percentile.

OpenSearch

Same search API.

Point the Elasticsearch source at OpenSearch. Only the labels in the console change.

Auth

Bearer, basic or API key.

Loki takes a bearer token or basic auth, plus a tenant ID for multi-tenant setups. Elasticsearch also takes an API key.

Guardrails

A broken query tells you, instead of lying to you.

Some mistakes are refused when you save. The rest show up as no data with a reason, never as a quiet healthy.

A query that returns log lines

{app="checkout"} |= "payment_failed"

Refused

When you save

LogQL query must be a metric aggregation (count_over_time, rate, sum(...), …), not a raw log stream

Counting per stream instead of in total

count_over_time({app="checkout"} |= "payment_failed" [5m])

No data

Saves; caught on the agent

loki_multiple_series: one series per stream comes back, so there is no single number to judge. Wrap it in sum(...).

A search with nothing to read

{"query":{"range":{"@timestamp":{"gte":"now-5m"}}}}

Refused

When you save

query must contain an `aggs` (or `aggregations`) block that produces the metric value

An aggregation name that doesn't match

agg_name: error
aggs: { "errors": { "value_count": { "field": "level" } } }

No data

Saves; caught on the agent

es_agg_not_found, with the aggregation names the response did contain.

A token the agent host doesn't have

auth_mode: bearer
token_ref: LOKI_TOKEN

No data

On the agent

loki_auth_ref_missing, naming LOKI_TOKEN, instead of a silent 401.

Then

From here, it's just a metric.

Thresholds, the confirmation window, services, SLOs and drafted incidents work the same as for any other source. This is the console chart for the Loki query above.

Console / Metrics / Payment failures
Payment failures, average per minute, last 24h. Thresholds: healthy < 5, unhealthy > 25. Showing the most recent 200 of 289 rows.
Time (UTC)AverageMinMaxStatus
2026-11-12 09:50:00 UTC223Healthy
2026-11-12 09:55:00 UTC111Healthy
2026-11-12 10:00:00 UTC334Healthy
2026-11-12 10:05:00 UTC222Healthy
2026-11-12 10:10:00 UTC324Healthy
2026-11-12 10:15:00 UTC445Healthy
2026-11-12 10:20:00 UTC111Healthy
2026-11-12 10:25:00 UTC334Healthy
2026-11-12 10:30:00 UTC111Healthy
2026-11-12 10:35:00 UTC324Healthy
2026-11-12 10:40:00 UTC111Healthy
2026-11-12 10:45:00 UTC222Healthy
2026-11-12 10:50:00 UTC334Healthy
2026-11-12 10:55:00 UTC435Healthy
2026-11-12 11:00:00 UTC111Healthy
2026-11-12 11:05:00 UTC111Healthy
2026-11-12 11:10:00 UTC445Healthy
2026-11-12 11:15:00 UTC324Healthy
2026-11-12 11:20:00 UTC223Healthy
2026-11-12 11:25:00 UTC444Healthy
2026-11-12 11:30:00 UTC222Healthy
2026-11-12 11:35:00 UTC435Healthy
2026-11-12 11:40:00 UTC222Healthy
2026-11-12 11:45:00 UTC111Healthy
2026-11-12 11:50:00 UTC334Healthy
2026-11-12 11:55:00 UTC222Healthy
2026-11-12 12:00:00 UTC334Healthy
2026-11-12 12:05:00 UTC435Healthy
2026-11-12 12:10:00 UTC333Healthy
2026-11-12 12:15:00 UTC111Healthy
2026-11-12 12:20:00 UTC334Healthy
2026-11-12 12:25:00 UTC223Healthy
2026-11-12 12:30:00 UTC324Healthy
2026-11-12 12:35:00 UTC334Healthy
2026-11-12 12:40:00 UTC222Healthy
2026-11-12 12:45:00 UTC111Healthy
2026-11-12 12:50:00 UTC222Healthy
2026-11-12 12:55:00 UTC111Healthy
2026-11-12 13:00:00 UTC334Healthy
2026-11-12 13:05:00 UTC435Healthy
2026-11-12 13:10:00 UTC222Healthy
2026-11-12 13:15:00 UTC222Healthy
2026-11-12 13:20:00 UTC222Healthy
2026-11-12 13:25:00 UTC435Healthy
2026-11-12 13:30:00 UTC435Healthy
2026-11-12 13:35:00 UTC222Healthy
2026-11-12 13:40:00 UTC333Healthy
2026-11-12 13:45:00 UTC111Healthy
2026-11-12 13:50:00 UTC111Healthy
2026-11-12 13:55:00 UTC445Healthy
2026-11-12 14:00:00 UTC222Healthy
2026-11-12 14:05:00 UTC324Healthy
2026-11-12 14:10:00 UTC444Healthy
2026-11-12 14:15:00 UTC434Healthy
2026-11-12 14:20:00 UTC334Healthy
2026-11-12 14:25:00 UTC111Healthy
2026-11-12 14:30:00 UTC445Healthy
2026-11-12 14:35:00 UTC334Healthy
2026-11-12 14:40:00 UTC324Healthy
2026-11-12 14:45:00 UTC222Healthy
2026-11-12 14:50:00 UTC111Healthy
2026-11-12 14:55:00 UTC334Healthy
2026-11-12 15:00:00 UTC334Healthy
2026-11-12 15:05:00 UTC111Healthy
2026-11-12 15:10:00 UTC334Healthy
2026-11-12 15:15:00 UTC333Healthy
2026-11-12 15:20:00 UTC111Healthy
2026-11-12 15:25:00 UTC334Healthy
2026-11-12 15:30:00 UTC333Healthy
2026-11-12 15:35:00 UTC223Healthy
2026-11-12 15:40:00 UTC222Healthy
2026-11-12 15:45:00 UTC444Healthy
2026-11-12 15:50:00 UTC111Healthy
2026-11-12 15:55:00 UTC223Healthy
2026-11-12 16:00:00 UTC334Healthy
2026-11-12 16:05:00 UTC111Healthy
2026-11-12 16:10:00 UTC111Healthy
2026-11-12 16:15:00 UTC445Healthy
2026-11-12 16:20:00 UTC323Healthy
2026-11-12 16:25:00 UTC222Healthy
2026-11-12 16:30:00 UTC444Healthy
2026-11-12 16:35:00 UTC222Healthy
2026-11-12 16:40:00 UTC435Healthy
2026-11-12 16:45:00 UTC223Healthy
2026-11-12 16:50:00 UTC222Healthy
2026-11-12 16:55:00 UTC223Healthy
2026-11-12 17:00:00 UTC111Healthy
2026-11-12 17:05:00 UTC111Healthy
2026-11-12 17:10:00 UTC334Healthy
2026-11-12 17:15:00 UTC111Healthy
2026-11-12 17:20:00 UTC435Healthy
2026-11-12 17:25:00 UTC333Healthy
2026-11-12 17:30:00 UTC333Healthy
2026-11-12 17:35:00 UTC222Healthy
2026-11-12 17:40:00 UTC445Healthy
2026-11-12 17:45:00 UTC223Healthy
2026-11-12 17:50:00 UTC111Healthy
2026-11-12 17:55:00 UTC111Healthy
2026-11-12 18:00:00 UTC435Healthy
2026-11-12 18:05:00 UTC223Healthy
2026-11-12 18:10:00 UTC222Healthy
2026-11-12 18:15:00 UTC111Healthy
2026-11-12 18:20:00 UTC111Healthy
2026-11-12 18:25:00 UTC435Healthy
2026-11-12 18:30:00 UTC323Healthy
2026-11-12 18:35:00 UTC324Healthy
2026-11-12 18:40:00 UTC333Healthy
2026-11-12 18:45:00 UTC222Healthy
2026-11-12 18:50:00 UTC111Healthy
2026-11-12 18:55:00 UTC334Healthy
2026-11-12 19:00:00 UTC435Healthy
2026-11-12 19:05:00 UTC222Healthy
2026-11-12 19:10:00 UTC222Healthy
2026-11-12 19:15:00 UTC222Healthy
2026-11-12 19:20:00 UTC223Healthy
2026-11-12 19:25:00 UTC111Healthy
2026-11-12 19:30:00 UTC222Healthy
2026-11-12 19:35:00 UTC111Healthy
2026-11-12 19:40:00 UTC222Healthy
2026-11-12 19:45:00 UTC334Healthy
2026-11-12 19:50:00 UTC111Healthy
2026-11-12 19:55:00 UTC111Healthy
2026-11-12 20:00:00 UTC333Healthy
2026-11-12 20:05:00 UTC181521Degraded
2026-11-12 20:10:00 UTC393349Unhealthy
2026-11-12 20:15:00 UTC191721Degraded
2026-11-12 20:20:00 UTC121014Degraded
2026-11-12 20:25:00 UTC181622Degraded
2026-11-12 20:30:00 UTC342838Unhealthy
2026-11-12 20:35:00 UTC282334Unhealthy
2026-11-12 20:40:00 UTC262230Unhealthy
2026-11-12 20:45:00 UTC222Healthy
2026-11-12 20:50:00 UTC222Healthy
2026-11-12 20:55:00 UTC334Healthy
2026-11-12 21:00:00 UTC111Healthy
2026-11-12 21:05:00 UTC111Healthy
2026-11-12 21:10:00 UTC111Healthy
2026-11-12 21:15:00 UTC323Healthy
2026-11-12 21:20:00 UTC111Healthy
2026-11-12 21:25:00 UTC111Healthy
2026-11-12 21:30:00 UTC111Healthy
2026-11-12 21:35:00 UTC222Healthy
2026-11-12 21:40:00 UTC111Healthy
2026-11-12 21:45:00 UTC222Healthy
2026-11-12 21:50:00 UTC435Healthy
2026-11-12 21:55:00 UTC222Healthy
2026-11-12 22:00:00 UTC333Healthy
2026-11-12 22:05:00 UTC333Healthy
2026-11-12 22:10:00 UTC111Healthy
2026-11-12 22:15:00 UTC223Healthy
2026-11-12 22:20:00 UTC111Healthy
2026-11-12 22:25:00 UTC111Healthy
2026-11-12 22:30:00 UTC223Healthy
2026-11-12 22:35:00 UTC222Healthy
2026-11-12 22:40:00 UTC323Healthy
2026-11-12 22:45:00 UTC333Healthy
2026-11-12 22:50:00 UTC111Healthy
2026-11-12 22:55:00 UTC222Healthy
2026-11-12 23:00:00 UTC445Healthy
2026-11-12 23:05:00 UTC334Healthy
2026-11-12 23:10:00 UTC334Healthy
2026-11-12 23:15:00 UTC222Healthy
2026-11-12 23:20:00 UTC435Healthy
2026-11-12 23:25:00 UTC445Healthy
2026-11-12 23:30:00 UTC334Healthy
2026-11-12 23:35:00 UTC111Healthy
2026-11-12 23:40:00 UTC222Healthy
2026-11-12 23:45:00 UTC223Healthy
2026-11-12 23:50:00 UTC334Healthy
2026-11-12 23:55:00 UTC435Healthy
2026-11-13 00:00:00 UTC222Healthy
2026-11-13 00:05:00 UTC111Healthy
2026-11-13 00:10:00 UTC111Healthy
2026-11-13 00:15:00 UTC222Healthy
2026-11-13 00:20:00 UTC223Healthy
2026-11-13 00:25:00 UTC111Healthy
2026-11-13 00:30:00 UTC222Healthy
2026-11-13 00:35:00 UTC333Healthy
2026-11-13 00:40:00 UTC324Healthy
2026-11-13 00:45:00 UTC333Healthy
2026-11-13 00:50:00 UTC222Healthy
2026-11-13 00:55:00 UTC334Healthy
2026-11-13 01:00:00 UTC222Healthy
2026-11-13 01:05:00 UTC435Healthy
2026-11-13 01:10:00 UTC223Healthy
2026-11-13 01:15:00 UTC333Healthy
2026-11-13 01:20:00 UTC445Healthy
2026-11-13 01:25:00 UTC111Healthy
2026-11-13 01:30:00 UTC334Healthy
2026-11-13 01:35:00 UTC445Healthy
2026-11-13 01:40:00 UTC334Healthy
2026-11-13 01:45:00 UTC223Healthy
2026-11-13 01:50:00 UTC111Healthy
2026-11-13 01:55:00 UTC334Healthy
2026-11-13 02:00:00 UTC223Healthy
2026-11-13 02:05:00 UTC222Healthy
2026-11-13 02:10:00 UTC333Healthy
2026-11-13 02:15:00 UTC111Healthy
2026-11-13 02:20:00 UTC435Healthy
2026-11-13 02:25:00 UTC111Healthy
Start freeElasticsearch guide

Keep reading

Loki sourceElasticsearch sourceMetricsDatabasesThe agent

Connect one metric. See its record in ten minutes.

Start freeRead the docs