Watch Owl alert rules

Watch Owl alert rules fire notifications when a host metric crosses a threshold and holds it. Rules are evaluated server-side against the metrics your agents push, so there is no per-host configuration — define a rule once, target it at all hosts or a subset, and route the firings to your integrations.

Find them at System Monitor → Alerts. Evaluation runs every 60 seconds.

How rules work

Each rule answers four questions:

1. What metric is being watched (CPU, memory, disk, network, patches, etc).
2. What threshold triggers the rule (operator + value, like > 90).
3. How long the threshold must hold before the rule fires (the duration).
4. Which hosts the rule applies to (all, by tag, or a single host).

When the condition holds for duration_seconds straight, the rule fires — an event row is created in the timeline and the configured integrations dispatch a notification. When the metric drops back below the threshold, the event resolves automatically and a "resolved" notification goes out.

After a fire, the rule enters a cooldown (default: 1 hour) before it can fire again for the same host. This prevents flapping alerts from spamming the on-call.

Available metrics

disk_percent requires a mount filter (e.g. /, /var, /data) — without one, a single noisy mount on one host could fire the rule for the wrong reason. Network metrics similarly take an interface filter (e.g. eth0, ens5).

Operators

gt, gte, lt, lte, eq, ne — applied to the numeric or boolean threshold you set. For boolean metrics like reboot_required, use eq true.

Scope: which hosts the rule covers

Tag-based scoping is the most flexible — tag your hosts on enrollment and you can add or remove machines from a rule's scope just by editing the host, without touching the rule.

Duration and cooldown

• Duration is the "must hold for" window before firing. Set it to 0 to fire immediately on the first sample that crosses the threshold; set it to 300 (5 minutes) to require sustained breach. For noisy metrics like CPU, longer durations cut false positives sharply.
• Cooldown is the post-fire silence period. After a rule fires for a host, it won't fire again for that host until the cooldown expires (default: 3600 seconds / 1 hour). The rule still evaluates and the event still auto-resolves — the cooldown only suppresses a new fire.

Routing alerts to integrations

Each rule has an integration list — pick one or more configured integrations from your Integrations page. When the rule fires (or resolves), every selected integration receives the event. See the Integrations overview for the full set: Email, Discord, Slack, Microsoft Teams, ntfy, and custom webhooks.

A rule with no integrations selected still records events in the timeline — useful if you only want a UI-visible audit trail without a notification.

Event timeline

System Monitor → Alerts → Events shows every fire and resolve, scoped to your organization. Each event records:

• Which rule fired and which host triggered it.
• The exact metric value at the moment of fire (e.g. CPU 94.2%).
• The value at resolution.
• Per-integration dispatch status (sent, failed, skipped) so you can confirm the notification actually went out.

Events are kept indefinitely for post-incident review. If a host is removed from your organization, its historical events are retained — the host UUID is stored on the event, not a foreign key.

Examples

Fire if any production database host is over 90% disk for 10 minutes:

Name:     prod-db disk near full
Metric:   disk_percent
Operator: gt
Threshold: 90
Mount filter: /var/lib/postgresql
Duration: 600
Scope:    Tagged → env=prod, role=db
Cooldown: 3600

Fire immediately if any host requires a reboot for security:

Name:     Reboot pending
Metric:   reboot_required
Operator: eq
Threshold: true
Duration: 0
Scope:    All hosts
Cooldown: 86400  // once per day per host

Fire if a host hasn't checked in for 5 minutes:

Name:     Host offline
Metric:   host_offline_seconds
Operator: gt
Threshold: 300
Duration: 0
Scope:    All hosts

Best practices

• Start broad, narrow with tags. Begin with a single "all hosts" rule per metric, then split into env- or role-tagged variants once you know which thresholds make sense where.
• Always set a duration on noisy metrics. CPU and memory spike briefly during normal workloads. Requiring the threshold to hold for five minutes is what separates a real problem from a build job.
• Use cooldown to throttle, not silence. If you're tempted to set a 24-hour cooldown to stop spam, that's usually a sign the rule's threshold or duration is wrong — fix those first.
• Pair host_offline_seconds with pending_updates. Together they cover both "host is down" and "host is up but neglected" — the two failure modes of a Linux server fleet.

Limitations

• Watch Owl metrics only. Alert rules cannot be defined against uptime monitors, SSL expiry or domain expiry. There is no per-monitor alerting elsewhere in the product either — a failing check dispatches nothing today. Uptime alerting is the top item on the roadmap.
• No escalation policies and no on-call rotations. Every integration a rule routes to is notified at the same moment. If you need escalation, route to a webhook and let your own tooling handle it.
• No SMS and no phone calls. Not built.
• No maintenance windows on rules. To silence a rule, disable it from the rules list. A scheduled maintenance window on a status page does not mute alert rules — the two are unconnected.

See also

• Watch Owl overview — what the agent collects.
• Notifications overview — the six channels and what reaches them.
• Automatic incidents — what a failing uptime check does instead.