Monitoring: checks now run from several vantage points

Monitoring now probes your services from a cluster of probers running on separate nodes. Until now there was exactly one prober – and it cannot tell whether your service is really gone or whether it just cannot reach it at the moment.

What this changes

• A failure only counts as an outage of your service once every reporting prober confirms it.
• If even one prober reaches your target, the matter is settled: the failure is recorded as a problem on my side and excluded from your availability. No incident, no alert.
• A prober whose own reference check is failing does not get a vote. One broken node therefore cannot talk the whole cluster into an outage.
• If nobody can vote, everything stays as before: the failure counts against the check's own failure threshold, exactly as it used to. Monitoring never falls silent.
• In the console you can see, per check, which prober measured what and where they disagreed.

Empty DNS answers are no longer an outage

• When a resolver answered NOERROR but held no record of the queried type, that used to be treated as “the name does not exist”. An MX check on a zone without mail, or a CNAME check on a zone apex, therefore sat at red permanently while every other DNS tool disagreed.
• An empty answer is a property of the zone, not a fault of the target. It now counts as a successful query.
• If you want exactly that to alert, say so with an assertion on the answer – those are still evaluated on top.

Domains are re-verified periodically

• A domain's proof of ownership used to run exactly once. It is now repeated every 24 hours, and within minutes after a failed attempt.
• If the proof keeps failing for more than 48 hours, the domain is suspended and its checks are paused. As soon as verification succeeds again, everything resumes automatically.
• Domains whose ownership is already established through another module (DynDNS hostnames, for example) are excluded from this.

A note on data retention

• The finest tiers of the time series have become shorter: individual probes 1 day, 5-minute values 7 days. Hourly values (1 year) and daily values (5 years) remain unchanged.
• Analyses over longer ranges are unaffected. Only the very fine-grained look back does not reach as far as announced in the update of 11 August.

If something is missing or looks off, do send me a note.