Loading
Lab 6 of 6
Inject 800 ms of latency, watch the p95 alert, then clear it.
The rule HighP95Latency fires when p95 of /search is above 0.5 s for 1 minute. The wait stops one slow request from paging anyone. A second rule fires when 5xx responses exceed 1%.
You inject an 800 ms delay. What happens within about two minutes?
800 ms is over the 500 ms SLO, held for the 1 minute window, so the page fires. Clear the fault and it resolves.
make up
make fault-slow
# watch Alertmanager http://localhost:9093 and Grafana http://localhost:3300
make fault-clearDefined in the Makefile and monitoring/alerts.yml. Not verified on this page: I have not recorded a live firing here.
Run make fault-500 instead and note which rule fires. Change the 1m window to 5m and say what you trade.