Postmortem -
Read details
Apr 2, 07:18 UTC
Resolved -
From 11:50 PT->12:04am PT a significant percentage of metadata plane traffic returned 500 response codes with db login failures; this impacted:
- offline query triggers
- offline query status polling
- scheduled resolver execution
- very rarely, auth token exchange
- UI access
Root cause was an error in an automated password rotation process which dropped an 'old' password too early.
Apr 2, 07:17 UTC