Sporadic Email Failures on EU Production Instance
Description
During the two windows below, email alerts triggered by actions in Viedoc were sent intermittently. Most were delivered as expected. A subset were delayed, and some were not delivered at all.
Email alerts in Viedoc are not sent by the application at the moment a user saves data. They are produced by background processing that runs immediately afterwards and decides, based on your study's configuration, which alerts need to be sent. During these two windows that background processing did not always complete, so the alerts it would have produced were delayed or never generated.
Data entered and saved by users during these periods was stored as expected. The interruption affected work handled by particular processing servers rather than the platform as a whole, which is why alerts were intermittent rather than stopping entirely.
Cause
Viedoc's background processing runs across several servers working in parallel. Each server needs a working connection to a shared internal service that Viedoc uses to coordinate this work safely - for example, to ensure two servers never process the same subject at the same moment. During these windows, individual servers lost that connection and did not re-establish it. An affected server continued to accept work but could not complete it. Other servers running at the same time were unaffected, which is why the impact was intermittent and why the platform continued to appear healthy overall.
Corrective action
Preventative action
Increased monitoring
We have increased the monitoring and added alerting that detects this specific failure on any individual server much faster, preventing this to fail silently.
Underlying cause - ongoing investigation
The investigation into why these connections failed remains open, and we are treating it as separate from the mitigations above.