Occasional delay in accessing user details page on EU instance

Incident Report for Viedoc

Postmortem

Background:

As part of the Viedoc 4.78 release on 2024-03-16, all user data was moved to a new dedicated database. This transition uncovered a slowdown issue due to a missing index, particularly affecting user detail page load times in Viedoc Admin. Detected on 2024-03-18, this was caused by an overlooked index in the user login history table. This scenario was not anticipated during testing due to its occurrence under specific high-load conditions, as observed in the production instances.

Resolution:

To address this, we added the necessary index to the login history table at 16.51 PM CET on 2024-03-18, promptly restoring normal response times and preventing future recurrences.

Impact:

From 09.00 AM to 16.51 PM CET on 2024-03-18, users experienced delays and, in some cases, timeouts when accessing user details pages.

We apologize for any inconvenience this may have caused.

Posted Mar 20, 2024 - 07:56 UTC

Resolved

During the day 2024-03-18 there have been occasional delays accessing the details page of other users from Viedoc Admin. This led to slower response times and in some cases requests timed out so that an error message was displayed on the screen. This issue was solved 16:51 CET. Performance is back up and the issue is resolved now. Users should be able to access user details page as expected.

Posted Mar 18, 2024 - 16:03 UTC

This incident affected: Viedoc 4 - Europe (Main portal (clinic/admin/designer)).