Viedoc service issue report

Report author

Claes Aspman
Manager Quality Assurance
Viedoc Technologies AB

Report issued

2022-07-01

Description

Occasional slow performance and rare timeouts resulting in an application error on public endpoints; Viedoc Clinic, Viedoc Me, and Viedoc API.

Background services suffered occasional disturbance resulting in some exports and imports being stuck in queue and some pdfs not being generated.

Instances affected

Viedoc version 4 - Japan instances

Duration

JST: 2022-06-28 21:15 -- 2022-06-30 23:14
UTC: 2022-06-28 12:15 -- 2022-06-30 14:04

Cause

The underlying network infrastructure of the Japanese instances experienced intermittent network failures and packet loss affecting a subset of the Viedoc 4 application servers.

Impact

Availability only. Data integrity and data confidentiality not affected.

Corrective action

The underlying network disturbance was mitigated by the infrastructure provider.

Data exports stuck in queue were reinitiated.

Data import affected only one customer, that customer has been informed.

Missing pdfs will be re-generated when data is exported.

Preventive action

New non-functional Viedoc requirements will be implemented to improve how Viedoc manage this type of network disturbances.

Posted Jul 01, 2022 - 10:23 UTC

Resolved

We have seen no more issues since performance was stabilized.
An incident report will be posted in due time.

Posted Jul 01, 2022 - 05:45 UTC

Monitoring

Performance have stabilized, but we continue to monitor.

Posted Jun 30, 2022 - 16:04 UTC

Update

The underlying infrastructure provider for Viedoc is currently deploying a hotfix to mitigate the issue.
Another update will be posted when we have more information.

Posted Jun 30, 2022 - 13:59 UTC

Identified

Identified issues impacting performance in other parts of Viedoc.
The underlying infrastructure provider for Viedoc has an ongoing incident causing intermittent network failures and packet loss.

Posted Jun 30, 2022 - 12:56 UTC

Investigating

There are currently occasional issues with data exports, user log exports and pdf creation on save. We are investigating these issues.

Posted Jun 30, 2022 - 09:29 UTC

This incident affected: Viedoc 4 - Japan (Main portal (clinic/admin/designer), Subject portal (ViedocMe), Application programming interface (API), Background processes (import/export/revisions/alerts/archive/disposal)).