Viedoc service issue report

Report author

Ted Lähteenmäki, Manager IT Operations
Viedoc Technologies AB
Phone: +46 736 92 26 12
Email: ted.lahteenmaki@viedoc.com

Report issued

2021-03-05

Description

Degraded performance on Viedoc 4 Japan instance.

Instances affected

Viedoc 4 - Japan (Background processes (import/export/revisions/alerts/archive/disposal)).

Impact

1. Exports (Clinic)
2. PrintForms (Designer)
3. PrintWorkflow (Designer)
4. PDF creation (forms + study events)
For PDF’s not been created during the incident the client needs to perform a PDF export.

Duration

2021-02-26 03:27 UTC - 2021-02-26 06:58 UTC

Cause

An incident at our service provider Microsoft caused failures for Viedoc services attempting to utilize storage resources in Microsoft Azure.

Microsoft has published a preliminary RCA but in the light of new information their investigation continues.
Please find the latest information at: https://status.azure.com/en-us/status/history

Corrective action

Microsoft recovered unhealthy nodes which restored resources to the service.

Preventive action

Microsoft will improve detection and alerting for their load balancing and reduce the maximum size utilization levels to ensure increased resource headroom in case of multiple unexpected events.

Posted Mar 05, 2021 - 14:01 UTC

Resolved

Incident resolved.

Microsoft: "Preliminary Root Cause: During the window of impact, an increase in utilisation was observed and this combined with the sudden loss of backend instance meant an operational threshold was reached. This in turn caused failures for customers and Azure services attempting to utilise Storage resources."

Microsoft Tracking ID: PLWV-BT0

Posted Feb 26, 2021 - 10:23 UTC

Update

Microsoft has identified the underlying cause and are actively pursuing multiple mitigation workstreams and recovery is currently in progress.

We continue to monitor the Viedoc services, which are all as of now Operational.

Posted Feb 26, 2021 - 07:28 UTC

Monitoring

The issue has been resolved, monitoring and waiting on confirmation from Microsoft.

Posted Feb 26, 2021 - 07:06 UTC

Identified

An issue with Azure storage accessibility in Japan East has been identified at our service provider, Microsoft Azure. Microsoft is currently investigating the issue. https://status.azure.com/en-us/status

The issue impact Viedoc functionality such as export, file upload and PDF generation.

Updated 05:57 UTC

Microsoft has identified a possible root cause and are actively determining mitigation steps.

Updated 06:19 UTC

Posted Feb 26, 2021 - 05:25 UTC

Investigating

We are currently investigating this issue.

Posted Feb 26, 2021 - 04:01 UTC

This incident affected: Viedoc 4 - Japan (Background processes (import/export/revisions/alerts/archive/disposal)).