Description
Between the late afternoon of 2024-08-20 and noon 2024-08-21 some of the documents (blank CRF, annotated CRF, Workflow) could not be generated in Viedoc Designer. Users who attempted to generate these documents were shown the document generation being stuck at 0%. Once the worker was re-started and corrupt print jobs were removed, the worker was functioning as expected again.
Instances affected
https://v4.viedoc.net/; https://v4training.viedoc.net
Duration
Between the late afternoon of 2024-08-20 and noon 2024-08-21.
Cause
As preparation for some technical infrastructure upgrades, the designer worker handling print jobs was re-started in the afternoon of 2024-08-20. When re-started, a print job that was ongoing when the worker was re-started ended up in a corrupt state, not being possible to process by the worker. This blocked the queue and no other print jobs were executed in Designer. Around 9AM on 2024-08-21 the first report came in from a customer that the print jobs were not being processed.
Impact
Between re-starting the worker and resetting the status of the corrupt print job, no print jobs handled by this worker were processed. Once the status of the corrupt print job was reset, the queue was again being processed as expected. Thus, the impact for the customers was a delay in getting these specific print jobs completed.
Corrective action
When the print job queue was reviewed, the invalid print job blocking the queue was detected. This status of this job was then reset so that the worker would start over. After this action the print job queue was processed as normal again and print jobs that had previously not been completed were executed.
Preventive action
Since this issue was triggered by the manual intervention or re-starting the worker, the initial preventive action is just to manually monitor to see that print jobs start executing as expected in case the worker needs to be re-started again. For the future, improved automatic monitoring and re-try attempts will be evaluated.