0047606: Missing timeout on Azure Blob Storage operations

Openbravo Issue Tracking System - Modules

View Issue Details

Project

Category

View Status

Date Submitted

Last Update

0047606

Modules

Integrations with Azure Blob Storage

public

2021-08-30 11:31

2021-09-15 14:40

Reporter

sebastien_liron

Assigned To

jarmendariz

Priority

urgent

Severity

major

Reproducibility

random

Status

closed

Resolution

fixed

Platform

OS Version

Product Version

Target Version

2.50

Fixed in Version

Merge Request Status

Regression date

Regression introduced by commit

Regression level

Review Assigned To

Support ticket

OBNetwork customer

Regression introduced in release

Summary

0047606: Missing timeout on Azure Blob Storage operations

Description

In our use of the Azure JDK we are not defining timeouts on any of the upload/download/delete file operations.

If there is any problem when working with that API and the server does not respond, the thread executing that code will get stack, and its database transaction will remain open.

an import entry was blocked by this request for at least 4 hours :

Relevant stack trace:

Import Entry - 5" 0000884 daemon prio=5 os_prio=0 cpu=1105246.52ms elapsed=810940.05s tid=0x00007f27d404b000 nid=0x3720 waiting on condition [0x00007f271498b000]
java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)

parking to wait for <0x000000009a1fab18> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.locksupport.park(java.base@11.0.11/locksupport.java:194)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.parkandcheckinterrupt(java.base@11.0.11/abstractqueuedsynchronizer.java:885)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.doacquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1039)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.acquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1345)
        at java.util.concurrent.countdownlatch.await(java.base@11.0.11/countdownlatch.java:232)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
        at reactor.core.publisher.Mono.block(Mono.java:1494)
        at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:94)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:182)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:153)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:129)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.putObject(AzureStorageClient.java:122)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.upload(AzureStorageClient.java:195)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:111)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:82)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation.uploadFile(AzureStorageAttachImplementation.java:45)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation$Proxy$_$$_WeldClientProxy.uploadFile(Unknown Source)
        at org.openbravo.client.application.attachment.AttachImplementationManager.upload(AttachImplementationManager.java:183)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.attachFileToOrder(ProcessPendingItem.java:118)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.processPendingItem(ProcessPendingItem.java:73)
        at com.openbravo.norauto.order.integration.integration.NOROrdSynchronization.saveRecord(NOROrdSynchronization.java:41)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.saveRecord(DataSynchronizationProcess.java:201)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:156)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:88)
        at org.openbravo.mobile.core.process.MobileImportEntryProcessorRunnable.processEntry(MobileImportEntryProcessorRunnable.java:54)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.doRunCycle(ImportEntryProcessor.java:373)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.run(ImportEntryProcessor.java:294)
        at java.util.concurrent.executors$runnableadapter.call(java.base@11.0.11/executors.java:515)
        at java.util.concurrent.futuretask.run(java.base@11.0.11/futuretask.java:264)
        at java.util.concurrent.threadpoolexecutor.runworker(java.base@11.0.11/threadpoolexecutor.java:1128)
        at java.util.concurrent.threadpoolexecutor$worker.run(java.base@11.0.11/threadpoolexecutor.java:628)
        at java.lang.thread.run(java.base@11.0.11/thread.java:829)

Steps To Reproduce

There are still no steps to reproduce, the problem with the Azure server seems to be random so far.

But looking at the APIs that are being used on the AzureStorageClient class, it is clear that no timeouts are being defined.

Proposed Solution

there are 2 options to check:
- Close the connection before start the upload
- Add a timeout so if the upload takes long (5 hours is a lot) interrupt the process

Additional Information