0047606: Missing timeout on Azure Blob Storage operations

Description

In our use of the Azure JDK we are not defining timeouts on any of the upload/download/delete file operations.

If there is any problem when working with that API and the server does not respond, the thread executing that code will get stack, and its database transaction will remain open.

an import entry was blocked by this request for at least 4 hours :

Relevant stack trace:

Import Entry - 5" 0000884 daemon prio=5 os_prio=0 cpu=1105246.52ms elapsed=810940.05s tid=0x00007f27d404b000 nid=0x3720 waiting on condition [0x00007f271498b000]
java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)

parking to wait for <0x000000009a1fab18> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.locksupport.park(java.base@11.0.11/locksupport.java:194)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.parkandcheckinterrupt(java.base@11.0.11/abstractqueuedsynchronizer.java:885)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.doacquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1039)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.acquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1345)
        at java.util.concurrent.countdownlatch.await(java.base@11.0.11/countdownlatch.java:232)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
        at reactor.core.publisher.Mono.block(Mono.java:1494)
        at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:94)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:182)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:153)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:129)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.putObject(AzureStorageClient.java:122)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.upload(AzureStorageClient.java:195)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:111)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:82)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation.uploadFile(AzureStorageAttachImplementation.java:45)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation$Proxy$_$$_WeldClientProxy.uploadFile(Unknown Source)
        at org.openbravo.client.application.attachment.AttachImplementationManager.upload(AttachImplementationManager.java:183)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.attachFileToOrder(ProcessPendingItem.java:118)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.processPendingItem(ProcessPendingItem.java:73)
        at com.openbravo.norauto.order.integration.integration.NOROrdSynchronization.saveRecord(NOROrdSynchronization.java:41)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.saveRecord(DataSynchronizationProcess.java:201)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:156)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:88)
        at org.openbravo.mobile.core.process.MobileImportEntryProcessorRunnable.processEntry(MobileImportEntryProcessorRunnable.java:54)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.doRunCycle(ImportEntryProcessor.java:373)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.run(ImportEntryProcessor.java:294)
        at java.util.concurrent.executors$runnableadapter.call(java.base@11.0.11/executors.java:515)
        at java.util.concurrent.futuretask.run(java.base@11.0.11/futuretask.java:264)
        at java.util.concurrent.threadpoolexecutor.runworker(java.base@11.0.11/threadpoolexecutor.java:1128)
        at java.util.concurrent.threadpoolexecutor$worker.run(java.base@11.0.11/threadpoolexecutor.java:628)
        at java.lang.thread.run(java.base@11.0.11/thread.java:829)

Steps To Reproduce

There are still no steps to reproduce, the problem with the Azure server seems to be random so far.

But looking at the APIs that are being used on the AzureStorageClient class, it is clear that no timeouts are being defined.

Proposed Solution

there are 2 options to check:
- Close the connection before start the upload
- Add a timeout so if the upload takes long (5 hours is a lot) interrupt the process

Tags

NOR

Attached Files

Relationships [ Relation Graph ] [ Dependency Graph ]

related to	defect	0047649		closed	jarmendariz	cannot overwrite attachemnt in Azure
related to	defect	0047650		closed	jarmendariz	missing timeout for Amazon S3 upload

Notes
(0131574) hgbot (developer) 2021-09-06 12:18	Merge Request created: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/merge_requests/6 [^]

(0131641) hgbot (developer) 2021-09-08 09:17	Directly closing issue as related merge request is already approved. Repository: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage [^] Changeset: 3b48046e6f792366bc082d19cfa905139539a690 Author: Javier Armendáriz <javier.armendariz@openbravo.com> Date: 2021-09-08T09:13:32+02:00 URL: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/commit/3b48046e6f792366bc082d19cfa905139539a690 [^] Fixed ISSUE-47606: Setting a timeout value for all azure operations Adding a timeout field to azure configuration tab with a default value of 60 seconds. This timeout will be used by all azure operations. This also adds a timeout exception used to display a descriptive message to the user when the operation times out. --- A src/org/openbravo/service/integration/azure/storage/exception/OperationTimeoutException.java M src-db/database/model/tables/OBAZS_CONFIGURATION.xml M src-db/database/sourcedata/AD_COLUMN.xml M src-db/database/sourcedata/AD_ELEMENT.xml M src-db/database/sourcedata/AD_FIELD.xml M src-db/database/sourcedata/AD_MESSAGE.xml M src/org/openbravo/service/integration/azure/storage/AzureStorageClient.java M src/org/openbravo/service/integration/azure/storage/attachments/AzureStorageAttachImplementation.java M src/org/openbravo/service/integration/azure/storage/process/AzureStorageValidationClient.java ---

(0131642) hgbot (developer) 2021-09-08 09:17	Merge request merged: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/merge_requests/6 [^]

Issue History
Date Modified	Username	Field	Change
2021-08-30 11:31	sebastien_liron	New Issue
2021-08-30 11:31	sebastien_liron	Assigned To	=> AugustoMauch
2021-08-30 11:31	sebastien_liron	OBNetwork customer	=> OBPS
2021-09-01 16:15	AugustoMauch	version	2.50 =>
2021-09-01 16:15	AugustoMauch	Summary	lock on import entry using azure attachments => Missing timeout on Azule Blob Storage operations
2021-09-01 16:15	AugustoMauch	Description Updated	View Revisions
2021-09-01 16:15	AugustoMauch	Steps to Reproduce Updated	View Revisions
2021-09-06 09:17	AugustoMauch	Assigned To	AugustoMauch => jarmendariz
2021-09-06 09:18	AugustoMauch	Status	new => scheduled
2021-09-06 12:18	hgbot	Merge Request Status	=> open
2021-09-06 12:18	hgbot	Note Added: 0131574
2021-09-08 08:35	alostale	Summary	Missing timeout on Azule Blob Storage operations => Missing timeout on Azure Blob Storage operations
2021-09-08 08:59	alostale	Relationship added	related to 0047649
2021-09-08 09:01	alostale	Relationship added	related to 0047650
2021-09-08 09:02	hgbot	Merge Request Status	open => approved
2021-09-08 09:17	hgbot	Resolution	open => fixed
2021-09-08 09:17	hgbot	Status	scheduled => closed
2021-09-08 09:17	hgbot	Note Added: 0131641
2021-09-08 09:17	hgbot	Note Added: 0131642
2021-09-15 14:40	rafaroda	Tag Attached: NOR