Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0047606
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Modules] Integrations with Azure Blob Storagemajorrandom2021-08-30 11:312021-09-15 14:40
Reportersebastien_lironView Statuspublic 
Assigned Tojarmendariz 
PriorityurgentResolutionfixedFixed in Version
StatusclosedFix in branchFixed in SCM revision
ProjectionnoneETAnoneTarget Version2.50
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Regression date
Regression introduced by commit
Regression level
Review Assigned To
Regression introduced in release
Summary

0047606: Missing timeout on Azure Blob Storage operations

DescriptionIn our use of the Azure JDK we are not defining timeouts on any of the upload/download/delete file operations.

If there is any problem when working with that API and the server does not respond, the thread executing that code will get stack, and its database transaction will remain open.

an import entry was blocked by this request for at least 4 hours :

Relevant stack trace:

Import Entry - 5" 0000884 daemon prio=5 os_prio=0 cpu=1105246.52ms elapsed=810940.05s tid=0x00007f27d404b000 nid=0x3720 waiting on condition [0x00007f271498b000]
java.lang.Thread.State: WAITING (parking)

at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)

parking to wait for <0x000000009a1fab18> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.locksupport.park(java.base@11.0.11/locksupport.java:194)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.parkandcheckinterrupt(java.base@11.0.11/abstractqueuedsynchronizer.java:885)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.doacquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1039)
        at java.util.concurrent.locks.abstractqueuedsynchronizer.acquiresharedinterruptibly(java.base@11.0.11/abstractqueuedsynchronizer.java:1345)
        at java.util.concurrent.countdownlatch.await(java.base@11.0.11/countdownlatch.java:232)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
        at reactor.core.publisher.Mono.block(Mono.java:1494)
        at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:94)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:182)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:153)
        at com.azure.storage.blob.BlobClient.uploadFromFile(BlobClient.java:129)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.putObject(AzureStorageClient.java:122)
        at org.openbravo.service.integration.azure.storage.AzureStorageClient.upload(AzureStorageClient.java:195)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:111)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachmentClient.upload(AzureStorageAttachmentClient.java:82)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation.uploadFile(AzureStorageAttachImplementation.java:45)
        at org.openbravo.service.integration.azure.storage.attachments.AzureStorageAttachImplementation$Proxy$_$$_WeldClientProxy.uploadFile(Unknown Source)
        at org.openbravo.client.application.attachment.AttachImplementationManager.upload(AttachImplementationManager.java:183)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.attachFileToOrder(ProcessPendingItem.java:118)
        at com.openbravo.norauto.order.integration.processes.ProcessPendingItem.processPendingItem(ProcessPendingItem.java:73)
        at com.openbravo.norauto.order.integration.integration.NOROrdSynchronization.saveRecord(NOROrdSynchronization.java:41)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.saveRecord(DataSynchronizationProcess.java:201)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:156)
        at org.openbravo.mobile.core.process.DataSynchronizationProcess.exec(DataSynchronizationProcess.java:88)
        at org.openbravo.mobile.core.process.MobileImportEntryProcessorRunnable.processEntry(MobileImportEntryProcessorRunnable.java:54)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.doRunCycle(ImportEntryProcessor.java:373)
        at org.openbravo.service.importprocess.ImportEntryProcessor$ImportEntryProcessRunnable.run(ImportEntryProcessor.java:294)
        at java.util.concurrent.executors$runnableadapter.call(java.base@11.0.11/executors.java:515)
        at java.util.concurrent.futuretask.run(java.base@11.0.11/futuretask.java:264)
        at java.util.concurrent.threadpoolexecutor.runworker(java.base@11.0.11/threadpoolexecutor.java:1128)
        at java.util.concurrent.threadpoolexecutor$worker.run(java.base@11.0.11/threadpoolexecutor.java:628)
        at java.lang.thread.run(java.base@11.0.11/thread.java:829)
Steps To ReproduceThere are still no steps to reproduce, the problem with the Azure server seems to be random so far.

But looking at the APIs that are being used on the AzureStorageClient class, it is clear that no timeouts are being defined.
Proposed Solutionthere are 2 options to check:
- Close the connection before start the upload
- Add a timeout so if the upload takes long (5 hours is a lot) interrupt the process
TagsNOR
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]
related to defect 0047649 closedjarmendariz cannot overwrite attachemnt in Azure 
related to defect 0047650 closedjarmendariz missing timeout for Amazon S3 upload 

-  Notes
(0131574)
hgbot (developer)
2021-09-06 12:18

 Merge Request created: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/merge_requests/6 [^]
(0131641)
hgbot (developer)
2021-09-08 09:17

 Directly closing issue as related merge request is already approved.

Repository: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage [^]
Changeset: 3b48046e6f792366bc082d19cfa905139539a690
Author: Javier Armendáriz <javier.armendariz@openbravo.com>
Date: 2021-09-08T09:13:32+02:00
URL: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/commit/3b48046e6f792366bc082d19cfa905139539a690 [^]

Fixed ISSUE-47606: Setting a timeout value for all azure operations

Adding a timeout field to azure configuration tab with a default value of 60 seconds. This timeout will
be used by all azure operations. This also adds a timeout exception used to display a descriptive message
to the user when the operation times out.

---
A src/org/openbravo/service/integration/azure/storage/exception/OperationTimeoutException.java
M src-db/database/model/tables/OBAZS_CONFIGURATION.xml
M src-db/database/sourcedata/AD_COLUMN.xml
M src-db/database/sourcedata/AD_ELEMENT.xml
M src-db/database/sourcedata/AD_FIELD.xml
M src-db/database/sourcedata/AD_MESSAGE.xml
M src/org/openbravo/service/integration/azure/storage/AzureStorageClient.java
M src/org/openbravo/service/integration/azure/storage/attachments/AzureStorageAttachImplementation.java
M src/org/openbravo/service/integration/azure/storage/process/AzureStorageValidationClient.java
---
(0131642)
hgbot (developer)
2021-09-08 09:17

 Merge request merged: https://gitlab.com/openbravo/product/pmods/org.openbravo.service.integration.azure.storage/-/merge_requests/6 [^]

- Issue History
Date Modified Username Field Change
2021-08-30 11:31 sebastien_liron New Issue
2021-08-30 11:31 sebastien_liron Assigned To => AugustoMauch
2021-09-01 16:15 AugustoMauch version 2.50 =>
2021-09-01 16:15 AugustoMauch Summary lock on import entry using azure attachments => Missing timeout on Azule Blob Storage operations
2021-09-01 16:15 AugustoMauch Description Updated View Revisions
2021-09-01 16:15 AugustoMauch Steps to Reproduce Updated View Revisions
2021-09-06 09:17 AugustoMauch Assigned To AugustoMauch => jarmendariz
2021-09-06 09:18 AugustoMauch Status new => scheduled
2021-09-06 12:18 hgbot Note Added: 0131574
2021-09-08 08:35 alostale Summary Missing timeout on Azule Blob Storage operations => Missing timeout on Azure Blob Storage operations
2021-09-08 08:59 alostale Relationship added related to 0047649
2021-09-08 09:01 alostale Relationship added related to 0047650
2021-09-08 09:17 hgbot Resolution open => fixed
2021-09-08 09:17 hgbot Status scheduled => closed
2021-09-08 09:17 hgbot Note Added: 0131641
2021-09-08 09:17 hgbot Note Added: 0131642
2021-09-15 14:40 rafaroda Tag Attached: NOR

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker