Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue DetailsJump to Notes ] Issue History ] Print ]
ID
0048427
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] A. Platformmajorhave not tried2022-01-19 13:042022-11-28 23:54
ReporteralostaleView Statuspublic 
Assigned Toalostale 
PrioritynormalResolutionfixedFixed in VersionPR22Q3
StatusclosedFix in branchFixed in SCM revision
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned To
Web browser
ModulesCore
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0048427: high load of low priority import entries can cause higher priority entries not to be processed

DescriptionHaving the following import entry types:

1. OBUIELS_Data1
  It has a higher priority (1)

2. OBUIELS_Data2
  It has a lower priority (2)
  It takes long to process each entry
  Its entries span their keys to consume all the threads
  It has a high inflow of entries to process

The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time.

This issue is caused because:

1. 1st Import Entry manger loop starts:
1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop

2. After wait time elapsed, 2nd loop starts:
2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done.
2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processor runnables that were created in the previous cycle and are still processing. Because of this, these runnables, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done.
Steps To ReproduceThis scenario can be simulated with Entry Load Simulator [1] and the following config:

** Import Entry config
  import.batch.size                     5000
  import.number.of.threads              11
  import.max.task.queue.size            1000
  import.wait.time                      600
  import.processing.capacity.per.second 330
** OBUIELS_Data1
  Num of Entries 10
  Period         200
  Cycles         50
  Delay          50
  Exec Time      50
  # of Keys      50
** OBUIELS_Data2
  Num of Entries 200
  Period         100
  Cycles         15
  Delay          0
  Exec Time      750
  # of Keys      10


[1] https://gitlab.com/openbravo/tools/platform/org.openbravo.util.ieloadsim [^]
TagsNOR
Attached Filespng file icon 48427- not-fixed.png [^] (18,079 bytes) 2022-01-19 13:05

- Relationships Relation Graph ] Dependency Graph ]
related to defect 0048418 newTriage Platform Conn Openbravo ERP import entries can be processed twice if their key is not deterministic 
related to defect 0045145 closedranjith_qualiantech_com Retail Modules Priority not defined for Session Import types 
related to defect 0049963 closedTriage Platform Base Openbravo ERP When a ImportEntry is not assigned to a runnable it should not be included in hte entryCount 
related to defect 0047219 closedcaristu Modules Use different runnables for EDL processes that should be executed in a specific thread 

-  Notes
(0134406)
hgbot (developer)
2022-01-20 09:02

Merge Request created: https://gitlab.com/openbravo/product/openbravo/-/merge_requests/506 [^]
(0136453)
hgbot (developer)
2022-04-11 09:29

Directly closing issue as related merge request is already approved.

Repository: https://gitlab.com/openbravo/product/openbravo [^]
Changeset: f5b8202f8017ca4403c9965f8671e87e54535d69
Author: Asier Lostalé <asier.lostale@openbravo.com>
Date: 11-04-2022 07:01:11
URL: https://gitlab.com/openbravo/product/openbravo/-/commit/f5b8202f8017ca4403c9965f8671e87e54535d69 [^]

fixed BUG-48427: low prio entries can get precedence over high prio

Having scheduled in a given ImportEntryProcessRunnable a set of import
entries (all of them sharing same type of data and the same key). If the
processing of those entries does not finalize when the next ImportEntry
cycle looks for new entries to process and there are entries for the
same type of data and key, they will be scheduled to be executed in the
already running ImportEntryProcessRunnable. This makes these newly
scheduled entries to take precedence over new entries of other types of
data with a supposedly higher priority that require a new ImportEntryProcessRunnable.

Now ImportEntryProcessRunnable instances are linked to the
ImportEntryManager's execution cycle, so new entries for the same type
and key are not scheduled if there are still entries with the same key
and type scheduled from a previous cycle.

---
M src/org/openbravo/service/importprocess/ImportEntryManager.java
M src/org/openbravo/service/importprocess/ImportEntryProcessor.java
---
(0136454)
hgbot (developer)
2022-04-11 09:29

Merge request merged: https://gitlab.com/openbravo/product/openbravo/-/merge_requests/506 [^]

- Issue History
Date Modified Username Field Change
2022-01-19 13:04 alostale New Issue
2022-01-19 13:04 alostale Assigned To => Triage Platform Conn
2022-01-19 13:04 alostale Modules => Core
2022-01-19 13:04 alostale Triggers an Emergency Pack => No
2022-01-19 13:05 alostale File Added: 48427- not-fixed.png
2022-01-19 13:06 alostale Relationship added related to 0048418
2022-01-19 13:20 alostale Description Updated View Revisions
2022-01-20 08:59 alostale Description Updated View Revisions
2022-01-20 09:02 hgbot Note Added: 0134406
2022-01-20 11:08 alostale Assigned To Triage Platform Conn => alostale
2022-02-01 07:24 alostale Assigned To alostale => Triage Platform Base
2022-03-22 15:24 rafaroda Tag Attached: NOR
2022-03-22 15:24 rafaroda Issue Monitored: rafaroda
2022-03-22 15:25 rafaroda Relationship added related to 0045145
2022-03-28 16:22 AugustoMauch Assigned To Triage Platform Base => alostale
2022-03-28 16:22 AugustoMauch Status new => acknowledged
2022-04-04 14:17 rafaroda Relationship added related to 0047219
2022-04-11 09:29 hgbot Resolution open => fixed
2022-04-11 09:29 hgbot Status acknowledged => closed
2022-04-11 09:29 hgbot Fixed in Version => PR22Q3
2022-04-11 09:29 hgbot Note Added: 0136453
2022-04-11 09:29 hgbot Note Added: 0136454
2022-08-12 10:36 alostale Relationship added related to 0049963
2022-11-28 23:54 eugeni Issue Monitored: eugeni


Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker