0048427: high load of low priority import entries can cause higher priority entries not to be processed

Anonymous | Login

Project:

News | My View | View Issues | Roadmap | Summary

View Revisions: Issue #48427		[ Back to Issue ]
Summary	0048427: high load of low priority import entries can cause higher priority entries not to be processed

Revision	2022-01-20 08:59 by alostale
Description	Having the following import entry types: 1. OBUIELS_Data1 It has a higher priority (1) 2. OBUIELS_Data2 It has a lower priority (2) It takes long to process each entry Its entries span their keys to consume all the threads It has a high inflow of entries to process The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time. This issue is caused because: 1. 1st Import Entry manger loop starts: 1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop 2. After wait time elapsed, 2nd loop starts: 2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done. 2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processor runnables that were created in the previous cycle and are still processing. Because of this, these runnables, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done.

Revision	2022-01-19 13:20 by alostale
Description	Having the following import entry types: 1. OBUIELS_Data1 It has a higher priority (1) 2. OBUIELS_Data2 It has a lower priority (2) It takes long to process each entry Its entries span their keys to consume all the threads It has a high inflow of entries to process The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time. This issue is caused because: 1. 1st Import Entry manger loop starts: 1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop 2. After wait time elapsed, 2nd loop starts: 2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done. 2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processors that were created in the previous cycle and are still processing. Because of this, these processors, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done.

Revision	2022-01-19 13:06 by alostale
Description	Having the following import entry types: 1. OBUIELS_Data1 It has a higher priority (1) 2. OBUIELS_Data2 It has a lower priority (2) It takes long to process each entry Its entries span their keys to consume all the threads It has a high inflow of entries to process The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time.