Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Revisions: Issue #48427 Back to Issue ]
Summary 0048427: high load of low priority import entries can cause higher priority entries not to be processed
Revision 2022-01-20 08:59 by alostale
Description Having the following import entry types:

1. OBUIELS_Data1
  It has a higher priority (1)

2. OBUIELS_Data2
  It has a lower priority (2)
  It takes long to process each entry
  Its entries span their keys to consume all the threads
  It has a high inflow of entries to process

The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time.

This issue is caused because:

1. 1st Import Entry manger loop starts:
1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop

2. After wait time elapsed, 2nd loop starts:
2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done.
2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processor runnables that were created in the previous cycle and are still processing. Because of this, these runnables, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done.
Revision 2022-01-19 13:20 by alostale
Description Having the following import entry types:

1. OBUIELS_Data1
  It has a higher priority (1)

2. OBUIELS_Data2
  It has a lower priority (2)
  It takes long to process each entry
  Its entries span their keys to consume all the threads
  It has a high inflow of entries to process

The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time.

This issue is caused because:

1. 1st Import Entry manger loop starts:
1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop

2. After wait time elapsed, 2nd loop starts:
2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done.
2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processors that were created in the previous cycle and are still processing. Because of this, these processors, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done.
Revision 2022-01-19 13:06 by alostale
Description Having the following import entry types:

1. OBUIELS_Data1
  It has a higher priority (1)

2. OBUIELS_Data2
  It has a lower priority (2)
  It takes long to process each entry
  Its entries span their keys to consume all the threads
  It has a high inflow of entries to process

The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time.


Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker