Project:
View Issue Details[ Jump to Notes ] | [ Issue History ] [ Print ] | |||||||
ID | ||||||||
0016623 | ||||||||
Type | Category | Severity | Reproducibility | Date Submitted | Last Update | |||
defect | [Openbravo ERP] I. Performance | minor | always | 2011-04-03 13:57 | 2011-07-14 18:44 | |||
Reporter | eintelau | View Status | public | |||||
Assigned To | alostale | |||||||
Priority | normal | Resolution | fixed | Fixed in Version | ||||
Status | closed | Fix in branch | Fixed in SCM revision | 16ec818d352b | ||||
Projection | none | ETA | none | Target Version | 3.0MP1 | |||
OS | Any | Database | Any | Java version | ||||
OS Version | Database version | Ant version | ||||||
Product Version | SCM revision | |||||||
Review Assigned To | ||||||||
Web browser | ||||||||
Modules | User Interface Selector | |||||||
Regression level | ||||||||
Regression date | ||||||||
Regression introduced in release | ||||||||
Regression introduced by commit | ||||||||
Triggers an Emergency Pack | No | |||||||
Summary | 0016623: CustomQuerySelectorDatasource is very inefficient when filtering using a foreign key reference | |||||||
Description | If a selector has a field defined with a reference that is a foreign key reference (e.g. Table, TableDir, etc), then the query generated to filter the results by that field is extremely inefficient. The generated query treats the id's as text fields and does a comparison using C_IGNORE_ACCENT(a) = C_IGNORE_ACCENT(b). When dealing with a large dataset can increase the query time by an order of magnitude or more. | |||||||
Proposed Solution | Add special case for references that are foreign key so that the generated filter does a direct comparison of id with supplied value. | |||||||
Tags | No tags attached. | |||||||
Attached Files | 16623.patch [^] (1,645 bytes) 2011-04-03 13:59 [Show Content]
16623v2.patch [^] (5,322 bytes) 2011-04-06 02:52 [Show Content] | |||||||
Relationships [ Relation Graph ] [ Dependency Graph ] | |||||||||||||||||||||||||||||||||||||||||||
|
Notes | |
(0035633) eintelau (developer) 2011-04-06 02:49 |
After some more investigation and tuning I have found some other parts of the query generation that are very inefficient. 1) Filtering for string domain types is done using the C_IGNORE_ACCENT on the column being filtered. This means that indexes, etc can't be used & the ignore accent function has to be evaluated for every row in the table. With a large dataset this leads to queries taking orders of magnitude longer (from < 0.5 sec to > 20 seconds on my test data). Even marking the ignore accent function as immutable rather than volatile doesn't make the query performant. 2) Filtering for ID columns is done as a string and uses the ignore accent / like clause. This should be done as a straight equality comparison. Updated patch attached with fixes for these issues. The changes are - Remove use of C_IGNORE_ACCENT and replace with lower (for case insensitive comparision) - Add new domain type class (UniqueIdDomainType extends StringDomainType) for ID columns - Update ID Reference to use new domain type - When creating filter clause for UniqueIdDomainType use straight equality comparison instead of LIKE. |
(0037183) aldorisioj (reporter) 2011-05-18 23:05 |
We have observed this issue here at SAIC as well. We're running MP24 on Oracle 11gR2. We determined that this problem is caused by the query performing a full table scan whenever searching for Products because the selector invoked the C_IGNORE_ACCENT function. Rather than switching the function from C_IGNORE_ACCENT to LOWER or performing any other source code changes, we believe the most efficient way to resolve this issue is to create a function-based index for C_IGNORE_ACCENT. This would preserve the functionality provided by C_IGNORE_ACCENT but also address the performance concern. To make this happen, C_IGNORE_ACCENT must first be made deterministic. Which may be accomplished by recompiling the stored procedure with that command as such: CREATE OR REPLACE FUNCTION C_IGNORE_ACCENT(v_str IN VARCHAR2) RETURN VARCHAR2 DETERMINISTIC AS BEGIN RETURN(TRANSLATE(upper(C_TRIM(v_str)), 'ÁÀÂÄÉÈÊËÍÌÎÏÓÒÔÖÚÙÛÜÑ#Ç', 'AAAAEEEEIIIIOOOOUUUUNNC')); END C_IGNORE_ACCENT ; You can then create the function-based index with the following command (in Oracle): CREATE INDEX IDX_PROD_IGN_ACCENT_VALUE ON M_PRODUCT ( C_IGNORE_ACCENT (VALUE) ); A similar command is available for PostgreSQL. |
(0037184) jonalegriaesarte (reporter) 2011-05-18 23:37 |
Reported by a professional subscriber too |
(0037288) iciordia (manager) 2011-05-20 17:09 |
Asier, it seems that the Business Partner selector is the only component in Openbravo 3 that uses c_ignore_accent. The fix to this issue is just don't use c_ignore_accent from this selector. Ismael |
(0038683) hgbot (developer) 2011-07-01 14:03 |
Repository: erp/devel/pi Changeset: 16ec818d352b85ad7e64eea9dd95a69383c62a28 Author: Asier Lostalé <asier.lostale <at> openbravo.com> Date: Fri Jul 01 14:03:32 2011 +0200 URL: http://code.openbravo.com/erp/devel/pi/rev/16ec818d352b85ad7e64eea9dd95a69383c62a28 [^] fixed bug 16623: Don't use ignore accent in CustomQuerySelectorDatasource --- M modules/org.openbravo.userinterface.selector/src/org/openbravo/userinterface/selector/CustomQuerySelectorDatasource.java M src-db/database/sourcedata/AD_REFERENCE.xml A src/org/openbravo/base/model/domaintype/UniqueIdDomainType.java --- |
(0038684) hgbot (developer) 2011-07-01 14:12 |
Repository: erp/devel/pi Changeset: 11cdbe7be970b1ca7f31b3b8782082304fdca9db Author: Asier Lostalé <asier.lostale <at> openbravo.com> Date: Fri Jul 01 14:11:17 2011 +0200 URL: http://code.openbravo.com/erp/devel/pi/rev/11cdbe7be970b1ca7f31b3b8782082304fdca9db [^] related to issue 16623: fixed file format --- M src/org/openbravo/base/model/domaintype/UniqueIdDomainType.java --- |
(0038852) hudsonbot (developer) 2011-07-06 02:42 |
A changeset related to this issue has been promoted main and to the Central Repository, after passing a series of tests. Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/f1c481b1a146 [^] Maturity status: Test |
(0038853) hudsonbot (developer) 2011-07-06 02:42 |
A changeset related to this issue has been promoted main and to the Central Repository, after passing a series of tests. Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/f1c481b1a146 [^] Maturity status: Test |
(0039071) shuehner (administrator) 2011-07-14 18:44 |
Reviewed/tested on pi/pgsql (rev: 5ba902b0d15e). String comparison now done using case-insensitive like without using the c_ignore_accent function at all. Also two new cases added for fk/id-references to use straight equality without the like overhead. |
Issue History | |||
Date Modified | Username | Field | Change |
2011-04-03 13:57 | eintelau | New Issue | |
2011-04-03 13:57 | eintelau | Assigned To | => dbaz |
2011-04-03 13:57 | eintelau | Modules | => User Interface Selector |
2011-04-03 13:59 | eintelau | File Added: 16623.patch | |
2011-04-04 10:56 | dbaz | Assigned To | dbaz => mtaal |
2011-04-05 14:07 | shuehner | Category | B. User interface => I. Performance |
2011-04-05 17:19 | shuehner | Relationship added | blocks 0016611 |
2011-04-06 02:49 | eintelau | Note Added: 0035633 | |
2011-04-06 02:52 | eintelau | File Added: 16623v2.patch | |
2011-04-11 10:29 | alostale | Status | new => scheduled |
2011-04-11 15:40 | mtaal | Assigned To | mtaal => alostale |
2011-05-18 22:22 | aldorisioj | Issue Monitored: aldorisioj | |
2011-05-18 23:05 | aldorisioj | Note Added: 0037183 | |
2011-05-18 23:37 | jonalegriaesarte | Note Added: 0037184 | |
2011-05-19 08:27 | alostale | Assigned To | alostale => mtaal |
2011-05-20 17:07 | iciordia | Assigned To | mtaal => alostale |
2011-05-20 17:07 | iciordia | Target Version | => 3.0MP0 |
2011-05-20 17:09 | iciordia | Note Added: 0037288 | |
2011-05-20 17:23 | alostale | Relationship added | duplicate of 0016891 |
2011-05-20 17:42 | alostale | Relationship replaced | related to 0016891 |
2011-06-02 10:53 | dmitry_mezentsev | Target Version | 3.0MP0 => 3.0MP1 |
2011-06-17 13:14 | gorkaion | Relationship added | blocks 0016373 |
2011-06-17 13:14 | gorkaion | Relationship deleted | blocks 0016373 |
2011-06-17 13:14 | gorkaion | Relationship added | related to 0016373 |
2011-07-01 14:03 | hgbot | Checkin | |
2011-07-01 14:03 | hgbot | Note Added: 0038683 | |
2011-07-01 14:03 | hgbot | Status | scheduled => resolved |
2011-07-01 14:03 | hgbot | Resolution | open => fixed |
2011-07-01 14:03 | hgbot | Fixed in SCM revision | => http://code.openbravo.com/erp/devel/pi/rev/16ec818d352b85ad7e64eea9dd95a69383c62a28 [^] |
2011-07-01 14:12 | hgbot | Checkin | |
2011-07-01 14:12 | hgbot | Note Added: 0038684 | |
2011-07-06 02:42 | hudsonbot | Checkin | |
2011-07-06 02:42 | hudsonbot | Note Added: 0038852 | |
2011-07-06 02:42 | hudsonbot | Checkin | |
2011-07-06 02:42 | hudsonbot | Note Added: 0038853 | |
2011-07-14 18:44 | shuehner | Note Added: 0039071 | |
2011-07-14 18:44 | shuehner | Status | resolved => closed |
2011-08-16 13:21 | dbaz | Relationship added | related to 0018283 |
2011-08-16 13:21 | dbaz | Relationship added | related to 0018282 |
2011-08-19 12:56 | dbaz | Relationship added | related to 0018321 |
2011-11-09 16:50 | dbaz | Relationship added | related to 0019027 |
Copyright © 2000 - 2009 MantisBT Group |