Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0035832Openbravo ERPY. DBSourceManagerpublic2017-04-25 13:272017-06-02 07:29
ReporterAugustoMauch 
Assigned ToAugustoMauch 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in Version3.0PR17Q3 
Merge Request Status
Review Assigned Toalostale
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0035832: Support using several threads to import and export the sample data
DescriptionThe import and export sample data processes can benefit from using several threads, the same way it was done in this issue [1] for index creation and function standardization

[1] https://issues.openbravo.com/view.php?id=34654 [^]
Steps To Reproduce-
Proposed Solution
Additional Information
TagsPerformance
Relationships
related to defect 0036111 closed AugustoMauch Retail Modules Support multithread in the export.store.data task 
related to defect 0038269 closed alostale Openbravo ERP parallel export.sampledata can get stuck 
causes defect 0036573 closed alostale Openbravo ERP import sample data can get stuck when executed by multiple threads 
causes defect 0037103 closed AugustoMauch Openbravo ERP Import.sample.data does not immediately stop when a file cannot be imported 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2017-04-25 13:27AugustoMauchNew Issue
2017-04-25 13:27AugustoMauchAssigned To => AugustoMauch
2017-04-25 13:27AugustoMauchModules => Core
2017-04-25 13:27AugustoMauchTriggers an Emergency Pack => No
2017-04-27 11:24AugustoMauchStatusnew => scheduled
2017-04-27 11:25AugustoMauchNote Added: 0096216
2017-05-25 16:26hgbotCheckin
2017-05-25 16:26hgbotNote Added: 0096813
2017-05-25 16:26hgbotCheckin
2017-05-25 16:26hgbotNote Added: 0096814
2017-05-25 16:26hgbotCheckin
2017-05-25 16:26hgbotNote Added: 0096815
2017-05-25 16:26hgbotStatusscheduled => resolved
2017-05-25 16:26hgbotResolutionopen => fixed
2017-05-25 16:26hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^]
2017-05-25 16:27AugustoMauchReview Assigned To => alostale
2017-05-25 16:29hgbotCheckin
2017-05-25 16:29hgbotNote Added: 0096816
2017-05-25 16:29hgbotFixed in SCM revisionhttp://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^] => http://code.openbravo.com/erp/devel/pi/rev/49ac427a0d6689181afa29b95d7384f78eab2dca [^]
2017-05-25 17:27AugustoMauchRelationship addedrelated to 0036111
2017-05-29 17:43hudsonbotCheckin
2017-05-29 17:43hudsonbotNote Added: 0096993
2017-06-02 07:29alostaleNote Added: 0097086
2017-06-02 07:29alostaleStatusresolved => closed
2017-06-02 07:29alostaleFixed in Version => 3.0PR17Q3
2017-06-02 07:29alostaleTag Attached: Performance
2017-06-02 07:30alostaleNote Edited: 0097086bug_revision_view_page.php?bugnote_id=0097086#r15268
2017-06-02 07:30alostaleNote Edited: 0097086bug_revision_view_page.php?bugnote_id=0097086#r15269
2017-08-02 10:03alostaleRelationship addedcauses 0036573
2018-01-24 09:07alostaleRelationship addedcauses 0037103
2018-04-04 14:07alostaleRelationship addedrelated to 0038269

Notes
(0096216)
AugustoMauch   
2017-04-27 11:25   
The project is being developed in these repos:

https://code.openbravo.com/erp/devel/dbsm-multithread-import-export-sample-data [^]
https://code.openbravo.com/erp/devel/pi-multithread-import-export-sample-data [^]
(0096813)
hgbot   
2017-05-25 16:26   
Repository: erp/devel/dbsm-main
Changeset: c6fa7da9db9bca6babcdb7e1f95accfb73ed1da8
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Wed Apr 26 17:47:03 2017 +0200
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/c6fa7da9db9bca6babcdb7e1f95accfb73ed1da8 [^]

Related with issue 35832: ImportSampleData now supports multithreading

The ImportSampleData class now supports defining the number of threads that will be used to import the files. This can be done by setting the max.threads parameter in the ant task that ends up invoking the ImportSampleData task, for instance:

ant install.source -Dmax.threads=1

It uses the default Platform's implementation, so if the max.threads parameter is not provided, it will use nAvailableCPUs/2.

---
M src/org/openbravo/ddlutils/task/ImportSampledata.java
---
(0096814)
hgbot   
2017-05-25 16:26   
Repository: erp/devel/dbsm-main
Changeset: bfdec7f711d33cc1417b174588dcad41214fb5b4
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Thu Apr 27 10:24:13 2017 +0200
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/bfdec7f711d33cc1417b174588dcad41214fb5b4 [^]

Related with issue 35832: ExportSampleData now supports multithreading

The ExportSampleData class now supports defining the number of threads that will be used to import the files. This can be done by setting the max.threads parameter in the ant task that ends up invoking the ImportSampleData task, for instance:

ant export.sample.data -Dclient='The White Valley Group' -Dmodule=org.openbravo

It uses the default Platform's implementation, so if the max.threads parameter is not provided, it will use nAvailableCPUs/2.

Two additional changes have been done to support multithreading properly:
- To prevent mixing the log between threads, now the log does not show
   * Exporting table <tableName>
   * <n> records exported
   Now it will show <n> records exported from table <tableName>
- If the dataset being exported has more than one dataset table that references the same table, the first dataset table will be exported to <tableName>.xml and the second to <tableName>_1.xml. If no records are exported from the first dataset table <tableName>.xml will be deleted but the second one will still be named <tableName>_1.xml. With the previous version, given that the dataset tables were exported sequentially one at a time, it was checked if the previous one had been deleted, and if so the second one would have been named <tableName>.xml. Now that several tables can be exported in parallel, each thread must know in advance the final name of the file where it will export data to.

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
M src/org/apache/ddlutils/platform/postgresql/PostgreSqlDatabaseDataIO.java
M src/org/openbravo/ddlutils/task/ExportSampledata.java
---
(0096815)
hgbot   
2017-05-25 16:26   
Repository: erp/devel/dbsm-main
Changeset: ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Tue May 23 15:34:59 2017 +0200
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^]

Fixes issue 35832: Removes method that creates an ExecutorService

A method that creates an ExecutorService was duplicated in the ImportSampledata and ExportSampledata classes. Now they just use Executors.newFixedThreadPool(platform.getMaxThreads()); The only difference is that if the number of threads is 1, the removed method would have used Executors.newSingleThreadExecutor(); instead of Executors.newFixedThreadPool(threads); The only difference is that the object created with Executors.newSingleThreadExecutor cannot be configured to use additional threads.

---
M src/org/openbravo/ddlutils/task/ExportSampledata.java
M src/org/openbravo/ddlutils/task/ImportSampledata.java
---
(0096816)
hgbot   
2017-05-25 16:29   
Repository: erp/devel/pi
Changeset: 49ac427a0d6689181afa29b95d7384f78eab2dca
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Thu May 25 16:25:36 2017 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/49ac427a0d6689181afa29b95d7384f78eab2dca [^]

Fixes issue 35832: Multithread support in import/export sample data

Now the processes that import/export sample data support multithread. To do so, the max.threads parameter must be passed to the ant task. If the parameter is not passed it will use half the available CPUs. For instance, to use 4 threads to import the sample data in an install source, this command should be used:

  ant install.source -Dmax.threads=4

---
M src-db/database/build.xml
M src-db/database/lib/dbsourcemanager.jar
---
(0096993)
hudsonbot   
2017-05-29 17:43   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/1ee70113bdc4 [^]
Maturity status: Test
(0097086)
alostale   
2017-06-02 07:29   
(edited on: 2017-06-02 07:30)
code reviewed + tested as project

https://docs.google.com/spreadsheets/d/1aVVhb-NYUfzx4oYMZmNRyy7y2nBIkQpvSAbCo3MlSVE/edit [^]