0035832: Support using several threads to import and export the sample data

Notes
(0096216) AugustoMauch (administrator) 2017-04-27 11:25	The project is being developed in these repos: https://code.openbravo.com/erp/devel/dbsm-multithread-import-export-sample-data [^] https://code.openbravo.com/erp/devel/pi-multithread-import-export-sample-data [^]

(0096813) hgbot (developer) 2017-05-25 16:26	Repository: erp/devel/dbsm-main Changeset: c6fa7da9db9bca6babcdb7e1f95accfb73ed1da8 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Wed Apr 26 17:47:03 2017 +0200 URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/c6fa7da9db9bca6babcdb7e1f95accfb73ed1da8 [^] Related with issue 35832: ImportSampleData now supports multithreading The ImportSampleData class now supports defining the number of threads that will be used to import the files. This can be done by setting the max.threads parameter in the ant task that ends up invoking the ImportSampleData task, for instance: ant install.source -Dmax.threads=1 It uses the default Platform's implementation, so if the max.threads parameter is not provided, it will use nAvailableCPUs/2. --- M src/org/openbravo/ddlutils/task/ImportSampledata.java ---

(0096814) hgbot (developer) 2017-05-25 16:26	Repository: erp/devel/dbsm-main Changeset: bfdec7f711d33cc1417b174588dcad41214fb5b4 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Thu Apr 27 10:24:13 2017 +0200 URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/bfdec7f711d33cc1417b174588dcad41214fb5b4 [^] Related with issue 35832: ExportSampleData now supports multithreading The ExportSampleData class now supports defining the number of threads that will be used to import the files. This can be done by setting the max.threads parameter in the ant task that ends up invoking the ImportSampleData task, for instance: ant export.sample.data -Dclient='The White Valley Group' -Dmodule=org.openbravo It uses the default Platform's implementation, so if the max.threads parameter is not provided, it will use nAvailableCPUs/2. Two additional changes have been done to support multithreading properly: - To prevent mixing the log between threads, now the log does not show * Exporting table <tableName> * <n> records exported Now it will show <n> records exported from table <tableName> - If the dataset being exported has more than one dataset table that references the same table, the first dataset table will be exported to <tableName>.xml and the second to <tableName>_1.xml. If no records are exported from the first dataset table <tableName>.xml will be deleted but the second one will still be named <tableName>_1.xml. With the previous version, given that the dataset tables were exported sequentially one at a time, it was checked if the previous one had been deleted, and if so the second one would have been named <tableName>.xml. Now that several tables can be exported in parallel, each thread must know in advance the final name of the file where it will export data to. --- M src/org/apache/ddlutils/io/DatabaseDataIO.java M src/org/apache/ddlutils/platform/postgresql/PostgreSqlDatabaseDataIO.java M src/org/openbravo/ddlutils/task/ExportSampledata.java ---

(0096815) hgbot (developer) 2017-05-25 16:26	Repository: erp/devel/dbsm-main Changeset: ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Tue May 23 15:34:59 2017 +0200 URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^] Fixes issue 35832: Removes method that creates an ExecutorService A method that creates an ExecutorService was duplicated in the ImportSampledata and ExportSampledata classes. Now they just use Executors.newFixedThreadPool(platform.getMaxThreads()); The only difference is that if the number of threads is 1, the removed method would have used Executors.newSingleThreadExecutor(); instead of Executors.newFixedThreadPool(threads); The only difference is that the object created with Executors.newSingleThreadExecutor cannot be configured to use additional threads. --- M src/org/openbravo/ddlutils/task/ExportSampledata.java M src/org/openbravo/ddlutils/task/ImportSampledata.java ---

(0096816) hgbot (developer) 2017-05-25 16:29	Repository: erp/devel/pi Changeset: 49ac427a0d6689181afa29b95d7384f78eab2dca Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Thu May 25 16:25:36 2017 +0200 URL: http://code.openbravo.com/erp/devel/pi/rev/49ac427a0d6689181afa29b95d7384f78eab2dca [^] Fixes issue 35832: Multithread support in import/export sample data Now the processes that import/export sample data support multithread. To do so, the max.threads parameter must be passed to the ant task. If the parameter is not passed it will use half the available CPUs. For instance, to use 4 threads to import the sample data in an install source, this command should be used: ant install.source -Dmax.threads=4 --- M src-db/database/build.xml M src-db/database/lib/dbsourcemanager.jar ---

(0096993) hudsonbot (developer) 2017-05-29 17:43	A changeset related to this issue has been promoted main and to the Central Repository, after passing a series of tests. Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/1ee70113bdc4 [^] Maturity status: Test

(0097086) alostale (manager) 2017-06-02 07:29 edited on: 2017-06-02 07:30	code reviewed + tested as project https://docs.google.com/spreadsheets/d/1aVVhb-NYUfzx4oYMZmNRyy7y2nBIkQpvSAbCo3MlSVE/edit [^]

Issue History
Date Modified	Username	Field	Change
2017-04-25 13:27	AugustoMauch	New Issue
2017-04-25 13:27	AugustoMauch	Assigned To	=> AugustoMauch
2017-04-25 13:27	AugustoMauch	Modules	=> Core
2017-04-25 13:27	AugustoMauch	Triggers an Emergency Pack	=> No
2017-04-27 11:24	AugustoMauch	Status	new => scheduled
2017-04-27 11:25	AugustoMauch	Note Added: 0096216
2017-05-25 16:26	hgbot	Checkin
2017-05-25 16:26	hgbot	Note Added: 0096813
2017-05-25 16:26	hgbot	Checkin
2017-05-25 16:26	hgbot	Note Added: 0096814
2017-05-25 16:26	hgbot	Checkin
2017-05-25 16:26	hgbot	Note Added: 0096815
2017-05-25 16:26	hgbot	Status	scheduled => resolved
2017-05-25 16:26	hgbot	Resolution	open => fixed
2017-05-25 16:26	hgbot	Fixed in SCM revision	=> http://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^]
2017-05-25 16:27	AugustoMauch	Review Assigned To	=> alostale
2017-05-25 16:29	hgbot	Checkin
2017-05-25 16:29	hgbot	Note Added: 0096816
2017-05-25 16:29	hgbot	Fixed in SCM revision	http://code.openbravo.com/erp/devel/dbsm-main/rev/ed7342a7fb3b14d10f02b9aae965e0149b3fbfdb [^] => http://code.openbravo.com/erp/devel/pi/rev/49ac427a0d6689181afa29b95d7384f78eab2dca [^]
2017-05-25 17:27	AugustoMauch	Relationship added	related to 0036111
2017-05-29 17:43	hudsonbot	Checkin
2017-05-29 17:43	hudsonbot	Note Added: 0096993
2017-06-02 07:29	alostale	Note Added: 0097086
2017-06-02 07:29	alostale	Status	resolved => closed
2017-06-02 07:29	alostale	Fixed in Version	=> 3.0PR17Q3
2017-06-02 07:29	alostale	Tag Attached: Performance
2017-06-02 07:30	alostale	Note Edited: 0097086	View Revisions
2017-06-02 07:30	alostale	Note Edited: 0097086	View Revisions
2017-08-02 10:03	alostale	Relationship added	causes 0036573
2018-01-24 09:07	alostale	Relationship added	causes 0037103
2018-04-04 14:07	alostale	Relationship added	related to 0038269