Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0033875Openbravo ERPY. DBSourceManagerpublic2016-09-01 13:322016-12-16 18:39
ReporterAugustoMauch 
Assigned ToAugustoMauch 
PrioritynormalSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in Version3.0PR17Q1 
Merge Request Status
Review Assigned Toalostale
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0033875: Importing/Exporting sample data using PostgreSQL's COPY instead of current XML
DescriptionCurrently we use XML to export and import Openbravo's sampledata. To export we extract the database contents ourselves and then use ddlutil's DataWriter to export them to database. To import we use the DataReader, that parses the XML en loads the data in the database.

PostgreSQL provides the COPY command [1], using it would have at least two advantages:
- The exported sample data is much smaller since the CSV format has less overhead that XML (one particular uncompressed sampledata weighted 4.5G in XML versus 1.2G in CSV, 147M vs 127M compressed).
- Both the export and import process are much faster. Metrics will be added here when they are available.

Project: http://wiki.openbravo.com/wiki/Projects:Import_Export_Sample_Data_Using_PostgreSQL_Copy [^]

[1] https://www.postgresql.org/docs/9.5/static/sql-copy.html [^]
Steps To Reproduce-
Proposed Solution
Additional Information
TagsNo tags attached.
Relationships
has duplicate feature request 0034094 closed AugustoMauch Openbravo ERP Support exporting and importing sampledata using PostgreSQL's COPY 
related to defect 0034639 closed AugustoMauch Retail Modules ExportStoreData class has to be updated after the latest changes done in ExportsampleData 
causes defect 0034638 closed AugustoMauch Openbravo ERP Changing the visibility of DatabaseDataIO.writeDataForTableToXML has broken a job 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2016-09-01 13:32AugustoMauchNew Issue
2016-09-01 13:32AugustoMauchAssigned To => AugustoMauch
2016-09-01 13:32AugustoMauchModules => Core
2016-09-01 13:32AugustoMauchTriggers an Emergency Pack => No
2016-10-28 13:47AugustoMauchRelationship addedhas duplicate 0034094
2016-11-25 17:41hgbotCheckin
2016-11-25 17:41hgbotNote Added: 0091859
2016-11-25 17:41hgbotStatusnew => resolved
2016-11-25 17:41hgbotResolutionopen => fixed
2016-11-25 17:41hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^]
2016-11-25 17:56hgbotCheckin
2016-11-25 17:56hgbotNote Added: 0091860
2016-11-25 17:56hgbotFixed in SCM revisionhttp://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^] => http://code.openbravo.com/erp/devel/pi/rev/f169cafac730141372142a78fbc68e040c12b7c1 [^]
2016-11-25 17:58AugustoMauchReview Assigned To => alostale
2016-11-26 13:57AugustoMauchRelationship addedcauses 0034638
2016-11-26 14:03hgbotCheckin
2016-11-26 14:03hgbotNote Added: 0091864
2016-11-26 14:04hgbotCheckin
2016-11-26 14:04hgbotNote Added: 0091866
2016-11-26 14:13AugustoMauchRelationship addedrelated to 0034639
2016-11-30 17:00hgbotCheckin
2016-11-30 17:00hgbotNote Added: 0091936
2016-11-30 17:03hgbotCheckin
2016-11-30 17:03hgbotNote Added: 0091937
2016-12-02 08:41alostaleDescription Updatedbug_revision_view_page.php?rev_id=13949#r13949
2016-12-02 08:41alostaleNote Added: 0092005
2016-12-02 08:41alostaleStatusresolved => closed
2016-12-02 08:41alostaleFixed in Version => 3.0PR17Q1
2016-12-16 18:39hudsonbotCheckin
2016-12-16 18:39hudsonbotNote Added: 0092663
2016-12-16 18:39hudsonbotCheckin
2016-12-16 18:39hudsonbotNote Added: 0092665
2016-12-16 18:39hudsonbotCheckin
2016-12-16 18:39hudsonbotNote Added: 0092672

Notes
(0091859)
hgbot   
2016-11-25 17:41   
Repository: erp/devel/dbsm-main
Changeset: 492821683ddcdb8f8f43859373169c52abdff39f
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Nov 25 17:37:31 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^]

Fixes issue 33875: Sample data can be exported/imported with pgsql copy format

The export.sample.data ant task can now export the sample data in PostgreSQL`s copy format when the data is stored in a PostgreSQL database, see http://wiki.openbravo.com/wiki/How_to_export_sample_data [^]

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
M src/org/openbravo/ddlutils/task/DatabaseUtils.java
M src/org/openbravo/ddlutils/task/ExportSampledata.java
M src/org/openbravo/ddlutils/task/ImportSampledata.java
M src/org/openbravo/ddlutils/util/OBDataset.java
M src/org/openbravo/ddlutils/util/OBDatasetTable.java
A src/org/apache/ddlutils/io/DataSetTableExporter.java
A src/org/apache/ddlutils/io/DataSetTableQueryGenerator.java
A src/org/apache/ddlutils/io/DataSetTableQueryGeneratorExtraProperties.java
A src/org/apache/ddlutils/platform/postgresql/PostgreSqlDatabaseDataIO.java
---
(0091860)
hgbot   
2016-11-25 17:56   
Repository: erp/devel/pi
Changeset: f169cafac730141372142a78fbc68e040c12b7c1
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Nov 25 17:55:35 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/f169cafac730141372142a78fbc68e040c12b7c1 [^]

Fixes issue 33875: Sample data can be exported/imported with pgsql copy format

The export.sample.data ant task can now export the sample data in PostgreSQL`s copy format when the data is stored in a PostgreSQL database, see http://wiki.openbravo.com/wiki/How_to_export_sample_data [^]

---
M src-db/database/build.xml
M src-db/database/lib/dbsourcemanager.jar
---
(0091864)
hgbot   
2016-11-26 14:03   
Repository: erp/devel/dbsm-main
Changeset: 9ee6ed582c36f9a342f3afd2324dfc890ef34239
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Sat Nov 26 14:02:36 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/9ee6ed582c36f9a342f3afd2324dfc890ef34239 [^]

Fixes issue 34638, related with issue 33875: Reverts method visibility change

The DatabaseDataIO.writeDataForTableToXML method was made private because it was not being used by any class other than DatabaseDataIO inside dbsm, but I failed to notice that it was being used in an external module (Easy Extensible Attributes).

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
---
(0091866)
hgbot   
2016-11-26 14:04   
Repository: erp/devel/pi
Changeset: 127865df93ddcdb773414120a0bbcae1bb7736c8
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Sat Nov 26 14:03:59 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/127865df93ddcdb773414120a0bbcae1bb7736c8 [^]

Fixes issue 34638, related with issue 33875: Reverts method visibility change

The DatabaseDataIO.writeDataForTableToXML method was made private because it was not being used by any class other than DatabaseDataIO inside dbsm, but I failed to notice that it was being used in an external module (Easy Extensible Attributes).

---
M src-db/database/lib/dbsourcemanager.jar
---
(0091936)
hgbot   
2016-11-30 17:00   
Repository: erp/devel/dbsm-main
Changeset: 6342c7ef4c4b48b6d9605a41eaabfc044b0cd1de
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Wed Nov 30 15:26:35 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/6342c7ef4c4b48b6d9605a41eaabfc044b0cd1de [^]

Related with issue 33875: Inlines method to avoid working with closed resultset

The DatabaseDataIO.getDatasetTableDataIterator method, which was only used in one place, returned a Iterator<DynaBean>, which was an iterator of a ResultSet. The problem was that the ResultSet was being closed inside the getDatasetTableDataIterator method, so the returned iterator could not work.

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
---
(0091937)
hgbot   
2016-11-30 17:03   
Repository: erp/devel/pi
Changeset: 83e12075fade2e0dfff7c6528f879031758f3e8e
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Wed Nov 30 17:02:36 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/83e12075fade2e0dfff7c6528f879031758f3e8e [^]

Related with issue 33875: Inlines method to avoid working with closed resultset

The DatabaseDataIO.getDatasetTableDataIterator method, which was only used in one place, returned a Iterator<DynaBean>, which was an iterator of a ResultSet. The problem was that the Re
sultSet was being closed inside the getDatasetTableDataIterator method, so the returned iterator could not work.

---
M src-db/database/lib/dbsourcemanager.jar
---
(0092005)
alostale   
2016-12-02 08:41   
code reviewed and tested within project
(0092663)
hudsonbot   
2016-12-16 18:39   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test
(0092665)
hudsonbot   
2016-12-16 18:39   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test
(0092672)
hudsonbot   
2016-12-16 18:39   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test