Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0033875
TypeCategorySeverityReproducibilityDate SubmittedLast Update
feature request[Openbravo ERP] Y. DBSourceManagermajorhave not tried2016-09-01 13:322016-12-16 18:39
ReporterAugustoMauchView Statuspublic 
Assigned ToAugustoMauch 
PrioritynormalResolutionfixedFixed in Version3.0PR17Q1
StatusclosedFix in branchFixed in SCM revisionf169cafac730
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Merge Request Status
Review Assigned Toalostale
OBNetwork customerNo
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0033875: Importing/Exporting sample data using PostgreSQL's COPY instead of current XML

DescriptionCurrently we use XML to export and import Openbravo's sampledata. To export we extract the database contents ourselves and then use ddlutil's DataWriter to export them to database. To import we use the DataReader, that parses the XML en loads the data in the database.

PostgreSQL provides the COPY command [1], using it would have at least two advantages:
- The exported sample data is much smaller since the CSV format has less overhead that XML (one particular uncompressed sampledata weighted 4.5G in XML versus 1.2G in CSV, 147M vs 127M compressed).
- Both the export and import process are much faster. Metrics will be added here when they are available.

Project: http://wiki.openbravo.com/wiki/Projects:Import_Export_Sample_Data_Using_PostgreSQL_Copy [^]

[1] https://www.postgresql.org/docs/9.5/static/sql-copy.html [^]
Steps To Reproduce-
TagsNo tags attached.
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]
has duplicate feature request 0034094 closedAugustoMauch Openbravo ERP Support exporting and importing sampledata using PostgreSQL's COPY 
related to defect 0034639 closedAugustoMauch Retail Modules ExportStoreData class has to be updated after the latest changes done in ExportsampleData 
causes defect 0034638 closedAugustoMauch Openbravo ERP Changing the visibility of DatabaseDataIO.writeDataForTableToXML has broken a job 

-  Notes
(0091859)
hgbot (developer)
2016-11-25 17:41

 Repository: erp/devel/dbsm-main
Changeset: 492821683ddcdb8f8f43859373169c52abdff39f
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Nov 25 17:37:31 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^]

Fixes issue 33875: Sample data can be exported/imported with pgsql copy format

The export.sample.data ant task can now export the sample data in PostgreSQL`s copy format when the data is stored in a PostgreSQL database, see http://wiki.openbravo.com/wiki/How_to_export_sample_data [^]

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
M src/org/openbravo/ddlutils/task/DatabaseUtils.java
M src/org/openbravo/ddlutils/task/ExportSampledata.java
M src/org/openbravo/ddlutils/task/ImportSampledata.java
M src/org/openbravo/ddlutils/util/OBDataset.java
M src/org/openbravo/ddlutils/util/OBDatasetTable.java
A src/org/apache/ddlutils/io/DataSetTableExporter.java
A src/org/apache/ddlutils/io/DataSetTableQueryGenerator.java
A src/org/apache/ddlutils/io/DataSetTableQueryGeneratorExtraProperties.java
A src/org/apache/ddlutils/platform/postgresql/PostgreSqlDatabaseDataIO.java
---
(0091860)
hgbot (developer)
2016-11-25 17:56

 Repository: erp/devel/pi
Changeset: f169cafac730141372142a78fbc68e040c12b7c1
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Nov 25 17:55:35 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/f169cafac730141372142a78fbc68e040c12b7c1 [^]

Fixes issue 33875: Sample data can be exported/imported with pgsql copy format

The export.sample.data ant task can now export the sample data in PostgreSQL`s copy format when the data is stored in a PostgreSQL database, see http://wiki.openbravo.com/wiki/How_to_export_sample_data [^]

---
M src-db/database/build.xml
M src-db/database/lib/dbsourcemanager.jar
---
(0091864)
hgbot (developer)
2016-11-26 14:03

 Repository: erp/devel/dbsm-main
Changeset: 9ee6ed582c36f9a342f3afd2324dfc890ef34239
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Sat Nov 26 14:02:36 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/9ee6ed582c36f9a342f3afd2324dfc890ef34239 [^]

Fixes issue 34638, related with issue 33875: Reverts method visibility change

The DatabaseDataIO.writeDataForTableToXML method was made private because it was not being used by any class other than DatabaseDataIO inside dbsm, but I failed to notice that it was being used in an external module (Easy Extensible Attributes).

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
---
(0091866)
hgbot (developer)
2016-11-26 14:04

 Repository: erp/devel/pi
Changeset: 127865df93ddcdb773414120a0bbcae1bb7736c8
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Sat Nov 26 14:03:59 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/127865df93ddcdb773414120a0bbcae1bb7736c8 [^]

Fixes issue 34638, related with issue 33875: Reverts method visibility change

The DatabaseDataIO.writeDataForTableToXML method was made private because it was not being used by any class other than DatabaseDataIO inside dbsm, but I failed to notice that it was being used in an external module (Easy Extensible Attributes).

---
M src-db/database/lib/dbsourcemanager.jar
---
(0091936)
hgbot (developer)
2016-11-30 17:00

 Repository: erp/devel/dbsm-main
Changeset: 6342c7ef4c4b48b6d9605a41eaabfc044b0cd1de
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Wed Nov 30 15:26:35 2016 +0100
URL: http://code.openbravo.com/erp/devel/dbsm-main/rev/6342c7ef4c4b48b6d9605a41eaabfc044b0cd1de [^]

Related with issue 33875: Inlines method to avoid working with closed resultset

The DatabaseDataIO.getDatasetTableDataIterator method, which was only used in one place, returned a Iterator<DynaBean>, which was an iterator of a ResultSet. The problem was that the ResultSet was being closed inside the getDatasetTableDataIterator method, so the returned iterator could not work.

---
M src/org/apache/ddlutils/io/DatabaseDataIO.java
---
(0091937)
hgbot (developer)
2016-11-30 17:03

 Repository: erp/devel/pi
Changeset: 83e12075fade2e0dfff7c6528f879031758f3e8e
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Wed Nov 30 17:02:36 2016 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/83e12075fade2e0dfff7c6528f879031758f3e8e [^]

Related with issue 33875: Inlines method to avoid working with closed resultset

The DatabaseDataIO.getDatasetTableDataIterator method, which was only used in one place, returned a Iterator<DynaBean>, which was an iterator of a ResultSet. The problem was that the Re
sultSet was being closed inside the getDatasetTableDataIterator method, so the returned iterator could not work.

---
M src-db/database/lib/dbsourcemanager.jar
---
(0092005)
alostale (viewer)
2016-12-02 08:41

 code reviewed and tested within project
(0092663)
hudsonbot (viewer)
2016-12-16 18:39

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test
(0092665)
hudsonbot (viewer)
2016-12-16 18:39

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test
(0092672)
hudsonbot (viewer)
2016-12-16 18:39

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/dc8bf00badd0 [^]
Maturity status: Test

- Issue History
Date Modified Username Field Change
2016-09-01 13:32 AugustoMauch New Issue
2016-09-01 13:32 AugustoMauch Assigned To => AugustoMauch
2016-09-01 13:32 AugustoMauch OBNetwork customer => No
2016-09-01 13:32 AugustoMauch Modules => Core
2016-09-01 13:32 AugustoMauch Triggers an Emergency Pack => No
2016-10-28 13:47 AugustoMauch Relationship added has duplicate 0034094
2016-11-25 17:41 hgbot Checkin
2016-11-25 17:41 hgbot Note Added: 0091859
2016-11-25 17:41 hgbot Status new => resolved
2016-11-25 17:41 hgbot Resolution open => fixed
2016-11-25 17:41 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^]
2016-11-25 17:56 hgbot Checkin
2016-11-25 17:56 hgbot Note Added: 0091860
2016-11-25 17:56 hgbot Fixed in SCM revision http://code.openbravo.com/erp/devel/dbsm-main/rev/492821683ddcdb8f8f43859373169c52abdff39f [^] => http://code.openbravo.com/erp/devel/pi/rev/f169cafac730141372142a78fbc68e040c12b7c1 [^]
2016-11-25 17:58 AugustoMauch Review Assigned To => alostale
2016-11-26 13:57 AugustoMauch Relationship added causes 0034638
2016-11-26 14:03 hgbot Checkin
2016-11-26 14:03 hgbot Note Added: 0091864
2016-11-26 14:04 hgbot Checkin
2016-11-26 14:04 hgbot Note Added: 0091866
2016-11-26 14:13 AugustoMauch Relationship added related to 0034639
2016-11-30 17:00 hgbot Checkin
2016-11-30 17:00 hgbot Note Added: 0091936
2016-11-30 17:03 hgbot Checkin
2016-11-30 17:03 hgbot Note Added: 0091937
2016-12-02 08:41 alostale Description Updated View Revisions
2016-12-02 08:41 alostale Note Added: 0092005
2016-12-02 08:41 alostale Status resolved => closed
2016-12-02 08:41 alostale Fixed in Version => 3.0PR17Q1
2016-12-16 18:39 hudsonbot Checkin
2016-12-16 18:39 hudsonbot Note Added: 0092663
2016-12-16 18:39 hudsonbot Checkin
2016-12-16 18:39 hudsonbot Note Added: 0092665
2016-12-16 18:39 hudsonbot Checkin
2016-12-16 18:39 hudsonbot Note Added: 0092672

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker