Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0020623
TypeCategorySeverityReproducibilityDate SubmittedLast Update
design defect[Openbravo ERP] A. Platformmajorhave not tried2012-05-28 12:372013-08-06 11:34
ReporteralostaleView Statuspublic 
Assigned Toshankarb 
PrioritylowResolutionfixedFixed in Version3.0MP26
StatusclosedFix in branchpiFixed in SCM revision091b2c16deaf
ProjectionnoneETAnoneTarget Version3.0MP25
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Merge Request Status
Review Assigned Toalostale
OBNetwork customerNo
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0020623: 32K limit for attachments

DescriptionIn a linux with ext3 file system it is not possible to create attachments for more than 32K records.

This problem is caused by 2 facts:
*File system limitation: ext3 allows to create up to 32K subdirectories within a directory.
*The way OB manages attachments: it creates a subdirectory within attachments directory for each record with attachments.
Steps To Reproduce-Try to created more than attachments for more than 32K records.
Proposed SolutionCurrent structure is:

- Table1-record1
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
- Table1-record2
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
- Table2-record1
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
-...

Proposed solution is to split subdirectories in tables + have a limit of 32K within each of them:

*Alternative 1:
Use the same approach squid does: http://serverfault.com/a/46386 [^]

It is the same approach as proposed in issue 0012488

Splitting file name based on its record UUID.

If this alternative is chosen, it would require to decide how the actual split would be done.

*Alternative 2:

- Table1
  |-- Group 1
  | |-- Record 1
  | | |-- Attachment 1
  | | |-- ...
  | |-- Record 2
  | | |-- Attachment 1
  | | |-- ...
  | |-- ...
  | |-- Record 31999
  |-- Group 2
  | |-- Record 1
  | | |-- Attachment 1
  | | |-- ...
  | |-- Record 2
  | | |-- Attachment 1
  | | |-- ...
  | |-- ...
  | |-- Record 31999
  |--...
- Table 2
- ...

This way it would support attachments for 1.024M (32^2K) records on each table, 32K attachment for record.

Note that currently, c_file stores just name of attachment without path as it is calculated based on table + record id.

To maintain backwards compatibility:
-If implementing solution 1. Add a new boolean to c_file, this is defaulted to N. When N the old approach is used, for new attachments value is Y and file name is splitted.
-If implementing solution 2. A new "path" field should be added to c_file. This path is relative to attachments directory and only would be used for new attachements. Old attachments (identified by this field being null) would still work in the old way.
-Prepare a migration tool. This tool to be optionally executed would move attachments from existent structure to new one.
-Create a new system level preference (by default not set), when set, save attachments in old way

**Regression risk**
New attachments would follow a different approach to determine their location. This would make to fail modules trying to read new attachments in the old way.
TagsNo tags attached.
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]
has duplicate design defect 0012488 closediperdomo When there are attachemnts for more than 31998 documents it is not possible to create new ones 
related to design defect 0024289 closedshankarb Retrieving attachments API Change 
related to defect 0024373 closedshankarb api-check failure at build 4114 
causes defect 00293133.0PR15Q2 closedalostale Download attachments selecting several records is not working fine 

-  Notes
(0052328)
AugustoMauch (administrator)
2012-09-24 12:47

 Effort: 5
Impact: low
Plan: long
(0059378)
alostale (viewer)
2013-06-12 09:59

 32K subdirectory limitation in ext3 file system is doubled to 64K in ext4
(0059933)
shankarb (viewer)
2013-07-09 08:50

 Alternative 2 is going to be used to fix this issue.
(0059961)
hgbot (developer)
2013-07-13 09:16

 Repository: erp/devel/pi
Changeset: bed326b6ddf6f08b10cf22cd7403aff84c0e3750
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Thu Jul 11 11:00:28 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^]

Fixes Issue 0020623, Fixes Issue 24289: 32K limit for attachments

The attachment model has been changed as explained in the wiki[1].
A new ant task has been added to migrate attachments from old model to new model.

[1] http://wiki.openbravo.com/wiki/Attachments [^]

---
M build.xml
M modules/org.openbravo.client.application/src/org/openbravo/client/application/window/AttachmentsAH.java
M src-db/database/model/tables/C_FILE.xml
M src-db/database/sourcedata/AD_COLUMN.xml
M src-db/database/sourcedata/AD_ELEMENT.xml
M src-db/database/sourcedata/AD_REF_LIST.xml
M src/org/openbravo/erpCommon/businessUtility/TabAttachments.java
M src/org/openbravo/erpCommon/businessUtility/TabAttachments_data.xsql
M src/org/openbravo/erpCommon/utility/reporting/ReportManager.java
A src/org/openbravo/erpCommon/utility/MigrateAttachments.java
---
(0059964)
shankarb (viewer)
2013-07-13 09:22

 Reopened to add review assigned to.
(0059966)
hgbot (developer)
2013-07-15 06:52

 Repository: erp/devel/pi
Changeset: 8362ae2c2b8f2ea9d8ceeb889bfb23bd7ed1fd3d
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Mon Jul 15 10:21:16 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/8362ae2c2b8f2ea9d8ceeb889bfb23bd7ed1fd3d [^]

Related to Issue 20623 : Fixed Javadoc issues

The case of the arguments in the javadoc where not matching with the method arguments which caused warnings in generate.java.doc.
Matched javadoc params with method parameters.

---
M src/org/openbravo/erpCommon/businessUtility/TabAttachments.java
---
(0059967)
hgbot (developer)
2013-07-15 08:53

 Repository: erp/devel/pi
Changeset: fd17b88f8379c2fef84dcad8e1090fb946834ba8
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Mon Jul 15 12:21:34 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/fd17b88f8379c2fef84dcad8e1090fb946834ba8 [^]

Related to issue 20623 : Fix api checks
An additional argument was added to the Insert method in TabAttachments_data that failed api tests.
Added an additional sql method for insert to maintain backward compatibility.

---
M src/org/openbravo/erpCommon/businessUtility/TabAttachments_data.xsql
---
(0060039)
hudsonbot (viewer)
2013-07-16 14:26

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060042)
hudsonbot (viewer)
2013-07-16 14:27

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060043)
hudsonbot (viewer)
2013-07-16 14:27

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060094)
alostale (viewer)
2013-07-18 13:53

 Reopening it because migration task is not working:

ant migrate.attachments
Buildfile: /home/asier/ws/pi-pg/openbravo/build.xml

set.code.rev:

code.rev:

init:

migrate.attachments:
     [java] Exception in thread "main" java.lang.NoSuchMethodError: main

BUILD FAILED
/home/asier/ws/pi-pg/openbravo/build.xml:375: Java returned: 1

Total time: 6 seconds
(0060095)
hgbot (developer)
2013-07-18 15:18

 Repository: erp/devel/pi
Changeset: 091b2c16deaf05748b09c64c6c70831ab432a478
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Thu Jul 18 18:46:02 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/091b2c16deaf05748b09c64c6c70831ab432a478 [^]

Fixes Issue 20623 : Modifications made to ant task

The ant task definition has been modified from java type to taskdef to enable execution without main method.

---
M build.xml
M src/org/openbravo/erpCommon/utility/MigrateAttachments.java
---
(0060096)
alostale (viewer)
2013-07-18 16:07

 public class AttachmentsTest extends BaseTest {
  public void testEx() throws ClientProtocolException, IOException {
    System.out.println("Starting...");
    long t = System.currentTimeMillis();
    for (int i = 0; i < 65000; i++) {
      if (i % 100 == 0) {
        System.out.println(" created " + i + " in " + (System.currentTimeMillis() - t) + " ms");
      }
      String fileDir = "/opt/openbravo/attachments/"
          + TabAttachments.getAttachmentDirectoryForNewAttachments("100", SequenceIdData.getUUID());

      File dir = new File(fileDir);
      dir.mkdirs();
      File f = new File(dir, "t");
      f.createNewFile();
    }
  }
}
(0060097)
alostale (viewer)
2013-07-18 16:09

 Code reviewed + tested.

Executed test:

-https://issues.openbravo.com/view.php?id=20623#c60096 [^] in linux ext4 file system, works fine creating 65000 attachmets
-having the backwards compatibility preference works fine and the attachments are created in the old way
-migration task works fine
(0060131)
hudsonbot (viewer)
2013-07-19 18:00

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/488f9bf0ef52 [^]

Maturity status: Test
(0060511)
alostale (viewer)
2013-08-06 11:34

 ext4 has no actual limit for number subdirectories: http://kernelnewbies.org/Ext4#head-97cbed179e6bcc48e47e645e06b95205ea832a68. [^]

In any case, this fix improves:

-ext3: supports for more than 32K attachments
-other file systems: improves directory management as not all attachements are saved in the same directory level

- Issue History
Date Modified Username Field Change
2012-05-28 12:37 alostale New Issue
2012-05-28 12:37 alostale Assigned To => alostale
2012-05-28 12:37 alostale Modules => Core
2012-05-28 12:37 alostale OBNetwork customer => No
2012-05-28 12:54 iperdomo Relationship added has duplicate 0012488
2012-05-28 13:35 alostale Proposed Solution updated
2012-05-28 13:36 alostale Proposed Solution updated
2012-09-24 12:47 AugustoMauch Note Added: 0052328
2012-09-24 12:47 AugustoMauch Priority normal => low
2012-09-24 12:47 AugustoMauch Status new => scheduled
2013-05-21 11:48 jonalegriaesarte Triggers an Emergency Pack => No
2013-05-21 11:48 jonalegriaesarte Target Version => 3.0MP25
2013-06-12 09:59 alostale Note Added: 0059378
2013-06-19 12:05 alostale Proposed Solution updated
2013-06-19 12:16 alostale Assigned To alostale => shankarb
2013-07-09 08:50 shankarb Note Added: 0059933
2013-07-13 09:11 shankarb Relationship added related to 0024289
2013-07-13 09:16 hgbot Checkin
2013-07-13 09:16 hgbot Note Added: 0059961
2013-07-13 09:16 hgbot Status scheduled => resolved
2013-07-13 09:16 hgbot Resolution open => fixed
2013-07-13 09:16 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^]
2013-07-13 09:22 shankarb Note Added: 0059964
2013-07-13 09:22 shankarb Status resolved => new
2013-07-13 09:22 shankarb Resolution fixed => open
2013-07-13 09:22 shankarb Review Assigned To => alostale
2013-07-13 09:22 shankarb Status new => scheduled
2013-07-13 09:22 shankarb fix_in_branch => pi
2013-07-13 09:22 shankarb Status scheduled => resolved
2013-07-13 09:22 shankarb Resolution open => fixed
2013-07-13 09:23 shankarb Issue Monitored: alostale
2013-07-15 06:52 hgbot Checkin
2013-07-15 06:52 hgbot Note Added: 0059966
2013-07-15 08:53 hgbot Checkin
2013-07-15 08:53 hgbot Note Added: 0059967
2013-07-16 14:26 hudsonbot Checkin
2013-07-16 14:26 hudsonbot Note Added: 0060039
2013-07-16 14:27 hudsonbot Checkin
2013-07-16 14:27 hudsonbot Note Added: 0060042
2013-07-16 14:27 hudsonbot Checkin
2013-07-16 14:27 hudsonbot Note Added: 0060043
2013-07-18 13:53 alostale Note Added: 0060094
2013-07-18 13:53 alostale Status resolved => new
2013-07-18 13:53 alostale Resolution fixed => open
2013-07-18 15:18 hgbot Checkin
2013-07-18 15:18 hgbot Note Added: 0060095
2013-07-18 15:18 hgbot Status new => resolved
2013-07-18 15:18 hgbot Resolution open => fixed
2013-07-18 15:18 hgbot Fixed in SCM revision http://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^] => http://code.openbravo.com/erp/devel/pi/rev/091b2c16deaf05748b09c64c6c70831ab432a478 [^]
2013-07-18 16:07 alostale Note Added: 0060096
2013-07-18 16:09 alostale Note Added: 0060097
2013-07-18 16:09 alostale Status resolved => closed
2013-07-18 16:09 alostale Fixed in Version => 3.0MP26
2013-07-18 16:27 alostale Relationship added related to 0024373
2013-07-19 18:00 hudsonbot Checkin
2013-07-19 18:00 hudsonbot Note Added: 0060131
2013-08-06 11:34 alostale Note Added: 0060511
2015-03-18 08:53 alostale Relationship added causes 0029313

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker