Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0020623Openbravo ERPA. Platformpublic2012-05-28 12:372013-08-06 11:34
Reporteralostale 
Assigned Toshankarb 
PrioritylowSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target Version3.0MP25Fixed in Version3.0MP26 
Merge Request Status
Review Assigned Toalostale
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0020623: 32K limit for attachments
DescriptionIn a linux with ext3 file system it is not possible to create attachments for more than 32K records.

This problem is caused by 2 facts:
*File system limitation: ext3 allows to create up to 32K subdirectories within a directory.
*The way OB manages attachments: it creates a subdirectory within attachments directory for each record with attachments.
Steps To Reproduce-Try to created more than attachments for more than 32K records.
Proposed SolutionCurrent structure is:

- Table1-record1
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
- Table1-record2
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
- Table2-record1
  |-- Attachment 1
  |-- Attachment 2
  |-- ...
-...

Proposed solution is to split subdirectories in tables + have a limit of 32K within each of them:

*Alternative 1:
Use the same approach squid does: http://serverfault.com/a/46386 [^]

It is the same approach as proposed in issue 0012488

Splitting file name based on its record UUID.

If this alternative is chosen, it would require to decide how the actual split would be done.

*Alternative 2:

- Table1
  |-- Group 1
  | |-- Record 1
  | | |-- Attachment 1
  | | |-- ...
  | |-- Record 2
  | | |-- Attachment 1
  | | |-- ...
  | |-- ...
  | |-- Record 31999
  |-- Group 2
  | |-- Record 1
  | | |-- Attachment 1
  | | |-- ...
  | |-- Record 2
  | | |-- Attachment 1
  | | |-- ...
  | |-- ...
  | |-- Record 31999
  |--...
- Table 2
- ...

This way it would support attachments for 1.024M (32^2K) records on each table, 32K attachment for record.

Note that currently, c_file stores just name of attachment without path as it is calculated based on table + record id.

To maintain backwards compatibility:
-If implementing solution 1. Add a new boolean to c_file, this is defaulted to N. When N the old approach is used, for new attachments value is Y and file name is splitted.
-If implementing solution 2. A new "path" field should be added to c_file. This path is relative to attachments directory and only would be used for new attachements. Old attachments (identified by this field being null) would still work in the old way.
-Prepare a migration tool. This tool to be optionally executed would move attachments from existent structure to new one.
-Create a new system level preference (by default not set), when set, save attachments in old way

**Regression risk**
New attachments would follow a different approach to determine their location. This would make to fail modules trying to read new attachments in the old way.
Additional Information
TagsNo tags attached.
Relationships
has duplicate design defect 0012488 closed iperdomo When there are attachemnts for more than 31998 documents it is not possible to create new ones 
related to design defect 0024289 closed shankarb Retrieving attachments API Change 
related to defect 0024373 closed shankarb api-check failure at build 4114 
causes defect 00293133.0PR15Q2 closed alostale Download attachments selecting several records is not working fine 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2012-05-28 12:37alostaleNew Issue
2012-05-28 12:37alostaleAssigned To => alostale
2012-05-28 12:37alostaleModules => Core
2012-05-28 12:54iperdomoRelationship addedhas duplicate 0012488
2012-05-28 13:35alostaleProposed Solution updated
2012-05-28 13:36alostaleProposed Solution updated
2012-09-24 12:47AugustoMauchNote Added: 0052328
2012-09-24 12:47AugustoMauchPrioritynormal => low
2012-09-24 12:47AugustoMauchStatusnew => scheduled
2013-05-21 11:48jonalegriaesarteTriggers an Emergency Pack => No
2013-05-21 11:48jonalegriaesarteTarget Version => 3.0MP25
2013-06-12 09:59alostaleNote Added: 0059378
2013-06-19 12:05alostaleProposed Solution updated
2013-06-19 12:16alostaleAssigned Toalostale => shankarb
2013-07-09 08:50shankarbNote Added: 0059933
2013-07-13 09:11shankarbRelationship addedrelated to 0024289
2013-07-13 09:16hgbotCheckin
2013-07-13 09:16hgbotNote Added: 0059961
2013-07-13 09:16hgbotStatusscheduled => resolved
2013-07-13 09:16hgbotResolutionopen => fixed
2013-07-13 09:16hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^]
2013-07-13 09:22shankarbNote Added: 0059964
2013-07-13 09:22shankarbStatusresolved => new
2013-07-13 09:22shankarbResolutionfixed => open
2013-07-13 09:22shankarbReview Assigned To => alostale
2013-07-13 09:22shankarbStatusnew => scheduled
2013-07-13 09:22shankarbfix_in_branch => pi
2013-07-13 09:22shankarbStatusscheduled => resolved
2013-07-13 09:22shankarbResolutionopen => fixed
2013-07-13 09:23shankarbIssue Monitored: alostale
2013-07-15 06:52hgbotCheckin
2013-07-15 06:52hgbotNote Added: 0059966
2013-07-15 08:53hgbotCheckin
2013-07-15 08:53hgbotNote Added: 0059967
2013-07-16 14:26hudsonbotCheckin
2013-07-16 14:26hudsonbotNote Added: 0060039
2013-07-16 14:27hudsonbotCheckin
2013-07-16 14:27hudsonbotNote Added: 0060042
2013-07-16 14:27hudsonbotCheckin
2013-07-16 14:27hudsonbotNote Added: 0060043
2013-07-18 13:53alostaleNote Added: 0060094
2013-07-18 13:53alostaleStatusresolved => new
2013-07-18 13:53alostaleResolutionfixed => open
2013-07-18 15:18hgbotCheckin
2013-07-18 15:18hgbotNote Added: 0060095
2013-07-18 15:18hgbotStatusnew => resolved
2013-07-18 15:18hgbotResolutionopen => fixed
2013-07-18 15:18hgbotFixed in SCM revisionhttp://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^] => http://code.openbravo.com/erp/devel/pi/rev/091b2c16deaf05748b09c64c6c70831ab432a478 [^]
2013-07-18 16:07alostaleNote Added: 0060096
2013-07-18 16:09alostaleNote Added: 0060097
2013-07-18 16:09alostaleStatusresolved => closed
2013-07-18 16:09alostaleFixed in Version => 3.0MP26
2013-07-18 16:27alostaleRelationship addedrelated to 0024373
2013-07-19 18:00hudsonbotCheckin
2013-07-19 18:00hudsonbotNote Added: 0060131
2013-08-06 11:34alostaleNote Added: 0060511
2015-03-18 08:53alostaleRelationship addedcauses 0029313

Notes
(0052328)
AugustoMauch   
2012-09-24 12:47   
Effort: 5
Impact: low
Plan: long
(0059378)
alostale   
2013-06-12 09:59   
32K subdirectory limitation in ext3 file system is doubled to 64K in ext4
(0059933)
shankarb   
2013-07-09 08:50   
Alternative 2 is going to be used to fix this issue.
(0059961)
hgbot   
2013-07-13 09:16   
Repository: erp/devel/pi
Changeset: bed326b6ddf6f08b10cf22cd7403aff84c0e3750
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Thu Jul 11 11:00:28 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/bed326b6ddf6f08b10cf22cd7403aff84c0e3750 [^]

Fixes Issue 0020623, Fixes Issue 24289: 32K limit for attachments

The attachment model has been changed as explained in the wiki[1].
A new ant task has been added to migrate attachments from old model to new model.

[1] http://wiki.openbravo.com/wiki/Attachments [^]

---
M build.xml
M modules/org.openbravo.client.application/src/org/openbravo/client/application/window/AttachmentsAH.java
M src-db/database/model/tables/C_FILE.xml
M src-db/database/sourcedata/AD_COLUMN.xml
M src-db/database/sourcedata/AD_ELEMENT.xml
M src-db/database/sourcedata/AD_REF_LIST.xml
M src/org/openbravo/erpCommon/businessUtility/TabAttachments.java
M src/org/openbravo/erpCommon/businessUtility/TabAttachments_data.xsql
M src/org/openbravo/erpCommon/utility/reporting/ReportManager.java
A src/org/openbravo/erpCommon/utility/MigrateAttachments.java
---
(0059964)
shankarb   
2013-07-13 09:22   
Reopened to add review assigned to.
(0059966)
hgbot   
2013-07-15 06:52   
Repository: erp/devel/pi
Changeset: 8362ae2c2b8f2ea9d8ceeb889bfb23bd7ed1fd3d
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Mon Jul 15 10:21:16 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/8362ae2c2b8f2ea9d8ceeb889bfb23bd7ed1fd3d [^]

Related to Issue 20623 : Fixed Javadoc issues

The case of the arguments in the javadoc where not matching with the method arguments which caused warnings in generate.java.doc.
Matched javadoc params with method parameters.

---
M src/org/openbravo/erpCommon/businessUtility/TabAttachments.java
---
(0059967)
hgbot   
2013-07-15 08:53   
Repository: erp/devel/pi
Changeset: fd17b88f8379c2fef84dcad8e1090fb946834ba8
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Mon Jul 15 12:21:34 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/fd17b88f8379c2fef84dcad8e1090fb946834ba8 [^]

Related to issue 20623 : Fix api checks
An additional argument was added to the Insert method in TabAttachments_data that failed api tests.
Added an additional sql method for insert to maintain backward compatibility.

---
M src/org/openbravo/erpCommon/businessUtility/TabAttachments_data.xsql
---
(0060039)
hudsonbot   
2013-07-16 14:26   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060042)
hudsonbot   
2013-07-16 14:27   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060043)
hudsonbot   
2013-07-16 14:27   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9a5d5983399f [^]

Maturity status: Test
(0060094)
alostale   
2013-07-18 13:53   
Reopening it because migration task is not working:

ant migrate.attachments
Buildfile: /home/asier/ws/pi-pg/openbravo/build.xml

set.code.rev:

code.rev:

init:

migrate.attachments:
     [java] Exception in thread "main" java.lang.NoSuchMethodError: main

BUILD FAILED
/home/asier/ws/pi-pg/openbravo/build.xml:375: Java returned: 1

Total time: 6 seconds
(0060095)
hgbot   
2013-07-18 15:18   
Repository: erp/devel/pi
Changeset: 091b2c16deaf05748b09c64c6c70831ab432a478
Author: Shankar Balachandran <shankar.balachandran <at> openbravo.com>
Date: Thu Jul 18 18:46:02 2013 +0530
URL: http://code.openbravo.com/erp/devel/pi/rev/091b2c16deaf05748b09c64c6c70831ab432a478 [^]

Fixes Issue 20623 : Modifications made to ant task

The ant task definition has been modified from java type to taskdef to enable execution without main method.

---
M build.xml
M src/org/openbravo/erpCommon/utility/MigrateAttachments.java
---
(0060096)
alostale   
2013-07-18 16:07   
public class AttachmentsTest extends BaseTest {
  public void testEx() throws ClientProtocolException, IOException {
    System.out.println("Starting...");
    long t = System.currentTimeMillis();
    for (int i = 0; i < 65000; i++) {
      if (i % 100 == 0) {
        System.out.println(" created " + i + " in " + (System.currentTimeMillis() - t) + " ms");
      }
      String fileDir = "/opt/openbravo/attachments/"
          + TabAttachments.getAttachmentDirectoryForNewAttachments("100", SequenceIdData.getUUID());

      File dir = new File(fileDir);
      dir.mkdirs();
      File f = new File(dir, "t");
      f.createNewFile();
    }
  }
}
(0060097)
alostale   
2013-07-18 16:09   
Code reviewed + tested.

Executed test:

-https://issues.openbravo.com/view.php?id=20623#c60096 [^] in linux ext4 file system, works fine creating 65000 attachmets
-having the backwards compatibility preference works fine and the attachments are created in the old way
-migration task works fine
(0060131)
hudsonbot   
2013-07-19 18:00   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/488f9bf0ef52 [^]

Maturity status: Test
(0060511)
alostale   
2013-08-06 11:34   
ext4 has no actual limit for number subdirectories: http://kernelnewbies.org/Ext4#head-97cbed179e6bcc48e47e645e06b95205ea832a68. [^]

In any case, this fix improves:

-ext3: supports for more than 32K attachments
-other file systems: improves directory management as not all attachements are saved in the same directory level