Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0040211Openbravo ERPA. Platformpublic2019-02-15 08:062019-11-12 15:48
Reporteralostale 
Assigned Toalostale 
PriorityurgentSeveritycriticalReproducibilityrandom
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in Version3.0PR19Q2 
Merge Request Status
Review Assigned Tocaristu
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0040211: Servlets randomly become unavailable on 1st request
DescriptionIn environments with high concurrency, some Servlets can randomly become unlivable on their first request. When this occurs, the servlet cannot be recovered without restarting Tomcat, the effect is server returning HTTP 404 (unavailable) for this Servlet.
Steps To ReproduceIt cannot be easily reproduced, we've detected this problem in some customers with high concurrency occurring once a year.

When this happens this can be seen in localhost.log:

11-Feb-2019 23:08:24.695 INFO [ajp-nio-127.0.0.1-8009-exec-11] org.apache.catalina.core.ApplicationContext.log Marking servlet [P43ADA4F09EAD4034A8F9B1F4759407B5] as unavailable
11-Feb-2019 23:08:24.696 SEVERE [ajp-nio-127.0.0.1-8009-exec-11] org.apache.catalina.core.StandardWrapperValve.invoke Allocate exception for servlet [P43ADA4F09EAD4034A8F9B1F4759407B5]
 java.lang.NullPointerException
        at org.jboss.weld.environment.servlet.inject.AbstractInjector.inject(AbstractInjector.java:54)
        at org.jboss.weld.environment.tomcat7.WeldInstanceManager.newInstance(WeldInstanceManager.java:24)
        at org.jboss.weld.environment.tomcat7.WeldForwardingInstanceManager.newInstance(WeldForwardingInstanceManager.java:63)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1044)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:773)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:133)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:487)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1463)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:844)
Proposed SolutionThis problem seems to be caused by this Weld issue [1]. Backporting it to our current Weld version (1.1.24) is not feasible as it depends, at least, on some other refactors [2].

The proper solution would be to upgrade Weld 0040193.

Temporary workaround
This problem should not occur if we eagerly initialize all Servlets in a single thread on Tomcat start. I've done a quick test by manually editing web.xml adding load-on-start to all servlets to measure the impact: startup time increases 1-2 seconds and retained heap ~12MB, so I don't think this should be problematic.

Note 0040192 is a dependency for this workaround, as XmlEngine defined as Servlet throws an exception on Tomcat startup:

SEVERE: StandardWrapper.Throwable
java.lang.NullPointerException
    at java.base/java.io.File.<init>(File.java:276)
    at org.openbravo.xmlEngine.XmlEngine.init(XmlEngine.java:87)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5362)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
    at or40192g.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1700)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1690)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)

---
[1] https://github.com/weld/core/commit/e0ae6c40b4864674046edc1e3375eed973613c02 [^]
[2] https://github.com/weld/core/commit/325d5806dd567b04c00c51f5a036fa0f1e1c8552 [^]
Additional Information
TagsNo tags attached.
Relationships
related to feature request 0040193 closed caristu upgrade Weld to version 3.1.0.Final 
depends on defect 0040192 closed alostale XmlEngine should not be a Servlet 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2019-02-15 08:06alostaleNew Issue
2019-02-15 08:06alostaleAssigned To => alostale
2019-02-15 08:06alostaleModules => Core
2019-02-15 08:06alostaleTriggers an Emergency Pack => No
2019-02-15 08:07alostaleRelationship addedrelated to 0040193
2019-02-15 08:17alostaleProposed Solution updated
2019-02-15 08:17alostaleRelationship addeddepends on 0040192
2019-02-18 08:53hgbotCheckin
2019-02-18 08:53hgbotNote Added: 0109888
2019-02-18 08:53hgbotStatusnew => resolved
2019-02-18 08:53hgbotResolutionopen => fixed
2019-02-18 08:53hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/5318eb8145c5b04f82d59adc609d150e9b69c1fa [^]
2019-02-18 09:00alostaleReview Assigned To => caristu
2019-02-22 10:56caristuNote Added: 0110024
2019-02-22 10:56caristuStatusresolved => closed
2019-02-22 10:56caristuFixed in Version => 3.0PR19Q2
2019-03-28 14:56hudsonbotCheckin
2019-03-28 14:56hudsonbotNote Added: 0110820
2019-04-02 18:06hgbotCheckin
2019-04-02 18:06hgbotNote Added: 0110970
2019-05-21 15:16hudsonbotCheckin
2019-05-21 15:16hudsonbotNote Added: 0111957
2019-11-12 15:48alostaleSeveritymajor => critical

Notes
(0109888)
hgbot   
2019-02-18 08:53   
Repository: erp/devel/pi
Changeset: 5318eb8145c5b04f82d59adc609d150e9b69c1fa
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Feb 15 09:11:12 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/5318eb8145c5b04f82d59adc609d150e9b69c1fa [^]

fixed bug 40211: Servlets randomly become unavailable on 1st request

  Implemented temporary workaround until Weld gets updated. Now all Servlets that
  do not specify any startup order, are eagerly loaded on server startup. This is
  done by a single thread preventing in this way race conditions in Wel's buggy
  code.

  It's possible to prevent concrete Servlets to be load on startup by setting to
  a negative number their load on startup field.

---
M src-wad/src/org/openbravo/wad/Wad.java
M src-wad/src/org/openbravo/wad/web.xml
---
(0110024)
caristu   
2019-02-22 10:56   
Reviewed + tested
(0110820)
hudsonbot   
2019-03-28 14:56   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test
(0110970)
hgbot   
2019-04-02 18:06   
Repository: erp/devel/pi
Changeset: 211ad7c835e5e14e63568c737b975d4ea4edc3a4
Author: Carlos Aristu <carlos.aristu <at> openbravo.com>
Date: Thu Mar 07 18:47:52 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/211ad7c835e5e14e63568c737b975d4ea4edc3a4 [^]

[weld31] Revert change for bug 40211

  Fix for issue[1] was a temporary solution due to an issue in weld which is already solved in version 3.1.0. Therefore now the fix is reverted.

  [1] https://issues.openbravo.com/view.php?id=40211 [^]
  [2] https://github.com/weld/core/commit/e0ae6c40b4864674046edc1e3375eed973613c02 [^]

---
M src-wad/src/org/openbravo/wad/Wad.java
M src-wad/src/org/openbravo/wad/web.xml
---
(0111957)
hudsonbot   
2019-05-21 15:16   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9b8f37d9d85e [^]
Maturity status: Test