Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0040211
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] A. Platformcriticalrandom2019-02-15 08:062019-11-12 15:48
ReporteralostaleView Statuspublic 
Assigned Toalostale 
PriorityurgentResolutionfixedFixed in Version3.0PR19Q2
StatusclosedFix in branchFixed in SCM revision5318eb8145c5
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned Tocaristu
Web browser
ModulesCore
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0040211: Servlets randomly become unavailable on 1st request

DescriptionIn environments with high concurrency, some Servlets can randomly become unlivable on their first request. When this occurs, the servlet cannot be recovered without restarting Tomcat, the effect is server returning HTTP 404 (unavailable) for this Servlet.
Steps To ReproduceIt cannot be easily reproduced, we've detected this problem in some customers with high concurrency occurring once a year.

When this happens this can be seen in localhost.log:

11-Feb-2019 23:08:24.695 INFO [ajp-nio-127.0.0.1-8009-exec-11] org.apache.catalina.core.ApplicationContext.log Marking servlet [P43ADA4F09EAD4034A8F9B1F4759407B5] as unavailable
11-Feb-2019 23:08:24.696 SEVERE [ajp-nio-127.0.0.1-8009-exec-11] org.apache.catalina.core.StandardWrapperValve.invoke Allocate exception for servlet [P43ADA4F09EAD4034A8F9B1F4759407B5]
 java.lang.NullPointerException
        at org.jboss.weld.environment.servlet.inject.AbstractInjector.inject(AbstractInjector.java:54)
        at org.jboss.weld.environment.tomcat7.WeldInstanceManager.newInstance(WeldInstanceManager.java:24)
        at org.jboss.weld.environment.tomcat7.WeldForwardingInstanceManager.newInstance(WeldForwardingInstanceManager.java:63)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1044)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:773)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:133)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:487)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1463)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:844)
Proposed SolutionThis problem seems to be caused by this Weld issue [1]. Backporting it to our current Weld version (1.1.24) is not feasible as it depends, at least, on some other refactors [2].

The proper solution would be to upgrade Weld 0040193.

Temporary workaround
This problem should not occur if we eagerly initialize all Servlets in a single thread on Tomcat start. I've done a quick test by manually editing web.xml adding load-on-start to all servlets to measure the impact: startup time increases 1-2 seconds and retained heap ~12MB, so I don't think this should be problematic.

Note 0040192 is a dependency for this workaround, as XmlEngine defined as Servlet throws an exception on Tomcat startup:

SEVERE: StandardWrapper.Throwable
java.lang.NullPointerException
    at java.base/java.io.File.<init>(File.java:276)
    at org.openbravo.xmlEngine.XmlEngine.init(XmlEngine.java:87)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5362)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
    at or40192g.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1700)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1690)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)

---
[1] https://github.com/weld/core/commit/e0ae6c40b4864674046edc1e3375eed973613c02 [^]
[2] https://github.com/weld/core/commit/325d5806dd567b04c00c51f5a036fa0f1e1c8552 [^]
TagsNo tags attached.
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]
related to feature request 0040193 closedcaristu upgrade Weld to version 3.1.0.Final 
depends on defect 0040192 closedalostale XmlEngine should not be a Servlet 

-  Notes
(0109888)
hgbot (developer)
2019-02-18 08:53

 Repository: erp/devel/pi
Changeset: 5318eb8145c5b04f82d59adc609d150e9b69c1fa
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Feb 15 09:11:12 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/5318eb8145c5b04f82d59adc609d150e9b69c1fa [^]

fixed bug 40211: Servlets randomly become unavailable on 1st request

  Implemented temporary workaround until Weld gets updated. Now all Servlets that
  do not specify any startup order, are eagerly loaded on server startup. This is
  done by a single thread preventing in this way race conditions in Wel's buggy
  code.

  It's possible to prevent concrete Servlets to be load on startup by setting to
  a negative number their load on startup field.

---
M src-wad/src/org/openbravo/wad/Wad.java
M src-wad/src/org/openbravo/wad/web.xml
---
(0110024)
caristu (developer)
2019-02-22 10:56

 Reviewed + tested
(0110820)
hudsonbot (developer)
2019-03-28 14:56

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test
(0110970)
hgbot (developer)
2019-04-02 18:06

 Repository: erp/devel/pi
Changeset: 211ad7c835e5e14e63568c737b975d4ea4edc3a4
Author: Carlos Aristu <carlos.aristu <at> openbravo.com>
Date: Thu Mar 07 18:47:52 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/211ad7c835e5e14e63568c737b975d4ea4edc3a4 [^]

[weld31] Revert change for bug 40211

  Fix for issue[1] was a temporary solution due to an issue in weld which is already solved in version 3.1.0. Therefore now the fix is reverted.

  [1] https://issues.openbravo.com/view.php?id=40211 [^]
  [2] https://github.com/weld/core/commit/e0ae6c40b4864674046edc1e3375eed973613c02 [^]

---
M src-wad/src/org/openbravo/wad/Wad.java
M src-wad/src/org/openbravo/wad/web.xml
---
(0111957)
hudsonbot (developer)
2019-05-21 15:16

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9b8f37d9d85e [^]
Maturity status: Test

- Issue History
Date Modified Username Field Change
2019-02-15 08:06 alostale New Issue
2019-02-15 08:06 alostale Assigned To => alostale
2019-02-15 08:06 alostale Modules => Core
2019-02-15 08:06 alostale Triggers an Emergency Pack => No
2019-02-15 08:07 alostale Relationship added related to 0040193
2019-02-15 08:17 alostale Proposed Solution updated
2019-02-15 08:17 alostale Relationship added depends on 0040192
2019-02-18 08:53 hgbot Checkin
2019-02-18 08:53 hgbot Note Added: 0109888
2019-02-18 08:53 hgbot Status new => resolved
2019-02-18 08:53 hgbot Resolution open => fixed
2019-02-18 08:53 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/5318eb8145c5b04f82d59adc609d150e9b69c1fa [^]
2019-02-18 09:00 alostale Review Assigned To => caristu
2019-02-22 10:56 caristu Note Added: 0110024
2019-02-22 10:56 caristu Status resolved => closed
2019-02-22 10:56 caristu Fixed in Version => 3.0PR19Q2
2019-03-28 14:56 hudsonbot Checkin
2019-03-28 14:56 hudsonbot Note Added: 0110820
2019-04-02 18:06 hgbot Checkin
2019-04-02 18:06 hgbot Note Added: 0110970
2019-05-21 15:16 hudsonbot Checkin
2019-05-21 15:16 hudsonbot Note Added: 0111957
2019-11-12 15:48 alostale Severity major => critical

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker