SAP Knowledge Base Article - Preview

3022737 - Backoffice not accessible with Internal Server Error on one node with blocked threads on AbstractApplicationContext.refresh

Symptom

1. Visible CPU usage permanent raise or or short spike (around 1h), correlated with Eden Space memory raise, and drop in Survivor Space memory.

2. Visible spikes in Backoffice's sessions and sometimes correlated with Storefront's sessions. O/I activity may have visible spikes too.

3. In Dynatrace among events you find deployments for backoffice's widget lib in unexpected (not planned) moments.

4. Slowly raising threads in ajp-nio

5. Threaddumps reveal BLOCKED threads AbstractApplicationContext.refresh

"ajp-nio-8010-exec-4" #230 daemon prio=5 os_prio=0 cpu=811049.80ms elapsed=233281.52s allocated=177G defined_classes=828 tid=0x000055bf3ba4d000 nid=0x1023 waiting for monitor entry  [0x00007ff56cbc9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:518)
 - waiting to lock <0x000000064fbefc70> (a java.lang.Object)
 at com.hybris.backoffice.BackofficeApplicationContextInitializer.sessionCreated(BackofficeApplicationContextInitializer.java:54)
 at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:388)
 at org.apache.catalina.session.StandardSession.setId(StandardSession.java:360)
 at org.apache.catalina.session.StandardSession.setId(StandardSession.java:341)
 at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:656)
 at org.apache.catalina.connector.Request.doGetSession(Request.java:3087)
 at org.apache.catalina.connector.Request.getSession(Request.java:2476)
 at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:896)
 at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
 at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:240)
 at de.hybris.platform.servicelayer.web.SessionFilter.doFilter(SessionFilter.java:86)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at de.hybris.platform.servicelayer.web.session.HybrisSpringSessionFilter.doFilter(HybrisSpringSessionFilter.java:74)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at com.hybris.cockpitng.modules.spring.filter.ExternalModuleContextClassLoaderFilter.doFilter(ExternalModuleContextClassLoaderFilter.java:37)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at de.hybris.platform.servicelayer.web.RedirectWhenSystemIsNotInitializedFilter.doFilter(RedirectWhenSystemIsNotInitializedFilter.java:101)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at de.hybris.platform.servicelayer.web.TenantActivationFilter.doFilter(TenantActivationFilter.java:83)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at de.hybris.platform.servicelayer.web.Log4JFilter.doFilter(Log4JFilter.java:44)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at com.hybris.backoffice.filter.responseheaders.BackofficeResponseHeadersFilter.doFilter(BackofficeResponseHeadersFilter.java:31)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.processStandardFilterChain(AbstractPlatformFilterChain.java:207)
 at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.doFilterInternal(AbstractPlatformFilterChain.java:184)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at de.hybris.platform.servicelayer.web.XSSFilter.processPatternsAndDoFilter(XSSFilter.java:358)
 at de.hybris.platform.servicelayer.web.XSSFilter.doFilter(XSSFilter.java:306)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:476)
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 - locked <0x000000064fbefab0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.8/ThreadPoolExecutor.java:1128)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.8/ThreadPoolExecutor.java:628)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(java.base@11.0.8/Thread.java:834)

   Locked ownable synchronizers:
 - <0x0000000429ee44b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Threads occupation confirmed in Dynatrace's stackraces 


6. Errors similar to below may show up in logs showing arising problems with widget configuration loading.

ERROR [ajp-nio-8010-exec-247] [10.32.126.190] [DefaultCockpitWidgetEngine]  Could not create widget view

 java.lang.IllegalStateException: zip file closed

 at java.util.zip.ZipFile.ensureOpen(ZipFile.java:915) ~[?:?]

 java.util.zip.ZipFile.getEntry(ZipFile.java:347) ~[?:?]

 java.util.zip.ZipFile$1.getEntry(ZipFile.java:1121) ~[?:?]

 java.util.jar.JarFile.getEntry0(JarFile.java:578) ~[?:?]

 java.util.jar.JarFile.getEntry(JarFile.java:508) ~[?:?]

 java.util.jar.JarFile.getJarEntry(JarFile.java:470) ~[?:?]

 com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.findAndCache(WidgetClassLoader.java:434) ~[cockpitcore-19.05.12-RC5.jar:?]

 com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.findResourceInDir(WidgetClassLoader.java:493) ~[cockpitcore-19.05.12-RC5.jar:?]

 com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.getResourceAsStream(WidgetClassLoader.java:207) ~[cockpitcore-19.05.12-RC5.jar:?]

 com.hybris.cockpitng.util.labels.CockpitComponentDefinitionLabelLocator.loadDefinitionLabels(CockpitComponentDefinitionLabelLocator.java:193) ~[cockpitframework-19.05.12-RC5.jar:?]

 com.hybris.cockpitng.util.labels.CockpitComponentDefinitionLabelLocator.locate(CockpitComponentDefinitionLabelLocator.java:155) ~[cockpitframework-19.05.12-RC5.jar:?]

 org.zkoss.util.resource.impl.LabelLoaderImpl.loadLabels(LabelLoaderImpl.java:277) ~[zcommon-8.6.0.1.jar:8.6.0.1]

 org.zkoss.util.resource.impl.LabelLoaderImpl.getSegmentedLabels(LabelLoaderImpl.java:165) ~[zcommon-8.6.0.1.jar:8.6.0.1]

 org.zkoss.util.resource.impl.LabelLoaderImpl.getSegmentedLabels(LabelLoaderImpl.java:155) ~[zcommon-8.6.0.1.jar:8.6.0.1]

 org.zkoss.util.resource.Labels.getSegmentedLabels(Labels.java:124) ~[zcommon-8.6.0.1.jar:8.6.0.1]

 
Errors similar to below may start to show up frequently and regard various backoffice extensions. 

ERROR [ajp-nio-8010-exec-247] [10.32.126.190] [WidgetClassLoader]  error loading resource 'com/hybris/cockpitng/core/security/WidgetAuthorizationService.class' from jar 'couponbackoffice.jar'

 java.nio.file.NoSuchFileException: /opt/hybris_PACKAGE/data/backoffice/widgetlib/deployed/couponbackoffice.jar

  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]

  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]

  at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]

  at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]

  at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]

  at java.nio.file.Files.readAttributes(Files.java:1763) ~[?:?]


Read more...

Environment

Any environment with backoffice application deployed on one or multiple nodes with significant number of user accessing it.

Keywords

cockpitng backoffice reset login start scope resourceloader resource cache widgetclassloader widget config NoSuchFileException DefaultCockpitWidgetEngine BackofficeApplicationContextInitializer AbstractApplicationContext.refresh TBD , KBA , CEC-COM-ADM-BO , Backoffice , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP for Me (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.