Symptom
1. Visible CPU usage permanent raise or or short spike (around 1h), correlated with Eden Space memory raise, and drop in Survivor Space memory.
2. Visible spikes in Backoffice's sessions and sometimes correlated with Storefront's sessions. O/I activity may have visible spikes too.
3. In Dynatrace among events you find deployments for backoffice's widget lib in unexpected (not planned) moments.
4. Slowly raising threads in ajp-nio
5. Threaddumps reveal BLOCKED threads AbstractApplicationContext.refresh
"ajp-nio-8010-exec-4" #230 daemon prio=5 os_prio=0 cpu=811049.80ms elapsed=233281.52s allocated=177G defined_classes=828 tid=0x000055bf3ba4d000 nid=0x1023 waiting for monitor entry [0x00007ff56cbc9000] java.lang.Thread.State: BLOCKED (on object monitor) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:518) - waiting to lock <0x000000064fbefc70> (a java.lang.Object) at com.hybris.backoffice.BackofficeApplicationContextInitializer.sessionCreated(BackofficeApplicationContextInitializer.java:54) at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:388) at org.apache.catalina.session.StandardSession.setId(StandardSession.java:360) at org.apache.catalina.session.StandardSession.setId(StandardSession.java:341) at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:656) at org.apache.catalina.connector.Request.doGetSession(Request.java:3087) at org.apache.catalina.connector.Request.getSession(Request.java:2476) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:896) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:240) at de.hybris.platform.servicelayer.web.SessionFilter.doFilter(SessionFilter.java:86) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at de.hybris.platform.servicelayer.web.session.HybrisSpringSessionFilter.doFilter(HybrisSpringSessionFilter.java:74) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at com.hybris.cockpitng.modules.spring.filter.ExternalModuleContextClassLoaderFilter.doFilter(ExternalModuleContextClassLoaderFilter.java:37) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at de.hybris.platform.servicelayer.web.RedirectWhenSystemIsNotInitializedFilter.doFilter(RedirectWhenSystemIsNotInitializedFilter.java:101) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at de.hybris.platform.servicelayer.web.TenantActivationFilter.doFilter(TenantActivationFilter.java:83) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at de.hybris.platform.servicelayer.web.Log4JFilter.doFilter(Log4JFilter.java:44) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at com.hybris.backoffice.filter.responseheaders.BackofficeResponseHeadersFilter.doFilter(BackofficeResponseHeadersFilter.java:31) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:299) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.processStandardFilterChain(AbstractPlatformFilterChain.java:207) at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.doFilterInternal(AbstractPlatformFilterChain.java:184) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at de.hybris.platform.servicelayer.web.XSSFilter.processPatternsAndDoFilter(XSSFilter.java:358) at de.hybris.platform.servicelayer.web.XSSFilter.doFilter(XSSFilter.java:306) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:476) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) - locked <0x000000064fbefab0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.8/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.8/ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(java.base@11.0.8/Thread.java:834)
Locked ownable synchronizers: - <0x0000000429ee44b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
|
Threads occupation confirmed in Dynatrace's stackraces
6. Errors similar to below may show up in logs showing arising problems with widget configuration loading.
ERROR [ajp-nio-8010-exec-247] [10.32.126.190] [DefaultCockpitWidgetEngine] Could not create widget view
java.lang.IllegalStateException: zip file closed
at java.util.zip.ZipFile.ensureOpen(ZipFile.java:915) ~[?:?]
java.util.zip.ZipFile.getEntry(ZipFile.java:347) ~[?:?]
java.util.zip.ZipFile$1.getEntry(ZipFile.java:1121) ~[?:?]
java.util.jar.JarFile.getEntry0(JarFile.java:578) ~[?:?]
java.util.jar.JarFile.getEntry(JarFile.java:508) ~[?:?]
java.util.jar.JarFile.getJarEntry(JarFile.java:470) ~[?:?]
com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.findAndCache(WidgetClassLoader.java:434) ~[cockpitcore-19.05.12-RC5.jar:?]
com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.findResourceInDir(WidgetClassLoader.java:493) ~[cockpitcore-19.05.12-RC5.jar:?]
com.hybris.cockpitng.core.persistence.packaging.WidgetClassLoader.getResourceAsStream(WidgetClassLoader.java:207) ~[cockpitcore-19.05.12-RC5.jar:?]
com.hybris.cockpitng.util.labels.CockpitComponentDefinitionLabelLocator.loadDefinitionLabels(CockpitComponentDefinitionLabelLocator.java:193) ~[cockpitframework-19.05.12-RC5.jar:?]
com.hybris.cockpitng.util.labels.CockpitComponentDefinitionLabelLocator.locate(CockpitComponentDefinitionLabelLocator.java:155) ~[cockpitframework-19.05.12-RC5.jar:?]
org.zkoss.util.resource.impl.LabelLoaderImpl.loadLabels(LabelLoaderImpl.java:277) ~[zcommon-8.6.0.1.jar:8.6.0.1]
org.zkoss.util.resource.impl.LabelLoaderImpl.getSegmentedLabels(LabelLoaderImpl.java:165) ~[zcommon-8.6.0.1.jar:8.6.0.1]
org.zkoss.util.resource.impl.LabelLoaderImpl.getSegmentedLabels(LabelLoaderImpl.java:155) ~[zcommon-8.6.0.1.jar:8.6.0.1]
org.zkoss.util.resource.Labels.getSegmentedLabels(Labels.java:124) ~[zcommon-8.6.0.1.jar:8.6.0.1]
|
Errors similar to below may start to show up frequently and regard various backoffice extensions.
ERROR [ajp-nio-8010-exec-247] [10.32.126.190] [WidgetClassLoader] error loading resource 'com/hybris/cockpitng/core/security/WidgetAuthorizationService.class' from jar 'couponbackoffice.jar'
java.nio.file.NoSuchFileException: /opt/hybris_PACKAGE/data/backoffice/widgetlib/deployed/couponbackoffice.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1763) ~[?:?]
|
Read more...
Environment
Any environment with backoffice application deployed on one or multiple nodes with significant number of user accessing it.
Product
SAP Commerce Cloud all versions
Keywords
cockpitng
backoffice
reset
login
start
scope
resourceloader
resource
cache
widgetclassloader
widget
config
NoSuchFileException
DefaultCockpitWidgetEngine
BackofficeApplicationContextInitializer
AbstractApplicationContext.refresh
TBD , KBA , CEC-SCC-CDM-BO , Backoffice & Product Content Management , Problem
About this page
This is a preview of a SAP Knowledge Base Article. Click
more to access the full version on SAP for Me (Login
required).
Search for additional results
Visit SAP Support Portal's SAP Notes and KBA Search.