2555893 - AS Java slow startup - Deploy Parallel Start Threads

AS Java startup is hanging at "starting apps" phase and takes much time to start (sometimes more than an hour).

In dev_server you can see the startup process:

F [Thr 140736938428160] Sun Sep 10 15:26:17 2017
F [Thr 140736938428160] *** LOG => State changed from 0 (Initial) to 1 (Waiting to start).
...
F [Thr 140736938428160] *** LOG => State changed from 1 (Waiting to start) to 2 (Starting framework).
...
F [Thr 140736938428160] Sun Sep 10 15:29:45 2017
F [Thr 140736938428160] *** LOG => State changed from 2 (Starting framework) to 10 (Starting apps).
...
F [Thr 140729733011200] Sun Sep 10 16:05:13 2017
F [Thr 140729733011200] *** LOG => State changed from 10 (Starting apps) to 3 (Running).

You execute thread dumps according to SAP Note 1095473. Then you analyze these thread dumps in Eclipse Memory Analyzer tool according to the following SAP document: Analyzing Thread Dumps.

Thread_Dump_Analsysis under "Affected Threads' Merged Activity Stacks" shows that com.sap.engine.services.deploy service is responsible for this scenario. See examples:

"com.sap.engine.services.servlets_jsp.server.deploy.ApplicationThreadInitializer.processTldsInDir"
"com.sap.engine.services.deploy.server.prl.ParallelApplicationStartTraverser$StartApplicationPriviligedAction"

Moreover under "Lock Tree of Affected Threads" you can see "Deploy Parallel Start Threads":

If you analyze thread dumps manually, in server log files (e.g. dev_server or std_server) you can see many similar threads "locked" and/or "waiting to lock" like this:

"Deploy Parallel Start Thread 5" #1901 prio=5 os_prio=0 cpu=328.13 [reset 328.13] ms elapsed=9.58 [reset 9.58] s allocated=23659176 B (22.56 MB) [reset 23659176 B (22.56 MB)] defined_classes=7
io= file i/o: 17849/0 B, net i/o: 779197/250842 B, files opened:2, socks opened:0  [reset file i/o: 17849/0 B, net i/o: 779197/250842 B, files opened:2, socks opened:0 ]
user="deploy_service" application="tc?bl?txmanager?plb" isapplicationthread="true" tid=0x00000015c79ba800 nid=0x5d74 / 23924 runnable  [_thread_in_native (_at_safepoint), stack(0x00000015dcc50000,0x00000015dce50000)] [0x00000015dce4b000]
   java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(Native Method)
 at java.net.SocketInputStream.read([BII)I(SocketInputStream.java:129)
...
 at com.microsoft.sqlserver.jdbc.TDSChannel.read([BII)I(IOBuffer.java:1782)
 at com.microsoft.sqlserver.jdbc.TDSReader.readPacket()Z(IOBuffer.java:4838)
 - locked <0x00000000d86fd730> (a com.microsoft.sqlserver.jdbc.TDSReader)
...
 - locked <0x000000008da266b8> (a java.lang.Object)
...
 - locked <0x0000000085752058> (a com.sap.security.core.imp.User)
...
 - locked <0x000000008e43da20> (a com.sap.jms.client.session.JMSQueueSession)
...

"Deploy Parallel Start Thread 2" #2496 prio=5 os_prio=0 cpu=66030.00 [reset 65880.00] ms elapsed=1032.61 [reset 953.28] s allocated=2249796576 B (2.10 GB) [reset 2239208952 B (2.09 GB)] defined_classes=4417
io= file i/o: 16283317/16916 B, net i/o: 7899564/6495680 B, files opened:1428, socks opened:2  [reset file i/o: 16204997/16916 B, net i/o: 7845612/6459365 B, files opened:1424, socks opened:2 ]
user="deploy_service" isapplicationthread="true" tid=0x00007ffc74010000 nid=0x9636 / 38454  pthread-id=140724011980544 waiting for monitor entry  [_thread_blocked (_at_safepoint), stack(0x00007ffcdc9ed000,0x00007ffcdcbee000)] [0x00007ffcdcbea000] top_bci: 0
   java.lang.Thread.State: BLOCKED (on object monitor)
 at com.sap.engine.core.classload.impl0.LoadContextImpl.createClassLoader([Ljava/lang/ClassLoader;[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;I)Ljava/lang/ClassLoader;(LoadContextImpl.java:1002)
 - waiting to lock <0x00000007464a98f8> (a com.sap.engine.core.classload.impl0.LoadContextImpl)
...

SAP NetWeaver 7.1
SAP NetWeaver 7.2
SAP NetWeaver 7.3
SAP NetWeaver 7.3 EHP1
SAP NetWeaver 7.4
SAP NetWeaver 7.5

Deploy Parallel start thread, Deploy Parallel start threads, Deploy Parallel Start Thread 5, waiting to lock, locked, Initial_Application_Start_Threads, config tool, configtool, deploy service, Lock Tree of Affected Threads, locks, enqueue locks, com.sap.engine.services.deploy, com.sap.engine.services.deploy service, AS Java slow startup, AS java slow, AS Java hangs, AS Java is hanging, starting apps, deploy.ApplicationThreadInitializer, ParallelApplicationStartTraverser, deadlock, deadlocks, dangling locks, dangling lock, Thread_Dump_Analsysis, threaddump, thread, thread dump, thread dumps, thread, Affected Threads' Merged Activity Stacks , KBA , BC-JAS-DPL , Deployment , Problem