2754662 - SUM upgrade/Installation of S/4HANA fails due to Enqueue problem

1. Following error occurs during SUM upgrade of S/4HANA 1709 to 1809 in the phase MAIN_NEWBAS/JOB_RSI18N_UPGUCLANG:

Directory: /usr/sap/<SID>/SUM/abap/log
Name:UPGUCLANG.log

1 ETQ201 Entering upgrade-phase "MAIN_NEWBAS/JOB_RSI18N_UPGUCLANG" ("20190116165515")
……..
1 ETQ359 RFC Login to: System="PC2", AsHost="<hostname>" Nr="01", Client="<number>", GwHost="<hostname>", GwService="<gateway service>"
…
2 ETQ373 parameter "BATCHINSTANCE" = "<instance_name>"
1EETQ235 Call of function module "SUBST_START_BATCHJOB" by RFC failed (error-status "5")
2EETQ360 RFC of "SUBST_START_BATCHJOB" failed:
……
1EETQ399 Last error code set is: RFC system PC2 nr 01 function 'SUBST_START_BATCHJOB'
1EETQ399XRFC failed with code 5 key BATCH_SCHEDULING_FAILED:
1EETQ399XID:BT Type:E Number:389 RSI18N_UPGUCLANG
1EETQ203 Upgrade phase "JOB_RSI18N_UPGUCLANG" aborted with errors ("20190116165536")

(Here <instance_name> is the server where job is executed)

Following errors can be seen when you logon to the server used in the above phase:

SM12==>Extras==>Diagnosis

Lock management in a standalone server
Lock management operation mode
HOST:<blank>
INST: <instance_number>

Error: System error during lock operation
Error: Unexpected error occurred in lock operation. SY-SUBRC = 8

SM21:

31.01.2019 15:45:02 <instance_name> DIA <work process number> 000 DDIC @5C\Qvery high priority@ GEH Enqueue: Transfer error while reading lock entries
31.01.2019 15:45:02 <instance_name> DIA <work process number> 000 DDIC @5C\Qvery high priority@ GEH Enqueue: Transfer error while reading lock entries
31.01.2019 15:45:02 <instance_name> DIA <work process number> 000 DDIC @SR\Qinformation@ GZZ > ENQUE_READ exception code: 8

Checking work process trace dev_w<work process number> in the work directory:

E Thu Jan 31 15:45:02:848 2019
E *** ERROR => ENSA_Connect(): AlHARegisterComponent() failed with error 254 [ensaclnt.c 835]
E {root-id=005056B91FC91EE989A821ADEA7C02CB}_{conn-id=00000000000000000000000000000000}_0
E *** ERROR => EncCliCon:hostname unknown [enccli.c 299]
E *** ERROR => EnsaCliICreateHandle: cannot get handle from layer 0 (EncCli): rc=-2 [ensacli.c 1034]
E *** ERROR => EnsaCliLayerGetHandle: failed to create new handle (7) [ensacli.c 285]
E *** ERROR => ENSA_CommInit: failed to connect to server at /sapdp01 through layer EncCli (rc=7) [ensaclnt.c 764]
E *** ERROR => ENSA_Connect(): ENSA_CommInit() returned error -2 [ensaclnt.c 848]
E *** ERROR => ENSA_GetComBuffer: ENSA_Connect() ,connect to enqueue server failed. [ensaclnt.c 1719]
E *** ERROR => Enqueue: no REMOTESA-Buffer available (nil) 0 [enxxhead.c 2898]

2. Following error occurs during SUM upgrade of S/4HANA 1709 to 1809, in the phase MAIN_NEWBAS/STARTSAP_TBUPG:

Directory: /usr/sap/<SID>/SUM/abap/log
Name:SAPup.ECO

EXECUTING E:\usr\sap\<SID>\<instance\exe\sapcontrol.exe -prot PIPE -host <hostname> -nr <instance_number> -function StartWait 300 10
18.12.2018 22:44:38
Start
OK
18.12.2018 22:49:40
StartWait
FAIL: Timeout
SAPup> Process with PID 7684 terminated with status 2 at 20181218224940!
…
EXECUTING E:\usr\sap\<SID>\<instance\exe exe\sapcontrol.exe -format script -prot PIPE -host <hostname> -nr <instance_number> -function GetProcessList

18.12.2018 22:49:40
GetProcessList
OK
0 name: disp+work.EXE
0 description: Dispatcher
0 dispstatus: YELLOW
0 textstatus: Request handling without progress

Checking work directory of instance reported in the above error:

dev_disp

.....
Fri Jan 4 16:07:19:037 2019
DpUpdateStatusFileWith: state=YELLOW, reason=Server in state STARTING
Fri Jan 4 16:07:21:053 2019
DpUpdateStatusFileWith: state=YELLOW, reason=Request handling without progress
*** WARNING => DpRequestProcessingCheck: potential request processing problem detected (1. check) [dpxxwp.c 4991]
..
********** SERVER SNAPSHOT 1 (Reason: Request handling without progress / Time: Fri Jan 04 16:12:23 2019) - begin ******
| 0|988 |DIA |WP_NEW | | | | | |10 | | | | | | | |
| 1|5112 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 2|6160 |DIA |WP_NEW | | | | | | |10 | | | | 
.......

All the work processes stay in NEW status only.

Checking work process traces, dev_w*, we can see:

E Fri Jan 4 16:12:20:834 2019
E EnqChannel(-1): connecting to ZE48-V-SAP42:50119 (timeout=5000 ms)
....
M Fri Jan 4 16:12:21:866 2019
M ***LOG Q0I=> NiPConnect2: <IP address>:5<instance_number>19: connect (10061: WSAECONNREFUSED: Connection refused) [D:/depot/bas/773_REL/src/base/ni/nixxi.cpp 3456]
.....
M Fri Jan 4 16:12:22:189 2019
M *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 9/sock 656
(SI_ECONN_REFUSE/10061; I4; ST; <IP address>:5<instance_number>19: [nixxi.cpp 3456]
E *** ERROR => EnqChannel(-1): failed to connect NI on hdl-1 (rc=-10 => NIECONN_REFUSED) [EnqChannel.c 171]

3. You get the below error during installation of ABAP Foundation 1709 at the "Install Instance basics of ASCS<intance_number>" when the ASCS (ABAP Central Service) instance is started by SWPM.

SWPM error screen:

An error occurred while processing option Foundation 1709 on SAP NetWeaver AS for ABAP 7.52, version for SAP HANA > SAP HANA Database > Installation > Application Server ABAP > Standard System > Standard System(Last error reported by the step: Instance CS1/ASCS01 [PARTIAL] did not start after 5:10 minutes. Giving up). You can now:
Choose Retry
to repeat the current step. 
Choose Log Files
to get more information about the error. 
Stop the option and continue later.
Log files are written to /tmp/sapinst_instdir/S4HANA1709/FNDN/HDB/INSTALL/STD/ABAP.

The output of sapcontrol shows that Enqueue server in YELLOW status.

sapcontrol -nr <ASCS_instance_number> -function GetProcessList

14.02.2019 09:34:47
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2019 02 13 11:43:18, 21:51:29, 14018
enserver, EnqueueServer, YELLOW, Running but not responding, 2019 02 13 11:43:18, 21:51:29, 14019
gwrd, Gateway, GREEN, Running, 2019 02 13 11:43:18, 21:51:29, 14020

SAP NetWeaver with Standalone enqueue server (ENSA 1.0) or Standalone enqueue server 2 (ENSA 2.0)

SAP Netweaver, SAP NW, ENQ, Standalone Enqueue Server 2, enq/enable, ASCS, ABAP Central Service, ENSA, configuration, ENSA 1.0 and ENSA 2.0 profile parameters, wrong, incorrect, setting, stuck, stop, SWPM, not working , KBA , BC-CST-EQ , Enqueue , Problem