3128523 - Frequent cluster related problems and jgroups errors after customizing the jgroups-tcp.xml

This article is specifically targeted towards cluster related problems that happen with a custom configuration, e.g.

cluster.broadcast.method.jgroups.configuration=custom-jgroups-tcp.xml

1. After making this change, you may notice that pods are dropping out of the cluster more often and not rejoining. In hac > Monitoring > Cluster, not all pods will be listed and you may even find inconsistent results depending on which backoffice pod you are looking from.
   
   
2. In the logs, you will find more errors of the form: "WARN","loggerName":"org.jgroups.protocols.TCP","message":"JGRP000032: hybrisnode-24: no physical address for hybrisnode-19, dropping message"
   
   
3. Still in the logs, you might also find errors such as: "WARN","loggerName":"org.jgroups.protocols.pbcast.GMS","message":"hybrisnode-31: too many JOIN attempts (10): becoming singleton"
   
   
4. If you manage to catch the issue in real time and run probe as per 2507555 - How to use JGroups Probe to diagnose hybris cluster issues, you will notice that one or more pods have different coordinators and belong to a different cluster island than the rest of the pods.
   
   
5. When logging in to hac and navigating to Platform > System > System properties tab, you do not find the properties below:
   

   hybris.jgroups.remove_all_data_on_view_change hybris.jgroups.write_data_on_find

SAP Commerce Cloud for versions below:

• 1905 on patch level 6 or greater.
• Any major version after 1905 (e.g. 2005, 2011, etc.) on any patch level.

cluster,jgroups,view,coordinator,jgroupsping , KBA , CEC-SCC-CLA-ENV-EMG , Environment Management , Problem