Information

Content

<< DEPRECATED ON AUGUST 2022. PLEASE CHECK How to troubleshoot Mule cluster communication issues on hybrid or on-prem environments >>

GOAL

There are some cases in which, after creating a cluster, the cluster hasn't been correctly created. It is also possible that in a running cluster the nodes stop being connected. This could happen due to networking issues between the servers and you can verify it following this procedure. If you are using Runtime Manager, the nodes could automatically inform that there are communications issues between them (see Related links section).

PROCEDURE

In order to verify that the cluster is working as expected, and all the nodes are registered to it, we'll need to review the mule_ee logs.

For Mule 3.7.x to Mule 4.1.x versions

The information about the current cluster composition will be found in the mule_ee logs.
The logs will show the list of members of the cluster and their IPs, if the cluster is properly created, then you should see all the nodes listed.
The logs should look like the following:
Cluster OK log:

Members [2] {
    Member [<IP-Node1>]:5701 this
    Member [<IP-Node2>]:5701
}

If at some point in time this node is disconnected you will find a log entry like this one:
Cluster NOT OK log:

Members [1] {
    Member [<IP-Node1>]:5701 this
}

For Mule 4.2.x and higher versions

From 4.2.x version onwards the cluster logs are not printed in the mule_ee logs automatically.
To print the cluster logs in mule_ee logs, enable the below packages to DEBUG in $MULE_HOME/conf/log4j2.xml file.

<AsyncLogger name="com.mulesoft.mule.runtime.module.cluster" level="DEBUG"/>
<AsyncLogger name="com.hazelcast.internal.cluster" level="DEBUG"/>

The logs will show the list of members of the cluster and their IPs, if the cluster is properly created, then you should see all the nodes listed.
The logs in the master node should look like the following:

Sending member list to the non-master nodes:
Members {size:2, ver:2} [
    Member [172.17.0.2]:5701 - 8b154059-7a51-48eb-8f3d-96030a63141b this
    Member [172.17.0.3]:5702 - 77a359d3-3789-4647-874a-7adcfc2a66fb
]

The logs in a non-master node should look like the following:

com.hazelcast.internal.cluster.impl.MembershipManager: [172.17.0.3]:5702 [3886699] [3.12] Cannot publish member list to cluster.
Is-master: false, joined: true , mastership claim in progress: false