GOAL
You want to enable Hazelcast Health Monitor logs every X seconds.
PROCEDURE
You will need to add the following JVM properties into the Cluster Node wrapper.conf:
wrapper.java.additional.<n>=-Dhazelcast.health.monitoring.level=NOISY
wrapper.java.additional.<n>=-Dhazelcast.health.monitoring.delay.seconds=5
In this case the following entry will be logged into the mule_ee.log by 5 seconds:
INFO 2019-03-29 16:05:45,805 [hz.893983305.HealthMonitor] com.hazelcast.internal.diagnostics.HealthMonitor: [10.8.20.153]:5701 [68d0baf0-c88c-4784-a65c-ac27b869b59d] [3.10.4] processors=12, physical.memory.total=16.0G, physical.memory.free=45.7M, swap.space.total=5.0G, swap.space.free=842.3M, heap.memory.used=750.6M, heap.memory.free=255.9M, heap.memory.total=1006.5M, heap.memory.max=3.6G, heap.memory.used/total=74.58%, heap.memory.used/max=20.62%, minor.gc.count=8, minor.gc.time=106ms, major.gc.count=2, major.gc.time=166ms, load.process=9.89%, load.system=0.00%, load.systemAverage=2.18, thread.count=101, thread.peakCount=101, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=6, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0
INFO 2019-03-29 16:05:50,809 [hz.893983305.HealthMonitor] com.hazelcast.internal.diagnostics.HealthMonitor: [10.8.20.153]:5701 [68d0baf0-c88c-4784-a65c-ac27b869b59d] [3.10.4] processors=12, physical.memory.total=16.0G, physical.memory.free=42.1M, swap.space.total=5.0G, swap.space.free=838.8M, heap.memory.used=758.9M, heap.memory.free=247.6M, heap.memory.total=1006.5M, heap.memory.max=3.6G, heap.memory.used/total=75.40%, heap.memory.used/max=20.84%, minor.gc.count=8, minor.gc.time=106ms, major.gc.count=2, major.gc.time=166ms, load.process=8.52%, load.system=NaN%, load.systemAverage=2.09, thread.count=101, thread.peakCount=101, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=6, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0
INFO 2019-03-29 16:05:55,815 [hz.893983305.HealthMonitor] com.hazelcast.internal.diagnostics.HealthMonitor: [10.8.20.153]:5701 [68d0baf0-c88c-4784-a65c-ac27b869b59d] [3.10.4] processors=12, physical.memory.total=16.0G, physical.memory.free=37.2M, swap.space.total=5.0G, swap.space.free=838.8M, heap.memory.used=771.4M, heap.memory.free=235.1M, heap.memory.total=1006.5M, heap.memory.max=3.6G, heap.memory.used/total=76.64%, heap.memory.used/max=21.19%, minor.gc.count=8, minor.gc.time=106ms, major.gc.count=2, major.gc.time=166ms, load.process=8.67%, load.system=33.33%, load.systemAverage=1.92, thread.count=101, thread.peakCount=101, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=6, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0
ADDITIONAL NOTES
If you have already added the system properties under the Wrapper.conf file and restarted the runtime, but the Hazelcast Health Monitoring logs are still not showing, please check if you are still missing the following AsyncLogger config under the log4j2.xml from the Mule Runtime.
<AsyncLogger name="com.hazelcast.internal.diagnostics.HealthMonitor" level="INFO"/>
