Salesforce

How to troubleshoot communication issues between the Mule Runtime Agent and the OnPrem server

Printable View
« Go Back

Information

 
Content

GOAL

If you are facing communication issues between the Mule Runtime Agent and the OnPrem server (discrepancies between what is seen in the OnPrem server and in Runtime Manager, metrics not pushed to the dashboard, etc.) there are some key points to ensure that everything has been met for the optimal behaviour of the Mule Runtime Agent.
 

PROCEDURE

1) Ensure you are using the latest Agent version.

You can identify your Mule runtime manager agent by following How to know the Mule Runtime Agent Version, and to know which is the latest version of the connector you can check the Release Notes.
Note that Mule 3.x uses Agent plugin 1.x, and Mule 4.x uses Agent plugin 2.x
You can get the latest agent from the Support Download Center.
To install the new version follow this guide.

Information to collect:
  • <RUNTIME_HOME>/logs/mule_ee.log and <RUNTIME_HOME>/logs/mule_agent.log
  • <RUNTIME_HOME>/conf/mule-agent.yml
NOTE: If the Agent version is updated from older versions(prior to 1.12.0) to later versions(1.12.0 and above) for Mule 3.x, then please refer to Issues with Runtime Manager after upgrading Runtime Agent from older versions to later versions(1.12.0 and above) in Mule 3 and make sure the configuration is correct under mule-agent.yml


2) Check that the expiration dates (license and certificate) of the server have not been reached.

Inside Runtime Manager, go to your Server Settings and check:
User-added image
If the license has expired, follow this documentation.
If the certificate has expired, follow this Applications and Servers showing as unknown and disconnected state in Anypoint Runtime Manager due to expired key pair.

NOTE: If the server is part of a Cluster, you won't be able to see those values through the UI. Instead, you will need to follow this procedures to check the license and the certificate. (How to check Mule Agent certificate expiration date)



3) Check that you have connectivity against all required hosts. 

Here is the list of all required hosts needed to be allowlisted. Make Network connectivity testing against all those endpoints.
If not able to establish connection to any of the hosts above contact your Networking team.

NOTE1: Some domains may add another IP address in the future for stability and reliability, so make sure to whitelist the domain, not the current IP addresses
NOTE2: Take into account, that some hosts have static IP assigned, but some others use dynamic IP only.

Information to collect: the output of each connectivity test.



4) Check that the certificate used against Runtime Manager is not getting intercepted.

To ensure that your firewall or proxy is not intercepting the certificate run the following command depending on the agent version you are using:
  • Agent version < 1.12.0 or 2.2.0:
$ openssl s_client -connect mule-manager.anypoint.mulesoft.com:443 -ign_eof

...
-----END CERTIFICATE-----
subject=/C=US/ST=CA/L=San Francisco/O=Mulesoft/OU=Mulesoft/CN=mule-manager.anypoint.mulesoft.com/emailAddress=devops@mulesoft.com
issuer=/C=US/ST=CA/L=San Francisco/O=Mulesoft/OU=Mulesoft/CN=Mulesoft/emailAddress=devops@mulesoft.com
---
...
  • Agent version >= 1.12.0 or 2.2.0:
$ openssl s_client -connect runtime-manager.anypoint.mulesoft.com:443 -ign_eof

...
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=salesforce.com, inc./CN=runtime-manager.anypoint.mulesoft.com
issuer=/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
---
...
If openssl tool is not available to you or if you are behind a proxy please use the corresponding method described in this article: How to check the Server Certificate of a particular Host

If the output is not the expected contact your Networking team.

Information to collect: the output of the openssl command.
 

5) Ensure mule-agent.yml file is not corrupted or empty

Check that <MULE_HOME>/conf/mule-agent.yml file is not empty, and does not have unknown characters.
If that's the case, follow the following article: Mule agent yml file corrupted or zero bytes length
 

If all the above has been met, and still facing the issue raise a ticket to the Support Team attaching all the "Information to collect" mentioned in each step.
You may use the Support information collector to collect most of the requested information:
  • Runtime Manager screenshot on your browser(if you are seeing unexpected Server/Application status)
  • Mule Runtime's logs folder (logs) - (Point 1)
  • Mule Agent installation details. mule-agent.yml and expiration details (agent) - (Point 1 and 2)
  • Anypoint platform whitelisting check (platformconns) - (Point 3)
  • Point 4 has to be collected manually.
Sample usage for this: 
support-collector download -p <mule_pid> -o logs, agent, platformconns
Attachments
Powered by