When XML-RPC calls from a remote backup service failed only on VPN connections and the port forwarding fix that made backups reliable again

As businesses and individuals increasingly rely on digital infrastructure to safeguard their data, remote backup services have become vital components of any IT strategy. These services depend on consistent, reliable communication with remote servers using standardized protocols such as XML-RPC. However, challenges can arise when the network topology is complex — such as in environments using VPNs. One puzzling issue emerged when XML-RPC calls from a remote backup service consistently failed during VPN connections, even though they worked seamlessly on standard internet connections.

TL;DR

Remote backup services using XML-RPC encountered failures while connected via VPN. The issue was traced to improper port forwarding on the VPN gateway, which blocked incoming XML-RPC communication. By identifying the required ports and implementing correct port forwarding rules, the issue was resolved. Backups became fully reliable again, both on and off VPN.

The Nature of the Problem

Remote backup applications often use XML-RPC—a remote procedure call protocol encoded in XML—for coordinating file transfers and validating backup statuses. In most instances, these communications happen over well-known ports and are managed through firewalls by either whitelisting the remote client’s IP or using SSH tunnels. However, in a particular case, XML-RPC calls that worked in normal internet scenarios were failing when the backup client was connected over a corporate VPN.

Operations like retrieving server status, initiating scheduled backups, or restoring files would result in timeouts or “403 Forbidden” style HTTP errors when attempted over VPN. These RPC calls never reached the intended backup server, essentially rendering backup operations inoperable when connected to the internal network.

Initial Troubleshooting

When analyzing failed XML-RPC communications, the immediate suspects were network-level blocks and application misconfigurations. The IT team verified the following:

• Firewall Settings: Both local and network firewalls were reviewed to ensure that communication over the RPC-specific ports (typically port 80 or 443) was permitted.
• Backup Service Logs: Application logs showed that the XML-RPC calls failed to establish HTTP connections, timing out without receiving any meaningful response from the server.
• Manual RPC Invocation: Attempts to manually query the backup server’s XML-RPC interface via tools like ‘curl’ or ‘Wget’ also timed out over VPN but worked on standard connections.

Despite confirming that the remote backup server was operational and listening, the VPN pathway proved to be the variable. Curiously, traffic routing didn’t change substantially according to the routing table, and DNS resolution was functioning identically in both scenarios.

Pinpointing the Issue

The breakthrough came with the recognition that VPN clients were routed through a single NAT (Network Address Translation) gateway inside the corporate firewall. While outbound HTTPS traffic was generally allowed, the backup application’s XML-RPC calls attempted to reach specific ports or expected certain headers not handled correctly by the VPN gateway.

Packet inspection tools such as Wireshark and TCPdump were deployed on both the client and server side. These revealed that XML-RPC packets were not even arriving at the backup server when initiated over the VPN. This pointed to a filtering or redirection problem at the gateway level.

Further investigation into the VPN configuration files and firewall at the network edge revealed that:

• The VPN appliance had restrictive NAT rules.
• Only certain ports (e.g., 80, 443 for web traffic) were forwarded correctly through to external services.
• Custom or lesser-known ports, specifically those used by the backup software for XML-RPC, were being blocked or dropped silently.

The Port Forwarding Fix

To resolve the issue, the network team implemented a series of port forwarding rules on the VPN gateway. These rules forwarded incoming and outgoing traffic for the required XML-RPC ports to the appropriate destinations on the remote backup server. The following steps were followed:

1. Identified all the TCP ports used by the backup software’s XML-RPC engine (e.g., 8080, 8731).
2. Configuring the VPN firewall/NAT settings to allow this traffic through without rewriting IP headers in a restrictive way.
3. Testing the XML-RPC connection using debug tools to ensure ports were mapped and responses properly received.

Once in place, these rules allowed RPC calls to traverse the VPN market cleanly, resulting in fully restored communication between client and server, even over VPN.

Testing and Validation

After making the firewall and port forwarding changes, the team initiated a series of tests to validate the fix:

• Successful start and completion of automated nightly backups over VPN.
• Responsive and reliable XML-RPC ping requests to the server made from VPN-connected endpoints.
• Restoration processes were tested to confirm both read and write RPC instructions executed as expected.

This rigorous verification ensured that backups would now function reliably for users on VPN—especially important for remote workers who default to VPN access for security and compliance reasons.

Preventative Measures Moving Forward

To avoid such issues in the future, several best practices were adopted:

• Comprehensive Firewall Documentation: Maintain clear documentation of all service ports required by applications, including backup clients.
• VPN Testing Suites: Include connectivity testing in routine VPN quality assurance checks.
• Alert-Driven Monitoring: Configure alerts for failed backup jobs so that silent issues are surfaced quickly.
• Cross-Department Coordination: Encourage collaboration between network and application teams to address full-stack connectivity obstacles.

Conclusion

The resolution of XML-RPC failures over VPN highlights the importance of understanding the end-to-end data path in networked applications. Even widely-used protocols like XML-RPC can behave unpredictably when underlying infrastructure components—like VPN gateways and firewalls—are not uniformly configured. By implementing appropriate port forwarding rules and validating VPN configurations, the backup system returned to its expected reliability, safeguarding enterprise data effectively no matter the client location.

FAQ

What is XML-RPC used for in backup services?

XML-RPC is a protocol allowing a backup client to communicate with a server for tasks like initiating data transfers, checking status, and restoring files. It’s often used due to its simplicity and wide support in multiple programming languages.

Why did backups fail over VPN but work outside it?

The VPN gateway lacked correct port forwarding configurations. While it allowed general web traffic, it blocked the ports needed for the XML-RPC calls used by the backup services, causing failures over VPN only.

How does port forwarding fix XML-RPC communication issues?

Port forwarding ensures that incoming and outgoing network requests on specific ports are directed to the appropriate server. Without this, the VPN or NAT might block or drop traffic, thinking it’s unauthorized or irrelevant.

Can VPN encryption interfere with XML-RPC?

No, VPN encryption in itself doesn’t interfere. The issue typically lies with how the VPN handles routing or filtering, not the encryption layer.

What tools help diagnose XML-RPC over VPN issues?

Logging tools on the backup client, network packet analyzers like Wireshark or TCPdump, and VPN/firewall logs are all invaluable in diagnosing these problems.