Intermittent and excessive loss of UDP traffic

This is a call for help to get an understanding of what happened. We are still flabbergasted and confused.

Lets start from the beginning. We have s system consisting of a number of servers that primarily is communicating with each other through TCP. In very few occasions we use UDP. The system is running in a virtual environment (VMware). From one of the services, lets call it US (Udp Sender) we send UDP to another server UR (Udp Receiver).  The US is one or more and the UR is behind a load balancer even though it currently is a single server.

After upgrading VMware from 5.0 to 5.1 the UDP packages on the UR end are lost excessively and intermittent. We are using RHEL 5.5 in a quite old version as the guest OS. Checking with tcpdump on the US side we see that all servers send packages as expected. Through divide and concur we rule out switches, load balancers, firewall, physical NICs until we on have the virtual NIC left.

So far so good - now for the strange. In an attempt to replicate the error in a simpler and easier to test fashion I wrote a small python script sending the exact same packages. Through tcpdump we verify that the packages are identical (and the UR is also fooled by the packages). The strange thing is that the packages sent by the java app at the US end are lost as before while the same packages sent from US using the python script aren't - not even once. The catching java packages and python packages was done with the following command: tcpdump -c 2 -nn -Ss0 -i eth0 udp and UR-loadbalancer-IP

 The fix of the problem was to switch the e1000 guest NIC driver to a VMXNET 3 driver. We are still waiting on both VMware and RedHat to clarify the issue.

If someone can explain why packages sent by java are sometimes lost while identical packages sent by python aren't. The obvious would be that the packages aren't identical but as far as I can tell they are....




Comments

Post a Comment

Popular posts from this blog

Possible SYN flooding on port 3306 (MySQL)

The system setup is such that the MySQL servers that put the "Possible SYN flooding on port 3306" in the log files only are exposed to system internal backend services. These in turn aren't exposed to the wild wild web. Fronting the system we have the servers publishing services to internet. Thus I was kind of stunned when the log messages started to appear and even though we had done a release of the system I found it far fetched that we should start to DOS our self. So why did the messages appear? Two different error messages could be identified in the log files and they seem to be related, especially since the Java servers with link failure do communicate with the MySQL servers. Possible SYN flooding on port 3306 @ MySQL server Communication link failure @ Java backend servers After some tcpdumping, head scratching and googling I think I have it down to the root cause, hidden in how TCP works in general, and the OS config in combi...
Read more

Part 1 - Disaster Recovery with SRM and vSphere Replication

Image
Abstract  The company I work for has needs for disaster recovery but there is also external demands for it. The company not being very big has so far not been able to implement any automated disaster recovery solution although we do have a documented disaster recovery plan. The plan regardless of how good or bad it is isn't tested in a long time and the RTO (Recovery Time Objective) is wished by management to be days but I can't see that it is anything else than weeks. Since we virtualized our production system I've been looking at VMware vCenter Site Recovery Manager as a driver but the cost array based replication has stopped any attempts dead in their tracks. VMware vSphere® Replication has been around for a while now and we hope that it's hardened enough for a slightly bigger production critical implementation. In this article I will try to document and explain how we experiment, design and implement disaster recovery and even more important, in my opinion...
Read more

Part 2 - Disaster Recovery with SRM and vSphere Replication

Image
In the previous article we went through the installation and configuration of the SRM and vSphere infrastructure. The time has now come to actually doing some tests and failover some VM's.  In the simple scenario I expect everything to go smoothly but there are a few things that I'm concerned about at this point since the protected environment I'm ultimately is failing over isn't so simple. It has multiple dvSwitches, vLans, load balancer, firewall, ldap, a set of test drivers to verify the integrity of the system and access to the system in a sensible way for administrators so there are a few things that needs to be resolved. Protected Setup Test Failover The small test In this scenario I do a test failover of a single machine and verifies that it starts and that I can log into it. The steps are: Setup the machine running RHEL 5.5.  Install VMware tools  Kick of the vSphere Replication Create a Protection Group and ... no that didn...
Read more