How Notes performance can degrade on bad networks: and one way to fix it

Anthony Holmes  10 November 2008 06:50:19 PM

Excessive network fragmentation can cause some aspects of Notes performance to slow to a crawl: Dealing with attachments and dealing with some views and opening some types of documents. The bandwidth between you and the Notes application may be very good, but bad network packet fragmentation can cause times to extend from seconds to minutes: and ultimately lead to "server not responding" or "network operation did not complete in a reasonable amount of time" errors.

In one of my earlier blog postings, I mentioned how much bandwidth a Notes client will typically need. For this discussion, I'll assume you've got enough bandwidth but things are still slow.

So if it's not bandwidth, what could cause slowness? Recently an ISP was found to be slowing traffic on certain ports and included Notes' port (1352) along with the Peer to Peer programs they were wanting to throttle. That ISP has since stopped throttling Notes, but it's always possible that another ISP might get the same idea. But we'll assume that's not the cause of the problems you are seeing.

Which brings us to fragmentation. Network fragmentation, not disk fragmentation.

All traffic over TCP is broken into packets. There's a bit of overhead associated with each packet, so an efficient network needs to strike a balance when selecting packet size. The goal is to keep them reasonably large so there's little wastage sending header packets, but not so large that one user can hold up other traffic or, if data gets lost then a lot of data needs to be re-sent.

The largest packet size that TCP on ethernet allows is 1500 bytes.

This size is established by the sending PC, each router along the way and the receiving PC. In the real olden days, 1500 bytes was rather large: according to Wikipedia, it could take a second for a 14.4k modem to transmit a single packet of that size. A slow network might deliberately choose to break packets into smaller sizes so that it didn't spend an eternity (a second or so) looking after just one user.

Nowadays, most ethernet networks normally use packet sizes of 1500 bytes or just a little bit smaller.

But: if the packet size being used across the network drops much below about 1400 and if the PC has the usual TCP configuration, then the performance of Notes starts to become exponentially slower for some transactions (attachments, some views, some documents).

My gut feeling is that excessively small fragments are usually a result of extremely overloaded routers, or routers that are starting to fail, but haven't stopped working yet. Unlike most computing hardware, network hardware sometimes degrades without breaking down completely.

There's an easy way to test to see if you've got a packet size problem:

Ping the destination using packets of a size you choose using a flag to prevent them being broken. If you ping doesn't get through, try again with a lower packet size until eventually your packets are small enough to be allowed through.

ping -f -l 1500

If you get a response from the destination, there's no problem. If pings of 1500 bytes don't get through, reduce the number:

ping -f -l 1400

Eventually, you'll probably reach a point where your pings get through: Let's say it happens at 1100. That's a sign that you've got a problem.

(It might be that packet sizes are only dropping under certain conditions, so you might need to do the ping test at different times of the day.)

The solution? Get the broken network component (usually a router) fixed. The responsible network engineer ought to recognise the problem and agree to fix it. Some might claim that other traffic (eg HTTP servers) isn't slowed. That might be true but a) the router is processing a lot more header information than it needs to process and b) small packets mean that all sorts of traffic is being slowed, some more than others. So something does need to be fixed on the network.

In the worst scenario (eg a network engineer who refuses to budge, or a case where you can't contact anybody to make a fix (such as when the problem is at some distant overseas location)) then there's a workaround: You can arbitrarily reduce the default size for packets (Maximum Transmission Unit MTU) on the end user PC.This can be done in the Operating System (eg Windows) for the user. It's not a desirable thing to do because it reduces the overall efficiency of your networking. But, if you have a packet problem, the difference it makes can be truly magical.

(The instructions for changing MTU size varies with different versions of Windows/different operating systems, so dig around your advanced TCP settings or search for your OS and the terms MTU and TCP on the web.)



There's a fnal issue that can degrade Notes performance: Latency that's unrelated to packet fragmentation. John Lamb and Peter Lew's classic book, Lotus Notes and Domino 5 Scalable Network Design says "Client/Server applications that were developed for a LAN are now being pushed onto the wide area network. For many of these applications, a single transaction causes many round-trip flows back and forth 'under the covers'... Although each individual round-trip flow by itself takes well under a second, the sum of many turnaround flows looks like a long response time to the user."

Unfortunately, there's no magical client side fix for latency problems. The only place it can be fixed is by optimising the network (or, perhaps, using iNotes, since browser traffic has fewer turnaround flows).


Further information:


Lotus Software Knowledge Base Document

Title:        How to ping by packet size to establish MTU
Doc #:        1086718
URL:        http://www.ibm.com/support/docview.wss?rs=899&uid=swg21086718



Lotus Software Knowledge Base Document

Title:        Tips for Enhancing and Troubleshooting Notes 6 Client Performance
Doc #:        7006235
URL:        http://www.ibm.com/support/docview.wss?rs=899&uid=swg27006235


Lotus Software Knowledge Base Document

Title:        Troubleshooting Script: How to Diagnose Notes and Domino TCP/IP Issues
Doc #:        7003388
URL:        http://www.ibm.com/support/docview.wss?rs=899&uid=swg27003388


Below: Gauges from the Submarine USS Pampanito. I included this picture because Submarines go "Ping".  :-)

• Comments [1]