Re: Odd tracepath and ping behavior when using a 9000 byte MTU

John Stewart <john.stewart@noyb.com.invalid> wrote:

> Summary:
> I've recently switched my network to gigabit. All the hardware is 9k
> jumbo frame clean. I can switch all the machines to a 9000 MTU and
> everything works great. But, if I run tracepath on my linux machine
> it decides the path MTU should be 1492. However if I set the MTU to
> 8996 or any MTU where size= 8x+4 then tracepath works fine.

> Details:
> MTU is set to 9000 via:
> ifconfig eth0 mtu 9000

> I run ttcp to send a few 32768 byte buffers and check the size of the
> on wire packets on the receiving end with ethereal. They're 9014
> bytes just like they're supposed to be.

> now I run tracepath and get the following output:
> tracepath 192.168.1.100
> 1: 192.168.1.103 (192.168.1.103) 0.099ms pmtu 9000
> 1: 192.168.1.103 (192.168.1.103) 0.018ms pmtu 8166
> 1: 192.168.1.103 (192.168.1.103) 0.015ms pmtu 4352
> 1: 192.168.1.103 (192.168.1.103) 0.016ms pmtu 2002
> 1: 192.168.1.103 (192.168.1.103) 0.013ms pmtu 1492
> 1: 192.168.1.100 (192.168.1.100) 0.347ms reached
> Resume: pmtu 1492 hops 1 back 1

> For some reason the sending machine seems to have decided it has to
> fragment the packets. And from the output it appears that it is doing
> it before it reaches the wire. From this point on the ttcp+ethereal
> test shows the on wire packet size as 1506. This behavior persists
> until I change the MTU. I.e. I think that it is remembering the
> "discovered" MTU for that route and using it for all subsequent
> connections.

If the sending machine is using PMTU Discovery as described in RFC 1191
then the receiving machine likely advertised a MSS of 8996, corresponding
to an MTU of 9014. The receiving machine starts with it's own 9000 MTU
and then tries, in descending order, the "plateaus" for the various
types of physical networks listed in that RFC which have MTUs less than
9000 and 9014, starting with 8166 (RFC 1042). When it finds one that
is an integral multiple of 8996

> But if I set the MTU to 8996. Or for that matter to any size between
> say 2k and 9k that is of the form size=8*x+4 (8996=8*1124+4) then
> tracepath works fine. example:
> ifconfig eth0 MTU 8996
> tracepath 192.168.1.100
> 1: 192.168.1.103 (192.168.1.103) 0.114ms pmtu 8996
> 1: 192.168.1.100 (192.168.1.100) 0.778ms reached
> Resume: pmtu 8996 hops 1 back 1

> Both before and after running tracepath the ttcp+ethereal test shows
> the on wire received packet size to be: 9010

> Some info on the machines in question:
> sending machine:
> motherboard: Tyan Tiger motherboard
> processors: dual Opteron 240
> NIC: Onboard intel gigabit
> Running: Gentoo Linux with kernel 2.6.4-r1 for AMD64
> Using the e1000 driver that came with the kernel with default
> settings.

> receiving machine:
> motherboard: MSI Pro266-TD Master-LR
> processors: Dual Intel tualatin PIIIs at 1.26 gHz
> NIC: Intel pro/1000 MT desktop adapter
> Running: Windows 2000
> Using the 6.01.03 drivers from Intel.

> switch: SMC 8508T

> And here's the results of additional testing using ping with the don't
> fragment flag set:

> immediately after setting an MTU of 9000 I do the following:
> ping -c 4 -M do -s 8968 192.168.1.100
> which succeeds:
> 8976 bytes from 192.168.1.100: icmp_seq=1 ttl=128 time=1.09 ms

> and then this:
> ping -c -4 -M do -s 8970 192.168.1.100
> which fails:
> From 192.168.1.103 icmp_seq=1 Frag needed and DF set (mtu = 8166)

> At this point the ttcp+ethereal test shows an on wire packet size of
> 8180 bytes.

> try again with the MTU at 8996:
> ping -c 4 -M do -s 8968 192.168.1.100
> success:
> 8976 bytes from 192.168.1.100: icmp_seq=1 ttl=128 time=1.12 ms

> ping -c 4 -M do -s 8970 192.168.1.100
> fails:
> From 192.168.1.103 icmp_seq=1 Frag needed and DF set (mtu = 8996)

> But notice it hasn't changed the MTU.

> Since the problem occurs with both ping and tracepath I'd guess it's
> not happening at the application level which leaves the network stack
> and the NIC driver.

> Does anyone have any ideas?
> Can anyone reproduce this?

> Thanks,

> John Stewart





--
Clifford Kite Email: "echo xvgr_yvahk-ccc@ri1.arg|rot13"
PPP-Q&A links, downloads: http://ckite.no-ip.net/
/* Bluffing in a poker game can win big; bluffing in a newsgroup
only attracts sharks. */

Clifford Kite <kite@see.signature.id> wrote:
> John Stewart <john.stewart@noyb.com.invalid> wrote:

>> Summary:
>> I've recently switched my network to gigabit. All the hardware is 9k
>> jumbo frame clean. I can switch all the machines to a 9000 MTU and
>> everything works great. But, if I run tracepath on my linux machine
>> it decides the path MTU should be 1492. However if I set the MTU to
>> 8996 or any MTU where size= 8x+4 then tracepath works fine.

>> Details:
>> MTU is set to 9000 via:
>> ifconfig eth0 mtu 9000

>> I run ttcp to send a few 32768 byte buffers and check the size of the
>> on wire packets on the receiving end with ethereal. They're 9014
>> bytes just like they're supposed to be.

>> now I run tracepath and get the following output:
>> tracepath 192.168.1.100
>> 1: 192.168.1.103 (192.168.1.103) 0.099ms pmtu 9000
>> 1: 192.168.1.103 (192.168.1.103) 0.018ms pmtu 8166
>> 1: 192.168.1.103 (192.168.1.103) 0.015ms pmtu 4352
>> 1: 192.168.1.103 (192.168.1.103) 0.016ms pmtu 2002
>> 1: 192.168.1.103 (192.168.1.103) 0.013ms pmtu 1492
>> 1: 192.168.1.100 (192.168.1.100) 0.347ms reached
>> Resume: pmtu 1492 hops 1 back 1

>> For some reason the sending machine seems to have decided it has to
>> fragment the packets. And from the output it appears that it is doing
>> it before it reaches the wire. From this point on the ttcp+ethereal
>> test shows the on wire packet size as 1506. This behavior persists
>> until I change the MTU. I.e. I think that it is remembering the
>> "discovered" MTU for that route and using it for all subsequent
>> connections.

> If the sending machine is using PMTU Discovery as described in RFC 1191
> then the receiving machine likely advertised a MSS of 8996, corresponding
> to an MTU of 9014. The receiving machine starts with it's own 9000 MTU
> and then tries, in descending order, the "plateaus" for the various
> types of physical networks listed in that RFC which have MTUs less than
> 9000 and 9014, starting with 8166 (RFC 1042). When it finds one that
> is an integral multiple of 8996

Sorry, forget this. I was trying to learn while composing a reply and
accidentally posted it before it was ready. In reality, it's likely
that it wouldn't have ever been completed and so been discarded.

--
Clifford Kite Email: "echo xvgr_yvahk-ccc@ri1.arg|rot13"
PPP-Q&A links, downloads: http://ckite.no-ip.net/