This is an deep dive into understanding the layer 2 and layer 3 MTU settings, as well as understanding what MTU value is actually being used when you are sending pings with different sizes. Theres more to it than actually meets the eye on the CLI.
Ive strung a couple of switches together to setup test environment. The 10 gig interfaces will be set to use an MTU of 9216, then Ill discuss what this means at layer 2 and 3.
6500-to-6880
Configuration
6500
interface TenGigabitEthernet1/1
no switchport
mtu 9216
ip address 1.1.1.1 255.255.255.0
6880
interface TenGigabitEthernet1/1/1
no switchport
mtu 9216
ip address 1.1.1.2 255.255.255.0
As you can see, both sides have their layer 2 MTU value set to 9216. This value is actually the maximum allowed size of the Ethernet frame before the layer 2 headers and FCS value (i.e. the CRC check) are are added.
A traditional 802.3 datagram is 1518 bytes large, which would break down into:
Destination address = 6 bytes
Source address = 6 bytes
Type/Length = 2 bytes
FCS = 4 bytes
Data/Payload = 1500 Bytes
However additional fields can be added to expand the size of the layer 2 frame above the traditional 1518 bytes, such as: VLAN tagging (4 bytes), QinQ (4bytes), ISL trunks (30 bytes) etc. It just appears that Cisco have taken these values out of the equation, making it much more simple to configure the MTU value. Moral of the story is: When setting the MTU at layer 2, its exclusive of the layer 2 headers and CRC.
Whats interesting on Cisco devices is that configuring the layer 2 MTU, it also automatically adjusts the layer 3 IP MTU value; as shown in the output below.
6500#sh ip int te1/1 | i MTU
MTU is 9216 bytes
6880#sh ip int te1/1/1 | i MTU
MTU is 9216 bytes
Thats a bit confusing right? Considering that the the IP header is 20 bytes, you would have thought that the IP MTU value would actually be 9236. However what Cisco devices do, is add the 20 byte IP header to a 9196 byte payload into the total configured layer 3 PDU size. So when you see the IP MTU is set to 9216 byes, this is the total layer 3 packet size (i.e. the payload + the IP header, as a combined value).
Cisco Pings
This methodology is also carried over to their ping requests. So when I send an icmp echo between the devices with a size of 9216 bytes, this is the total layer 3 PDU size as it is encapsulated into a layer 2 frame. In the ping below, Im sending a total layer 3 sized PDU (inclusive of the IP and ICMP header) of 9216.
#ping 1.1.1.2 size 9216 df-bit
Type escape sequence to abort.
Sending 5, 9216-byte ICMP Echos to 1.1.1.2, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
So let me elaborate on what I mean by breaking down the packet structure for this ping:
Ethernet Frame size (exclusive of header and CRC) = 9216 bytes
IP MTU size = 20 bytes
ICMP header = 8 bytes
ICMP payload = 9188 bytes
So actually the ping size is 9188 bytes. But when you issue the ping command and set the size of it to 9216 on the CLI, this actually defines the total ping size inclusive of ICMP and IP headers. You absolutely need to be aware of this for Cisco products. If you were to send a ping with an adjusted size in operating systems such as Linux, Microsoft Windows, and Juniper, they all actually refer to the to the ICMP payload size. For example. I am on a Windows laptop at the moment, connected hard wired into a switch that has the interface IP MTU set to 1500 bytes. If I ping the switch with the df-bit set, at size 1472 it works, but at 1473 it fails. Remember the test below is on a Windows machine to a Cisco switch with the standard MTU settings (1500 for both).
C:>ping -l 1472 10.15.0.254 -f -n 1
Pinging 10.15.0.254 with 1472 bytes of data:
Reply from 10.15.0.254: bytes=1472 time=3ms TTL=255
Ping statistics for 10.15.0.254:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 3ms, Average = 3ms
C:>ping -l 1473 10.15.0.254 -f -n 1
Pinging 10.15.0.254 with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Ping statistics for 10.15.0.254:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss)
So this would be broken down into:
Ethernet Frame size (exclusive of header and CRC) = 1500 bytes
IP MTU size = 20 bytes
ICMP header = 8 bytes
ICMP payload = 1472 bytes
Summary:
The #mtu 9216 value sets the maximum possible layer 2 payload (i.e. exclusive of layer 2 headers and CRC check).
The #ip mtu 9216 value sets the maximum possible layer 3 PDU (i.e. inclusive of the IP header & ICMP header, if using ICMP).
In my topology these are set to the same value. However, there will be times where the IP MTU will need to be lowered to accomodate for things like GRE or IPSec headers (something non-standard, unlike ICMP, which would require additional modification). The layer 2 MTU will never change, but the IP MTU value certainly will need reducing for things like tunnels.