In my previous post i explanined the how the missing labels can forward the route traffichow the missing labels can forward the route traffic to another locations. This scenario is also like the same but in this customer end to end is pinging but our core is not pinging.
A figure given is typical SP network in which MPLS is used and loopbacks are used for label advertisements. By default labels are generated for all the RIB routes and being distributed to all its adjacent peers. But in the current scenario labels are only advertised for loopbacks only and the same loopbacks are being used for MP-BGP. OSPF is used as IGP. MPLS core is having two route reflectors and every PE is having peering with each RR to provide redundancy to customers. Customer is having two branches; HO is located near PE1 and branch office is located near to PE3. Static routing is used between PE-CE. A vrf TEST is used for customer. Route reflectors are loaded with 12.4(15) T1 ios. PE1 & PE2 are using 12.2(31) SB13 ios and PE3 is 2800 series router which is using the 12.3 sp ios. RR1 is having smallest router id than RR2, so always RR1 route will be preferred.
Customer HO is already provisioned and serving all the other branch locations. A new branch is being provisioned named CPE-A which is directly connected with PE3. Prior to its provisioning the same vrf TEST is configured on PE2 with no ip address and able to receive all the routes from HO. Without deleting the vrf from PE2, we configured the vrf TEST on PE3 with static route of CPE-A lan pointing towards wan address. Static routes are also configured at CPE-A end pointing towards PE wan. Once the provisioning completed we are able to ping PE-CE interfaces successfully. But remote locations are still unreachable. There after we check the route updates on PE1 for WAN BRANCH subnet and astonished to see that the routes are coming from both RRs but it prefers the RR2 route. But it should not be like this it is showing because it should select the RR1 update for routing the traffic. Another thing which is seen really a weird that the routes update is coming from both the RRs but with different next hop. In the given snap shot you can see that the second path is selected as best path which is coming from RR2 and PE address is 10.1.1.2. Actually there is a vrf configured on PE2 but with no ip address. First update is correct update but it does not prefer. I thought there might be some problem other it cannot happen like the way it is going on. Right now customer as per us customer is down because it is not reachable in our clod but the same instant a call received from customer end and he tells the link starts working. Customer call really intrigues me. I think how this can possible? Sub optimal routing makes customer end to end live.
On PE1 update for 10.10.4.0/24 is coming from PE2 and advertising by RR2 but it should be RR1. Router is receiving 1000 as out label for lan route and next hop is PE2. On PE2 1000 is used as in label and 2000 is used as out label and on PE3 2000 label is used for customer lan. It clears the fact that labels can do anything which I have already described in “Missing Labels Cause Packet Drops”.
Below are the commands used for trouble shooting
a) Show ip bgp vpnv4 vrf TEST x.x.x.x – This command is used to check the updates from teh different Route reflectors and tells which is selected as best with which label will be tagged during the forwarding of route.
b) Show ip route vrf TEST x.x.x.x – This command is used to see which route is installing in the vrf routing table.
c) Show mpls forwarding label detail – This is really a awesome command which can help to see the mpls vpnv4 label with vrf name also. If you are getting the route with [V] it means it is VPN route else it is IGP route.
Need to clear the BGP session with RR2. After that proper updates are received. We are getting these types of issues on daily basics but every time we need to clear the bgp session on receiving end router which is not good one. Now we are planning to upgrade the ios of RR because T1 is having 2 critical bugs in which routes are not going to flush from PE.