The rule in Cisco’s Doc CD is defined as:

“Rule 1 of RPF checking in MSDP is applied when the sending MSDP peer is also an i(M)BGP peer. When Rule 1 is applied, the RPF check proceeds as follows:

The peer searches the BGP Multicast Routing Information Base (MRIB) for the best path to the RP that originated the SA message. If a path is not found in the MRIB, the peer then searches the Unicast Routing Information Base (URIB). If a path is still not found, the RPF check fails.

If the previous search succeeds (that is, the best path is found), the peer then determines the address of the BGP neighbor for this best path, which will be the address of the BGP neighbor that sent the peer the path in BGP update messages.

If the IP address of the sending MSDP peer is the same as the BGP neighbor address (that is, the address of the BGP peer that sent the peer the path), then the RPF check succeeds; otherwise it fails.

Notes
The BGP neighbor address is not the same as the next-hop address in the path. Because i(M)BGP peers do not update the next-hop attribute of a path, the next-hop address usually is not the same as the address of the BGP peer that sent us the path.

The BGP neighbor address is not necessarily the same as the BGP router ID of the peer that sent the peer the path.”

In lamens terms, this rule checks that the receiver RP of the multicast group is using the MSDP peer that is in its best path towards the source. To show this, I’m going to give an example below.

In this topology what that rpf rule means, is that R5 will check the BGP best path to 1.1.1.1. Whatever BGP peer it goes through to reach 1.1.1.1, should be the peer that has been configured as a MSDP neighbor.

Because we R5 has a BGP & MSDP peer with both neighbors in the path towards the source, it means he will receive two MSDP SA’s about the same source (one from R4 and one from R3). There will be an RPF check on both SA’s, but only one of the rpf checks will be successful. Let’s start by checking the best path to the source on R5.

R5#sh ip bgp 1.1.1.1
BGP routing table entry for 1.1.1.1/32, version 79
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to update-groups:
1
2 1, (Received from a RR-client)
24.24.24.2 (metric 2) from 4.4.4.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, internal
2 1, (Received from a RR-client)
23.23.23.2 (metric 2) from 3.3.3.3 (33.33.33.33)
Origin IGP, metric 0, localpref 100, valid, internal, best
Before digressing, I want to point out some important stuff in the bgp output. 23.23.23.2 is the bgp next-hop towards 1.1.1.1, 3.3.3.3 is the neighbor that advertised this bgp route (i.e. the address in the #neighbor 3.3.3.3 remote-as 3 command), and 33.33.33.33 is the router-id of the neighbor that advertised this information.
Ok, the best path is via our neighbor 3.3.3.3. The reason is because of the lower router-id. This means that the RPF interface towards the source is via neighbor R3. I can verify this by checking the rpf interface that was selected, and performing an mtrace to 1.1.1.1.

R5#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: FastEthernet0/0
RPF neighbor: ? (35.35.35.3)
RPF route/mask: 1.1.1.1/32
RPF type: unicast (bgp 3)
RPF recursion count: 2
Doing distance-preferred lookups across tables

R5#mtrace 1.1.1.1
Type escape sequence to abort.
Mtrace from 1.1.1.1 to 35.35.35.5 via RPF
From source (?) to destination (?)
Querying full reverse path…
0 35.35.35.5
-1 35.35.35.5 PIM [1.1.1.1/32]
-2 35.35.35.3 PIM [1.1.1.1/32]
-3 23.23.23.2 PIM [1.1.1.1/32]
-4 12.12.12.1 PIM [1.1.1.1/32]
-5 1.1.1.1
This means that the MSDP neighbor 4.4.4.4 should be having rpf failures when he advertised the information about 1.1.1.1 in the SA message. We can check this by enabling the following debug on R5 & sending a ping to the multicast group from the source (i.e. forcing all routers from the source to send SA’s to R5).

R5#debug ip msdp peer
MSDP Peer debugging is on
R5#
*Mar 1 02:50:56.091: MSDP(0): 3.3.3.3: Received 120-byte msg 239 from peer
*Mar 1 02:50:56.091: MSDP(0): 3.3.3.3: SA TLV, len: 120, ec: 1, RP: 2.2.2.2, with data
*Mar 1 02:50:56.095: MSDP(0): 3.3.3.3: Peer RPF check passed for 2.2.2.2, used IBGP peer
*Mar 1 02:50:56.095: MSDP(0): WAVL Insert SA Source 12.12.12.1 Group 239.1.1.1 RP 2.2.2.2 Successful
*Mar 1 02:50:56.099: MSDP(0): Forward decapsulated SA data for (12.12.12.1, 239.1.1.1) on Loopback1
*Mar 1 02:50:56.103: MSDP(0): 4.4.4.4: Forward 120-byte SA (12.12.12.1, 239.1.1.1) from 3.3.3.3 to 4.4.4.4
*Mar 1 02:50:56.103: MSDP(0): 3.3.3.3: Received 120-byte msg 240 from peer
*Mar 1 02:50:56.107: MSDP(0): 3.3.3.3: SA TLV, len: 120, ec: 1, RP: 1.1.1.1, with data
*Mar 1 02:50:56.107: MSDP(0): 3.3.3.3: Peer RPF check passed for 1.1.1.1, used IBGP peer
*Mar 1 02:50:56.107: MSDP(0): WAVL Insert SA Source 1.1.1.1 Group 239.1.1.1 RP 1.1.1.1 Successful
*Mar 1 02:50:56.111: MSDP(0): Forward decapsulated SA data for (1.1.1.1, 239.1.1.1) on Loopback1
*Mar 1 02:50:56.115: MSDP(0): 4.4.4.4: Forward 120-byte SA (1.1.1.1, 239.1.1.1) from 3.3.3.3 to 4.4.4.4
*Mar 1 02:50:56.123: MSDP(0): 4.4.4.4: Received 120-byte msg 241 from peer
*Mar 1 02:50:56.123: MSDP(0): 4.4.4.4: SA TLV, len: 120, ec: 1, RP: 2.2.2.2, with data
*Mar 1 02:50:56.123: MSDP(0): 4.4.4.4: Peer RPF check failed for 2.2.2.2, used IBGP route’s peer 3.3.3.3
As you can see, we got the expected results. Hopefully you found this useful in your studying for MSDP.