In the Cisco DOC CD, it describes the 3rd Rule of the MSDP RPF check as.

“Rule 3 of RPF checking is applied when the sending MSDP peer is not a (M)BGP peer at all. When Rule 3 is applied, the RPF check proceeds as follows:

The peer searches the BGP MRIB for the best path to the RP that originated the SA message. If a path is not found in the MRIB, the peer then searches the URIB. If a path is still not found, the RPF check fails.
If the previous search succeeds (that is, the best path to the RP that originated the SA message is found), the peer then searches the BGP MRIB for the best path to the MSDP peer that sent the SA message. If a path is not found in the MRIB, the peer then searches the URIB. If a path is still not found, the RPF check fails.
If the first autonomous system in the best path to the RP is the same as the autonomous system of the sending MSDP peer, then the RPF check succeeds; otherwise it fails.”
My version of this rule is:

“If you have an MSDP peer, but it’s not a BGP peer, make sure the MSDP peer is the in the BGP best path towards the source RP.”

Let’s take this into a topology and have a look.

R6 is not a BGP peer with R4, however, R6 and R4 are indeed MSDP peers. This means, MSDP Rule 3 applies, so R4 must be in the bgp bestpath towards the multicast source R1 (1.1.1.1). Let’s break down step 1 above, “The peer searches the BGP mRIB or uRIB for the best path to the RP that originated the SA message”. The originating RP is R1. The steps below confirm R1’s RP address, and that R6 has a BGP route to it.

R1#sh ip pim rp
Group: 239.1.1.1, RP: 1.1.1.1, next RP-reachable in 00:01:25
Group: 224.0.1.40, RP: 1.1.1.1, next RP-reachable in 00:00:04

R6#sh ip bgp 1.1.1.1
BGP routing table entry for 1.1.1.1/32, version 58
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
4 2 1
5.5.5.5 (metric 2) from 5.5.5.5 (5.5.5.5)
Origin IGP, metric 0, localpref 300, valid, internal, best
So that’s step 1 done. Next, let’s confirm step 2, “the peer then searches the BGP MRIB or uRIB for the best path to the MSDP peer that sent the SA message”. OK. Our MSDP peer that sent us this SA here, is R4 (ip address 45.45.45.1). So we need to check the BGP mRIB or uRIB for the best path to it.

R6#sh ip bgp 45.45.45.1
BGP routing table entry for 45.45.45.0/30, version 69
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
4
5.5.5.5 (metric 2) from 5.5.5.5 (5.5.5.5)
Origin IGP, metric 0, localpref 300, valid, internal, best
Ok, so we have a path to it. This best path to our MSDP peer is via R5 (5.5.5.5) towards AS4. The final step states: “If the first autonomous system in the best path to the RP is the same as the autonomous system of the sending MSDP peer, then the RPF check succeeds; otherwise it fails”. Ok, let’s backtrack to the #sh ip bgp 1.1.1.1 output above. We can see that the first AS in the best path to the RP (1.1.1.1) is AS 4. We need to make sure that this is the same AS as the MSDP peer that sent us the SA. From the #sh ip bgp 45.45.45.1, we can see that this is also AS 4. So the RPF check succeeds. On R6, if I enable #debug ip msdp peer, then make R1 (the source) send a couple of multicast packets, we will see that the RPF check is performed & succeeds.

R1#ping 239.1.1.1 so lo0 repeat 2

Type escape sequence to abort.
Sending 4, 100-byte ICMP Echos to 239.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1

Reply to request 0 from 56.56.56.2, 168 ms
Reply to request 1 from 56.56.56.2, 148 ms
Reply to request 1 from 56.56.56.2, 148 ms

R6#
*Mar 1 08:52:44.153: MSDP(0): 35.35.35.1: Received 120-byte msg 1016 from peer
*Mar 1 08:52:44.153: MSDP(0): 35.35.35.1: SA TLV, len: 120, ec: 1, RP: 1.1.1.1, with data
*Mar 1 08:52:44.157: MSDP(0): 35.35.35.1: RPF check failed for 1.1.1.1
*Mar 1 08:52:44.157: MSDP(0): 45.45.45.1: Received 120-byte msg 1017 from peer
*Mar 1 08:52:44.161: MSDP(0): 45.45.45.1: SA TLV, len: 120, ec: 1, RP: 1.1.1.1, with data
*Mar 1 08:52:44.161: MSDP(0): 45.45.45.1: RPF check passed for 1.1.1.1, Peer is best in the closest AS
*Mar 1 08:52:44.165: MSDP(0): WAVL Insert SA Source 1.1.1.1 Group 239.1.1.1 RP 1.1.1.1 Successful
R6#
*Mar 1 08:52:44.165: MSDP(0): Forward decapsulated SA data for (1.1.1.1, 239.1.1.1) on Loopback1
*Mar 1 08:52:44.169: MSDP(0): 35.35.35.1: Forward 120-byte SA (1.1.1.1, 239.1.1.1) from 45.45.45.1