No category

Download A Methodology for Troubleshooting Inter

Transcript

A Methodology for
Troubleshooting
Inter-domain
IP Multicast
Engineering Workshops
1
Problems Addressed
• The main types of problems addressed in this
section are topology/reachability problems – the
packets aren’t flowing.
• The source and receiver are assumed to be in two
different AS’s. Troubleshooting multicast within
your own campus network is a subset of
interdomain troubleshooting.
• Because it is the most common today, we assume
ASM. Many problems would go away with SSM.
• We will mention some things about performance
issues at the end, and list some tools/references.
Engineering Workshops
1
Why the need for a “methodology”?
• Most engineers don’t troubleshoot multicast
problems as often as unicast.
• As we have learned, multicast is receiver-driven
(somewhat backwards).
• The problem can be far from the symptom.
• The same symptom can have many different causes,
at different places in the path.
Engineering Workshops
Overview
1
Engineering Workshops
1
Engineering Workshops
6
/
*
2
*
(
'
*
2
*
(
/
,
! .
'
(/
*
*
)
2
?
1
$
1
=
(
*
*
1
.
#
:
/5
*/
<
(
2
=!
%+
*
*
)
(
&'
%*
/
3
%
'
)
!
*
&'
*
(
)
)
&'
/
$
9
9
.
/5
/
.
/9
8
>
/.
6
<
(
%(
'
/
.
1
/
1
/
<
(
*/
<
.
(
6 /5
7
&'
2
)*
!
4
(!
*
2
*
/
3
)
32
*'
5
/
#
1
(
'&
*
)
*)
+
0/.
! -
(
<'
'
.
;
*
+
(
*
2!
,
:
#
*
*
()
&'
3
)
)
&'
/
.
.
/9
8
% $
! #
"!
What is the problem?
1
Engineering Workshops
Gather Information
• End-users seem to have trouble reporting multicast
•
•
•
•
problems in our language.
Performance issue vs. topology/reachability issue?
Was it working recently then stopped working, or
has one site gotten nothing at all from another site?
Is the problem intermittent, cyclic, or steady-state?
User education about how to report a problem before
a problem happens is very helpful!
Engineering Workshops
1
Gather Information
• Pick ONE direction (that is the problem, or seems
representative of the problem).
• Identify source end and receiving end.
C
B A
FED
@
C
B A
• Recall multicast is unidirectional in nature…
FED
C
B A
@
C
B A
Implies almost nothing about…
Engineering Workshops
1
1
Gather Information
Now that you have a direction, you will need:
• A constantly active source IP address
• A constantly active receiver IP address
• The group address
It is virtually impossible to debug a
multicast problem without specifying
all of these!!!
Engineering Workshops
Gather Information
OK – we know the IP addresses for
•the
problem source, receiver, and
group, and that the source and
receiver are active.
Move on to step 2…
Engineering Workshops
1
H
H
H
G
1
Engineering Workshops
Verify Receiver Interest
• Because of the way multicast distribution trees
are built, it is almost always easier to debug a
problem by starting at the receiver. If you are the
sender, you are pretty much working blind.
• Recall in ASM, group interest on a subnet is
indicated by a host sending out (multicast) an
IGMPv2 membership report.
• The DR (designated router) on a segment is
responsible for listening to that report, and
forwarding a PIM ( * , G) join towards the RP.
• For this step, all we need to do is verify which router is
the DR, and check that it knows it has interested
listeners for that group on the interface facing the
receiver. Stop there. Don't worry about getting to the
Engineering Workshops
RP at this point.
1
Verify Receiver Interest
• You might think you know which router is the DR, but you
should not proceed until it has been verified. It only takes a
couple seconds.
• To verify the DR, log into the router you think should be
routing multicast for the receiver.
1) Find/verify the interface that serves the receiver’s
subnet.
2) Check that there is no other PIM router that thinks it is
the DR for the subnet.
• Although in our workshop lab our first-hop routers are
Ciscos, the following examples show both Junipers and
Ciscos.
Engineering Workshops
1
K
I
L
IJ
MN
b
a
e
Q
e
a
u
I
h
f n
L
I
I
SO
L
g
I
KR
O
O h
S
fS
f
S
f
R
f
L_g
I
i
_ Q
Ow
T
O
g
_
l
a
W
O Q
I
S
K_
K
gS
f
\]^
g
f
g
_f
_
O
g
I
L
K_
l
a
a
ta
Rf
\]^
k
e
j
Q
X
X
X
a
V
V
Zv
W
[
Z
W
UZ
l U
I
L
q
u
` q
Kf
SO
\]^
S
s
b
gS
e
k
j
f
g
_f
_
O
a
e
a
t
h
g
f Q
W
X W
Y
X W
Y
X W
Y
l
M
Q
O n
i
_f
\]^
l
a
S_f
a
fa
L
i
Lf
qr
p
o
m
M
Q
n
Q
g T
S
X
`g
O
d
iO
U
X
X
VU
[
Z
UZ
V
XYW
X
L_
k
j
h
fQe
X I
d
cbI
PS
O T
_
a
O Q
L
S`
a
f
O T
_Q
_Q
\]^
\]^
I
I
L
MN
UT
X
X
VU
[
Z
UZ
V
XYW
SR
RQ
POM
K
IJ
x
y
| {
zy
xy
Verify Receiver Interest
1
1) Cisco: find the right interface:
Engineering Workshops

¦
Y
~¥
~
¡

}

}
~
¤

~
}

}

}~
1) Juniper: find the right interface:
Engineering Workshops
§
¨
«¨ ª
§¨©
Verify Receiver Interest
1
×
Ø
Û Ú
ÙØ
×Ø
Ë
Í
Í
Í
Í
Í
ÏÐ
°
Ñ
Ë
ÒÐ Ë
±
ÐË
cÓÄ
Öº
ÔÕ½
±
±
Ç
ÊÂ
¯
Å
À
¯¬
ÀÃ
Â
ÇÉ
¬
Ç
ÆÈ
ÆÇ
¬
¯
Â
»
¾
»
¼½ À
Ë
Á
¿¸
ÀÅ
¬
Ä
Ä
¬ÂÃ
Ë
Ë
Ð°
Ì
±
°
Í ÌË
Î
Á
»
¿¸
¼¾
»
¼½º
»º
¶
¹¸·
µ´
³²±°
®¯
¬
Verify Receiver Interest
1
1) Nortel: find the right interface:
Engineering Workshops
Ü
ß
ÞÝ
Ü
áà
â
éä
æ
æè
ô
Ü
ö
ø÷
ÿï
õ
ú
çè
ø÷
å
Ü
å
Ü
Ü
Ü
çè
à
çâ
ý
÷
øý
ú
ü
ü
â
ú ù
Ü
é çè
Ü
ä
ú ÷
à
â
èö
ç
ç
âó
é ö
ß
æ
Ü
Ü
ß
ë
Þô
Þ
Ü
çè
ü
ü
â
â
çâ
ý
÷
øý
ú ÷
ú ù
Ü
é çè
ÞÝ
éè
ã
ä
ç
âó
Þô
é
Ü
ß
ë
ç
Þô
Þ
å
ô
ÿ
Þ
ç
Ü
ô
Ü
è
éç
Ü
ß
ë
ç
ÿ
ù
öâ
ó
à
à
âó
çé
Ü
ß
ë
â
ð
ö
èó
è
äâ
ç
ç
âó
ßë
è
é çè
æ
æ â
ô
ß
æ
Ü
æ
Þô
Þ
ÿ
ê
Ü
ç
ç
þ
î
þ
â
â
â
èó
è
ß
æ
ô
ò Ü
ßë
ÿï
õ
ä
è
ÿï
õ
õ
ö
ð
Üè
ô
æ
Ü
ç
ç
ã â
Þô
é
Ü
Ü
ë
ßë
æ
Þô
æ
è
è
Ü
å
Ü
ß
é çè
æ
Üè
æ
ô
ã
Ü
é
ÞÝ
éè
ç
ù
ù
ù
÷
óâ
â
éè ð
å
å
Ü
ß
è
Ü
æ
ô
ë
è
Üè
Ü
å
Üè
æ
ô
ã
Ü
é
ÞÝ
éè
ß
å
ç
ù
÷
ÿï
õ
ö
â
â
Ü
æ
ô
ë
è
Üè
Ü
Þô
å
éè
ÞÝ
éè
â
â
ù
÷
ü
ö
çâ
ç
ÿï
õ
Ü
æ
ô
ë
Üè
Ü
ß
é çè
æ
ù
ö
óâ
â
ÞÝ
éè
ÿï
õ
è
ç
ÿï
õ
â
Ü
ü
æ â
ô
Ü
éè
é çè
Þô
é
æ
é
Þé
è
ç
ç
ÿï
õ
â
à
Ü
ü
æ â
ô
Ü
éè
Ü
ô
æ
é
Þé
ð
ö
èó
è
â
ßë
è
é çè
æ
æ â
ô
ß
æ
Ü
ÿï
õ
ê
ç
õ
Ü
ø÷
ü
ü
þü
ý
÷
øý
ú ÷
ú ù
û
ú
ö
í
Ü â
Ü
è
é ö
ß
æè
é çè
æ
Þ
ç
ç
ñ
Ü
æè
ã
Ü
ô
ôë
ô
ã
é
æè
ç
ç
óâ
ó
â
â
îí
ì
éè à
ò ã
Þ
ð
â
ä
ß â
ï
àâ
áà
ßë
ä
ä
è
é çè
æ
îí
ì
â
ã â
åä
ã â
Ü
Ü
ß
Ü
ÞÝ
ê
Verify Receiver Interest
1
2) Cisco: verify DR for that interface:
Engineering Workshops
e
m
i
lk
no
V l
Ug
jf
U
a
W
W
hi
c
[
d
04
BC
N
N
I
I
H
F
FS
[
8 I
N
J
J
[ 8 H
Y W
Z
H
I
[
Y W 8
Z
[
Y W
Z
g
/fe
bU
:
2
M<
^_
`a
F
F
F
G
G
8 I 8
/ML
W\
X
X
D
R4
/^]
[
Y W
Z
V
TU
04
:
2
Q
H
6
I
6H
N
M<
/ML
N
IQ
IG
HG
8
JH
O
IG
HG
F
04
:
H
6
I
6H
N
M<
2
P
/ML
ED1
ED1
F
2
2
IG
HG
F
04
:
2
H
6
I
6H
N
M<
/ML
KJ
8 H
ED1
2
12
1
7
<
0
0
:4
2
.
A=
1
41
12
BC
?
6
-
@
<
>
>
4>
=
94
2;
8
5
12
:
41
0
92
34
.
0
-7
6
/.-
!
',
+*)
(
&'
%#
$#
"
Verify Receiver Interest
1
2) Juniper: verify DR for that interface:
Engineering Workshops
¥

¬£
«
ª

¥
¤
¤

¢
¥ ¢
¢ ¥
¢ ¥
¤
¡
£

¡
¢ ¥ ¢Z
¦

¥
£

§¨

§¨
¥
¤
¤
¢Z ¢ ¤

¢ ¤ ¢ ¤
¤

¤
¤
¢ ¤ ¢ ¤
¤

¡
¢
Z
¢ ¡ ¢

¢
Z
¢
¥©

/
/

v
|

}
}
~
~}
{z
y
ut
xw
s
rq
p
Verify Receiver Interest
1
2) Nortel: verify DR for that interface:
Engineering Workshops
Verify Receiver Interest
• SO… now you are sure you are on your
receiver’s DR.
• Remember, multicast is receiver-driven.
• QUESTION: Does the DR know that there
are interested receivers of the group on
your host’s subnet??
• Look at IGMP for the group in question.
Engineering Workshops
1
Ó
ÏØ
Óï
Ì
Ú
ÒÕ
ÃÕ
Ì
ã
á
â
á
á
ß
ß
æ
ß
á
â
éâ
â
ç
ß
ß
æ
è
ãê
âæ
â
éâ
â
ß
ç
ß
çæ
æ
à
Ô
æÍ
çæ
â
Í
Â
ÌÕ
Ù
Ì
Î
ÊÌ
Î
ÐÇ
ÊÞ
Ó
ØÝ
Ó
Ê
Î
ÐÕ
ÜÛÚ
ÑÊ
Ð
ËÊ
Ø×
Î
ÊÌ
Ó
Ó
É
â
Ö
ÎÍ
Ê
Ð
É
Ï
Ç
ÈÎÃ
Øå
ä
â
â
ãâ
æ
æ
á
á
ß
à
ßà
æ
Í
ãí
ì
îæ
ÉÊ
Î
Ì
ÊÔ
ëØ
â
â
ãâ
ß
á
á à
é
á
á
ðÔ á
á
à
ßà
Ï
Ä
Í
Ð
Ó
Î
ÔÕ
ÊÒ
Ñ
Ê
Ð
Ç
ÈÎÃ
ÊÌ
Ë
ÉÊ
É
ÈÇÆ
Å
ÄÃ
Â
¿
¿
¿
°
®
±
®¯
µ
¼·
º»
¸
¹·
À
À
ÁÀ
½
¾
½¾
¸¶
·¶
´
³²
ñ
+ýü
ñ
ø
÷ò
ôó
ö
õ
ñ
ñò
ôó
ù
ø
ûôú
Verify Receiver Interest
1
On the DR (Cisco):
If receiver’s interface is in this list, you are OK. You
might want to watch for a while to ensure no
timeouts are occurring.
Engineering Workshops
FE
1
1
06
+
1
!0
/
I
C
C-
(
&
@
+
).
+ )
,
)
)
*
*
!
20
9
+,)
.
=
>3
3
+,)
J
J
If receiver’s interface is in this list, you are OK.
You might want to watch for a while to ensure
no timeouts are occurring.
Engineering Workshops
Q
PK
ML
O
M S
JVUT
ML
N
JK
R
Q
On the DR (Juniper):
3
)=
3
-
+ -
5-
'
8
$<;
%
4
34
<D
#
A
&
2
+ 3
&
ÿ
#7
)
:
=
+ )
,
+ )
,
+ )
,
&
%
!
&
2
<?
&
0
( F
H
?
AB@
9G
%
$#"
!
0
þÿ
A
!
Verify Receiver Interest
1
1
Verify Receiver Interest
ie
hb
f
ge
fd
v
l

v
l

B
t{
y~
wx
j
y
u

}

{

s
{
z|

z{
m
rq
y
pon
sx
t
jw
lm

vs
v

sn
kj
]
c
`
ed
ba
_^
\[
Z
W
YX
On the DR (Nortel):
If receiver’s interface is in the list for the group, you
are OK.
You might want to watch for a while to ensure
no timeouts are occurring.
Engineering Workshops
Verify Receiver Interest
• What if your interface isn’t listed with that
group, even though everything else about the
DR looked fine??
• You have a problem!
– Host OS / driver problem
– Application problem
– Broken IGMP snooping switches in the
middle
– Try tcpdump on the host - can you see the
IGMP membership reports on the wire?
(Remember, they don't have to come from
that particular host.)
Engineering Workshops
1
Verify Receiver Interest
• If your receiver’s DR knows it has
listeners of your group on that
interface, you are done this step.
Move on to step 3…
Engineering Workshops
1

1
Engineering Workshops
Verify knowledge of active source
• This is often the most complex part – the bulk
of your work could be here. As we have
learned, a lot has to happen for the receiver’s
DR to know about a particular source.
• You MAY have view this from both ends
– The receiver’s RP
– The source’s RP
• For most interdomain cases, these RPs will not
be the same, and MSDP will be involved.
Engineering Workshops
1
à
à
õ
ô
ò
å
è
è
ò
ø
ä
ì
ëá
è
Ôëè
ç
ç
ü
òú
éê
éê
ñ
à
à
å
ô
ò
ø
õ
á
ì
á
ëá
è
Ôëè
è
åá
ëá
á
ê
éê
ìê
ò
ÿ
Ôëè
ç
ü
ú
ÿ
ÿ
þ
ô
û
ò
ð
à
ë
ÿë
þ
ñ
ã
ã
ã
ú
òú
û
ò
í
á
ñ
ä
ä
ã
ã
ã
è à
á
á
á
è
åí
ø à
îï
ð à
ÿ ã
ÿ
ã
úþ
ó
úþ ã
úþ
ó
ò
úþ
ü
ý
ûü
óÿ
Bóþ
úù à
ëô
òó
ä
á
è
á
í
í
ø÷
öõ
åä
ì
îï
ì
ëá
è
è
è
Ôëè
ç
å
ä
ä
åä
á
â
áâ
æ
ßÞ
ñð
éê
Â
Ä±
±
¾
Ç
¸
Å
Ý
¸Ö
¾Ü
Ð±
µ³
Ç
º«
±
¾
É
Ú
¸¹
¾Ê
µ±
¾
Ã Á
Ã
À
¾
µ±
¾
¶
µ³
Ç
º«
±
¾
Ì
Ì
Ñ²
¾
ÛÚ
ÅÕ
±
¶
Ç²
¾
ÐÅ
À
¶
Ç
¼
Ñ²
¾
®×
±
¸»
´º²
« Ò
»
º² Â Â
±
¾ ¾· Ò
Ç ³ ÔÓÊ
¾ ¾
Ç
Ì
µ³ Ó ² ±
¾ ¾
¶²
Ê ¾
Ì
° ¸Ä Ê
µ ® Ó
¶» Ç
À ³¾ µ ²
Ö

ÆÅÄ Ä
Â
¾
Á¬
³ Ã
Ö
µÇ
²
Ì Õ
ÔØÊ ´³²
Â
µÇ
¾ ¸Ö Õ
¶
·
¶ ± Á
Ã
Ø
²± ¾ ·
³ Å
¸
¾Ù Ç
±
Ê
¬Ä
¸Í
´º²
Ç
³
¾
¾
·¬
·
¶
¾
±
Â
Â
Â
Â
®
À
¾
¾Á
Ä
¾

É
³
º
¾
¾
±
¸É
º
Å
ÇÈ
¸
¶
¶
µÇ
Ä
Ä
ÆÅÄ
¶
º
Á
°
³
µ
¸Ë
Ã °
Â
Ë
Ã Ê
Ã
²± Ã
Â
Ä ½
Á¬
º
Ç
µ± Ã
¾
¿
º
± Â
Ç
¾·
ÎÃ
»
¶²
Ï ±
¾
Â
Ç
¬
Ç
¸
Ì
°
Ð µ
Î Ã »
¸Í
¼
´º²
Â
¼ ¬Ä
´Ñ²
¾
¼
Ç Â
½
²±
®·
º ¶
º ¾
±
´º² Ã
»
Â
®
¬
Ã
Ã Ê
Â
Â
Ã ¼
Â
Í
¬
¶»
µ
¿
°±
±
¸¹·
®
±
´º²
»
¼
µ
½
p¾°
µ¶
´³²
¯®
«¬
¥
¥
¥
¥
¥
¥

¡¢
£
¦
¨ª
¨©
¨¦
¦
¨¦
¦
¦
§¦
£
¤
£¤
B
B

Verify knowledge of active source
1
• If the DR does NOT know about the source,
we may only see a ( * , G) entry on a Cisco DR,
and we have some work to do.
Engineering Workshops
* ()
%
$"#
&'
B
A
#G
!
D
=
#
=F
<
ED<
, C
-
5
6 #5
/
-
, A
2
,
?
2
2
Engineering Workshops
7
()
78
7(
(
7(
0
6#5
/
& -
/
,
,
!
$
2
(
(
4(
0
10
2 1
3
2
"
%
-. #
,
+
* ()
%
$"#
&'
!
<=>
:; "
#
!
/
!9
@
,
Verify knowledge of active source
1
• If the DR does NOT know about the source,
we may see nothing on a Juniper DR, and we
have some work to do.
Verify knowledge of active source
• If the DR does NOT know about the source,
Nc
_
bN
N
_
_ P
3
^
P
a
R
_
Y
N
N
^
_
_ P
3
P
W
Y
]
\[
ZT
`N
XY
ZT I
^
.XV
[
W
.WV
M
n
V
RW
R
\
l
m.gf
R
\ V
\[
V
Yk
[
ij
T
H
QP
O MN
I
KLJ
Z
X
g[
T
L
hf
Z
ag
[
e
f
R
U.TS
QP
d
X V
O MN
I
KLJ
we may see nothing on a Nortel DR, and we
have some work to do.
Engineering Workshops
1
Verify knowledge of active source
• Recall that knowledge of active sources is first
spread through a given PIM domain by pergroup RP-rooted shared distribution trees.
• Current practice is to set the Source Path
Tree (SPT) threshold to zero, so that (S,G)
state is created by on the first packet sent
through the RP.
• But if the RPT doesn’t get built properly, the
SPT never will!
Engineering Workshops
1
Verify knowledge of active source
• So, first, we will work back from the receiver’s
DR to its RP, to be sure that the RPT branch is
built correctly.
• Second, we will check to see if the receiver’s RP
knows about the source.
• Third, we will check with the source end for
their RP’s knowledge and advertisement of the
source.
• Last, we will troubleshoot MSDP as needed to
make sure knowledge of the source can get
from one RP to the other.
• The following page has a rough flowchart for
later reference.
Engineering Workshops
1
Verify knowledge of active source
Yes, but
Go to
Recv DR know of source?
still no
step 4
No
traffic
Is RPT built correctly recv DR to recv RP?
Yes
No
Troubleshoot RPF, PIM
Recv RP know of source?
Yes
No
Source RP know of source?
No
Yes
Troubleshoot source DR to RP Troubleshoot MSDP
Engineering Workshops
1
Ò
íß
Ó
Ó
×
Ö Ì
ð
Ó
Ö Ó
Ò
Ð
öÝ à
í
ÔÕ
Ê
Ò
ÔÕ
ë
ã
Ð
Ì
×
Ì
Ö Ì
Ó
Ö Ó
Ó
ÐÌ
Ö Ì
Ì
Ö Ó
Ê ì
íß
Õ
ÔÕ
×Õ
öÝà
í
íÝ
è
ì
íå
Ü
Öë
ß ê
çì
Ýí
ìë
ë
ì õ
Ýí
Þ
å ê
ë
ê
ï
å ê
Þè
Û
Î
Î
Î
Ê
Ö
í
å
å
Ï
ðÏ
ï
Ó
Ì
Ì
Ì
Ó
Ê
Î
Î
Î
Ö
Ï
Ì
Ó
Ì
Ø
Ø
ÔÕ
Ñ
Þß
ãâ
áà
Ý ÜÛ

ÐÏ
×
ÙÚ
×
Ö Ì
Ø
Ê
Ó
Ó
Ö Ó
Ò
Ð
Ï
Ï
ÐÏ
Ì
Ó
Û
Ê
Î
Î
Î Í

ÌÍ
Ì
ÐØ
ã
ÙÚ
Ýî Ü
çì
Ýí
ìë
å ê
Þ
å ê
é
çè
ë
ô
ã
Ï
Þ
Ý ê
ó
è
å
Ê ì
Ê
Ýî Ü
æåä
ò ñ
Ê ÈÉ
Ë
®

±
¤

¨
¨
«
Ç
³
Ä
¨ µ
E£
¨

¯
±
£
°
£À
¬
'
»¨ Æ

® ¨

ª
¨
¡

·
¡
±¨
°

» ÅÄ
¼
¨
°¿

ª¡
±
±
·
¦
¼
¨
¤

¨
¨
·
À

Â¡

µ
¨ Ã
±¨
±
Â µ
¬ À
ª¡
¥
·

±
¤

¨
¥
.¤
£¥
Á

¨
¨¢

±
£
°
¨¢
¡
£À
.
° ¯
¨

±
£ ¯
µ
¾
¨

¬
·
®
®
«

«
¢
¿
¬ ¿
¾
¡¨
±
¯

½
¾µ
¨
¬ ½
¦
¤

±
¦
¼.
¨
.¤
£¸
Engineering Workshops
¥
.¤
¢
¤
¤
¹
»±£

¬ º
¨
±¨

¯
¯
¬
® ¹±
®
.¤
£¸
¬ ¸

®
®
®
®
¤
±
¨

¥
¡¨
§

¦
¯
¬ ¦
¬

±
¡
¨
¥
¨¢
¬ ©
¡¨
§
µ

¬ ¢
¬ ¢
¬

¥
¨«
®
®

²
¨
¨

£¶
¬ ¶

¤
¤
£ ³
¬ ³
´

® °
±£
¬¡
±
®¡¨
¯
¯
° ¯
¬ ¯
®¡¨
¤
¬ «

ª¡
©

¦
§
¨
¥
.¤

E£¢
¡
.

s
o
–
We can first just look at the ( * , G) entry on the receiver's DR.
–
If that doesn't look right, we can look to see how it learned
about the RP with
{
~ }
3
|u
wq
z
z
y t
r
u
xu
v
wu
v t
r
u
u t
r
q p
r
Verify knowledge of active source
1
• Does the DR have the right RP (Cisco)?
ý
"
ø
÷ú
û
$ $
$
%
û
$
úû
ýü
þÿ
øû
ùú
÷ø
)
$
$
)
"
"
û
þ
÷ú
ø
#
(
ø
þ
"
÷ú
'
ùø
û
&
ú
û
#
ø
ùø
%
%
"
ú
÷
û
!
ú
û
û
ø
÷
$
ø
þ
ù
ÿ
÷
øû
ù
ø
û
ý
û
ø
÷
ù
ú
ý
þÿ
ýü
øû
ùú
÷ø
Verify knowledge of active source
• Does the DR have the right RP (Juniper)?
Engineering Workshops
1
0
/.
21
-
,+
*
E
E
E
E
E
E
.
.
.
C
.
DC
1
C
Q.
C
1P
C
C
C
C
FC
F
C
DC
-O
*N
*N
G
*
@ G
*
M
JM
@ G
*
*
M
JM
G
*
+
H
@
I
G
*
H
?
:
?
-
G
*
KL
=7
<
;J
B8
97
G
E
E
E
:;
!97
8
!87
!54
3
21
/.
B8
A@?
F
1
C
C
C
C
FC
C
DC
<
=>7
6
0
-
,+
*
Verify knowledge of active source
• Does the DR have the right RP (Nortel)?
Engineering Workshops
1
Verify knowledge of active source
• What if the RP is wrong?
– A common problem is that auto-RP and/or PIMv2
R
U
U
Y
Z
Z
UY
W
Z
Z
UY
^]
\
X
X
[
X
X
W
V
US
!TR
S
!SR
YX
BSR may be running without the admin's knowledge
(on Ciscos they are on by default when PIM-SM is
enabled, and Junipers listen to them). Information
can leak from a neighboring AS! These take
precedence over anything you statically configure.
Hint: use
– Auto-RP and BSR are complex, and could have any
one of a number of problems. We recommend static
configuration in most campus networks, Anycast-RP
in backbone/transit networks.
– Might just be a typo in entering the static RP
address.
Engineering Workshops
1
Verify knowledge of active source
• Now that you are sure of what the RP is (and it
is correct), starting at the receiver’s DR, work
your way back to the receiver’s RP:
• Check that the RPF is pointing the way you
expect.
• Check that PIM is configured and working
properly on the interface. A common problem
is PIM is not turned on for a particular
interface.
• You may also want to double-check that each
router has ( * , G) state for the group you are
debugging.
Engineering Workshops
1

£¨
£

¨
¡
ª«

¢
¢
¤
¥

¢¤

!
¬

¢
¥
±

!
¥

¢
µA´

!
³
¢

¢
¢
¡
²
®

¢
!

!
¥

¢
!
£
£

!
!

¨§

q
z
r
ox
{
~v

>}{
y
v|
u
ys
y
p
!wr
uv
!tr
s
!sr
!po
n
–
¥ ¢ °

®¯

b
g
hi
_
_
el
j
m
k
k
dc
ed
!dc
!a`
_
f
–

¢°

Verify knowledge of active source
1
Cisco:
Engineering Workshops
"
9
9
9
<
9
<
C
C
5
<
B
5
/A?
F
;
<,
<
$
@
?
C
<,
B
G
;
?
E
?
/A?
@
<
B
9
;
5
<,
B
;
=;
B
<:
;:
9
/A?
@
?
<
<
C;
E
?
=
5
<,
B
;
>=
& ;
87
& ; & < & < & <
;
;
9 B ;
<
? B
& ; & < & & <
C
>= &
D;
9
&
=
>C & ; = & <
?
/A?
@
?
D
9
<C
<:
<:
:
&
;:
9
;:
9
87
:
87
%
*
$
0
0
(
(
+
,
,
56
430
"2
'"
.1
+
/.-
",
+
!"
&
#
(
"
'
!"
ÿ
îú
îú
î
þ×
óî
ð
à
÷
×
êé
Ö
×Ú
øý
$ %
(
)
"
ì
÷
Ù
é
ÙÚ
ÖÙ
ôü
û
ô
ç
òí
î
òí
ð ó
ð
íî
í
ð ï
ñ
íî
ú
Ü
ç
Ú
Ú
õ
ö
!øè
×
Ú
ã
Öë
ã
ê
Ú
íî
í
íî
áò
í
òí
ð ó
ð
ç
Ú
é!è
æØ
å
Ùä
ã
âá
Ü
ÝÞ
ÜÛ
Ö ðñï
×è
ø
ó
ï
áî
ù í
ñ
×
×Ú
ØÙ
Ö×
Ýô à ß
×ç
ê
ã
ê
Ú
!éè
ð Ú
÷
æ
ì
Í
Ê
Õ
Ì
ËÔ
½¼
Ä
ÅÆ
¶
¶
ÁÈ
À
É
Ç
Ç
Â¾
ÁÂ
¶½
!¿¾
»º
!ÓÏ À
ÑÒ
!ÐÏ
Î
¹
¶
!ÌË ¸!·
Ê
Ã
–
–
& < &
7
3
Verify knowledge of active source
1
Juniper:
Engineering Workshops

z
z

]

m
kn
i
mv
m
ku

il
t
{
x

z
z
x~
]

|
|
m
rn
r
m
op
{
|}x
zy
x
mn
z
w
jl
k
s
s
q
h
_
^g
fa
ej
i
ca
Z
`
h
_
^g
fa
de
ca
b
ba
_^
]
YX
\[
W
VU
T
K
S
J
IR
QL
OP
NL
M
ML
JI
H
–
p
h
_i
Nortel:
Verify knowledge of active source
1
Engineering Workshops
Verify knowledge of active source
•
Next Big Question: Does the receiver's RP have
knowledge of the active source?
•
Since we already checked that the RPT is correct, it
probably doesn’t, or the DR would have likely had (S,G)
information.
•
If it doesn’t, but has ( * , G) only, and no MSDP SA
(source-active) cache entry for that source, we will
have to find out some information about the source
end of things, then troubleshoot MSDP.
•
Note it does not matter which peer you get an SA from
as long as it is accepted and in the cache. However, if
you are going to open a ticket with an upstream, you
might as well figure out who you expect to accept it
from.
Engineering Workshops
1
Verify knowledge of active source
The SA originates from the source’s RP, and is readvertised/ flooded by MSDP peers along the way.
•
Some sites have estimated that about half of their
multicast problems are problems associated with
missing MSDP SA information.

º¹

¾
¾
¾
¯®
·´ ¬± °
µ ·
µ
¸²
µ ²
²
µ ²
µ ² ª¬
¶£²´
µ
²³´

§¨
º¹
º¹

¡
¢£¡
¢
¥¤
¢
¤
¤¦
º¹
º¹
¾
¾
»
½¼
»
½¼
»
½¼
»
½¼

¾
»
½¼
§¨
»
½¼
§¨
§¨
º¹

•

The objective here will be to get an MSDP sourceactive about the source to our receiver’s RP.

•
Engineering Workshops
1
"
"
)
&
&
(
'&%
*
!
$
"
# å
å
Í
Ì
á
ÄÍ
ÛÉ
â
â
Ù
Æ
ÊÆ
Ä
Í
Ì
Þ
æ
Ù
è
ø
Ôè
ÁÔ
Ü×
Éà
Õ Ñÿ
Ð Õ Î
Ð
Ð Õ
Ð
Ð Õ ÒÑ
Õ
Ð
Î
å
Ý
á å
Ü×
å Ù
Ëú
Í
âÄ
Ìâ
À
Ù
â
ÛÚ
Ñ
Í
Ê
Û
Ä
ÛÚ
ì
ÏÎ
Ï
È Ð
ÇÝÀ
Î
Þ Ð
ÒÑ
ÝÇÀ
Ñ
ÌÍ Ð
Ñ
É
þ
ì å
Ú
ÝÖ å
Õ
Ï
Õ
Õ
Õø
Ôè
ø
è
Õø
Ææ
ÛÉ
À
à
À
ÍÅ
Ì
Ìù
ÁÔ
Á
ÊÉ
Ý
É
Û
Þ
é
Ë÷
Ì
Þ
Ê
ÇÝÀ
Þ
ÇÝÀ
ÌÍ
É
ýü
ì
ä
ð
É
ÌÍ
Ì
ÊÉ
ÊÆ
ÛÚ
Ì
û
ô
Íú
Í
ÝÖ
ÌÍ
Í
ÌÍ
Ì
Ø
ê
æ
ø
Êè
Í
Û
å ã
å
â
Û
Ä
ÛÚ
É
ÝÖ
ÌÍ
ì
ì
ð
À
Ä
ÉÀ
Í
Æ
ùø
Æó
Ì
À
ÈÍ
âÄ
É
ÈÍ
ß
Ë÷
Ì
Þ
Ê
ÇÝÀ
Þ
ö ó
ÝÀ ä ä
ÌÍ
í ó
É Ê
Ü
ÇÝÀ å ã
ì Í
ÛÚ
è
Í
Ö
Ø ä
ì ã
Ù ß ÍÜ
Û ä è Ú
Þ
âÄ Û ò Í
Ì è
Û
ô
æ
ÉÞ
ä
ËÊ
ËÊ É
ô Æ Ú
Í
ÛÉ å
è
Á
î Ææ
Í
ÛÉ
Í ä
Ú
Íæ À
ÁÄ Ý
ì
è ÇÚÀ
À
Ú Ì ÇÝÀ
ô
ÍÅ Þ Ê
Ä
è
Ì Ì
Ê
ÆÜ
Ø
ã å Ì
ß
ä è
õ
Û
Ì ä
Û
ÚØ
ÉÞ
Û
ËÊ Ä
Æ Ì
Ëß
Ý
Ý
Í
å Ù
î
í
×
É
ÈÍ
å
ä
ÌÍ
òÍ
É
ÊÉ
Û
ÛÝ
Ì
ê
æ
Àè
è
ñè
ã×
Ø
ñ
ì
ä
ä
ä
Ê
ÄÍ
ä
Û
ÊÚ
ë
ë
Ì
í
ä
Ë
Ý
É
Í
×
è
à å Ù å
À
Ì ×
ß
×æ
ß
ä
ä
æ
ÄÍ
ÛÉ
Ææ
ÄÍ
Ý
ã
Íã
Þ
è
Ù
Engineering Workshops
å
í å å
ÇÝÀ Ü ç
ä ä
×æ
Ü× ç
ß
èÀ
å
ä
à ÉÀ
Ø À
Ì
ä
Éé
ÄÍ Æ
Ø
æ Ì å
ã× å
Ä
á
Ú
ä
É
Í ä
æ
Û ÍÜ æ
ÌÍ
Ø
è Þ
À Éé
Í Ä Æ
Ý ÌÍ å
É
ÉÌ
ê
îå
ì ä
Ù ê
Û ÊÝ
å Þ ÝÍ
Ú
ÌÍ
ß å
À
ÝÀ
Ê
ðÊ
É
ËÜ
Ý
Ý
ÇÝÀ
å Þ
ä
ï
âÄ
Û
á
Ù
Ì
Ü
Ð
Ð
Ð
Ð
Ì
ÇÝÀ
Þ
ß Ð
Û Ð
à
ÍÙ
ËÊ
Ä
Û
Ì
ÇÚÀ
ËØ
Ö×
ËÊ
ÈÉ
ÌÍ
ÑÕ
ÓÔ
ÓÑ
Ñ
ÓÑ
Ñ
Ñ
ÒÑ
Ï
Î
Î
Î
ÏÎ
ÇÆÀ
Å
¿
ÁÂÀ
ÃÀ
Ä
Verify knowledge of active source
1
On the receiver’s RP:
Verify knowledge of active source
• Recall it is MSDP's job to flood source-active
advertisements between peers so that an RP in
one PIM domain can know about active sources in
another.
• MSDP SA advertisements are accepted/forwarded
or rejected based on MSDP "peer-RPF" rules
covered earlier in this workshop.
• Remember, the information being tested against
the peer-RPF rules is the originating RP's IP address.
Not the IP of the source itself, but its RP.
• We need to trace the source-RP via the peer-RPF
rules from our receiver's RP out into our
neighbor's AS.
Engineering Workshops
1
Verify knowledge of active source
• But… how do we know the source’s RP if we
run only the receiver network?
– You may have to pick up phone and walk
them through verifying the source’s DR and
finding the group-to-RP mapping there.
– Get them to tell you they have verified the
source is sending, and the IP of their RP is
___.
– You might want to have them look to see
that they mark the mroute as a candidate
for MSDP advertisement while you're there.
(Example - next slide.)
Engineering Workshops
1
/,
+
-.,
R
R
<
09
G5
2S
G5
M
A
A
Z
R
N
8
M
HC
m
Ag
6
0
;
:
l
EU
1
5 1
9
8
<
<
G
N0
N
N
J
`
K
l
@
;
A
A
:
E
Ag
@U
-@
:
Z
R
R
89
8
GF
5
89
8
O
j
W
S
g
D
6 U
9
G
65
62
c
9i
9
89
IB
9
P 8
_
N
G
0
GF
Z
_
hg
,
0
5 ,
9
2
49
8
2b
N0
,
P
J
G
GF
5
9
c
c
U
,
U
E
9 1
F8
-0
G5
G5
J
9
89
I ,
N0
-
8
3I,
6
J
7f
Z
Z
R
6
?:
=
O
D
\
6
I
G
2
76
U
U
J
8
O
D
8
U
3I,
9S
8
P K
Q
P
2
76
5J
G
G
U
I3,
9
[
e
P K
Q
P
5J
R
R
=
=
>=
;
;:
:
Z
5
49
K
A:
<
<
<
R
?
=A
?@
?=
5
89
IB
G
Z
<
<
<
=
k
:
?=
=
9
8
<
8h
,
L
,
8
9 1
J
V
?=
I
I
5
G
9
8
P E
n
,
E
GF
5 L
N8
0
GF
9
89
5
89
5
3I,
3I,
J
J
4
6
3I,
J
3I,
6
8
F
IB
k
:
N:
?
N
U
g
U
Ag
-:
A
-
65
A
A:
@
Ag
D;
K
`
7f
?=
R
R
P
P
5
9
S
,
8
d
9H
8
c
8
7K
I
I
9
0
G
E
E
8
3F,
7D
65
2
0
6
F ,
2S
9
F
F
76
a
F
9H
B
Z
R
R
P
P
89
I
49
09
a9
5
65
G
GI
8
8
W
S
,U
U
O
H
b
b
,
`U
OC
D
`
Z
R
^
[
5
49
K
J
3I,
,
I ,
6
C
7H
I
I
\
_6
5
]
R
R
R
P
P
P
\
8
[
[
,
L
89
9
9
58
I
G
5
F
OC
D
D
K
S
CS
U
3I,
6
09
8
P K
Q
CS
K
R
R
R
R
7
9
09
I
5
G
6F
8
8
HC
H
C
C
Y
G
5
0
89
J
J
9H
P M
P
P
P
P
Y
E
,
,
L
U
E
Z
R
R
R
R
R
J
,U
E
D
F
I
6I
9
89
W
P W
X
2
76
5V
P0
2
76
5V
S
S
U
5 ,
T
P T
09
G5
2S
P S
09
I
9O
P O
Q
N0
G
M
E
8
H
G
K
L
9 E
J
8
3I,
76
0
G
8
3F,
7D
BC
<
<
<
<
<
<
1
76
45
89
?
=A
?@
?=
=
?=
=
=
>=
;
:
:
;:
32,
0
/,
+
-.,
r
On the source’s RP to show generating an SA:
u
st
vw
p o
q
Verify knowledge of active source
1
Engineering Workshops
1
Verify knowledge of active source
• Now we have the source/originating RP's IP
address.
• The idea here is we are trying to figure out
which of our MSDP peers we should expect to
get knowledge of the actual source from.
– If the source RP is an MSDP peer of our RP, the

{
¥
¨
¦§
¤

£ ¢
{
¡

{

¬
«ª
©ª

{

xz
~

{

~ }
{
|
z y
{
x
source RP is the RPF peer.
– If we look at
, the
MSDP peer in the adjacent AS is the RPF peer.
– In practice, in most campus networks,
and
will usually get you going in the right
direction.
Engineering Workshops
Ò
Á
Ê
à
È
¶
Â
ßµ
Â
Á
Ë
Å
Û
Ê
ÏÛ
É
ÏÛ
Ï
ÖÅ
®
Ä
Ä
Âä
¼
¼
¼
´
Ú
Ú
Ú
Ú
ß
°
´
Å
½
¸½
3Î´
Ë
¯
¶Á
¯
¶
Ë
Ë
Ë
Å
½
½
½
¸½
Æ
Ç
´
Æ
µ Æ
¯
Â Æ
Î
¶
Ä
¿Ù
ã

Á ´
¶
â
·
º
º
º
º
º
º
º
º
º
Ä
¶Â
½
½
Ê
¸
à
Ê
¹Û
¸
à
ÓÊ
É
É
Ó
Ó¸
É
Ó
Ó¸
É
Ê
¸
¼
¼
¼
Í
Ì
·
¸
¼
¼
¹¼
¸
ÓÛ
É
¸
Û
Ê
Á
ÓÛ
É
¸
¸
Û
Û
Ê
É½
ÏÉ
Ï
ÓÛ
É
¸
¸
¸
½
É½
È
à
È
ÏÛ
ÖÅ
Ï
®
Ä
Ä
Âä
¼
¼
´
ß
°
Î
¿Ù
ã
¶Á
¯
¶
¯
²Ë
Ë
Ë
Ë
Å
Å
¹
½
¸½
½
Ñ
Æ
Ç
´
Æ
3µÆ
Â Æ
Ú ¯
Ú
Ú
Ú
´

Á ´
¶
â
·
Ä
¶Â
½
½
¼
É¹
¼
¹¼
½¼
È
¼
¼
¼
¼
½¼
¹¸
Ó¸
É
Ê
¸
¹¸
Ó¸
É
º ¸
»
º
º
º
º
º
º
º
º
Ê
¸
Í
Ì
·
¸
¸
¼
¹¼
¸
ÓÛ
É
É
Û
Û
¼
¼
²Ë
ÒÄ
µ
³
Ñ
3Ä´
Ò
Ï
®
²
µ
¶Â
¶
µ

Ë
Ë
ÂÅ
Ë
Ç
Å
²3´
¶
Ë
á
° Ç
Ú
Ú
Æ
´
Ì
Å
Å
±
¸
Ñ
Æ
¯
Ë
À
Þ
Þ
ÜÝ
à
½¼
½
Æ
Í
Æ
¯ Ñ
²Ë
Ë
Ï
ß
¼
¯ Ñ
¯
°¯
Ï
Å
² ³
¯
¿
Å
ÁË
Ë
¹¸
½
½
°
Á
Ë
¼
Ô
È
ÓÛ
Ó
¸
ÓÊ
Â ´
¶²
Ú ¸È
»
º
º
¹¸
º ¸
»
¶
Â ·
Å
Æ
¯ Ñ

Å
Á ´
¶Â
Ö¶
®
¿Ù
Ø
µ
Ä
¹¸
º
¼
¹¼
¸
½¼
¹¸
º ¸
»
º
³
Ñ
3µ´
²±
¯
°¯
®
Ç
Ë
Ë
Ë
Ë
Å
Ç
Å
²Ë
Æ
¯ Ñ
²
²
¶Â
¯Î
® Õ
µ
²
Â Æ
Â
¶
¶
¶
Òµ
Î
¯Á
² ´

3Á´
Â×
·
Á
Â
Î
Á
ë ê
ì
é
çè
æå
½
Â ´
¶²
ÎË
ÏÅ
®
®
¶
À
¾¿
µ
Ä
ÏË
ÖÅ
µ
À
¾¿
Ñ
Å
¯²
Ä Ô
Ë
º
º
¸È
½
½
¹¸
º ¸
»
¹¸
¼
Ô
Ï Õ
®
¶Â
À
¾¿
¯Á
º
º
º
°
Â
¯Á
º ¶
º
À
´
Æ
³Ñ
´
½
¹¸
Ó¸
É
Ê
¸
¼
¼
Í
Ì
Æ
Â ³
Î
¯Á

º Ë
Ò
Ò

Ë
ÁË
¶
Ä
Ï¶
Â
Á
¾¿
Ë
¯Á
ÏË
¯Î
¶
Å
Ð
Ê
¸½
Á ´
À
¾¿
Æ
·
º
º
º
º
º
º
Á
¯µ
¶
¯
²
²
Ë
Å
Å
Å
¼
É¹
¸
¼
¹¼
½¼
È
½¼
Ê
É½
Í
Ì
Æ
ÈÇ
3²Æ
¶
ÃÂ·
Á
Å
Â ´
¯
¶Ä
ÃÂ·
3Á´
À
¾¿
º
º
³
¶µ
¹¸
¸
¹¼
½¼
¹¸
¼
3µ´
²±
º ¸
»
¯
°¯
®
·
Verify knowledge of active source
1
Engineering Workshops
G
F
MP
S
QE
P
RQ
M
K
LM
M
RQ
M
LK
NOM
FL
J
HI
F
ED
•
@
&C
$6
AB@
-= ".
-= <
-.
0=."
<*
=
-#;
= !
"*
= !
"*
= ?0="
0=-"
/ +$
?= ?0="
0=-"
/ +$
?-
+ + ".
= .
0="
<= -/ -:
".
/
.; #" %
<# +)2
0="
-" $6
/ * BA@
7?- @
&C
= ?= "
= ?>
?-*
= .
0="
<= -/ -:
/
.; #" %
<# +)2
0="
?-# $6
/ BA@
7?- @
&C
= ?= "
= ?>
?-*
".
?-
"*
-#;
0=."
<?
=
= !
= ?0="
0=-"
/ +$
+ = .
0="
<= -/ -:
/
.; #" %
<# +)2
0="
</ *> $6
BA@
7?- @
&C
= ?= "
= ?>
?-*
".
?-
"*
This is a circular buffer, so it's hit-or-miss...
On a Juniper, turn on MSDP traceoptions and search the file.
Engineering Workshops
*
""
= *
= *
?"
""
= >
= ?0="
0=-"
/ +$
+ = .
0="
<= -/ -:
/
.; 543 &
#" <# % 0=" 4
-/ *> 6 7
%)
8
?-7 $
= - /
?- 9
= " $
?=> 8)6
-?* :
ü
ú
ûÿ
!"# ! $ % $ '( &
')
$
+,**
+,**
/0-.
(1% &
+
"*
**
2
$ &
ÿú
%
û÷
þ
ÿ
ÿ
ÿ
ú
ö÷
ú
ÿö
þ
þû
ýüõû
ø
õ
øù
õ
ô òó
ñð
ïî
í
•
+2
%)
Verify knowledge of active source
1
Assuming we do not have an entry for the source and group
in our receiver RP's SA-cache, we might be able to see if we
are getting a reasonable SA advertisement but rejecting it:
Verify knowledge of active source
• If we are getting an SA from what we think should be
the RPF peer, yet rejecting it, we need to work
through the MSDP peer-RPF rules to figure out why.
Possible reasons:
– We've configured to use only the multicast RIB, yet
we have no MBGP route to the originating RP.
Check that the source network is advertising the
route to the RP in MBGP and we are accepting it
(policy misconfigurations).
– We have MBGP running, but not MSDP, with a
peer that appears to have a better route to the
originating RP than who we think is the RPF peer.
– incorrectly configured default peer.
– bugs, voodoo, who knows!
Engineering Workshops
1
Verify knowledge of active source
• Assuming you are not getting an SA from the
peer you think should be the RPF peer, you
may need to open a ticket with your upstream
provider or peer. You can give them the
following:
– We are not getting an SA for <source IP address>
– The group address is <group address>
– The source’s RP is <source RP IP address>
– We expected to get this from
<MSDP peer’s IP address>
• Also report if you are not getting the MBGP
Engineering Workshops
route.
1
Verify knowledge of active source
• Other than just turning the problem over to your
upstream provider, for many Internet2
campuses, Abilene core routers will be in the
path.
• It is sometimes helpful to go to the router proxy
closest to the source and check for the SA-cache
entry for the source/group in question there.
• If there is no entry there, it is not too surprising
your campus is not getting a valid SA. (We have a
screenshot at the end of these slides.)
http://loadrunner.uits.iu.edu/%7Erouterproxy/abilene/
Engineering Workshops
1
Verify knowledge of active source
• Since you have already checked your path back
1
from the receiver to your RP, you should then
get (S,G) state on the receiver’s DR when you fix
rejecting a received SA, or your upstream
provider or peer resolves the ticket concerning a
missing SA.
Move on to step 4…
Engineering Workshops
f
j
Uq
YW
iU W
p
`a_
[
kU
[
^]\m
n
fY
[U
fY
[
`
Y
Znm
kZYl
_
]j
i`
`
d_
o
[\
Yc
g\
Y
fW
U
iW
Y[
]\
YW
[\
de_
Y[
fY
gh\
Y
[
Yc
]
`W\
U
[b
]^\
`a_
[
UVT
W
ZYX
Overview Refresher!
1
Engineering Workshops
s
r
1
Engineering Workshops
ÿ
è
ð
ð
ð
Ù
ä
í
õ
öÐ
Ù
×Ð
ÖÔÕ
ò
÷
÷ø

(

t
w
vtu
}
V|{
z
yx

¡¢ V~

£
¤¥¦ |

©ª¨

ª¬«

We now have (S,G) state on the receiver’s DR.
•
Next, we need to check to see if traffic is actually flowing…
(Cisco example)
If this is zero, you still have a problem.
Engineering Workshops
ó
Ð Ï ´µ,³ §
¢
ÒÓ Ñ ¶µ
·

ÖÔÕ »º¸¹

¼ ¯®
× ÐØ ¾ ½´
º ±
° §
Ù
¥¦
5ÕÚ ÁÀ ¿
ÐÛ
Â ²
Þ ÜÝ ¿ Ã
¢
ßà ¾º ½´
Û
á ¿
Ä
â ÂÃ
ÜÝ À ¿ ÐÏ

ÒÓÑ µÅÆ
À
Ó ÇÆ ¢
ã
Õä ´º
Ø

¸
å OÉÈ Ä
æÏ
ç¯à ¼
Ö ÂÃ
× ¿
Ü ËÊ¹
èéÓ Ü
Û Æ
ÌÄ
Ð ¹Í
êçÒ ´
Ü Î¹
ëâ À ¿
âÐ
è µÅÆ
Ò
Ð À
Ô ÇÆ
ì
•
ù
ùõ
ó
è õ
Ó
í
ÖÕ
ä
ä
ò
Ù ðñîïï
úò ð
î
ò
óò
úò ð
ò
î
è ò
úû
Ö ôÕ
ò Ó
úõ
Ý
ï ÔÒ
î
ÖÔÕ
þýü
×Ð
ÿ Ù
ÔÒ
Ó
Ö ôÕ
ÖÕ
Ó
Trace forwarding state back
1
%&
'(
)*
-,+
"#!
$ $ <. "" C2 3" K
/$ . $2 G )
%$ )2 2 = 2$ = C 8
+ =12
M H 1 %2
3" D '
N . +I
+ 1 +I
33" . 6 +
O " L**5
7
6
J
:4
+ 6
66
< ?F : G
4;
1 E
+ 2H
"
5I D
4 1
D 0 E$
*:+ 9 8
:* 0 . C
6 * $
:* 12
6 4 3$
:; 2
455 B$ <
. ) 0
6
6 4 3 =
7*
>?
6 *
*
@A
:*
8
6 4
:;
6
:*+
2
92 "
2 6 *
:*
9
3$2
' 12
0$
".
%&
'(
)*
-,+
"
.&
/
0)
1"
$. 0
2
%$.
0
3
455
6
6 4
7*
6 *
*
"#!
$ Trace forwarding state back
1
• Here’s how to check if traffic is flowing on a Juniper:
If this is zero, you still have a problem.
Engineering Workshops
u
q
fe
in X
V Tu
rZ
i
RS
lS
jP
SZ h m
RP
^h
Sf
oVT
^Z
Sf
^ ik
V
l
\v p
^ b
`b
^ `
``
^ `
w]
q
s k Z [V
fS Si
aat j f
jk
l Xm
^ nVf
oP
^S
RX
^ Ro
nm
R
]\ p
_^\
]\
^ `
^ ]a
]\b
rn
So
Z[
e
fX
nh
^ `
^ ]a
]\b
]\
^_\
]\
ZY
[

~
}
On a Cisco:
nP k
RY y
P
So
ZV
PP
Sh
V kV
nZ [
Zn m
Zn
jY
on
Sf
PhX
m
fiX
cd cd
•
cd cd dc
e e e
Zn zY h ZV
RoZ rn Rn h
P XV f xg
f R X SP
oV oSP yr
Rf h ^ b
rh V p ^ b
b PY [ ^ bb
a{ bx
vq
•
|V
cd QP
e RS
fX P U T
[V
Zg PVT
Sh W
f XV YX
Trace forwarding state back
1
Start on your receiver’s DR.
This time, RPF back towards the actual source IP address (as
opposed to the source RP).
Engineering Workshops

_
¨

¡
¢
£
¥¤
•

¯¤ ¬
¤

# £
#
¥ ¥¤ ¥ ©

ª
§° ¤ ¡«

(
(

¨ ©

- ©
¤ -

¦§
± ®

´
On a Juniper:
·
µ¶
²³
Trace forwarding state back
1
You are looking to see how you are expecting the
SPT tree to be built, where you actually expect
the packet flow to come from.
Engineering Workshops
*
)
)
(
(
(
'
"
(
'
"
"
!
)
'
"
(
"
*
*
"
"
"
"
"
"
6
$
;
:
9
8
6
6
%
7
%
6
(
2
-
&5
%
(
+
#.
-
!
)
)
#-
+ *
,
(
'
!
"
!
Engineering Workshops
"
!
'
!
$
%
1/
0
%
%
-
23
0
/
$
4
0
/
/
&
þý Ù ÑàÓ
ÿ
ü ó Ú íÓ
ùúðð ëìç ê
ðîù áÓ ÕÚ
ûï
ìøô çÝæ
ðà éèã Õ
Ó
îï ÒÓ ÐÏ º¸¹
ñòð #ÖÕ Ñ »
ðñ óó ÙÚ ×Ø ÔÓ#ÖÒÕ ¸ ½¼
óó ß ×Ø ¾
ñ ó ààÚ ÙÚ À¿
ôõ ááÓ ÝÜÛ Á
Ø ÃÂ
#ÓÞ Ã
Ð
íÞÜ â
ÄÂ
â ö ÚÓ ã
ÅÆ
ö÷ äÜ
ÇÂ
åÓ
À ¿È
É
ÌÊ Ë
ÅÍ
ÍÎ
•
3
#
Trace forwarding state back
1
Cisco:
Work your way back towards the source IP, looking for PIM
problems along the way.
Juniper:
Trace forwarding state back
1
K
@
<L
M
>
=
K
L
I
B=
>
>
IJ
K
?
>
@
I
<L
>G
>G
B=
<=
<=
IJ
K
@?
@?
>
<=
H
<=
H
E
G F
D
E
G F
D
= C
D
B
= C
D
<
<
>
<=
@?
A
>
<=
@?
A
B
• Log into that upstream router and check state there with:
R
NZ
R
T
`
a Y
_
R
S
RQ
P
TO
S
]
R
NZ
P
TO
^
P
NO
^
\
]
P
NO
\
RQ
P
NO
Q
T
[
Z Y
D
X
Q
P
W
T
S
N
O U
D
V
P
NO
RQ
• Or (Juniper):
• Look to see if the downstream router is in the outgoing
interface list, and to see if you see a positive traffic rate.
• Hopefully you will work your way back to a router that is
seeing the traffic flow.
Engineering Workshops
Å

ÌË
¡

£
±
°

º»
µ
·
¶µ
ÇÆ
Engineering Workshops
ÏÎ
ÏÎ
Í
Í
Á
¸¹
ÄÂ
¿Ã
Â
ÃÂ
À¿
ÁÀ
¾
¾½
´³²

É È
Ê

¢
¯

¨
£

}

£ª

}

}£

¢
}
£¤

£¦
¥
e

§
1©

~
|}
t
t
t
t
t
t
h
{
on
z
p
qp
on
lm
r
wy
u
wx
wu
u
wu
u
u
vu
r
s
rs
j i
k
g
cf
b
dec
üï
î
ö
þ
ø ø
ü
î
ü
ñ ÷
îò
ê
ø
ö
ê
ï ì
ë
îí
ü
ò ì
ë
ö
ó
ò ì
ñó
é û
ë
ý
ÿ
ø
ö
÷
è
÷
ðç
è
÷
çð
õ
õ
ê
òì
ë
ö
ó
ò ì
ó
ü
ò
îò
îí
ü
ñ
í
ò æ
ý
ø
ø
ïó
î
ô
ø
ê
ý
ÿ
ë
îí
ü
ö ö
ò æ
ë
ö
ë
ö
1ñè
îë
ö
ü
ñ
ñ þ
ë
ø
ö
îë
ö
ï
îí
ü
ò æ
ë
ö
ý
ý
ï
üö
ö
ì
ì
ë
ø
üï
ÿ
ø
ö
þ
þ
ê
í
ï
ëì
ö üö
î
ü
î
ïó
î
îí
ü
ò ì
ë
ö
ó
ò ì
ý
ø
ùí
ý
é
ïó
î
ñó
ö
é û
ë
ì
ô
ú
ë
÷
ê
é
ë
î
ó
õ
ö ê
Ê
ô
ò ì
ë
1ñð
îï
í ì
ëê
éè
æç
à
à
à
à
à
à
Õ
ÝÜ
ÛÚ
ØÙ
Þ
ãå
á
ãä
ãá
á
ãá
á
á
âá
Þ
ß
Þß
× Ö
k
Ô
ÑÓ
Ð
ÒeÑ
Trace forwarding state back
1
o
_S
S
S_
IV
D
O
Z
v
ru
t^
e
_
Qn
UD
Q
U
Z
Z
DZ
IV
E
D
n
nF
_
_V
Qn
C
]
O
sD
QO
F
G
F
F
YG
T
T
d
m
qpV
o
U
IV
c
rb
h
h
C
i
j
l
c
j
e
el
b
b
lb
b
ej
ib
jk
Y
f
h
h
gb
e
e
e
fe
b
ck
d
d
d
d
d
jk
j
D W
Z
nF
OV
E
DV
OZ
_S
d
c
cb
V
V
T
Z
W`
Y
V
J
^
F]
E
U
E
V
Z
Z
M
F
a
_ F
_Z
U
D[
\
K
O[
RQ
R
U
X F
SO
U
O
RQ
S
RQ
PON
W
HI
E
LM
JK
DC
FG
=
=
=
-
#
@
@
7
;
9
(
!
=
=
=
>0
Engineering Workshops
0
0
;
<
:
4
+*
9"
4
"
"!
;<
7
7
*
$(
8
7/
%
0
A1
AB
A0
0
A0
!?
6
0
231
4 ,
5
(
/
9"
9
9
:*
98
*
4
(
"
- "
.
+
$*
)('
,
!
& !
$%
#"
!
Trace forwarding state back
1
Juniper:
Trace forwarding state back
•
If you get to a point where the upstream router
IS showing it is receiving the packets, but your
downstream is not, you need to figure out why
those packets are getting lost.
• ACLs?
• Broken IGMP snooping switch in the middle?
• PIM problem?
Engineering Workshops
1
Trace forwarding state back
•
You may work this back to the edge of your area
of responsibility, and may have to open a ticket
with your upstream to continue the process
towards the source. Give them:
• The active source IP address
• The group address
• The circuit / link towards which your router
•
has sent the (S,G) join
The fact that you are not receiving packets for
that (S,G) on that shared link.
Engineering Workshops
1

x
|z
xz

~
x
~

|
~x

|
~

|
z

}
|
}

~
|

|~
z
|z

~
|

|
~|

~
|

x
~

~
| {
}
z
x w
y
Summary
1
Engineering Workshops
1

}

y
Summary
•Pick a direction
•Active source and receiver IP addresses
•Group address
Engineering Workshops
1
¡£
¨ ¤

¡£ ¢
¤
¦¥
£
Summary
•Identify the DR for the receiver.
•Verify the DR knows of interest in that group.
•Check that the DR is not receiving traffic.
Engineering Workshops
1
°
´

¸ ·
}
¶

º
¾ ´®
µ´
³ ²
}
½
¯
±°

»
¼¯
¹º
® ¬
¢
Summary
mean fixing multicast reachability
• Might
topology or PIM state.
• Probably will involve MSDP SA debugging.
Engineering Workshops
1
Ê
É ÈÇ

À
Î
Â
Á
ÃÌ
ÁË
Ì
Í
Æ
Á
Å Ä

À
ÂÃ
ÀÁ ¿
¢
Summary
•Trace forwarding state from receiver’s DR.
•Work towards the actual source.
•Verify reachability, PIM state, and whether
traffic is flowing at each step.
Engineering Workshops
A word on troubleshooting
performance problems...
• Performance problems in multicast inherit virtually all the
problems associated with unicast performance issues, which
you know how to troubleshoot:
• packet loss due to congestion.
• latency/jitter due to queueing, traffic shaping devices,
interleaving, etc.
• duplex problems, cable issues, etc.
• Users often neglect to look at their host performance. Video
apps can drive the CPU to where it cannot handle the load.
• It is usually more fruitful to look to the above issues before
spending a lot of time looking at timers and such in
multicast protocols.
Engineering Workshops
1
1
Tools
• Beacon
http://dast.nlanr.net/Projects/Beacon/
– The beacon is an application to monitor multicast
reachability and performance among beacon-group
participants. Participants both send and receive on a
known group.
– The results are displayed with receivers on the hosts as
receivers on the vertical axis and sources on the horizontal
axis.
– There is a new version out that is RTP-based. By default
it runs on a different group than old beacon, and runs at a
much slower rate (~1pps vs. ~9pps for old beacon):
http://dast.nlanr.net/Projects/Beacon/newbeacon/
Engineering Workshops
Tools
1
http://dast.nlanr.net/Projects/Beacon/
Engineering Workshops
1
Tools
• If the beacon is broken, that gives you higher
•
•
•
•
confidence the problem is not just user error or host
issues.
It is sometimes possible to use the beacon as the
constantly active source and receiver for debugging.
However, many times the beacon can be fine yet
multicast is broken for a different group.
It will not catch new/transient problems with source
knowledge or state creation (the tree has been built).
Encourage sites you collaborate with to participate in
a beacon group!
Engineering Workshops
Tools
•
1
Example: GEANT http://beaconserver.geant.net:9999
Engineering Workshops
Tools
1
• Some web tools exist to look at peer’s routers.
• Again, the Abilene router proxy:
http://loadrunner.uits.iu.edu/%7Erouterproxy/abilene/
• Also, some looking-glass pages include multicast
information as queries you can run:
http://www.nordu.net/connectivity/looking-glass/lg.cgi
• You can get the proxy code free from IU after signing a
license agreement. You can freely download the
looking glass code and modify it yourself if you would
like to make your network visible to others.
Engineering Workshops
Tools
1
Engineering Workshops
Tools
1
Engineering Workshops
1
Tools
• rtpqual
ftp://ftp.ee.lbl.gov/rtpqual.c
– Simple Multiprotocol Multicast Signal Quality Meter
– very useful for establishing a receiver (even if the
multicast is not using RTP)
– also useful for finding packet loss problems and
whether they are periodic or not
Engineering Workshops
Information Online
1
•
tutorial-style paper at:
http://multicast.internet2.edu/almeroth.pdf
•
http://www.ncne.nlanr.net/documentation/faq/mcast_eng_faq.html
•
http://dast.nlanr.net/Projects/Beacon/
•
GEANT: http://www.dante.net/nep/GEANT-MULTICAST/
links to some troubleshooting docs and monitoring tools
•
ftp://ftpeng.cisco.com/ipmulticast.html
•
http://www.sprint.net/multicast/faq.html
•
Abilene router proxy:
http://loadrunner.uits.iu.edu/%7Erouterproxy/abilene/
Engineering Workshops
1
Lab 5: MSDP & Interdomain ASM
Engineering Workshops