How to work BGP synchronize and next hop self with Nexus?

How to work BGP synchronize and next hop self with Nexus?

 

I have already written about BGP concept simply in this another blog in Korean. For the BGP synchronization, I have also written in this blog in Korean. For Next hop self concept, this blog will be helpful. Please do not worry even if you can not read Korean. In this post, I will reproduce this synchorization and next-hop-self with nexus switches.

 

1. Pre-requisite

 

This is my environment for this post. I will configure step by step with below. For "OSPF1 + AS1" part, 

s1 (lo1 110.0.0.1)	s2 (lo1 110.0.0.2)	s3 (lo1 110.0.0.3)
feature ospf feature bgp interface Ethernet1/2   no switchport   ip address 100.21.1.12/16   ip router ospf 1 area 0.0.0.1   no shutdown interface Ethernet1/3   no switchport   ip address 100.31.1.13/16   ip router ospf 1 area 0.0.0.1   no shutdown interface loopback0   ip address 110.0.0.1/32   ip router ospf 1 area 0.0.0.1 router ospf 1   router-id 110.0.0.1	feature ospf feature bgp interface Ethernet1/1   no switchport   ip address 100.42.2.11/16   no shutdown interface Ethernet1/2   no switchport   ip address 100.21.2.12/16   ip router ospf 1 area 0.0.0.1   no shutdown interface loopback0   ip address 110.0.0.2/32   ip router ospf 1 area 0.0.0.1 router ospf 1   router-id 110.0.0.2 router bgp 1   router-id 110.0.0.2   log-neighbor-changes   address-family ipv4 unicast     network 110.0.0.2/32   neighbor 100.42.4.11     remote-as 2     log-neighbor-changes     update-source loopback0     address-family ipv4 unicast   neighbor 110.0.0.3     remote-as 1     update-source loopback0     address-family ipv4 unicast	feature ospf feature bgp interface Ethernet1/1   no switchport   ip address 100.63.3.11/16   no shutdown interface Ethernet1/3   no switchport   ip address 100.31.3.13/16   ip router ospf 1 area 0.0.0.1   no shutdown interface loopback0   ip address 110.0.0.3/32   ip router ospf 1 area 0.0.0.1 router ospf 1   router-id 110.0.0.3 router bgp 1   router-id 110.0.0.3   address-family ipv4 unicast     network 110.0.0.3/32   neighbor 100.63.6.11     remote-as 3     update-source loopback0     address-family ipv4 unicast   neighbor 110.0.0.2     remote-as 1     update-source loopback0     address-family ipv4 unicast

For "OSPF2 + AS2" part, 

s4 (lo1 110.0.0.4)

s5 (lo1 110.0.0.5)

feature ospf feature bgp interface Ethernet1/1   no switchport   ip address 100.42.4.11/16   no shutdown interface Ethernet1/2   no switchport   ip address 100.54.4.12/16   ip router ospf 2 area 0.0.0.2   no shutdown interface loopback0   ip address 110.0.0.4/32 router ospf 2   router-id 110.0.0.4 router bgp 2   router-id 110.0.0.4   log-neighbor-changes   address-family ipv4 unicast     network 110.0.0.4/32   neighbor 100.42.2.11     remote-as 1     log-neighbor-changes     address-family ipv4 unicast   neighbor 110.42.2.11     address-family ipv4 unicast

feature ospf feature bgp interface Ethernet1/2   no switchport   ip address 100.54.5.12/16   ip router ospf 2 area 0.0.0.2   no shutdown interface loopback0   ip address 110.0.0.5/32   ip router ospf 2 area 0.0.0.2 router ospf 2   router-id 110.0.0.5

For "OSPF3 + AS3" part, 

s6 (lo1 110.0.0.6)

s7 (lo1 110.0.0.7)

feature ospf feature bgp interface Ethernet1/1   no switchport   ip address 100.63.6.11/16   no shutdown interface Ethernet1/2   no switchport   ip address 100.76.6.12/16   ip router ospf 3 area 0.0.0.3   no shutdown interface loopback0   ip address 110.0.0.6/32 router ospf 3   router-id 110.0.0.6 router bgp 3   router-id 110.0.0.6   log-neighbor-changes   address-family ipv4 unicast     network 110.0.0.6/32   neighbor 100.63.3.11     remote-as 1     address-family ipv4 unicast

feature ospf feature bgp interface Ethernet1/2   no switchport   ip address 100.76.7.12/16   ip router ospf 3 area 0.0.0.3   no shutdown interface loopback0   ip address 110.0.0.7/32   ip router ospf 3 area 0.0.0.3 router ospf 3   router-id 110.0.0.7

This is simple configuration. At this time, it is not perfect in traffic flow. In some case, it can not send each other. For example, S4 (110.0.0.4) and S6 (110.0.0.6) can not transfer the packets each other.

 

2. About BGP synchroization.

 

To verify the BGP synchronization, I have to look the BGP table on S3 switch.

s3# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 16, Local Router ID is 110.0.0.3 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>i110.0.0.2/32       110.0.0.2                         100          0 i *>l110.0.0.3/32       0.0.0.0                           100      32768 i   i110.0.0.4/32       100.42.4.11                       100          0 2 i *>e110.0.0.6/32       100.63.6.11                                    0 3 i

This is so strange. Because I did not "no synchorization" in BGP configuration, "110.0.0.4" should not be displayed. In this blog, the concept of the synchronization will be explained.

I used nexus switch image for this generation. In this instruction, there are default settings. By these values, the BGP table on S3 switch displayed S4 information.

 

3. About Next-Hop-Self.

 

This is not vaild status, even if the BGP table show S4 information. It looks like normal. 

s3# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 16, Local Router ID is 110.0.0.3 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>i110.0.0.2/32       110.0.0.2                         100          0 i *>l110.0.0.3/32       0.0.0.0                           100      32768 i   i110.0.0.4/32       100.42.4.11                       100          0 2 i *>e110.0.0.6/32       100.63.6.11                                    0 3 i

I need to check the "Next Hop" status. "100.42.4.11" is the IP address between S2 and S4. In the routing table on S3, there is no information.

s3# show ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF  100.21.0.0/16, ubest/mbest: 1/0     *via 100.31.1.13, Eth1/3, [110/80], 1d03h, ospf-1, intra 100.31.0.0/16, ubest/mbest: 1/0, attached     *via 100.31.3.13, Eth1/3, [0/0], 1d04h, direct 100.31.3.13/32, ubest/mbest: 1/0, attached     *via 100.31.3.13, Eth1/3, [0/0], 1d04h, local 100.63.0.0/16, ubest/mbest: 1/0, attached     *via 100.63.3.11, Eth1/1, [0/0], 1d04h, direct 100.63.3.11/32, ubest/mbest: 1/0, attached     *via 100.63.3.11, Eth1/1, [0/0], 1d04h, local 110.0.0.1/32, ubest/mbest: 1/0     *via 100.31.1.13, Eth1/3, [110/41], 1d03h, ospf-1, intra 110.0.0.2/32, ubest/mbest: 1/0     *via 100.31.1.13, Eth1/3, [110/81], 1d03h, ospf-1, intra 110.0.0.3/32, ubest/mbest: 2/0, attached     *via 110.0.0.3, Lo0, [0/0], 1d04h, local     *via 110.0.0.3, Lo0, [0/0], 1d04h, direct 110.0.0.6/32, ubest/mbest: 1/0     *via 100.63.6.11, [20/0], 1d02h, bgp-1, external, tag 3

It is not valid because the nexthop is not valid. To resolve this, there is the 2 ways. First, add the static route. Second, Next-hop-self.

# Add static route on s3 s3(config)# ip route 100.42.0.0/16 110.0.0.1

And then, I can see the BGP table. The status should be changed by valid.

s3# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 17, Local Router ID is 110.0.0.3 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>i110.0.0.2/32       110.0.0.2                         100          0 i *>l110.0.0.3/32       0.0.0.0                           100      32768 i *>i110.0.0.4/32       100.42.4.11                       100          0 2 i *>e110.0.0.6/32       100.63.6.11                                    0 3 i

Now, I will configure "next-hop-self" on s2 switch after remove static route on s3 switch.

# Remove static route on s3 s3(config)# no ip route 100.42.0.0/16 110.0.0.1    # Add next hop self configuratoin on s2 s2(config)# router bgp 1 s2(config-router)# neighbor 110.0.0.3 s2(config-router-neighbor)# address-family ipv4 unicast s2(config-router-neighbor-af)# next-hop-self s2(config-router-neighbor-af)# end

Now look at the s3 switch with "show ip bgp" command

s3# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 19, Local Router ID is 110.0.0.3 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>i110.0.0.2/32       110.0.0.2                         100          0 i *>l110.0.0.3/32       0.0.0.0                           100      32768 i *>i110.0.0.4/32       110.0.0.2                         100          0 2 i *>e110.0.0.6/32       100.63.6.11                                    0 3 i

There are 2 change. First, the status is changed as the valid. Second the next hop ip address is changed from "100.42.4.11" to "110.0.0.2". This "next-hop-self" feature make the "ebgp" existance clear. Now I will do again in s3 switch for next step.

s3(config)# router bgp 1 s3(config-router)# nei 110.0.0.2 s3(config-router-neighbor)# address-family ipv4 unicast s3(config-router-neighbor-af)# next-hop-self s3(config-router-neighbor-af)# exit

 

4. Troubleshooting.

 

Look at routing table and bgp table on the s4 switch and s6 switch. From s4 switch,

s4# show ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF  100.42.0.0/16, ubest/mbest: 1/0, attached     *via 100.42.4.11, Eth1/1, [0/0], 1d04h, direct 100.42.4.11/32, ubest/mbest: 1/0, attached     *via 100.42.4.11, Eth1/1, [0/0], 1d04h, local 100.54.0.0/16, ubest/mbest: 1/0, attached     *via 100.54.4.12, Eth1/2, [0/0], 1d04h, direct 100.54.4.12/32, ubest/mbest: 1/0, attached     *via 100.54.4.12, Eth1/2, [0/0], 1d04h, local 110.0.0.2/32, ubest/mbest: 1/0     *via 100.42.2.11, [20/0], 1d02h, bgp-2, external, tag 1 110.0.0.3/32, ubest/mbest: 1/0     *via 100.42.2.11, [20/0], 1d02h, bgp-2, external, tag 1 110.0.0.4/32, ubest/mbest: 2/0, attached     *via 110.0.0.4, Lo0, [0/0], 1d04h, local     *via 110.0.0.4, Lo0, [0/0], 1d04h, direct 110.0.0.5/32, ubest/mbest: 1/0     *via 100.54.5.12, Eth1/2, [110/41], 00:40:49, ospf-2, intra 110.0.0.6/32, ubest/mbest: 1/0     *via 100.42.2.11, [20/0], 00:03:43, bgp-2, external, tag 1 s4# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 12, Local Router ID is 110.0.0.4 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>e110.0.0.2/32       100.42.2.11                                    0 1 i *>e110.0.0.3/32       100.42.2.11                                    0 1 i *>l110.0.0.4/32       0.0.0.0                           100      32768 i *>e110.0.0.6/32       100.42.2.11                                    0 1 3 i

From s6 switch,

s6# show ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF  100.63.0.0/16, ubest/mbest: 1/0, attached     *via 100.63.6.11, Eth1/1, [0/0], 1d04h, direct 100.63.6.11/32, ubest/mbest: 1/0, attached     *via 100.63.6.11, Eth1/1, [0/0], 1d04h, local 100.76.0.0/16, ubest/mbest: 1/0, attached     *via 100.76.6.12, Eth1/2, [0/0], 1d04h, direct 100.76.6.12/32, ubest/mbest: 1/0, attached     *via 100.76.6.12, Eth1/2, [0/0], 1d04h, local 110.0.0.2/32, ubest/mbest: 1/0     *via 100.63.3.11, [20/0], 1d02h, bgp-3, external, tag 1 110.0.0.3/32, ubest/mbest: 1/0     *via 100.63.3.11, [20/0], 1d02h, bgp-3, external, tag 1 110.0.0.4/32, ubest/mbest: 1/0     *via 100.63.3.11, [20/0], 00:10:28, bgp-3, external, tag 1 110.0.0.6/32, ubest/mbest: 2/0, attached     *via 110.0.0.6, Lo0, [0/0], 1d04h, local     *via 110.0.0.6, Lo0, [0/0], 1d04h, direct 110.0.0.7/32, ubest/mbest: 1/0     *via 100.76.7.12, Eth1/2, [110/41], 00:44:12, ospf-3, intra s6# show ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 15, Local Router ID is 110.0.0.6 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2    Network            Next Hop            Metric     LocPrf     Weight Path *>e110.0.0.2/32       100.63.3.11                                    0 1 i *>e110.0.0.3/32       100.63.3.11                                    0 1 i *>e110.0.0.4/32       100.63.3.11                                    0 1 2 i *>l110.0.0.6/32       0.0.0.0                           100      32768 i

By these informations, I may send traffic each other. However, I can not. Please look at the below, I send the traffic from s4 to s6.

s4# ping 110.0.0.6 source-interface loopback 0 PING 110.0.0.6 (110.0.0.6): 56 data bytes Request 0 timed out   --- 110.0.0.6 ping statistics --- 2 packets transmitted, 0 packets received, 100.00% packet loss

I have to check the routing table s2 and s3 also. They have also information.

s2# show ip route 110.0.0.4/32, ubest/mbest: 1/0     *via 100.42.4.11, [20/0], 1d03h, bgp-1, external, tag 2 110.0.0.6/32, ubest/mbest: 1/0     *via 110.0.0.3, [200/0], 00:09:44, bgp-1, internal, tag 3 s3# show ip route 110.0.0.4/32, ubest/mbest: 1/0     *via 110.0.0.2, [200/0], 00:14:14, bgp-1, internal, tag 2 110.0.0.6/32, ubest/mbest: 1/0     *via 100.63.6.11, [20/0], 1d02h, bgp-1, external, tag 3

Now, only s1 switch has been left. In s1 switch, I did not configure for BGP. Because of this, s1 can not get any information for s4 and s6. Now I will add static route simply on s1 switch.

s1(config)# ip route 110.0.0.4/32 110.0.0.2 s1(config)# ip route 110.0.0.5/32 110.0.0.2 s1(config)# ip route 110.0.0.6/32 110.0.0.3 s1(config)# ip route 110.0.0.7/32 110.0.0.3

After this configuration add, I can send the traffic

s4# ping 110.0.0.6 source-interface loopback 0 PING 110.0.0.6 (110.0.0.6): 56 data bytes 64 bytes from 110.0.0.6: icmp_seq=0 ttl=251 time=20.858 ms 64 bytes from 110.0.0.6: icmp_seq=1 ttl=251 time=12.801 ms 64 bytes from 110.0.0.6: icmp_seq=2 ttl=251 time=25.59 ms 64 bytes from 110.0.0.6: icmp_seq=3 ttl=251 time=13.339 ms 64 bytes from 110.0.0.6: icmp_seq=4 ttl=251 time=12.694 ms --- 110.0.0.6 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 12.694/17.056/25.59 ms

 

Reference

 

[ 1 ] https://blog.naver.com/happy_jhyo/70151060540

[ 2 ] https://blog.naver.com/happy_jhyo/221284566856

[ 3 ] http://www.nnk.com.au/index.php?option=com_content&view=article&id=22:bgp-synchronization&catid=3:articles&Itemid=5

[ 4 ] https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/unicast/configuration/guide/l3_cli_nxos/l3_bgp.html