How does the flannel work?
Recently, I am studying kubernetis. During studying, I have known about the flannel. There is some instruction to reproduct simply. I will follow this instruction.
1. Pre-requisite
There are some preparation. Docker and Etcd should be installed before. Here is instruction for installation of Docker. I will install community engine on 2 nodes running "ubuntu 16.04".
sudo apt-get remove docker docker-engine docker.io sudo apt-get update sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update sudo apt-get install docker-ce |
And then, I will install Etcd. I have already written how to install and use Etcd in this post. However, it is more simple in this instruction. In my case, I will run the command in "root directory"
wget https://github.com/coreos/etcd/releases/download/v3.0.12/etcd-v3.0.12-linux-amd64.tar.gz tar zxvf etcd-v3.0.12-linux-amd64.tar.gz cd etcd-v3.0.12-linux-amd64 |
After installation, I will edit the "/etc/hosts" file before running this Etcd. Please note that host name has only single IP address on this file.
vi /etc/hosts 147.75.65.69 node1 147.75.65.63 node2 |
Each nodes can communicate with each other with these IP addresses.
# At node 1 nohup ./etcd --name docker-node1 --initial-advertise-peer-urls http://node1:2380 \ --listen-peer-urls http://node1:2380 \ --listen-client-urls http://node1:2379,http://localhost:2379 \ --advertise-client-urls http://node1:2379 \ --initial-cluster-token etcd-cluster \ --initial-cluster docker-node1=http://node1:2380,docker-node2=http://node2:2380 \ --initial-cluster-state new& # At node 2 nohup ./etcd --name docker-node2 --initial-advertise-peer-urls http://node2:2380 \ --listen-peer-urls http://node2:2380 \ --listen-client-urls http://node2:2379,http://localhost:2379 \ --advertise-client-urls http://node2:2379 \ --initial-cluster-token etcd-cluster \ --initial-cluster docker-node1=http://node1:2380,docker-node2=http://node2:2380 \ --initial-cluster-state new& |
Now, I am ready to start the flannel configuration.
cd etcd-v3.0.12-linux-amd64 ./etcdctl cluster-health member 43bb846a7344a01f is healthy: got healthy result from http://node2:2379 member a9aee06e6a14d468 is healthy: got healthy result from http://node1:2379 cluster is healthy |
2. Flannel installation.
Download and install the flannel command on each nodes. In my case, I will run the command in "root directory"
wget https://github.com/coreos/flannel/releases/download/v0.6.2/flanneld-amd64 -O flanneld && chmod 755 flanneld |
I will make configuration file to create the network topology.
vi flannel-network-config.json { "Network": "172.16.0.0/12", "SubnetLen": 24, "SubnetMin": "172.16.16.0", "SubnetMax": "172.31.247.0", "Backend": { "Type": "vxlan", "VNI": 172, "Port": 8889 } } |
In this documentation, there are the meaning of the above parameters. Especially, "SubnetLen" is IP address range allocated in each host. Flannel use the configuration from Etcd, /coreos.com/network/config. I will set the configuration on Node 1
# At node 1 cd etcd-v3.0.12-linux-amd64/ ~/etcd-v3.0.12-linux-amd64$ ./etcdctl set /coreos.com/network/config < ../flannel-network-config.json |
I can check if the configuration is set or not on Node 2.
# At node 2 cd etcd-v3.0.12-linux-amd64/ ~/etcd-v3.0.12-linux-amd64$ ./etcdctl get /coreos.com/network/config | jq . { "Network": "172.16.0.0/12", "SubnetLen": 24, "SubnetMin": "172.16.16.0", "SubnetMax": "172.31.247.0", "Backend": { "Type": "vxlan", "VNI": 172, "Port": 8889 } } |
Now, I am ready to start the flannel. Before start flannel, I have to look at my network interface status.
# At node 1 nohup sudo ./flanneld -iface=bond0 & # At node 2 nohup sudo ./flanneld -iface=bond0 & |
After start flannel, I can see new interface which is named with "flannel.VNI". In this case, It should be flannel.172.
flannel.172 Link encap:Ethernet HWaddr 82:41:de:d4:77:d3 inet addr:172.16.75.0 Bcast:0.0.0.0 Mask:255.240.0.0 inet6 addr: fe80::8041:deff:fed4:77d3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:8 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) |
Also, I can get some information from Etcd.
cd etcd-v3.0.12-linux-amd64/ ~/etcd-v3.0.12-linux-amd64# ./etcdctl ls /coreos.com/network/subnets /coreos.com/network/subnets/172.16.68.0-24 /coreos.com/network/subnets/172.16.75.0-24 |
This mean that each host has these subnets each. I can see more detail.
cd etcd-v3.0.12-linux-amd64/ ~/etcd-v3.0.12-linux-amd64# ./etcdctl get /coreos.com/network/subnets/172.16.68.0-24 | jq . { "PublicIP": "147.75.65.63", "BackendType": "vxlan", "BackendData": { "VtepMAC": "7a:ac:15:15:2b:61" } } |
This is configuration for the flannel. I can also see the what flannel network is assigned with "/var/run/flannel/subnet.env". Please note that this file will be used for next step, docker daemon configuration.
cat /var/run/flannel/subnet.env FLANNEL_NETWORK=172.16.0.0/12 FLANNEL_SUBNET=172.16.68.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false |
3. Docker daemon configuration.
Docker does not use flannel default. It has swarm mode to make overlay network. Therefore, it is necessary to change to use this flannel as default network module. At first, I have to stop the Docker daemon on each hosts, node1 and node2.
# Node 1 and Node 2 sudo service docker stop |
I will restart Docker daemon with this flannel configuration.
# Node 1 and Node 2 source /run/flannel/subnet.env sudo ifconfig docker0 ${FLANNEL_SUBNET} sudo dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} & |
Before restart Docker daemon, there is "docker0" interface default which has "172.17.0.1" IP address. It will be changed with the network what I defined.
# Before restart with flannel configuration ifconfig docker0 docker0 Link encap:Ethernet HWaddr 02:42:f6:10:ac:49 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) # After restart with flannel configuration ifconfig docker0 docker0 Link encap:Ethernet HWaddr 02:42:f6:10:ac:49 inet addr:172.16.75.1 Bcast:172.16.75.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) |
4. Create the Test container and Start it.
Now, I will create 2 container for the test.
# At the node 1 sudo docker run -d --name test1 busybox sh -c "while true; do sleep 3600; done" # At the node 2 ssudo docker run -d --name test2 busybox sh -c "while true; do sleep 3600; done" |
Depends on the container properties, the container can be stopped after exit. "sh -c "while true; do sleep 3600; done" make this container keep alive for 1 hour. It is enough for the test.
5. Analysis container networks.
In this post, I will explain how to work in docker swarm mode. It is good to analysis the network topology of docker. At first, go to "/var/run/docker", there is the "netns" directory. There is the network configuration of the container.
# At node 1 and node 2 cd /var/run/ ln -s docker/netns/ netns |
After this symbolic link, I can see the network namespace list like below. "ip netns list" show the namespace ID.
ip netns list faf9928b897f (id: 0) |
I can see more detail information with this ID. "ip netns exec" show the same result with "docker exec". Thus I can see the same result with "docker exec test1 ip -d addr show"
# At the Node 1 ip netns exec b5380e6b336a ip -d addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 11: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 02:42:ac:10:4b:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 172.16.75.2/24 brd 172.16.75.255 scope global eth0 valid_lft forever preferred_lft forever # At the Node 2 ip netns exec faf9928b897f ip -d addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 02:42:ac:10:44:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 172.16.68.2/24 brd 172.16.68.255 scope global eth0 valid_lft forever preferred_lft forever |
6. Test and Troubleshooting
Now, I have know that "172.16.75.2" is container IP address on Node 1 and "172.16.68.2" is container IP address on Node 2. Flannel offers the overlay network between hosts. So, I will send ICMP (ping) from node1 to node2
ip netns exec b5380e6b336a ping 172.16.68.2 PING 172.16.68.2 (172.16.68.2) 56(84) bytes of data. ^C --- 172.16.68.2 ping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5040ms |
Hmm. It does not work. From Docker 1.13 default iptables policy for FORWARDING is DROP,
# At Node 1 and Node 2 sudo iptables -P FORWARD ACCEPT |
Wow, I can send ICMP each other.
ip netns exec b5380e6b336a ping 172.16.68.2 ip netns exec b5380e6b336a ping 172.16.68.2 -c 4 PING 172.16.68.2 (172.16.68.2) 56(84) bytes of data. 64 bytes from 172.16.68.2: icmp_seq=1 ttl=62 time=0.364 ms 64 bytes from 172.16.68.2: icmp_seq=2 ttl=62 time=0.310 ms 64 bytes from 172.16.68.2: icmp_seq=3 ttl=62 time=0.319 ms 64 bytes from 172.16.68.2: icmp_seq=4 ttl=62 time=0.308 ms --- 172.16.68.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2998ms rtt min/avg/max/mdev = 0.308/0.325/0.364/0.026 ms |
This is the flannel.
Reference
[ 1 ] https://docs.docker.com/install/linux/docker-ce/ubuntu/
[ 2 ] https://docker-k8s-lab.readthedocs.io/en/latest/docker/docker-etcd.html
[ 3 ] https://docker-k8s-lab.readthedocs.io/en/latest/docker/docker-flannel.html
[ 4 ] https://github.com/coreos/flannel/blob/master/Documentation/configuration.md
'Network Engineering > Docker and Kubernetes Learning' 카테고리의 다른 글
| How to install Kubernetes with external Etcd manually? (0) | 2019.05.22 |
|---|