netgo

How to outbound external connection of GN3 devices over remote server?

I use GNS3 simulator to study network feature sometimes. There is many instruction over internet. In this post, I will introduce how to use NAT fuction to connect to interet for outbound traffic. 

1. Pre-requisite.

I have local GNS3 client and remote server over Packet.net and AWS Baremetal cloud. In my case, my remote server is the Linux (Ubuntu 16.04) host. After then, I uploaded CiscoNX v9000 in my remote server.  

This is the my environment to re-produce in this post.

2. About NAT appliance (DHCP feature).

In GNS3, there is NAT appliance. It has the role to connect the physical interface on remote server virtually. This NAT appliance offers DHCP feature to assign IP address. This IP address determined the next hop against the Cisco switch/router. Thus, the Cisco switch/router should have DHCP feature. 

After this configuration, I can verify the interface status with "show ip int br" command

And "ip route 0.0.0.0/0 192.168.122.1" is written. In the Linux host (remote server), I can see the interfaces like below.

With this interfaces, the packet will transfer to external side. I can see the masqurade rule in "iptables table" like below.

Now, I can send traffic to outside on switch 1. Please noteh that it is not possible on switch 2.

3. Configuration NAT Inside and Outside

Switch 1 can send the packet to outside, however switch 2 can not. Because "192.168.122.0/24" network can only be masquraded in the remote server. This is the reason why the switch 1 has NAT feature. In this instruction, there is the explation for Cisco NAT concept. 

Inside source translation is for inside --> outside traffic. Outside source translation is for outside --> inside traffic. At first, I will make the direction on Cisco switch 1.

4. Create Rule (NAT Policy) for outbound traffic.

In this post, I handle only outbound traffic. The main factor is that IP address for all traffic to outside should be changed with Switch 1 interface ethernet 1/1 IP address.  In this post, It will be useful. I want all traffic to be sent.

5. Configure inside network (Switch 2).

So far, I made NAT firewall. From this part, it can assume internet network. However, I only use single switch/router simply. 

This is normal configuration. There is nothing special. After default configuration. I can verify the outbound connection like below over switch 2.

It works now.

Reference 

[ 1 ] https://docs.gns3.com/1c2Iyiczy6efnv-TS_4Hc7p11gn03-ytz9ukgwFfckDk/index.html

[ 2 ] https://docs.gns3.com/appliances/cisco-nxosv9k.html

[ 3 ] https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/interfaces/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_7x_chapter_01100.html#concept_F5BFDE98ABA749859F5AD710AC61EFCD

[ 4 ] https://onlyit.tistory.com/entry/How-to-use-a-nat-node-in-gns3

How does the multicast (PIM sparse mode) work with MSDP? 

In this post, I will generate the multicast environment with Nx9000v on GNS3. I will handle the multicast with PIM sparse mode.

1. Pre-requiste for Test environments

I will produce this environment to generate the multicast.

Before I make Multicast environment, I need to prepare NAT for outbound traffic and Multicast test scripts. This is the sample configuration for NAT outbound traffic.

Also I attach the multicast example below

2. Configuration for default environment.

2-1. Configure Core swtich/router  

I will make full-mesh connectivity. There are four switch/router. I will name as s1, s2, s3 and s4. On those switch/router, I will run OSPF routing to announce loopback addresses for Router ID and RP point. 

In s1 and s2, there are 2 kinds of loopback addresses. One is for the switch/router ID, the other is for the RP address. For loopback address for RP, I will utilize "ghost loopback address" method. Please look at the loopback interface 1 in s1 and s2.

S1 is the main RP server in normal status. However, it will be changed to s2 when S1 is failed without IP address change. Because of this, I need MSDP (Multicast Source Discovery Protocol). I will handle in later. To announce the network with subnet of loopback interface into OSPF, "Point to Point" is necesary. Now I can see the routing table on S3 and S4 switch/router.

In red text above, "100.100.100.100" is the RP address for multicast. At this time, the destination should be s1. It will be changed to s2 by "100.100.100.0/24", when s1 is failed.

2-2. Configure the Host (PC, Labtop) network.

s3 and s4 is the gateway for each host network. In my scenario, I will use "10.0.0.0/24" for PC1 and "20.0.0.0/24" for PC2.

I have already done for the gateway above.

Now I will configure the PC network interface. In my case I will use "Ubuntu Desktop". In this instruction, I can see how to install "ubuntu" in GNS3. After installation, I can open the console like below.

After login, I can set the network interface like below. I need to update file in "/etc/network/interfaces".

After set both of hosts, I can send ICMP packet each other.

However, there is one more thing necessary. To Join and receive the multicast packet. I need to define the IGMP version on Linux host sometimes. This instruction will be helpful.

In my case, Host 2 and Host 3 will be receiver, therefore it will be like below

3. Configuraton for Multicast environment.

3-1. Configure for default multicast 

In fact, I have already done for multicast routing, if you follow sample configuration. This instruction will be helpful to understand how to configure.

Now, I can test multicast routing with sample script. I can get the result like below.

I success to send multicast traffic. However, I still leave MSDP. "99.99.99.99" is the RP which is S1. I worry the S1 fail situation. MSDP is one of good solution. Please read next step.

3-2. Configure for MSDP.

Before, I configure for MSDP. Any information was not shared each other. (S1 and S2 are peer connection for MSDP). I run command "show ip msdp route" and "show ip msdp sa-cache".

Because of this, the routing information should be registered again on S2, when S1 is failed. Now, I will follow this instruction. At first, I will MSDP configuration to share source information between s1 and s2.

In above, there is "ip msdp cache-sa-state" command for caching. However, there is no command in Nx9000v in my case. Therefore, my configuration should be below.

I can check the status with "show ip msdp summary" 

I can also verify the status with again "show ip msdp route" and "show ip msdp sa-cache".

In fact, I can not understand this effect.

3-3. Verify the MSDP effect.

Before MSDP configuration, I will shutodown loopback 1 (RP) of S1 switch/router. In S2 switch/router, there is no multicast source information.

After sender request, the source information is updated. This is normal case

Now, I will configure MSDP. I will shutdown loopback 1 of switch/router S1 again. In S2,

Multicast routing information is updated directly.

Sample Configurations

Reference

[ 1 ] https://www.osboxes.org/ubuntu/#ubuntu-14_04-vmware

[ 2 ] https://docs.gns3.com/1RpM21DdgL2pCNtIrw6e9Yf52tRjYZIe46LMeESl-jy4/

[ 3 ] https://www.cisco.com/c/en/us/td/docs/ios/12_2/ip/configuration/guide/fipr_c/1cfmsdp.html#wp1000963

[ 4 ] https://createnetech.tistory.com/40?category=672584

[ 5 ] https://pymotw.com/2/socket/multicast.html

[ 6 ] https://sites.google.com/site/miclinuxcorner/technology/multicast-routing

[ 7 ] https://www.cisco.com/c/en/us/support/docs/ip/ip-multicast/9356-48.html#msdp

In this post, I will install the kubernetes with external etcd. Most of instrcution come from here. I will follow step by step as the beginner.

1. Pre-requsite

At first I need to prepare the hosts for Kubernetes components. In my case, I have 5 nodes, one for master, one for external etcd and three for workers. After I prepare these nodes. I need to update "/etc/hostname" and "/etc/hosts" files propertly. For example, I have to change hostname to "m1" as master node.

After this, I need to log-out and log-in again.

Kubernetes use Docker engine. Therefore, I need to install Docker. I will follow this instruction. In my case, I will run this kubernetes over Ubuntu 16.04. This is overview.

I need to check docker info. It will be like below. I need to check "Cgroup Driver: cgroupfs" as default.

From this instruction, I will change this driver from "cgroupsfs" to "systemd". This change affect lots of issue during Kubernetes installation. Therefore I need to remember this.

After this, Cgroup Driver will be changed 

2. Install kubelet, kubeadm and kubectl

Kubelet (daemon on each node has the role to handle conatiner), kubeadm (daemon on each node has the role to setup master and workers) and kubectl (daemon on each node has the role to manage the cluster) are necessary components.

After installation, I need to check the directories which are created during the installation. "/etc/kubernetes/" and "/var/lib/kubelet/pki/" and "/etc/systemd/system/kubelet.service.d/" are created. 

To work Kubernetes normally, Kubelet work normally at first. I can check the status with "systemctl status kubelet". At this moment, it is not activate.

To see more details, I will look at the logs in "/var/log/syslog". From logs, I can check I need "config.yaml" file. Keep this issue and go to next step. This file will be created with "kubectl init" command later.

I need to match the cgroup driver with the value "systemd". In the instruction, there is the comment like below.

However there are no "/etc/default/kubelet" and "/var/lib/kubelet/kubeadm-flags.env". So I can not update this with this instruction. Please look at the result from "systemctl status kubelet". I can see "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf". This is the loaded file when kubelet is started.

I will add the red text, Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd". And restart the kubelet with "systemctl daemon-reload" and "systemctl restart kubelet". However, it is still not activated due to no config.yaml.

3. Install external etcd cluster

I have alread posted in here. To create the etcd cluster, It is not easy. However, I can make these with this instruction easliy. This etcd will be created as one of container application which are managed by Kubernetes. On the external etcd node, I have to run command below.

 Please note that there is no option for cgroup drivers. Therefore, I need to change something.

The file name start with 20-etcd-service-manager.conf due to "20", so it has higher priority rather than 10-kubeadm.conf. After running this, I can check the "systemctl status kubelet" and "/var/log/syslog". The kubelet does not works with some error. This error is different from above. The first error above is happend after 10-kubeadm.conf loading. However this error is happend during 20-etcd-service-manager.conf loading. So I have to solve this error at first.

There are some error and warning messages. To solve this, I have to create "kubeadmcfg.yaml" from this instruction. In my case, I will difference file name like below

"controlPlaneEndpoint: 147.75.94.251:6443" is master IP address and Port for API server. This parameter solve this warning "No api server defined - no events will be sent to API server.". 147.75.92.69 ip address is external etcd node interface IP addresss which is announced for outside. In this yaml file, some options are included. Therefore, I have to revise 20-etcd-service-manager.conf file like below. 

Even if I added more options in yaml file, I do not handle root cause why the kubelet fail. In the syslog, "failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false" is the main reason. To solve this, there are 3 option.

"swapoff -a" is temporary option. Comment "swap" in "/etc/fstab" is permenant option. Adding " --fail-swap-on=false" in 20-etcd-service-manager.conf is another permenant option. After then, the status of kubelet will be like this.

Now, I am ready to deploy etcd container deployment. I need certificate and private key for etcd cluster to work well. These values are located in "/etc/kubernetes/pki/etcd". However, you do not have this directory at this time. As the instruction, these file will be generated with command below.

If you are want to make cluster of etcd nodes, I have copy these files on each etcd nodes. However, it is not enough to work well. I need more certiface and key.

after then, serveral key will be generated. In fact, some of key is not necessary if you run single external etcd node. Now, I can run etcd container with command below

I can also check the health status.

4. Install master nodes

There are 2 instructions for master node, single master and cluser masters. To setup the master node, I need configuration file. I will refer from this instruction. This is the sample configuration yaml file what I have

With this configuration above, I can run "kubeadm init". Please note that "mode: ipvs" will change the kube-proxy mode to work with "ipvsadm" kerneal module. In the post what I wrote, It will be undertand how kube-proxy work with this part. This is overview to install and load kernal module

Sometimes, some module are not loaded. So I need to do manually with "modeprobe". If these module are not loaed when kubeadm init is started, I can see this message "kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh]" 

Now, it will load the file "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf".

With this configuration I can run "kubeadm init" 

before run kubeadm init, I need to "swapoff -a". I can get some error message without "swapoff -a"

Anyway, I will get this result when I make master node succefully.

I need more steps. First I configure for "kubectl" enable.

After this, I can use "kubectl" command. Sometimes, master status can be "NotReady". At this time, I should check the status of pods with "kubectl get pods --all-namespace"

From this instruction, I need to POD network to solve this issue.

To install POD network, look at this instuction.

In my case, I will use "Flannel".

And then, the master status is normal

Oh It is work now. See status with more information

There is something strange. "cgroupDriver: cgroupfs" is strange. It should be "systemd", becuase I update "Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" in "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf". I think it is not work. I will update for this

I will run "kubeadm reset" and "kubeadm init" again. Now

However, this is not all of things. I have alread made external etcd. I have not configure for this yet. Default, etcd conatiner is located on each master node. They are not clustered.

From this instruction, there are serveral step for this. At first I need to copy the certificate and key from etcd node.

Now, I need to update my configuration file

Now, I will run "kubeadm reset" and "kubeadm init" again. Please note that the certificate and key will be remove when I run kubeadm reset. Therefore, I need to copy again from etcd node

After I copy the certificate and key, I can check the status again

etcd is not existed on master node. 

5. Setup the Worker nodes

It is almost same. I need docker engin and kubeadm, kubectl, kubelet. I have already explain about these above. After creating master node, I can see the message like below

First is for adding master node, second is for adding worker node. I need some information such as token.

With these values, I can run command on each worker node. I will check the status of the cluster.

Now I made the kubernetes cluster.

Reference

[ 1 ] https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

[ 2 ] https://docs.docker.com/install/linux/docker-ce/ubuntu/

[ 3 ] https://kubernetes.io/docs/setup/cri/

[ 4 ] https://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/ 

[ 5 ] https://createnetech.tistory.com/17?category=679927

[ 6 ] https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[ 7 ] https://kubernetes.io/docs/setup/independent/high-availability/

[ 8 ] https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

[ 9 ] https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

[ 10 ] https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd-nodes

[ 11 ] https://createnetech.tistory.com/36?category=672585

[ 12 ] https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#is-kube-proxy-writing-iptables-rules

How does the flannel work?

Recently, I am studying kubernetis. During studying, I have known about the flannel. There is some instruction to reproduct simply. I will follow this instruction.

1. Pre-requisite

There are some preparation. Docker and Etcd should be installed before. Here is instruction for installation of Docker. I will install community engine on 2 nodes running "ubuntu 16.04".

And then, I will install Etcd. I have already written how to install and use Etcd in this post. However, it is more simple in this instruction. In my case, I will run the command in "root directory"

After installation, I will edit the "/etc/hosts" file before running this Etcd. Please note that host name has only single IP address on this file.

Each nodes can communicate with each other with these IP addresses.

Now, I am ready to start the flannel configuration.

2. Flannel installation.

Download and install the flannel command on each nodes. In my case, I will run the command in "root directory"

I will make configuration file to create the network topology.

In this documentation, there are the meaning of the above parameters. Especially, "SubnetLen" is IP address range allocated in each host. Flannel use the configuration from Etcd, /coreos.com/network/config. I will set the configuration on Node 1

I can check if the configuration is set or not on Node 2.

Now, I am ready to start the flannel. Before start flannel, I have to look at my network interface status.

After start flannel, I can see new interface which is named with "flannel.VNI". In this case, It should be flannel.172.

Also, I can get some information from Etcd.

This mean that each host has these subnets each. I can see more detail. 

This is configuration for the flannel. I can also see the what flannel network is assigned with "/var/run/flannel/subnet.env". Please note that this file will be used for next step, docker daemon configuration.

3. Docker daemon configuration.

Docker does not use flannel default. It has swarm mode to make overlay network. Therefore, it is necessary to change to use this flannel as default network module. At first, I have to stop the Docker daemon on each hosts, node1 and node2.

I will restart Docker daemon with this flannel configuration.

Before restart Docker daemon, there is "docker0" interface default which has "172.17.0.1" IP address. It will be changed with the network what I defined.

4. Create the Test container and Start it.

Now, I will create 2 container for the test. 

Depends on the container properties, the container can be stopped after exit. "sh -c "while true; do sleep 3600; done" make this container keep alive for 1 hour. It is enough for the test.

5. Analysis container networks.

In this post, I will explain how to work in docker swarm mode. It is good to analysis the network topology of docker. At first, go to "/var/run/docker", there is the "netns" directory. There is the network configuration of the container.

After this symbolic link, I can see the network namespace list like below. "ip netns list" show the namespace ID.

I can see more detail information with this ID. "ip netns exec" show the same result with "docker exec". Thus I can see the same result with "docker exec test1 ip -d addr show"

6. Test and Troubleshooting

Now, I have know that "172.16.75.2" is container IP address on Node 1 and "172.16.68.2" is container IP address on Node 2. Flannel offers the overlay network between hosts. So, I will send ICMP (ping) from node1 to node2

Hmm. It does not work. From Docker 1.13 default iptables policy for FORWARDING is DROP,

Wow, I can send ICMP each other.

This is the flannel.

Reference

[ 1 ] https://docs.docker.com/install/linux/docker-ce/ubuntu/

[ 2 ] https://docker-k8s-lab.readthedocs.io/en/latest/docker/docker-etcd.html

[ 3 ] https://docker-k8s-lab.readthedocs.io/en/latest/docker/docker-flannel.html

[ 4 ] https://github.com/coreos/flannel/blob/master/Documentation/configuration.md

How SSL/TLS handshake can be done?

In this post, I will analysis the SSL/TLS packet. In fact, I have some chance to see RDP packets. Look at the below. This is RDP packet captures. 

I wrote "How to calculate the sequence number" in this post. In this post, I will only handle how SSL/TLS handshake can be done. Before SSL/TLS handshake, TCP handshake should be established.

1. Client Hello

At first, Client send the "Client Hello" packet. In this packet, there are three important information. Random number, Session ID and Cipher suites. 

Random number is used to generate "pre-master key" with another random number from server. This "pre-master key" will be used to generate "master key" which encrypt and decrypt the packets.

Cipher suites is the list which the client can support. Thus, the server will select one of this lists.

2. Server Hello.

After receive the client hello, server send the "Server Hello" packet to client. In this packet, there are three important information. Random number, Cipher suite and Certificate with Public key.

Random number is used to generate "pre-master key" with another random number from client.

Cipher suite is the selected item which is one of list from client.

Certificates is the very important parameters. In this values, "Public Key" is included. This "Public Key" is used to encrypt the "pre-master key" before transfer to server.

3. Client Key Exchange

In this step, Client know both random values of client and server. Therefore, client generate the "Pre-master key". Also, client can know public key because of the received certificates. So, client sent the packet which "Pre-master key" is included in. It is encrypted by public key.

4. Server Response.

Finally, the server knows "pre-master key", after decrypting received packet. The server and client will be generate "master key" each by some algorithm. This "master key" is used for encrypt and decrypt the data packet.

5. Data Packet with Encryption.

So, the Data packets are encrypted by this master key. I can see the SSL layer in the packets like below. Data will be encrypted.

Reference

[ 1 ] https://opentutorials.org/course/228/4894

[ 2 ] http://createnetech.tistory.com/25

How to calculate sequence number of the TCP/IP packets?

I am the network engineer. Recently, I have some change to remember the sequence number of the TCP/IP packets. Someone include me think that this is not easy. 

1. General Packet Structure.

IP header and TCP header have generally 20 Byte size of the packets. Data payloads can be maximum 1460 Byte size of the packets. 

MSS is the data size, which determine how much can be send at one time. MTU is sum of TCP, IP and MSS (Data). 

MSS = Maximum TCP Segment Size in one Packet – usually it is 1460 + 20 (TCP Header) + 20 (IP Header) = MTU =1500 + 18 (DLC Header) and you have a full frame of 1518 bytes.

2. IP header Structure

I have told "IP header is 20 Byte". However, IP header is not fixable. It can be increased by optional values up to 60 Byte.  In this header, there are three point which I need to focus on.

Length field shows how much size of IP header. Identification field is one of mark. This is the unique value which is not changed from source to destination.  In this link, I can see more detail of the Protocol field.

3. TCP header structure.

TCP header is also not fixable. It can be max 60 Byte. In TCP header, there are sequence number, acknowledge number and window size value.

Windows size value is determined by server environment, such as allocated memory of operating system. It can be increased or decreased. If I suffer from "Zero Window" issue, I have to check the buffer size of host.

4. DLC header structure.

This header shows MAC address generally. In Ethernet field, 0x0800 means IPv4 and 0x08dd means IPv6.

5. Packet sequence analysis for SYN and FIN

For SYN-ACK / FIN-ACK handshake, it is important to add +1 value, even if length of data is zero. Client sent packet with sequence number 0. Therefore, the expect ACK number should 1. Server will send the packet with sequence number 0 and ACK number is 1. For this packet, the expect ACK number should also is 1. Finally, client send last ACK packet with sequence number 1 and ACK number 1.

6. Packet sequence analysis for Data 

For Data, It is little bit different with above. It add only length of data. Look at the below. Fist, sequence number 380 + data length 213 is Expect ACK with number 593.

Second, sequence number 1306 + data length 373 is 1679 with ACK 593 which come from above.

Final, sequence number 593 which equals with ACK number will transfer with ACK 1679.

7. Optional ,SACK (Selective ACK) 

For effective transmission, the selective ACK mechanism is existed. Look at the below. 172.18.0.74 is sender. This sender send the packet with sequence number, 5943, 7403, 8863, 10323, 11784, 13243, and 14703. The data length is 1460. Thus, there is no loss to transfer.

However, Loot at the below. At this time, ACK number is important value. "ACK=10323, SLE=13243, SRE=14703" message means 11783 packet does not exist in the receiver. 

In the last, re-transmission for sequence 11783 is happen. The ACK number with 11783 is shown.

Reference

[ 1 ] https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml

[ 3 ] https://www.networkcomputing.com/network-security/network-troubleshooting-tcp-sack-analysis/2073731643

How to Configuration VXLAN in Ubuntu 16.04

In this post, I will configure VXLAN example. I will also utilize the linux-bridge to define L2 domain. Test environment is looks like below. 

The concepts are difficult to understand. However, the steps are not difficult.

1. Install the Linux Bridge and configuration.

In this step, I will create Linux Bridge and Interface on each hosts. The IP address in the same broadcasting is set on each interface.

apt-get install bridge-utils

brctl addbr vbr0

ip address add 192.168.0.1/24 dev vbr0

ifconfig vbr0 up

2. Configure VXLAN with Unicast

I will create VTEP interface with the command below. I can check the detail information with “-d” option.

3. Add VXLAN interface on Linux Bridge

However, it is not enough to communicate over tunnel. In this case, the traffic of “192.168.10.0/24” can not pass over the Linux Bridge. Thus, It is necessary for VXLAN interface to attach on the Linux Bridge.

4. Testing and analysis

I will do ping with one of “192.168.10.0/24” IP address. 

And I will dump packet during sending the packets. From the result, I can confirm “ICMP packets are encapsulated over VXLAN”

Reference Links

[ 1 ] https://serverfault.com/questions/777179/configuring-vxlan-unicast-in-linux

[ 2 ] https://www.tldp.org/HOWTO/BRIDGE-STP-HOWTO/set-up-the-bridge.html

[ 3 ] https://www.kernel.org/doc/Documentation/networking/vxlan.txt

[ 4 ] https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/

[ 5 ] http://www.codeblogbt.com/archives/301596

How to Configure “ipvsadm” in Ubuntu 16.04

I recently the existence of this “ipvsadm” which used as the load-balancer.
“ipvsadm” is referred linux kernel load-balancer, which is also called “LVS, Linux Virtual Server”.
This LVS has 3 mode such as DR (Direct Routing), Tunnel and Masquerade. In this post, I will handle DSR and Masquerade (Network Address Translation, NAT).

Direct Routing : The option value is default with “-g”. The packet is send without modifying. The servers receives the packets from “ipvsadm” response to client directly.

Network Address Translation (NAT) : This option is adapted with “-m”. The packet is send with modifying the destination IP address. (The source IP address is not modified). The servers have to response the “ipvsadm”. Usually, The servers indicate “ipvsadm” as the default gateway.

My test environment is set on Ubuntu 16.04. I used AWS IaaS.

1. DR mode
 

1-1. ipvsadm configuration

Enable the IP forwarding, because the “ipvsadm” has the role to transfer and distribute received packets. To enable, edit “net.ipv4.ip_forward=1“ in “/etc/sysctl.conf” and run “sysctl -p /etc/sysctl.conf” or “sysctl –p” to apply this.
It can be done with, echo 1 > /proc/sys/net/ipv4/conf/all/forwarding, alternatively.

Configure virtual server, there are two steps. First, create the virtual server with traffic distribute method such as round-robin. Second, register servers to distribute the packets.
ipvsadm -C 
ipvsadm -A -t 10.10.0.244:80 -s rr
ipvsadm -a -t 10.10.0.244:80 -r 10.10.0.233:80 -g
After this configuration, I can confirm the status with “ipvsadm –Ln”, “ipvsadm –Lcn”, and “ipvsadm -l –-stats”

 
“ipvsadm –Ln” show the mapping information with forward method. In this case, the received packet with “10.10.0.244:80” will be routed to “10.10.0.233:80”.
 

“ipvsadm –Lcn” show the current session information. At this time, there is no con-current connection now.
 

“ipvsadm -l –-stats” show the information for in/out traffic information.

1-2. Servers configuration

In DR mode, the server received the packet without modifying. And the server response to the client directly. However, the packet drop can be happened in client side, because the client receive the packet from the server with server’s IP address. To resolve this issue, the server need to set the loopback interface with service IP address. In this case, the service IP address should be “10.10.0.244”.
ifconfig lo:0 10.10.0.244 netmask 255.255.255.255
 

LVS Direct Routing works by forwarding packets to the MAC address of servers. In this case, we have to consider “Linux ARP flux” problem. The server should not answer ARP request for “10.10.0.244”. For, this, I added in “/etc/sysctl.conf”.
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
 

1-3. Testing and analysis

Client send the request with “curl http://10.10.0.244” and get the response from server. I dumped the “ipvsadm” and “server”.
Loot at the “ipvsadm” result. I can see there is no change of source and destination IP address. 
 

This is something strange. “ipvsadm” has “10.10.0.244”, so it looks like sending to myself. This is the DR mode property, which works by MAC address of servers. Look at the connection information with “ipvsadm –Lcn”, the destination IP address can be shown.
 

At this time, what happened in the server, Look at the below. The packet was received with “10.10.0.244”. And response to this IP address. More important thing is response packet to client. The server send the packet, which has “10.10.0.244” as the source IP address. Because of this, the client does not dropt the packet.
 

2. Network Address Translation Mode

In NAT mode, the response should be return to the “ipvsadm”. However, the source IP address does not modified and sent to the server. NAT mode only modify the destination IP address.

2-1. ipvsadm configuration

Enable the IP forwarding, because the “ipvsadm” has the role to transfer and distribute received packets. To enable, edit “net.ipv4.ip_forward=1“ in “/etc/sysctl.conf” and run “sysctl -p /etc/sysctl.conf” or “sysctl –p” to apply this.
It can be done with, echo 1 > /proc/sys/net/ipv4/conf/all/forwarding, alternatively.
 
Configure virtual server, there are two steps. First, create the virtual server with traffic distribute method such as round-robin. Second, register servers to distribute the packets.
ipvsadm -C
ipvsadm -A -t 10.10.0.244:80 -s rr
ipvsadm -a -t 10.10.0.244:80 -r 10.10.0.233:80 –m
“ipvsadm –Ln” show the forward method is changed from “Route” to “Masq”
 

2-2. Server configuration

Server received the packet which is modified. Remember “ipvsadm” does not change the source IP address. In this case, the response will be return to client directly.
 
I use same network topology above. Therefore, “ipvsadm” and server are located on the same network. So, I can add some “static route” to transfer the response to “ipvsadm”.
route add -net 10.10.1.0 netmask 255.255.255.0 gw 10.10.0.244
 

2-3. Testing and analysis

Look at the “ipvsadm” packet flow from TCP dump. It show that the destination IP address is modified from 10.10.0.244 to 10.10.0.233. In response, the source IP address is also modified from 10.10.0.233 to 10.10.0.244.
 

Look at the server packet flow. The server do only normal processing. 

2-4. “ipvsadm” with SNAT (L3 mode, Proxy mode)

So far, “ipvsadm” and server are located on the same network. Therefore, It will be easy to construct LVS with NAT mode, using “static routing” method on server side. However, “ipvsadm” and servers can be located on different network.
For L3 environment, “ipvsadm” have modify the source IP address when the packet sent to server. I will add some rule in “iptables”.
Before, we add this rule, we need to add some configure in “/etc/sysctl.conf”. The iptables does not work without this options below.
net.ipv4.vs.conntrack = 1
net.ipv4.vs.snat_reroute = 1
 

After this, I add the rule into iptables with “-m”.
iptables -t nat -A POSTROUTING -o eth0 --dst 10.10.0.233 -m ipvs --ipvs --vaddr 10.10.0.244 --vport 80 --vmethod masq -j SNAT --to-source 10.10.0.244
 

Then, we can see the packet flow with TCP dump. The source IP is not client IP address, any more. The source IP address will be modified to send the server.

3. ipvsadmin with MARK of iptables

Occasionally, we need to use the MARK configuration of iptables. The PREROUTING will be used for this. Two steps are necessary. First, the received packet from client should be marked with iptables. Second, the marked packet should be distributed to servers.
To mark at the packet, I have to use mangle table. Mangle table is used for mark and QoS. In this case, I insert the rule like below
iptables  -A PREROUTING -t mangle -d 10.10.0.244/32 -j MARK --set-mark 1
 

And then, I edit the “ipvs” configuration.
ipvsadm -C 
ipvsadm -A -f 1  -s rr
ipvsadm -a -f 1 -r 10.10.0.233:0 –m
After then, I can see some change are happened. “FWM 1” mean MARK information in iptables.
 

Reference Links

[ 1 ] http://www.ultramonkey.org/papers/lvs_tutorial/html/
[ 2 ] http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.persistent_connection.html
[ 3 ] https://techiess.com/2010/09/09/load-balancing-dsr-direct-server-return/
[ 4 ] https://www.cyberciti.biz/faq/ubuntu-linux-add-static-routing/
[ 5 ] https://terrywang.net/2016/02/02/new-iptables-gotchas.html
[ 6 ] https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1641918
[ 7 ] http://www.loadbalancer.org/blog/enabling-snat-in-lvs-xt_ipvs-and-iptables/
[ 8 ] http://manpages.ubuntu.com/manpages/trusty/man8/ipvsadm.8.html