How to use etcd (multi-machine cluster TLS/SSL security mode) in Ubuntu?

How to use etcd (multi-machine cluster TLS/SSL security mode) in Ubuntu?

From this post, I run multi-machine cluster basic mode. Now, I will run multi-machine cluster with TLS/SSL security. I am not the security engineer. I am not friendly about TLS/SSL concepts. I studied about key and chain concept before, however it still difficult to understand it. In this documentation, there are 2 things required,  a unique key pair (member.crt, member.key) and shared cluster CA certificate (ca.crt). In this post, I address how to create these.

1. Overview the command for multi-machine cluster with TLS/SSL.

From this documentation, the commands are look like below.

etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \   --listen-peer-urls https://172.22.0.96:2380 \   --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.96:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \   --cert-file=/path/to/infra0-client.crt --key-file=/path/to/infra0-client.key \   --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \   --peer-cert-file=/path/to/infra0-peer.crt --peer-key-file=/path/to/infra0-peer.key etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \   --listen-peer-urls https://172.22.0.33:2380 \   --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.33:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \   --cert-file=/path/to/infra1-client.crt --key-file=/path/to/infra1-client.key \   --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \   --peer-cert-file=/path/to/infra1-peer.crt --peer-key-file=/path/to/infra1-peer.key etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \   --listen-peer-urls https://172.22.0.133:2380 \   --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.133:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \   --cert-file=/path/to/infra2-client.crt --key-file=/path/to/infra2-client.key \   --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \   --peer-cert-file=/path/to/infra2-peer.crt --peer-key-file=/path/to/infra2-peer.key

From command line above, there is something unknown like below. I need certificate. However, I do not know how and what can I obtain these.

--client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \   --cert-file=/path/to/infra2-client.crt --key-file=/path/to/infra2-client.key \   --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \   --peer-cert-file=/path/to/infra2-peer.crt --peer-key-file=/path/to/infra2-peer.key

"cfssl" can do for these. In this Git link, there are introduction about "cfssl" like below.

This demonstrates using Cloudflare's cfssl to easily generate certificates for an etcd cluster.

2. Download and install the "cfssl"

In this documentation, there are installation guide and usage. 

mkdir ~/bin curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x ~/bin/{cfssl,cfssljson}  export PATH=$PATH:~/bin

It is so simple. I verify the version.

# cfssl version Version: 1.2.0 Revision: dev Runtime: go1.6

3. Create root certificate (CA).

In the command, there are 2 root certificate, ca-client.crt and ca-peer.crt. It will be shared between all hosts. Thus, I create these certificate on single host and copy into others. To create root certificate, I need 2 files, "config.json" and "csr.json". The contents is here.

# vi ca-client-config.json {   "signing": {     "default": {         "usages": [           "signing",           "key encipherment",           "server auth",           "client auth"         ],         "expiry": "8760h"     }   } } # vi ca-client-csr.json { "CN": "CA Client", "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "Honest Achmed's Used Certificates", "OU": "Hastily-Generated Values Divison", "L": "San Francisco", "ST": "California", "C": "US" } ] }

I create "ca-client-config.json" and "ca-client-csr.json" for ca-client certificate. I have to create more for ca-peer certificate. CN name should be different.

# vi ca-peer-config.json {   "signing": {     "default": {         "usages": [           "signing",           "key encipherment",           "server auth",           "client auth"         ],         "expiry": "8760h"     }   } } # vi ca-peer-csr.json { "CN": "CA Peer", "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "Honest Achmed's Used Certificates", "OU": "Hastily-Generated Values Divison", "L": "San Francisco", "ST": "California", "C": "US" } ] }

Now, I can generate with files above. Please, note that I will obtain some files which is named "ca.csr", "ca-key.pem" and "ca.pem"

# cfssl gencert -initca ca-client-csr.json | cfssljson -bare ca-client - # cfssl gencert -initca ca-peer-csr.json | cfssljson -bare ca-peer -

I can verify this is normally good or not. 

# openssl x509 -noout -text -in ca-client.pem # openssl x509 -noout -text -in ca-peer.pem

It's good. I can generate intermediate certification with this certificate. Therefore I need to copy these files into other hosts. I move files on each directory like below. (I create these directory on every hosts)

# mkdir ~/cert/clientm # cp ca-client* ~/cert/client/ # ls ~/cert/client/ ca-client-key.pem ca-client.pem ca-client-config.json # mkdir ~/cert/peer # cp ca-peer* ~/cert/peer/ # ls ~/cert/peer/ ca-peer-key.pem  ca-peer.pem

 

In this documentation, there are several security models. Example 2 (Red) and Example 3(Blue) are matched with the command above.

etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \   --listen-peer-urls https://172.22.0.96:2380 \   --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.96:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \   --cert-file=/path/to/infra0-client.crt --key-file=/path/to/infra0-client.key \   --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \   --peer-cert-file=/path/to/infra0-peer.crt --peer-key-file=/path/to/infra0-peer.key

At first, In example 2, I need server to run "etcd" and I need client certificate to get response. And second, In example 3, I need peer certificate to communicate between hosts.

4. Create server, client and peer certificate.

Look at this post, there are way to generate each certificate. I create server certificate. To create, I need to "server.json" file. In this Git, there are sample JSON file. For server.json, hosts information are most important.

IP's currently in the config should be replaced/added with IP addresses of each cluster node, please note 127.0.0.1 is always required for loopback

# cat server.json {   "CN": "infra0-client",   "hosts": [     "172.22.0.96",     "127.0.0.1"   ],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] } # cat server.json {   "CN": "infra1-client",   "hosts": [     "172.22.0.33",     "127.0.0.1"   ],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] } # cat server.json { "CN": "infra2-client", "hosts": [ "172.22.0.133", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }

I will generate server certificate. Please note that the command above use the name "infra-client", however, this is server certification, it is not client certificate.

# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra0-client # mv infra0-client* client/ # cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra1-client # mv infra1-client* client/ # cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra2-client # mv infra2-client* client/

"-profile=server" option makes server certificate. Next, I create the peer certificate. Also I need "peer.json" file. This file is similar with "server.json". However, CommonName (CN) should be different.

# cat peer.json {   "CN": "infra0-peer",   "hosts": [     "172.22.0.96",     "127.0.0.1"   ],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] } # cat peer.json {   "CN": "infra1-peer",   "hosts": [     "172.22.0.33",     "127.0.0.1"   ],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] } # cat peer.json { "CN": "infra2-peer", "hosts": [ "172.22.0.133", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }

I generate peer certificate with peer.json file above. "-profile=peer" is adopted at this time.

# cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra0-peer # mv infra0-peer* peer/ # cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra1-peer # mv infra1-peer* peer/ # cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra2-peer # mv infra2-peer* peer/

I create server certificate and peer certificate. Therefore, I can run "etcd" with both certificate. However, I need client certificate to obtain response. I will use same root certificate of the server. I need "client.json" file.

# cat client.json { "CN": "infra0", "hosts": [""], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] } # cat client.json {   "CN": "infra1",   "hosts": [""],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] } # cat client.json {   "CN": "infra2",   "hosts": [""],   "key": {     "algo": "ecdsa",     "size": 384   },   "names": [     {       "O": "autogenerated",       "OU": "etcd cluster",       "L": "the internet"     }   ] }

I generate peer certificate with client.json file above. "-profile=client" is adopted at this time.

# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra0 # cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra1 # cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra2

5. Run etcd in security mode.

I ready all certificate to run in security mode. Final command looks like below.

etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \   --listen-peer-urls https://172.22.0.96:2380 \   --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.96:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=client/ca-client.pem \   --cert-file=client/infra0-client.pem --key-file=client/infra0-client-key.pem \   --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \   --peer-cert-file=peer/infra0-peer.pem --peer-key-file=peer/infra0-peer-key.pem etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \   --listen-peer-urls https://172.22.0.33:2380 \   --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.33:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=client/ca-client.pem \   --cert-file=client/infra1-client.pem --key-file=client/infra1-client-key.pem \   --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \   --peer-cert-file=peer/infra1-peer.pem --peer-key-file=peer/infra1-peer-key.pem etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \   --listen-peer-urls https://172.22.0.133:2380 \   --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.133:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --client-cert-auth --trusted-ca-file=client/ca-client.pem \   --cert-file=client/infra2-client.pem --key-file=client/infra2-client-key.pem \   --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \   --peer-cert-file=peer/infra2-peer.pem --peer-key-file=peer/infra2-peer-key.pem

At this time, I do not know command line to put and get. However, there is sample curl command in this documentation (Example 2).

curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo -XPUT -d value=bar -v

# curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo -XPUT -d value=bar -v * Trying 172.22.0.96... * Connected to 172.22.0.96 (172.22.0.96) port 2379 (#0) * found 1 certificates in client/ca-client.pem * found 592 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: infra0-client (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: L=the internet,O=autogenerated,OU=etcd cluster,CN=infra0-client * start date: Fri, 19 Oct 2018 16:34:00 GMT * expire date: Sat, 19 Oct 2019 16:34:00 GMT * issuer: C=US,ST=California,L=San Francisco,O=Honest Achmed's Used Certificates,OU=Hastily-Generated Values Divison,CN=CA Client * compression: NULL * ALPN, server did not agree to a protocol > PUT /v2/keys/foo HTTP/1.1 > Host: 172.22.0.96:2379 > User-Agent: curl/7.47.0 > Accept: */* > Content-Length: 9 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 9 out of 9 bytes < HTTP/1.1 201 Created < Content-Type: application/json < X-Etcd-Cluster-Id: e3631873cee60c62 < X-Etcd-Index: 17 < X-Raft-Index: 26 < X-Raft-Term: 47 < Date: Fri, 19 Oct 2018 17:33:13 GMT < Content-Length: 90 < {"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}} # curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo {"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}}

I can see the result {"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}}. It's works now.

6. Run automatic certificate mode

So far, I do so many step to run in security mode. It's is not simple. Because of this, etcd offers automatic certificate mode. In this mode, etcd create certificate automatically. "--auto-tls" and "--peer-auto-tls" are replaced instead of part for certificate.

etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \   --listen-peer-urls https://172.22.0.96:2380 \   --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.96:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --auto-tls \   --peer-auto-tls etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \   --listen-peer-urls https://172.22.0.33:2380 \   --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.33:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --auto-tls \   --peer-auto-tls etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \   --listen-peer-urls https://172.22.0.133:2380 \   --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \   --advertise-client-urls https://172.22.0.133:2379 \   --initial-cluster-token etcd-cluster-1 \   --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \   --initial-cluster-state new \   --auto-tls \   --peer-auto-tls

In Example 4 of this documentation, "curl -k https://127.0.0.1:2379/v2/keys/foo -Xput -d value=bar -v" command is used. (This site also useful)

# curl -k -X "PUT" https://127.0.0.1:2379/v2/keys/seoul -d value='seoul' {"action":"set","node":{"key":"/seoul","value":"seoul","modifiedIndex":18,"createdIndex":18}} # curl -k https://127.0.0.1:2379/v2/keys/seoul {"action":"get","node":{"key":"/seoul","value":"seoul","modifiedIndex":18,"createdIndex":18}} # curl -k -X "DELETE" https://127.0.0.1:2379/v2/keys/seoul

Now, I can use with simple model.

Reference

[ 1 ] http://createnetech.tistory.com/16

[ 2 ] http://createnetech.tistory.com/14

[ 3 ] http://createnetech.tistory.com/12

[ 4 ] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md

[ 5 ] https://coreos.com/etcd/docs/latest/op-guide/security.html

[ 6 ] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/security.md

[ 7 ] https://coreos.com/os/docs/latest/generate-self-signed-certificates.html

[ 8 ] https://github.com/etcd-io/etcd/tree/master/hack/tls-setup

[ 9 ] https://coreos.com/etcd/docs/latest/demo.html

[ 10 ] https://www.mkyong.com/web/curl-put-request-examples/