How to use etcd (multi-machine cluster TLS/SSL security mode) in Ubuntu?
From this post, I run multi-machine cluster basic mode. Now, I will run multi-machine cluster with TLS/SSL security. I am not the security engineer. I am not friendly about TLS/SSL concepts. I studied about key and chain concept before, however it still difficult to understand it. In this documentation, there are 2 things required, a unique key pair (member.crt, member.key) and shared cluster CA certificate (ca.crt). In this post, I address how to create these.
1. Overview the command for multi-machine cluster with TLS/SSL.
From this documentation, the commands are look like below.
etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \ --listen-peer-urls https://172.22.0.96:2380 \ --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.96:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \ --cert-file=/path/to/infra0-client.crt --key-file=/path/to/infra0-client.key \ --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \ --peer-cert-file=/path/to/infra0-peer.crt --peer-key-file=/path/to/infra0-peer.key
etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \ --listen-peer-urls https://172.22.0.33:2380 \ --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.33:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \ --cert-file=/path/to/infra1-client.crt --key-file=/path/to/infra1-client.key \ --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \ --peer-cert-file=/path/to/infra1-peer.crt --peer-key-file=/path/to/infra1-peer.key
etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \ --listen-peer-urls https://172.22.0.133:2380 \ --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.133:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \ --cert-file=/path/to/infra2-client.crt --key-file=/path/to/infra2-client.key \ --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \ --peer-cert-file=/path/to/infra2-peer.crt --peer-key-file=/path/to/infra2-peer.key |
From command line above, there is something unknown like below. I need certificate. However, I do not know how and what can I obtain these.
--client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \ --cert-file=/path/to/infra2-client.crt --key-file=/path/to/infra2-client.key \ --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \ --peer-cert-file=/path/to/infra2-peer.crt --peer-key-file=/path/to/infra2-peer.key |
"cfssl" can do for these. In this Git link, there are introduction about "cfssl" like below.
This demonstrates using Cloudflare's cfssl to easily generate certificates for an etcd cluster. |
2. Download and install the "cfssl"
In this documentation, there are installation guide and usage.
mkdir ~/bin curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x ~/bin/{cfssl,cfssljson} export PATH=$PATH:~/bin |
It is so simple. I verify the version.
# cfssl version Version: 1.2.0 Revision: dev Runtime: go1.6 |
3. Create root certificate (CA).
In the command, there are 2 root certificate, ca-client.crt and ca-peer.crt. It will be shared between all hosts. Thus, I create these certificate on single host and copy into others. To create root certificate, I need 2 files, "config.json" and "csr.json". The contents is here.
# vi ca-client-config.json { "signing": { "default": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } }
# vi ca-client-csr.json {
"CN": "CA Client",
"key": {
"algo": "ecdsa",
"size": 384
},
"names": [
{
"O": "Honest Achmed's Used Certificates",
"OU": "Hastily-Generated Values Divison",
"L": "San Francisco",
"ST": "California",
"C": "US"
}
]
} |
I create "ca-client-config.json" and "ca-client-csr.json" for ca-client certificate. I have to create more for ca-peer certificate. CN name should be different.
# vi ca-peer-config.json { "signing": { "default": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } }
# vi ca-peer-csr.json {
"CN": "CA Peer",
"key": {
"algo": "ecdsa",
"size": 384
},
"names": [
{
"O": "Honest Achmed's Used Certificates",
"OU": "Hastily-Generated Values Divison",
"L": "San Francisco",
"ST": "California",
"C": "US"
}
]
} |
Now, I can generate with files above. Please, note that I will obtain some files which is named "ca.csr", "ca-key.pem" and "ca.pem"
# cfssl gencert -initca ca-client-csr.json | cfssljson -bare ca-client - # cfssl gencert -initca ca-peer-csr.json | cfssljson -bare ca-peer - |
I can verify this is normally good or not.
# openssl x509 -noout -text -in ca-client.pem # openssl x509 -noout -text -in ca-peer.pem |
It's good. I can generate intermediate certification with this certificate. Therefore I need to copy these files into other hosts. I move files on each directory like below. (I create these directory on every hosts)
# mkdir ~/cert/clientm # cp ca-client* ~/cert/client/ # ls ~/cert/client/ ca-client-key.pem ca-client.pem ca-client-config.json
# mkdir ~/cert/peer # cp ca-peer* ~/cert/peer/ # ls ~/cert/peer/ ca-peer-key.pem ca-peer.pem |
In this documentation, there are several security models. Example 2 (Red) and Example 3(Blue) are matched with the command above.
etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \ --listen-peer-urls https://172.22.0.96:2380 \ --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.96:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \ --cert-file=/path/to/infra0-client.crt --key-file=/path/to/infra0-client.key \ --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \ --peer-cert-file=/path/to/infra0-peer.crt --peer-key-file=/path/to/infra0-peer.key |
At first, In example 2, I need server to run "etcd" and I need client certificate to get response. And second, In example 3, I need peer certificate to communicate between hosts.
4. Create server, client and peer certificate.
Look at this post, there are way to generate each certificate. I create server certificate. To create, I need to "server.json" file. In this Git, there are sample JSON file. For server.json, hosts information are most important.
IP's currently in the config should be replaced/added with IP addresses of each cluster node, please note 127.0.0.1 is always required for loopback |
# cat server.json { "CN": "infra0-client", "hosts": [ "172.22.0.96", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }
# cat server.json { "CN": "infra1-client", "hosts": [ "172.22.0.33", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }
# cat server.json
{
"CN": "infra2-client",
"hosts": [
"172.22.0.133",
"127.0.0.1"
],
"key": {
"algo": "ecdsa",
"size": 384
},
"names": [
{
"O": "autogenerated",
"OU": "etcd cluster",
"L": "the internet"
}
]
} |
I will generate server certificate. Please note that the command above use the name "infra-client", however, this is server certification, it is not client certificate.
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra0-client # mv infra0-client* client/
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra1-client # mv infra1-client* client/
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=server server.json | cfssljson -bare infra2-client # mv infra2-client* client/ |
"-profile=server" option makes server certificate. Next, I create the peer certificate. Also I need "peer.json" file. This file is similar with "server.json". However, CommonName (CN) should be different.
# cat peer.json { "CN": "infra0-peer", "hosts": [ "172.22.0.96", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }
# cat peer.json { "CN": "infra1-peer", "hosts": [ "172.22.0.33", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }
# cat peer.json
{
"CN": "infra2-peer",
"hosts": [
"172.22.0.133",
"127.0.0.1"
],
"key": {
"algo": "ecdsa",
"size": 384
},
"names": [
{
"O": "autogenerated",
"OU": "etcd cluster",
"L": "the internet"
}
]
} |
I generate peer certificate with peer.json file above. "-profile=peer" is adopted at this time.
# cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra0-peer # mv infra0-peer* peer/
# cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra1-peer # mv infra1-peer* peer/
# cfssl gencert -ca=peer/ca-peer.pem -ca-key=peer/ca-peer-key.pem -config=peer/ca-peer-config.json -profile=peer peer.json | cfssljson -bare infra2-peer # mv infra2-peer* peer/ |
I create server certificate and peer certificate. Therefore, I can run "etcd" with both certificate. However, I need client certificate to obtain response. I will use same root certificate of the server. I need "client.json" file.
# cat client.json
{
"CN": "infra0",
"hosts": [""],
"key": {
"algo": "ecdsa",
"size": 384
},
"names": [
{
"O": "autogenerated",
"OU": "etcd cluster",
"L": "the internet"
}
]
}
# cat client.json { "CN": "infra1", "hosts": [""], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] }
# cat client.json { "CN": "infra2", "hosts": [""], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] } |
I generate peer certificate with client.json file above. "-profile=client" is adopted at this time.
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra0
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra1
# cfssl gencert -ca=client/ca-client.pem -ca-key=client/ca-client-key.pem -config=client/ca-client-config.json -profile=client client.json | cfssljson -bare infra2 |
5. Run etcd in security mode.
I ready all certificate to run in security mode. Final command looks like below.
etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \ --listen-peer-urls https://172.22.0.96:2380 \ --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.96:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=client/ca-client.pem \ --cert-file=client/infra0-client.pem --key-file=client/infra0-client-key.pem \ --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \ --peer-cert-file=peer/infra0-peer.pem --peer-key-file=peer/infra0-peer-key.pem
etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \ --listen-peer-urls https://172.22.0.33:2380 \ --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.33:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=client/ca-client.pem \ --cert-file=client/infra1-client.pem --key-file=client/infra1-client-key.pem \ --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \ --peer-cert-file=peer/infra1-peer.pem --peer-key-file=peer/infra1-peer-key.pem
etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \ --listen-peer-urls https://172.22.0.133:2380 \ --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.133:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --client-cert-auth --trusted-ca-file=client/ca-client.pem \ --cert-file=client/infra2-client.pem --key-file=client/infra2-client-key.pem \ --peer-client-cert-auth --peer-trusted-ca-file=peer/ca-peer.pem \ --peer-cert-file=peer/infra2-peer.pem --peer-key-file=peer/infra2-peer-key.pem |
At this time, I do not know command line to put and get. However, there is sample curl command in this documentation (Example 2).
curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo -XPUT -d value=bar -v |
# curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo -XPUT -d value=bar -v
* Trying 172.22.0.96...
* Connected to 172.22.0.96 (172.22.0.96) port 2379 (#0)
* found 1 certificates in client/ca-client.pem
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: infra0-client (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC
* certificate version: #3
* subject: L=the internet,O=autogenerated,OU=etcd cluster,CN=infra0-client
* start date: Fri, 19 Oct 2018 16:34:00 GMT
* expire date: Sat, 19 Oct 2019 16:34:00 GMT
* issuer: C=US,ST=California,L=San Francisco,O=Honest Achmed's Used Certificates,OU=Hastily-Generated Values Divison,CN=CA Client
* compression: NULL
* ALPN, server did not agree to a protocol
> PUT /v2/keys/foo HTTP/1.1
> Host: 172.22.0.96:2379
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 9
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 9 out of 9 bytes
< HTTP/1.1 201 Created
< Content-Type: application/json
< X-Etcd-Cluster-Id: e3631873cee60c62
< X-Etcd-Index: 17
< X-Raft-Index: 26
< X-Raft-Term: 47
< Date: Fri, 19 Oct 2018 17:33:13 GMT
< Content-Length: 90
<
{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}}
# curl --cacert client/ca-client.pem --cert infra0.pem --key infra0-key.pem -L https://172.22.0.96:2379/v2/keys/foo {"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}} |
I can see the result {"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":17,"createdIndex":17}}. It's works now.
6. Run automatic certificate mode
So far, I do so many step to run in security mode. It's is not simple. Because of this, etcd offers automatic certificate mode. In this mode, etcd create certificate automatically. "--auto-tls" and "--peer-auto-tls" are replaced instead of part for certificate.
etcd --name infra0 --initial-advertise-peer-urls https://172.22.0.96:2380 \ --listen-peer-urls https://172.22.0.96:2380 \ --listen-client-urls https://172.22.0.96:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.96:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --auto-tls \ --peer-auto-tls
etcd --name infra1 --initial-advertise-peer-urls https://172.22.0.33:2380 \ --listen-peer-urls https://172.22.0.33:2380 \ --listen-client-urls https://172.22.0.33:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.33:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --auto-tls \ --peer-auto-tls
etcd --name infra2 --initial-advertise-peer-urls https://172.22.0.133:2380 \ --listen-peer-urls https://172.22.0.133:2380 \ --listen-client-urls https://172.22.0.133:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://172.22.0.133:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster infra0=https://172.22.0.96:2380,infra1=https://172.22.0.33:2380,infra2=https://172.22.0.133:2380 \ --initial-cluster-state new \ --auto-tls \ --peer-auto-tls |
In Example 4 of this documentation, "curl -k https://127.0.0.1:2379/v2/keys/foo -Xput -d value=bar -v" command is used. (This site also useful)
# curl -k -X "PUT" https://127.0.0.1:2379/v2/keys/seoul -d value='seoul' {"action":"set","node":{"key":"/seoul","value":"seoul","modifiedIndex":18,"createdIndex":18}}
# curl -k https://127.0.0.1:2379/v2/keys/seoul {"action":"get","node":{"key":"/seoul","value":"seoul","modifiedIndex":18,"createdIndex":18}}
# curl -k -X "DELETE" https://127.0.0.1:2379/v2/keys/seoul |
Now, I can use with simple model.
Reference
[ 1 ] http://createnetech.tistory.com/16
[ 2 ] http://createnetech.tistory.com/14
[ 3 ] http://createnetech.tistory.com/12
[ 4 ] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md
[ 5 ] https://coreos.com/etcd/docs/latest/op-guide/security.html
[ 6 ] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/security.md
[ 7 ] https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
[ 8 ] https://github.com/etcd-io/etcd/tree/master/hack/tls-setup
[ 9 ] https://coreos.com/etcd/docs/latest/demo.html
[ 10 ] https://www.mkyong.com/web/curl-put-request-examples/
AWS_Region_ICMP.xlsx