Docs Home

Etcd

This section describes how to set up and run etcd, which stores OpenSQL's cluster configuration information and settings.

Enviroment configuration

The settings for etcd communication to be installed on the three nodes are mandatory.

Enviroment configuration etcd.env 

# /etc/etcd/etcd.env 
# mandatory configuration
ETCD_NAME=<ETCD_NODE_NAME>

# Initial cluster configuration
ETCD_INITIAL_CLUSTER=<ETCD_NODE_NAME>=http://<NODE1_IP>:2380,<ETCD_NODE_NAME2>=http://<NODE2_IP>:2380, ...
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLSUTER_TOKEN=etcd-cluster

# Peer configuration
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://<NODE_IP>:2380
ETCD_LISTEN_PEER_URLS=http://<NODE_IP>:2380

# Client/server configuration
ETCD_ADVERTISE_CLIENT_URLS=http://<NODE_IP>:2379
ETCD_LISTEN_CLIENT_URLS=http://<NODE_IP>:2379,http://127.0.0.1:2379

#data dir
ETCD_DATA_DIR=/var/lib/etcd

Member and cluster information

Environment Variable
Example
Description

ETCD_NAME

node1

Specifies the name of a unique node within the ETCD cluster. There must not be more than one node with the same name in the cluster and it must match the node name specified in the section describing cluster information later.

ETCD_INITIAL_CLUSTER

node1=http://172.18.0.5:2380,

node2=http://172.18.0.6:2380,

node3=http://172.18.0.7:2380

A list of URLs for communication between peers of ETCD nodes within a cluster separated by commas, used when initializing an ETCD cluster. Must match the ETCD_INITIAL_ADVERTISE_PEER_URLS value defined for each node.

ETCD_INITIAL_CLUSTER_STATE

new

Decides whether to start a new cluster or add this node to an existing cluster. Use new if this is the first node in a new cluster, and existing if this is a new node being added to an existing cluster.

ETCD_INITIAL_CLUSTER_TOKEN

my-etcd-cluster

Nodes with the same token value, which is a unique identifier used for initializing the ETCD cluster, can participate in the cluster.

ETCD_INITIAL_ADVERTISE_PEER_URLS

http://172.18.0.5:2380

A list of peer URLs of this node that are to be made public (advertised) to other nodes in order to communicate between ETCD nodes. A URL must be specified so that other nodes can access this node.

ETCD_LISTEN_PEER_URLS

http://172.18.0.5:2380

A list of URLs that this node's ETCD server will listen to for peer-to-peer communication.

ETCD_LISTEN_CLIENT_URLS

http://172.18.0.5:2379,

https://192.168.0.31:2379,

http://127.0.0.1:2379

A list of URLs that this node's ETCD server will listen to for peer-to-peer communication.

ETCD_ADVERTISE_CLIENT_URLS

https://192.168.0.31:2379

A list of server URLs for this node to be advertised to clients. When cluster member information is retrieved using the ETCD API, this value is displayed as Client Addrs.

ETCD_DATA_DIR

/var/lib/etcd

ETCD's data directory

TLS authentication

To restrict access to ETCD to users with valid client certificates by issuing TLS certificates and to encrypt end-to-end communication with clients, configure the following items.

It is divided into [client-server] authentication and [server-server] peer authentication, and the same TLS certificate set can be used in both cases.

Environment Variable
Example
Description

ETCD_TRUSTED_CA_FILE

/etc/etcd/pki/etcd-ca.pem

This is the signing entity for TLS certificates used for communication between clients and servers, and specifies the certificate path of the root CA (certification authority).

ETCD_CERT_FILE

/etc/etcd/pki/node3.pem

Specifies the certificate path to be used for communication between the client and server.

ETCD_KEY_FILE

/etc/etcd/pki/node3-key.pem

Specify the path of the key file to be used for communication between the client and server.

ETCD_PEER_TRUSTED_CA_FILE

/etc/etcd/pki/etcd-ca.pem

This is the signing entity for the TLS certificate used for peer communication between servers, and specifies the root CA certificate path.

ETCD_PEER_CERT_FILE

/etc/etcd/pki/node3-peer.pem

Specify the certificate path to be used for peer communication between servers.

ETCD_PEER_KEY_FILE

/etc/etcd/pki/node3-peer-key.pem

Specify the path of the key file to be used for peer communication between servers.

Example etcd preferences for a 3-node cluster

If you have the following node addresses and are configuring a cluster with etcd nodes

  • node1 172.176.0.2

  • node2 172.176.0.3

  • node3 172.176.0.4

#/etc/etcd/etcd.env
## Name the etcd node for this server as node1 
ETCD_NAME=node1

## set the names of all etcd nodes in the cluster and their accessible peer URLs
ETCD_INITIAL_CLUSTER=node1=http://172.176.0.2:2380,node2=http://172.176.0.3:2380,node3=http://172.176.0.4:2380
ETCD_INITIAL_CLSUTER_TOKEN=etcd-cluster1
## In the initial cluster configuration, all 3 nodes are set to 'new' and working. Later modified to
existing.
ETCD_INITIAL_CLUSTER_STATE=new

## Specifies the peer URLs used to communicate between etcd nodes. The default port is 2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.176.0.2:2380
ETCD_LISTEN_PEER_URLS=http://172.176.0.2:2380

# Client/server configuration
ETCD_ADVERTISE_CLIENT_URLS=http://172.176.0.2:2379
ETCD_LISTEN_CLIENT_URLS=http://172.176.0.2:2379,http://127.0.0.1:2379

# data dir
ETCD_DATA_DIR=/var/lib/etcd

Execution

Execute etcd with Systemd

Register and manage etcd as a systemd service file on each of the three nodes as shown below.

$ sudo vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd-member.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd/etcd.env
ExecStart=/usr/bin/etcd
Type=notify
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
Nice=-10
IOSchedulingClass=best-effort
IOSchedulingPriority=2
MemoryLow=200M

[Install]
WantedBy=multi-user.target

An example of registering as a Systemd service is shown below. Use the daemon- reload command of the command line tool systemctl to update the configuration with the newly defined etcd.service service definition.

$ sudo systemctl daemon-reload

Once registered, you can enable the service as shown in the example below. Enabled services are automatically started when the system boots.

$ sudo systemctl enable etcd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

Services can be started, stopped, or restarted directly, as shown in the examples below. Even enabled services don't work until explicitly started by a user or rebooted, so run them directly as needed.

$ sudo systemctl start etcd.service

## Example of stopping the service
$ sudo systemctl stop etcd.service

## Example of restarting the service
$ sudo systemctl restart etcd.service

You can check the status of running services as shown in the example below.

$ sudo systemctl status -l etcd
● etcd.service - etcd
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2025-03-17 16:49:45 KST; 3 weeks 2 days ago
     Docs: https://github.com/coreos/etcd
 Main PID: 4763 (etcd)
   CGroup: /system.slice/etcd.service
           └─4763 /usr/bin/etcd

Apr 09 17:46:30 node2 etcd[4763]: {"level":"warn","ts":"2025-04-09T17:46:30.542+0900","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"402.056395ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/service/opensql/\" range_end:\"/service/opensql\" ","response":"range_response_count:9 size:6173"}

Execute etcd with Command

Alternatively, you can also execute it directly from the command line, as shown in the example below.

$ etcd --name 'node1' \
    --data-dir '/var/lib/etcd' \
    --initial-cluster 'node1=http://172.18.0.2:2380,node2=http://172.18.0.3:2380,node3=http://172.18.0.4:2380' \
    --initial-cluster-token 'etcd-cluster1'
    --initial-cluster-state 'new'

Note

Etcd is recommended to be registered as Systemd.

Verification of configuration

Check the status of the etcd cluster on the three nodes by executing the command below.

$ etcdctl member list -w=table
+------------------+---------+-------+----------------------+-------------------------+------------+
|        ID        | STATUS  | NAME  |      PEER ADDRS      |      CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+----------------------+-------------------------+------------+
| bfac432fd36a61d5 | started | etcd3 | http://opensql3:2380 | http://178.176.0.3:2379 |      false |
| c41cad57348d886f | started | etcd2 | http://opensql2:2380 | http://178.176.0.2:2379 |      false |
| dc3ff6e1d56a1012 | started | etcd1 | http://opensql1:2380 | http://178.176.0.4:2379 |      false |
+------------------+---------+-------+----------------------+-------------------------+------------+

PreviousComponents configurationNextPatroni

Last updated