Docs Home

Etcd

This document describes how to use etcdctl, a command line tool installed with ETCD3, to check the status of ETCD3 cluster members and endpoint health, query the values of keys that meet certain conditions, and test the read/write performance of the cluster.

Verify the installation of the command-line tool etcdctl

etcdctl, a command-line tool for managing ETCD3 clusters, is provided with the etcd server binary. You can check if etcdctl is installed on a node and its version.

$ which etcdctl
/usr/local/bin/etcdctl

$ etcdctl version
etcdctl version: 3.5.6
API version: 3.5

There are two versions of etcdctl's API, v2 and v3, and you can specify which version of the API to call when running etcdctl by setting the environment variable ETCDCTL_API to 2 or 3, respectively.

Starting with v3.4.0, the v3 version of the API is used by default, and this document also describes how to use the v3 version of the API.

Common options

This section describes common options that can be specified when running the etcdctl command.

  • --endpoints : etcdctl Specify the endpoint URLs of the ETCD3 gRPC servers to access as a list separated by commas ,. If not specified, the localhost 127.0.0.1:2379 of the running environment is recognized as the default gRPC endpoint URL and communication is attempted. ◦ An invalid Endpoint URL returns an error as shown below.

$ etcdctl member list
Error:  dial tcp 127.0.0.1:2379: connect: connection refused

  • -w, --write-out : Options for setting the format of output values include fields, json, protobuf, simple, and table .

  • --key : Specifies the path to the Client Key file to use for TLS.

  • --cert : Specifies the path to the Client certificate file to use for TLS authentication.

  • --cacert : : Specifies the path to the certificate authority (CA) certificate file to use for TLS.

Example

$ ETCDCTL_API=3 etcdctl --endpoints="https://192.168.0.10:2379,https://192.168.0.11:2379" \
  --key="./etcd-client-key.pem" \
  --cert="./etcd-client-crt.pem" \
  --cacert="./etcd-ca-crt.pem" \
  member list

Note

https://github.com/etcd-io/etcd/issues/9600

View Cluster status

Get a list and status of the cluster's members and endpoint URLs

View cluster members

etcdctl member list in the etcdctl member list, the URL for advertised peer communication list and a list of URLs for client communication.

$ etcdctl member list
670b863301943618, started, node1, http://192.168.0.10:2380, http://192.168.0.10:2379, false
7825d7b04510b842, started, node3, http://192.168.0.11:2380, http://192.168.0.11:2379, false
c8245114d55ec576, started, node2, http://192.168.0.12:2380, http://192.168.0.12:2379, false

$ etcdctl member list -w table
+------------------+---------+-------+--------------------------+--------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+----------------------------+------------------------+------------+
| 670b863301943618 | started | node1 | http://192.168.0.10:2380 | http://192.168.0.10:2379 |      false |
| 7825d7b04510b842 | started | node3 | http://192.168.0.11:2380 | http://192.168.0.11:2379 |      false |
| c8245114d55ec576 | started | node2 | http://192.168.0.12:2380 | http://192.168.0.12:2379 |      false |
+------------------+---------+-------+----------------------------+------------------------+------------+

  • ID : A unique identifier for that node, used by the Raft algorithm to elect an ETCD leader node.

  • STATUS : A value indicating whether the node has booted and successfully participated in the cluster's leader election. started or unstarted. For ETCD3 nodes that have not yet been started, it has the value unstarted.

  • NAME : The unique name of the node, specified by the --name argument when starting the ETCD3

  • PEER ADDRS : A list of URLs for this node's advertised peer communications.

  • CLIENT ADDRS : A list of URLs for this node's advertised client communications.

  • IS LEARNER : The member has learned the snapshot and WAL replication status from the ETCD cluster. Indicates if it is a Learner node that supports but does not participate in the leader election quorum.

View Cluster endpoint

You can check the access URL, database size, leader status, and Raft information of all member nodes in the cluster with etcdctl endpoint status .

$ etcdctl endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.0.10:2379 | 670b863301943618 |  3.5.21 |  168 kB |      true |      false |         6 |        520 |                520 |        |
| http://192.168.0.11:2379 | c8245114d55ec576 |  3.5.21 |  168 kB |     false |      false |         6 |        520 |                520 |        |
| http://192.168.0.12:2379 | 7825d7b04510b842 |  3.5.21 |  168 kB |     false |      false |         6 |        520 |                520 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

  • ENDPOINT : A list of URLs for the advertised client communication of this node.

  • ID : Same as ENDPOINT

  • VERSION : Indicates the version of the ETCD3 server running on this node.

  • DB SIZE : Indicates the size of the ETCD3 database stored on disk.

  • IS LEADER : Indicates whether the node is the Raft Leader for this cluster.

  • IS LEARNER : Same as IS LEADER

  • RAFT TERM : Current term of office (Term) according to the Raft algorithm. The value is incremented by 1 each time a new Leader is elected.

  • RAFT INDEX : Indicates the log entry location of the write operations stored on this node. The node with the highest Raft Index (i.e., the most up-to-date), which is referenced when electing a leader, has priority when electing a leader.

  • RAFT APPLIED INDEX : Indicates the location of the log entry that is applied to the local environment key-value store of the node and is readable (Applied). Depending on the CPU and disk environment of the node, the Raft Applied Index may have a value less than the Raft Index (i.e., a data write operation) may occur, in which case Data Consistency may not be maintained depending on client settings. The behavior depends on the value of the etcdctl get flag --consistency. -- consistency=s (Serializable, default) fetches data from a point in time in the past.

  • ERRORS : : Represents the problem detected on the endpoint of this node in the form of a message.

View Data 데이터 조회

View data stored in an ETCD3 cluster.

View with Prefix

You can specify the --prefix option as an argument to the etcdctl get command to retrieve keys that match a specific preceding string.

$ etcdctl get --prefix "/opensql/opensql/members" -w simple
/opensql/opensql/members/pg-1
{"conn_url":"postgres://192.168.131.12:5432/postgres","api_url":"http://192.168.131.12:8008/patroni","state":"running","role":"replica","version":"4.0.5","proxy_url":"postgres://192.168.131.15:6432/postgres","xlog_location":223510016,"replication_state":"streaming","timeline":3}
/opensql/opensql/members/pg-2
{"conn_url":"postgres://192.168.131.13:5432/postgres","api_url":"http://192.168.131.13:8008/patroni","state":"running","role":"primary","version":"4.0.5","proxy_url":"postgres://192.168.131.15:6432/postgres","xlog_location":223510016,"timeline":3}
/opensql/opensql/members/pg-3
{"conn_url":"postgres://192.168.131.14:5432/postgres","api_url":"http://192.168.131.14:8008/patroni","state":"running","role":"replica","version":"4.0.5","proxy_url":"postgres://192.168.131.15:6432/postgres","xlog_location":223510016,"replication_state":"streaming","timeline":3}

View with Range

You can get a list of keys that fall betweentwo points in the key index space, precisely [key, range_end), by specifying a specific key value key and a different key range_end as arguments to the etcdctl get command.

  • All keys stored in ETCD3 are replaced by an array of bytes and stored in the Index space, and are indexed case-sensitively. aa < ab a\xff < b

  • Keys matching range_end are not included. (Exclusive)

$ etcdctl get "/opensql/opensql/members/pg-1" "/opensql/opensql/members/pg-3"
/opensql/opensql/members/pg-1
{"conn_url":"postgres://192.168.131.12:5432/postgres","api_url":"http://192.168.131.12:8008/patroni","state":"running","role":"replica","version":"4.0.5","proxy_url":"postgres://192.168.131.15:6432/postgres","xlog_location":223510016,"replication_state":"streaming","timeline":3}
/opensql/opensql/members/pg-2
{"conn_url":"postgres://192.168.131.13:5432/postgres","api_url":"http://192.168.131.13:8008/patroni","state":"running","role":"primary","version":"4.0.5","proxy_url":"postgres://192.168.131.15:6432/postgres","xlog_location":223510016,"timeline":3}

$ etcdctl get "/opensql/opensql/failover" "/opensql/opensql/historz"
/opensql/opensql/failover
{}
/opensql/opensql/history
[[1,223438192,"no recovery target specified","2025-04-18T14:27:03.269756+09:00","pg-1"],[2,223438824,"no recovery target specified","2025-04-21T15:13:37.632010+09:00","pg-2"]]

Database maintenance

ETCD3 uses Multi-Version Concurrency Control (MVCC) based on revision information to control concurrent client access and maintain data consistency. If the cluster is maintained for a long period of time and there are many operations such as Patroni Switchover / Failover, there may be a problem that the DB size continues to increase even though the actual data size stored by Patroni in the ETCD3 cluster is small.

Specify which revisions to clean up

The etcdctl compact command specifies a specific revision to define revisions from a point in time before that as no longer referenced. This behavior is a cluster-wide operation and can be executed without specifying a separate endpoint.

## Check Endpoint Status to get the most recent revision information maintained by each node.
$ etcdctl endpoint status -w json | jq | grep 'revision'
        "revision": 447871,
        "revision": 447871,
        "revision": 447871,

## Marks revisions prior to this revision as no longer referenced.
$ rev=447871
$ etcdctl compact $rev
compacted revision 447871

Disk defragmentation

The etcdctl defrag command deletes unreferenced revisions from the DB File to free up disk space. Without a separate --endpoints option, only the local node's disk space is freed. If you specify all endpoint URLs in the cluster with --endpoints option, it will free up disk space on each host.

$ etcdctl defrag --endpoints "http://192.168.0.10:2379,http://192.168.0.11:2379,http://192.168.0.12:2379"
Finished defragmenting etcd member[http://192.168.0.10:2379]
Finished defragmenting etcd member[http://192.168.0.11:2379]
Finished defragmenting etcd member[http://192.168.0.12:2379]

$ etcdctl endpoint status --endpoints "..."
http://192.168.0.10:2379, 670b863301943618, 3.5.21, 25 kB, true, false, 8, 448000, 448000, 
http://192.168.0.11:2379, c8245114d55ec576, 3.5.21, 25 kB, false, false, 8, 448000, 448000, 
http://192.168.0.12:2379, 7825d7b04510b842, 3.5.21, 25 kB, false, false, 8, 448000, 448000,

Write performance test

You can test the overall write throughput of your cluster for one minute with the etcdctl check perf command.

$ etcdctl check perf
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooom! 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.291186s
PASS: Stddev is 0.029467s
PASS

--load option allows you to specify a different number of clients to request and a different number of requests per second.

  • s : 50 Clients

  • m : 200 Clients

  • l : 500 Clients

  • xl : 1000 Clients

The output will have a PASS / FAIL Criteria.

  • More than 90% of the requests generated will have Throughput.

  • All requests will be processed within 500 ms.

  • • The standard deviation (stddev) of the time it took to process the request is 100 ms or less.

$ etcdctl check perf
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooom! 100.00% 1m0s
PASS: Throughput is 150 writes/s
Slowest request took too long: 0.535645s
PASS: Stddev is 0.079037s
FAIL

$ etcdctl check perf --load="xl"
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooom! 100.00% 1m0s
FAIL: Throughput too low: 3668 writes/s
Slowest request took too long: 0.645609s
Stddev too high: 0.105516s
FAIL

If the load imposed by the performance test is heavy, it is recommended to perform compact and defrag operations because the ETCD database size can increase significantly due to DB revision history.

Recovery

ETCD3 manages all of the cluster's configuration, including itself, such as the cluster nodes' connection URLs, cluster tokens, and listen URLs, like a database snapshot. If you change the configuration of a node in an already configured ETCD3 cluster and restart it with only the key-value data preserved, it will generally not work.

If you want to change the environmental configuration of an already configured ETCD3 cluster to restart it, such as a failure caused by a service outage that requires a scale-in, a scale-out by adding new nodes, or a change in the IPv4 address of a host that requires the cluster to be reset, you should use the command line tool etcdctl to perform the recovery.

Veryfy snapshot

For recovery, you need a snapshot of the ETCD3 node that contains the key-value data you want to import. A snapshot of ETCD3 corresponds to a data file in a relational database and stores not only the key- value data stored in the ETCD3 cluster at a specific point in time, but also the configuration information and raft state of the cluster. You can import a database file of ETCD3 stored in the file system or create it from a running ETCD3 cluster.

Check in the filesystem

The ETCD3 data path ETCD_DATA_DIR is located in the subpath member/snap/db.

$ ls -l $ETCD_DATA_DIR/member/snap/
total 168
-rw-------. 1 opensql opensql 16805888 Apr 18 14:29 db

$ cp $ETCD_DATA_DIR/member/snap/db ./mysnapshot.db

Generate from a running ETCD3

You can take a snapshot of the current point in time from the ETCD3 server using the snapshot save command in the command line tool etcdctl .

  • Only one ETCD3 server URL should be given as the ENDPOINT argument, even if a cluster of multiple nodes is running

$ ETCDCTL_API=3 etcdctl --endpoints=${ENDPOINT} snapshot save mysnapshot.db

{"level":"info","ts":"2025-04-21T12:18:30.406283+0900","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"mysnapshot.db.part"}
{"level":"info","ts":"2025-04-21T12:18:30.407142+0900","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2025-04-21T12:18:30.407162+0900","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"192.168.131.12:2379"}
{"level":"info","ts":"2025-04-21T12:18:30.432538+0900","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2025-04-21T12:18:30.480983+0900","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"192.168.131.12:2379","size":"168 kB","took":"now"}
{"level":"info","ts":"2025-04-21T12:18:30.481039+0900","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"mysnapshot.db"}
Snapshot saved at mysnapshot.db

Recovering data from snapshots

You can create ETCD3 cluster data from a snapshot using the snapshot restore command in the command line tool etcdctl.

  • As with the initial configuration of an ETCD3 cluster, all of the arguments included in the Database

    Snapshot during initial setup of a newly configured cluster must be given as command line arguments.

    • --name : Node name of the ETCD3 server to be repaired

    • --initial-cluster : Node name of all ETCD3 servers in the cluster and endpoint URL for

      peer communication

    • --initial-cluster-token : Initialize Token to join the cluster

    • --initial-advertise-peer-urls : This node's Endpoint URL for peer communication

  • --skip-hash-check option is used to skip verifying the data integrity of the original snapshot being recovered.

    • For snapshots created by a running ETCD3 server with the snapshot save command, data integrity is preserved and can be restored normally without this option.

    • ETCD3 data db files copied from the filesystem contain cluster meta-information at runtime and cannot be recovered normally without this option. Skip data integrity verification by giving the --skip- hash-check option.

$ ETCDCTL_API=3 etcdctl snapshot restore ./mysnapshot.db \
  --name node3 \
  --initial-cluster node1=http://192.168.0.8:2380,node2=http://192.168.0.9:2380,node3=http://192.168.0.10:2380 \
  --initial-cluster-token new-etcd-cluster \
  --initial-advertise-peer-urls http://192.168.0.10:2380 \
  --skip-hash-check \
  # ...

Deprecated: Use `etcdutl snapshot restore` instead.

snapshot/v3_snapshot.go:248	restoring snapshot	{"path": "member/snap/db", "wal-dir": "node3.etcd/member/wal", "data-dir": "node3.etcd", "snap-dir": "node3.etcd/member/snap", "stack": "go.etcd.io/..." }
membership/store.go:141	Trimming membership information from the backend...
membership/cluster.go:421	added member	{"cluster-id": "154dfe96307df6f0", "local-member-id": "0", "added-peer-id": "3dfe6fc7fff49d22", "added-peer-peer-urls": ["http://192.168.0.8:2380"]}
membership/cluster.go:421	added member	{"cluster-id": "154dfe96307df6f0", "local-member-id": "0", "added-peer-id": "7f846315e3b9872d", "added-peer-peer-urls": ["http://192.168.0.9:2380"]}
membership/cluster.go:421	added member	{"cluster-id": "154dfe96307df6f0", "local-member-id": "0", "added-peer-id": "c0ea9022befd3eaa", "added-peer-peer-urls": ["http://192.168.0.10:2380"]}
snapshot/v3_snapshot.go:269	restored snapshot	{"path": "member/snap/db", "wal-dir": "node3.etcd/member/wal", "data-dir": "node3.etcd", "snap-dir": "node3.etcd/member/snap"}

  • • If the data is successfully recovered, a new cluster is created inside the directory named

    ${ETCD_NAME}.etcd.

$ ls -l
total 0
drwx------. 3 root root  20 Mar 17 14:48 node1.etcd

$ ls -l node1.etcd/
drwx------. 2 root root 246 Mar 17 14:41 snap
drwx------. 2 root root 257 Mar 17 14:41 wal

  • • Restart the ETCD3 service based on the recovered data directory. The arguments used to recover the data must match the arguments used to restart the service. The created ${ETCD_NAME}.etcd directory should be located in the member/ subdirectory

$ rm -rf $ETCD_DATA_DIR/member

$ cp -r node1.etcd $ETCD_DATA_DIR/member

## Check the contents of the etcd.env file referenced in the Service definition
$ vi /etc/etcd/etcd.env
ETCD_NAME=node1

ETCD_INITIAL_CLUSTER=node1=http://192.168.0.8:2380,node2=http://192.168.0.9:2380,node3=http://192.168.0.10:2380
ETCD_INITIAL_CLUSTER_TOKEN=new-etcd-cluster
ETCD_INITIAL_CLUSTER_STATE=new
  --initial-cluster-token new-etcd-cluster \
  --initial-advertise-peer-urls http://192.168.0.8:2380 \

$ systemctl restart etcd.service

Note

https://github.com/etcd-io/etcd/blob/v3.4.19/Documentation/op-guide/recovery.md

PreviousPatroniNextTLS authentication

Last updated