[Cockroach] 노드 추가/삭제

1. 노드 추가 및 삭제

1-1. 노드 추가

 

1-1-1. Certs 생성

Node용 Certificates 생성

cockroach cert create-node 172.16.8.4 crdbsrv-4 node4 localhost 127.0.0.1 172.16.8.4 crdbsrv-7 nodelb --certs-dir=certs --ca-key=my-safe-directory/ca.key

 

1-1-2. 노드 추가
기본 노드 Start와 동일하고 listen-addr 옵션만 추가됨

cockroach start --certs-dir=/app/certs \
                     --advertise-addr=172.16.8.4 \
                     --listen-addr=172.16.30.4 \
                     --join=172.16.8.1,172.16.8.2,172.16.8.3 \
                     --cache=.25 \
                     --max-sql-memory=.25 \
                     --store=/app/cockroach-data

[root@crdbsrv-4 app]# cockroach start --certs-dir=/app/certs --advertise-addr=172.16.8.4 --listen-addr=172.16.30.17 --join=172.16.8.113,172.16.8.115,172.16.8.116 --cache=.25 --max-sql-memory=.25 --store=/app/cockroach-data
*
* INFO: initial startup completed
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
CockroachDB node starting at 2020-11-30 04:18:27.60690325 +0000 UTC (took 0.2s)
build:               CCL v20.2.0 @ 2020/11/09 16:01:45 (go1.13.14)
webui:               https://172.16.30.17:8080
sql:                 postgresql://root@172.16.8.4:26257?sslcert=%2Fapp%2Fcerts%2Fclient.root.crt&sslkey=%2Fapp%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fapp%2Fcerts%2Fca.crt
RPC client flags:    cockroach <client cmd> --host=172.16.8.4:26257 --certs-dir=/app/certs
logs:                /app/cockroach-data/logs
temp dir:            /app/cockroach-data/cockroach-temp090812048
external I/O path:   /app/cockroach-data/extern
store[0]:            path=/app/cockroach-data
storage engine:      pebble
status:              initialized new node, joined pre-existing cluster
clusterID:           4be9abfd-d43c-4d87-9e31-26faf6110658
nodeID:              4
^C

 

 

1-2. 노드 삭제

 

1-2-1. decommission 수행

수행 후 해당 노드에 속한 replicas가 모두 다른 노드로 rebalancing이 되어야 수행이 완료됨

cockroach node decommission --self --certs-dir=/app/certs --host=172.16.30.4

[root@crdbsrv-4 app]# cockroach node decommission --self --certs-dir=/app/certs --host=172.16.8.4

  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        8 |        true        | decommissioning |    false
(1 row)
............
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        6 |        true        | decommissioning |    false
(1 row)
............
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        4 |        true        | decommissioning |    false
(1 row)
........
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        3 |        true        | decommissioning |    false
(1 row)
...........
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        2 |        true        | decommissioning |    false
(1 row)
.............
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        1 |        true        | decommissioning |    false
(1 row)
.............
  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   4 |  true   |        0 |        true        | decommissioning |    false
(1 row)

  id | is_live | replicas | is_decommissioning |   membership   | is_draining
-----+---------+----------+--------------------+----------------+--------------
   4 |  true   |        0 |        true        | decommissioned |    false
(1 row)

No more data reported on target nodes. Please verify cluster health before removing the nodes.
[root@crdbsrv-4 app]#

 

1-2-2. Cockroach process kill

[root@crdbsrv-4 app]# ps -ef|grep cock
root     23249     1  6 15:19 pts/1    00:00:24 cockroach start --certs-dir=/app/certs --advertise-addr=172.16.8.119 --listen-addr=172.16.30.17 --join=172.16.8.113,172.16.8.115,172.16.8.116 --cache=.25 --max-sql-memory=.25 --store=/app/cockroach-data
root     23578 22523  0 15:25 pts/1    00:00:00 grep --color=auto cock
[root@crdbsrv-4 app]# kill -9 23249
[root@crdbsrv-4 app]# ps -ef|grep cock
root     23587 22523  0 15:25 pts/1    00:00:00 grep --color=auto cock
[root@crdbsrv-4 app]#

 

 

1-2-3. 확인

DB Console 등에서 상태를 확인하면 decommission 상태로 내려오기까지 시간이 소요됨

-> 해당 설정을 줄이면 이 시간을 줄일 수 있음

 

파리미터 server.time_until_store_dead 

root@172.16.8.1:26257/defaultdb> show cluster setting server.time_until_store_dead;
  server.time_until_store_dead
--------------------------------
  00:05:00
(1 row)

Time: 1ms total (execution 1ms / network 1ms)

root@172.16.8.1:26257/defaultdb> set cluster setting server.time_until_store_dead = '1m0s';
ERROR: cannot set server.time_until_store_dead to less than 1m15s: 1m0s
root@172.16.8.1:26257/defaultdb> set cluster setting server.time_until_store_dead = '1m15s';
SET CLUSTER SETTING

Time: 26ms total (execution 24ms / network 2ms)

root@172.16.8.1:26257/defaultdb> show cluster setting server.time_until_store_dead;
  server.time_until_store_dead
--------------------------------
  00:01:15
(1 row)

Time: 1ms total (execution 0ms / network 1ms)

root@172.16.8.1:26257/defaultdb>