0. 참고 자료
www.cockroachlabs.com/docs/v20.2/deploy-cockroachdb-on-premises
Deploy CockroachDB On-Premises | CockroachDB Docs
www.cockroachlabs.com
1. 목표
여러 노드에 Secure 모드로 구성
- 구성
. 3개 node (crdbsrv-1, crdbsrv-2, crdbsrv-3)에 cockroach 설치 및 구성
2. Check Requirements
2-1. Production checklist
참고 : www.cockroachlabs.com/docs/v20.2/recommended-production-settings
2-2. Topology Pattern
참고: www.cockroachlabs.com/docs/v20.2/topology-patterns
3. Prepare
3-1. Time sync
개요
- Scale-Out DB에서 각 노드간의 Timestamp가 굉장히 중요함 (Time Gap 발생하는 경우 node down 발생)
- DB Console에도 Time gap 현황을 제공
3-1-1. ntp 설정
install ntp
root@crdbsrv-2 app]# yum install ntp
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.kakao.com
* extras: mirror.kakao.com
* updates: mirror.kakao.com
Resolving Dependencies
--> Running transaction check
---> Package ntp.x86_64 0:4.2.6p5-29.el7.centos.2 will be installed
--> Processing Dependency: ntpdate = 4.2.6p5-29.el7.centos.2 for package: ntp-4.2.6p5-29.el7.centos.2.x86_64
--> Processing Dependency: libopts.so.25()(64bit) for package: ntp-4.2.6p5-29.el7.centos.2.x86_64
--> Running transaction check
---> Package autogen-libopts.x86_64 0:5.18-5.el7 will be installed
---> Package ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=================================================================================================================
Package Arch Version
=================================================================================================================
Installing:
ntp x86_64 4.2.6p5-29.el7.centos.2
Installing for dependencies:
autogen-libopts x86_64 5.18-5.el7
ntpdate x86_64 4.2.6p5-29.el7.centos.2
Transaction Summary
=================================================================================================================
Install 1 Package (+2 Dependent packages)
Total download size: 701 k
Installed size: 1.6 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): autogen-libopts-5.18-5.el7.x86_64.rpm
(2/3): ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm
(3/3): ntpdate-4.2.6p5-29.el7.centos.2.x86_64.rpm
-----------------------------------------------------------------------------------------------------------------
Total
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : autogen-libopts-5.18-5.el7.x86_64
Installing : ntpdate-4.2.6p5-29.el7.centos.2.x86_64
Installing : ntp-4.2.6p5-29.el7.centos.2.x86_64
Verifying : ntpdate-4.2.6p5-29.el7.centos.2.x86_64
Verifying : ntp-4.2.6p5-29.el7.centos.2.x86_64
Verifying : autogen-libopts-5.18-5.el7.x86_64
Installed:
ntp.x86_64 0:4.2.6p5-29.el7.centos.2
Dependency Installed:
autogen-libopts.x86_64 0:5.18-5.el7 ntpdate.x86_64 0:4.2.6p5-29.el7.ce
Complete!
ntp 설정
[root@crdbsrv-2 app]# systemctl start ntpd
[root@crdbsrv-2 app]# ntpd -q
[root@crdbsrv-2 app]# systemctl stop ntpd
[root@crdbsrv-2 app]# vi /etc/ntp
ntp/ ntp.conf
[root@crdbsrv-2 app]# vi /etc/ntp.conf
[root@crdbsrv-2 app]# more /etc/ntp.conf
~~
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
~~
[root@crdbsrv-2 app]# firewall-cmd --add-service=ntp --permenent
-bash: firewall-cmd: command not found
[root@crdbsrv-2 app]# systemctl start ntpd
[root@crdbsrv-2 app]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
send.mx.cdnetwo .INIT. 16 u - 64 0 0.000 0.000 0.000
x.ns.gin.ntt.ne .INIT. 16 u - 64 0 0.000 0.000 0.000
ip.0xt.ca .INIT. 16 u - 64 0 0.000 0.000 0.000
195.219.205.9 .INIT. 16 u - 64 0 0.000 0.000 0.000
[root@crdbsrv-2 app]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
3-1-2. timezone 설정
Timezone 설정
[root@crdbsrv-2 sysconfig]# timedatectl
Local time: Wed 2020-11-11 02:04:50 UTC
Universal time: Wed 2020-11-11 02:04:50 UTC
RTC time: Wed 2020-11-11 02:04:50
Time zone: UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
[root@crdbsrv-2 sysconfig]# timedatectl list-timezones|grep Seoul
Asia/Seoul
[root@crdbsrv-2 sysconfig]# timedatectl set-timezone Asia/Seoul
[root@crdbsrv-2 sysconfig]# date
Wed Nov 11 11:05:32 KST 2020
[root@crdbsrv-2 sysconfig]# timedatectl
Local time: Wed 2020-11-11 11:05:39 KST
Universal time: Wed 2020-11-11 02:05:39 UTC
RTC time: Wed 2020-11-11 02:05:38
Time zone: Asia/Seoul (KST, +0900)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
[root@crdbsrv-2 sysconfig]#
3-1-3. binary copy
cockroach binary file은 미리 최신 버전으로 download -> 각 서버에 copy
Copy the binary into your PATH so you can execute cockroach commands from any shell:
cp -i cockroach-v20.2.0.linux-amd64/cockroach /usr/local/bin/
[root@crdbsrv-1 app]# cp cockroach-v20.2.0.linux-amd64/cockroach /usr/local/bin/
mkdir -p /usr/local/lib/cockroach
cp -i cockroach-v20.2.0.linux-amd64/lib/libgeos.so /usr/local/lib/cockroach/
cp -i cockroach-v20.2.0.linux-amd64/lib/libgeos_c.so /usr/local/lib/cockroach/
[root@crdbsrv-1 app]# mkdir -p /usr/local/lib/cockroach
[root@crdbsrv-1 app]# cp -i cockroach-v20.2.0.linux-amd64/lib/libgeos.so /usr/local/lib/cockroach/
[root@crdbsrv-1 app]# cp -i cockroach-v20.2.0.linux-amd64/lib/libgeos_c.so /usr/local/lib/cockroach/
Verify
[root@crdbsrv-1 app]# which cockroach
/usr/local/bin/cockroachdemo
[root@crdbsrv-1 app]# cockroach demo
#
# Welcome to the CockroachDB demo database!
#
# You are connected to a temporary, in-memory CockroachDB cluster of 1 node.
#
# This demo session will attempt to enable enterprise features
# by acquiring a temporary license from Cockroach Labs in the background.
# To disable this behavior, set the environment variable
# COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING=true.
#
# Beginning initialization of the movr dataset, please wait...
#
# The cluster has been preloaded with the "movr" dataset
# (MovR is a fictional vehicle sharing company).
#
# Reminder: your changes to data stored in the demo session will not be saved!
#
# Connection parameters:
# (console) http://127.0.0.1:39873
# (sql) postgres://root:admin@?host=%2Ftmp%2Fdemo714389800&port=26257
# (sql/tcp) postgres://root:admin@127.0.0.1:33343?sslmode=require
#
#
# The user "root" with password "admin" has been created. Use it to access the Web UI!
#
# Server version: CockroachDB CCL v20.2.0 (x86_64-unknown-linux-gnu, built 2020/11/09 16:01:45, go1.13.14) (same version as client)
# Cluster ID: e9b04e29-9dce-4114-8c5e-50f94e77d5d1
# Organization: Cockroach Demo
#
# Enter \? for a brief introduction.
#
root@127.0.0.1:33343/movr> show tables;
schema_name | table_name | type | estimated_row_count
--------------+----------------------------+-------+----------------------
public | promo_codes | table | 0
public | rides | table | 0
public | user_promo_codes | table | 0
public | users | table | 0
public | vehicle_location_histories | table | 0
public | vehicles | table | 0
(6 rows)
Time: 19ms total (execution 19ms / network 0ms)
root@127.0.0.1:33343/movr> \q
[root@crdbsrv-1 app]#
4. Certificates
4-1. 개요
Node 생성, DB 접속 모두 미리 생성된 Certificates를 통하여 접속 해야 함
- 아래는 Manual대로 수행한 내용
- Manual대로 하지 않으면 약간 편리한 것들
. certificates 용 directory를 /app/ 아래 위치가능 -> 이후 DB 사용 시 절대경로를 주면 됨
. certificates는 CA, Node용, Client용이 있으며
Node용 manual에는 각 node별로 각각 생성하도록 되어 있으나
모든 Node의 모든 IP를 한꺼번에 등록하여 각 Node에 동일한 Node용 certificate를 두고 사용가능
4-1-1. Make directory
각 node에 /app directory를 기본 base directory로 사용함
rm -rf certs my-safe-directory
mkdir certs
mkdir my-safe-directory
[root@crdbsrv-1 app]# mkdir certs
[root@crdbsrv-1 app]# mkdir my-safe-directory4-2. Certificates 생성
4-1-2. Create the CA certificate and Key
cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
[root@crdbsrv-1 app]# cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
[root@crdbsrv-1 app]# tree certs/ my-safe-directory/
certs/
└── ca.crt
my-safe-directory/
└── ca.key
0 directories, 2 files
[root@crdbsrv-1 app]#
4-1-3. 본문Create the certificate and key for the first node
node1번에 대한 certificate 생성하여 node1번에 저장 (아래 IP list에서 node1에 대한 모든 IP 등록하면 됨)
cockroach cert create-node <node1 internal IP address> \
<node1 external IP address> \
<node1 hostname> \
<other common names for node1> \
localhost 127.0.0.1 \
<load balancer IP address> \
<load balancer hostname> \
<other common names for load balancer instances> \
--certs-dir=certs --ca-key=my-safe-directory/ca.key
[root@crdbsrv-1 app]# cockroach cert create-node 172.16.8.1 crdbsrv-1 node1 localhost 127.0.0.1 172.16.8.7 crdbsrv-7 nodelb --certs-dir=certs --ca-key=my-safe-directory/ca.key
[root@crdbsrv-1 app]# tree certs/ my-safe-directory/ certs/
├── ca.crt
├── node.crt
└── node.key
my-safe-directory/
└── ca.key
0 directories, 4 files
[root@crdbsrv-1 app]#
4-1-4. Upload the CA certificate and node certificate and key to the first node
생성된 파일을 node1으로 옮기고 node2,3용 certificates를 생성하기 위해서 기존 directory를 삭제
ssh <username>@<node1 address> "mkdir certs"
scp certs/ca.crt certs/node.crt certs/node.key <username>@<node1 address>:~/certs
rm certs/node.crt certs/node.key
[root@crdbsrv-1 app]# ssh root@crdbsrv-1 "mkdir certs"
Password:
[root@crdbsrv-1 app]# scp certs/ca.crt certs/node.crt certs/node.key root@crdbsrv-1:~/certs
Password:
ca.crt 100% 1111 742.7KB/s 00:00
node.crt 100% 1216 1.3MB/s 00:00
node.key 100% 1679 1.7MB/s 00:00
[root@crdbsrv-1 app]# rm certs/node.crt certs/node.key
[root@crdbsrv-1 app]#
4-1-5. Node2, Nod3용 ceritificates 생성 및 적용
위의 4-1-3, 4-1-4를 반복하여 node2,node3용 certificate 생성
※ node용 certificates를 생성할 때 전체 node에 대한 모든 IP를 등록하여 node용 certificates는 공동으로 사용 가능
4-1-6. Client 용 Certificates 생성
cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key
ssh <username>@<workload address> "mkdir certs"
scp certs/ca.crt certs/client.root.crt certs/client.root.key <username>@<workload address>:~/certs
[root@crdbsrv-1 app]# cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key
[root@crdbsrv-1 app]# ssh root@crdbsrv-8 "mkdir certs"
Password:
[root@crdbsrv-1 app]# scp certs/ca.crt certs/client.root.crt certs/client.root.key root@crdbsrv-8:~/certs
Password:
ca.crt 100% 1111 936.6KB/s 00:00
client.root.crt 100% 1099 1.1MB/s 00:00
client.root.key 100% 1679 1.4MB/s 00:00
[root@crdbsrv-1 app]#
[root@crdbsrv-8 app]# tree /root/certs
/root/certs
├── ca.crt
├── client.root.crt
└── client.root.key
0 directories, 3 files
[root@crdbsrv-8 app]#
5. DB 구성 및 수행
5-1. 개요
각 Node별 instance를 각각 start 한 후 initialize를 통해서 하나의 Cluster로 묶게 됨
- Start 명령에서 Manual에서는 --background로 수행하도록 되어 있으나 처음에는 해당 옵션을 지우고
수행하는 것이 좋음 (어떤 Node로 등록이 되었는지 등의 정보 확인 가능)
- 정상 동작 하는 것을 확인 한 후에 --background 옵션 사용하는 것이 편리함
5-2. 개별 Node Start
Start Cockroach on Node1
cockroach start --certs-dir=certs --advertise-addr=<node1 address> --join=<node1 address>,<node2 address>,<node3 address> --cache=.25 --max-sql-memory=.25 --background
[root@crdbsrv-1 app]# cockroach start --certs-dir=certs --advertise-addr=172.16.8.1 --join=172.16.8.1,172.16.8.2,172.16.8.3 --cache=.25 --max-sql-memory=.25
*
* WARNING: [n?] advertise address "172.16.8.1" not in node certificate (IP=172.16.8.2,127.0.0.1,172.16.8.8; DNS=crdbsrv-3,node2,localhost,crdbsrv-7,nodelb; CN=node)
* advertise SQL address "172.16.8.1" not in node certificate (IP=172.16.8.3,127.0.0.1,172.16.8.3; DNS=crdbsrv-3,node2,localhost,crdbsrv-7,nodelb; CN=node)
* Secure client connections are likely to fail.
* Consider extending the node certificate or tweak --listen-addr/--advertise-addr/--sql-addr/--advertise-sql-addr.
*
*
* INFO: initial startup completed
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
*
* WARNING: The server appears to be unable to contact the other nodes in the cluster. Please try:
*
* - starting the other nodes, if you haven't already;
* - double-checking that the '--join' and '--listen'/'--advertise' flags are set up correctly;
* - running the 'cockroach init' command if you are trying to initialize a new cluster.
*
* If problems persist, please see https://www.cockroachlabs.com/docs/v20.2/cluster-setup-troubleshooting.html.
*
Start Cockroach on Node2, Node3
더보기
ndoe2, node3도 수행
[root@crdbsrv-2 ~]# cockroach start --certs-dir=certs --advertise-addr=172.16.8.2 --join=172.16.8.1,172.16.8.2,172.16.8.3 --cache=.25 --max-sql-memory=.25
*
* INFO: initial startup completed
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
*
* WARNING: The server appears to be unable to contact the other nodes in the cluster. Please try:
*
* - starting the other nodes, if you haven't already;
* - double-checking that the '--join' and '--listen'/'--advertise' flags are set up correctly;
* - running the 'cockroach init' command if you are trying to initialize a new cluster.
*
* If problems persist, please see https://www.cockroachlabs.com/docs/v20.2/cluster-setup-troubleshooting.html.
*
[root@crdbsrv-3 ~]# cockroach start --certs-dir=certs --advertise-addr=172.16.8.3 --join=172.16.8.1,172.16.8.2,172.16.8.3 --cache=.25 --max-sql-memory=.25
*
* INFO: initial startup completed
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
*
* WARNING: The server appears to be unable to contact the other nodes in the cluster. Please try:
*
* - starting the other nodes, if you haven't already;
* - double-checking that the '--join' and '--listen'/'--advertise' flags are set up correctly;
* - running the 'cockroach init' command if you are trying to initialize a new cluster.
*
* If problems persist, please see https://www.cockroachlabs.com/docs/v20.2/cluster-setup-troubleshooting.html.
*
5-3. Initialize
Initialize
[root@crdbsrv-1 app]# cockroach init --certs-dir=certs --host=172.16.8.1
Cluster successfully initialized
6. DB Console 접속
6-1. 개요
DB Console을 접속하기 위해서 필요한 사항
- 접속용 ID가 필요함 (Node Start 등은 모두 certificates를 통하지만...)
6-2. DB 접속 테스트
6-2-1. 단순 접속 테스트
cockroach sql --certs-dir=certs --host=<address of load balancer>
[root@crdbsrv-1 app]# ./cockroach sql --certs-dir=certs --host=172.16.8.1
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.7 (x86_64-unknown-linux-gnu, built 2020/10/12 16:04:22, go1.13.9) (same version as client)
# Cluster ID: 0200f715-2980-43cb-98ac-a0f387f77f9c
#
# Enter \? for a brief introduction.
#
root@172.16.8.1:26257/defaultdb>
root@172.16.8.1:26257/defaultdb> CREATE DATABASE securenodetest;
CREATE DATABASE
Time: 7.411241ms
root@172.16.8.1:26257/defaultdb> SHOW DATABASES;
database_name
------------------
defaultdb
postgres
securenodetest
system
(4 rows)
Time: 2.514852ms
root@172.16.8.1:26257/defaultdb>
root@172.16.8.1:26257/defaultdb> \q
6-2-2. DB Console용 ID 생성
[root@crdbsrv-1 app]# cockroach sql --certs-dir=certs --host=172.16.8.1
CREATE USER citec WITH PASSWORD 'citec';
root@172.16.8.1:26257/defaultdb> CREATE USER bench WITH PASSWORD 'bench';
CREATE ROLE
Time: 91.857375ms
root@172.16.8.1:26257/defaultdb> grant admin to bench;
GRANT
Time: 193ms total (execution 71ms / network 122ms)
7. HA Proxy 설정
7-1. 개요
HAProxy is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster
7-2. HA Proxy Config파일 생성
Cockroach를 통해 HA Proxy config 파일 생성
7-2-1. Generate config files for HA proxy
cockroach gen haproxy --certs-dir=certs --host=<address of any node>
cockroach gen haproxy --certs-dir=certs --host=172.16.8.1
[root@crdbsrv-1 ~]# more haproxy.cfg
global
maxconn 4096
defaults
mode tcp
# Timeout values should be configured for your specific use.
# See: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-timeout%20connect
timeout connect 10s
timeout client 1m
timeout server 1m
# TCP keep-alive on client side. Server already enables them.
option clitcpka
listen psql
bind :26257
mode tcp
balance roundrobin
option httpchk GET /health?ready=1
server cockroach1 172.16.8.1:26257 check port 8080
server cockroach2 172.16.8.2:26257 check port 8080
server cockroach3 172.16.8.3:26257 check port 8080
[root@crdbsrv-1 ~]#
7-2-2. Copy To Proxy Server
$ scp haproxy.cfg <username>@<haproxy address>:~/
7-2-3. HA Proxy 설치
더보기
[root@haproxySrv ~]# yum install haproxy
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirror.kakao.com
* extras: mirror.kakao.com
* updates: mirror.kakao.com
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/4): base/7/x86_64/group_gz | 153 kB 00:00:00
(2/4): extras/7/x86_64/primary_db | 206 kB 00:00:00
(3/4): base/7/x86_64/primary_db | 6.1 MB 00:00:03
(4/4): updates/7/x86_64/primary_db | 4.5 MB 00:00:05
Resolving Dependencies
--> Running transaction check
---> Package haproxy.x86_64 0:1.5.18-9.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================
Package Arch Version Repository Size
===========================================================================================================
Installing:
haproxy x86_64 1.5.18-9.el7 base 834 k
Transaction Summary
===========================================================================================================
Install 1 Package
Total download size: 834 k
Installed size: 2.6 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/haproxy-1.5.18-9.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for haproxy-1.5.18-9.el7.x86_64.rpm is not installed
haproxy-1.5.18-9.el7.x86_64.rpm | 834 kB 00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-8.2003.0.el7.centos.x86_64 (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : haproxy-1.5.18-9.el7.x86_64 1/1
Verifying : haproxy-1.5.18-9.el7.x86_64 1/1
Installed:
haproxy.x86_64 0:1.5.18-9.el7
Complete!
[root@haproxySrv ~]#
7-2-4. HA Proxy 수행
haproxy -f haproxy.cfg
[root@haproxySrv ~]# haproxy -f haproxy.cfg
7-2-5. HA Proxy를 통한 접속 확인
[root@app ~]# cockroach sql --certs-dir=certs --host=haproxySrv
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.2.0 (x86_64-unknown-linux-gnu, built 2020/11/09 16:01:45, go1.13.14) (same version as client)
# Cluster ID: 1b558b2a-ebe3-490a-8e3e-1973777beeb1
#
# Enter \? for a brief introduction.
#
root@haproxySrv:26257/defaultdb> show database;
database
-------------
defaultdb
(1 row)
Time: 2ms total (execution 1ms / network 1ms)
root@haproxySrv:26257/defaultdb> \q'Database' 카테고리의 다른 글
| [Cockroach] 노드 추가/삭제 (0) | 2020.12.12 |
|---|---|
| [Cockroach] Architecture - Read/Write (0) | 2020.12.12 |
| [HANA] Scale-out sr_takeover 수행 단계 분석 (0) | 2020.08.12 |
| [HANA] Monitoring 시스템 - 기능 추가 (LogAnalyzer) - Sample (0) | 2020.08.11 |
| [HANA] Monitoring 시스템 - 기능 추가 (LogAnalyzer) (0) | 2020.08.10 |
