系统版本:CentOS7
节点规划:
热数据节点: 192.168.2.4
温数据节点: 192.168.2.190
PS:这里就没分 hot warm cold 这种三级存储,我们一般使用 hot cold 2种即可。
我们这里使用 tar.gz 通用二进制文件
useradd es
# cat /etc/security/limits.d/elasticsearch.conf
es - nofile 65535 es - noproc 65535 es - memlock unlimited
cd /usr/local/elasticsearch-7.4.2/
mkdir data logs
chown es.es /usr/local/elasticsearch-7.4.2/ -R
su - es
cd /usr/local/elasticsearch-7.4.2/
热数据节点: 192.168.2.4 的配置如下:
cat config/elasticsearch.yml
cluster.name: my-application node.name: node-1 node.attr.rack: r1 node.attr.temperature: hot path.data: ./data/ path.logs: ./logs node.master: true node.data: true node.ingest: true bootstrap.memory_lock: true network.host: 0.0.0.0 http.port: 9200 cluster.initial_master_nodes: - 192.168.2.4:9300 - 192.168.2.190:9300 discovery.seed_hosts: - 192.168.2.4 - 192.168.2.190 gateway.recover_after_nodes: 1 #action.destructive_requires_name: true ############# xpack 的配置项 #################### #xpack.security.enabled: true #xpack.security.transport.ssl.enabled: true xpack.security.audit.enabled: true xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed,connection_denied, tampered_request, run_as_denied, run_as_granted xpack.security.audit.logfile.emit_node_host_address: true xpack.security.audit.logfile.emit_node_host_name: true xpack.sql.enabled: true xpack.ilm.enabled: true
温数据节点: 192.168.2.190
cat config/elasticsearch.yml
cluster.name: my-application node.name: es11 node.attr.rack: r1 node.attr.temperature: warm path.data: ./data/ path.logs: ./logs node.master: true node.data: true node.ingest: true bootstrap.memory_lock: true network.host: 0.0.0.0 http.port: 9200 cluster.initial_master_nodes: - 192.168.2.4:9300 - 192.168.2.190:9300 discovery.seed_hosts: - 192.168.2.4 - 192.168.2.190 gateway.recover_after_nodes: 1 #action.destructive_requires_name: true ############# xpack 的配置项 #################### #xpack.security.enabled: true #xpack.security.transport.ssl.enabled: true xpack.security.audit.enabled: true xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed,connection_denied, tampered_request, run_as_denied, run_as_granted xpack.security.audit.logfile.emit_node_host_address: true xpack.security.audit.logfile.emit_node_host_name: true xpack.sql.enabled: true xpack.ilm.enabled: true
启动脚本:
# cat start.sh
# es 依赖高版本的jdk 需要先export下 export JAVA_HOME=./jdk/ nohup ./bin/elasticsearch >/dev/null 2>&1 &
停止脚本:
# cat stop.sh
kill $(jps| grep Elasticsearch | awk '{print $1}')
创建索引,并将数据搬迁到hot节点:
curl -H 'Content-Type: application/json' -X PUT http://localhost:9200/index-2019.10.19?pretty curl -H 'Content-Type: application/json' -X PUT http://localhost:9200/index-2019.10.19/_settings -d ' { "index.routing.allocation.require.temperature": "hot" }'
如果要将 index-2019.10.19 的数据搬迁到温节点,我们使用下面的这个命令就行
curl -H 'Content-Type: application/json' -X PUT http://localhost:9200/index-2019.10.19/_settings -d ' { "index.routing.allocation.require.temperature": "hot" }'
流程跑通后,我们可以写个脚本,将7天前的索引,打标签,存放到es的warm节点(大容量HDD磁盘):
#!/bin/bash day=$(date +"%Y.%m.%d" -d -7day) # echo ${day} curl -H 'Content-Type: application/json' -X PUT http://192.168.2.4:9200/*-${day}/_settings -d ' { "index.routing.allocation.require.temperature": "warm" }'
另外, 在es7里面 提供 index-lifecycle-management 这个功能, 我们在kibana 界面里面就可以进行配置。 具体可以查阅es官方的文档(个人还是比较喜欢用上面的这种脚本的方案)。