elk 情况搭建以及php测试样例

1.java情况安装

elk需要java情况。这里简单贴出 centos下 java情况设置

#jdk1.8下载#下载地点 https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.htmlwget https://download.oracle.com/otn/java/jdk/8u301-b09/d3c52aa6bfa54d3ca74e617f18309292/jdk-8u301-linux-aarch64.tar.gztar -zxvf jdk-8u301-linux-x64.tar.gzmv jdk-8u301-linux-x64 javamv java /usr/local/#情况变量设置vim /etc/profile#javaexport JAVA_HOME=/usr/local/javaexport PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JRE_HOME=$JAVA_HOME/jre#source /etc/profilejava -version #测试java是否成功2.elasticsearch 安装

2.1 用户和用户组创建

elasticsearch 默认是非root用户启动我这里添加 elk 用户和用户组 并创建elk的目次

groupadd elkuseradd -m -g elk elk# elk 安装目次在 /user/local/elk 目次下mkdir elk2.2 elasticsearch安装以es根本设置和跨域设置等

es 安装设置请注意磁盘容量 假如剩余量很小会报错，假如启动报错，请检察日志

#下载elasticsearchwget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.0-linux-x86_64.tar.gztar -zxvf elasticsearch-7.13.0-linux-x86_64.tar.gz#修改es设置文件cd /usr/local/elk/elasticsearch/configvim elasticsearch.ymlhttp.port: 9200node.name: node-1cluster.initial_master_nodes: ["node-1"]#network.host: 0.0.0.0  #假如允许外网访问这里需要改为0.0.0.0#添加跨域设置http.cors.enabled: truehttp.cors.allow-origin: "*"cluster.routing.allocation.disk.threshold_enabled: truecluster.routing.allocation.disk.watermark.flood_stage: 5gbcluster.routing.allocation.disk.watermark.low: 30gbcluster.routing.allocation.disk.watermark.high: 20gb2.3 安装i k分词器

ik分词器要和 es版本同等

cd /usr/local/elk/elasticsearch/bin./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.13.0/elasticsearch-analysis-ik-7.13.0.zip2.4 es启动

#切换用户启动essu elkbin/elasticsearch -d#测试es是否启动curl 127.0.0.1:9200#{  "name" : "node-1",  "cluster_name" : "elasticsearch",  "cluster_uuid" : "RzObV5d0Ro-aj5uxOMZE6g",  "version" : {    "number" : "7.13.0",    "build_flavor" : "default",    "build_type" : "tar",    "build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",    "build_date" : "2021-05-19T22:22:26.081971330Z",    "build_snapshot" : false,    "lucene_version" : "8.8.2",    "minimum_wire_compatibility_version" : "6.8.0",    "minimum_index_compatibility_version" : "6.0.0-beta1"  },  "tagline" : "You Know, for Search"}2.5安装 elasticsearch-head

2.1 依赖node情况

这里不做赘述 请自行百度安装

2.2 安装elasticsearch-head

git clone https://github.com/mobz/elasticsearch-head.gitcd elasticsearch-headnpm installnpm audit fixnpm audit fix --forcenpm run start访问地点：http://localhost:9100/3.kibana 安装

3.1 安装

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.13.0-darwin-x86_64.tar.gztar -zxvf kibana-7.13.0-darwin-x86_64.tar.gzmv kibana-7.13.0-darwin-x86_64 /usr/local/elk/kibana#汉化vim /usr/local/elk/kibana/config/kibana.ymli18n.locale: "zh-CN"#启动kibanacd /usr/local/elk/kibananohup bin/kibana &3.2 设置外网可访问

阿里云等服务器，请确保端口开放

vim /usr/local/elk/kibana/config/kibana.ymlserver.port: 5601server.host: "0.0.0.0"elasticsearch.hosts: ["http://127.0.0.1:9200"]kibana.index: ".kibana"4.安装logstash

mysql-connector-java-8.0.26.jar
这里需要主要。需要倒入这个包

#下载地点https://artifacts.elastic.co/downloads/logstash/logstash-7.13.0-darwin-x86_64.tar.gz#解压后进入目次#下令行进入bin/logstash -e "input { stdin {} } output { stdout {} }"#mysql-connector-java-8.0.26.jarcd /usr/local/elk/logstash/jars4.2设置文件处理demo.conf

input {  stdin {  }  jdbc {    #数据库信息设置    jdbc_connection_string => "jdbc:mysql://localhost:3306/test?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf-8"    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"    jdbc_driver_library => "/usr/local/elk/logstash/jars/mysql-connector-java-8.0.26.jar"    jdbc_user => "root"    jdbc_password => "root"    jdbc_paging_enabled => "true"    jdbc_page_size => 10000    jdbc_default_timezone => "Asia/Shanghai"    #执行sql文件位置    statement_filepath => "/usr/local/elk/logstash/config/demo.sql"    schedule => "* * * * *"    lowercase_column_names => "false"  }}output {  elasticsearch {    hosts => ["http://localhost:9200"]    index => "index_name"    document_id => "%{infoid}"    #user => "elastic"    #password => "changeme"  }  stdout {    codec => json_lines  }}4.3sql处理

这里的sql请根据本身的业务自行编写 使用logstash同步数据到es中

4.4启动logstash

bin/logstash -f config/user.conf5 注意点：

1. logstash 假如多次启动报错请删除 data目次下的 .lock文件
2. es报错或者卡顿。请注意磁盘大小 和jvm参数设置
3. kibana访问请确定外网端口是否开放
4. logstash 大批量倒入数据请确定添加了 mysql-connector-java-8.0.26.jar

解决es只能查询10000条数据的问题
PUT index_name/_settings?preserve_existing=true{  "max_result_window": "1000000"}GET index_name/_search{ "query": {         "match_all": {}                 },         "track_total_hits":true}下一期整理es在php中如何使用
如有错误请及时指出。

转发了

转发了