Elastic Stack之Redis集群使用

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰 

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

　　本篇博客数据流走向：FileBeat ===》Redis  ===》logstash ===> elasticsearch。 

 

一.安装Redis服务器

1>.安装repo源

[root@node105 ~]# [root@node105 ~]# yum -y install epel-releaseLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.cn99.com * extras: mirror.lzu.edu.cn * updates: mirror.lzu.edu.cnResolving Dependencies--> Running transaction check---> Package epel-release.noarch 0:7-11 will be installed--> Finished Dependency ResolutionDependencies Resolved=================================================================================================================================================================================================== Package                                             Arch                                          Version                                     Repository                                     Size===================================================================================================================================================================================================Installing: epel-release                                        noarch                                        7-11                                        extras                                         15 kTransaction Summary===================================================================================================================================================================================================Install  1 PackageTotal download size: 15 kInstalled size: 24 kDownloading packages:epel-release-7-11.noarch.rpm                                                                                                                                                |  15 kB  00:00:00     Running transaction checkRunning transaction testTransaction test succeededRunning transactionWarning: RPMDB altered outside of yum.  Installing : epel-release-7-11.noarch                                                                                                                                                        1/1   Verifying  : epel-release-7-11.noarch                                                                                                                                                        1/1 Installed:  epel-release.noarch 0:7-11                                                                                                                                                                       Complete![root@node105 ~]#

2>.安装redis服务器

[root@node105 ~]# [root@node105 ~]# yum -y install redisLoaded plugins: fastestmirrorepel/x86_64/metalink                                                                                                                                                        | 7.9 kB  00:00:00     epel                                                                                                                                                                        | 4.7 kB  00:00:00     (1/3): epel/x86_64/group_gz                                                                                                                                                 |  88 kB  00:00:00     (2/3): epel/x86_64/updateinfo                                                                                                                                               | 959 kB  00:00:01     (3/3): epel/x86_64/primary_db                                                                                                                                               | 6.6 MB  00:00:01     Loading mirror speeds from cached hostfile * base: mirrors.cn99.com * epel: mirrors.yun-idc.com * extras: mirror.lzu.edu.cn * updates: mirror.lzu.edu.cnResolving Dependencies--> Running transaction check---> Package redis.x86_64 0:3.2.12-2.el7 will be installed--> Processing Dependency: libjemalloc.so.1()(64bit) for package: redis-3.2.12-2.el7.x86_64--> Running transaction check---> Package jemalloc.x86_64 0:3.6.0-1.el7 will be installed--> Finished Dependency ResolutionDependencies Resolved=================================================================================================================================================================================================== Package                                        Arch                                         Version                                              Repository                                  Size===================================================================================================================================================================================================Installing: redis                                          x86_64                                       3.2.12-2.el7                                         epel                                       544 kInstalling for dependencies: jemalloc                                       x86_64                                       3.6.0-1.el7                                          epel                                       105 kTransaction Summary===================================================================================================================================================================================================Install  1 Package (+1 Dependent package)Total download size: 648 kInstalled size: 1.7 MDownloading packages:warning: /var/cache/yum/x86_64/7/epel/packages/jemalloc-3.6.0-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY                           ]  0.0 B/s |    0 B  --:--:-- ETA Public key for jemalloc-3.6.0-1.el7.x86_64.rpm is not installed(1/2): jemalloc-3.6.0-1.el7.x86_64.rpm                                                                                                                                      | 105 kB  00:00:00     (2/2): redis-3.2.12-2.el7.x86_64.rpm                                                                                                                                        | 544 kB  00:00:00     ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total                                                                                                                                                              564 kB/s | 648 kB  00:00:01     Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7Importing GPG key 0x352C64E5: Userid     : "Fedora EPEL (7) 
      
       " Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5 Package    : epel-release-7-11.noarch (@extras) From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7Running transaction checkRunning transaction testTransaction test succeededRunning transaction  Installing : jemalloc-3.6.0-1.el7.x86_64                                                                                                                                                     1/2   Installing : redis-3.2.12-2.el7.x86_64                                                                                                                                                       2/2   Verifying  : redis-3.2.12-2.el7.x86_64                                                                                                                                                       1/2   Verifying  : jemalloc-3.6.0-1.el7.x86_64                                                                                                                                                     2/2 Installed:  redis.x86_64 0:3.2.12-2.el7                                                                                                                                                                      Dependency Installed:  jemalloc.x86_64 0:3.6.0-1.el7                                                                                                                                                                    Complete![root@node105 ~]#
      

3>.修改Redis服务器的默认配置

[root@node105 ~]# [root@node105 ~]# grep bind /etc/redis.conf   | grep -v ^#　　　　　　　　　　　　　　　　　　　　　　#指定绑定的ip地址为服务器的所有网卡！bind 0.0.0.0 [root@node105 ~]# [root@node105 ~]# grep requirepass /etc/redis.conf   | grep -v ^#　　　　　　　　　　　　　　　　　　#开启认证requirepass yinzhengjie.org.cn[root@node105 ~]# [root@node105 ~]#

4>.启动redis服务器

[root@node105 ~]# [root@node105 ~]# systemctl enable redisCreated symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /usr/lib/systemd/system/redis.service.[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# systemctl start redis[root@node105 ~]# [root@node105 ~]# systemctl status redis● redis.service - Redis persistent key-value database   Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)  Drop-In: /etc/systemd/system/redis.service.d           └─limit.conf   Active: active (running) since Mon 2019-03-11 23:17:20 CST; 4s ago Main PID: 4102 (redis-server)   CGroup: /system.slice/redis.service           └─4102 /usr/bin/redis-server 0.0.0.0:6379Mar 11 23:17:20 node105.yinzhengjie.org.cn systemd[1]: Starting Redis persistent key-value database...Mar 11 23:17:20 node105.yinzhengjie.org.cn systemd[1]: Started Redis persistent key-value database.[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# ss -tnl | grep 6379LISTEN     0      128          *:6379                     *:*                  [root@node105 ~]# [root@node105 ~]#

 

二.使用FileBeat结合redis收集日志文件案例

1>.安装FileBeat软件

　　详情请参考：　。

2>.修改FileBeat的配置文件并启动

[root@node105 ~]# [root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$|  #"filebeat.prospectors:- input_type: log  paths:    - /var/log/httpd/access_log  output.redis:  enabled: true  hosts: ["node105.yinzhengjie.org.cn:6379"]  key: filebeat  password: yinzhengjie.org.cn  db: 0  datatype: list[root@node105 ~]# [root@node105 ~]#

3>.查看redis服务器

[root@node105 ~]# [root@node105 ~]# redis-cli -a yinzhengjie.org.cn127.0.0.1:6379> 127.0.0.1:6379> KEYS *1) "filebeat"127.0.0.1:6379> 127.0.0.1:6379> LINDEX filebeat 0"{\"@timestamp\":\"2019-03-11T15:34:37.114Z\",\"beat\":{\"hostname\":\"node105.yinzhengjie.org.cn\",\"name\":\"node105.yinzhengjie.org.cn\",\"version\":\"5.6.12\"},\"input_type\":\"log\",\"message\":\"80.129.1.1 - - [11/Mar/2019:23:05:48 +0800] \\\"GET /test42.html HTTP/1.1\\\" 200 8 \\\"-\\\" \\\"curl/7.29.0\\\"\",\"offset\":134141,\"source\":\"/var/log/httpd/access_log\",\"type\":\"log\"}"127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat(integer) 1540127.0.0.1:6379> [root@node105 ~]# redis-cli -a yinzhengjie.org.cn

4>.配置logstash的redis输入插件（参考链接：）

[root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf input {    redis {        host => "node105.yinzhengjie.org.cn"        port => 6379        password => "yinzhengjie.org.cn"        db => 0        key => "filebeat"        data_type => "list"    }}filter {    grok {        match => { "message" => "%{HTTPD_COMBINEDLOG}" }        remove_field => ["message","beat"]     }    date {        match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]        remove_field => "timestamp"    }    geoip {        source => "clientip"        target => "geoip"        database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"    }}output {    elasticsearch {        hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]        index => "logstash-%{+YYYY.MM.dd}"        document_type => "httpd_access_logs"    }}[root@node105 ~]#

[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -tWARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaultsCould not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the consoleConfiguration OK[root@node105 ~]#

5>.启动logstash并查看redis服务器

[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;donePage 34Page 27Page 7Page 41
Not Found
The requested URL /test58.html was not found on this server.
Page 48Page 26Page 40
Not Found
The requested URL /test54.html was not found on this server.
Not Found
The requested URL /test53.html was not found on this server.
Page 8Page 20
Not Found
The requested URL /test59.html was not found on this server.
Page 43Page 15Page 46Page 24Page 19
Not Found
The requested URL /test55.html was not found on this server.
Page 32Page 30Page 20Page 33Page 8Page 26
Not Found
The requested URL /test54.html was not found on this server.
Page 25Page 37Page 20Page 23Page 19Page 37Page 47Page 34Page 10Page 9Page 10Page 22Page 23Page 37
Not Found
The requested URL /test53.html was not found on this server.
Page 49Page 19Page 21Page 6
Not Found
The requested URL /test53.html was not found on this server.
Page 15Page 4Page 27Page 19Page 29Page 8
Not Found
The requested URL /test51.html was not found on this server.
Page 33
Not Found
The requested URL /test58.html was not found on this server.
Page 49Page 23Page 27Page 30Page 9Page 1Page 44Page 31Page 12Page 46Page 26Page 18Page 6Page 17Page 27Page 5Page 49Page 14
Not Found
The requested URL /test58.html was not found on this server.
Page 36Page 27Page 32Page 16Page 25Page 15Page 4Page 48Page 25Page 13Page 32Page 6Page 1Page 35Page 25Page 6
Not Found
The requested URL /test53.html was not found on this server.
Page 34Page 49Page 31Page 15
Not Found
The requested URL /test55.html was not found on this server.
Page 12
Not Found
The requested URL /test60.html was not found on this server.
Page 15
Not Found
The requested URL /test51.html was not found on this server.
Page 34
Not Found
The requested URL /test55.html was not found on this server.
Page 26Page 8
Not Found
The requested URL /test56.html was not found on this server.
Not Found
The requested URL /test53.html was not found on this server.
Page 36Page 41Page 49Page 20
Not Found
The requested URL /test51.html was not found on this server.
Page 27Page 5Page 49Page 23Page 45
Not Found
The requested URL /test60.html was not found on this server.
Page 6Page 26
Not Found
The requested URL /test57.html was not found on this server.
Page 5Page 43Page 46Page 16Page 39Page 47Page 22Page 25Page 15
Not Found
The requested URL /test55.html was not found on this server.
Page 21Page 21Page 11Page 26
Not Found
The requested URL /test59.html was not found on this server.
Page 27Page 4Page 49Page 5Page 5Page 7Page 12Page 20Page 12
Not Found
The requested URL /test53.html was not found on this server.
Not Found
The requested URL /test57.html was not found on this server.
Page 8Page 18Page 30Page 27Page 46Page 34Page 17Page 5Page 12Page 26Page 37Page 13
Not Found
The requested URL /test55.html was not found on this server.
Not Found
The requested URL /test54.html was not found on this server.
Not Found
The requested URL /test57.html was not found on this server.
Not Found
The requested URL /test55.html was not found on this server.
Not Found
The requested URL /test52.html was not found on this server.
Page 44Page 23Page 11Page 27Page 16Page 19Page 18Page 6Page 34
Not Found
The requested URL /test59.html was not found on this server.
Page 19Page 25Page 45Page 36Page 23Page 1Page 41Page 47Page 24Page 45
Not Found
The requested URL /test55.html was not found on this server.
Page 43Page 48Page 28Page 13Page 30
Not Found
The requested URL /test57.html was not found on this server.
Not Found
The requested URL /test52.html was not found on this server.
Page 19
Not Found
The requested URL /test58.html was not found on this server.
Page 45
Not Found
The requested URL /test56.html was not found on this server.
Page 43Page 13Page 6Page 33Page 17Page 1Page 17Page 8Page 42Page 12
Not Found
The requested URL /test55.html was not found on this server.
Page 22Page 15Page 43Page 48Page 46
Not Found
The requested URL /test52.html was not found on this server.
Page 9Page 29Page 18
Not Found
The requested URL /test52.html was not found on this server.
Page 34Page 37Page 13Page 13Page 18
Not Found
The requested URL /test59.html was not found on this server.
Page 15Page 12Page 33Page 39Page 5Page 40Page 16Page 10Page 18Page 14Page 31Page 42Page 4Page 18Page 11
Not Found
The requested URL /test53.html was not found on this server.
Page 14Page 22
Not Found
The requested URL /test55.html was not found on this server.
Page 38Page 46Page 23Page 46
Not Found
The requested URL /test51.html was not found on this server.
Page 44Page 36Page 22
Not Found
The requested URL /test58.html was not found on this server.
Not Found
The requested URL /test54.html was not found on this server.
Page 32Page 15Page 50Page 23Page 4Page 11Page 46Page 10Page 45Page 15Page 31Page 5Page 42
Not Found
The requested URL /test52.html was not found on this server.
Page 40
Not Found
The requested URL /test56.html was not found on this server.
Not Found
The requested URL /test60.html was not found on this server.
Page 40Page 17Page 33Page 11Page 16Page 44Page 15Page 1Page 7Page 13Page 42Page 18Page 30Page 22Page 36Page 37Page 30Page 13Page 12Page 6Page 1Page 39Page 45Page 9Page 16Page 47Page 20Page 12Page 4Page 24
Not Found
The requested URL /test52.html was not found on this server.
Page 18Page 4Page 27
Not Found
The requested URL /test54.html was not found on this server.
Page 6Page 47Page 1Page 5Page 4Page 20Page 14
Not Found
The requested URL /test57.html was not found on this server.
Not Found
The requested URL /test51.html was not found on this server.
Not Found
The requested URL /test56.html was not found on this server.
Page 6Page 26Page 42Page 34Page 38Page 4Page 7Page 15Page 19Page 38Page 15Page 8Page 21Page 47Page 17Page 28Page 47^C[root@node103 ~]#

[root@node105 ~]# redis-cli -a yinzhengjie.org.cn127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat(integer) 5438127.0.0.1:6379>

[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaultsCould not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[root@node105 ~]# redis-cli -a yinzhengjie.org.cn127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat(integer) 0127.0.0.1:6379>

[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100  9035  100  9035    0     0   428k      0 --:--:-- --:--:-- --:--:--  441k{  "took": 11,  "timed_out": false,  "_shards": {    "total": 15,    "successful": 15,    "skipped": 0,    "failed": 0  },  "hits": {    "total": 1445,    "max_score": 1.8790942,    "hits": [      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAEi",        "_score": 1.8790942,        "_source": {          "request": "/test55.html",          "geoip": {            "ip": "100.242.1.1",            "latitude": 37.751,            "country_name": "United States",            "country_code2": "US",            "continent_code": "NA",            "country_code3": "US",            "location": {              "lon": -97.822,              "lat": 37.751            },            "longitude": -97.822          },          "offset": 25922,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:02:33.000Z",          "response": "404",          "bytes": "209",          "clientip": "100.242.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAEq",        "_score": 1.8790942,        "_source": {          "request": "/test52.html",          "geoip": {            "timezone": "Europe/Paris",            "ip": "82.234.1.1",            "latitude": 48.6942,            "continent_code": "EU",            "city_name": "Brunoy",            "country_name": "France",            "country_code2": "FR",            "country_code3": "FR",            "region_name": "Essonne",            "location": {              "lon": 2.4922,              "lat": 48.6942            },            "postal_code": "91800",            "region_code": "91",            "longitude": 2.4922          },          "offset": 26692,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:02:41.000Z",          "response": "404",          "bytes": "209",          "clientip": "82.234.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAEt",        "_score": 1.8790942,        "_source": {          "request": "/test52.html",          "geoip": {            "timezone": "Africa/Accra",            "ip": "45.209.1.1",            "latitude": 8,            "country_name": "Ghana",            "country_code2": "GH",            "continent_code": "AF",            "country_code3": "GH",            "location": {              "lon": -2,              "lat": 8            },            "longitude": -2          },          "offset": 26979,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:02:44.000Z",          "response": "404",          "bytes": "209",          "clientip": "45.209.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAE9",        "_score": 1.8790942,        "_source": {          "request": "/test58.html",          "geoip": {            "timezone": "Asia/Tokyo",            "ip": "111.89.1.1",            "latitude": 34.9833,            "continent_code": "AS",            "city_name": "Shobara",            "country_name": "Japan",            "country_code2": "JP",            "country_code3": "JP",            "region_name": "Hiroshima",            "location": {              "lon": 132.9833,              "lat": 34.9833            },            "postal_code": "727-0321",            "region_code": "34",            "longitude": 132.9833          },          "offset": 28514,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:01.000Z",          "response": "404",          "bytes": "209",          "clientip": "111.89.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAFH",        "_score": 1.8790942,        "_source": {          "request": "/test51.html",          "geoip": {            "timezone": "Europe/Madrid",            "ip": "83.47.1.1",            "latitude": 36.54,            "continent_code": "EU",            "city_name": "Fuengirola",            "country_name": "Spain",            "country_code2": "ES",            "country_code3": "ES",            "region_name": "Malaga",            "location": {              "lon": -4.6247,              "lat": 36.54            },            "postal_code": "29640",            "region_code": "MA",            "longitude": -4.6247          },          "offset": 29476,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:11.000Z",          "response": "404",          "bytes": "209",          "clientip": "83.47.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAFY",        "_score": 1.8790942,        "_source": {          "request": "/test53.html",          "geoip": {            "timezone": "Europe/Rome",            "ip": "94.164.1.1",            "latitude": 42.5245,            "continent_code": "EU",            "city_name": "Piansano",            "country_name": "Italy",            "country_code2": "IT",            "country_code3": "IT",            "region_name": "Provincia di Viterbo",            "location": {              "lon": 11.8298,              "lat": 42.5245            },            "postal_code": "01010",            "region_code": "VT",            "longitude": 11.8298          },          "offset": 31107,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:28.000Z",          "response": "404",          "bytes": "209",          "clientip": "94.164.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAFd",        "_score": 1.8790942,        "_source": {          "request": "/test55.html",          "geoip": {            "timezone": "Africa/Tunis",            "ip": "154.106.1.1",            "latitude": 34,            "country_name": "Tunisia",            "country_code2": "TN",            "continent_code": "AF",            "country_code3": "TN",            "location": {              "lon": 9,              "lat": 34            },            "longitude": 9          },          "offset": 31590,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:33.000Z",          "response": "404",          "bytes": "209",          "clientip": "154.106.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN80lXxXllWpXYAFq",        "_score": 1.8790942,        "_source": {          "request": "/test56.html",          "geoip": {            "ip": "33.65.1.1",            "latitude": 37.751,            "country_name": "United States",            "country_code2": "US",            "continent_code": "NA",            "country_code3": "US",            "location": {              "lon": -97.822,              "lat": 37.751            },            "longitude": -97.822          },          "offset": 32832,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:46.000Z",          "response": "404",          "bytes": "209",          "clientip": "33.65.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN85S3WCT5NaOiwHV",        "_score": 1.8790942,        "_source": {          "request": "/test57.html",          "geoip": {            "timezone": "America/Bogota",            "ip": "179.19.1.1",            "latitude": 4.5981,            "country_name": "Colombia",            "country_code2": "CO",            "continent_code": "SA",            "country_code3": "CO",            "location": {              "lon": -74.0758,              "lat": 4.5981            },            "longitude": -74.0758          },          "offset": 33600,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:03:54.000Z",          "response": "404",          "bytes": "209",          "clientip": "179.19.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      },      {        "_index": "logstash-2019.03.11",        "_type": "httpd_access_logs",        "_id": "AWltN85T3WCT5NaOiwHc",        "_score": 1.8790942,        "_source": {          "request": "/test51.html",          "geoip": {            "city_name": "Central",            "timezone": "Asia/Hong_Kong",            "ip": "13.94.1.1",            "latitude": 22.2909,            "country_name": "Hong Kong",            "country_code2": "HK",            "continent_code": "AS",            "country_code3": "HK",            "region_name": "Central and Western District",            "location": {              "lon": 114.15,              "lat": 22.2909            },            "region_code": "HCW",            "longitude": 114.15          },          "offset": 34272,          "auth": "-",          "ident": "-",          "input_type": "log",          "verb": "GET",          "source": "/var/log/httpd/access_log",          "type": "log",          "tags": [            "beats_input_codec_plain_applied"          ],          "referrer": "\"-\"",          "@timestamp": "2019-03-11T14:04:01.000Z",          "response": "404",          "bytes": "209",          "clientip": "13.94.1.1",          "@version": "1",          "beat": {            "name": "node105.yinzhengjie.org.cn",            "hostname": "node105.yinzhengjie.org.cn",            "version": "5.6.12"          },          "host": "node105.yinzhengjie.org.cn",          "httpversion": "1.1",          "user_agent": "\"curl/7.29.0\""        }      }    ]  }}[root@node101 ~]#

 

三.使用一个FileBeat加载多种不同的日志案例

1>.编写FileBeat的配置文件

[root@node105 ~]# [root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"filebeat.prospectors:- input_type: log  paths:    - /var/log/httpd/access_log  fields:    log_type: access- paths:    - /var/log/httpd/error_log  fields:    log_type: errorsoutput.redis:  enabled: true  hosts: ["node105.yinzhengjie.org.cn:6379"]  key: filebeat  password: yinzhengjie.org.cn  db: 0  datatype: listlogging.level: debug[root@node105 ~]# [root@node105 ~]#

2>.编写logstash的配置文件

[root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf input {    redis {        host => "node105.yinzhengjie.org.cn"        port => 6379        password => "yinzhengjie.org.cn"        db => 0        key => "filebeat"        data_type => "list"    }}filter {    if [fields][log_type] == "access" {    grok {        match => { "message" => "%{HTTPD_COMBINEDLOG}" }        remove_field => ["message","beat"]     }    date {        match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]        remove_field => "timestamp"    }    geoip {        source => "clientip"        target => "geoip"        database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"    }     }}output {    if [fields][log_type] == "access" {    elasticsearch {        hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]        index => "logstash-%{+YYYY.MM.dd}"        document_type => "httpd_access_logs"    }    } else {    elasticsearch {                hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]                index => "logstash-%{+YYYY.MM.dd}"                document_type => "httpd_error_logs"        }    }}[root@node105 ~]# [root@node105 ~]#

[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -tWARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaultsCould not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the consoleConfiguration OK[root@node105 ~]#

3>.启动logstash服务

[root@node105 ~]# [root@node105 ~]# mkdir -pv /yinzhengjie/logstash/conf.dmkdir: created directory ‘/yinzhengjie/logstash’mkdir: created directory ‘/yinzhengjie/logstash/conf.d’[root@node105 ~]# [root@node105 ~]# mv /etc/logstash/conf.d/* /yinzhengjie/logstash/conf.d/[root@node105 ~]# [root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# ll /etc/logstash/conf.d/total 4-rw-r--r--. 1 root root 1112 Mar 12 21:45 redis-condition-elasticsearch.conf[root@node105 ~]#

[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# systemctl start logstash[root@node105 ~]# [root@node105 ~]# systemctl status logstash● logstash.service - logstash   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)   Active: active (running) since Tue 2019-03-12 21:46:06 CST; 5s ago Main PID: 3200 (java)   CGroup: /system.slice/logstash.service           └─3200 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8...Mar 12 21:46:06 node105.yinzhengjie.org.cn systemd[1]: Started logstash.Mar 12 21:46:06 node105.yinzhengjie.org.cn systemd[1]: Starting logstash...[root@node105 ~]# [root@node105 ~]#

[root@node105 ~]# [root@node105 ~]# ps -ef | grep logstash | grep -v greplogstash   3200      1 15 21:46 ?        00:00:51 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# netstat -untalp | grep 3200tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      3200/java           tcp6       0      0 172.30.1.105:48739      172.30.1.101:9200       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:46093      172.30.1.102:9200       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:50903      172.30.1.103:9200       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:42682      172.30.1.105:6379       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:50900      172.30.1.103:9200       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:46090      172.30.1.102:9200       ESTABLISHED 3200/java           tcp6       0      0 172.30.1.105:48736      172.30.1.101:9200       ESTABLISHED 3200/java           [root@node105 ~]# [root@node105 ~]#