About

由于環境基于CentOS 6.7 x64，rsyslog本身就是OS的組件，由于本文使用rsyslog作為統一日志采集器，需要與kafka進行數據交付，而原始支持kafka組件是在v8.7.0以后版本才支持，而系統自帶的rsyslog是版本是v5的，因此需要對rsyslog版本進行升級。

有網的方法比較簡單，通過yum命令即可，官網參考官網 http://www.rsyslog.com/rhelcentos-rpms/ 即可安裝指定版本。

安裝說明

由于鑒于內網環境無Internet無法使用yum進行安裝，因此選擇離線rpm方式升級rsyslog（rpm包下載地址：http://rpms.adiscon.com/ ）。

此處有坑：按習慣，一般使用官網提供的最新的穩定版/v8-stable/epel-7，然而由于CentOS 6.7 glibc版本為2.12而epel-7依賴的libgt-0.3.11-1.el7.x86_64、libfastjson-0.99.2-1.el7.x86_64、libestr-0.1.10-1.el7.x86_64幾個組件需要 libc.so.6: version `GLIBC_2.14，CentOS 6.7升級后會提示bash版本有問題，因此此處選用/v8-stable/epel-6版本（ps：此坑是個人問題，centos6.x本來就應該使用epel-6，因此此坑可忽略）

需要下載的rpm

可在http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/ 獲取

json-c-0.11-4.el6.x86_64.rpm
libfastjson-0.99.2-1.el6.x86_64.rpm
libgt-0.3.11-1.el6.x86_64.rpm
libestr-0.1.10-1.el6.x86_64.rpm
liblogging-1.0.5-1.el6.x86_64.rpm
librdkafka1-0.8.5-0.x86_64.rpm
rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
rsyslog-8.19.0-1.el6.x86_64.rpm

rpm安裝

執行安裝前先給沒個文件加上運行屬性 chmod +x

rpm -ivh  json-c-0.11-4.el6.x86_64.rpm
rpm -ivh libfastjson-0.99.2-1.el6.x86_64.rpm
rpm -ivh libgt-0.3.11-1.el6.x86_64.rpm
rpm -ivh libestr-0.1.10-1.el6.x86_64.rpm
rpm -ivh liblogging-1.0.5-1.el6.x86_64.rpm
rpm -ivh librdkafka1-0.8.5-0.x86_64.rpm

#這邊使用 -U 參數是為了直接升級本地rsyslog
rpm -Uvh rsyslog-8.19.0-1.el6.x86_64.rpm

#rsyslog-kafka組件安裝依賴 rsyslog-8.19.0及librdkafka，該命令最后執行
rpm -ivh rsyslog-kafka-8.19.0-1.el6.x86_64.rpm

驗證安裝

安裝完后執行 rsyslogd -v, 驗證rsyslog是否升級成功。

# rsyslogd -v
rsyslogd 8.19.0, compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              No
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        Number of Bits in RainerScript integers: 64
 
See http://www.rsyslog.com for more information.

查看/lib64/rsyslog/（32位系統為 /lib/rsyslog）中是否存在 omkafka.so，驗證rsyslog-kafka是否安裝成功。

配置說明

rsyslog官網是個大坑，能路過盡量路過。

接下來說明下配置，首先聲明下本文使用rsyslog的目的是作為應用的log collector，應用使用log4x(log4net:log4net.Appender.RemoteSyslogAppender, log4j:org.apache.log4j.net.SyslogAppender)，通過UDP協議將日志傳輸到rsyslog。

rsyslo的默認的配置文件 /etc/rsyslog.config

# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced
# things, RainerScript configuration is suggested.

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously done by rklogd)
#module(load"immark")  # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  /var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
 
 
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

接下來說明配置修改：

配置 input

開啟接收UDP消息

module(load="imudp") # 加載imudp模塊
input(type="imudp" port="514") # 配置UDP監聽的端口號

配置 output

開啟數據轉發kafka

 module(load="omkafka") # 配置加載omkafka模塊
 # 轉發到kafka消息格式
template(name="rsyslogToKafka" type="list" option.json="on"){
        constant(value="{")
        #constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
        constant(value="\"message\":\"") property(name="msg" format="json")
        constant(value="\",\"host\":\"") property(name="hostname")
        constant(value="\",\"severity\":\"") property(name="syslogseverity")
        constant(value="\",\"severity-text\":\"") property(name="syslogseverity-text")
        constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
        constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
        constant(value="\",\"type\":\"application\"")
        constant(value=",\"recieve-from\":\"") property(name="fromhost-ip")
        constant(value="\",\"recieve-at\":\"") property(name="timegenerated" dateFormat="rfc3339")
        constant(value="\",\"report-at\":\"") property(name="timereported" dateFormat="rfc3339")
        constant(value="\"}")
}

# 配置kafka相關信息及模板
action( type="omkafka"
        topic="test"
        broker="11.4.74.26:9092"
        template="rsyslogToKafka"
)