Note: If you are also using Kibana as your front end, you will need to add a MimeType of “application/json” for the extension .json to IIS.
We are pushing all of our logs into Elasticsearch using Logstash. IIS was the most painful part of the process so I am writing up a few gotchas for Logstash 1.3.3 and IIS in general.
The process is relatively straight forward on paper:
- Logstash monitors the IIS log and pushes new entries into the pipeline
- Use a grok filter to split out the fields in the IIS log line (more on this below)
- Push the result into Elasticsearch
Firstly there is a bug in the Logstash file input on windows (doesn’t handle files named the same in different directories) which results in partial entries being read. To remedy this you need to get IIS to generate a single log file per server (default is per website). Once that is done we can read the IIS logs with this config
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| file { | |
| type => "iis" | |
| path => "C:/inetpub/logs/LogFiles/W3SVC/*.log" | |
| } | |
| } |
Once we have IIS log lines pumping through the veins of Logstash, we need to break down the line into its component fields. To do this we use the Logstash Grok filter. In IIS the default logging is W3C but you are able to select the fields you want outputed. The following config works for the default fields and [bytes sent] so we can see bandwidth usuage. The Heroku Grok Debugger is a lifesaver for debugging the Grok string (paste an entry from your log into it and then paste you GROK pattern in)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| filter{ | |
| grok { | |
| match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"] | |
| } | |
| } |
Below is the complete IIS configuration for logstash. There are a few other filters we use to enrich the event sent to logstash as well as a conditional to remove IIS log comments.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| file { | |
| type => "iis" | |
| path => "C:/inetpub/logs/LogFiles/W3SVC/*.log" | |
| } | |
| } | |
| filter { | |
| #ignore log comments | |
| if [message] =~ "^#" { | |
| drop {} | |
| } | |
| grok { | |
| match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"] | |
| } | |
| #Set the Event Timesteamp from the log | |
| date { | |
| match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ] | |
| timezone => "Etc/UCT" | |
| } | |
| ruby{ code => "event['kilobytes'] = event['bytes'] / 1024.0" } | |
| #https://logstash.jira.com/browse/LOGSTASH-1354 | |
| #geoip{ | |
| # source => "clienthost" | |
| # add_tag => [ "geoip" ] | |
| #} | |
| useragent { | |
| source=> "useragent" | |
| prefix=> "browser" | |
| } | |
| mutate { | |
| remove_field => [ "log_timestamp"] | |
| } | |
| } | |
| output { | |
| elasticsearch { | |
| host => "127.0.0.1" | |
| } | |
| } | |
This Hours Eternity 