X-Forwarded-For in IIS and ELK

Now that I got my public site logs in to my ELK stack I wanted to get the remote IP tied to my IIS log. I use NGINX to act as a proxy so all of the remote ip addresses were my NGINX server and not the client. I was already passing the remote IP from NGINX back to IIS in an x-forwarded-for header and now I needed to get that into ELK.

So, I fiddled with it and found that IIS in Server 2016 supports adding in arbitrary fields into the log files. It supports a few different data sources, namely, Request Header. It also supports Server Variable and Response Header.

I want to put the value of the X-Forwarded-For header onto the IIS log entry, so I added that custom field to the logging fields in IIS. I used X-Forwarded-For for the Field Name and Source and left the Source Type as Request Header. The Field Name box is a drop down, but you can free form type text into it. The drop down is a list of well-known headers.

To add the header to the logs,

1. Open IIS Manager
2. Expand your server
3. Expand Sites
4. Click your web site
5. Double click Logging
6. Click Select Fields
7. Click Add Field... at the bottom of the popup
8. Put X-Forwarded-For into the Field Name and Source text boxes.
9. Leave Source Type as Request Header
10. Click OK out of the Add Custom Field box
11. Click OK out of the W3C Logging Fields box
12. Click Apply on the right.

Now that we have the new field in the logs we will need to modify the grok pattern that ElasticSearch uses to parse the logs. To see how to do that, please go through my previous post: File Beat and IIS with multiple sites

For this exercise, since I am setting up all of my IIS sites the same way on all of my servers, I will also remove all of the extra grok patterns for performance. My server is relatively low power and I need to squeeze every extra cycle I can out of the poor CPU's.

The new and only grok pattern that I am using is:

%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.proxy_ip} HTTP/%{NUMBER:iis.access.http_version} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms} %{IPORHOST:iis.access.remote_ip}

If you compare it to the one I created in the previous post, I added a new field, iis.access.proxy_ip and moved the iis.access.remote_ip to the end of the list. This effectively created a new field where the remote_ip was, calling it the proxy_ip, since it contains the proxy address it made sense. At work, with multiple proxies I could definitely see this being a benefit since we have multiple load balancers in multiple data-centers. Since I kept the same name for the iis.access.remote_ip attribute all of the dashboards continued to work the same way (except they actually worked now). To get rid of the exclamation point on the new field in the Kibana dashboard I did need to go to the management of the Kibana index and refresh the field list.