Elastic Search, Beats and a reverse proxy
The problem is because the HOST header isn't being sent (or it's incorrect) so the proxy doesn't know what to do with the request.
I finished moving my Elastic Stack to Docker, which is now fronted by a reverse proxy, HAProxy. As soon as I did that, my beats stopped working and started to receive a 503 service unavailable.
The problem is because the Host header isn't being sent (or it's incorrect) so the proxy doesn't know what to do with the request. This is typical of any reverse proxy. I've worked with quite a few different types, and they all seem to return a 503 when they don't know how to handle the request.
For the fix, edit the beat config file and add the Host header to the output.elasticsearch /headers section.
For example:
output.elasticsearch:
headers:
HOST: elasticsearch.example.com
And here is my full filebeat.yml file:
filebeat.inputs:
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "https://kibana.example.com:443"
ssl.verification_mode: full
ssl.certificate_authorities: ["c:\\Program Files\\WinFileBeat\\example_com-root.crt"]
output.elasticsearch:
hosts: ["elasticsearch.example.com:443"]
protocol: "https"
ssl.verification_mode: full
ssl.certificate_authorities: ["c:\\Program Files\\WinFileBeat\\example_com-root.crt"]
headers:
HOST: elasticsearch.example.com
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
setup.template.overwrite: true
setup.template.enabled : true
setup.ilm.pattern: "{now/M{yyyy.MM}}-000001"
setup.ilm.enabled: true
For reference, this is the error I was seeing in my logs:
2019-07-24T18:23:10.292-0600 ERROR elasticsearch/elasticsearch.go:255 Error connecting to Elasticsearch at https://elasticsearch.example.com:443: 503 Service Unavailable: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
2019-07-24T18:23:10.292-0600 ERROR fileset/factory.go:131 Error loading pipeline: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://elasticsearch.example.com:443: 503 Service Unavailable: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
]