I finished moving my Elastic Stack to Docker, which is now fronted by a reverse proxy, HAProxy. As soon as I did that, my beats stopped working and started to receive a 503 service unavailable.

The problem is because the Host header isn't being sent (or it's incorrect) so the proxy doesn't know what to do with the request. This is typical of any reverse proxy. I've worked with quite a few different types, and they all seem to return a 503 when they don't know how to handle the request.

For the fix, edit the beat config file and add the Host header to the output.elasticsearch /headers section.

For example:

output.elasticsearch:
  headers:
    HOST: elasticsearch.example.com

And here is my full filebeat.yml file:

filebeat.inputs:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:
  host: "https://kibana.example.com:443"
  ssl.verification_mode: full
  ssl.certificate_authorities: ["c:\\Program Files\\WinFileBeat\\example_com-root.crt"]

output.elasticsearch:
  hosts: ["elasticsearch.example.com:443"]
  protocol: "https"
  ssl.verification_mode: full
  ssl.certificate_authorities: ["c:\\Program Files\\WinFileBeat\\example_com-root.crt"]
  headers:
    HOST: elasticsearch.example.com

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

setup.template.overwrite: true
setup.template.enabled : true
setup.ilm.pattern: "{now/M{yyyy.MM}}-000001"
setup.ilm.enabled: true

For reference, this is the error I was seeing in my logs:

2019-07-24T18:23:10.292-0600	ERROR	elasticsearch/elasticsearch.go:255	Error connecting to Elasticsearch at https://elasticsearch.example.com:443: 503 Service Unavailable: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2019-07-24T18:23:10.292-0600	ERROR	fileset/factory.go:131	Error loading pipeline: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://elasticsearch.example.com:443: 503 Service Unavailable: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
]