Easy Health Checks in ASP.Net Core
As we write more services we need a consistent and easy way of monitoring them.
As we write more services we need a consistent and easy way of monitoring them.
The more you monitor your systems the more you will realize that just watching logs for errors is not a viable solution. One of the reasons is that you generally won't know of an issue until a request fails. Preemptive monitoring with health checks are far superior to a reactive approach of monitoring like log watching.
I'm not saying monitoring logs is bad. It's still a good thing to do and has a great deal of value. I just don't feel that its the best for overall application state monitoring.
This post isn't about how to check a specific part of your application or what you need to check for, however, the final example is checking the state of 2 URLs. It is up to you on how you determine the different states. This is about how to expose that health check information in an easy and consistent way throughout your systems.
When it comes to the state of an application, there's really 3 different states, up, degraded and down. Simple applications may only have up and down.
Up, the easiest state, means that your application is functioning properly.
Degraded is a bit more subjective. For us it means that some part of the application isn't functioning for whatever reason. Could be a queue is getting backed up, requests are taking a little longer than usual or maybe a feature of an application isn't working. But the core functionality of the application is still working. The average user shouldn't be seeing error messages at this point.
Down is also a bit subjective, but for us it is a core part of application isn't functioning as expected and the average end user will most likely be seeing an error message.
An example of the difference, lets take a home banking application. Many different parts and pieces to it. If everything works, it's up, however, if just a piece, like paying bills, is down, we considered it degraded. People can still get in and do everything else. The core application is still responding. However if there was some underlying issue that prevented the user from logging in, then the system was down because the core application was unavailable.
As usual, the companion code to this is available on GitHub at https://github.com/veccsolutions/Vecc.HealthCheckExample. I broke each change out into a separate branch for easy viewing of what was needed. And it's not a lot.
We're going to start out with an empty ASP.Net Core web site.
Adding a simple health check
Code: https://github.com/veccsolutions/Vecc.HealthCheckExample/compare/master...1-BasicHealthCheck
Now that we have a basic web application add the first NuGet package, Microsoft.Extensions.Diagnostics.HealthChecks. This will give you the ability to expose a very basic health check. It will report one of 3 states, Healthy, Degraded and Unhealthy. That's all it will return, no detail about the state of that health check. We'll go over adding more detail in the next part. This package will expose endpoints that will return a 200 status code for a system that Healthy or Degraded and a 503 for a system that is Unhealthy. This is invaluable for a health check in Kubernetes or other similar system that makes determinations by HTTP status codes.
With the package added, we need to add the individual health checks. They go in startup.cs.
First, the part that every poster out there seems to forget, the using statements.
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
Next we add the services to the ConfigureServices method, like this with a few checks.
You use the .AddHealthChecks() extension method on services and then call the .AddCheck extension method to configure each of the checks. There's a number of different ways of doing a health check, but we're keeping it simple. We'll add a ping which will always return health, and 3 remote checks. Each will return one of the 3 different supported states.
services.AddHealthChecks()
.AddCheck("ping", () => new HealthCheckResult(HealthStatus.Healthy, "pong"), new string[] { "ping" })
.AddCheck("remote1", () => new HealthCheckResult(HealthStatus.Healthy, "always healthy"), new string[] { "remote" })
.AddCheck("remote2", () => new HealthCheckResult(HealthStatus.Degraded, "always degraded"), new string[] { "remote" })
.AddCheck("remote3", () => new HealthCheckResult(HealthStatus.Unhealthy, "always unhealthy"), new string[] { "remote" })
The final ConfigureServices should look like this:
public void ConfigureServices(IServiceCollection services)
{
services.AddControllersWithViews();
services.AddHealthChecks()
.AddCheck("ping", () => new HealthCheckResult(HealthStatus.Healthy, "pong"), new string[] { "ping" })
.AddCheck("remote1", () => new HealthCheckResult(HealthStatus.Healthy, "always healthy"), new string[] { "remote" })
.AddCheck("remote2", () => new HealthCheckResult(HealthStatus.Degraded, "always degraded"), new string[] { "remote" })
.AddCheck("remote3", () => new HealthCheckResult(HealthStatus.Unhealthy, "always unhealthy"), new string[] { "remote" })
}
After we have that we need to add the route endpoints in the Configure method. I wanted each service to have a ping endpoint so I can quickly tell if it is reachable. I don't want it to go out to other services because this endpoint could be called very frequently and you can easily run into a recursion loop which would be bad (service a checks service b, service b checks service a, rabbit hole).
Add the endpoints with:
endpoints.MapHealthChecks("/hc/ping", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ping"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecks("/hc/remote", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("remote"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
Your final Configure method should now look like this:
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapHealthChecks("/hc/ping", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ping"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecks("/hc/remote", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("remote"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecksUI();
endpoints.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
});
}
We now have a static ping endpoint and another remote endpoint that pretends to hit 3 other services. You can get to them by going to /hc/ping and /hc/remote. /hc/ping should say Healthy and /hc/remote should show Unhealthy.
That's it for the simple health check.
Adding more detailed health check results
Having those health checks will help a lot, but it's not the best. Mostly because we won't know why our application is in a degraded or unhealthy state. That makes it very difficult to figure out what is wrong. We need those endpoints to return information about the checks that were performed.
Fortunately, it's easy. A nice group of people built a NuGet package that does just that. It's the AspNetCore.HealthChecks.UI.Client NuGet package. So go ahead and bring that in. Next we'll add another using statement, then a property to the options in the endpoints.
The using statement:
using HealthChecks.UI.Client;
The property:
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
Your Configure method should look like this now.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapHealthChecks("/hc/ping", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ping"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecks("/hc/remote", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("remote"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecksUI();
endpoints.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
});
}
Start your application and take a look at those health check endpoints. There is a lot more information now. Things like what check ran, the result of that check, the output of that check and how long it took. They will still return the same HTTP status codes that depict the state of your application as the basic health check above.
Adding a nice user friendly UI
Next up is displaying those health checks. We need a good way to visualize it, because we're people and looking at raw ugly JSON isn't very much fun and it's hard to see at a glance what state a system is in.
Luckily for us, the same group of people that made the NuGet package that output the detailed health check information also built a package that will visualize it for us. It's just as easy to implement as the other one. There are 2 NuGet packages that you will need. The first contains the frontend application itself, AspNetCore.HealthChecks.UI. The second is the backend storage for things like historical results. I'm not building that out in this demo so I'll just use their in-memory package, AspNetCore.HealthChecks.UI.InMemory.Storage.
Now that we have the packages installed we need to add the services required and configure the health check endpoints. In your ConfigureServices method add the following:
services.AddHealthChecksUI((options)=>
{
options.AddHealthCheckEndpoint("ping", "/hc/ping");
options.AddHealthCheckEndpoint("remote", "/hc/remote");
})
.AddInMemoryStorage();
It should now look like this:
public void ConfigureServices(IServiceCollection services)
{
services.AddControllersWithViews();
services.AddHealthChecks()
.AddCheck("ping", () => new HealthCheckResult(HealthStatus.Healthy, "pong"), new string[] { "ping" })
.AddCheck("remote1", () => new HealthCheckResult(HealthStatus.Healthy, "always healthy"), new string[] { "remote" })
.AddCheck("remote2", () => new HealthCheckResult(HealthStatus.Degraded, "always degraded"), new string[] { "remote" })
.AddCheck("remote3", () => new HealthCheckResult(HealthStatus.Unhealthy, "always unhealthy"), new string[] { "remote" });
services.AddHealthChecksUI((options)=>
{
options.AddHealthCheckEndpoint("ping", "/hc/ping");
options.AddHealthCheckEndpoint("remote", "/hc/remote");
})
.AddInMemoryStorage();
}
After the services are added we need to set up the endpoint, in the Configure method add this:
endpoints.MapHealthChecksUI();
It should now look like this:
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapHealthChecks("/hc/ping", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ping"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecks("/hc/remote", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("remote"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
endpoints.MapHealthChecksUI();
endpoints.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
});
}
Now run the application and head to /healthchecks-ui.
That's all there is to adding a nice frontend on top of those great health check endpoints.
A simple URI health check
Code: https://github.com/veccsolutions/Vecc.HealthCheckExample/compare/3-EasyUI...4-SimpleUriCheck
This is just a quick example of using a package from the same people who wrote the UI and detailed health check writer. Add their AspNetCore.HealthChecks.Uris NuGet package. You can then use the new extension methods added to the result of AddHealthCheck in the ConfigureServices method.
Example:
services.AddHealthChecks()
.AddCheck("ping", () => new HealthCheckResult(HealthStatus.Healthy, "pong"), new string[] { "ping" })
.AddCheck("remote1", () => new HealthCheckResult(HealthStatus.Healthy, "always healthy"), new string[] { "remote" })
.AddCheck("remote2", () => new HealthCheckResult(HealthStatus.Degraded, "always degraded"), new string[] { "remote" })
.AddCheck("remote3", () => new HealthCheckResult(HealthStatus.Unhealthy, "always unhealthy"), new string[] { "remote" })
.AddUrlGroup(new Uri("https://idontexist.frakkingsweet.com"), "broken", HealthStatus.Unhealthy, new string[] { "remote" })
.AddUrlGroup(new Uri("https://www.google.com"), "google", HealthStatus.Degraded, new string[] { "remote" });
This is what my ConfigureServices looks like now:
public void ConfigureServices(IServiceCollection services)
{
services.AddControllersWithViews();
services.AddHealthChecks()
.AddCheck("ping", () => new HealthCheckResult(HealthStatus.Healthy, "pong"), new string[] { "ping" })
.AddCheck("remote1", () => new HealthCheckResult(HealthStatus.Healthy, "always healthy"), new string[] { "remote" })
.AddCheck("remote2", () => new HealthCheckResult(HealthStatus.Degraded, "always degraded"), new string[] { "remote" })
.AddCheck("remote3", () => new HealthCheckResult(HealthStatus.Unhealthy, "always unhealthy"), new string[] { "remote" })
.AddUrlGroup(new Uri("https://idontexist.frakkingsweet.com"), "broken", HealthStatus.Unhealthy, new string[] { "remote" })
.AddUrlGroup(new Uri("https://www.google.com"), "google", HealthStatus.Degraded, new string[] { "remote" });
services.AddHealthChecksUI((options)=>
{
options.AddHealthCheckEndpoint("ping", "/hc/ping");
options.AddHealthCheckEndpoint("remote", "/hc/remote");
})
.AddInMemoryStorage();
}
Conclusion
This was so simple that at first I didn't think I did it right.
I'm really excited to use this in future projects and add it to existing ones.
To get more information on the available health check helper packages you can check out their Git repository at https://github.com/xabaril/AspNetCore.Diagnostics.HealthChecks. There is a ton of easy to use health checks already built.