Resiliency

Configure policies and monitor app and sidecar health

Distributed applications are commonly comprised of many microservices, with dozens - sometimes hundreds - of instances scaling across underlying infrastructure. As these distributed solutions grow in size and complexity, the potential for system failures inevitably increases. Service instances can fail or become unresponsive due to any number of issues, including hardware failures, unexpected throughput, or application lifecycle events, such as scaling out and application restarts. Designing and implementing a self-healing solution with the ability to detect, mitigate, and respond to failure is critical.

Resiliency Policies

The Dapr API calls a resiliency policy can be applied toA dashed boundary named Your solution holds five applications, each an app hexagon with a Dapr sidecar docked on it. App A sits in the middle: its sidecar gets state from a state store on the left, publishes to a message broker on the right, calls a method on App B below it, and calls an actor on App C. An input binding triggers App A from above. From the broker, two subscribe lines run to App D and App E, and App E invokes an output binding to a system outside the boundary. App B retrieves a secret from a secret store. Seven white callout cards, each with a dashed leader to the call it names, give the resiliency target for every one: state management, publish and subscribe, service invocation, actors, secret management and the two bindings, with the target keys apps, actors, and components outbound or inbound.Your solutionExternalsystemTriggerState storeGet statePub/subPublishSubscribeCall actorCall methodSecret storeRetrieve secretExternalsystemApp AApp BApp CApp DApp EState managementcomponents: outboundInput bindingcomponents: inboundPublish and subscribecomponents: outbound, inboundService invocationappsActor callsactorsSecret managementcomponents: outboundOutput bindingcomponents: outbound
Download SVG

Dapr provides a capability for defining and applying fault tolerance resiliency policies to your application. You can define policies for following resiliency patterns:

  • Timeouts
  • Retries/back-offs
  • Circuit breakers

These policies can be applied to any Dapr API calls when calling components with a resiliency spec.

App Health Checks

Dapr probing the health of the app it servesAn application hexagon named my-app on the left, its Dapr sidecar on the right. The app provides a health endpoint, drawn as a ball on a stalk beside it and named /healthz. An arrow runs from the sidecar to that endpoint, labelled "Health probe, every 5 seconds", with a note that a 200 to 299 response is healthy. Below, a dashed card headed "Turn it on" holds a dapr run command with the app health check flags. Beside it a second card, "After 3 failures in a row", lists what Dapr stops doing on the application behalf: pub/sub subscriptions, input bindings, service invocation and actor types.my-app/healthzHealth probe,every 5 seconds200-299 is healthyTurn it ondapr run \  --app-id my-app \  --app-protocol http \  --enable-app-health-check \  --app-health-check-path=/healthzAfter 3 failures in a rowpub/sub subscriptions stopinput bindings stopservice invocation short-circuitsactor types unregister
Download SVG

Applications can become unresponsive for a variety of reasons. For example, they are too busy to accept new work, could have crashed, or be in a deadlock state. Sometimes the condition can be transitory or persistent.

Dapr provides a capability for monitoring app health through probes that check the health of your application and react to status changes. When an unhealthy app is detected, Dapr stops accepting new work on behalf of the application.

Read more on how to apply app health checks to your application.

Sidecar Health Checks

A platform probing the health of the Dapr sidecarAn application hexagon named my-app on the left, joined by a plain line to its Dapr sidecar on the right. The sidecar provides a health endpoint, drawn as a ball on a stalk beyond it and named /v1.0/healthz. A Kubernetes mark sits further right, with an arrow to that endpoint labelled "Readiness and liveness". Below, a dashed card headed "The endpoint" gives GET http://localhost:<daprPort>/v1.0/healthz and its two answers: 204 when Dapr is healthy and 500 when it is not. Beside it a note says the endpoint is for infrastructure probes and not for your application code to depend on.my-app/v1.0/healthzReadinessand livenessThe endpointGET http://localhost:<daprPort>/v1.0/healthz204  Dapr is healthy500  Dapr is not healthyWho calls itthe platform, for its probes,and an SDK waiting for thesidecar to be ready - neveryour own application code
Download SVG

Dapr provides a way to determine its health using an HTTP /healthz endpoint. With this endpoint, the daprd process, or sidecar, can be:

  • Probed for its health
  • Determined for readiness and liveness

Read more on about how to apply dapr health checks to your application.

Next steps