Resilience & Health Checks

Connections drop, servers restart, transient errors happen. WitRPC ships two complementary mechanisms for surviving them, plus health check integration for observing connection state in production.

The two work independently and combine naturally: reconnection restores the channel, retries repeat the call.

Automatic reconnection

With auto-reconnection enabled, the client monitors its connection and re-establishes it after a drop:

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithJson();
    options.WithEncryption();
    options.WithAutoReconnect();   // defaults
});

Behavior is configurable:

options.WithAutoReconnect(reconnect =>
{
    reconnect.MaxAttempts = 10;                        // 0 = unlimited
    reconnect.InitialDelay = TimeSpan.FromSeconds(1);
    reconnect.MaxDelay = TimeSpan.FromMinutes(2);
    reconnect.BackoffMultiplier = 2.0;
    reconnect.ReconnectOnDisconnect = true;            // also reconnect when the server closes the connection
});

Delays grow exponentially between attempts (InitialDelay × BackoffMultiplier^n, capped at MaxDelay), so with the defaults the sequence runs 1s, 2s, 4s, 8s and onward up to two minutes. Backoff prevents a fleet of clients from hammering a server that just came back up.

Reconnection callbacks

Three callbacks expose the process, which is where applications refresh their state after an interruption:

options.WithAutoReconnect(reconnect =>
{
    reconnect.OnReconnecting = (attempt, delay) =>
        Console.WriteLine($"Reconnection attempt {attempt}, waiting {delay}...");

    reconnect.OnReconnected = () =>
    {
        Console.WriteLine("Reconnected.");
        // Refresh state the client may have missed while offline
    };

    reconnect.OnReconnectionFailed = lastException =>
    {
        Console.WriteLine($"Reconnection failed: {lastException?.Message}");
        // Alert the user, switch to offline mode
    };
});

Connection state and manual control

The client exposes its state through ConnectionState:

switch (client.ConnectionState)
{
    case ReconnectionState.Connected:     /* normal operation */          break;
    case ReconnectionState.Reconnecting:  /* attempts in progress */      break;
    case ReconnectionState.Disconnected:  /* idle, not reconnecting */    break;
    case ReconnectionState.Failed:        /* attempts exhausted */        break;
}

await client.StopReconnectionAsync() cancels ongoing attempts, which belongs in shutdown paths. WithoutAutoReconnect() disables the feature explicitly.

Retry policies

A retry policy repeats an individual call that failed with a retryable error. Since 3.0 the policy is deliberately conservative in two ways.

Only client-local failures are retried by default. The retryable statuses out of the box are Timeout (the call did not come back in time) and TransportError (the connection failed mid-call). Both are outcomes where a second attempt can genuinely succeed. A service fault (InternalServerError, the implementation threw) is never retried by default: re-running failed business logic must be an explicit decision, made through RetryableStatuses or RetryOn<TException>().

Only methods declared idempotent are retried. With no declarations the policy is inert, so a command never silently runs twice:

options.WithRetryPolicy(retry =>
{
    retry.MaxRetries = 3;
    retry.InitialDelay = TimeSpan.FromMilliseconds(500);
    retry.MaxDelay = TimeSpan.FromSeconds(5);
    retry.BackoffType = BackoffType.Exponential;

    // Declare what is safe to re-execute:
    retry.MarkIdempotent(nameof(IMyService.GetStatus), nameof(IMyService.ListItems));
    // or, deliberately: retry.RetryAllMethods = true;
});

BackoffType selects how delays grow between attempts: Fixed keeps them constant, Linear grows them arithmetically, Exponential (the default) doubles them under the configured multiplier. An OnRetry callback (Action<Exception?, int, TimeSpan>) fires before each attempt, useful for logging.

Why a retried call never executes twice

Every invocation carries an InvocationId that stays stable across retries. When the first attempt did reach the server and only its response was lost, the server recognizes the retry as a duplicate and answers it from a bounded per-connection cache instead of re-executing the method. Declaring a method idempotent therefore says "a retry of this call is safe", and the de-duplication makes it safe even in the lost-response case that classic retry schemes get wrong.

The declaration is still yours to make: which operations are reads, which are state-setting updates, and which must never repeat (payments, counters, appends) is a property of your service semantics. The framework's default of retrying nothing until told keeps the dangerous case opt-in.

Health checks

The OutWit.Communication.Client.HealthChecks package plugs WitRPC clients into ASP.NET Core health checks. It works with clients registered through the dependency injection integration, referencing them by name:

builder.Services.AddHealthChecks()
    .AddWitRpcClient("MainClient");   // name of the DI-registered client

The check reports the named client's connection health through the standard health endpoint, alongside databases and other dependencies, so existing monitoring picks it up without special handling. Optional parameters set the check name, failure status, and tags. Registering named clients is covered in Dependency Injection →.

Putting it together

A production client typically enables all three:

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMemoryPack();
    options.WithEncryption();
    options.WithAccessToken(token);

    options.WithAutoReconnect(r =>
    {
        r.MaxAttempts = 0;                            // keep trying
        r.OnReconnected = () => RefreshApplicationState();
    });

    options.WithRetryPolicy(r =>
    {
        r.MaxRetries = 3;
        r.BackoffType = BackoffType.Exponential;
        r.MarkIdempotent(nameof(IMyService.GetOrders), nameof(IMyService.GetStatus));
    });
});

Reconnection keeps the channel alive across outages, retries absorb transient call failures, and the health check makes both visible to monitoring. What remains application work: refreshing missed state in OnReconnected, and declaring which methods are idempotent, since without those declarations the retry policy stays inert by design.