Operate the Orleans Execution Model

Derived page. The behaviour described here is specified by the orleans-execution capability under openspec/specs/. That specification is the source; this page explains and illustrates it. Where the two disagree, the specification is right and this page is a bug.

Migrate to the Orleans Execution Model says what a host registers. This page says what an operator has to know once it runs.

A silo that dies hard, and a replacement on another address

Orleans keeps a membership table of silos. A silo that is killed — no graceful stop — leaves its row marked active. A new silo that starts on a different network address must reach every active silo in the table before it may join. It cannot reach the dead one, retries for MaxJoinAttemptTime (five minutes by default), and then fails with OrleansClusterConnectivityCheckFailedException. Shortening the IAmAlive settings does not change this: they decide when a stale entry is logged, not whether a joiner waits for it.

This is the shape an orchestrator produces when it replaces a crashed pod that was the cluster's only silo with a new pod on a new address. There are three ways out:

Run two silos. A surviving active silo votes the dead one out after its missed probes, and the replacement joins as soon as that is done. This is the answer Orleans is built around.
Restart on the same address. A silo that comes back on the address it had declares its older self dead on start and is ready at once, and a timer that came due while it was down fires. An orchestrator with stable network identities gives this for free.
Clean the membership table. Delete the dead silo's row from the membership table before the replacement starts, or run the reset while no silo of the cluster runs.

A silo that is stopped rather than killed leaves nothing behind.

The grain directory

The grains whose single activation must survive an unstable cluster — the store readers of projections and sagas, singleton work, timer owners, stateful processes and the heavy-work permits — are placed in the storage-backed directory the host registers with AddStrataraOrleans. Aggregate grains and command runners use the runtime's built-in directory: a second activation of an aggregate ends in a concurrency conflict at the store, which is the guarantee the bus workers have always relied on.

With Redis:

var redis = StackExchange.Redis.ConfigurationOptions.Parse(builder.Configuration.GetConnectionString("redis")!);

builder.UseOrleans(silo => silo
    .AddStrataraOrleans((s, name) => s.AddRedisGrainDirectory(name, options => options.ConfigurationOptions = redis)));

A silo without a directory under that name fails at start and names the call.

Kept commands

A recorded command whose handler keeps failing is resumed up to MessageRetryOptions.MaxDeliveryAttempts times and then kept: it stays in the outbox table with its attempt count and its last failure, and the commands after it are still resumed. Find the kept commands with:

SELECT id, aggregate_id, attempt_count, last_failure
FROM outbox_entry
WHERE kept_at IS NOT NULL;

Once the cause is fixed, return a kept command; it is resumed with its attempts starting over:

UPDATE outbox_entry
SET kept_at = NULL, attempt_count = 0
WHERE id = @id AND kept_at IS NOT NULL;

A partition that stops advancing

An entry a projection or saga cannot apply stops its partition. The checkpoint stays before the entry, the failure is logged with the entry's identity and counted, and the entry is tried again on every wake-up and poll — nothing after it in that partition advances until it passes. The event identifiers are listed in the log events schema. A missing prerequisite from another partition is retried under the preceding-fact policy in the same way.

Under the portable reader a partition also stops at an entry that has no partition position: a process appended it without PartitionCounterInterceptor. The logged failure names the entry, the interceptor and PartitionCounterBackfill. Add the interceptor to the write context of that process, then run PartitionCounterBackfill.RunAsync once; the partition continues from where it stopped. A host that refuses to start naming a partition counter beyond its partition count was configured with a lower count than the store was counted with; restore the count.

Reset what the model keeps

IExecutionModelReset clears everything the model keeps beside the event stream for the host's deployment: the reminders of its service, and with them every durable timer; the membership rows of its cluster; the checkpoints of the projections and sagas it registers; and the grain directory's entries, through a callback the host supplies because the directory is its choice. The event stream is never touched, and a host started afterwards rebuilds those checkpoints from it. The report counts what was removed of each.

var orleansDb = builder.Configuration.GetConnectionString("orleans")!;
builder.Services.AddStrataraExecutionModelReset<AppReadDbContext>(
    orleansDb,
    async (services, cancellationToken) =>
    {
        // With the Redis directory: remove the cluster's keys and report how many.
        var multiplexer = services.GetRequiredService<StackExchange.Redis.IConnectionMultiplexer>();
        var keys = multiplexer.GetServer(multiplexer.GetEndPoints()[0]).Keys(pattern: "*my-cluster*").ToArray();
        return keys.Length == 0 ? 0 : await multiplexer.GetDatabase().KeyDeleteAsync(keys);
    });

var report = await app.Services.GetRequiredService<IExecutionModelReset>().ResetAsync();

Run it while no silo of the cluster runs: a running silo writes its membership and reminders back.

Resolve it from the host's own composition — the one that calls AddStrataraProjectionGrains and AddStrataraSagaGrains. The checkpoints it removes are those of the projections and sagas registered there; a tool that registers none removes no checkpoint and reports zero. Another consumer's checkpoints in the same read store stay, and so do those of a projection the host no longer registers: nothing reads them, and removing them is a delete the host owns.

A checkpoint belongs to a consumer and a partition, not to a deployment. Two deployments can keep their checkpoints in one read store only when no projection name is registered by both. Every deployment's store-reading sagas read under one consumer, so at most one deployment sharing a read store runs AddStrataraSagaGrains; two would overwrite each other's positions.

Reminder profile and clocks

A deployed silo runs Orleans' reminder defaults. Keep-alive periods and the timer retry period are kept as reminders, so they must be at least the runtime's minimum reminder period; the host fails at start otherwise. Lowering ReminderOptions.MinimumReminderPeriod is for tests that need to observe a reminder within seconds.

A timer's due time is computed on the host that registered it and its tick runs on a silo with a clock of its own. A tick earlier than the due time by less than DurableTimerOptions.DueTolerance fires instead of waiting a whole retry period.

A timer fires once however long its handler runs. When a handler outlasts the reminder call's response timeout, the runtime delivers the next tick while it still runs; that tick does nothing, and the timer is unregistered once the handler has completed.

Heavy commands and their aggregate

A heavy command runs in the bounded heavy-work pool, not in its aggregate's activation, so a long unit does not hold back the aggregate's other commands. It therefore keeps no order with them: a command dispatched after it for the same aggregate does not wait for it, and where both append, the store's version check refuses the later one, which is resumed within MessageRetryOptions.MaxDeliveryAttempts like any failing command. Mark a command heavy only where it rarely meets a stream of other commands on its aggregate.

Table of Contents