Part VII - Software Platform Engineering
SYSE.30 - Make Software Builds Repeatable and Traceable
Status: Stable
Normativity: Guidance within the stated software-engineering use; examples are illustrative.
SYSE.30:1 - Problem frame
Use this pattern when two builds of “the same source” produce unexplained differences, a package cannot be rebuilt after a worker disappears, or nobody can recover which inputs produced a tested artifact. Start with one required output and one build attempt. Recover its source, resolved dependencies, instructions, toolchain and environment before interpreting a source label as a complete build identity.
Return the build procedure or the account that links a specified artifact to its build and inputs, according to the question. For a reproducibility claim, give the comparison from fresh executions under the stated conditions. Identify any input or difference that keeps the requested result from being established. Later tests and artifact claims can refer to the identified bytes. Rebuilding or delivering those bytes also requires access to the necessary inputs or artifact. Correct behavior and a trusted builder each require evidence for that claim.
If the needed artifact already exists and the task is to transfer that exact artifact, use SYSE.32. If separately developed changes still need integration into a checked mainline, use the direct CI source route in SYSE.31; reproducible isolated branches are not an integrated application.
SYSE.30:2 - Problem
A source revision omits many influences on a build: dependency resolution, generated inputs, tool versions, platform defaults and ambient state. “Run the same command again” can reuse yesterday’s output or resolve a different dependency today. The command may succeed in both cases without reproducing the claimed artifact.
Without an explicit output and comparison criterion, a team can also confuse repeatable execution, equivalent behavior and byte identity. Each supports a different later use.
SYSE.30:3 - Forces
| Force | Practical tension |
|---|---|
| Reconstruction | Recording more inputs helps recovery; irrelevant detail increases maintenance and hides the inputs that change output. |
| Fast builds | Caches reduce delay but can hide an undeclared dependency or a build that never ran. |
| Byte identity | Strict comparison detects unexplained variation; some outputs deliberately contain separately varying operational information. |
| Availability | Pinned inputs stabilize meaning, but a version name is not enough when its bytes later disappear. |
SYSE.30:4 - Solution
SYSE.30:4.1 - Fix the output and the claim
Name the artifacts needed by the next consumer: for example an executable package and its dependency bundle. Distinguish them from build logs and other ancillary results. State whether the current need is to repeat the procedure, explain provenance, compare functional behavior, or reproduce the specified artifacts byte for byte.
Use the Reproducible Builds meaning of a reproducible build only for the byte-identical specified artifacts under the declared source, environment and instructions. If a comparison deliberately ignores selected metadata, state that narrower equivalence and why it is enough for the receiving use; do not report the same result as byte reproducibility. Do not exclude a differing file merely to make the result pass.
SYSE.30:4.2 - Recover what the build actually consumed
Begin with the source revision and any uncommitted or generated input included in the attempt. Recover the exact build instructions, resolved dependency versions and content identities, toolchain, build configuration and relevant environment conditions. A dependency declaration and the dependencies resolved for this build are separate evidence.
Inspect influences that can change the selected output: local files, network downloads, compiler flags, environment variables, locale, paths, timestamps, random values and ordering. Remove an unnecessary source of variation when the operation permits it. Control or record the influences that still matter to the specified output. A container image can help identify a toolchain without describing every relevant host or external-service condition.
Retain the inputs or a usable authorized acquisition route for the intended reconstruction interval. A recorded URL that no longer supplies the identified bytes is a recovery gap. Reuse existing build metadata and configuration records when they answer these questions; no particular manifest format is required.
SYSE.30:4.3 - Construct one controlled execution
When a build procedure is needed, use an adequate existing one or turn the recovered instructions into an executable procedure with explicit inputs and outputs. For a controlled execution, start in a clean workspace and obtain the named inputs. Prevent an undeclared local file or an unbounded “latest” dependency from silently changing the result. Keep secrets outside public logs and use the authorized access mechanism.
Treat caches according to their role. An identified dependency cache can supply a known input; an old final package cannot demonstrate that the present build procedure works. For the reconstruction check, execute the relevant build work rather than copy a previously produced output. If complete isolation is expensive, state which remaining influences are controlled, observed or unresolved.
Link the resulting artifact identities to the execution that produced them and to the inputs it used. SYSE.13 supplies configuration identity. A successful exit without the promised output is not successful construction.
SYSE.30:4.4 - Compare independent repetitions and explain differences
When the receiving use needs reproducibility, compare applicable evidence from independent executions. If that evidence is missing, build again from the identified inputs in a fresh execution. Where the claim includes transfer between builders, use the relevant independent builder or environment, not just a second invocation on the same dirty worker.
Compare the specified output bytes, commonly through a suitable cryptographic digest followed by a useful difference inspection when they differ. Change one suspected influence at a time when that can discriminate the cause. Explain the variation before accepting an equivalence claim.
For incidental build timestamps, prefer removing them from the specified artifact or deriving a needed date from the named source. Retain the time the build ran in separate build evidence when it is needed. A tool-specific fixed-date mechanism works only where the selected tool supports it; changing a clock globally can affect behavior that depends on elapsed time. Rebuild and compare after the repair.
A matching pair is bounded evidence, not proof against every future environment. If a dependency, toolchain or output requirement changes, qualify the affected build again. Preserve a useful partial result: “the build runs from these inputs, but archive order remains uncontrolled” is more informative than “reproducible” with an unexplained exception.
SYSE.30:4.5 - Return the right result to the next use
Return the result for the receiving question: the procedure, the correspondence between a build, its inputs and its artifacts, or the comparison and conditions for a reproducibility claim. Include the parts needed to support that result. If an input is missing or a difference remains unexplained, identify it for the recipient.
Behavioral tests consume this artifact through SYSE.31. Artifact authenticity, trust expectations and promotion use SYSE.32. Actual runtime installation uses SYSE.41. None of those results is established by a build digest. Identify a package rebuilt for another environment and compare its bytes with those of the previously tested artifact before deciding which earlier evidence remains applicable.
SYSE.30:4.6 - Turn an inherited build into a maintainable construction
An inherited build often starts with one working developer command. Follow that command backward from the promised package. For each generated file, determine whether it is deliberately supplied as an identified input or produced by this build. Recover the producing actions for the latter; for each action, identify the files, tools and parameters that can affect its result. Continue until the remaining inputs are deliberately supplied source, acquired dependencies or declared environment conditions. This forms a build dependency graph: an edge says that one action needs another’s output, not merely that it happened later in a log. A generated header copied from a colleague’s machine must be accounted for through its producing action or as an identified supplied input obtainable for reconstruction; compilation success alone does not settle this.
Use the graph to decide what to control. A font compiler that downloads an unspecified font while building has an acquisition dependency. Obtain an identified permitted font first and provide it as an input, or retain the network dependency and its reconstruction limitation. A generator that reads the operator’s home directory has an ambient dependency: either make the required setting explicit or remove that read. Do not place credentials in the dependency archive; retain the authorized way to obtain the identified private input and the fact that future access is a reconstruction condition.
Separate input acquisition from transformation where that makes the construction more inspectable. Run the transformation with only the chosen input set and necessary services available, then inspect a failure for a genuinely missing prerequisite. A strict sandbox is useful for exposing hidden reads, but a successful sandboxed invocation is not a proof of deterministic output. A container can fix a user-space toolchain while still sharing a kernel or depending on a remote generator. Choose these controls for the actual claim instead of treating one packaging technology as a guarantee.
The same account governs caching and parallelism. A reused intermediate result must correspond to the relevant action, input contents, toolchain and effective parameters. If its lookup key ignores a compiler option that changes output, the cache can return a package that no action in this attempt produced. Independent actions may run together when their inputs are ready and they do not communicate through an unmodelled scratch file or mutable service. A build system can schedule a graph efficiently only to the extent that its declarations represent the dependencies between the actions.
When two clean builds disagree, locate the first differing intermediate result rather than rewriting all controls. Compare its inputs and execution influences, repair the contributing action, then rerun the affected downstream construction. When intermediates match but the final archive differs, examine the packaging action, including member ordering, file attributes and embedded signatures. If signing intentionally introduces fresh values, define a separately reproducible payload and a separately verified signed envelope before using that distinction; it does not make the entire signed file byte-identical.
A practical qualification uses the variations the promise covers: a fresh workspace without final-output cache, another supported worker, and relevant differences such as path, locale or time. Where stale cache reuse could conceal a changed input, use evidence that the build consumes the change, or vary that input in a fresh execution to test it. Keep a periodic reconstruction or an input-retention check only when it protects a real future use. The ordinary delivery path can reuse an already qualified artifact; reconstruction capability and artifact promotion solve different problems.
For example, a reporting tool generates a parser from a grammar, compiles it with a library, and packages the executable with templates. An inherited build tracks only the source files. The recovered construction adds the grammar generator, grammar, library bytes and template contents to their actual consuming actions. A template-only change need not regenerate the parser, but it must change or revalidate the packaged result. If an upgrade changes the generator’s output, earlier parser tests cannot be inherited solely because the hand-written source stayed unchanged. This reasoning applies to another language or build tool without assuming that its cache or dependency scanner is already correct.
SYSE.30:5 - Archetypal Grounding
In a constructed ParcelWorks case, the developer supplies service revision r17 and asks for the package that will be tested. Two workers report successful builds with different package digests. Both claim r17. The team first names the service package as the reproducible artifact; the execution log is separate evidence.
Inspection recovers build recipe b3, dependency set d9 and runner image i7 from both attempts. The package contains the same service files but also a generated line, “built at 10:02” in the first attempt and “built at 10:07” in the second. The source label was correct but incomplete: wall-clock time changed a specified output.
The application does not use that line for business behavior, and exact execution time is needed only for diagnosis. The team removes the line from the package and retains it in the separate build evidence. A required source-version label remains r17. Two fresh executions of the repaired recipe, with the same d9 and i7, produce identical specified package bytes. The result names that tested input set and comparison.
Now change the case: a dependency named “stable” resolves to different bytes on the second worker. Removing the timestamp cannot repair this difference. Identify which dependency changed and, for reconstructing the earlier package, which bytes that build used. Use their identity to obtain or pin the corresponding input where available. If the original bytes cannot lawfully be obtained, exact reconstruction remains unavailable even though a newer package could be built and tested as a different candidate.
A third case reuses the first worker’s final package from cache. Matching digests prove that the same cached bytes were returned, not that the procedure reconstructed them. A fresh execution is still needed for the reconstruction claim.
The team can now identify which input changed, which artifact was tested and what the comparison establishes. For a later test failure, establish whether the behavior, the test or its execution conditions explain the observation.
SYSE.30:6 - Bias-Annotation
Successful builds on a familiar worker can hide dependence on its local state. A focus on byte equality can also distract from malicious or incorrect but deterministic output. Include the environment transfer relevant to the real use, and keep correctness and builder trust as separate questions.
SYSE.30:7 - Conformance Checklist
- The specified output and comparison criterion are explicit before differences are judged.
- The account identifies the source, resolved inputs and relevant environment used by the build, or the specific gap that limits recovery.
- For a claim about controlled construction, identify the clean execution, its stated conditions and its output. To trace an existing artifact, identify the execution that produced it and its inputs to the extent established.
- A reconstruction check does not substitute a cached final artifact for execution.
- Differences are explained or returned as a bounded gap; ignored metadata is not silently removed from the claim.
- Correctness, authenticity, mainline integration and deployment remain separate results.
SYSE.30:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| A source commit is treated as the complete build input. | Recover resolved dependencies, generated inputs and relevant environment conditions. |
| Two successful exits are called reproducibility. | Compare the specified outputs under an explicit criterion. |
| A build report names an available version but its bytes cannot be recovered. | Preserve the identified input or state the unavailable reconstruction dependency. |
| Equal digests are accepted as proof of safe software. | Use the separate behavior and trust questions for the actual artifact. |
SYSE.30:9 - Consequences
Differences become diagnosable and tested bytes remain identifiable. Controlled inputs and retention cost storage, maintenance and sometimes build speed. Apply stronger reconstruction controls when the receiving claim needs them; an already verified artifact can be transferred without rebuilding it when the receiving conditions permit. Check its identity and the evidence the destination needs.
SYSE.30:10 - Rationale
A build is an operation over more than a source-tree name. Recovering its actual inputs closes a practical explanation gap, while an explicit output criterion prevents a weaker comparison from being reused as a stronger result. Independent reconstruction and artifact trust then contribute different evidence.
SYSE.30:11 - SoTA-Echoing
For constructing recoverable builds, adopt the specified-input and byte-comparison criterion in the Reproducible Builds definition, with its environment recording and timestamp treatments. Adapt the isolation reasoning in Bazel Hermeticity without requiring Bazel. These professional approaches already address hidden dependencies and nondeterminism.
The construction here connects the chosen build claim to an inherited dependency graph, cache reuse, diagnosis of differing intermediates and the downstream tested artifact. A hermetic execution and a byte-reproducible result answer different questions. A retained artifact is often the economical delivery input; independent reconstruction costs additional execution and input retention and is selected for the claim that needs it.
Reopen the affected construction when a required input disappears, a toolchain or generator changes, an independent builder disagrees, or the receiving use needs another output/equivalence claim.
SYSE.30:12 - Relations
SYSE.13 supplies configuration identity. SYSE.31 explains change feedback. When checked branches still need a shared-mainline result, its §4.5 gives the DORA continuous-integration method. SYSE.32 covers artifact-specific trust and promotion, SYSE.33 reconstructible development and test conditions, and SYSE.41 runtime deployment. SYSE.4 governs reliance on evidence for a named engineering claim.
SYSE.30:End
SYSE.31 - Keep Software Feedback and Continuous Integration Fast and Trustworthy
Status: Stable
Normativity: Guidance within the stated software-engineering use; examples are illustrative.
SYSE.31:1 - Problem frame
Use this pattern when developers wait too long to learn whether a change works, rerun failures until they disappear, or receive green checks that do not answer the current engineering question. Start with one change and the decision its feedback must support. Identify what each check can establish, on which source or artifact, under which conditions, and who can act on failure.
The first result is a usable feedback arrangement: maintained checks, interpretable outcomes and a bounded response to a failed or missing result. The application team supplies the expected behavior; a software platform can make the feedback available without inventing that behavior or owning the release decision.
If adequate build and tests already exist and only integration of a small change is missing, go directly to the DORA continuous-integration Method in section 4.5. If the build inputs themselves cannot be recovered, use SYSE.30; if the conditions cannot be reconstructed, use SYSE.33.
SYSE.31:2 - Problem
A slow result arrives after the developer has changed context or several changes have accumulated. An unreliable result makes repetition cheaper than understanding. A fast but irrelevant result is equally dangerous: the suite can pass on one branch, one fixture or one old artifact while the integrated application fails.
The aim is to distinguish a usable change from a product regression, a broken test or environment, or a case that still lacks evidence, early enough to act on the result.
SYSE.31:3 - Forces
| Force | Practical tension |
|---|---|
| Speed and reach | Small checks give early feedback; some behavior needs a running application, real integration or human exploration. |
| Stable tests and real variation | Controlled fixtures aid diagnosis; removing genuine concurrency or environment variation can remove the defect being sought. |
| Local ownership and cooperation | Developers need to repair feedback quickly while testers and domain specialists contribute different acceptance knowledge. |
| Availability and maintenance | Keeping every test forever increases burden; dropping a test can remove the only evidence for an important claim. |
SYSE.31:4 - Solution
SYSE.31:4.1 - Connect a check to the decision
Name the changed behavior, the assertion under test and the result consumer. Recover the application team’s acceptance meaning rather than deriving it from current implementation. A test that repeats the implementation’s mistaken assumption is not an independent answer to the user question.
Bind each result to the source or artifact actually exercised, relevant configuration, test input and conditions. Distinguish a test description from a completed run. A missing run, lost observation or wrong artifact cannot be reported as a pass.
Begin with checks that discriminate the consequential failures for the current change. When necessary behavior is not specified well enough to judge, return that question to its owner. The number of existing tests cannot settle the missing requirement.
SYSE.31:4.2 - Put feedback where it can change the next action
Use fast focused tests for failures they can detect. Retain integration, acceptance, performance and exploratory work where their distinct results affect reliance. Choose the execution arrangement from those dependencies, not from a fixed proportion of test types.
Run relevant checks close enough to a change that the result remains actionable. Recover the actual waiting, execution and diagnosis time before choosing an improvement: a faster test does not repair a long queue or an unintelligible failure report. Parallel execution is useful only when isolation and capacity preserve the test’s meaning.
A slower test can remain necessary. Make its unresolved result visible to the decision that needs it instead of presenting early fast checks as full acceptance.
SYSE.31:4.3 - Make a failure reproducible and interpretable
Retain the tested revision/artifact, inputs, relevant environment and failure observation. Compare a suspected regression with the previous qualified behavior under matching conditions when that comparison can distinguish the cause.
Investigate variable outcomes. Shared mutable fixtures, uncontrolled clocks, order dependence and unavailable services can corrupt feedback. A nondeterministic product race is still a product defect; calling a test “flaky” does not settle which object failed.
Control a suspected source of interference and repeat a discriminating case. Change only what the comparison can account for. A rerun may help diagnosis, but a later pass does not erase an earlier failure under supported conditions.
SYSE.31:4.4 - Decide who repairs which failure
| Observed problem | Immediate response |
|---|---|
| The tested product violates the agreed behavior. | Return the failing case to the application team and withhold the reliance that required it. |
| The test asserts an obsolete or incorrect expectation. | Resolve the expectation with its owner, repair the test and re-exercise the behavior. |
| The environment or provider failed before the behavior could be observed. | Return that failure for environment/platform repair; retain the missing product evidence. |
| The result cannot be attributed to the candidate or conditions. | Recover attribution or rerun the right case; do not transfer the old pass. |
Route the failure to whoever can establish or repair the missing result under the required authority. Platform operators can repair provision; they do not acquire authority to change application behavior, withdraw a source change or approve release merely because they operate the test service.
If a test must be quarantined, identify the claim whose evidence was lost, its current consequence and a usable temporary replacement or stop. Name the repair responsibility and the condition for restoring the check. Quarantine is not an unlimited exemption or a way to improve the displayed pass rate.
SYSE.31:4.5 - Construct a stream of small integrations into the current mainline
Continuous integration is a way of keeping separately developing changes usable together. A server that builds isolated branches supplies feedback but does not create that combined result. The DORA CI Method is a directly usable starting treatment; the construction here explains its joins with the artifact, environment and recovery Methods in this framework.
First choose the integration boundary that matters to the application. Name the shared mainline and the components whose combination must work. One repository can contain independently deployable components, and several repositories can still contain a tightly coupled application. Choose checks for the behavior, interfaces and conditions those components must preserve together. A repository boundary is not evidence that two changes can be released independently.
Break a large change into increments that leave the mainline usable under its existing supported behavior. A preparatory increment may add an unused adapter; a later increment may switch one caller to it; removal waits until no supported caller needs the old route. Each increment should preserve the agreed behavior or openly change it under its acceptance conditions. A feature flag can separate installation from activation only if the disabled state really preserves the required behavior and the enabled path can be qualified. A dormant flag does not make an incompatible schema change harmless.
For one increment, take a current mainline base, construct the intended combined change, and run the relevant fast feedback against that combination. Resolve any incompatibilities in that combination before integration. When another change advances the mainline, update the candidate combination and re-establish the affected feedback. A merge queue is one possible implementation: it serializes or rebuilds candidate combinations against the current mainline state they are intended to join.
After integration, establish the build and feedback results for the resulting shared revision through the authoritative path. Reuse qualified evidence for that same combination and run the missing checks. Associate its produced artifact with that combined revision and its own results. An earlier artifact from the branch can be reused only when it is the same qualified subject under the needed conditions. The team must be able to find the current mainline result and its unresolved checks without reconstructing several private branch histories.
When the integration breaks, protect other development from treating the broken state as a usable base. Make restoration the immediate shared task. If a narrow repair can promptly restore the intended combination, integrate and check it. Otherwise the authorized team can withdraw the offending change and verify the resulting mainline. Withdrawing source changes does not undo an already executed external effect or data migration; those returns have their own state boundary. Keep the failed observation and unresolved business conflict so that a later attempt does not repeat it.
Choose integration frequency and feedback latency together. If a team accumulates several days of interacting changes because every run takes hours, asking for more merges alone creates a queue of unresolved combinations. First find the constraining feedback and improve its execution, split a genuinely independent check, or redesign the change so that each smaller increment remains useful. Slow necessary evidence can run separately while its affected release claim stays open. The sought gain is a short interval to a trustworthy combined result, not a mandated merge count.
For the range example in section 5, A and B are individually small but incompatible in business meaning. No branch strategy can choose whether a minimum of 8 or a maximum of 5 is intended. CI makes that conflict visible while it is still bounded. By contrast, adding a disabled new form with a lossless old-format adapter can be a legitimate preparatory increment, because the old behavior and storage contract remain usable while the new form is being developed.
SYSE.31:4.6 - Keep the complete feedback path usable as changes accumulate
Arrange checks by what they need and which decision they can change. A source-level invariant may run before packaging. A binary compatibility check consumes the produced package. An interaction with a real database needs a qualified running environment. A user-interface task can need browser behavior and human interpretation that a backend check cannot supply. Put the cheaper informative failure first where these dependencies permit; preserve the later question even when the early stage is green.
Use a small representative change to build this path before multiplying its variants. Give each stage a way to identify the candidate, obtain its actual inputs and return an actionable failure. When the first stage passes and a later stage is waiting, the current result is partly qualified. Do not let a single green label erase the pending contribution. Nor should every slow stage unnecessarily block unrelated editing or an independently qualified artifact.
Derive test selection from affected behavior and dependency reach. If a shared schema generator changes, all consumers of its changed output may be affected even if their hand-written files are untouched. If a test-selection mechanism cannot recover that reach, include a wider check or leave the conclusion narrower. Inspect escaped regressions for missing reach as well as missing assertions. Reducing test time by assuming unproven independence simply moves detection to a later, more expensive use.
For example, ParcelWorks can check lossless address split/join quickly with boundary values; check its write adapter and trigger in the actual supported database arrangement; then exercise display and save in the browser. These are complementary observations. Passing the first does not make the other two redundant. A new runner that halves unit-test time but starves the database stage has not necessarily shortened time to a releasable result; include queue and diagnosis delay when choosing that change.
SYSE.31:4.7 - Verify that the arrangement improves useful feedback
Use evidence from a known passing case, a meaningful failing change and a failure of the feedback mechanism. Exercise any missing case. Check that the responsible participants or systems can distinguish the outcomes and recover the tested subject. Inspect late-discovered defects: did the arrangement omit a needed question, use the wrong conditions, or fail to deliver its result in time?
Return the checks, the questions and conditions they cover, any missing evidence and how each failure will be handled, including who can act and which use must wait. Judge an improvement by the failures detected and repaired in time for use.
SYSE.31:5 - Archetypal Grounding
In a constructed ParcelWorks case, two address-service tests share the same fixture row. One test removes it during cleanup while another still reads it. The second test sometimes fails on unchanged application bytes; rerunning the suite merely changes the timing.
The developer first preserves the failing order and row identity. The fixture is then made independent for each attempt, with cleanup limited to that attempt’s data. Repeating the conflicting order no longer destroys another test’s input. A deliberately broken address transformation still fails its assertion. The repaired feedback therefore removed fixture interference without deleting the behavior check.
Now consider a different failure: an application permits parcel weights between a minimum and maximum, with the declared invariant minimum <= maximum. Mainline M0 has the range 1..10. Change A raises the minimum to 8; change B, developed from M0, lowers the maximum to 5.
| Tested state | Range | Feedback and consequence |
|---|---|---|
| M0 | 1..10 | The invariant passes. |
| A on M0 | 8..10 | A’s branch passes. |
| B on M0 | 1..5 | B’s branch passes. |
| B combined with already integrated A | 8..5 | The invariant fails despite no textual merge conflict. |
| Withdraw B while retaining A | 8..10 | The rebuilt integrated result passes the invariant again. |
Testing B against the actual current mainline would reject it before integration. If stale branch feedback was nevertheless used, the post-integration check finds the bad combination and blocks artifact qualification. The authorized application team withdraws B and verifies the restored result. The team must resolve the conflicting business intent about the permitted parcel-weight range before B is attempted again.
Finally, fast data-transformation tests can pass while a new address form hides a non-empty display line. The application-level acceptance or exploratory result remains necessary for that user behavior. Running more fast unit tests cannot substitute for the missing observation.
The team can distinguish what failed and who can act on it. It knows which change and behavior each passing check exercised.
SYSE.31:6 - Bias-Annotation
Teams can optimize the measured feedback time by excluding slow but necessary questions, or optimize pass rate by suppressing real failures. Developers may also assume that a familiar fixture represents all supported use. Inspect escaped defects and missing observations, including the perspective of application users and testers.
SYSE.31:7 - Conformance Checklist
- Each relied-on check has a concrete question, subject and applicable conditions.
- The arrangement retains distinct slow or human evidence where the decision needs it.
- Product, test, environment and missing-observation failures receive different responses.
- Quarantine exposes the lost assurance and a replacement or stop.
- Integration feedback concerns the current combined mainline, not only isolated branches.
- Repair work has an assigned maintainer; source withdrawal and release require the applicable authorization.
SYSE.31:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Rerun until green. | Preserve and explain the failed observation; use repetition diagnostically. |
| Delete the slowest test to meet a time target. | Recover the claim it answered and retain an adequate result or explicit limitation. |
| Two green branches imply a green mainline. | Exercise the current combined revision and restore it when the integration fails. |
| Infrastructure failure is reported as product success because no assertion failed. | Mark the product observation missing and repair the feedback mechanism. |
SYSE.31:9 - Consequences
Useful feedback can shorten the interval between a change and an informed response. It requires continuing test and environment maintenance and can expose missing acceptance knowledge. A clearer failure report or a smaller integrated change can shorten the wait for a useful answer.
SYSE.31:10 - Rationale
Feedback is useful only when it helps answer the current question and arrives where someone can act on it. Subject attribution, credible failure meaning and repair ownership protect that connection. Mainline integration supplies an additional result that branch-local build and tests cannot establish.
SYSE.31:11 - SoTA-Echoing
For prompt feedback and a usable combined mainline, adopt and adapt DORA Test automation and Continuous integration. Those methods combine small integrations, maintained tests and rapid restoration. The Continuous Delivery deployment pipeline arranges successive evidence across an integrated delivery path.
Sections 4.5–4.7 construct those joins for the shared base, artifact and environment. Compatible preparatory increments compete with a longer branch; fast scoped tests compete with broader slower checks. Select from interaction risk, feedback delay and recoverability, retaining a pending release question where necessary. Include the wait for the combined result when comparing feedback time.
Reopen test selection or integration structure when architecture, generated dependencies or supported behavior changes, regressions escape, or restoration no longer fits the working cadence.
SYSE.31:12 - Relations
SYSE.30 supplies recoverable build inputs and SYSE.33 supplies development and test conditions. For small-change integration, see the DORA method in §4.5. SYSE.32 uses artifact-specific evidence, SYSE.41 explains deployment, and SYSE.36 distinguishes observations of platform tasks from application tasks. SYSE.4 addresses reliance on evidence and SYSE.14 release decisions.
SYSE.31:End
SYSE.32 - Promote Verified Artifacts without Rebuilding
Status: Stable
Normativity: Guidance within the stated software-engineering use; examples are illustrative.
SYSE.32:1 - Problem frame
Use this pattern when a tested software package must cross an environment, storage service or provider boundary and someone needs to rely on it being the same qualified result. Start with the artifact that was exercised, the evidence needed at the destination, and the conditions under which that evidence remains applicable.
The first result is an artifact-specific basis for promotion, or an identity, evidence or trust condition that has not been met. Promotion here means making the selected artifact available for a next use without silently replacing it. Use SYSE.41 for installation and verification of the resulting runtime when that work is needed.
If an unchanged artifact is already available at the consumer with adequate identity and applicable evidence, use that result. If the missing result is a recoverable build, use SYSE.30. If it is application acceptance, obtain the relevant engineering feedback through SYSE.31. A storage transfer cannot supply either.
SYSE.32:2 - Problem
A version label or source revision can remain unchanged while the bytes change. A later stage may resolve different dependencies, embed another configuration or receive a substituted package. Tests of the earlier package then accompany a different result.
Authenticity is a separate question. A signature can be valid while the signer, source, build procedure or artifact is not the one the consumer expects. Conversely, a package can have the expected digest while the only evidence for that expectation came through an untrusted channel.
SYSE.32:3 - Forces
| Force | Practical tension |
|---|---|
| Reuse and local adaptation | Reusing tested bytes preserves their identity; qualify the destination configuration when the next use involves running the artifact there. |
| Convenience and evidence reach | Mutable names simplify selection but can hide substitution after a test or check. |
| Verification and trust | Cryptographic checks establish particular relationships; apply the trust and acceptance conditions that govern the consumer’s use. |
| Continuity and change | Old evidence can be reused for an unchanged subject, but a new artifact or material condition requires evidence applicable to that subject and its material conditions. |
SYSE.32:4 - Solution
SYSE.32:4.1 - Recover the artifact and the claim being carried
Identify the exact package, image or other deliverable that the prior activity produced and exercised. Use an identity appropriate to its form, such as a digest of the specified bytes, together with a retrievable immutable object where available. State which bytes the identity covers; a directory name or a mutable image tag is not that definition.
Recover the evidence needed for the next decision. Distinguish functional tests, build-input traceability, vulnerability or policy checks, and authenticity. For each result used, identify its subject and the conditions under which it applies. A test of source code does not automatically cover a separately built binary.
When evidence is absent or attributed only to an ambiguous label, return that gap. The team may re-establish evidence for the selected artifact or select the artifact already covered. Relabeling the new output with the old version is not a repair.
SYSE.32:4.2 - Separate the artifact from destination configuration
Keep destination-specific values in an independently identified configuration when the software supports that separation. Define the compatible artifact/configuration combination and protect secrets through the applicable authorized mechanism; do not publish them merely to make the build look reproducible.
If a destination requires a different compiled result, treat it as a different artifact and obtain its own evidence. Do not describe a rebuild as promotion of the previously tested bytes.
Preserve access to the selected artifact for the intended interval. A digest without obtainable bytes is not a usable delivery path. Establish whether the selected bytes will remain available and whether their distribution is permitted.
SYSE.32:4.3 - Transfer and verify the subject at the relying boundary
Determine where substitution can occur between the earlier check and the next use. Check the actual artifact delivered at the boundary where reliance is made, or establish a controlled chain that preserves both identity and the applicability of the earlier result. SYSE.28 helps place that control.
A failed transfer, incomplete download or digest mismatch prevents reliance on the earlier artifact evidence. Preserve the expected identity and inspect the actual result; do not repair the mismatch by silently changing the expectation to match the received package.
A matching digest establishes identity relative to the expected digest and its trust basis. It does not establish that the original artifact was correct, non-malicious or fit for this target.
SYSE.32:4.4 - Apply the authenticity Method when that claim is required
Use SLSA v1.2, Verifying artifacts with the consumer’s configured trust roots and expectations. Its operative verification includes the signature, the actual artifact’s correspondence to the provenance subject, the expected provenance type, and the recognized builder’s trust basis. Compare the claimed build with the expected builder, canonical source, build type and external parameters; a valid signature alone is insufficient.
Supply those expectations through the authorized trust arrangement. Unknown or unexpected parameters require the source Method’s explicit disposition, not silent acceptance. Use a verifier qualified for the deployed formats and trust system.
Distinguish missing provenance, invalid verification, an unexpected but correctly signed build, and unavailable verification. Each may require a different investigation or authorized exception decision, but none establishes the missing trust claim. If an authorized holder permits a bounded use without that assurance, retain the limitation and its scope.
SYSE.32:4.5 - Return a usable, bounded promotion basis
Use the selected artifact where it is already available to the consumer, or make it available there. Give the consumer the artifact’s identity, its applicable evidence, any separately specified configuration it requires, and any unresolved conditions. An existing delivery mechanism or record can supply these facts.
Use applicable results for the consequential substitution and missing-evidence cases, or exercise the cases that remain unresolved for the chosen mechanism. Establish that the consumer obtains the intended bytes and can distinguish failure from a completed verification. Reuse a result while its subject, required claim and material conditions remain the same.
If the next use requires deployment, give its operator the selected artifact, its verification results and the limits of any authorized exception for use with SYSE.41. For that deployment, establish the target’s effective configuration, dependencies and observed behavior. Use SYSE.14 when a separately authorized release decision is needed; artifact promotion does not widen exposure by itself.
SYSE.32:4.6 - Build a promotion mechanism whose evidence reaches actual use
Begin at the next consumer and work backward. Which object will it obtain, and who can change that selection between verification and use? Prefer selecting an immutable content identity for the consumption itself. Merely verifying a tag and then asking the runtime to fetch that mutable tag again leaves a substitution interval. Where direct immutable selection is unavailable, the delivery mechanism must retain and verify the selected object through the last mutable boundary; otherwise report the remaining identity gap.
Separate the package’s stable identity from its changing eligibility. A package can keep identical bytes while an expiring test condition, revoked builder trust, incompatible target or newly known defect changes whether it should be used. Keep evidence discoverable by artifact identity, with the scope and time of each result. A later policy decision can withdraw use without rewriting the artifact’s history. Conversely, a new policy label cannot turn an untested binary into the subject of an old passing test.
Choose where verification occurs from the substitution and trust question. Verification at publication into a registry can reject a bad producer result early. Consumer verification can additionally cover what arrives after storage or transit. A trusted controlled store may let several consumers reuse a verification result, provided the identity-preservation and trust conditions for that reuse hold. Reuse a sufficient result while its subject and material conditions remain applicable; repeat a check when it can resolve a remaining relevant question. Skipping the only check after an uncontrolled boundary loses assurance.
Multi-platform packaging makes the subject question concrete. An image index may select different platform-specific images. A test on one architecture supports the actual selected image and its conditions, not every image mentioned by the index. For a second target, establish which child artifact will run and obtain its applicable evidence. If a distribution system repackages files, determine whether the relied-on tests concern the retained payload, packaging behavior or the entire delivered bytes. Reuse only the results whose subjects and conditions survive that transformation.
Construct the failure path as part of the same mechanism. An absent artifact returns unavailable, a mismatch leaves the expected artifact identity unconfirmed, an unrecognized provenance returns an unresolved origin, and a failed required test returns an unqualified candidate. Each result should point to the contribution that can repair it. Retain the last permitted usable selection when that remains a valid option. Do not silently fall back to an older package whose permissions, compatibility or known defects have changed.
Suppose h1 passed on a supported Linux target and a release job creates a multi-platform bundle containing h1 and a newly compiled Windows package h3. The whole bundle is new even though h1’s bytes remain. The Linux consumer can reuse the applicable h1 behavior evidence after verifying its selection; the Windows consumer needs h3 evidence. A policy scanner rerun on the bundle might answer a new security question, but it cannot supply h3’s missing behavior check.
SYSE.32:5 - Archetypal Grounding
In a constructed ParcelWorks example, integrated source revision r17 produces artifact h1. Here h1 and h2 denote distinct artifact digests. The address-transformation tests exercised h1 under their stated conditions. A staging job later rebuilds r17 and obtains h2 because a build input changed.
The team wants to move the tested software to a candidate runtime. Its immediate question is whether the selected artifact carries the required evidence, not whether that runtime is already usable.
| Candidate situation | What is established | Next action |
|---|---|---|
| The consumer receives h1, and the required results concern h1 under applicable conditions. | Identity and those stated evidence relationships are preserved. | Return the promotion basis. To establish runtime usability, deploy and test the target separately. |
| The consumer receives h2 with h1’s test result. | The source label matches, but the tested artifact does not. | Select h1 or qualify h2 as a new candidate. |
| Provenance names h1 while the consumer receives h2. | Artifact/provenance correspondence fails. | Stop the dependent authenticity claim and inspect the mismatch. |
| Provenance is correctly signed but names an unexpected builder or source. | Signature validity does not meet the consumer’s expectation. | Resolve the unexpected origin through the configured trust decision. |
| The verifier is unavailable. | This attempt yields no verification result. | Use an applicable earlier result if the controlled chain covers this delivery. Otherwise restore verification or obtain an explicit bounded exception. |
For the normal branch, the consumer obtains the retained h1 bytes through the supported artifact store. The expected identity comes from the qualified build/evidence result, not merely from the same mutable label used for downloading. The consumer verifies the required identity and authenticity conditions. Destination configuration c2 remains separate and has not yet been exercised in a runtime.
The consumer has h1 with the named applicable evidence for the intended next use. To establish runtime usability, the operator uses SYSE.41 to install h1 with c2, observe the target state and perform the bounded deployment test. An application-form defect not covered by the earlier tests remains possible.
In the adverse branch, the staging system insists on rebuilding. The team can either change that delivery mechanism to consume retained h1 or explicitly qualify h2. It cannot retain the claim of unchanged-artifact promotion while taking the second route. If the old artifact is no longer retrievable, the first route is unavailable even though its digest and logs remain.
A second supplier signs a package with a valid key, but the supplied build uses an unapproved source location. Automatic acceptance based only on signature success would admit the wrong origin. The consumer rejects this build for its unapproved source despite the valid signature.
The test result concerns the artifact it exercised. Installation, target configuration, authenticity and release each need their own grounds.
SYSE.32:6 - Bias-Annotation
Teams can mistake familiar version names, a green pipeline or a prestigious builder label for evidence. Consumer trust expectations can also be so broad that verification accepts nearly anything the supplier signs. Inspect the consequential mismatch cases and the origin of the expectation, not only the success path.
SYSE.32:7 - Conformance Checklist
- The artifact identity covers the actual bytes or object form used by the consumer.
- Relied-on evidence concerns that artifact and remains applicable to the intended claim.
- Destination configuration is separate, or a changed artifact is openly treated as a new candidate.
- Transfer does not silently rebuild, substitute or relabel the result.
- Required authenticity uses configured trust and artifact/provenance correspondence, not signature success alone.
- Missing, invalid, unexpected and unavailable results remain distinguishable.
- The promotion basis does not claim actual runtime usability or release permission.
SYSE.32:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Rebuild the same source in every stage and reuse the first test result. | Carry the tested artifact, or qualify each different output on its own basis. |
| Compare two mutable labels. | Compare the delivered artifact against a qualified expected identity. |
| Accept any correctly signed provenance. | Apply the consumer’s expected origin, build and trust conditions as well. |
| Treat a missing verifier result as a successful control. | Expose the missing assurance and its dependent stop or authorized limitation. |
SYSE.32:9 - Consequences
The pattern can reduce unnoticed substitution and avoid unnecessary rebuilding between environments. It requires artifact retention, evidence attribution and maintained trust expectations. It can expose that an apparently convenient delivery mechanism changes the candidate at every stage; repairing that mechanism or qualifying each output costs work.
SYSE.32:10 - Rationale
Evidence travels through a valid subject-and-condition relationship, not through a shared label. Preserving the artifact makes that relationship easier to maintain. Separating destination configuration and actual deployment prevents a successful storage operation from standing in for an observed running System.
SYSE.32:11 - SoTA-Echoing
For artifact origin and integrity, adopt the consumer procedure in SLSA v1.2 Verifying artifacts, including configured trust and expectations. The procedure checks the delivered artifact against the provenance subject and the expected build, in addition to verifying the signature. Section 4.4 uses those checks for the origin claim. Application behavior requires its own evidence.
For promotion, adopt and adapt the same-artifact practice in DORA Deployment automation and the Continuous Delivery pipeline. Rebuilding per target can supply legitimately different outputs, but each then needs its own applicable evidence. Section 4.6 follows verification through the last mutable selection. For a multi-platform package, identify the selected member and reuse the evidence applicable to that member or to the package as a whole.
Compare consumer verification with a qualified shared verification boundary by substitution risk, trust, freshness and cost. Reopen reliance when packaging, retention, consumer selection, trust roots or applicable conditions change. An unchanged artifact identity does not imply unchanged permission to use it.
SYSE.32:12 - Relations
SYSE.30 supplies the build result and its inputs; SYSE.31 supplies engineering feedback. SYSE.13 identifies the artifact/configuration combination, and SYSE.28 places controls at the relying boundary. An operator can use the promotion result for deployment under SYSE.41. SYSE.4 governs evidence reliance, and SYSE.14 governs release.
SYSE.32:End
SYSE.33 - Provide Reconstructible Software Development and Test Environments
Status: Stable
Normativity: Guidance within the stated software-engineering use; examples are illustrative.
SYSE.33:1 - Problem frame
Use this pattern when development or a test works only on one prepared machine, fails because of shared mutable state, or cannot be repeated after an environment is lost. Start with the named task and the conditions it needs, including the resources, dependencies, data and access that must be available.
Return a usable environment with the conditions and limits established for its task, drawing on a suitable existing result or on a representative execution. If a required resource, permission or data source is unavailable, return the reconstructible description and identify the missing prerequisite.
Use an already adequate environment while it satisfies the task’s conditions. Exact identity of every incidental machine property is unnecessary unless the task depends on it. This pattern concerns software development and test conditions, not the complete qualification of a physical laboratory or production process.
SYSE.33:2 - Problem
A setup instruction can omit the package installed months ago, a manually altered service, a local credential or a shared fixture. Copying the machine then reproduces some of its history without explaining which conditions the task requires.
Infrastructure descriptions solve only part of the problem. A declared database is not its retained contents, a secret reference is not granted access, and a running machine is not evidence that the intended test can execute. Partial creation and configuration drift make these distinctions consequential.
SYSE.33:3 - Forces
| Force | Practical tension |
|---|---|
| Fidelity and cost | A task needs representative conditions, not an unnecessarily expensive copy of every production component. |
| Isolation and cooperation | Independent attempts avoid interference while some integration questions require controlled shared dependencies. |
| Reconstruction and state | Replaceable infrastructure can be recreated; persistent information needs a separate retention and restoration basis. |
| Automation and authority | A mechanism may know the desired state without permission or capacity to produce it. |
SYSE.33:4 - Solution
SYSE.33:4.1 - Define the environment for its task
Name the task, the software/configuration it exercises, its relevant dependencies, and the observations that will establish readiness for that use. Distinguish a local editing environment, a build worker, an integration test environment and a deployment target when their requirements differ.
Recover the current working conditions from a representative attempt. Identify the runtime and toolchain, resolved packages or images, network/service dependencies, resource needs, test data and access references. Separate relevant constraints from incidental machine history. Return any unresolved condition that could change what the test establishes.
Choose an equivalence criterion appropriate to the use. A behavior test may require the same database semantics and fixture content without requiring the same machine identifier. A timing claim may require additional resource and load conditions. An environment suitable for the first task is not automatically suitable for the second.
SYSE.33:4.2 - Separate description, provision and information
Keep the desired configuration under a retrievable versioned identity together with the supported creation or reconciliation procedure. Name actual resources independently so that their current state can be observed and their ownership established.
Distinguish replaceable resources, persistent state, disposable fixtures and credentials. Use synthetic data or data explicitly permitted for this task. Supply credentials through the authorized access mechanism; a reference can be retained without exposing the secret itself.
Decide which information may be discarded, which must be retained, and what a reconstruction is expected to restore. If continued use depends on preserved data, obtain the relevant recovery result through SYSE.34 or the qualified storage Method. Recreating an empty database does not satisfy a promise to restore its former contents.
SYSE.33:4.3 - Choose a creation and change mechanism
Compare a controlled push procedure with a pull-and-reconcile arrangement for the actual resources and operating conditions. A push procedure can be adequate for a bounded short-lived test environment. Continuous reconciliation may help where drift and continuing desired state are material. Neither choice removes the need to observe the resulting environment.
The OpenGitOps v1.0.0 principles call for versioned declarative desired state, automatic retrieval and continuing attempts to reconcile observed state. Use it where those conditions fit. Do not infer successful convergence from a committed declaration or an agent’s presence.
For either mechanism, define the operation’s bounded target, permitted changes and behavior after partial completion. Know how to identify resources already created by an attempt. Repeat a step only when its replay behavior is qualified; a repeated creation request must not silently create a second independently chargeable or stateful resource.
SYSE.33:4.4 - Provide isolation, lifetime and assistance
Give each independent attempt enough isolation to protect the meaning of its work. This may concern files, database rows, queues, network names or whole resources. Shared use is possible when the interference conditions are understood and controlled.
Set an appropriate lifetime and identify who is responsible for maintenance, assistance and decisions about retained information. A temporary environment must not disappear in the middle of promised work merely because a default expiry was reached. Conversely, indefinite retention of unused environments consumes resources and can preserve unnecessary access.
Limit cleanup to the resources and data that belong to this environment and that you are authorized to remove. If ownership or retained-data consequences are uncertain, return the uncertainty before deleting. A broad naming convention alone is not proof that every matching resource belongs to the current attempt.
SYSE.33:4.5 - Exercise clean creation, change and partial failure
Use applicable observations from a clean creation and representative task when available. Where evidence is missing, establish the required resources, effective configuration and reachable dependencies. Create the environment from the retained description if clean creation still needs to be demonstrated, and run the representative task if its result is missing. A successful provisioning job without the task result establishes less.
When the arrangement promises configuration change, change one relevant configuration and verify both the intended new condition and the continuing task. If a manual intervention was needed, either make its necessary effect part of the supported procedure or identify the still-manual dependency. Do not let a reconciler and an unrecorded repair repeatedly undo each other.
Use a qualified response to consequential partial creation, or exercise that failure to establish the response. A permitted retry can converge on the intended resources if its contract covers every remaining state, including concurrent changes and duplicate chargeable resources. Otherwise determine which resources were created before acting on them, or hold the affected step. Retain, reconcile or remove resources only under the applicable ownership, access and data conditions. State any remaining uncertainty about the interrupted attempt.
Give the recipient the description and procedure, lifetime and support arrangements. For a usable environment, identify the instance exercised, its task and the conditions it supports. For a blocked task, identify the missing prerequisite and what has been established so far. Existing configuration and operating records can carry these facts.
SYSE.33:4.6 - Choose a fidelity boundary and make reconstruction converge
Construct the environment from the question the task must answer. For each interaction that could change that answer, choose whether the task needs the actual dependency, a controlled substitute or no dependency at all. A substitute is adequate only for the behavior it represents. An in-memory data store can make a transformation test fast while omitting collation, isolation or persistence behavior that a migration test needs. Identify those omitted questions and send them to the environment that can exercise them; do not require production duplication for a test whose answer does not depend on it.
Develop supported environment profiles when several tasks share the same consequential conditions. Each profile states what its users may infer: for example local functional feedback, real-database integration, or representative performance. Generate these profiles from shared configuration where feasible, but retain their intentional differences. A single universal environment can become slow and expensive; uncontrolled differences between ad hoc environments can invalidate comparisons. The deciding question is which differences affect the receiving claim.
Treat reconstruction as convergence from a known starting condition. Establish how the mechanism identifies an existing resource, inspects its actual state and chooses create, change, leave, or stop. Declarative desired state is useful only when those operations are supported for the relevant resource and state. A tool’s state record can help associate resources with requests, but it is not the resource itself. If the record is lost while the resource survives, recover that association before creating another resource with the same business purpose.
Choose how to handle drift. Replacing a disposable worker is often simpler than understanding every manual change it accumulated. A shared stateful dependency may require an in-place repair that retains information and service continuity. Reconcile only the resources the mechanism is authorized and responsible for managing. When emergency work deliberately changes a managed value, either adopt the new intended value through the supported change path or restore the former value after mitigation; two competing controllers must not keep undoing one another. Continuing reconciliation adds value when ongoing drift matters; an observed one-shot construction is sufficient for many short-lived tests.
Information and access make this more than infrastructure recreation. A usable retained-data environment needs a recoverable data source, a restore/import procedure, compatible schema and a way to verify the required state. Credentials need valid scope and availability at use time. Provisioned capacity needs to last through the promised task. Keep the machine-reconstruction procedure when another prerequisite for the task is missing. Identify what it reconstructs and what still blocks the task.
In a changed ParcelWorks use, a fast local substitute treats text comparison differently from the production database. Keep it for pure split/join feedback whose contract is unaffected. Move the persistence comparison to a profile using the relevant database semantics, and include the differing text case. If the new purpose is a latency guarantee, neither profile is sufficient merely because it uses the right database version: reproduce or justify the relevant load and resource envelope through SYSE.40. The action changes with the test question, while independently useful environments remain usable.
SYSE.33:5 - Archetypal Grounding
In a constructed ParcelWorks example, address-service tests pass on a developer’s machine but fail on a shared worker. Two causes are suspected: an unrecorded runtime version and tests that overwrite the same address row.
The team defines a bounded integration-test environment t7. Its description identifies the application artifact, runtime image, compatible database version, required text-handling settings, synthetic address fixtures and authorized test-service access reference. Each independent attempt gets its own fixture namespace. Production data and production credentials are outside this environment’s permission.
For the normal construction, the provisioner creates the named resources and loads the synthetic fixtures. The test reads and changes an address containing a line separator, then checks the exact resulting display text. The team observes the effective runtime/database conditions rather than assuming that the description was applied. A fresh t8 can exercise the same task without copying the developer’s home directory or t7’s mutable fixture state.
The relevant equivalence is the qualified behavior and conditions, not equality of t7 and t8’s resource identifiers. A later performance claim would need a new qualification of capacity and load; this successful functional test does not supply it.
The team next changes the configured test-service endpoint from one authorized sandbox to another. The environment’s effective configuration and dependency test must reflect the new endpoint. If the desired file changes but the running process retains the old value, the change is not complete.
| Adverse observation | Correct interpretation | Bounded response |
|---|---|---|
| Creation times out; lookup finds the worker but no database. | The attempt partially took effect. | Reuse the identified worker and create the missing database only under qualified replay conditions. |
| Lookup itself is unavailable. | Resource state is unknown. | Restore observation or obtain assistance before another potentially duplicating creation. |
| The database exists but the fixture-loading permission is absent. | Infrastructure exists; the test environment is not usable for its named task. | Return the permission gap. |
| A declaration requests the new endpoint, but the process uses the old one. | Desired and actual state differ. | Repair the application/reconciliation step and re-exercise the dependency. |
| An old test database contains retained results needed by a user. | “Temporary” does not establish permission to discard those results. | Resolve their disposition before cleanup. |
For this short-lived use, the team selects a controlled creation procedure with explicit observation rather than installing a permanent reconciliation service. Another environment with continuing drift could reasonably make the other choice. The test’s result, failure recovery and data limits govern that decision.
The next user can reconstruct the environment and exercise the task without depending on the original machine’s hidden setup, or identify the missing prerequisite.
SYSE.33:6 - Bias-Annotation
A familiar workstation can become an accidental standard, while a production-like environment can be assumed necessary for every task. Inspect which conditions affect the question. Automation can also make provision failures less visible by reporting success before access, fixture loading or a dependent service has been exercised.
SYSE.33:7 - Conformance Checklist
- The environment is defined for a named task and an appropriate equivalence criterion.
- Desired configuration, actual resources, persistent information and access are distinguished.
- Hidden setup is removed from the supported creation path or exposed as a remaining dependency.
- Isolation and shared-state conditions preserve the meaning of independent attempts.
- Clean creation and consequential partial failure have usable responses; relevant configuration changes are qualified when that capability is promised.
- A real task result, not only provisioning completion, supports readiness for the stated use.
- Cleanup respects resource ownership, retained information and actual permission.
SYSE.33:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Clone the one working machine indefinitely. | Recover required conditions and exercise a clean construction. |
| Declare the environment in Git and report it ready. | Observe actual state and run the named task. |
| Treat infrastructure recreation as data restoration. | Obtain and exercise the separate information-recovery procedure. |
| Retry all creation after timeout. | Recover partial effects and repeat only qualified operations. |
SYSE.33:9 - Consequences
Reconstructible environments can reduce hidden dependencies and make failures easier to compare. They require maintained descriptions, controlled fixture/access arrangements and attention to partial effects. Isolation can increase cost; excessive fidelity can slow feedback without improving the question being tested.
SYSE.33:10 - Rationale
Readiness depends on conditions available for the task. Separate the desired configuration, resources observed in use and information that must survive reconstruction. Use the representative task result to establish usability, or identify the missing prerequisite. For a claim that prior information has been restored, compare the recovered state with what the task requires.
SYSE.33:11 - SoTA-Echoing
For repeatable environment provision, adapt the shared-procedure and configuration practices in DORA Deployment automation and the Continuous Delivery pipeline. Those approaches already treat environment variation as an engineering concern. Sections 4.5–4.6 derive the necessary fidelity from the task instead of requiring either a universal production clone or unrelated local setups.
Adopt and adapt OpenGitOps v1.0.0 when ongoing pull and reconciliation answer the drift problem. Compare it with a controlled one-shot or push procedure: continuing controllers can maintain desired state but add responsibility and need a clear ownership boundary. For either arrangement, establish that the data and access are usable, and observe whether the mechanism brings the environment to the state required for the promised task.
Reopen the profile or provision mechanism when provider behavior, dependency semantics, data permissions, expiry or the receiving task changes. Infrastructure reconstruction alone does not establish information restoration or performance under a new resource envelope.
SYSE.33:12 - Relations
SYSE.26 supplies the supported practitioner path, SYSE.30 supplies recoverable builds, and SYSE.31 consumes credible test conditions. SYSE.13 identifies configuration; SYSE.34 supplies persistent-data compatibility and recovery. If application deployment is needed, use the deployment method in SYSE.41. SYSE.29 governs migration or retirement of an environment path.
SYSE.33:End
SYSE.34 - Change Software and Persistent Data with a Recovery Boundary
Status: Stable
Normativity: Guidance within the stated software-and-data change use; examples are illustrative.
SYSE.34:1 - Problem frame
Use this pattern when a software change can alter stored information, its interpretation, or which old and new consumers can use it. Start with the values that must survive, the permitted writers and readers, and the point at which a previous executable would cease to understand the resulting state.
For an unchanged data contract, return the compatibility result the task requires. For a transition, return a bounded change-and-recovery procedure identifying compatible states, any required write or synchronization mechanism, validation, the limits of reversal and operation, and who can authorize the relevant change or remaining loss. If a required data fact, database-specific Method or permission is missing, identify that gap.
Do not undertake a migration merely because an executable changes. An unchanged data contract may need only a compatibility check. Conversely, a schema that looks unchanged can conceal a changed meaning. Where a migration is needed, choose a procedure supported by the storage system and an acceptable limit on service interruption.
SYSE.34:2 - Problem
“Add fields, copy data, switch readers” leaves important work unspecified. Old writers may continue changing the original representation after a copy. Two new authorities can disagree. A retried backfill may overwrite a legitimate later write with a cached earlier value.
Returning the old executable is not enough when the new software has removed information or written values the old software cannot interpret. Even a successful database restore can omit later business events or repeat external effects. Establish which state can be recovered and how to account for later events.
SYSE.34:3 - Forces
| Force | Practical tension |
|---|---|
| Continuing service and simple change | Coexistence reduces coordinated interruption but adds compatibility and synchronization work. |
| New meaning and reversibility | A richer representation can improve use while making an exact return impossible. |
| Bounded migration and concurrency | Small batches control load; concurrent writes still need an explicit ordering rule. |
| Technical recovery and acceptable loss | A restore mechanism can work while its time or information loss remains unacceptable to the people relying on the service. |
SYSE.34:4 - Solution
SYSE.34:4.1 - Recover meaning, consumers and authority
Describe the old and proposed values using the application’s meaning, including null, empty, ordering, precision and identity distinctions that matter. Define a correspondence and any inverse. Identify inputs that cannot be represented or carried back without loss; do not hide them behind a convenient conversion.
Find every relevant writer and reader, including jobs, imports, replicas, recovery tools and infrequent clients. Establish which database or storage edition, constraints, triggers, isolation and permission conditions the procedure will encounter. For an existing installation, check its effective configuration.
Obtain the required meaning and acceptance limits from their holders. The platform can support the mechanism without deciding which information may be lost or how long the business can operate without the service.
SYSE.34:4.2 - Choose a change strategy and its states
Compare a bounded coordinated interruption with compatible coexistence. A short interruption can be simpler and safer for a small service; coexistence is useful only when its compatibility can be maintained. For either strategy, qualify the recovery actions available from its intermediate states, including forward repair where it is needed.
For parallel change, add the needed representation without prematurely removing the old contract, qualify coexistence, migrate consumers, and contract only after the old use has been resolved. The historical Parallel Change Method supplies that strategy, not an engine-specific synchronization mechanism.
Identify the admissible readers and writers, the data changes and the available return at each intermediate state. For an interrupted step, qualify a continuation for the states the interruption can leave or identify the condition that requires it to stop.
SYSE.34:4.3 - Construct the write and backfill mechanism
Choose one authoritative representation during coexistence when that can satisfy the use, and derive the other views through a specified transactional or otherwise qualified mechanism. If independent writers are required, obtain an explicit conflict/order resolution Method; the instruction “write both” does not supply one.
Select a backfill rule that accounts for concurrent updates. Use the database’s qualified locking, current-row or version-check behavior so that an old observation cannot silently replace a later legitimate write. Bound work and resource effects, including locks, storage growth and effects on serving traffic.
Identify which data changes and progress records commit together and which dependent or external effects can occur separately. After a lost acknowledgement, use an existing confirmed result if it suffices. Replay is available when the operator is authorized and the qualified procedure preserves the required result for every still-possible outcome of the earlier attempt. Include partial execution, later legitimate writes and effects such as a payment, notification or physical movement. If a repeat could duplicate an effect or overwrite a newer value, establish the relevant state first or hold the affected operation. To attribute an effect to a particular attempt, use evidence tied to that attempt or report the attribution as unknown. Check the resulting state against the needs of the reader or operation that will use it.
SYSE.34:4.4 - Validate coexistence and the actual recovery claim
Derive the required histories from the selected transition and its supported uses. Exercise ordinary and boundary values, the writers/readers and concurrency that occur in that strategy, and consequential partial-failure or replay cases. Verify information correspondence as well as successful execution.
Do not activate a consumer that relies on a new representation before its required population and continuing-write invariant are established. If some data is intentionally outside the scope, ensure that the consumer can identify and handle that boundary.
Use evidence that the proposed return works from the states it must support, and test any consequential state for which that evidence is missing. Returning an executable, restoring information and reconciling later external events are different operations. Where restoration is promised, use a suitable database restore Method and backup or archive material. Establish the loss and time the restoration achieves against the authorized limits.
SYSE.34:4.5 - Execute within the boundary and separate contraction
When the change is to be carried out, execute the qualified increment under the required permission, with observations that show when to stop. A timeout or inconsistent state suspends dependent reliance; it does not justify blindly restarting the whole procedure.
Retain the old contract while the selected return depends on it. Removing old fields, accepting non-invertible new values or changing the write authority is a later state change with its own consumer and recovery question. Use SYSE.29 to arrange migration or continued support for remaining users, decide what must happen to retained state, and fulfil or obtain permission to end support commitments that outlive the technical transition.
For a planned change, return the qualified procedure, its conditions and limits. After execution, return the state actually achieved, its evidence and any remaining limit to the work that needs them. An engineered procedure is not evidence that it has already run, and a successful migration is not general release permission.
SYSE.34:4.6 - Derive the transition from compatibility in both time directions
Treat each intermediate state as a combination of running readers and writers, stored representations and in-flight work. Ask two different questions: can the next version use the state it will receive, and can the intended recovery version use the state the next version may create? Successful upgrade alone does not answer return. Include asynchronous messages and queued jobs: they can carry an older contract after its producer has stopped running.
Construct the transition backward from the desired new use. Identify the first action that requires new information, a new representation or a new interpretation. Before that action, establish every consumer capability needed for coexistence. One useful arrangement is to deploy readers that understand both forms while writers still emit the old form, then change writers after the dual-reading capability is present where needed. Another retains one old representation as authoritative and adapts new use to it, as in section 5. Both have costs: dual readers must remain correct for both meanings, while an adapter can limit which new values are expressible.
Compare those arrangements with a coordinated interruption. If all consumers can be stopped within an acceptable interval, a verified backup and a bounded offline transformation may need less mechanism than online coexistence. For that interruption, prevent late writes, define the population to transform, and decide how to recover from partial completion. A large table, strict availability need or independently upgraded clients can make that arrangement infeasible. Choose the strategy whose interruption, recovery and compatibility conditions the service can support.
Common mechanisms for a continuously updated dataset include transactional projection, change-stream replication and a bounded write fence. A transaction can update the authoritative value and its derived representation together. A qualified change stream can carry updates after a known snapshot position, with ordering, replay and gap detection that the consumer understands. A bounded write fence can create a stable interval for transformation and switch. Merely running a bulk copy while writers remain active supplies none of these guarantees. A change-stream design additionally needs a rule for duplicates, reordering, deletion and catching up before the new reader depends on it. Obtain the storage-specific implementation for the chosen mechanism.
Do not treat coexistence completion as permission to remove old support. Find what can still return old data: a scheduled job, retained message, offline client, dormant disaster-recovery image or a restore from an earlier backup. A contraction may require keeping a conversion on the restore path even after no live client uses it. Select a compatibility horizon from those supported uses and retention commitments. This horizon can differ from the time needed to observe a canary or to finish the ordinary application rollout.
For a claimed return, use an applicable qualification from the state of greatest relevant exposure, or test the return from that state. In a coexistence strategy, include old and new writes; include partial migration and lost acknowledgement where they can affect the return. Follow which representation each reader sees and which effect a repeated operation could duplicate. If return would discard legitimate later events, decide whether those events can be replayed or reconciled and who may accept the remaining loss. A backup can supply older data without supplying this reconciliation. Keep a useful forward-repair option when exact reversal is unavailable.
For an unlike case, a job-message format gains a new optional field. First release workers able to read both formats while producers continue sending the old one. Then change a bounded producer population and observe completion of whole jobs. Retiring old workers is insufficient evidence to remove old-format reading: queued or archived jobs may still contain it. Conversely, rolling producers back does not remove already emitted new-format messages. If old workers cannot read those messages, recovery must retain compatible workers, translate messages with qualified meaning, or drain the queue under an agreed restriction. This is the same compatibility reasoning with a different persistence mechanism; the PostgreSQL trigger in section 5 is not its implementation.
SYSE.34:5 - Archetypal Grounding
SYSE.34:5.1 - Define a reversible address correspondence
In a constructed ParcelWorks example, one non-partitioned PostgreSQL 18 table has an immutable row ID and a non-null Unicode text value D, stored as delivery_address. Old applications read and replace that whole value. The new form edits the first display line L separately from the remaining display text T. It does not infer postal structure.
Split D at its first line-feed character, LF. If no LF exists, set L = D and T = null. Otherwise L is the text before that LF and T is all text after it, including any further line feeds. Reconstruct D as L when T is null, or L + LF + T otherwise. L must contain no LF. Preserve every character; do not normalize whitespace.
| Old D | New L | New T | Reconstructed meaning |
|---|---|---|---|
| 12 Oak St | 12 Oak St | null | No line separator was present. |
| 12 Oak St followed by LF | 12 Oak St | empty text | The trailing separator is preserved. |
| 12 Oak St, LF, North, LF, Depot | 12 Oak St | North, LF, Depot | Remaining lines stay in the tail. |
| Empty text | empty text | null | Empty but non-null old text remains representable. |
This correspondence makes old and new text mutually recoverable within the stated domain. Postal parsing, normalization or an additional field that cannot be joined back is outside it and reopens the recovery decision.
SYSE.34:5.2 - Keep one write authority during coexistence
The selected authority remains D. Old applications write D directly. The new application’s adapter validates L/T, joins them into D and writes D; it does not independently write derived columns.
A row-level BEFORE INSERT OR UPDATE trigger computes L/T from NEW.delivery_address and returns the modified row. The PostgreSQL 18 trigger behavior supplies same-transaction execution and returned-row semantics. This construction excludes other address-writing triggers, replicated writers that bypass this trigger and external side effects of these row updates. If those premises are not established, this mechanism is not yet qualified.
The authorized operator briefly fences new address writes and drains in-flight writers while installing the schema and trigger. Ordinary application roles must not bypass the mechanism or independently write the derived columns. The PostgreSQL 18 privilege rules require inspection of table grants, inherited rights, ownership and superuser access; revoking a column privilege does not cancel a broad table grant. Resume old writers only after the actual configuration satisfies those conditions.
SYSE.34:5.3 - Backfill current rows rather than cached values
Backfill operates in bounded Read Committed transactions over stable ID ranges. For each named ID, it performs the equivalent of:
UPDATE address
SET delivery_address = delivery_address
WHERE id = the_named_id;
Here the_named_id is a bound value for the immutable row identity, not literal executable SQL. The trigger derives from the row being updated. The procedure never writes a D/L/T tuple cached by an earlier SELECT.
Under PostgreSQL 18 Read Committed, an updater waiting on another updater proceeds against the qualifying updated row after that transaction commits. The immutable ID keeps this case’s selection stable. This is not a general guarantee for arbitrary multi-row predicates.
Advance the batch position only after commit. An abort leaves the changes rolled back and the position unadvanced. If the acknowledgement is lost, the authorized operator can replay the range from current values whether the first transaction committed or aborted. This case excludes other update effects and preserves values written in the meantime. The replay establishes no history for the unacknowledged attempt.
| Constructed history | Result |
|---|---|
| Backfill derives A, then an old writer commits B. | The trigger derives B with that write; both representations describe B. |
| Old writer B holds the row while backfill waits. | Backfill acts on the committed current B and does not restore A. |
| Backfill commits A, acknowledgement is lost, old writer commits B, then the range is replayed. | Replay derives B from current D. |
| New writer submits L = PO Box 7 and T = North Depot. | The adapter writes their joined D; the trigger reconstructs the same pair. |
| A backfill transaction fails before commit. | Its derived-field changes are rolled back and the range remains unfinished. |
Enable L/T-dependent readers only after the scoped backfill and a correspondence check establish their required population. Continuing writes must preserve the same invariant. The table illustrates these histories. For deployment, use qualification results from exercises of the relevant schema, roles, concurrent operations and recovery arrangement. Exercise any case still missing.
SYSE.34:5.4 - State the return and its end
This release retains D, the trigger and both reader contracts. Subject to the application’s other compatibility conditions, the old executable can return while preserving legitimate address writes made through the new form. It is not necessary to undo those writes.
Contraction is separate. Before removing the old contract, fence and drain all old writers and compatibility adapters, select the new write authority and validate its consumers. Once a lossy transformation or incompatible new write has occurred, an old binary alone cannot restore the prior usable state.
Suppose a later contraction removes a trailing LF and retains neither the original D nor another copy of the absent-versus-empty tail distinction. Both 12 Oak St (T = null) and 12 Oak St followed by LF (T = empty text) then become 12 Oak St: whether a separator existed is lost, and returning the old executable cannot reconstruct it. An exact-return promise therefore requires retaining D or that lossless distinction before this change; accepting its removal requires the holder of that decision right to authorize a narrower promise. After the information is gone, exact restoration needs an independently retained, qualified recovery source and is unavailable without one. A qualified forward repair must meet the accepted result; it is not an inverse obtainable from the transformed value alone.
A distinct fallback uses PostgreSQL 18 point-in-time recovery. It needs a suitable base backup and continuous required WAL archive; a logical dump is not a substitute for that mechanism. Restore the whole cluster into an appropriately isolated recovery arrangement, inspect its state and evaluate achieved loss/time. Choosing an earlier recovery point does not reconcile all later business events.
If an archive gap prevents that fallback, its recovery claim stops. A qualified forward repair may still be possible, and an independent non-data-changing artifact test can continue.
The team can explain which state remains usable after each change, why concurrent writes are preserved, and where “return to the old version” ceases to mean recovery.
SYSE.34:6 - Bias-Annotation
Success on a static sample can hide concurrency, inherited privileges and rare representations. Teams can also prefer zero downtime without pricing the complexity of coexistence. Inspect real writers and adverse histories, and compare a bounded interruption with a more elaborate online construction.
SYSE.34:7 - Conformance Checklist
- Old/new meaning and information-loss boundaries are explicit.
- All relevant writers/readers and engine-specific conditions are accounted for.
- Where old and new representations coexist, the procedure identifies the write authority or authorities and how updates are synchronized. With independent writers, specify how their conflicts and order are resolved.
- Where backfill is used, it cannot overwrite a later legitimate write with a stale observation.
- Consequential partial failure, lost acknowledgement and any allowed replay have qualified behavior.
- A new reader starts only with the population and continuing invariant it needs.
- Each promised executable return, data restore or later-event reconciliation is qualified for its own result.
- Contraction and permission to accept loss are not inferred from successful deployment.
SYSE.34:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Read old values, copy them later, then prefer any non-null new field. | Account for intervening writes and establish a maintained correspondence before new-reader reliance. |
| Make both old and new fields authoritative. | Choose one authority or supply a qualified conflict-resolution Method. |
| Repeat the entire migration after a timeout. | Apply the qualified replay rule to the affected step. If safe continuation depends on an unresolved effect, establish that effect first or hold the step. |
| Reinstall an old binary after irreversible data loss. | Use a qualified forward repair or information recovery with accepted limits. |
SYSE.34:9 - Consequences
The pattern makes the conditions and limits of migration and recovery claims explicit. Compatible coexistence can reduce interruption while adding temporary schema, adapter and maintenance cost. Some changes remain irreversible or require a planned outage; exposing that fact is more useful than promising a rollback that cannot preserve the information.
SYSE.34:10 - Rationale
Compatibility is a relation between actual representations and consumers over time. A transition must maintain that relation under the writers that remain active. Whether recovery is possible therefore depends on the resulting state and available permitted operations, not merely on the availability of an earlier executable.
SYSE.34:11 - SoTA-Echoing
For application/data transitions, adopt and adapt DORA Database change management and the foundational 2014 Parallel Change treatment. Compatible intermediate states and versioned migration are established professional Methods. A coordinated interruption remains a serious alternative when its cost is acceptable and online coexistence is harder to support.
The DORA address example is insufficient as a concurrency implementation when old writers remain active: generic dual writing and null fallback do not determine authority, atomicity or exact text correspondence. Sections 5.2–5.3 construct one bounded alternative from PostgreSQL 18’s trigger, privilege and isolation contracts. Section 4.6 derives other arrangements from actual readers, writers, in-flight work and recovery horizons. A change stream or write fence needs its own implementation qualification; the trigger does not generalize to every store.
Reopen the transition when representation meaning, writer population, engine behavior, side effects or supported return/restore paths change. For each promised return, establish which state can be recovered by the available procedure. Check compatibility for an executable return, the retained source for data restoration, and how later events will be reconciled when they matter.
SYSE.34:12 - Relations
SYSE.13 identifies software and data configurations. SYSE.27 and SYSE.29 address consumer compatibility and migration; SYSE.33 supplies controlled test conditions. SYSE.41 consumes the data-change result during deployment, SYSE.35 respects it during exposure recovery, and SYSE.38 uses it when an incident has partial data effects. Use SYSE.14 when a separately authorized release decision is needed.
SYSE.34:End
SYSE.35 - Control Software Release Exposure with User-Relevant Signals
Status: Stable
Normativity: Guidance within the stated software-release use; examples are illustrative.
SYSE.35:1 - Problem frame
Use this pattern when a software change can reach a bounded population before wider use, and observations from that population can change the next exposure decision. Start with the deployed candidate and its deployment-test result from SYSE.41 or a current equivalent, not merely a package in storage.
The first result is an exposure-and-evaluation arrangement: the affected task, comparison, bounded population, observations, decision conditions and qualified recovery action. Once observations exist, it yields a bounded proceed, stop or inconclusive result. The release holder decides whether to authorize exposure and whether the evidence is sufficient for the proposed next step.
Do not require a canary for every change. A useful evaluation may be impossible when the relevant event is indivisible, too rare within the allowed interval, or has an irreversible consequence that cannot be bounded. Use a different qualified risk-reduction Method in those cases. A deployment whose actual state is still unknown returns to SYSE.41.
SYSE.35:2 - Problem
A small deployment can still produce a large consequence through shared state or dependencies. A quiet dashboard can also reflect the wrong user task, absent instrumentation or too few relevant observations rather than a good change.
A comparison is misleading when candidate and control differ in workload, configuration or observation. Successful platform delivery does not demonstrate successful use of the delivered application. Exposure control must therefore connect the runtime identity to the user task being assessed and the action available on each outcome.
SYSE.35:3 - Forces
| Force | Practical tension |
|---|---|
| Limited impact and informative observation | A tiny population reduces exposure but may never exercise the changed behavior. |
| Comparability and ordinary use | Controlled comparisons improve attribution while users, workloads and shared dependencies continue changing. |
| Prompt response and causal certainty | A credible harmful effect may justify reducing exposure before the full cause is known. |
| Technical reversibility and retained effects | Routing can change quickly while data writes, messages or other consequences persist. |
SYSE.35:4 - Solution
SYSE.35:4.1 - Recover the candidate, task and permission
Identify the observed artifact/configuration and the interval for which its deployment result holds. Confirm the dependencies and data state required for the intended exposure. For a claim about a user task, use deployment-test or other evidence that covers the task itself.
Name the service and users whose experience is at issue. Apply SYSE.36 to that subject and task population. A platform deployment-path objective and an application-use objective are separate even if the same dashboard displays both.
Obtain the relevant behavior meaning, reliability objective where applicable and exposure limit. Identify who can authorize the selected exposure. Carry it out under an applicable authorization. If only isolated testing is permitted, construct the wider arrangement without executing wider exposure.
SYSE.35:4.2 - Choose the comparison and exposure unit
Choose a population or unit that can receive the candidate without invalidating the comparison: for example, a supported user cohort or a bounded set of work items. Identify the comparison basis and explain why its observations answer the same outcome question. For concurrent exposure, identify the control population.
Account for task mix, resource conditions, data, time variation and other changes. Where requests from one task can cross candidate and control, preserve enough attribution to understand the result. A shared database or dependency can transmit the candidate’s effects to the control; check whether the separation needed for this comparison holds.
Compare bounded exposure with alternatives such as a controlled representative exercise, independent feature activation through a qualified feature-flag or experiment configuration, or a planned cutover. Choose an arrangement that can both limit consequence and supply useful evidence. Increase exposure only when the additional information justifies the consequence within the authorized limit.
SYSE.35:4.3 - Define observations and three outcomes
Select observations that can reveal the consequential failures of the changed user task. Include correctness, delay and missing results as the task requires. Infrastructure indicators can help explain effects but do not automatically define user success.
Set the evaluation interval, observation delay and adequate-evidence conditions before judging the candidate. Use an appropriate statistical or domain Method when a quantitative inference requires it. A handful of successful attempts cannot establish a rare-failure probability from the reported percentage alone.
| Outcome | Meaning and dependent action |
|---|---|
| Proceed basis | The stated evidence conditions and acceptance comparison are met for this population and interval; the authorized next exposure may be considered. |
| Stop basis | A stated unacceptable effect or violated operating condition has been observed; apply the qualified restriction or recovery. |
| Inconclusive | Required observations, attribution or discrimination are insufficient; do not convert absence of a failure signal into a pass. |
An inconclusive result can lead to a longer permitted observation, a better measurement, a different representative exercise or abandonment of this exposure strategy. Each changes an identified missing condition; repeatedly waiting on irrelevant traffic does not.
SYSE.35:4.4 - Qualify the stopping action
Determine what reducing exposure does. It may stop new task assignments, drain current tasks, change routing or disable a qualified feature. Observe the resulting state; a sent command is not proof that the candidate is no longer affecting users.
Use SYSE.34 for retained data and interface conditions. Returning traffic to an old executable is allowed only while that executable remains compatible with the actual state. Identify effects that persist after routing changes and who can resolve them.
Make the response available to the person or mechanism authorized to stop exposure. If the proposed stop requires unavailable access, an unsupported old runtime or a data restoration that has not been qualified, repair that premise before relying on bounded recovery.
SYSE.35:4.5 - Observe, act and retain the limits
During the permitted exposure, check that observations concern the named candidate and task and the comparison basis in use. Respond to credible harm without waiting for a complete causal account when the qualified stop is justified. Continue diagnosis through SYSE.38 as needed.
After reducing exposure, verify the user result and residual effects. After an adequate favorable result, return only the scope the evidence supports. Wider use, a different workload or changed state may need another decision.
Retain the useful observations and any unresolved question in the working release account.
SYSE.35:4.6 - Construct an exposure sequence that can answer the release question
Start with the consequential difference introduced by the candidate. A changed form needs attempts using that form and its meaningful data variants; a changed scheduler needs enough whole jobs to expose assignment and completion behavior. Choose a unit over which the difference and its effects remain interpretable. For a stateless read it may be a request. For a multistep edit it may be a session or user cohort. For a job that runs for hours it may be a complete job and its output. Randomly distributing individual requests can spoil a comparison when one task then crosses incompatible versions or shares mutable intermediate state.
Choose the comparison for the available system. Concurrent candidate and control populations can reduce time-related differences, but only if task assignment, workload and dependencies make their results comparable. Before/after observation can be useful when concurrent operation is impossible; account for intervening demand and environmental changes and retain its weaker attribution where those cannot be controlled. An isolated replay can exercise a rare harmful case without live exposure, provided recorded inputs and simulated dependencies represent the behavior being judged.
Separate absolute protection from relative comparison. If candidate and control both fail because a shared database is saturated, their similar error rates are not a healthy result. Keep an absolute task or service limit that can stop the exposure even when relative comparison is neutral. Conversely, a control’s unrelated failure should not automatically condemn the candidate: investigate whether the compared populations and conditions still answer the chosen question. A release observation can justify protecting users before it establishes which component caused the harm.
Choose successive exposure bounds from the information and consequence at each step. A first step can establish installation and basic behavior; another can cover ordinary task variants; another can reveal sustained resource or dependency effects. Reuse applicable observations rather than replaying every earlier exercise, but obtain the evidence introduced by the wider scope. The fraction of traffic alone is insufficient: ten percent of an infrequent task can yield no useful observation, while one percent of a high-impact write stream can already create extensive persistent effects.
Select the duration from task completion, observation delay and the failure mechanism. A latency comparison can stabilize sooner than a memory leak, cache invalidation, scheduled job or end-of-day reconciliation. Exclude or model warm-up behavior deliberately instead of letting it silently decide the result. Make sure the metric window contains candidate observations from the intended interval; a long aggregate dominated by the previous version can hide a new failure. For a rare-failure probability, obtain a suitable statistical design rather than choosing a convenient count of successes. For a known unacceptable semantic defect, one demonstrated occurrence may already be decisive under the acceptance rule.
At each step, decide what a favorable result permits next and which condition requires a pause, stop or return. Make widening an explicit operation with the required authority. A fully automated sequence is useful when these predicates and actions are qualified. A human decision is useful when the next step needs interpretation; supply the evidence and alternatives rather than a color asking for reflex approval. If stopping cannot bound the relevant harm, replace the exposure strategy before running it.
For ParcelWorks, a session-bound cohort keeps a complete edit on one form version. The first permitted exercise includes absent, empty and non-empty address tails, because those distinguish the changed behavior; general browsing cannot substitute for them. After those cases pass, a later step can study ordinary edit load and response time. For an asynchronous importer, use whole import batches and compare validated output after completion instead. In that case a five-minute window containing only job starts cannot establish successful import, even if every start request returned promptly. The same construction changes its unit and duration while retaining the result question.
SYSE.35:5 - Archetypal Grounding
In a constructed ParcelWorks case, SYSE.41 has established a candidate application h2/c2 on two isolated instances, with a successful bounded deployment test. The old h1/c1 pool serves ordinary traffic. The data construction from SYSE.34 retains the old address contract, so a qualified return remains possible. A release holder permits a limited application exposure with a defined stop.
The changed task is viewing, editing and saving an address. The application owner supplies its meaning: the new form must display the first line and remaining text correctly and preserve the submitted value, including an empty versus absent tail. The common outcome comparison is correct address use under each form’s supported contract, not identical screen layout.
The arrangement identifies the candidate/control configuration for each relevant attempt. A version/snapshot or controlled observation distinguishes later legitimate edits from corruption. The exposure remains bounded until the agreed behavioral and observation conditions are met; an HTTP response code alone cannot establish the address result.
In the first constructed interval, nine general application requests reach h2/c2, but none uses the changed address form. The relevant task has no observations. At the same time, the platform has completed ten supported deployment requests successfully within its own agreed response bound. Those are good platform observations, not evidence about the application form.
| Observation | Exposure conclusion |
|---|---|
| Nine candidate requests, none exercising the changed form. | Inconclusive for the form change; relevant evidence is missing. |
| Ten successful platform deployment tasks. | No change to the application conclusion; this is another measured subject. |
| A permitted form attempt hides a non-empty tail, although the backend stores it correctly. | The application acceptance condition fails; stop or reduce exposure within the qualified boundary. |
| Candidate/control identity is absent from the form observation. | Attribution is missing; do not infer a favorable comparison. |
| The required representative cases succeed under the agreed comparison and observation conditions. | A bounded favorable basis exists for the named next decision, not a universal correctness claim. |
In the adverse form attempt, the candidate was deployed as requested. The defect is therefore not evidence that the platform failed to install h2/c2. The authorized response keeps or restores ordinary routing to the qualified old pool, handles in-flight work according to the selected rule and verifies that the old form still reads and preserves the current data. Diagnosis of the hidden tail can then proceed without continuing unnecessary exposure.
If the state had crossed SYSE.34’s incompatible boundary, that routing return would no longer be a sufficient recovery. The release holder would need the qualified forward repair or restoration result rather than a misleading “rollback complete” message.
The Argo Rollouts analysis model provides a concrete implementation comparison: an inconclusive analysis can pause progression for a decision. Its status does not choose the right metric or guarantee that missing observations were configured correctly.
Insufficient evidence can justify holding exposure at its present limit while the application result remains open.
SYSE.35:6 - Bias-Annotation
Release pressure can turn every neutral signal into permission to proceed. A convenient cohort can also avoid the very behavior being changed. Inspect relevance, missing observations and shared effects, and preserve a legitimate inconclusive outcome even when the planned release interval is ending.
SYSE.35:7 - Conformance Checklist
- The deployed candidate and its bounded deployment result are available.
- Subject, users, task and acceptance meaning are distinct from platform-delivery success.
- The comparison uses relevant, attributable observations sufficient for the conclusion claimed, or reports which observations or attribution are missing and returns an inconclusive result.
- Proceed, stop and inconclusive conditions have different consequences.
- The stop/recovery action is authorized and compatible with actual retained state.
- Wider exposure is not inferred from a quiet dashboard or a controller’s success label.
- Residual effects and restored user behavior are observed after stopping.
SYSE.35:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Count all application traffic as evidence for a rare changed task. | Observe the relevant eligible attempts or return insufficient evidence. |
| Treat platform deployment success as application acceptance. | Measure the application task under its own meaning and configuration. |
| Stop routing and assume every effect vanished. | Account for in-flight work, retained data and shared dependencies. |
| Automatically accept missing or empty metric results. | Configure and exercise the intended missing-evidence disposition. |
SYSE.35:9 - Consequences
Bounded exposure can reveal defects while limiting affected use. It costs comparison design, observation and recovery capability, and it can delay a decision when relevant traffic is sparse. When the result is inconclusive, identify the missing evidence and decide whether to obtain it through another observation, use a different exercise or stop pursuing this exposure.
SYSE.35:10 - Rationale
Exposure is useful as an engineering move only when it produces relevant discrimination and supports a qualified response. Runtime identity, task meaning and recovery conditions connect the experiment to actual consequences. A comparison alone does not create permission to impose those consequences.
SYSE.35:11 - SoTA-Echoing
For bounded release evaluation, adopt and adapt the 2018 SRE Canarying Releases treatment. It explains candidate/control selection, contamination, timing, absolute protection and long-running work. Section 4.6 applies those considerations to choosing a task unit and a sequence whose observations can decide the next exposure.
Adopt an explicit inconclusive outcome, concretely supported by Argo Rollouts analysis. The tool can enact a pause but cannot establish task meaning or adequate evidence. Compare concurrent canarying with replay, before/after observation and planned cutover according to representativeness, shared effects, consequence and available stopping action.
Reopen the arrangement when the changed behavior, workload, telemetry, data compatibility or recovery permission changes. A favorable smaller exposure supports the comparison for the observed population and interval. Wider use depends on its own conditions and permission.
SYSE.35:12 - Relations
SYSE.41 supplies the observed deployment; SYSE.36 supplies matching task measurement and objectives. SYSE.34 bounds stateful return, SYSE.38 supports diagnosis/restoration, and SYSE.40 addresses overload that can contaminate exposure. SYSE.4 governs evidence reliance and SYSE.14 the release decision.
SYSE.35:End
SYSE.36 - Measure User Tasks and Set Software-Service Reliability Objectives
Status: Stable
Normativity: Guidance within the stated software-service measurement use; examples are illustrative.
SYSE.36:1 - Problem frame
Use this pattern when a software service looks healthy in its infrastructure dashboard while users cannot complete their tasks, or when a reliability target exists without an agreed meaning or response. Start with one named service, its users, configuration and task class, then define what a good result means to those users.
Use the task definition and evidence for the service in question. If both a platform service for developers and an application for its users are being assessed, establish each measurement separately. The first result is a task-level measurement definition and exercised instrumentation, or a measurement gap. A service-level objective, SLO, additionally needs an agreed target, time/event basis and policy for acting on the resulting error budget.
If adequate measurement and an agreed objective already exist, use them for the relevant decision. A one-off correctness check need not become a permanent SLO. This pattern does not define the application’s business meaning, certify every use of the service or turn reliability into a complete measure of user value.
SYSE.36:2 - Problem
A backend counts only the requests it receives. Failures at the user entry can disappear from its denominator. An HTTP success response can accompany the wrong application result. Metrics collected without version or task attribution can mix unlike uses and conceal the changed behavior.
A displayed target has a different defect when nobody agrees what it protects or what to do when it is missed. The measurement then becomes a reporting number, while engineering priorities remain disconnected from user experience.
SYSE.36:3 - Forces
| Force | Practical tension |
|---|---|
| User meaning and affordable observation | The important result may be harder to observe than an existing server counter. |
| Comparability and diversity | An aggregate aids decisions while task classes, users and configurations can have different needs. |
| Reliability and other value | A more reliable service can still be hard to use, slow to improve or not worth providing. |
| Detail and responsible data use | Diagnosis benefits from context while privacy, access, retention and instrumentation cost constrain collection. |
SYSE.36:4 - Solution
SYSE.36:4.1 - Define the subject and the good task result
Name the service, user population, supported task class and relevant configuration. Recover the task’s acceptance meaning from the responsible product or practice owner. A platform team can implement observation without acquiring authority to redefine application behavior.
Define an eligible attempt, its beginning and end, and the condition for a good result. Include timing, correctness, freshness or other properties only as the task needs them. Separate unsupported or invalid requests under a stated rule; do not recategorize a valid but failed attempt merely to improve the denominator.
Distinguish a user attempt from internal retries or backend jobs. Several internal operations can serve one attempt, and a failed attempt may produce no backend job. Decide how cancellation, abandonment and incomplete observation affect the stated question rather than excluding them silently.
SYSE.36:4.2 - Separate measurement meaning from implementation
Write the service-level indicator, SLI, specification in terms of the user outcome before choosing a counter or query. Then identify how the available instrumentation could observe that specification and where it would miss or distort it.
For an event-ratio indicator, define both good events and eligible events over the same population and interval. For a time-based indicator, define the evaluated periods and their conditions. Do not combine request counts, user minutes and task outcomes in one ratio without a justified measurement Method.
If the important semantic result is not observable, return that gap. An easier proxy can be reported under its own narrower meaning, but it cannot silently replace the required result. A dependency-health result provides evidence about the conditions checked on that dependency; an HTTP success status shows that the server reported success.
SYSE.36:4.3 - Construct and exercise the observation path
Place observation where material entry, execution and result failures can be detected. Connect the relevant metrics, logs or traces to the right subject and configuration using sufficient correlation. The OpenTelemetry signals model supplies interoperable signal categories for observing the task. Determine success from the user’s required outcome.
Use applicable observations of the instrumentation on a known good task, a meaningful failure and a loss of observation. Exercise any case not yet covered, including a pre-backend failure when the service path admits it. Check delay, duplication, sampling and aggregation effects on the definition. Sampled diagnostic traces do not by themselves establish a complete event denominator.
Keep missing data visible. If eligible attempts are known but their results are not, retain that distinction. If the entry population is unobserved, even the denominator may be unknown. Apply an agreed conservative policy where needed, but distinguish that policy disposition from an observed service failure.
Use the authorized data-handling arrangement. Collect only what the question requires, and avoid exposing secrets or unrestricted personal task content. A correctness check may run within an authorized application boundary and emit a bounded result; it need not export the input values.
SYSE.36:4.4 - Agree an objective and its use
When a new or revised objective is needed, use observed task difficulty, user needs, failure consequences and feasible provision to propose a target and interval. Current performance can inform a starting proposal but does not determine what users should accept. Avoid importing another service’s percentage or assuming that a more demanding number is always a better choice.
Agree a new or revised target with those who depend on or provide the service, including whoever can make the relevant trade-offs. Define the error budget in the same event/time basis and the actions that follow material consumption or exhaustion. Those actions need authority and capacity; a dashboard configuration does not create either.
A policy can prioritize repair, restrict a class of changes or require a decision before further risk. Choose its scope for the task and consequences. The budget is not permission to cause any kind of harm up to a numerical allowance, and it does not cancel independent constraints.
If agreement, meaningful timing or adequate observation is missing, return a proposal and its unresolved condition rather than claiming an operative SLO.
SYSE.36:4.5 - Interpret and improve without changing the subject
Report the observed value or supported bounds with the subject, population, interval and coverage limits. When the required outcome cannot be observed, report the measurement gap. Distinguish measured reliability, proposed or agreed target, delivery speed, user satisfaction and improvement value.
Give platform-task evidence to those making the corresponding provider or practitioner decision. They can use SYSE.25 when choosing a platform improvement. Send application evidence to its application and exposure owners. For exposure evaluation or an alert, use observations for the service and population in question. Use its objective when the decision depends on one.
Compare reported reliability with the difficulties users encounter. Revise the definition or implementation when it misses consequential failures; retain enough continuity to explain why old and new values differ. Do not improve the apparent history by quietly changing who or what is counted.
SYSE.36:4.6 - Build a reliability model and an executable policy
Begin with the tasks whose failure changes the user’s ability to obtain the service’s result. Follow each task from its entry point through dependencies to the result the user receives. Choose an initial set of indicators that distinguishes those failures. Availability and latency may be enough for a simple lookup; a data pipeline can additionally need freshness and correctness. More indicators are useful only when they reveal a different consequential failure or support a different decision. A convenient infrastructure metric can remain a diagnostic signal without becoming another user objective.
Choose aggregation from the tolerated loss. A global ratio can hide repeated failure for a small supported population. Separate that population when its consequences or required response differ, or show a relevant distribution alongside the aggregate. Conversely, multiplying objectives for every internal component can make the policy impossible to interpret. Locate the task outcome first, then use component signals to investigate why it changed. When several properties jointly define a good attempt, count that attempt once according to the joint predicate; summing separate failure counts can double-count the same failed task.
Where outcomes are partly unknown but entry is complete, report the possible range before adopting a policy disposition. Let N > 0 be the number of eligible attempts, with G known good, B known bad and U unknown results. If those outcome categories are disjoint and G + B + U = N, the actual good fraction lies between G/N and (G + U)/N. With no eligible attempts, an event ratio for the interval is undefined. The range is an observation bound, not a probability estimate. For 1,000 entries with 985 good, 5 bad and 10 unknown, the range is 98.5% to 99.5%. A 99% objective is not decided by those observations. A conservative operational policy can act on the lower bound; it must not describe all ten unknown outcomes as observed failures. If entry itself is missing, this calculation no longer bounds the whole population.
Develop the objective and policy together. Estimate what loss users can tolerate over a meaningful interval, what the provider can supply, and what changes are available when the budget is at risk. Compare a feasible initial objective with a more demanding one by the additional engineering, delay and cost it requires. An aspirational target can guide improvement, but it is not an agreed current service promise merely because it appears on a dashboard. For a new service, use a provisional definition and observed trials to negotiate that promise; retain separately any minimum obligation that already applies.
Make the policy executable through recognizable states. Under ordinary conditions, continue permitted delivery while observing the agreed result. Under material rapid loss, use the available mitigation and response arrangement. Under exhausted budget or repeated misses, direct the relevant capacity to reliability work or restrict the changes that add the implicated risk. State how an exception is decided and what evidence permits ordinary operation to resume. Do not automatically ban a repair because it is a change, or freeze an unaffected platform because another application’s budget is exhausted. The policy’s holder needs authority over the resources and decisions it names.
Exercise that policy on a kind of decision the service faces before treating it as operational. If the last incident had exhausted the proposed budget, which work would have changed, who could have changed it, and would that action have protected the task? If no action follows, either the objective is informational or its decision arrangement is incomplete. If every harmless variation triggers expensive intervention, reconsider the definition, target or response with the affected users; do not quietly hide the observations. Then revise from actual incidents, complaints and changes in supported use.
An unlike pipeline example makes the adaptation clear. A nightly accounting export can be available all day while producing yesterday’s incomplete result. Define the eligible run, source cutoff, completeness and due time, and observe the delivered output against them. The provider’s HTTP availability remains useful for diagnosis, but adding more successful HTTP calls does not improve the export’s reliability. If the task changes to an interactive query, build a new timing and event definition instead of carrying over a daily-run percentage.
SYSE.36:5 - Archetypal Grounding
SYSE.36:5.1 - Measure two different software services
In a constructed ParcelWorks example, the platform offers a supported deployment path to developers. It also deploys an address application used by parcel operators. The two services can fail independently.
For the platform, eligible means one authorized request for an already verified artifact/configuration in a supported class. Good means that the requested runtime identity and bounded deployment-test result reach the requester within the response limit agreed for that class. A failure before any backend job exists is still a failed eligible attempt. An unsupported language request remains a separately classified boundary, not a successful deployment.
For the application h2/c2, eligible means an authorized attempt to view, edit and save an address through the changed form. The application owner requires correct first-line/tail display and preservation of the submitted text under the correspondence in SYSE.34, within the application’s own agreed response bound. An empty tail is different from an absent tail.
| Measurement question | Needed observation | Gap that must remain visible |
|---|---|---|
| Did the platform deliver the requested runtime? | User-entry attempt, actual artifact/configuration, bounded test and elapsed response. | No entry observation, unknown target state or missing test result prevents the good claim. |
| Did the new form display the intended address? | Matching form/configuration and its relevant displayed components. | Backend storage success does not show what the user saw. |
| Did the acknowledged save preserve the submitted text? | The value/identity correspondence for that save under a qualified version, snapshot or controlled test. | A later legitimate edit must not be mistaken for corruption; an HTTP 200 is insufficient. |
| May either result be used for a timeliness objective? | The subject’s own agreed response bound and a qualified elapsed-time observation. | No agreed bound means timeliness is not yet qualified. |
The team constructs instrumentation for these definitions and exercises good, failed and missing-observation cases. If it cannot yet observe the form correspondence, it can report backend availability separately, but the application-success measurement remains incomplete.
In one constructed controlled test, entry A17 binds the attempt to h2/c2 and starts its response clock. Inside the authorized application/test boundary, the check verifies the displayed L = 12 Oak St, T = empty text and the acknowledged saved version’s exact correspondence to the submitted text ending in LF; later row changes are not substituted for that version. It returns A17, h2/c2, the bounded correctness result and elapsed time, without exporting address text. A qualified internal read retry against that version retains A17: two read calls and duplicate delivery of the same result do not become additional user attempts. If correctness and the agreed response bound both hold, A17 is one good eligible attempt. An entry A18 with no correlated completion is still one eligible attempt, but its outcome remains unknown, not success or an observed failure; any conservative policy disposition is labelled separately.
Suppose ten observed platform deployment attempts complete correctly within their own agreed response bound. Meanwhile, nine general application requests reach h2/c2 but none uses the changed form. The platform observation is favorable for those ten attempts. The application-form observation remains absent, so it does not supply a favorable exposure result.
In another constructed attempt, h2/c2 is installed correctly but its form hides a non-empty tail. The application condition fails while platform delivery remains correct. Conversely, unavailable CI workers can prevent new developer tasks while the old deployed application continues serving parcel operators successfully. Direct each failure to those responsible for its service. Account for it against that service’s budget when the eligible task and agreed objective apply.
These are invented observations for explaining the Method, not measured service results or proof of production instrumentation.
SYSE.36:5.2 - Turn a definition into a decision, not just a percentage
For a separate arithmetic illustration, suppose the responsible users/providers agree that at least 99% of a named service’s eligible tasks must satisfy its complete good-result definition over a specified 30-day window. They also agree how budget exhaustion changes the priority of reliability work. The target and policy are assumptions of this example.
With 1,000 fully observed eligible tasks, 995 good and 5 bad, the observed ratio is 99.5%. For that observed population, the 1% allowance is 10 bad tasks, so the five consume half of the nominal event budget. Further arrivals change the denominator and the allowance. Unused allowance does not permit violation of an independent data-integrity requirement.
Now suppose ten results are missing rather than fully observed. The old dashboard ratio no longer establishes the fraction of good tasks in this interval. Recover the observations or apply the agreed missing-data policy with that limitation visible. If the service sees only ten events in an hour, high-volume alert thresholds require a separate fit decision; the number 99% does not supply it.
For each service, distinguish a response to observed failure, a response to lost observation and a decision about an acceptable level of loss.
SYSE.36:6 - Bias-Annotation
Providers may count only the work their backend accepted, and application teams may choose the easiest available metric. Aggregate success can hide a small population with consistently poor outcomes. Check user-entry failures, task classes and missing semantic observations before accepting an apparently excellent ratio.
SYSE.36:7 - Conformance Checklist
- The subject, users, configuration and task class are named.
- The responsible owner supplies the good-result meaning.
- Eligibility and good outcome are defined independently of the convenient telemetry.
- For a claimed working measurement, applicable exercises cover good, failed and missing observations. If an important outcome cannot be observed, the result identifies that gap.
- Material pre-backend failures, retries and incomplete outcomes are accounted for.
- When an objective and response policy are required, they are agreed on a consistent event/time basis.
- Platform and application results remain separate even when they share infrastructure.
- A reliability measure is not presented as complete user value or release authority.
SYSE.36:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Use backend jobs as all user attempts. | Observe or bound failures at the user entry and explain remaining coverage. |
| Treat HTTP 200 as application correctness. | Observe the task result against its acceptance condition or state the missing semantic check. |
| Set a percentage with no agreed consequence. | Agree the objective and response policy with the service’s users and providers, including those authorized to make the relevant trade-offs. |
| Merge platform and application budgets. | Bind each measurement and policy to its own service and users. |
SYSE.36:9 - Consequences
Task-grounded measurement can expose failures that infrastructure monitoring misses and make reliability decisions more intelligible. It can require additional instrumentation, data-handling design and agreement. It may initially reduce apparent success because previously invisible failures or gaps become visible.
SYSE.36:10 - Rationale
A reliability objective is useful only when its measurement represents the service result people depend on and its policy changes an available decision. Defining meaning before instrumentation protects that connection. Separating subjects prevents success in an enabling service from being mistaken for success in the work it enables.
SYSE.36:11 - SoTA-Echoing
For user-relevant reliability, adopt and adapt the 2018 SRE Implementing SLOs treatment. User journeys, SLI specification versus implementation, feasible objectives and an agreed error-budget policy are part of that approach. Sections 4.1–4.6 develop their use across platform and application tasks, with explicit unknown-outcome bounds and policies tied to the authority and capacity needed to respond.
Adopt OpenTelemetry signal categories as observation means where useful, without treating a telemetry model as application acceptance meaning. Compare a few outcome indicators with a larger indicator set by consequential coverage and maintenance cost. A simpler initial implementation can be useful with its limits visible; an unobservable semantic requirement cannot be replaced silently by a convenient proxy.
Reopen the definition or policy when supported tasks, populations, instrumentation coverage, consequences or feasible provision change. A more demanding percentage is not automatically the better user/provider agreement.
SYSE.36:12 - Relations
SYSE.25 uses platform-task evidence for improvement choices. SYSE.31 and SYSE.41 provide feedback/deployment observations with their own limits. Use matching task evidence for exposure evaluation under SYSE.35 and for alerts under SYSE.37. A budget-risk alert also needs its service objective. SYSE.38 uses correlated evidence for diagnosis; SYSE.40 makes delayed and rejected work visible. SYSE.4 governs the resulting reliance.
SYSE.36:End
SYSE.37 - Alert on Actionable Software-Service Reliability Risk
Status: Stable
Normativity: Guidance within the stated software-service alerting use; examples are illustrative.
SYSE.37:1 - Problem frame
Use this pattern when alerts for a software-service loss arrive too often, too late or without an action someone can take. Start with the loss that matters, the time available to respond and who or what can act within that time.
The first result is an exercised alert rule with its intended response and known blind spots, or the missing observation or response capability. An alert rule alone is not an operating on-call arrangement.
For budget-risk alerting, use the same service, task population, configuration and objective defined through SYSE.36. Do not borrow another service’s SLO. A directly observed critical failure may need its own alert without waiting for an aggregate budget calculation. A dashboard trend that can wait for ordinary planning does not automatically warrant an interrupting notification.
SYSE.37:2 - Problem
A threshold can be technically correct yet practically useless. It may repeatedly notify about isolated harmless variation, wait longer than recovery allows, or keep firing after the problem has ended. Several rules can multiply notifications for one event.
The opposite failure is silence caused by lost telemetry, the wrong denominator or a missing recipient. A green alert panel then conceals both service risk and an inability to observe or respond.
SYSE.37:3 - Forces
| Force | Practical tension |
|---|---|
| Early detection and precision | Rapid interruption can protect users while excessive false alarms consume the attention needed for real failures. |
| Significant loss and current activity | A long window detects accumulated harm; a short window helps determine whether that harm is continuing. |
| Shared defaults and service fit | Reuse simplifies maintenance, but traffic and consequences can invalidate an inherited rule. |
| Notification and response | Detecting a condition is useful only when an available action can change its consequence. |
SYSE.37:4 - Solution
SYSE.37:4.1 - Choose the loss and the needed response
Identify the user task and the condition that merits action. Distinguish budget risk, a direct task or integrity failure, and loss of observation. They can have different urgency, evidence and recipients.
Recover the qualified observation and response basis; for budget-risk alerting, include the applicable objective and response policy. Determine how much loss may occur before a useful response is no longer possible, including measurement delay, notification, acknowledgement, diagnosis and mitigation time.
Choose an interrupting notification, ordinary work item or observation-only signal according to that response need. Identify the person or mechanism responsible for the response and the action they are authorized and able to take. If nobody can receive or perform the response, return that missing capability instead of declaring the alert operational.
SYSE.37:4.2 - Derive a budget-risk rule on the right basis
For an event-ratio SLO below 100%, let s be the agreed good-event target and e the observed bad-event fraction for the same eligible population. The normalized burn rate is e / (1 - s). Its meaning depends on the quality and coverage of those observations.
Choose the budget fraction p that warrants action and a decision window w within objective period T. Under a consistent event-rate assumption, the corresponding threshold is p × T / w. If traffic changes materially, relate the actual event counts to the objective’s budget rather than treating elapsed time as an exact share of that budget.
Where useful, pair the longer decision window with a shorter confirmation window to distinguish accumulated loss from continuing high consumption. Select additional slower-risk coverage only when its distinct response is needed. More windows increase rule and notification complexity.
A maximum possible burn rate can be below a proposed threshold for a loose objective. Check that the rule can fire for the failures it is intended to detect. For an objective of 100%, s = 1, so e / (1 - s) is undefined; use a different qualified alerting rule.
SYSE.37:4.3 - Qualify low-traffic and missing-data behavior
Examine ordinary quiet periods and rare task classes. One failure among a few attempts can produce a large ratio without demonstrating a sustained high-volume incident. Its practical significance still depends on the consequence of that one failure.
Choose a response fitted to that use: a direct failure alert, a longer justified observation, qualified synthetic probes or an agreed objective change may be appropriate. Do not dilute real user failures with successful artificial requests or unrelated services to reduce the reported failure rate.
Treat missing data separately from no bad events. A zero denominator, stale series or absent result is not automatically a zero burn rate. Exercise the chosen loss-of-observation rule and its response; it may merit urgent action even when the actual service state is unknown.
SYSE.37:4.4 - Exercise evaluation, delivery and action
Verify the rule’s inputs, subject labels, triggering and clearing conditions, and evaluation cadence in the deployed monitoring mechanism. For a ratio, check its numerator and denominator. For a windowed rule, check the windows; a vendor’s lookback or duration can mean something different from the intended decision window. Check that the configured rule implements the chosen logic.
Use applicable results from representative recorded or controlled synthetic histories. Exercise any uncovered failure, recovery, isolated variation or loss of telemetry that can change the selected rule’s response. Inspect when the rule fires and resets, which notifications arrive, and whether related rules duplicate or suppress a needed action.
Where a person or mechanism must respond, include the subject, affected use, evidence and first qualified response. For a notification, verify delivery through the route in service and any required acknowledgement without falsely declaring a production incident. A successful test message proves delivery, not the service-risk predicate.
SYSE.37:4.5 - Improve from actual response
After use, compare alerts with significant events and resulting actions. Find missed harm, noise, late detection, stale firing and unavailable response. Where a problem is observed, identify which measurement, rule or response arrangement needs repair and act on the supported finding.
Return the exercised rule and limits through the existing service configuration and operating account. SYSE.38 supplies diagnosis/restoration when an event is active. For an actual user-task failure, establish recovery from the task result. Acknowledgement or suppression alone does not establish it.
SYSE.37:4.6 - Choose rules with useful detection and reset behavior
Construct the rule from the response deadline, then test its behavior over time. An instantaneous error threshold notices a brief burst quickly but can interrupt people for effects that have already ended. A long-window average can detect accumulated loss while continuing to fire long after recovery. Requiring one threshold continuously for a long duration can miss a damaging intermittent failure whose gaps repeatedly reset the timer. Compare these alternatives against histories that represent the loss you need to catch, rather than selecting the query shape first.
A longer-window and shorter-window pair addresses a particular combination: enough loss has accumulated to justify action, and significant loss is still occurring. Use logical AND within that pair. If both fast and slower damage need different timely responses, construct separate pairs and combine the relevant alerting paths deliberately. A slow response can be an ordinary work item instead of a second page. Explain overlap to the recipient or deduplicate it by the affected event so that two valid rules do not create two competing responses.
In a constructed exercise, the fast rule notifies at minute 0 and a slower rule at minute 1 about the same overload event A. Retain both window observations under A and request one response. From minute 1, the only available duty engineer needs four minutes to shed permitted low-priority work and confirm relief; A needs that response completed by minute 7.
At minute 1, separate evidence also identifies integrity event B, whose qualified containment takes three minutes and must finish by minute 6. Each response needs the engineer throughout and has no qualified safe interruption; its duration includes recognition, the authorized decision, intervention and confirmation. A then B finishes at minutes 5 and 8; B then A finishes at minutes 4 and 8. Neither order protects both deadlines. The engineer holds the delegated incident authority and, under the agreed priority for preventing further corrupt writes, chooses B first; A’s timely protection remains unavailable even though its relief can finish at minute 8. Keep B’s evidence and response separate from A’s grouped notifications. Use SYSE.38 for the active response and SYSE.40 to develop A’s overload protection before a recurrence.
If B’s next write is instead scheduled for minute 9 and containment before it remains sufficient, A then B fits both limits with the same engineer and actions.
For example, under the same constant-rate, 30-day, 99.9% assumptions as section 5, choosing 10% budget consumption in three days yields 0.10 × 720 / 72 = 1: a 0.1% bad-event fraction. A selected six-hour confirmation window can show that this slower loss continues. A sustained 0.2% bad fraction has burn rate 2; it may warrant planned reliability work through this rule while remaining below the fast rule’s 14.4 threshold. The distinct values express distinct response needs. They are illustrative choices, not a reason to copy a three-day delay into a service whose harm becomes unacceptable within minutes.
Check the time available after detection. A correctly evaluated predicate may still be too late when collection, evaluation, delivery and mitigation together exceed the useful response interval. Shorten those delays, change the detection rule, or redesign the mitigation if it can no longer protect the result. A higher-sensitivity rule cannot compensate for an unavailable responder or a recovery procedure that takes longer than the remaining useful time.
Replay complete histories that can challenge the selected budget-risk rules: for example, a short spike, sustained or intermittent loss, slow degradation, recovery or telemetry outage. Compare the first firing time, clearing time, notifications and required action. Vary task volume where it affects the rule. At rapidly changing volume, p × T / w is no longer an exact statement about the fraction of the objective’s event budget consumed; use actual counts under the selected objective-window convention and state any remaining forecast assumption. A time-window burn alert can still be a useful risk heuristic under that limitation.
Check that the implementation can express the selected interval. For example, the linked Cloud Monitoring built-in SLO alerting procedure limits its lookback compliance period to 24 hours. The three-day calculation above cannot be copied directly into that mechanism. Obtain an aggregation or another implementation that supports the intended interval, or select and exercise a different rule that preserves the needed slower-loss coverage. Shortening the window changes the decision rule even if the threshold label stays the same.
Keep direct high-consequence conditions alongside budget reasoning when they answer another question. An observed wrong-bank-account payment can require immediate containment regardless of the aggregate availability budget. A monitoring pipeline failure can require restoration of visibility without asserting that the application is down. Each such rule needs its own observation and response basis. Combining all of them into one undifferentiated service-health page loses the decision the recipient must make.
SYSE.37:5 - Archetypal Grounding
Consider a constructed high-volume software service, separate from the sparse ParcelWorks address-form example. Its users/providers have agreed a 99.9% good-event objective over 30 days and a response policy that treats 2% budget consumption in one hour as an urgent risk.
For the illustrative constant eligible-event-rate basis, T = 720 hours, p = 0.02 and w = 1 hour. The threshold is 0.02 × 720 / 1 = 14.4. Multiplying by the allowed bad fraction 0.001 gives a bad-event threshold of 0.0144, or 1.44%. The selected rule requires that threshold to be exceeded in both the one-hour window and the most recent five-minute window.
| Constructed observation | Rule result and meaning |
|---|---|
| Both windows have a 2% bad-event fraction. | Burn rate is 20 in each window; this urgent condition fires. |
| The one-hour fraction remains 2%, but the recent five-minute fraction is zero with complete observation. | This active fast-burn condition no longer fires; accumulated budget use still exists. |
| The short-window series is missing. | The confirmation is unobserved, not favorable; the telemetry-loss response applies. |
| The metric accidentally includes another service’s successful requests. | The rule no longer measures the agreed subject and must be repaired. |
The recipient’s first qualified action is to inspect the affected task and current service state, then apply the available mitigation under the service’s response policy. The exercise verifies the predicate, notification route and recipient response separately. The rule covers fast budget consumption. Slower loss and direct integrity failures need their own response when they matter.
Now consider a low-volume class with ten eligible attempts in an hour and one failure. Its observed bad fraction is 10%; against a 99.9% objective, the normalized burn rate is 100. This arithmetic does not show that the same high-volume paging strategy fits the case. A single high-consequence failed task may merit direct action; an ephemeral retriable failure may warrant a different, explicitly agreed response.
Synthetic probes can reveal some unavailable paths before a real user arrives, but their successful requests must not erase the real task’s failed observation. If the rare task cannot be represented by the probe, that blind spot remains.
A deployment-path alert uses deployment-task observations; an address-application alert uses address-task observations. For a budget-risk rule, use the matching service objective. Ten successful deployments do not cancel an observed application defect, and unavailable build workers do not automatically spend the running application’s error budget.
A notification can identify which loss needs what response, while absence of usable data and absence of an available responder remain visible failures of the arrangement.
SYSE.37:6 - Bias-Annotation
Teams can tune thresholds to reduce page count while losing important failures, or insist that every fluctuation deserves interruption. Investigate the response value of alerts, not only their number. Recompute numerical examples using the service’s denominator and time basis.
SYSE.37:7 - Conformance Checklist
- An operational rule identifies the relevant service-task loss or observation gap and a usable response. If that response is unavailable, identify the missing capability.
- Budget-risk logic uses the matching measured objective and eligible population.
- Any thresholds or windows used follow the selected loss and response basis.
- Missing observations have an explicit disposition; count- or rate-based rules also address low traffic and impossible thresholds.
- The rule’s evaluation is exercised. Where the selected response requires notification, acknowledgement or action, each required link is exercised separately. If a needed link is unavailable, the result identifies it.
- Recovery and suppression do not hide remaining user harm or exhausted budget.
- Platform and application alerts do not exchange subjects or denominators.
SYSE.37:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Copy 14.4 into every alert. | For a budget-risk rule, derive its threshold from the matching objective, window and response need. For other losses, choose a trigger that fits the loss and available response. |
| Treat no samples as no failures. | Detect the observation gap and apply its own response. |
| Reduce noise by adding unrelated successful traffic. | Preserve the task population and repair the response fit instead. |
| Declare a rule operational after a test email arrives. | Exercise the risk predicate, delivery and the action the recipient can take. |
SYSE.37:9 - Consequences
Actionable alerting protects response time and practitioner attention. It requires maintained measurement, rule semantics and the capacity and authority to respond. A good arrangement can produce fewer interruptions while revealing more important blind spots; notification count alone cannot establish either improvement.
SYSE.37:10 - Rationale
An alert connects evidence of a consequential condition to an available intervention. Burn-rate arithmetic is one way to establish urgency, not the whole connection. Subject fidelity, missing-data behavior and response availability determine whether that arithmetic can protect the user’s task.
SYSE.37:11 - SoTA-Echoing
For actionable reliability alerts, adopt and adapt the 2018 SRE Alerting on SLOs comparison of detection, precision and reset behavior. Multiwindow rules, slower-loss coverage and low-traffic adaptations belong to this professional Method. Sections 4.2–4.6 derive selected rules and exercise their interaction with telemetry and response delays.
Compare the intended rule with Cloud Monitoring burn-rate semantics when using that provider. Implementation names and default durations do not choose the service’s loss tolerance. Additional rules cost maintenance and recipient attention; retain only distinctions that support a different needed action.
Reopen the arrangement when volume, objective, metric meaning, evaluation behavior or response capability changes. A correct burn calculation, a delivered notification and an effective mitigation are three separate results. Historical example thresholds do not establish any of them for a new service.
SYSE.37:12 - Relations
Use SYSE.36 for the matching task measurement and, when the rule concerns budget risk, the agreed objective. SYSE.38 supplies diagnosis/restoration; SYSE.40 supplies capacity protection where overload is the problem. SYSE.35 uses its own exposure evidence and stop conditions. SYSE.25 can use recurring platform-alert evidence to select improvement, without relying on alert volume alone to establish user value.
SYSE.37:End
SYSE.38 - Diagnose and Restore a Failed Software-Platform Task
Status: Stable
Normativity: Guidance within the stated software-platform recovery use; examples are illustrative.
SYSE.38:1 - Problem frame
Use this pattern when a practitioner cannot obtain a supported result from a software platform, or receives an uncertain result after a failed attempt. Start with that attempt, its intended outcome, the affected path stage and the current observable effects.
The first result can be the restored user task, a usable result with stated limits after mitigation, or a request to a specialist that identifies the unresolved cause or effect and the specific result needed. Mitigate the impact as soon as the available grounds permit, even while the cause remains uncertain.
When a qualified repair procedure covers the known failure, use it and verify the result. For a major incident that needs coordinated investigation, repair or communication, use section 4.5 with the local assignments.
SYSE.38:2 - Problem
“The platform is down” can conceal several different failures: a request never entered, work is waiting, a provider failed, a runtime changed only partly, or the result was produced but not returned. Blind retry can duplicate resources or repeat irreversible effects.
Diagnosis can also become an uncontrolled sequence of restarts and configuration changes. When the service later recovers, nobody knows which explanation survived, whether the original task completed or whether data was restored.
SYSE.38:3 - Forces
| Force | Practical tension |
|---|---|
| Restoration and explanation | Reducing current harm can be urgent while complete causal understanding takes longer. |
| Evidence and intervention | A useful repair changes the state being investigated; enough prior evidence should survive without prolonging avoidable harm. |
| Local action and shared effects | One task can fail because of a common dependency, and its retry can make other tasks worse. |
| Fast assistance and real authority | An available expert may understand the fix without permission to alter the affected service or information. |
SYSE.38:4 - Solution
SYSE.38:4.1 - Establish the failed task and actual impact
Recover the expected supported result, actual observation, attempt identity, relevant configuration and time. Distinguish the user’s report from a controller’s interpretation and from an observed target state.
Locate the failure sufficiently to choose a next action: entry, admission, waiting, execution, dependency, result validation or result delivery. Determine who and what are affected, what still works, and whether a useful old path or limited fallback remains.
Use the shared operating account when several participants need a common understanding of the current situation. OPS.3 distinguishes operating subjects, cases, queues and resources. OPS.4 helps participants attend to their current state. A platform log can supply observations about its own part of the work.
SYSE.38:4.2 - Reduce impact within a known boundary
Select a useful mitigation within the applicable risk limits, granted permission and current state. It may restrict new work, preserve an old serving path, isolate a failed component or use an already qualified alternative. State what remains unavailable or degraded.
Before repeating a state-changing operation, check whether an existing result can be recovered and whether the permitted retry is safe across the remaining possible states. If repetition could duplicate or corrupt an effect and observation cannot resolve that risk, hold the repeat and return the uncertainty for appropriate assistance. Preserve an unresolved historical outcome separately from a supported way to restore the user’s task.
Use SYSE.34 for uncertain data effects and SYSE.41 for partial deployment state. A restarted process does not prove restored information. A proposed fallback whose capacity, compatibility or authority is unknown is not yet a recovery path.
SYSE.38:4.3 - Discriminate plausible explanations
Form competing explanations for the observed failure and identify what would distinguish them. Use available read-only evidence when it suffices. A bounded, qualified experiment may be worthwhile when it could improve the recovery decision or the grounds for a needed causal claim.
Compare observations at the relevant path boundaries. Long queue waiting and short execution suggest a different problem from immediate admission followed by a dependency timeout. A correlation with a recent change is a lead, not proof that the change caused the failure.
Make one interpretable intervention at a time where practical. When urgent response requires several actions, retain their timing and effects and avoid claiming that the final action alone established causation. Preserve the useful configuration, attempt and failure evidence within its data-handling limits.
If the remaining question needs a specialist, return the observed failure, surviving alternatives, constraints and result needed.
SYSE.38:4.4 - Verify the user result and residual effects
After mitigation or repair, exercise the affected supported task or recover the original result. Observe identity, correctness and timing to the degree the task requires. A healthy process, an empty queue or an acknowledged alert is not by itself task recovery.
Check partial resources, in-flight work, retained state and consequences for other users. Preserve a bounded degraded result when full restoration is not yet established. Report what is restored, what remains limited and who is responsible for any remaining action.
Do not confuse a successful later attempt with completion of an earlier one. Resolve the earlier attempt’s effects or keep its uncertainty visible.
SYSE.38:4.5 - Use the major-incident response Method when coordination is needed
Apply PagerDuty, During an Incident for coordinated major-incident response, adapted to the organization’s actual severity decision, people and communication facilities. Its operative contribution is coordinated command, expert investigation/repair, a shared working account and internal/customer communication through assigned roles.
Identify the people or services assigned to lead and support the response, the technical actions they are permitted to take, who may communicate, and the available contact paths. Apply the source’s role-specific instructions to those assignments. Use sample commands or public-announcement guidance only where the local facilities and authority support them.
The first coordinated result is a managed response with assigned work and shared current information, followed by a bounded recovery or continuation decision. Refer missing assignments or permissions to whoever is authorized to provide them. Security incidents also need the appropriate specialist response.
SYSE.38:4.6 - Return the improvement question after restoration
Retain the explanations the evidence supports, any uncertainty and the useful failure history for subsequent learning. Distinguish the immediate mitigation from any further change needed to prevent recurrence.
Use SYSE.25 when the platform’s supported path needs a product improvement and SYSE.39 when repeated intervention is the burden to remove. A larger application, provider or operating decision goes to whoever can decide it. Do not keep an urgent response active merely to finish every later improvement.
SYSE.38:4.7 - Develop diagnosis from the task path and build recoverable operations
Use the task’s causal path to choose observations. An accepted request may wait before dispatch, execute on a worker, call a dependency, commit an effect and return a result. A missing result can arise at any of those boundaries. Find the last boundary at which the particular attempt is known to have progressed and the next boundary for which evidence is missing or adverse.
Build competing explanations at that boundary. If a worker has started but no dependency response arrives, consider a call that was never sent, an unavailable route, refused access, slow execution and lost response. Choose an observation that separates consequential alternatives. A local successful ping cannot establish that the application identity can perform the required database operation. A diagnostic call through the application’s actual route can be more informative, but its side effects and added load must be acceptable. Keep suggestive evidence distinguishable from an observation that rules out an explanation.
Use reduction when the path is large. Compare a known working and failing population for one consequential difference, or inspect a boundary dividing the path into two regions. A successful response from the middle can eliminate some upstream explanations only for the work the response establishes for that attempt. A failing response from a substitute environment may expose a defect while leaving the production incident unexplained. Preserve a bounded repeatable case when one exists, so invasive experiments can move to a suitable environment under SYSE.33.
Choose mitigation by what it can restore, not by the elegance of its diagnosis. Restricting a costly optional feature may restore the core task while its cause remains uncertain. Adding workers helps a worker bottleneck only if the dependency can accept the added demand. Restarting can remove transient state and erase the best failure evidence at the same time; obtain the inexpensive decisive observations first when user harm permits that delay. When urgent action must come first, record that limitation and do not manufacture a stronger causal claim afterward.
A repeated failure is also a design input for the operation itself. Give important state-changing operations a stable way to recover their result after a lost response. Where the implementation supports it, a retained operation identity can link the request, effects and result, allowing a caller to resume observation instead of creating another attempt. This mechanism needs a defined retention interval and qualified behavior under duplicate requests and concurrent consumers. Merely attaching a request ID to a log does not supply deduplication or recoverability.
Build recovery procedures from observed states and allowed transitions. For each relevant partial state, determine what can be inspected, which continuation is safe, which prior state remains usable and what evidence completes the user’s task. Use applicable exercises of interruption states, including loss of the controller while the target continues, and exercise any consequential state they leave uncovered. Keep access, dependency availability and recovery capacity within the supported operating arrangement. A dormant script with expired credentials cannot provide a return. Use the exercise results to improve the procedure and the observations it needs.
In ParcelWorks, the missing deployment result can require different continuations. Before dispatch, restoring capacity lets the original request proceed. After h2 has started with c1, the continuation first establishes that partial runtime and corrects its effective configuration. A third history can contain a completed deployment whose reply was lost: recover and return the existing result instead of redeploying. These histories require different actions even though the user sees the same timeout. An unrelated payment service applies the same reasoning to a possibly committed payment, but needs its own transaction and reconciliation contract; the deployment correction is not a transferable payment-retry recipe.
SYSE.38:5 - Archetypal Grounding
In a constructed ParcelWorks example, a developer requests deployment of a verified artifact/configuration through the supported path. The request is accepted but no usable result returns. The old application is still serving parcel operators correctly.
The responder recovers the request and its stage observations. The request has not been dispatched to an execution worker. All eight shared workers are occupied by long test jobs; new test jobs continue arriving. No target resources have been created for this request.
Two explanations initially matter: the deployment is waiting for capacity, or the deployment dependency is failing. The next observations distinguish them.
| Observation or bounded probe | Effect on the explanations |
|---|---|
| The attempt remains in the admission queue with no execution start. | The current delay occurs before a dependency call in this attempt. |
| A permitted diagnostic probe outside that busy queue reaches the dependency. | A general dependency outage is less consistent with the observed delay. |
| A deployment executes normally when a worker becomes available. | Worker contention is supported as the immediate constraint for this task. |
| The probe fails despite available execution capacity in an alternate history. | Dependency failure remains live and needs its own investigation. |
For the first history, the authorized operator stops admitting additional heavy test jobs, lets current safe-to-finish work drain, and reserves the next available worker for the supported delivery task under an already qualified capacity policy. SYSE.40 supplies the enduring capacity-protection arrangement.
The original deployment request is then processed without creating a duplicate attempt. Its runtime artifact/configuration and bounded deployment test are observed, and the developer receives the result. Queue waiting has been reduced for that task; the team does not claim that all backlog or every user class is restored.
A second attempt illustrates a different failure. Its outer job timed out after installation began. Readback finds h2 running with old configuration c1 on one candidate instance. Resubmitting the entire deployment would ignore that partial effect. The responder keeps the candidate out of ordinary routing and uses SYSE.41 to reconcile the target or return it under the qualified compatibility conditions. If readback is unavailable, the state remains unknown.
If that second attempt may also have written persistent data, SYSE.34 determines whether replay or old-version return is valid. Restarting the process cannot settle the data question. The request to the specialist names that unresolved write/state relationship while independent diagnosis of the queue can continue.
If the incident spreads across several services and requires coordinated communication, the assigned incident lead invokes the major-incident arrangement in section 4.5. A small queue failure resolved by the authorized on-call practitioner does not need that expansion.
Across these cases, verify what the affected attempt achieved and what result its user received. State which effects or outcomes remain unknown. To choose among different responses to the same timeout, use observations that distinguish the surviving explanations.
SYSE.38:6 - Bias-Annotation
Recent changes and familiar past incidents are tempting explanations. A dramatic infrastructure metric can distract from the user’s failed path stage. Check contrary evidence and preserved functionality; do not infer an application outage simply because its build or deployment platform is unavailable.
SYSE.38:7 - Conformance Checklist
- The failed attempt, supported result and actual impact are recovered.
- Proposed mitigation has a known state, compatibility and permission boundary.
- When the cause needs investigation, available observations or bounded tests distinguish the consequential alternatives or identify what further evidence is needed.
- Partial effects are resolved before potentially duplicating retries.
- A claim that the user task recovered has evidence from that task. A working process or an acknowledged alert alone cannot establish recovery. Where it remains unverified, the result identifies that limit.
- Major-incident roles and communications follow the roles and authority assigned for this incident.
- Residual limits and later improvement questions reach their appropriate holders.
SYSE.38:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Retry after every timeout. | Use a qualified recovery under its current conditions and authority. If a possible earlier effect could make repetition harmful, resolve it before retrying or hold the repeat. |
| Restart everything and name the last restart as the cause. | Preserve observations and distinguish mitigation from causal evidence. |
| Wait for complete root cause before any useful mitigation. | Apply a qualified impact-reducing action while preserving enough evidence. |
| Declare recovery because the controller is green. | Verify the affected user task and retained-state conditions. |
SYSE.38:9 - Consequences
Task-centered diagnosis can restore useful work sooner and produce a more discriminating follow-up question. It needs accessible state, knowledgeable responders and qualified recovery paths. Some uncertainty can remain after mitigation; that is preferable to either overstated causation or continued avoidable harm.
SYSE.38:10 - Rationale
A failure is experienced through an interrupted result, while its causes and effects can lie across several components. Recovering the attempt and partial state connects intervention to that result. Competing explanations and bounded tests reduce the chance that successful mitigation becomes a false causal story.
SYSE.38:11 - SoTA-Echoing
For technical diagnosis, adopt and adapt the 2016 SRE Effective Troubleshooting Method: triage, a model of the failing path, competing explanations and discriminating tests. Its troubleshooting advice supports mitigating the impact while the cause remains uncertain. Section 4.7 connects that reasoning to state-changing operations whose effects can outlive a failed request.
For a coordinated major incident, adopt the PagerDuty During an Incident Method through section 4.5’s local assignments, informed by the 2018 SRE Incident Response treatment. Its coordination is useful when scope and consequences require it; a bounded failure handled by one authorized practitioner need not acquire the same overhead.
Reopen the recovery procedure when task paths, access, dependency behavior, persistent effects or observation change. OPS.3 and OPS.4 supply the shared operating distinctions and account.
SYSE.38:12 - Relations
SYSE.26 identifies the supported user result. SYSE.36 and SYSE.37 provide matching observations and alerts. SYSE.34 bounds data recovery, SYSE.40 supplies capacity protection and SYSE.41 can establish the state of a deployment. OPS.3/OPS.4 supply operating distinctions and shared current attention when needed. SYSE.25 and SYSE.39 receive the later platform-improvement question.
SYSE.38:End
SYSE.39 - Decide Whether and How to Reduce the Total Burden of Repetitive Platform Work
Status: Stable
Normativity: Guidance within the stated platform-improvement use; examples are illustrative.
SYSE.39:1 - Problem frame
Use this pattern when a recurring manual or interrupt-driven platform activity grows with demand but produces little enduring improvement. Start with one recognizable activity and the user result it supports, then recover who repeatedly spends effort and why.
The first result is a justified choice about the recurring activity, or a comparison that identifies what prevents a choice. Retain the current method when a change does not earn its full cost. A selected intervention may remove a cause, simplify the task, change the product or automate a qualified operation. State how its total burden will be compared after use.
Do not classify all disliked work as avoidable repetition. Novel engineering, necessary human judgment, a real learning need and unavoidable physical work may deserve different treatment. If a current failure is harming users, restore the affected task through SYSE.38 before treating the episode only as an improvement opportunity.
SYSE.39:2 - Problem
A provider can appear more efficient by making users perform the same work themselves. Tickets disappear while practitioners spend more time diagnosing, copying values or learning internal machinery.
Automation can preserve the underlying cause and add its own maintenance, exceptions and failure risk. Counting automatic completions or saved operator clicks misses whether the whole supported task became less burdensome and remained correct.
SYSE.39:3 - Forces
| Force | Practical tension |
|---|---|
| Immediate relief and cause removal | A workaround can reduce today’s effort while delaying a more durable repair. |
| Provider efficiency and user work | Moving an operation to self-service can help users or merely transfer the burden. |
| Investment and uncertain recurrence | Automation costs time now; the future frequency and maintenance burden may change. |
| Uniform handling and judgment | Repetition invites standardization while exceptional or authorized decisions may remain different. |
SYSE.39:4 - Solution
SYSE.39:4.1 - Recover the recurring activity and its result
Name the trigger, user, intended result and recurring operations. Use observations of representative ordinary and exceptional attempts. Separate effort that is necessary to produce the result from effort caused by missing information, a broken interface, repeated repair or unnecessary coordination.
Count occurrence over a meaningful interval. Recover user and provider active effort, waiting, interruptions, rework and errors with enough consistency for the intended comparison. Do not add elapsed waiting to active labor as though they were the same quantity.
Include those who perform the work, including users to whom provider effort has shifted. Determine whether the activity requires unfamiliar diagnosis or necessary professional judgment, and whether a missing capability contributes to the difficulty. Return these distinctions before deciding whether the activity should be automated.
SYSE.39:4.2 - Find what makes the activity recur
Ask what condition recreates the work. Is a resource leaking, a result undiscoverable, an interface ambiguous, an input repeatedly incomplete, or a provider obligation missing? Compare plausible causes using evidence from the task.
Distinguish repair of a defect from automation of its symptom. A cleanup script may be a justified temporary mitigation, but its maintenance and risk remain part of the intervention. Do not erase required evidence or disable an independent control merely to reduce operator effort.
If the activity no longer serves an accepted need, consider ending it with whoever can decide whether it continues. If an obligation still applies, obtain its release or provide the required result another way.
SYSE.39:4.3 - Compare whole-task alternatives
Compare retaining the current method, simplifying the task, removing the cause, improving the supported interaction, and automating the qualified operation. Use SYSE.25 when this intervention competes with other platform improvements; use SYSE.24 only when the question concerns whole ways of obtaining the result.
Keep the user result and conditions comparable. Include construction, maintenance, exceptions, support, failure handling and work transferred to users or other providers over a stated horizon. If the evidence does not discriminate the options, retain the unresolved comparison. Choose a bounded probe when its attainable result can improve the choice enough to justify its full cost and delay; use C.11.DUA when that judgement is unresolved. Otherwise give the present bounded answer and continue only work whose conditions are already supplied.
Use effort arithmetic where the units are comparable, but keep reliability, risk, waiting and lost opportunities visible rather than forcing every consequence into one score. A low-frequency task can reasonably remain manual when automation would add more total burden.
SYSE.39:4.4 - Construct one bounded intervention
When an intervention is selected, construct a bounded change that can test the proposed gain without losing the required result. Retaining the current method or an unresolved comparison creates no intervention or trial requirement. For an interface change, SYSE.26 or SYSE.27 supplies the supported-use and compatibility work. For automation, define its input, permitted operation, result and handling of failures and exceptions.
Retain necessary judgment and authority. Retrieving an already authorized result is different from granting new access or accepting a nonconforming product.
Provide a qualified stop or return for unsupported cases and uncertain effects. Establish who can maintain and support the change. Keep any temporary workaround’s limits and removal condition visible.
SYSE.39:4.5 - Compare burden after actual use
For a selected intervention, use applicable observations of ordinary, exceptional and failed attempts, and exercise any consequential case they leave unanswered. Check that the same intended result is obtained and identify any work transferred to other participants.
Observe comparable use after the intervention, including maintenance and newly created tasks. Compare the proposed mechanism with the changes observed in use. A shorter ticket queue can result from users abandoning the path; that is not evidence of reduced task burden.
Return the bounded result and remaining uncertainty. Continue, revise or retire the intervention according to that evidence. If only a desk calculation or pilot exists, say so; it cannot establish lasting production benefit.
SYSE.39:4.6 - Design the intervention around recurrence, exceptions and continued support
Develop the cause of recurrence far enough to choose the change. If a user repeatedly asks whether a deployment finished, the missing contribution may be result retrieval, not deployment automation. If operators repeatedly clean a full disk, the useful change may be bounded log retention or a repaired producer, not a faster cleanup script. If each request needs a different professional decision, the repeated clerical preparation may be automatable while the decision remains human. These decompositions produce different implementations from the same count of tickets.
Consider a constructed 12-week case in which a platform must keep serving tasks and retain every event record in a readable 30-day archive. A defect repeatedly copies already archived, sealed log segments into scratch space. The archive supplies all supported uses of these records. The permitted cleanup removes only copies verified against the archive; unmatched content stays until the duty operator preserves and verifies it. Four ordinary cleanups take ten operator minutes each per week; one damaged scratch copy needs twenty minutes for manual comparison, preservation and cleanup. For these nonoverlapping triggers, the existing operator/archive path restores headroom within thirty minutes, including escalation, against forty minutes remaining before the disk fills.
Compare three arrangements with the same retained information and timely service. All figures below are active staff minutes. Each weekly total includes five minutes for the common archive’s support. The script additionally needs ten maintenance minutes and the same twenty-minute exception; the producer repair needs five support minutes and removes the five scratch-cleanup triggers. Initial costs include construction, tests, rollout and transition cleanup. The platform maintainer is authorized and available to implement and disable either change; the script uses the same bounded cleanup permission, and the duty operator remains available for exceptions. No option adds user work or purchased capacity in this case.
| Arrangement | Initial effort | Weekly effort | Total over 12 weeks |
|---|---|---|---|
| Retain the qualified manual cleanup | 0 | 4 × 10 + 20 + 5 = 65 | 780 |
| Automate the same permitted cleanup | 120 | 10 + 20 + 5 = 35 | 540 |
| Repair the duplicate-copy loop | 300 | 5 + 5 = 10 | 420 |
Choose the producer repair on these assumptions: it removes the recurrence while preserving the original event archive and exception support. If a confirmed replacement will retire this producer in two weeks while preserving the archive, and neither change has begun, the totals become 130, 190 and 320 minutes; retain manual cleanup for that remaining use. After investment, compare remaining effort instead of charging already incurred construction again.
For comparable effort, make the assumed recurrence explicit. Let C be the one-time construction effort, H the comparison horizon, f the number of relevant tasks per unit time, and e the recurring whole-task effort per task. A simple comparison is C + H × f × e plus maintenance and any separately counted exceptional work. Use the same boundary on both sides and avoid counting exceptions twice when e already includes them. This calculation supports a conditional effort choice. Waiting, skill loss, reliability and consequences retain their own descriptions unless a justified common valuation is available.
Test the choice against plausible changes before investing heavily. A solution that pays back only at today’s peak volume may be unattractive after demand falls. A provider interface about to retire can shorten the available horizon. A small increase in the exceptional fraction can dominate a fast ordinary path when every exception needs a specialist. Identify which uncertain values could change the choice or the conditions under which it pays off. Obtain further evidence when the expected improvement in that judgement warrants its cost. If no feasible intervention improves the relevant result within the available horizon, retaining the present supported method is a useful outcome.
Construct automation around the operation’s contract. Validate that the request names the right subject and the requested operation is authorized; obtain the current state; perform only the supported action; return its observed result. Separate unsupported input, known operational failure and unknown effect. An exception should carry enough context for the recipient to continue the same task without repeating the whole investigation. After an unknown-effect timeout, recover the original effects or obtain the qualified retry or recovery action; a fresh state-changing attempt is not justified by the timeout alone.
Choose how much autonomy the available evidence warrants. A read-only lookup can often go directly into ordinary use after its access and result behavior are qualified. A repair automation with state-changing effects may first produce a proposed action for a qualified operator, then execute a bounded class once those decisions and effects are demonstrated. Include the time and effort spent supervising the automation and reviewing its suggestions in the cost of the supported path.
Assign continued support to a maintainer with the capacity and access it requires. Identify who can change the integration when the provider changes, diagnose wrong results and retire a no-longer-useful workaround. Expose ordinary errors in a form those maintainers can act on. If the change creates a new queue of unowned exceptions, its apparent automation gain is incomplete. Review recurrence and support demand after use without imposing a measurement burden larger than the work being removed.
For the lookup case, a provider that later exposes a stable result API can make the custom search adapter unnecessary. Compare replacement with retaining the adapter over the remaining use. If replacement is justified over that horizon, preserve attempt identity, authorization and unknown-result behavior. Weigh its total work together with the service result it can support. If the same provider removes historical results before users need them, a faster API still fails the task: negotiate retention, provide an authorized retained result, or narrow the promise.
SYSE.39:5 - Archetypal Grounding
In a constructed platform example, developers repeatedly ask an operator to recover the result of an already completed, already authorized deployment attempt. The result exists, but the supported interface does not expose a stable attempt lookup.
For twenty illustrative retrievals in a week, the old path takes four minutes of user effort and six minutes of operator effort per retrieval. The result is the same identified deployment outcome in each case. Waiting and interruption effects are not yet measured and remain separate from the following active-effort calculation.
Alternative A is a page telling each developer how to perform the operator’s internal search. It removes the provider ticket, but takes ten minutes of user effort per retrieval. Alternative B exposes a bounded authenticated lookup for the original attempt. It returns the observed result or a precise missing/uncertain outcome, without starting another deployment.
| Constructed weekly arrangement | User effort | Provider effort | Continuing maintenance | Total active effort |
|---|---|---|---|---|
| Old path, 20 retrievals | 80 min | 120 min | 0 additional min | 200 min |
| A: users perform the internal search | 200 min | 0 min | 0 additional min | 200 min |
| B: 18 ordinary and 2 exceptional retrievals | 24 min | 12 min | 20 min | 56 min |
For B, each ordinary retrieval takes one user minute. Each of the two exceptions takes three user minutes and six provider minutes. Thus user effort is 18 × 1 + 2 × 3 = 24 minutes, with 12 provider minutes and 20 maintenance minutes.
A is not an active-effort reduction despite eliminating all twenty tickets. It may also change waiting or autonomy, but those gains have not been observed. B has a constructed recurring difference of 144 minutes while retaining exception assistance. Its initial construction costs eight hours, which must remain in any longer-horizon comparison.
With recurrence, task mix and the stated costs unchanged, a one-week horizon gives 200 minutes for the old path versus 480 + 56 = 536 minutes for B, including its eight-hour construction. A four-week horizon gives 4 × 200 = 800 minutes for the old path versus 480 + 4 × 56 = 704 minutes for B. The active-effort preference therefore changes from the old path to B with the horizon. Waiting, risk and future recurrence remain separate questions.
The team selects a bounded B probe. It uses the provider’s qualified same-attempt result lookup and existing authorization. An ordinary request returns the original runtime/configuration observation and test outcome. A lost result observation returns uncertainty and support; the lookup does not create a replacement deployment.
A request to see another user’s restricted result fails the applicable access check. A request for new deployment permission is outside this retrieval activity and goes to the holder of the deployment decision. The lookup trial supplies evidence about retrieving a result under an existing permission.
The probe exercises ordinary lookup, missing provider observation and unauthorized access. The subsequent comparison must measure real recurrence, user/provider effort, maintenance, error and support effects over the same task conditions. If exceptions dominate or a provider change makes the lookup expensive to maintain, the anticipated gain can disappear.
The comparison counts the work of completing the retrieval, whoever performs it. Moving twenty searches to the users removes twenty tickets while leaving total active effort at 200 minutes.
SYSE.39:6 - Bias-Annotation
Operator-visible work is easier to count than user search, interruptions or abandoned attempts. People may also prefer an interesting automation project to a simpler product repair. Ask whose effort disappeared, whose increased, and whether the required result remained available.
SYSE.39:7 - Conformance Checklist
- The recurring activity and supported user result are recognizable.
- Occurrence, user/provider effort and exceptional work are compared on a stated basis.
- Necessary judgment, learning and physical work are not mislabeled merely because they are inconvenient.
- Cause removal, simplification, retention and automation receive a fair comparison.
- Any selected intervention preserves permission, correctness and usable exception handling.
- Construction, continuing maintenance and transferred burden remain in the comparison.
- Claimed gain distinguishes calculation, pilot observation and continued use in operation.
SYSE.39:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Fewer tickets means less work. | Compare the whole task, including user and other-provider effort. |
| Automate a recurring symptom indefinitely. | Inspect the cause and retain the workaround’s cost and limits. |
| Treat every manual action as a defect. | Identify its contribution and whether human judgment remains necessary. |
| Count only the successful automatic path. | Include exceptions, failed attempts, support and maintenance. |
SYSE.39:9 - Consequences
A bounded intervention can release practitioner attention and reduce repeated failure or coordination. It requires observation beyond the provider’s own activity and may show that an attractive automation is not worth maintaining. Some manual work remains the better choice under the actual frequency and consequences.
SYSE.39:10 - Rationale
Burden belongs to the whole supported task, not only the component or team being optimized. Comparing preserved results and all contributing effort prevents local efficiency from becoming a hidden transfer. Cause removal can eliminate the recurring work that symptom handling still requires.
SYSE.39:11 - SoTA-Echoing
For reducing repeated operational burden, adopt and adapt the 2018 SRE Eliminating Toil treatment. The source includes root-cause removal, process simplification, construction and maintenance costs, and benefits beyond direct labor. A universal time quota or automation-first rule is not required by that professional Method.
Sections 4.3–4.6 and the lookup case carry the comparison across the platform user, provider, exceptions and future support. Compare retaining the current method, ending unnecessary work, repairing the cause and automating a qualified operation. The comparison horizon and demand can reverse an effort preference; a disappeared ticket cannot establish a whole-task gain.
Reopen the intervention when recurrence, task meaning, provider contracts, exception rate or maintenance capability changes. Keep waiting, reliability and effort distinct where no justified common valuation is available. A claim of lasting organizational savings needs evidence covering continued use, including support and maintenance.
SYSE.39:12 - Relations
SYSE.25 compares platform improvements; SYSE.26 and SYSE.27 change supported interactions. SYSE.24 compares whole obtaining alternatives when that larger question is live. SYSE.38 handles immediate failure, SYSE.36 supplies task observations, and SYSE.29 retires an intervention or path whose continued use is no longer justified.
SYSE.39:End
SYSE.40 - Protect Software Platform Capacity and Isolate Failure
Status: Stable
Normativity: Guidance within the stated software capacity-protection use; examples are illustrative.
SYSE.40:1 - Problem frame
Use this pattern when bursts, expensive work or a failing dependency prevent other users from obtaining supported platform results. Start with the constrained resource and the workload classes consuming it, including the control and recovery work needed to keep the service operable.
The first result is a tested capacity-protection arrangement for a stated operating envelope, or the proposed envelope and the conditions whose qualification is missing. Identify the admission, waiting, separation and deadline limits the protected task needs, limits on any retries, and how users can recognise a refusal or usable degraded result.
If ordinary demand already fits a qualified envelope and no consequential coupling exists, use that arrangement. If an active failure needs immediate mitigation, use SYSE.38. Continuing allocation of operating work remains an operating responsibility; this pattern engineers the means and limits.
SYSE.40:2 - Problem
Requests with the same name or arrival rate can consume very different CPU, memory, storage, connection or worker resources. A long queue can hide overload until every accepted task misses its useful deadline. Retries can multiply demand while the original work is still running.
Adding capacity may not repair a serial bottleneck, a downstream limit or a failure that affects every new instance. A protected front-end proxy can remain healthy while its upstream service fails and the user’s task cannot finish.
SYSE.40:3 - Forces
| Force | Practical tension |
|---|---|
| Utilization and isolation | Sharing resources is efficient until one class prevents another from completing. |
| Admission and user outcome | Rejecting early can preserve useful work while still being a failed attempt for the rejected user. |
| Waiting and usefulness | A queue smooths short bursts but can retain work that is no longer worth completing. |
| Recovery and amplification | Retry can recover a transient failure or intensify pressure and duplicate effects. |
SYSE.40:4 - Solution
SYSE.40:4.1 - Recover demand, resource limits and failure coupling
Identify supported task classes, their demand patterns and the resources they consume through the path. Include costly variants, concurrency, fan-out, dependencies and resource use during failure, not only average successful requests.
Use applicable measurements of the relevant resource constraints. Where a relevant limit is unknown, measure it directly when feasible. A requests-per-second figure can be useful for a stable workload but cannot stand in for changing request cost. Distinguish available capacity from a configured limit and from a provider’s promised future supply.
Locate shared failure effects. Heavy tests may occupy the workers needed for release control; retrying clients may overload a recovering dependency; the monitoring or cancellation path may depend on the same exhausted pool. Determine which useful work must remain available under the selected failure.
SYSE.40:4.2 - Choose the protected envelope and alternatives
Compare additional capacity, reduced offered work, a cheaper qualified result, per-user or workload isolation, and redesigning the execution path to remove an identified serialization point. Autoscaling is one possible mechanism; its start delay, quotas, dependency capacity and failure behavior remain conditions.
Choose a bounded demand/resource envelope and the supported degradation outside it. Apply the authorised decision on priority and resource trade-offs. If a new decision is needed, make it within your authority or obtain it from the holder. A queue can apply an authorised priority rule.
Define the useful outcome for rejected, delayed or degraded work. Preserve the distinction between an accepted task, a refusal before effects, an unfinished task and a partial result. Do not count overload rejection as user success merely because it protects the service.
SYSE.40:4.3 - Construct admission and bounded waiting
Admit work only under the selected resource and concurrency limits. Bound queues or waiting by the conditions that preserve task usefulness, including deadlines and cancellation behavior. Account for the resources consumed merely by holding or rejecting a request.
Separate workload or user classes where their coupling would violate the protected use. Reserve or independently provide control, health observation and recovery capacity when that is necessary. Verify that a shared downstream resource does not invalidate the apparent isolation.
Expose a useful response when work cannot be admitted. Give the user enough information to distinguish a refused attempt from one that may already have effects. A retry indication must reflect the qualified path, not encourage an immediate synchronized retry storm.
SYSE.40:4.4 - Bound retry and dependency failure
For each retrying layer, establish the operation’s replay behavior, deadline, authority and any required attempt identity. Resolve effects that could make repetition unsafe, or use a retry qualified across the remaining states. SYSE.34 and SYSE.41 supply the relevant data and runtime conditions.
Where retries are permitted, bound the total number of calls they can produce across layers. Delays and jitter can spread attempts but do not make an unbounded retry policy finite. Stop work that can no longer produce its permitted result rather than spending resources until an outer timeout hides it.
Protect both the component receiving pressure and the dependency receiving calls. A local resource monitor, a dependency concurrency limit and a fallback can address different failure mechanisms. Check that the selected controls protect the path through its dependencies.
SYSE.40:4.5 - Exercise overload, isolation and recovery
Use applicable tests of ordinary demand and of the bursts, expensive variants or dependency failures and slowdowns that could defeat the claimed envelope. Exercise a relevant untested condition where a safe, permitted test is feasible. Identify any condition that remains unqualified. Check the resource observations and the accepted, rejected, delayed and partial task outcomes.
Verify that the protected class still obtains its qualified result, queues stay bounded, and other tenants or downstream Systems are not made worse in an unaccepted way. Check the control/observation path under pressure; repair the arrangement if its state cannot be observed or admission cannot resume under the selected recovery conditions.
Verify what happens as pressure falls. Use the recovery observations or exercise the missing condition to establish that useful queued work drains, admission resumes and returning clients stay within the recovering service’s capacity. A lower resource chart alone does not show that users can work again.
Return the exercised envelope, chosen limits, response behavior and remaining blind spots. Use SYSE.36 to retain user-visible failure and delay in measurement, and SYSE.38 for unresolved active failures.
SYSE.40:4.6 - Derive limits from useful work and check the recovering system
Develop a capacity model at the limiting resource on the task’s path. For a reasonably stable task class, measure resource demand per completed task and its variation, then combine it with the anticipated arrival and concurrency pattern. Distinguish sustained demand from a burst and include service time while a dependency is slow. A mean CPU cost can help estimate a baseline; it does not bound a costly tail or a separate memory, connection or storage constraint. Establish which resource becomes limiting under each relevant workload mix.
Use those observations to choose headroom and an admission rule. Headroom covers selected variation and failure conditions, such as one worker unavailable or a dependency taking longer to respond. It is not a universal percentage. A limit based on active concurrency can protect a connection pool better than a request-rate cap when calls become slow. A rate cap can protect a service from excessive arrivals even when each finishes quickly. Some paths need both, together with a bounded waiting population. Follow the request into shared dependencies before concluding that either local limit is sufficient.
Relate waiting to the task’s remaining useful time. In a simple single-class example, two dedicated workers complete each job within a qualified two-second bound and have no hidden downstream wait. Four queued jobs behind two just-started jobs can complete in three waves, by six seconds. If the task must finish within five seconds from admission, accepting all six cannot guarantee that bound: when each job uses the permitted two seconds, the last pair is late. Reduce accepted work, shorten qualified service time, add usable capacity or change the promise with its holder. If service time is only an average, this deterministic bound is unavailable; qualify a probabilistic or observed service objective instead.
Compare strict reservation with borrowing. Strictly separate pools make a protected class easier to reason about but can leave capacity idle. Borrowing can improve utilization if the protected class can reclaim sufficient capacity within its response bound. Non-preemptible long jobs can make that reclaim promise false. Separate execution pools also do not separate a shared database lock, network limit or provider quota. Choose the mechanism at the point of interference, including priority propagation where the downstream system supports it.
Design admission and recovery as one feedback arrangement. When measured pressure rises, reduce offered work soon enough to preserve the selected useful tasks. When it falls, resume in a way that does not immediately recreate the overload. Hysteresis, a gradual increase or a limited trial population can prevent rapid oscillation, but each needs a trigger and observation appropriate to the mechanism. Autoscaling can provide later capacity without protecting the interval before new workers are ready; a scaling request is not available capacity.
Extend the two-worker example with a constructed recovery trace. Each job can still complete within two seconds, including dependencies and result delivery; scheduling is FIFO, and the admission gate’s acceptance and completion records are current. At second 0, four jobs are accepted and admission pauses: two start immediately and two queue. An empty-system observation sampled at second −1 arrives at second 1 because resource reporting takes two seconds. At second 1, six clients whose earlier requests were refused before effects return with original task deadlines at second 6. The gate admits or refuses this burst by second 1.1 using separately qualified control capacity. If the controller reopens admission for all six from the stale observation, jobs taking the permitted two seconds finish in pairs at seconds 2, 4, 6, 8 and 10. The last four newly accepted jobs miss both their original deadline and the five-second bound from admission.
Instead, use the current accepted work and its remaining service bound to admit at most two of the six in this recovery trial. The gate enforces that total across the simultaneous arrivals and closes the trial after those acceptances; the other four receive refusal before execution effects. The earlier jobs finish by seconds 2 and 4; the admitted new pair finishes by second 6, within five seconds of admission and its original deadline. In that same two-second history, at second 4 even the next pair queued behind the still-accepted pair has a completion bound of second 8. The four refused tasks with original deadline 6 therefore cannot be readmitted on that basis. A later trial can serve other offered work only if its remaining useful time fits.
If one worker becomes unavailable during the trial, the two-worker calculation no longer supplies the promised bound. Stop further admission and identify the affected accepted tasks for recovery through SYSE.38; preserving their deadlines now requires another feasible arrangement.
Keep the service accountable for the resulting user outcomes. A cheaper response counts as a good degraded result only if that meaning was agreed for the task; stale data can be acceptable for one display and unacceptable for another. Expiring a queued task does not cancel an effect that has already begun. A recovery exercise must show both restored new-task admission and resolution or explicit limitation of earlier accepted work. Track retries and resumed demand during that exercise: a healthy idle service can fail as soon as all waiting clients return together.
When ParcelWorks adds a test class with much larger memory demand, the six-heavy-slot arrangement may cease to be safe even though the job count is unchanged. Introduce the relevant memory bound or a distinct class, qualify its effect on the shared dependency, and reconsider the accepted waiting population. Preserve the control reservation only if it still protects the end-to-end control task.
SYSE.40:5 - Archetypal Grounding
A constructed ParcelWorks platform has eight execution slots. Long test jobs can occupy all eight, preventing short delivery-control tasks from running. The current question is how to preserve a bounded control path under a test burst, not how to maximize average worker occupancy.
The team first qualifies the resource demand of the supported test and control classes. It finds that the relevant slot isolation can be implemented, while database and artifact-store limits still require separate observation. The following numbers illustrate the chosen admission construction.
The candidate reserves six execution slots for heavy tests and two for control work. Waiting is bounded at twelve heavy tasks and four control tasks, subject to the task’s remaining useful deadline. A class does not silently borrow the other’s reserved execution capacity in this construction. For the following admission-count illustration, all requests meet the class, access and remaining-deadline conditions; timing qualification is a separate question below.
| Simultaneous burst into an initially empty arrangement | Running | Waiting | Refused before execution |
|---|---|---|---|
| 20 heavy test requests | 6 | 12 | 2 |
| 8 control requests | 2 | 4 | 2 |
Eighteen heavy and six control requests are accepted, while four requests receive explicit refusal without execution effects. The total of eight running tasks respects the slot limit. A refusal is not counted as a successful developer task.
The control queue is not a guarantee that all six accepted control requests will finish in time. Their task-duration/dependency envelope and remaining deadlines must be exercised. If that qualification fails, reduce admission, provide capacity or change the service promise; a short queue alone does not establish responsiveness.
For ordinary load, representative tasks complete and their results are observed. Under the burst, control work still has its reserved slots while the heavy class reaches its own limit. If all control tasks nevertheless block on an exhausted shared database connection pool, the apparent isolation fails at that downstream resource and the arrangement must be revised.
In a second adverse history, a client times out while a deployment may already have changed its target. The retry policy first queries the existing attempt. It does not treat every timeout as a refusal with no effects. Only a qualified repeat is allowed, with a bounded total attempt budget across client and service layers.
To make the cross-layer bound concrete, suppose one user attempt permits at most three service invocations and each invocation at most two downstream calls, counting initial calls in both limits. Independent local caps can therefore produce 3 × 2 = 6 calls to the constrained dependency. In the alternative construction, every invocation for that same attempt identity shares four downstream-call units and a ten-second deadline measured from the original entry; neither allowance resets on an outer retry. Each call must obtain one unit from the common allowance before it is sent. If otherwise qualified repeats use two calls in the first invocation and two in the second, no unit remains for a third invocation’s downstream call. Waiting and backoff also spend the common time; do not start a call that cannot fit the qualified remaining-time rule, and do not mistake the deadline for proof that earlier work has stopped. If the second state-changing call has an unknown effect, stop replay with two units still available and recover its state. These counts and times are illustrative bounds, not universal settings or evidence of response-time qualification.
After pressure falls, the test verifies that waiting useful work completes, expired work is resolved according to its rule, and admission resumes without a synchronized flood. Observation and recovery operations must themselves remain available.
An implementation comparison is Envoy’s overload-manager architecture: it connects resource-pressure observations to configured actions protecting the proxy. Its own-resource protection is distinct from upstream circuit breaking. The linked latest/development documentation is not a pinned production configuration; identify the deployed edition and establish which configured monitors and actions are qualified for the task. Qualify any missing part before relying on it.
State the limits on accepted work and on the response to overload. Even two named queues can compete for one downstream connection pool. Check whether the protected class can finish within the proposed or agreed envelope and whether admitted work can recover after a burst.
SYSE.40:6 - Bias-Annotation
Average throughput can conceal expensive requests and a small class losing all service. A provider can also report a healthy system after rejecting most users. Inspect task outcomes by relevant class, including recovery/control work, and retain the cost of reserved or unused capacity.
SYSE.40:7 - Conformance Checklist
- The constrained resources and workload/failure conditions are identified.
- The protected envelope has supporting grounds, and priority trade-offs reflect a decision by their authorised holder.
- Admission and waiting are bounded by capacity and task usefulness.
- Isolation includes consequential downstream and control-path dependencies.
- Any retry mechanism has a qualified effect model and aggregate amplification limit.
- Rejected, delayed and degraded work remains visible in the results reported for each affected user class.
- Ordinary use, overload and recovery are exercised before relying on the envelope.
SYSE.40:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Size everything by average requests per second. | Inspect resource use per task and supported expensive variants. |
| Add an unbounded queue to avoid rejections. | Bound waiting and expose work that cannot finish usefully. |
| Assume autoscaling solves a shared dependency limit. | Qualify the bottleneck and the time/capacity available to scale. |
| Retry at every layer until the task succeeds. | Qualify the repeat under the possible effect states and bound total retries, deadlines and offered load. |
SYSE.40:9 - Consequences
Capacity protection can preserve useful service during pressure and make refusals more intelligible. It can reduce peak utilization, require reserved resources and expose an inadequate service promise. Isolation has a maintenance and cost burden; additional capacity or reduced demand may be the better choice for a particular envelope.
SYSE.40:10 - Rationale
Overload is a relation between offered work, its resource cost and the available means of completion. An arrival-rate or component-health limit protects the task only when it controls the relevant resource demand and preserves the required completion. Bounded admission, separation and replay behavior preserve a meaningful task result or an explicit refusal when unlimited service is impossible.
SYSE.40:11 - SoTA-Echoing
For resource pressure, adopt and adapt the 2016 SRE Handling Overload treatment. Resource-sensitive demand, quotas, client throttling and differentiated criticality are part of the source’s approach to handling overload. Sections 4.1–4.6 connect those choices to useful task deadlines, retained effects and recovery of accepted work. Derive the limits for the service from its workload model.
Compare the chosen controls with Envoy’s overload-manager architecture and configuration when that mechanism is used. Local proxy protection differs from upstream protection. Strict reservation, reclaimable borrowing, extra capacity and reduced demand have different utilization and response consequences.
Reopen the envelope when request cost, workload mix, shared dependencies, retry layers or deployed mechanisms change. Latest documentation is not qualification of an installed version; a burst count alone is not a task-deadline guarantee.
SYSE.40:12 - Relations
SYSE.26 defines the supported task result, SYSE.33 supplies controlled exercise conditions, and SYSE.36 measures rejected/delayed work. SYSE.38 handles active failure. SYSE.34 and SYSE.41 qualify state-changing recovery. SYSE.24 compares whole obtaining alternatives when capacity changes that larger choice; operating allocation remains with the relevant OPS work.
SYSE.40:End
SYSE.41 - Deploy a Verified Software Artifact with the Target Configuration and Handle Partial Failure
Status: Stable
Normativity: Guidance within the stated software-deployment use; examples are illustrative.
SYSE.41:1 - Problem frame
Use this pattern to establish what artifact and configuration run on a named target or to deploy the required configuration. Start with the artifact-specific basis from SYSE.32 and the intended target and configuration from SYSE.13. For deployment, establish the required resources, dependencies, applicable data-change result from SYSE.34 and permission for the operation.
The first result is an observed deployed configuration with its deployment test, interval and limits, or a precise failed/unknown partial state. A promoted artifact, committed desired state or completed controller job can leave the requested version or configuration absent from the target.
If the required current runtime already exists with applicable observation and test evidence, use the existing runtime and its evidence. If artifact qualification or data compatibility is missing, return that condition before the dependent operation. This pattern does not supply general application acceptance, wider exposure permission or the whole-System usability decision.
SYSE.41:2 - Problem
Deployment changes several coupled things: installed software, effective configuration, dependencies and sometimes persistent state. Some changes can complete before a command times out. The controller can fail while one instance already runs the new binary with the old configuration.
Blindly repeating the whole job can duplicate resources or repeat a migration. Returning an old binary can also fail if the data or external contract has changed. The actual resulting runtime must therefore be recovered and tested independently of the controller’s summary.
SYSE.41:3 - Forces
| Force | Practical tension |
|---|---|
| Consistent procedure and different targets | Reuse reduces variation while environment-specific configuration and permissions remain real. |
| Bounded change and partial completion | A small deployment limits impact but can still leave mixed versions or states. |
| Fast return and compatibility | A previous runtime is useful only while it remains compatible with actual data and dependencies. |
| Technical success and reliance | Installation and a deployment test support a limited use, not every application behavior or release decision. |
SYSE.41:4 - Solution
SYSE.41:4.1 - Recover the intended and actual configuration
Identify the verified artifact and the applicable evidence. For an installation, establish that its bytes are obtainable. Identify the target, runtime configuration and required dependency/resource conditions. Keep configuration separate from the artifact where the software supports it; do not rebuild the tested package for the destination.
Inspect the actual target state before acting: what is running, what is installed, which configuration is effective, and whether an earlier attempt has already changed anything. A desired-state file or an old deployment record does not replace current observation when drift or partial effects are material.
Identify any last usable configuration and the conditions under which it can be restored. If there is no qualified fallback, state that limit. Before changing the target, establish permission for the intended operation and obtain any missing authority. Access to a deployment tool does not by itself authorize a different target, wider user exposure or an unqualified data change.
SYSE.41:4.2 - Construct a bounded deployment procedure
Choose the deployment unit and how existing useful service will be preserved where the architecture permits it. Define the order the dependencies require and the point at which a test or observation changes the next action.
Use DORA Deployment automation to organize supplied packages, environment configuration and a deployment procedure. Its operations prepare the target, install the package, apply required configuration and dependency changes, and perform a deployment test. Bind these operations to the mechanisms and permissions available on the target.
Account for each step’s partial and replay behavior. Preparation may create resources; installation may change only some targets; configuration may be written but not loaded. Include data operations only when SYSE.34 or an equivalent qualified result permits them. An outer deployment job is not authority to invent or repeat a migration.
SYSE.41:4.3 - Prepare, install and observe the actual runtime
Where the observed target still needs work, perform the required, authorised steps in order. These may include preparing resources and dependency access, installing or updating the selected artifact, and applying its matching configuration. Use the existing authorized secret mechanism and verify configuration without displaying the secret values.
Observe the result per target. Establish which artifact is running and which configuration is effective. A copied file or a requested controller value does not establish either result. Check the dependency state relevant to the named use. Distinguish running, failed, not yet attempted and unknown targets.
For a reconciliation mechanism, observe convergence and the process result. A declaration accepted into version control or a desired replica count is not evidence that the intended artifact/configuration is serving the required use.
SYSE.41:4.4 - Perform the deployment test
Use a test result applicable to the required installation and connection. If the evidence is missing and a permitted test can supply it, exercise the bounded task. Establish the tested artifact and configuration, target, time and outcome, or state what remains unknown.
Distinguish missing observation from failure and failure from a successful but narrower test. A process health check can show liveness without demonstrating an application write/read or dependency interaction. Select the test from the intended next use.
Return a successful runtime result only for the targets and conditions for which the test and observations establish the required result. If a smaller qualified subset is useful, state its capacity and limits rather than silently presenting it as the complete requested deployment.
SYSE.41:4.5 - Choose a safe continuation after partial failure
After a timeout or interruption, use available observations to establish which states the deployment could have reached. Select a permitted continuation, removal or return that preserves the intended result in those states. If the same step is qualified for all of them, it may be repeated. If the choice depends on which effect occurred, inspect the target or original attempt, or hold the affected step. Verify the resulting runtime separately before relying on it.
Keep an unqualified candidate out of ordinary reliance where the architecture and authority permit that separation. Preserve the working prior service rather than damaging it during an uncertain repair. If the old service is no longer available, state that constraint instead of assuming a rollback path.
Use SYSE.34 before returning binaries across a changed data boundary. If effects or readback remain unknown, stop the operation that could create duplicate resources, repeat external effects or corrupt stored data, and request the missing observation or specialist result. A “failed deployment” label does not establish that no state changed.
SYSE.41:4.6 - Return the deployed result to its consumers
Provide the observed artifact/configuration, targets, dependency/test result, interval and remaining limits through the existing runtime and working account. Distinguish an engineered deployment procedure from a completed execution of it.
SYSE.11 can use these integration observations to assess bounded usability of the resulting System. Use SYSE.35 when further exposure needs evaluation, and SYSE.14 when an authorized release decision is needed. A passed deployment test does not supply permission to widen use.
SYSE.41:4.7 - Choose the deployment arrangement and make interruption recoverable
Construct the procedure around the target’s possible intermediate states. Separate preparation that can occur while the old service runs from operations that change a serving process or shared data. Place each compatibility prerequisite before its first consumer. Put an observation after the operation whose result determines the next choice. This creates a dependency structure that can permit parallel preparation while preserving necessary ordering.
Choose the replacement arrangement from coexistence and capacity. A rolling replacement can keep some old instances available while updating others, provided old and new versions can coexist and the remaining capacity is sufficient. A separately prepared candidate pool can be tested before routing changes, at the cost of extra resources and shared-state analysis. A stop-and-replace operation can be simpler for a service that permits the interruption and cannot coexist with its successor. Each needs a result test and a recovery boundary.
Identify the operation that starts the candidate’s effects on live work. Copying a file, restarting a process, reloading configuration, changing routing and enabling a feature can have different effects. The installed artifact may remain dormant; conversely, a scheduled worker can start processing shared work immediately when it starts. If runtime construction itself activates the candidate, obtain the corresponding exposure authority and protection before that action. Separation of deployment and wider exposure is a useful architecture where it exists, not a factual claim about every application.
Use stable resource or operation identities when the task asks what the interrupted request did or a safe continuation depends on its earlier effects. Advance a durable progress indication only when its corresponding effect is established under the failures the mechanism can undergo. After a lost acknowledgement, an authorized retry can complete the required operation if it is qualified for every state still possible, including partial execution and relevant intervening changes. When its safety depends on resolving an earlier effect, use reliable readback or another qualified reconciliation. If no permitted safe continuation is available, stop the affected step and identify the state or assistance still needed.
Keep the common procedure separate from legitimate target parameters. A test environment can supply a smaller pool and controlled data while retaining the same installation and configuration-loading mechanism. If production uses an additional proxy, identity provider or storage mode, exercise the relevant join before relying on lower-environment success. Keep target-specific exceptions visible and supported instead of growing unrelated scripts whose behavior can no longer be compared. Simplifying those exceptions can be a better investment than automating every historical variation.
Construct the deployment test to discriminate the next required claim. A process readback can establish the loaded artifact when it distinguishes the candidate builds, a connection exercise can establish access, and a bounded write/read can establish a selected application interaction. The collection must cover the promised installation result without becoming a substitute for every acceptance or reliability check. Use reserved test data or another qualified containment mechanism when test effects would otherwise reach ordinary users. A test that sends an actual customer notification needs the corresponding permission; tagging the request synthetic does not contain that effect.
For a scheduled importer, the apparent candidate pool may share the live work queue. Starting its workers already changes which version processes real jobs. The team can bind a candidate to a qualified test queue and limited credentials for isolated deployment testing, or treat startup as bounded live exposure with compatible jobs and a stop plan. If neither arrangement is available, a planned interruption may be the supported choice. The same artifact promotion and runtime readback remain useful. In each deployment architecture, identify the operation that first exposes live work to the candidate.
SYSE.41:5 - Archetypal Grounding
In a constructed ParcelWorks deployment case, the platform already has a verified candidate artifact h2 and intended configuration c2. The old service has an observed serving pool at h1/c1. A separate two-instance candidate pool can be prepared, installed, configured, started and queried through an authorized operator interface.
The data state retains the old address contract under SYSE.34’s single-authority correspondence. A release holder permits deployment and a synthetic address write/read test in the isolated candidate pool. This permission does not include routing general users to it.
The team constructs the following procedure from those available capabilities.
| Operation | Result to establish |
|---|---|
| Recover the old and candidate pools’ current state. | h1/c1 remains usable for ordinary service; existing candidate effects, if any, are known. |
| Prepare candidate resources and dependency access. | The named candidate targets can reach the required database and artifact source under the permitted configuration. |
| Install retained h2 and apply c2. | Each candidate process is observed running h2 with effective c2. |
| Exercise the permitted address write/read. | The result preserves the selected D/L/T correspondence under the qualified data state. |
| Return the deployment result. | The observed targets, identity, test and limits are available to the next usability/exposure decision. |
In the normal constructed history, both instances run h2/c2, their required dependency is accessible and the bounded test succeeds. The old serving pool still receives ordinary traffic. The candidate is usable for the exercised test; further address-form acceptance and exposure remain separate.
Now consider a partial history. The first instance reports h2/c2. The second install request times out, but readback finds h2 running with c1 and a failing database connection test. The controller’s failure did not mean that the second instance stayed unchanged.
The operator keeps ordinary routing on the old pool and excludes the unqualified candidate. With the partial state known, the operator applies the authorized c2 correction to the second instance, verifies that the running process loads c2, and repeats the bounded deployment test. If the correction fails, the operator can retain a clearly bounded qualified subset or remove/return the candidate only under the selected capacity and recovery conditions.
If readback is unavailable, no successful correction is inferred. The unresolved target remains unknown and cannot be included in a qualified deployment result. Retrying every command would not repair the missing state knowledge.
Returning candidate processes to h1/c1 is available in this example only because the old data contract remains valid and the runtime restoration test succeeds. A migration that loses information required by the old data contract or the agreed recovery result would remove that basis. A new data migration is never repeated merely because the outer deployment job did not report success.
Report the artifact and effective configuration established on each target, the available test result and any unresolved state. The recipient can then decide whether the available runtime supports its next use.
SYSE.41:6 - Bias-Annotation
Deployment tools encourage attention to their own job state. That can hide mixed runtime configuration, unknown targets and a successful copy that was never loaded. Inspect the actual process and user-relevant test, including the continued validity of the fallback.
SYSE.41:7 - Conformance Checklist
- Before changing a target, the applicable artifact qualification, target and configuration, dependencies, data conditions and permissions are established. A missing prerequisite is stated for the operation it prevents.
- The starting state is established or its uncertainty stated. Any usable previous configuration and its return conditions are identified.
- The bounded procedure names relevant ordering and partial/replay behavior.
- For each target credited with successful deployment, observations establish the running artifact and effective configuration.
- For a claimed successful deployment, an applicable test covers the named use under permitted conditions. A missing test limits that claim.
- After partial failure, the selected continuation is authorized and qualified for the states still possible. A stateful return stays within its qualified boundary.
- The result states its interval, targets and limits without granting wider release authority.
SYSE.41:8 - Common Anti-Patterns and How to Avoid Them
| Misuse | Repair |
|---|---|
| Treat artifact promotion as deployment completion. | Establish which artifact and configuration are running and what the applicable deployment test shows. Recover missing evidence where possible. Deploy or correct the runtime when the required state is absent and the operation is authorised. |
| Report requested configuration as effective configuration. | Query what the running process uses. |
| Restart the whole deployment after a timeout. | Resolve effects that could make the next step unsafe. Repeat only steps qualified for the remaining states and permitted on the target. |
| Roll back binaries after incompatible data changes. | Establish the data recovery or forward repair available before relying on return. |
SYSE.41:9 - Consequences
The pattern makes partial deployment visible and supports safer continuation or return. A successful deployment claim needs applicable runtime observations and test results. A return to prior service needs qualified fallback conditions; missing evidence limits the corresponding conclusion. Some deployment mechanisms need improvement before they can distinguish copied, loaded, failed and unknown states.
SYSE.41:10 - Rationale
Deployment creates or changes the target runtime. Moving an artifact or declaring desired state supplies only part of that result. Observation and a bounded task test establish what the change produced. Separating that result from application acceptance and release preserves the reach of its evidence and authority.
SYSE.41:11 - SoTA-Echoing
For making tested software usable in a runtime, adopt and adapt DORA Deployment automation and the Continuous Delivery pipeline. Common procedures, same-artifact promotion, environment configuration and deployment tests come from these approaches. Sections 4.2–4.7 connect those operations to target-specific intermediate states and qualified recovery.
Compare rolling replacement, a separate candidate pool and planned interruption using coexistence, capacity, activation effects and recoverability. Repetition and order independence are capabilities to construct where needed, not properties granted by an automation tool. Actual readback and a retained operation/result relation cost engineering effort but permit a defensible continuation after lost acknowledgement.
Reopen the arrangement when configuration loading, target identity, provider behavior, data compatibility or exposure effects change. A scheduled worker can begin live work at startup; separation of deployment and exposure must be established for the architecture in use. A passed deployment test supplies its bounded runtime claim and no wider release authority.
SYSE.41:12 - Relations
SYSE.32 supplies artifact-specific promotion evidence, SYSE.13 identifies target/configuration, SYSE.33 supplies reconstructible environment conditions and SYSE.34 supplies data compatibility/recovery. SYSE.11 uses actual integration observations for bounded usability; SYSE.35 governs exposure evaluation and SYSE.14 the release decision. SYSE.36 observes the deployment task, while SYSE.38 handles an active failure.