fpf.tools
Library / First Principles Framework (FPF) - Core Conceptual Specification
Jump to passage
In this reading
Source · lines 27081–27092

Link to current text

Published source confirmed at last check

Source changed 2026-10-03 16:02:47 UTC · snapshot created 2026-10-03 16:03:51 UTC · last check 2026-10-03 16:25:20 UTC

A.15.1:6.2 - ETL pipeline (parallelism and retries)

  • Top work occurrence: ETL_Nightly_2025-08-11T01:00-01:47.
  • Actual method and containing-system relation: enactsMethod -> Nightly_ETL_Load@DataOps-2025; ETLWorkOccursWithinPlatformBoundary@WarehousePlatform(ETL_Nightly_2025-08-11T01:00-01:47, DataPlatform_Prod) obtains under the platform delimitation declared in ETLWorkBoundaryRelations@WarehousePlatform.
  • Dataset participation; resource stop: relation specification ETL-DATA-REL-2025, owned by ETLDataUseRelations@WarehousePlatform, declares SourceDatasetParticipatesInETLWork(dataset, work, extent) and DestinationDatasetParticipatesInETLWork(dataset, work, extent). The stipulated job facts make the first obtain for RawOrders_2025-08-11, ETL_Nightly_2025-08-11T01:00-01:47, and its extent, and the second for WarehouseOrders_2025-08-11 with the same Work and extent. Neither predicate means later analytics use or dataset transformation. The case names no direct cluster-time or storage-use predicate, so those optional claims return missing-governor[ETL-RESOURCE-USE].
  • Actual change, no connection yet: A.3.4 identifies WarehouseOrders_LoadTransformation_2025-08-11 as the bounded change of the exact dated warehouse partition across 01:00–01:47 under the declared source-snapshot and partition-write conditions. Before that boundary the partition lacks AcceptedOrdersRowSet_2025-08-11; after it, the project data-state relation to that row set obtains. This fixture declares neither a direct W-to-T predicate nor a complete A.6.RCD disposition-2 claim with a constructor, governed base predicates, participants, and case facts, so it returns missing-governor[ETL-WORK-TO-CHANGE]. Keep the Work, transformation, and dataset-participation facts; shared time, destination label, and post-state do not connect W to T.
  • A.13 Agent basis, performer System, and assignment: ETL_Runtime is independently admitted as the exact running System. BatchExecutionControllerSystemRole is a declared local agential kind whose membership criterion requires condition-sensitive regulation of the stated batch action against the batch completion and failure policy. The runtime’s scheduler, authoritative state, policy branches, retry/redirect behavior, and stop conditions support its classification under that local criterion at this grain; no Grade or autonomy-profile claim is used. TransformerRuntimeAssignment is the direct assignment species; ETL_Runtime_TransformerAssignment_2025-08-11 is the obtaining occurrence with holder ETL_Runtime, assigned-kind value BatchExecutionControllerSystemRole, and an extent covering the ETL interval. Actual trace facts then support the dated ETL action and Method enactment; F.6 relates that Work to the same assignment. Code, a model artifact, the assignment alone, or a successful output does not supply agency or Work.
  • Parallel parts: Extract_A ‖ Extract_B; Transform starts when either completes (overlap).
  • Retry: WarehouseWriteAttempt-1 ended at 01:36 without satisfying WarehousePartitionWriteComplete@ETL-2025; WarehouseWriteAttempt-2 then entered to satisfy that same condition for the same dated partition and source snapshot with a smaller batch. Local declaration ETLRetryRelations@WarehousePlatform defines RetriesWarehousePartitionWrite(later, earlier) over <U.Work, U.Work> by exactly those facts and permits one immediate failed predecessor. The relation obtains for the two named attempts; a generic retryOf token is not used.
  • B.1.4 temporal roll-up: ETLSLACoverageAggregation-2025 uses hull under ETLSLAHullPolicy-2025; ETLClusterUtilizationAggregation-2025 uses union under ETLClusterUnionPolicy-2025. They consume the named Work-part intervals and do not establish dataset participation, resource use, or change.
  • B.1.6 resource roll-up: no compute or storage aggregate is asserted until the ETL project declares cluster-time and storage-use predicates and supplies the exact Work, resource, value, unit, and extent participants. Until then return missing-governor[ETL-RESOURCE-USE]; the two dataset-participation relations remain valid.