workflow#

hydromodpy.workflow provides composable steps and an immutable PipelineState payload that flows from one step to the next. The Project facade and every workflow launcher (simulation, calibration, batch, comparison, testbed) reuse these steps so the same TOML always drives the same lifecycle.

Sub-modules#

  • workflow/internals/state.py – PipelineState[TPayload] generic dataclass, immutable, specialised by step (ValidatedState, ResolvedState, GeographicState, LoadedState, MeshedState, SetupState, OpenStoreState, SolverRanState, ExtractedState, DerivedState, ExportedState). Transit through state.advance(...).

  • workflow/internals/step.py – Step[TIn, TOut] Protocol (name, run(state) -> state, optional config_sections).

  • workflow/tracking/journal.py – WorkflowJournal, the DuckDB journal of executed steps; workflow/tracking/resume.py and workflow/tracking/heartbeat.py sit beside it. There is no checkpoint store and no pickle layer: resume reads artefacts, see Pipeline resume from artefacts.

  • workflow/internals/dependencies.py – earliest_affected_step resolves the first pipeline index affected by a dotted-path override (used by calibration to skip shared phases).

  • workflow/internals/derived.py – DerivedRegistry for ordered post-extraction calculations.

  • workflow/internals/manifest.py – ResolvedRunManifest, the step blueprint plus config digest written under .hmp/checkpoints/<run_id>/resolved_manifest.json and verified on resume.

  • workflow/runner.py – Pipeline runner that chains steps, manages the journal, and raises typed errors (StepError exposes step_name and run_id).

  • workflow/steps/ – the canonical step library.

  • workflow/pipelines/ – pre-assembled pipelines for the standard workflows.

Twelve canonical steps#

Defined under workflow/steps/ and re-exported from workflow/steps/__init__.py:

ValidateStep, ResolveStep, LoadDataStep, BuildGeographicStep, BuildMeshStep, SetupProcessStep, PrepareSolverStep, RunSolverStep, ExtractStep, DeriveStep, DisplayStep, ExportStep.

DisplayStep runs before ExportStep: figures are the last reader of a run, so they draw from the open store while the intermediate fields are still there. ExportStep then drops what reconciliation forced on (the per-cell budget) and seals the store.

Step contract#

class XyzStep:
    name: ClassVar[str] = "xyz"
    tin: ClassVar[type] = InputPayloadType
    tout: ClassVar[type] = OutputPayloadType
    config_sections: ClassVar[tuple[str, ...]] = ("flow.param",)

    def run(self, state: PipelineState) -> PipelineState:
        ...
        return state.advance(
            step_index=...,
            step_name=self.name,
            payload=...,
        )

The config_sections annotation declares which TOML subtrees the step reads. earliest_affected_step walks every step’s declarations to decide which phases can be skipped per calibration trial.

Key public symbols#

  • hydromodpy.workflow.internals.state.PipelineState

  • hydromodpy.workflow.internals.step.Step

  • hydromodpy.workflow.internals.manifest.ResolvedRunManifest

  • hydromodpy.workflow.internals.dependencies.earliest_affected_step

  • hydromodpy.workflow.internals.derived.DerivedRegistry

  • hydromodpy.workflow.runner.Pipeline

  • hydromodpy.workflow.steps.{ValidateStep, ResolveStep, ..., DisplayStep}

The v1 StepsLedger module is removed; workflow steps are now recorded in the project index (.hmp/index.duckdb) under the workflow_steps table (see Storage Layout).

Layer-matrix neighbours#

  • Allowed targets: core, schema, config, physics, data, spatial, simulation, solver, calibration, results, display, analysis, reporting and workflow.

  • Allowed sources: project and cli.

See also#