Sharing Workflows and Outputs

Introduction

Handing off analysis is usually painful: you zip up a folder of data files and a separate script, and the person on the other end has to figure out which file came from which step and how to regenerate them. Because a oryxflow task bundles the code, its parameters, and its output together, you can share the whole reproducible pipeline — the recipient runs the tasks that still need running and loads any result by name, no manual file-shuffling.

Common cases where you want to do this:

  • data engineers share cleaned, ready-to-use data with data scientists — without re-sending it every time it changes

  • vendors sharing data with clients — as a pipeline the client can re-run, not a one-off dump

  • teachers sharing data with students — everyone starts from the same reproducible outputs

oryxflow gives you three approaches, from simplest to most advanced — this page covers them in order:

  1. Share the data folder — version data/ with Git LFS (optionally split by env= so you hand off only what you mean to). The recipient clones and gets the exact outputs.

  2. Share code-free stubs — FlowExport hands over the outputs as loadable tasks without the run() code that produced them, for when the logic is private.

  3. Bridge separate flows — attach_flow lets one flow read another flow’s outputs at run time, across projects or environments.

Sharing the data itself: Git LFS

Because oryxflow writes every task output under data/, the simplest way to share results is to version that folder alongside your code with Git LFS. Then you version and share your data as easily as your code: a teammate clones the repo and gets the exact datasets each run produced, so nobody re-runs the expensive tasks just to obtain outputs someone already computed. This is the recommended approach for most teams.

The Claude Code plugin sets this up for you in one step with /oryxflow:init-gitlfs (it puts data/ under Git LFS and wires the .gitattributes). To do it by hand, install Git LFS and track the data directory:

git lfs install
git lfs track "data/**"
git add .gitattributes data/
git commit -m "Track data outputs with Git LFS"

Tracking data/** also picks up the small .oryxflow-code-status.json record oryxflow keeps inside the data directory (the code fingerprints behind Automatic code invalidation), so it travels with the outputs it describes — move or restore a data directory whole and the freshness information comes along. The separate .oryxflow/ event log (run history) is high-frequency exhaust, not a shared output: keep it gitignored and export history deliberately if a teammate needs it.

The next section refines this — splitting data/ by env= so you share only part of it. The later Export/Import and Attach sections cover the cases Git LFS doesn’t: handing off the task code to another project, or reading one flow’s outputs from inside another flow.

Separating environments with env=

Often you don’t want to share everything under data/ — production outputs, a colleague’s scratch experiments, and your own dev runs may all live there, and only some of it is worth handing off. The env= argument to Workflow keeps them in separate subfolders so you can be selective about what gets shared (Git-LFS-track or commit just the one you mean to).

Passing env='prod' writes all task output under data/env=prod/ instead of data/:

flow = oryxflow.Workflow(TaskTrain, env='prod')
flow.run()          # output now under data/env=prod/
# a separate dev environment, isolated from prod:
flow_dev = oryxflow.Workflow(TaskTrain, env='dev')
flow_dev.run()      # output under data/env=dev/

Because each environment is its own directory, you can share just the one you mean to — commit data/env=prod/ and leave data/env=dev/ out — without the environments overwriting each other or leaking. When you later read these outputs (a plain import or FlowImport, both below), pass the same env= to point at the environment you want:

flow_prod = oryxflow.Workflow(TaskTrain, env='prod')
df = flow_prod.outputLoad()     # reads data/env=prod/

Just make sure the environment you point at is the one whose outputs actually exist — reading env='prod' when the shared data was saved without an env (plain data/) will find the tasks incomplete and rerun them.

Sharing outputs without the code (FlowExport)

The point of exporting is to share output data without sharing the code that produced it. Often the run() logic is the sensitive part — a proprietary model, a paid data source, an internal cleaning routine — but the output is what a colleague, client, or student actually needs. FlowExport lets you hand over the results as a first-class oryxflow flow while keeping that logic private.

It works because the exported file contains only stub task definitions. For each task it emits the parent class (so the output format is known), the persists names, the path, the task_group, and the parameters — everything needed to locate and load the output — plus external=True, which tells oryxflow to treat the output as already-produced and never run the task. The original run() body is not included. The recipient wires the stubs into a Workflow (see Loading shared outputs in your project below) and calls outputLoad() to work with your results through oryxflow — parameter management and all — without ever seeing how they were made.

You can Export your tasks into a new File or print the tasks in the console. All parameters, paths, task_group will be exported.

class Task1(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(...)   # your private logic — NOT exported

@oryxflow.requires(Task1)
class Task2(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(...)   # your private logic — NOT exported

flow = oryxflow.Workflow(Task2)

# This will only export Task 2 to console
e = oryxflow.FlowExport(tasks=Task2())
e.generate()

# This will export All the flow (Task1, Task2) to a file
e = oryxflow.FlowExport(flows=flow, save=True, path_export='tasks_export.py')
e.generate()

The generated tasks_export.py holds stubs like this — note there is no run():

import oryxflow
import datetime

class Task1(oryxflow.tasks.TaskPqPandas):
    external=True
    persists=['data']

class Task2(oryxflow.tasks.TaskPqPandas):
    external=True
    persists=['data']

Ship this file together with the data/ directory (the Git-LFS approach above is an easy way), and the recipient can load every output through oryxflow without the source code.

Loading shared outputs in your project

The simplest way to use tasks_export.py is to drop it into your project next to the shared data/ directory and import it like any other module. The stubs are ordinary task classes, so you use them in a standard Workflow — point oryxflow at the data directory, then load:

import oryxflow
import tasks_export                     # the stub file you were given

oryxflow.set_dir('data/')               # the shared data directory

flow = oryxflow.Workflow(tasks_export.Task2)
flow.complete()                         # True — external stub sees the existing output file
df = flow.outputLoad()                  # load the results, no producer code needed

Because the stubs are external=True, oryxflow treats their output as already-produced: nothing runs, complete() is True as long as the output files are in place, and outputLoad() just reads them. If the tasks have parameters, pass them as usual to select which output you want (oryxflow.Workflow(tasks_export.Task2, {'country': 'US'})) — the same parameter → path resolution applies, so you never chase file paths by hand.

Loading shared outputs from another project (FlowImport)

The plain import above assumes tasks_export.py and its data/ sit inside your project. When the file and data live in another directory or project, FlowImport loads the module and resolves its data path for you, so you don’t have to copy anything in:

scraper = oryxflow.FlowImport(path='../another-project/', module='tasks_export.py', path_data='data/')
flow_import = oryxflow.Workflow(scraper.tasks.Task2, path=scraper.dirpath)
df = flow_import.outputLoad()

FlowImport returns an object exposing the imported task classes under .tasks and the resolved data directory under .dirpath; you pass that dirpath as the workflow’s path so the flow reads from the other project’s data/. It’s the same idea as the plain import — just without moving files between projects.

Reading another flow’s outputs (attach_flow)

In more complex projects, users need to import data from many sources. Flows can be attached together in order to access the data generated in one flow inside the other.

Attach a flow to a workflow with attach_flow(flow, name); every task in that workflow then sees it under self.flows[name], so a task’s run() can load another flow’s output without wiring an explicit requires() dependency.

requires vs attach_flow: the defining difference

The two mechanisms look similar — both let one task read another’s output — but they solve different problems, and the difference comes down to two things: whose data root the read uses, and whether oryxflow tracks it.

A normal @oryxflow.requires edge assumes both tasks live in one flow with one data root. When you build a Workflow, it pushes its path (and env) down onto every upstream task instance — that is how the whole DAG agrees on where data/ is. oryxflow then tracks that edge: if the upstream output is missing, it builds it.

attach_flow is the opposite on both counts:

  • the attached flow is a self-contained handle that keeps its own path/env, so self.flows['x'].outputLoad() reads from that flow’s data location — which can be a different project’s data/ or a different env=;

  • it is not tracked as a dependency — oryxflow will not build the source flow for you.

So the mental split is: requires = “this is part of my pipeline, same data root, build it if needed”; attach_flow = “reach into a separate flow that has its own lifecycle and data root, and read what it already produced.”

Why not just load it yourself?

The obvious alternative is to load the other flow’s output in your driver script and pass the DataFrame into your task. Don’t — it throws away the one thing oryxflow is for. In oryxflow an output’s location is a deterministic function of (task, parameters): you never type a path, you ask for a result and the engine computes where it lives. Hand-loading reintroduces exactly the path bookkeeping the library removes, and it gets worse the moment the source flow is parameterized:

  • a source with no parameters — you already have to know or hardcode one path;

  • a parameterized source — now there is a path per parameter combination, and to pass the right DataFrame you would have to reconstruct the source’s task_id → path mapping yourself for every parameter set, and keep it in sync as parameters change. That is reimplementing oryxflow’s parameter management by hand, in glue code.

attach_flow avoids all of this. Because you attach the flow object, not its data, the “ask by parameters, not paths” property survives across the flow boundary — the attached flow resolves its own output paths from its own parameters:

self.flows['scraped'].outputLoad(Task2)                 # single flow: its parameters
self.flows['scraped'].outputLoad(Task2, flow='2024')    # WorkflowMulti: pick a parameter set by name

You never touch a path, parameterized or not, cross-project or not.

class Task1(oryxflow.tasks.TaskCachePandas):
    def run(self):
        self.save(pd.DataFrame({'a': [1, 2]}))

class Task3(oryxflow.tasks.TaskCachePandas):
    def run(self):
        # read the attached flow's default-task output
        temp_flow_df = self.flows['flow'].outputLoad()
        self.save(temp_flow_df)

# Define both flows and run the source flow first so its output exists
flow = oryxflow.Workflow(Task1)
flow2 = oryxflow.Workflow(Task3)
flow.run()

# Attach the first flow to the second under the name 'flow', THEN run
# (attachment is propagated to the tasks when flow2.run() executes)
flow2.attach_flow(flow, 'flow')
flow2.run()

df = flow2.outputLoad()   # Task3's output, sourced from the attached flow

Step by step, here is what actually happens:

  1. flow.run() executes Task1 and saves its output (here into the in-memory cache, because Task1 is a TaskCachePandas). At this point flow is a live Workflow object that knows how to load that output — flow.outputLoad() would return the DataFrame.

  2. flow2.attach_flow(flow, 'flow') records the whole flow object — not its data — under the key 'flow' on flow2. Nothing runs yet; you are just registering that flow2 may need to reach into flow later.

  3. flow2.run() propagates that registration onto the task instances just before executing them: every task in flow2 gets self.flows = {'flow': flow}. This is why the attach must come before the run — a task created and run without it would have self.flows empty.

  4. Inside Task3.run(), self.flows['flow'] is the attached flow object, so self.flows['flow'].outputLoad() calls outputLoad() on it and returns Task1’s DataFrame. Task3 then saves it as its own output.

The key idea is that the link is between flows, resolved lazily at load time — Task3 never declares Task1 in its requires(). That keeps the two flows independent (separate projects, separate data/ directories, separate run/reset scopes) while still letting one read the other’s results. Crucially, unlike a requires edge, the attached flow keeps its own path/env, so self.flows['flow'].outputLoad() reads from that flow’s data root — which may be a different project’s data/ or a different env=. That is what makes it the right tool for the cross-project case above: import another project’s tasks, wrap them in a Workflow pointed at that project’s dirpath, attach it, and read its outputs by parameter without ever resolving a path.

# another project's flow, pointed at ITS data root (see FlowImport above)
scraper = oryxflow.FlowImport(path='../another-project/', module='tasks_export.py', path_data='data/')
flow_prod = oryxflow.Workflow(scraper.tasks.Task2, path=scraper.dirpath, env='prod')

# your flow whose task consumes the scraped data
flow_mine = oryxflow.Workflow(MyTask)
flow_mine.attach_flow(flow_prod, 'scraped')
flow_mine.run()          # inside MyTask.run(): self.flows['scraped'].outputLoad(Task2)

Why not just @oryxflow.requires(scraper.tasks.Task2) on MyTask? Because flow_mine would push your path onto the imported task, so it would look for the output under your data/ instead of ../another-project/data/ — find it missing, and try to rebuild it (needing that project’s raw inputs and code). attach_flow sidesteps that precisely because the attached flow retains its own path.

Two trade-offs to know:

  • Not tracked as a dependency. oryxflow will not build the source flow for you: if the output doesn’t exist yet, self.flows['flow'].outputLoad() raises rather than running it, so you must run the source flow first (as flow.run() does above).

  • Attached at a fixed parameter slice. attach_flow hands you the source flow configured at attach time — clean when the consumer wants a fixed slice (“the prod-2024 scrape”). If instead you want each instance of a parameterized consumer to automatically pull the matching parameter set from the source (consumer country=US → source country=US), that per-instance coupling is what a real @oryxflow.requires edge does best — parameters propagate upstream automatically and it is tracked and auto-built.

Decision rule:

  • Same project, and the upstream should follow the consumer’s parameters → @oryxflow.requires (parameters propagate, tracked, auto-built). Don’t reach for attach_flow to wire tasks that belong to one pipeline.

  • Separate data root (another project or another env ), or an independently-managed flow → attach_flow, which keeps parameter → path resolution across the boundary — the thing manual DataFrame passing destroys.