Transition to oryxflow

Most data-science code starts as a script: a chain of functions that read a file, transform it, and write the next file, wired together by hand at the bottom. It works until it doesn’t — you change one step and have to remember which downstream files are now stale, you re-run the whole thing (including the slow data pull) just to test a small change, and six months later you can’t tell which parameters produced which output.

oryxflow turns that script into a pipeline of tasks and takes over the bookkeeping. You get three things you were doing in your head before:

• No wasted recomputation — a task that has already produced its output is skipped, so re-running the pipeline only runs what actually changed (a small edit no longer re-pulls the raw data).

• Reproducibility — every output is tied to the task and parameters that produced it, so you always know how a result was made and can reproduce it exactly.

• Automatic parameter management — change a parameter and oryxflow reruns exactly the tasks that depend on it, and keeps the outputs for each parameter set side by side.

• Provenance you can query — oryxflow records what ran, when, and why, and (when you set a code_version) catches the classic trap of editing a task’s code while the cache serves the old output. “Is this stale?” and “did I already run this?” stop being things you track in your head.

Current Workflow Using Functions

Your code currently probably looks like the example below. How do you turn it into a oryxflow workflow?

import pandas as pd

def get_data():
    data = pd.read_csv('rawdata.csv')
    data = clean(data)
    data.to_pickle('data.pkl')

def preprocess(data):
    data = scale(data)
    return data

# execute workflow
get_data()
df_train = pd.read_pickle('data.pkl')
do_preprocess = True
if do_preprocess:
    df_train = preprocess(df_train)

Workflow Using oryxflow Tasks

In a oryxflow workflow, you define your own task classes and then execute the workflow by running the final downstream task which will automatically run required upstream dependencies.

The function-based workflow example will transform to this:

import oryxflow
import pandas as pd

class TaskGetData(oryxflow.tasks.TaskPqPandas):

    # no dependency

    def run(self): # from `def get_data()`
        data = pd.read_csv('rawdata.csv')
        data = clean(data)
        self.save(data) # save output data

class TaskProcess(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True) # optional parameter

    def requires(self):
        return TaskGetData() # define dependency

    def run(self):
        data = self.inputLoad() # load input data
        if self.do_preprocess:
            data = scale(data) # # from `def preprocess(data)`
        self.save(data) # save output data

flow = oryxflow.Workflow(TaskProcess)
flow.run() # execute task with dependencies
data = flow.outputLoad() # load output data

Learn more about Writing and Managing Tasks and Running Workflows.

Tip

The Claude Code plugin automates this transition: describe your existing script in plain language and it creates the task classes and wires the @oryxflow.requires dependencies. See Using oryxflow with Claude Code.

Design Pattern Templates for Machine Learning Workflows

See code templates for a larger real-life project at https://github.com/d6t/d6tflow-template. Clone & code!