Skip to content
Project number
Project / 018
Year
2026
Category
Data Engineering
Status
Active

FORMA

Repeatedly writing and debugging ETL logic made me wonder whether working with messy data could be more direct without removing engineering control. FORMA explores that idea as a visual data workbench.

Origin
Work problem → Engineering tool
Role
Designer and developer
Status
Active
Stack
TypeScript · React · React Flow · Python (pandas)

01 /The problem

Messy data, scattered tools.

Data engineering repeatedly requires inspecting, selecting, cleaning, transforming, validating, running and reviewing data through workflows that are often fragmented across code and tools.

  1. 01Work with the data directly

    See the source, select what matters and shape it, with a preview before every change.

  2. 02Build a reproducible pipeline

    Every change becomes a step in a pipeline that runs the same way each time.

  3. 03Keep engineering control

    The pipeline exports as readable Python that produces the same output, so the work is never locked inside the tool.

02 /Core loop

See, select, transform, verify, run, export.

The core loop is the experience of using FORMA: what a person does, in order, with a dataset in front of them.

  1. See
  2. Select
  3. Transform
  4. Verify
  5. Run
  6. Export

03 /From messy input to pipeline

One specification, two engines.

The pipeline is the engineering process underneath. The interface, the in-browser engine and the Python generator all work from the same pipeline specification, and a parity suite checks that both engines agree cell for cell.

  1. Source
  2. Select
  3. Extract
  4. Clean
  5. Transform
  6. Validate
  7. Load
FORMA Sources page: an Excel workbook with three sheets listed by row and column count, and the pipeline that uses it

Sources are stored once and referenced by pipelines. Every sheet of a workbook is its own source.

Real product · Built-in example project and sample data

04 /The workspace

Shape the data where you can see it.

Pipelines open on a canvas. For deeper work, panel workbenches put the source, the data preview, the step inspector, a before and after comparison and the data profile side by side. Selecting a column shows its profile, detected patterns and suggested actions.

  1. FORMA analyst workbench: pipeline steps, a data preview of invoice rows, a before and after comparison, the step inspector and a data profile

    01Analyst view: steps, the data preview, before and after for the selected step, and the data profile.

    Real product · Built-in example project and sample data
  2. FORMA extraction view: the raw spreadsheet in the source viewer, failed rows grouped by issue type with Fix, Ignore and Exclude actions, and the generated Python for the validation step

    02Extraction view: the raw spreadsheet, rows that failed with a reason and an action, and the generated Python for the step.

    Real product · Built-in example project and sample data

05 /Verify before run

Verify before it runs.

Validation is explicit: rules such as not blank, pattern, valid date, numeric bounds, allowed set and unique. Validation health is reported with the quality dimensions behind it, never as an unexplained score. Rows that cannot be transformed or validated confidently are held back instead of being loaded silently.

FORMA monitor view: run history, validation health with completeness, validity, uniqueness and consistency, run logs per step and the failed rows

Monitor view: run history, validation health by dimension, step-by-step run logs and the failed rows.

Real product · Built-in example project and sample data

06 /Run and review

Runs are visible events.

A run reports the source load and each completed step with real row counts and timings. Held-back rows go to a review queue, where each one can be corrected, kept, excluded or ignored. Decisions apply on the next run.

  1. FORMA pipeline canvas after a run: eleven steps from source to destination with row counts and timings, summarised as 1,001 input, 985 ready and 16 to review

    01After a run on the example project: 1,001 rows in, 985 ready and 16 held back for review.

    Real product · Built-in example project and sample data
  2. FORMA review queue: sixteen rows held back with their issue, and a panel to correct the value, keep the original, exclude the row or ignore the warning

    02The review queue for those 16 rows, with the original value and the reason for each.

    Real product · Built-in example project and sample data

07 /Export

Keep the code.

A pipeline exports as a single pipeline.py or a full project with requirements, configuration, a README and the pipeline specification, plus an Airflow DAG or a Prefect flow. The code has one function per step, and its comments match the visual pipeline.

FORMA export: options for a Python script, a Python project, Airflow or Prefect, beside the generated pandas code

Export: the project options beside the generated pandas code.

Real product · Built-in example project and sample data

08 /Current state

Active, with a clear edge.

FORMA is active. Every screen on this page is a capture of the running app on its built-in example project. The loop from source to export works end to end.

  1. Not in this build yet

    Collaboration, comments and approvals; SQL and Polars code targets; branching pipelines; an in-browser Python parity check; lineage and Git integration.

09 /Next project

Job Matching System T2

XGBoost-powered job matching with semantic analysis for accurate candidate–job matching.