Skip to main content
You use lakehouse pipelines to keep copied or derived tables available for analysis. A sync copies source data into a namespace. A transform project uses SQL models to produce derived tables in a target namespace. Use their asset pages to review runs, check freshness, and recover when repeated failures pause scheduled work.

​
Build a transform project

You need edit access to the target namespace to create a project from it. Transform projects are Beta.
  1. Open the target namespace and select Create transform project.
  2. In Models, add or edit your SQL model files. Set their sources and output definitions for the tables you need.
  3. Use Preview to inspect a sample result without building outputs.
  4. Select Check to capture and validate the draft. This checks model blocks, sources, and the target namespace; it does not run the models.
  5. Select Plan to review what a build changes. Inspect the changes before selecting Apply plan on the plan’s card in the Build panel. It appears for the newest successful plan when that plan has changes to apply.
  6. Alternatively, select Build to check, plan, and apply the draft in one workflow. Review the run details to confirm publication.

How to plan and publish a transform, then resume a project paused after repeated failures

  1. Check the draft
  2. Plan the change
  3. Apply the plan
  4. Review what published
  5. Read why it paused
  6. Resume the project
  7. Run it now

​
Add publication checks

  1. Add SQL audit files under audits/. An audit query returns rows that violate a rule. Attach the audit by name in a model’s audits field.
  2. Add YAML fixture tests under tests/ to define input rows and expected query output for a model.
  3. Declare grain in the model when a set of columns defines a unique row. The project adds checks for nulls and duplicate combinations on those columns.
  4. Run Plan or Build and review test and audit results. Fix failed fixture tests or blocking audits before publishing.
Fixture tests run during planning and applying. Blocking audit failures prevent output publication. A successful preview does not prove these checks pass.

​
Check freshness

  1. On a sync, select Edit definition and set Freshness target (hours). Select Save definition. If you leave the target empty, a scheduled sync uses twice the schedule’s longest gap.
  2. On a transform project, set Freshness target and Default schedule in its configuration. Build the changed draft to apply it to the deployment.
  3. Review Freshness and the Stale badge. Sync freshness measures time since a successful run copies rows. Transform freshness measures time since output publication.
For transforms, a default schedule overrides the schedule implied by the freshness target. A model’s own cron takes precedence over the default schedule.

​
Resume after repeated failures

  1. Open the paused sync or transform project and read its failure reason and run details. By default, a project pauses after five failed runs in a row.
  2. Fix the source connection, permissions, SQL, or publication check that fails.
  3. Select Resume. This resets the failure count and allows the schedule to run again. A transform’s next scheduled run is due right away. A sync waits for its next scheduled time unless you choose Run now.
  4. Use Run now on a sync or a built transform project to check the recovery.

​
Good to know

  • Readers can inspect shared pipeline assets. Editors operate them. Publishing transforms also requires source namespace access and edit access to the target namespace.
  • Failure pauses hold manual and scheduled runs until you resume.
  • Transform runs execute the deployed revision. Editing a draft does not change scheduled work until you build or apply it.
  • A completed computation does not by itself confirm publication. Review the publication outcome before using new outputs.