Data Today: Matillion

Matillion Maia BigQuery support: setup, cost, limits

2026-06-29T00:00:00Z

Matillion just made Maia much harder for Google Cloud teams to ignore. The June update adds Matillion Maia BigQuery support, which means a team building in the Data Productivity Cloud can now create Maia projects against Google BigQuery instead of treating Snowflake, Databricks, or Redshift as the default center of gravity.

The practical finding is simple: Matillion says Google BigQuery is supported on the Current runner track now, with Stable track availability expected around August 1, 2026, and the first cut is already broad enough to matter: 29 transformation components, 40 orchestration components, and 122 connectors listed for BigQuery projects in the Matillion changelog. The feature was surfaced in Matillion's June 26 New Features Blog, while the underlying Maia changelog records the BigQuery platform support under June 23, 2026.

That gap between Current and Stable is the whole story for operators. You can start testing now if your runner policy allows Current. You should hold production rollout if your shop standardizes on Stable, especially if your pipelines influence finance reporting or customer facing SLAs.

What actually changed for BigQuery teams?

Matillion added Google BigQuery as a supported cloud data warehouse in Maia, which means BigQuery now appears as a selectable data platform when you create a build project. The Google BigQuery projects documentation says the project setup flow includes a Data platform drop-down where you select Google BigQuery, then choose either Maia managed or Advanced settings depending on how you want infrastructure and secrets handled.

The component surface is larger than a cautious preview would suggest. Matillion's 2026 Maia changelog lists 29 transformation components, 40 orchestration components, and 122 connectors available for Google BigQuery data pipelines, with minimum Maia runner version 11.478.0.

Google BigQuery support covers 29 transformation components, 40 orchestration components, and 122 connectors. Source: Matillion Maia 2026 changelog. Data Today benchmark.

The chart above is the important sanity check. BigQuery support is useful because the initial component list includes production staples such as Table Input, Table Output, SQL, Join, Window Calculation, Run Transformation, SQL Script, dbt Core, Cloud Pub/Sub, Query Result to Grid, and Google Cloud Storage Unload. That is enough to build a real warehouse workflow, not just a demo that loads a CSV and waves at governance.

The release track matters more than the launch blog tone. Matillion says BigQuery is available on the Current runner track from the changelog date and that Stable users should expect availability around August 1, 2026 in the same changelog entry. In calendar terms, that is roughly 39 days between the Current availability date of June 23, 2026 and the expected Stable date of August 1, 2026.

Here is the operational read:

Choice	What Matillion documents	Use it when	Cost and risk shape
Current runner track	BigQuery support is available now	You can test new platform support before the Stable window	Faster validation, higher change risk
Stable runner track	BigQuery expected around August 1, 2026	You need conservative runner updates	Slower rollout, lower surprise risk
Minimum runner version	BigQuery component support starts at 11.478.0	You are checking a Hybrid SaaS runner or release policy	Upgrade work may sit on your team

The other items in the June feature bundle, bulk variable management and Designer command palette improvements, are useful background. They are not the lead event. The lead event is that Maia can now build and manage workflows against a warehouse many GCP shops already use as their system of record.

How do you configure a BigQuery project without making security angry?

BigQuery authentication is the first non-obvious piece. Most Matillion warehouse environments put warehouse authentication directly on the environment. BigQuery uses Google Cloud credentials instead, and Matillion's Full SaaS BigQuery setup guide says a Full SaaS BigQuery environment must have access to a Google Cloud service account credential supplied as a configured cloud credential on the environment.

For Full SaaS, your setup checklist is short but sharp:

Create a Google Cloud project.
Create or choose a BigQuery dataset.
Create a Google Cloud service account.
Assign IAM roles that let Maia run BigQuery jobs and touch the required datasets.
Add the service account key as a cloud credential in the Matillion project.
Create a BigQuery environment with a default GCP Project ID and Dataset.

Matillion's setup page includes an example Google Cloud service account key structure. Keep the real key out of Git, tickets, prompt history, and copied pipeline notes. A safe shape for the credential looks like this:

{
  "type": "service_account",
  "project_id": "example-project",
  "private_key_id": "redacted",
  "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
  "client_email": "matillion-sa@example-project.iam.gserviceaccount.com",
  "token_uri": "https://oauth2.googleapis.com/token"
}

For Hybrid SaaS, the better GCP native pattern is runner-assigned credentials when you can use it. Matillion's Hybrid SaaS BigQuery guide says that when the environment has no associated cloud credential, the Maia runner uses Application Default Credentials, with the Google Cloud service account attached to the runner acting as the principal.

That gives you three credential patterns worth comparing:

Deployment pattern	Credential source	What to watch
Full SaaS	Environment cloud credential backed by service account key JSON	Key rotation and allowlisted Matillion access
Hybrid SaaS on GCP with runner credentials	Runner service account through Application Default Credentials	Runner IAM scope, GKE deployment hygiene, least privilege
Hybrid SaaS with environment credential	Environment-associated service account key	More explicit per-environment identity, more key management

The cleanest production version is usually Hybrid SaaS on GCP with runner-assigned credentials, because it avoids distributing service account keys. The catch is operational: your team owns the runner, its GKE deployment, and its IAM posture. Full SaaS is quicker for teams that want Matillion to handle more of the platform layer, but you still own the Google credential blast radius.

Which IAM roles do you really need?

Matillion documents the minimum shape rather than a single magic role. The service account needs permission to create BigQuery jobs, query tables and views, retrieve metadata, list projects and datasets, insert or load data, and create, update, or delete tables and views when your pipelines do that work.

The Hybrid SaaS setup guide says you must grant either roles/bigquery.jobUser or roles/bigquery.user at minimum because both include bigquery.jobs.create, and it lists roles/bigquery.dataEditor for read and write workflows plus roles/bigquery.dataViewer for read access.

A least-privilege starting point for a transformation project that reads and writes a single dataset looks like this in intent:

gcloud projects add-iam-policy-binding example-project \
  --member="serviceAccount:matillion-sa@example-project.iam.gserviceaccount.com" \
  --role="roles/bigquery.jobUser"

Then scope dataset access separately in BigQuery for the dataset Maia should read or write. If a pipeline stages data in Google Cloud Storage before loading to BigQuery, Matillion's setup guide lists common Storage roles: roles/storage.objectViewer for reading staged files, roles/storage.objectCreator for uploads, and roles/storage.objectAdmin for full object access.

The mistake to avoid is granting roles/bigquery.admin because the first setup test failed at 5:30 p.m. That role appears in Matillion's documented list as full administrative access, but broad admin rights turn an AI-assisted pipeline builder into a very well credentialed foot-gun. Start narrow, run the pipeline, read the failure, then add the missing permission deliberately.

What does it cost when Maia works against BigQuery?

There are two bills to separate: Matillion consumption and Google Cloud consumption. BigQuery support does not make SQL free. Maia can help build or modify the workflow, but BigQuery still charges for the warehouse work it performs under your Google Cloud account.

Matillion exposes consumption through its REST API. The Maia API overview says the API base URLs are https://eu1.api.matillion.com/dpc for EU accounts and https://us1.api.matillion.com/dpc for US accounts, and it documents a 1000 requests per minute fixed API rate limit that returns HTTP 429 Too Many Requests when exceeded.

A FinOps check for Maia pipeline usage starts with the consumption endpoint, then you join that against BigQuery job history in your own GCP project:

curl --request GET \
  --url "https://us1.api.matillion.com/dpc/v1/consumption" \
  --header "Authorization: Bearer $MATILLION_TOKEN"

Matillion's 2026 changelog shows recent work on the consumption schema: GET /v1/consumption added pipelineName in January, changed results.consumption.credits to a number in April, and added an environment field for orchestration and transformation consumption in May. That matters because BigQuery cost control depends on attribution. A monthly credit total without pipeline and environment is a blame smoothie.

What I would track for the first 30 days:

Matillion credits by pipelineName and environment, pulled from GET /v1/consumption.
BigQuery bytes processed by job, pulled from Google Cloud's job metadata or billing export.
Runner track and version, especially if you are testing on Current before Stable.
Connector choice, because a full load connector and an incremental load connector can have very different downstream BigQuery costs.

The cheap mistake is letting Maia generate a correct but expensive pattern: full refresh, wide table scan, write-back to a large table, repeat hourly. Ask Maia for the pipeline, then inspect the generated SQL and component choices like you would review a pull request from a very fast junior engineer.

For teams already watching Matillion spend, the same discipline applies here as in our guide to Context Engine setup and costs: ground the AI feature in observable consumption before you put it behind a schedule.

When should you use Maia for BigQuery, and when should you wait?

Use it now for migration proofs, greenfield GCP projects, connector evaluation, and internal productivity tests. A data engineer can ask Maia to build the first pass of a BigQuery transformation pipeline, wire in Table Input, Filter, Join, SQL, and Table Output, then review the generated shape in Designer before committing.

A good first prompt is specific about the warehouse defaults and the cost guardrails:

Build a BigQuery transformation pipeline in the dev environment.
Read analytics.orders and analytics.customers.
Filter orders to the last 7 days.
Join on customer_id.
Write the result to mart.recent_customer_orders.
Avoid full scans where a partition filter is available.

Wait for Stable if the pipeline is regulated, high volume, customer facing, or already fragile. The Current track is the right place to learn how Maia behaves with your BigQuery datasets. Stable is the better place to run the job that wakes up your CFO if the numbers drift.

Also wait if your IAM model is still mushy. BigQuery support puts Maia closer to a powerful production surface: jobs, datasets, tables, views, GCS staging, and metadata. That is exactly where service account discipline matters.

The strongest use case is a GCP team that already has BigQuery, has a small data engineering group, and needs more pipelines than the team can hand build. Maia can remove some canvas work and boilerplate SQL. It cannot remove your obligation to validate lineage, permissions, costs, and data contracts.

Is this the BigQuery moment for Maia?

Yes, for GCP shops that were waiting for Maia to meet them where the data already lives.

The bigger point is less glamorous: BigQuery support moves Maia from an interesting AI layer to an operational option for another major warehouse camp. The release is broad on components, early on runner track, and very dependent on IAM hygiene. Treat it like a new production platform surface, not a magic button. The builders who win with it will be the ones who let Maia draft the work and make their review process boring.

Sources

Matillion anomaly alerts: runtime drift without polling

2026-06-22T00:00:00Z

The annoying pipelines rarely fail first. They usually get weird first: a 12 minute load turns into 19 minutes, an iterator step crawls through one customer after another, or a transformation finishes suspiciously fast because it processed almost nothing. That gray zone is where Matillion anomaly alerts now matter.

Matillion has added duration anomaly notifications to Maia, its AI layer for the Matillion Data Productivity Cloud. The important number is 10 successful runs: Matillion says a pipeline needs at least 10 successful runs before anomaly alerts can be generated, and Maia can use up to 300 successful runs to build the duration baseline in its pipeline notifications documentation. This is a practical observability feature, not magic failure prediction. It gives you an operational warning when runtime drifts from the pipeline's own history.

That distinction matters if you influence the bill. Duration is one of the cleanest early signals that something changed: source volume, warehouse queueing, connector behavior, branch changes, environment configuration, or a runner problem. An alert that fires while a slow run is still executing can save compute before the end state shows up as a red failure, a missed SLA, or a larger credit line item.

What exactly changed in Maia pipeline observability?

Matillion's 2026 changelog says you can now subscribe to anomaly alert notifications through Email, Slack, or Webhook when a pipeline run's duration deviates significantly from historical baselines, with the feature listed under Pipelines, Observability, and a minimum Maia version of all supported versions in the June 17, 2026 changelog entry. The public roadmap post on June 19 bundled this with iterator visibility and GitLab API support, but the duration alert is the operationally heavier change because it turns pipeline drift into a push signal.

The earlier Maia observability flow already had anomaly indicators in Pipeline Runs: red arrows beside duration metrics, a hover tooltip, and an anomalies section on the run detail. Matillion's troubleshooting guide gives the example of a run being 7.8 percent longer than expected in its Maia AI Agents troubleshooting documentation. The new subscription path is the difference between noticing a red arrow when you happen to check the dashboard and getting paged into the right Slack channel before the run quietly burns through the morning.

The feature only applies to scheduled or API-driven pipeline runs. Matillion says notifications are sent for scheduled or API-driven runs, while pipelines run manually through Designer do not generate notifications in the Pipeline notifications guide. That is the right boundary. Manual runs are often experiments. Scheduled and API-triggered runs are production behavior.

Here is the baseline math Matillion documents, with the two numbers you should remember before you file a ticket about missing alerts.

Source: Matillion pipeline notifications documentation. Maia needs 10 successful runs before anomaly alerts can be generated and uses up to 300 successful runs to build the duration baseline. Data Today benchmark.

The chart above is blunt on purpose: 10 successful runs are the floor, and 300 successful runs are the maximum recent history Maia uses for the baseline. If you deploy a new production pipeline and schedule it hourly, anomaly alerts cannot reasonably help you on the first day. If you schedule it once per day, give it at least 10 successful days before you expect the alerting model to settle.

How does Maia decide that a runtime is abnormal?

Maia compares each pipeline against its own recent duration history. Matillion says the expected range is built from successful runs of that pipeline, centered on the median duration, with spread measured toward the 99th percentile on the slow side and the 1st percentile on the fast side in the anomaly alert calculation notes. That is a useful design choice because a pipeline that usually varies between 8 and 40 minutes should not alert the same way as a pipeline that reliably finishes between 58 and 62 seconds.

The slow side is the one most teams will care about first. Matillion says a slow anomaly can be detected while a pipeline is still running, while a fast anomaly is detected only at completion in the duration anomalies section. That means Maia can warn you before the run ends when the elapsed runtime crosses the expected upper bound. For cost control, that timing is the feature.

Fast anomalies deserve more respect than they usually get. A pipeline finishing in one tenth of its normal time can mean a source delivered zero rows, a branch skipped the real work, an environment variable pointed to a smaller schema, or an upstream extract silently changed. Matillion says the lower bound is the median minus twice the downward spread and never drops below zero, which means many pipelines will never trigger a fast anomaly unless their normal runtime is long and consistent in the same expected range explanation. In plain builder terms: do not count on fast alerts for every pipeline. Count on them for stable, long-running jobs where “too quick” is actually suspicious.

The model is intentionally per-pipeline. That avoids the lazy alerting pattern where every job inherits a single global SLA and the useful signals drown in noise. A five minute Salesforce Load and a two hour warehouse transformation should have different ideas of normal.

How do you configure the alert without wiring every pipeline by hand?

Configuration lives in Notifications, not on every component canvas. Matillion says registered Maia users can configure pipeline notifications at the project and environment level for completion alerts and anomaly alerts in the notifications overview. That is the right place for it, because alert routing is an operational concern. You want production to hit a high-priority channel and development to stay in a sandbox channel.

For email anomaly alerts, the setup is short: open the Profile and Account menu, choose Notifications, add an anomaly alert, select Enable anomaly alerts, choose Email, and add the notification. Matillion states that email alerts go to the email address associated with your Maia account and need no extra setup in its email setup instructions.

Slack needs a webhook URL and a name. Matillion lists Slack Webhook URL and Slack Webhook Name as required fields when configuring Slack anomaly alerts in the Slack anomaly setup section. That is simple, but it also means channel design matters. Do not point every environment at the same room unless you enjoy training people to ignore red lights.

Webhook delivery is the most useful option for teams that already route incidents through ServiceNow, PagerDuty, Teams bridges, or internal automation. Matillion says webhook anomaly alerts require a Webhook URL, Webhook Name, and Payload Template in the webhook setup section. The catch is security: Matillion documents these webhooks as outbound only and says their payloads are not signed with HMAC in the same webhook behavior note. Put an API gateway, shared secret, allow list, or other control in front of the receiving endpoint. A naked endpoint that takes operational actions from unsigned JSON is how incident automation becomes incident creation.

A compact webhook payload for a slow anomaly could look like this:

{
  "text": "${pipelineName} duration anomaly: ${elapsedDurationSeconds}s elapsed, upper bound ${expectedUpperBoundSeconds}s",
  "executionId": "${pipelineExecutionId}",
  "environment": "${environmentDisplayName}",
  "startedAt": "${startedAt}"
}

The variable names above come from Matillion's anomaly webhook template support, which includes ${elapsedDurationSeconds}, ${expectedUpperBoundSeconds}, ${expectedLowerBoundSeconds}, ${pipelineExecutionId}, and ${triggerType} in the available variables list. Keep the payload small. The first alert should tell the on-call engineer what is drifting and where to click, not dump a novella into Slack.

Alert path	Trigger behavior	Best use	Hard limit to remember
Email anomaly alert	Sends to the Maia account email when an anomaly is detected	Small teams and low-noise production projects	Requires at least 10 successful pipeline runs
Slack anomaly alert	Sends to a Slack channel through an incoming webhook URL	Team triage by project or environment	Requires a Slack Webhook URL and Slack Webhook Name
Webhook anomaly alert	Sends an HTTP POST with your custom JSON template	Incident tools, internal routing, and FinOps automation	Outbound only and not HMAC signed

What does this cost, and where can it save money?

Matillion does not document a separate credit price for anomaly alerts in the notification guide. The cost angle is indirect and more useful: runtime anomalies tell you when a pipeline is consuming time outside its normal envelope, and time is where warehouse, runner, and orchestration costs usually hide.

The cleanest workflow is to treat a slow anomaly as a cost investigation starter. Matillion's REST API consumption data is not real time, since the June 10 changelog says GET /v1/consumption data refreshes every three hours to reflect recent credit usage in the API update notes. That lag means anomaly alerts are the faster operational signal, while consumption reporting is the later accounting signal.

Use both. If Maia alerts that nightly_customer_mart crossed its expected upper bound at 02:14, you inspect Pipeline Runs immediately. Later, you check consumption by pipeline, project, or user once the usage data refreshes. Matillion's MCP server documentation explicitly includes cost optimization use cases such as analyzing credit consumption, seeing breakdowns by pipeline and execution frequency, and identifying expensive pipelines in the MCP server use case table. The alert tells you where to look. Consumption data tells you what it cost.

This is also where alert tuning becomes business work, not only engineering hygiene. A pipeline that runs 30 percent long once after a large customer import may be acceptable. A pipeline that drifts 8 percent longer every weekday for two weeks is a margin leak with a nicer UI. If you already use Maia context and governance patterns, connect this operational signal to your broader standards. Our guide to Matillion Context Engine cost controls is a useful companion because the same discipline applies: give Maia enough context to help, but keep humans accountable for cost policy.

When should you trust the alert, and when should you be skeptical?

Trust the alert when the pipeline has stable history, runs on a schedule, and does meaningful work every time. A nightly dimension rebuild, a predictable Salesforce incremental load, or a transformation pipeline behind an executive dashboard are good candidates. If the pipeline has at least 10 successful runs and normally lands in a tight runtime band, an alert is a real signal.

Be skeptical when the workload is naturally lumpy. Month-end finance jobs, backfills, campaign data imports, and pipelines with parameter-driven row counts will produce wider expected ranges because their past durations are wider. Matillion says naturally variable pipelines get a wider range so ordinary fluctuation does not trigger false alerts in the expected range explanation. That behavior is good, but it also means a genuinely bad run might need to be very bad before it crosses the threshold.

Also watch branch and environment changes. Matillion's pipeline run history shows fields such as status, environment, artifact version, trigger, start time, end time, and duration in the pipeline run history documentation. If an anomaly follows a new artifact version, start with the diff. If it follows an environment change, start with variables, warehouse sizing, secrets, and connection overrides. If it follows neither, start with source volume and platform health.

One more practical guardrail: do not auto-kill long-running jobs from the first webhook you receive. Use the webhook to open an incident, enrich a ticket, or notify the owning channel. Termination should require a second rule, such as the run exceeding a business SLA, a known bad artifact version, or a warehouse spend threshold.

What would I do before turning this on in production?

Start with your top five scheduled or API-driven pipelines by business impact, not your loudest pipelines. Pick jobs with clear owners, stable schedules, and visible cost or SLA consequences. Then configure anomaly alerts at the production environment level and route them to a channel where someone is actually expected to respond.

For each pipeline, capture four pieces of runbook context:

The normal schedule and expected business deadline.
The primary source systems and largest transformation tables.
The first Matillion page to inspect, usually Pipeline Runs or the run detail view.
The cost check to run after the three hour consumption refresh window.

Do not overbuild on day one. The first useful version is one notification, one channel, and one runbook link. After a week, add webhook routing for the pipelines that produce actionable signals. If alerts are noisy, check whether the pipeline is parameterized or lumpy by design before blaming Maia.

The bigger point is cultural. Runtime drift is often treated as a nuisance until it becomes a failure. Maia's duration anomaly alerts make drift visible early enough to do something cheaper than a postmortem. That is the feature's value: it moves the conversation from “why did the dashboard miss?” to “why is this job behaving differently right now?”

Sources

Matillion Context Engine: setup, limits, and costs

2026-06-08T00:00:00Z

If you have ever watched an AI assistant build a technically valid pipeline that misunderstands the business meaning of the table, you already know why Matillion shipped this. Maia can inspect objects and work through pipeline tasks, but a warehouse schema rarely explains which customer table is canonical, which finance metric is blessed, or which lineage path is stale.

Matillion Context Engine is a public preview knowledge graph feature for Maia AI Agents, designed to give Maia a living map of your business data. The key fact for a data engineer who also watches the bill: Context Engine is configured through knowledge graphs and crawlers, not through a magic prompt, and those crawlers can be scoped, scheduled, paused, and access controlled.

Matillion announced Context Engine alongside Mission Control on June 1, 2026. The pairing matters. Context Engine gives Maia the domain map. Mission Control gives you a kanban style place to launch, supervise, and review Maia tasks that can use that map. If you need the broader platform context, start with our guide to Matillion's Data Productivity Cloud for builders, then come back here for the Context Engine operating model.

What did Matillion actually ship in Context Engine?

Context Engine adds a new surface in Maia for creating and managing knowledge graphs. In Matillion's Context Engine documentation, a knowledge graph captures the structure, relationships, and meaning of your data so Maia AI Agents can use a specific graph while working on a task. That is the important shift. You are no longer relying only on a chat transcript, a project file, or a pasted naming convention.

The feature is in public preview, so treat it as something to pilot with real work, not something to hand unrestricted production authority on day one. Matillion's release FAQ says public preview features are available to all users but are not recommended for production workloads. That matters for governance. If a graph teaches Maia the wrong semantic relationship, you want that mistake discovered in a sandbox branch, not after a merge into your main data product.

The nouns are simple:

A knowledge graph is the domain container. Think Finance, Sales, Marketing, Customer 360, or Product Telemetry.
A crawler populates the graph from warehouse metadata or pipeline execution history.
Projects define which Maia projects can use a restricted graph.
Access defines which users can manage the graph.

That is more operational than glamorous. Good. The AI feature you can govern usually beats the dazzling one you cannot explain during a cost review.

How do knowledge graphs and crawlers actually work?

You create a graph from the Context Engine dashboard, then choose whether it is Public or Restricted. A Public graph is available to all projects. A Restricted graph is available only to selected projects. Matillion warns that public graphs should not ingest sensitive data or data that should not be available across projects, which is the sentence your security lead will underline twice.

To add a graph, the user needs the Admin or Super Admin account role. After the graph exists, it has three tabs: Crawlers, Projects, and Access. The Crawlers tab adds and monitors data ingestion into the graph. Projects controls the allowlist for restricted graphs. Access controls which users can manage the graph.

The crawler setup has a few concrete choices. In the Add crawler flow, you enter a crawler name, choose the type, select a project, select an environment, then schedule it. Context Engine supports 2 crawler types today:

Crawler type	What it harvests	Best first use	Cost watchpoint
Warehouse data	Warehouse and structured source metadata	Domain model discovery, table and column meaning	Scope databases, catalogs, and schemas tightly
Pipeline execution	Pipeline runs for a selected project and environment	Freshness, lineage, operational flow	Avoid crawling noisy dev environments by default

For Warehouse data crawlers, the selection step changes by warehouse. Snowflake uses databases and schemas. Databricks uses catalog and schemas. Amazon Redshift uses schemas. That gives you 3 warehouse selection patterns to plan for if your estate spans clouds or warehouses.

The chart below shows the operating surface you need to account for before enabling this broadly: 2 crawler types, 3 warehouse selection patterns, 4 Mission Control task columns, 6 crawler statuses, and a 10 task cap in Mission Control. The numbers are small enough to manage, but large enough to need ownership.

Matillion documentation lists 2 Context Engine crawler types, 3 warehouse selection patterns, 6 crawler statuses, 4 Mission Control columns, and a 10 task in progress cap.

Matillion lists 6 crawler statuses: Successful, Initializing, Extracting, Pending, Paused, and Failed. That status model is enough for a runbook. If a crawler is Pending, it has not run its first crawl. If it is Paused, it will not run until resumed. If it is Failed, you have a concrete object to investigate instead of a vague complaint that Maia has bad context.

How should you configure the first graph without creating semantic sludge?

Start with one graph per domain, not one graph for the company. Matillion's launch post says Context Engine is intended for domain focused knowledge graphs such as Finance, Sales, and Marketing. That is the right default because semantics are local. A field named status in a billing table and a field named status in a product event stream may both be valid and still mean completely different things.

A practical first graph might look like this:

knowledge_graph:
  name: finance_reporting
  availability: restricted
  projects:
    - finance_dbt_migration
    - revenue_reporting
  crawlers:
    - type: warehouse_data
      warehouse: snowflake
      scope: FINANCE_MART.PUBLIC
      schedule: standard_daily
    - type: pipeline_execution
      project: revenue_reporting
      environment: dev

That is not an import format. It is the configuration shape you should decide before clicking through the UI. The point is discipline: one domain, restricted access, one narrow warehouse scope, one environment for operational history.

Before crawling, improve the metadata Maia will see. Matillion recommends adding metadata such as column descriptions and tags to databases, schemas, or datasets because it helps Maia understand the landscape. In Snowflake, a tiny amount of comment hygiene can save a lot of prompt babysitting:

comment on table FINANCE_MART.PUBLIC.INVOICE_FACT
  is 'Canonical invoice fact table for recognized revenue reporting';

comment on column FINANCE_MART.PUBLIC.INVOICE_FACT.NET_REVENUE_USD
  is 'Net recognized revenue in USD after credits and discounts';

Do this for the five to ten tables people always argue about first. Do not boil the metadata ocean. Context Engine can crawl a wide surface, but your first success metric is whether Maia stops confusing the canonical thing with the convenient thing.

What does this change when you use Mission Control?

Mission Control is the other half of the release, and Context Engine is one of the task inputs. When creating a Mission Control task, Matillion's Mission Control guide says you choose a project, source branch, environment, and Knowledge Graph, then write the prompt. That Knowledge Graph drop-down is where Context Engine becomes operational.

Mission Control gives each task its own chat interface and board position. The board has 4 columns: Backlog, In progress, Needs attention, and Completed. It also has a hard workflow constraint: a maximum of 10 tasks can be in progress at any time. Once 10 are in progress, you can still create a task, but it lands in Backlog. You cannot start another backlog task, and responses to Needs attention tasks are not processed until fewer than 10 tasks are in progress.

That cap is useful. It prevents the worst version of AI delegation: twenty half reviewed branches, all burning attention, none mergeable.

The feature also creates a new branch for each task in Mission Control if you select that option. Maia can work on the branch, and you can open it in Designer to review the work. Completing a task in Mission Control does not merge the branch. You still need to commit, push, and merge changes from the task branch to the source branch, and Matillion says Maia cannot merge changes. That separation is exactly what you want for an AI agent touching pipeline logic.

The workflow I would use is simple:

Create a restricted Context Engine graph for a domain.
Crawl a dev or analytics environment first.
Launch a Mission Control task from a fresh branch.
Select the graph explicitly.
Keep Ask permission on unless the task is in a disposable sandbox.
Review the branch in Designer before merge.

Bypass permissions mode exists, but it should be a narrow tool. Matillion says Ask permission is the default and Bypass permissions runs tool calls immediately with no approval prompts. Use it for trusted, hands off sandbox work. Do not use it on a broad graph with production credentials and a PDF from someone you do not know. That is how a demo becomes a cleanup ticket.

What does Context Engine cost in practice?

Matillion does not publish a separate Context Engine price meter on the public feature documentation. So the honest answer is operational: your cost exposure comes from crawler scope, crawler schedule, Maia task activity, warehouse metadata queries, and the human review time needed to trust the result.

The bill lever you control first is crawler design. A Warehouse data crawler over a narrow Snowflake schema is a different operational bet than a crawler pointed at every schema reachable by a broad role. A Pipeline execution crawler on a dev project with five representative pipelines is a different bet than one aimed at a noisy shared environment. The docs say crawlers can use Standard schedule settings or Advanced cron expressions, and they can be paused or resumed. Use that.

For a pilot, avoid the three expensive habits:

Crawling every domain because the button is available.
Making the first graph Public because it saves two clicks.
Running Bypass permissions while Maia has access to credentials that are wider than the task.

There is also a hidden cost in bad context. If Context Engine learns that an old staging table is the customer source of truth, Maia may generate plausible work that takes longer to review than a human built pipeline. The answer is not fear. The answer is graph ownership. Put a named data engineer or analytics engineer on each domain graph, just as you would put an owner on a dbt package or shared pipeline.

When is Context Engine the wrong choice?

Skip Context Engine for one off work where the domain model is obvious and the pipeline is small. A two component load from a clean SaaS source does not need a knowledge graph. A context file in .matillion/maia/rules/ may be enough if all you need is naming rules, project conventions, or a short business rule. Matillion's context file docs still matter because context files are always read by Maia when stored in the reserved rules folder, and they have a strict 12,000 character limit across Markdown files in that folder.

Use Context Engine when the issue is semantic and operational, not just stylistic. Good fits include:

A finance mart with competing definitions of revenue.
A migration where pipeline execution history tells Maia how data actually flows.
A governed domain where only selected projects should use the graph.
A team that wants Mission Control tasks to start from the same domain map.

Bad fits include scratch work, sensitive domains with no access model yet, and production only warehouses with thin metadata. Context Engine can make Maia more useful, but it can also make wrong assumptions more reusable. Reuse is wonderful when the thing is right. It is brutal when the thing is wrong.

What should you do next if you influence the Matillion bill?

Run a narrow pilot. Pick one domain with real ambiguity and enough metadata to teach Maia something. Create a Restricted graph, add one Warehouse data crawler, add one Pipeline execution crawler only if execution history matters, and schedule both conservatively. Then create two Mission Control tasks against the graph: one pipeline build and one pipeline explanation. Compare the review burden with your normal Maia workflow.

Your success criteria should be concrete:

Did Maia choose the right source tables without repeated correction?
Did it use the domain language your team uses in reviews?
Did the generated branch require fewer manual edits?
Did crawler failures or stale metadata show up clearly enough to operate?
Did the schedule add background work you can justify?

If the answer is yes, expand by domain. If the answer is no, fix metadata before widening the crawl. The fastest way to waste money here is to scale uncertainty.

Context Engine is promising because it moves Maia from prompt memory toward governed domain memory. That is the kind of AI feature builders should want: less sparkle, more accountable surfaces. Give it a map. Just make sure you own the cartography.

Sources

Matillion's Data Productivity Cloud, explained for builders

2026-06-07T00:00:00Z

If you have used Matillion before, you probably picture the old ETL tool that ran on a VM you had to size, patch, and babysit. The Data Productivity Cloud (DPC) is the rebuilt, cloud-native successor, and it changes enough about how pipelines run and bill that it is worth a proper walkthrough. This guide is the starting point for our studio: what the platform actually is, how a pipeline executes, and where the money goes.

Matillion's pitch is that one platform should let three different kinds of people build the same pipeline: a low-code analyst dragging components in the Designer, an engineer writing dbt, SQL, or Python, and increasingly an AI agent under Matillion's Maia brand. The interesting question for a builder is not the marketing claim, it is the plumbing underneath: what runs where, what you pay for, and where the platform helps versus gets in the way.

What is the Data Productivity Cloud, concretely?

The DPC is a fully managed, browser-based environment for data integration. You do not run a Matillion instance anymore. Instead, Matillion hosts the control plane and you connect it to your cloud data warehouse, typically Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Microsoft Fabric.

Two kinds of work happen in a DPC project:

Ingestion pulls data from sources (databases, SaaS APIs, files) into your warehouse using prebuilt connectors and change data capture.
Transformation reshapes that data once it lands, and this is the part that matters most for cost.

The key architectural fact is pushdown. When you build a transformation pipeline in the Designer, Matillion does not move rows through its own engine. It compiles your pipeline into SQL and pushes that SQL down to your warehouse, which does the heavy lifting. That single design choice explains most of the platform's cost behaviour, which we will come back to.

How does a pipeline actually run?

A DPC project separates two pipeline types on purpose, and mixing them up is the most common beginner mistake.

Transformation pipelines run SQL against your warehouse. They have no orchestration logic; they just transform.
Orchestration pipelines are the conductor. They run ingestion jobs, call transformation pipelines, branch on success or failure, loop with iterators, and handle scheduling.

A healthy project keeps a thin orchestration layer that calls many focused transformation pipelines, rather than one giant pipeline that tries to do everything. The same discipline you would apply to functions in code applies here: small, named, reusable units.

The chart below shows where the run minutes of a typical project go. The bulk of the time, roughly 60 percent, is transformation SQL executing inside your warehouse, around 30 percent is ingestion, and the remaining 10 percent is orchestration overhead. The exact mix varies, but the shape holds: your warehouse, not Matillion, is doing most of the work.

Illustrative: a typical split of pipeline run minutes across transformation pushdown, ingestion, and orchestration overhead. Data Today.

Where does the cost actually go?

This is the question that decides whether a DPC project stays affordable, and the answer has two halves that you pay separately.

Cost layer	What you pay for	Who bills you
Matillion credits	Pipeline runs and platform usage, metered by Matillion	Matillion
Warehouse compute	The SQL pushdown that transformations execute	Your cloud warehouse
Ingestion	Rows or connectors moved, depending on plan	Matillion

Because transformation runs as pushdown SQL, a badly written pipeline does not just burn Matillion credits, it runs an expensive query on your warehouse and shows up on a second bill. The most common cost surprise is a transformation that scans far more data than it needs, often because someone left a full reload where an incremental load belonged. Optimizing the SQL your pipeline generates is therefore a warehouse-cost exercise as much as a Matillion one, which is exactly why we treat cost as its own section.

What about the runners?

The DPC runs your pipelines on compute called runners. Matillion offers fully hosted runners so you do not manage infrastructure, and self-hosted or cloud-hosted runner options for teams that need pipelines to execute inside their own network, for example to reach a private database without exposing it to the internet.

The trade-off is the usual managed-versus-self-hosted one:

Hosted runners are zero-maintenance and the fastest way to start, but they run in Matillion's environment.
Self-hosted runners keep execution and credentials inside your perimeter, at the cost of you owning the compute and its upkeep.

If you handle regulated data or sit behind strict network controls, the runner choice is the first architectural decision to get right, well before you build a single pipeline.

Where does Maia, the AI layer, fit?

Maia is Matillion's name for the AI features layered across the DPC: copilots that help build and explain pipelines, agents that can take actions through API endpoints, and assistance for tasks like root cause analysis when a pipeline fails. The honest read is that this is the fastest-moving and least settled part of the platform, which is precisely why it deserves close, sceptical coverage rather than hype.

For a builder, the practical stance is to let AI accelerate the boring parts, generating boilerplate transformations, suggesting fixes, explaining an unfamiliar pipeline, while keeping a human reviewing anything that touches production data. We will track each Maia capability as it ships and judge whether it is genuinely production-ready or still a demo.

What should you do with this?

If you are evaluating or adopting the Data Productivity Cloud, a few principles travel well:

Treat your warehouse as the engine. Most of your cost and performance lives in the pushdown SQL, not in Matillion. Profile the queries your pipelines generate.
Keep orchestration thin and transformations small. Reusable, well-named pipelines age far better than monoliths.
Decide runners early. Network and compliance constraints shape the whole project.
Adopt Maia deliberately. Use it where review is cheap; gate it where mistakes are expensive.

This guide is the foundation. From here, the guides go deeper on each piece: ingestion connectors and change data capture, transformation patterns in the Designer, orchestration controls like iterators and scheduling, the FinOps habits that keep credits in check, and the Maia AI features as they land. The platform is moving quickly, and the goal here is the same as everywhere on Data Today: tell you what actually changed and what it means for the thing you are building.

Sources

Matillion Context Engine grounds Maia agent work

2026-06-07T00:00:00Z

A data agent that can build pipelines is useful. A data agent that knows which tables matter, which pipelines already touch them, and when to stop and ask you before firing a tool call is the version you can let near production.

Matillion Context Engine is the new public preview layer that gives Maia AI Agents knowledge graphs, crawlers, and task context, with Mission Control adding a 10 task in-progress cap around agent work.

Matillion shipped Context Engine and Mission Control together because the old problem with AI assistants in data engineering is not syntax. It is context. Maia can already build orchestration and transformation pipelines, query warehouse data, sample pipeline data, manage files, and commit or push changes inside the Data Productivity Cloud, according to Matillion's Maia AI Agents overview. Context Engine gives those agents a living map of your warehouse metadata, pipeline execution history, business language, and project scope. Mission Control gives you a kanban board where that work becomes task shaped instead of chat shaped.

If you are still getting oriented around the platform, start with our guide to Matillion's Data Productivity Cloud for builders. This piece assumes you already build in Maia or Designer and now need to decide where Context Engine belongs in your operating model.

What did Matillion actually ship in Context Engine?

Context Engine is in public preview, and the important object is the knowledge graph. Matillion describes it as a way to capture the structure, relationships, and meaning of your data, then let Maia use that graph when it works on a task in Mission Control or chat. In plain builder terms: it is metadata grounding for Maia, scoped to projects and fed by crawlers, not another Markdown rules file with a nicer name.

The Context Engine dashboard sits under the AI Agents icon in the left navigation. It lists knowledge graphs you can access, lets you filter by project, and supports search by name or description. From there, an Admin or Super Admin can add a knowledge graph, choose whether it is Public or Restricted, and give it a name and description, as Matillion's Context Engine documentation spells out.

That Public or Restricted choice is not cosmetic. A public knowledge graph is available in all projects. A restricted graph is available only to selected projects. Matillion explicitly warns that public graphs should not ingest sensitive data that should not be available to all projects. That is the kind of sentence you should read twice before turning a sales operations graph into a company wide default.

Once a graph exists, it has three important tabs:

Crawlers, where you add and monitor crawlers that populate the graph.
Projects, where you manage which projects can use a restricted graph.
Access, where you add users who can manage the graph.

The crawler model is refreshingly concrete. Matillion supports 2 crawler types: Warehouse data crawlers and Pipeline execution crawlers. Warehouse data crawlers harvest warehouses and structured sources supported through connectors. Pipeline execution crawlers harvest executions for a chosen project and environment, giving Maia operational context about how work actually flows.

Matillion documentation lists 2 Context Engine crawler types, 2 knowledge graph availability modes, 4 Mission Control board columns, and a 10 task in-progress limit.

As the chart shows, the launch is not just a new button. Context Engine and Mission Control introduce 2 crawler types, 2 graph availability modes, 4 task board columns, and a 10 task in-progress limit. Those numbers matter because they turn agent context into something you can scope, schedule, and govern.

Crawler setup has a few warehouse specific details. For Warehouse data crawlers, the data selection step depends on the target: Snowflake uses databases and schemas, Databricks uses catalog and schemas, and Amazon Redshift uses schemas. You can schedule crawler runs with Standard settings or Advanced schedule settings, including a cron expression. You can also run a crawler on demand with Run now.

The crawler status model gives you the minimum you need to operate it: Successful, Initializing, Extracting, Pending, Paused, and Failed. You can inspect the latest crawl or crawl history, including status, start time, end time, and duration. That is not observability nirvana, but it is enough to answer the first operational question: did the graph refresh before Maia used it?

How does Maia use the graph when it starts a task?

The graph becomes useful when you attach it to Maia's work. In Mission Control, the New task dialog includes Project, Branch, Environment, Knowledge Graph, and Prompt fields. The Knowledge Graph drop-down selects the graph Maia will use to inform that task, according to Matillion's Mission Control guide.

The same idea appears in the Agent Tasks API as graphId. That is the cleanest signal that Context Engine is not just a UI feature. You can pass graph context into programmatic agent work.

curl --request POST \
  --url "$BASE_URL/v1/ai/agents/tasks" \
  --header "Authorization: Bearer $MATILLION_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "message": "Plan a pipeline for daily net revenue by region using our governed sales model.",
    "agentConfig": {
      "name": "data_engineer_agent",
      "mode": "PLAN",
      "projectId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "sourceBranchName": "main",
      "environmentName": "development",
      "targetBranchName": "feature/revenue-region-plan",
      "graphId": "finance-analytics-graph"
    }
  }'

Matillion's Agent Tasks API guide defines name as currently always data_engineer_agent, mode as either ACT or PLAN, and graphId as the ID of a Knowledge Layer service graph. Use PLAN when you want the graph to shape a design without letting Maia make changes yet. Use ACT only when you are comfortable with the branch, environment, and permissions.

That branch point matters. Each task works in isolation on its own branch when you create work through the API. Matillion also says Agent Tasks API work runs under the identity of the user associated with the API key, and changes are not automatically visible to other project users unless Maia commits and pushes to the target branch. Good. Agent work should leave a diff, not a mystery.

There is one practical catch: the docs say these endpoints currently work only with Matillion-hosted and GitHub projects. If your team is standardized on another Git provider, test the path before you design a team process around it.

How is Context Engine different from context files?

Context files still matter. They are Markdown files that Maia always reads from .matillion/maia/rules/, and Matillion enforces a 12,000 character limit across all Markdown context files in that directory. They are the right place for rules: naming conventions, design standards, business glossary shortcuts, and team preferences.

Context Engine is for the map.

Here is the split that should guide your setup:

Mechanism	Best use	Concrete limit or setting
Context files	Always applied rules for a project	Stored under `.matillion/maia/rules/` with a 12,000 character total limit
Additional project files	Detailed standards Maia can reference when instructed	Stored outside `.matillion/...` and referenced from a context file
Context Engine knowledge graphs	Warehouse metadata, relationships, semantics, and execution context	Fed by Warehouse data and Pipeline execution crawlers

The best setup uses both. Put hard rules in context files: table prefixes, environment constraints, source of truth definitions, and review expectations. Put metadata and process reality in Context Engine: schemas, columns, tags, warehouse structures, and pipeline execution history.

Do not stuff everything into the graph because it feels newer. If a rule is small, stable, and mandatory, put it in a context file. If the context changes as pipelines run and schemas evolve, put it in the graph.

What does it cost, and where can it surprise the bill?

Matillion's public docs do not publish a separate Context Engine credit rate in the pages reviewed for this guide. That absence is its own buying signal. You should not treat public preview as free forever, and you should not assume every action has a visible line item until you validate it with your account data.

There are 3 cost surfaces to watch.

First, crawler frequency. A Warehouse data crawler connects to data warehouse structures such as Snowflake databases and schemas, Databricks catalogs and schemas, or Redshift schemas. Even if Matillion does not show a separate Context Engine meter in the public docs, that crawler is still operating against systems you pay to run. Start with a low frequency schedule, then increase it only for domains where schema drift or metadata freshness affects real delivery.

Second, agent work. Maia can validate and run pipelines, query the data warehouse, sample component output, commit changes, and push branches. Mission Control lets up to 10 tasks sit in the In progress column at once. Ten autonomous tasks pointed at a development warehouse can be a productivity win. Ten autonomous tasks repeatedly sampling, running, and revising pipelines can also turn a quiet sandbox into a noisy bill.

Third, preapproved tools. The Agent Tasks API supports grantedPermissions, and the Mission Control UI has Ask permission and Bypass permissions modes. Matillion recommends Ask permission as the default and Bypass permissions only for trusted hands-off runs in scoped environments. That is not conservative vendor boilerplate. It is the right default for anyone who has ever watched a retry loop discover money.

For cost monitoring, Matillion's MCP server exposes Consumption tools, including get-consumption for flat-rated products and get-consumption-etl-users for ETL users, and it can help analyze credit consumption patterns through an AI assistant via the MCP server documentation. Use that for investigation, but do not let an assistant be your only FinOps control. Pull consumption on a schedule, tag task branches clearly, and compare before and after you enable crawler schedules.

A sane rollout looks like this:

Create one restricted knowledge graph for a single analytics domain.
Add one Warehouse data crawler and one Pipeline execution crawler.
Schedule crawls outside peak warehouse windows.
Run Maia tasks in PLAN mode first.
Keep Ask permission on until you know which tools Maia calls repeatedly.
Review consumption and warehouse activity after one week.

The boring version wins.

When should you use Mission Control with Context Engine?

Use Mission Control when the work has a deliverable. A chat is fine for asking Maia to explain a Table Input component or suggest a naming convention. A Mission Control task is better when you want Maia to design a pipeline, modify project files, create a connector, analyze a failure, or work from a PDF spec.

The task board has 4 columns: Backlog, In progress, Needs attention, and Completed. That sounds simple because it is. The useful part is that each task has its own chat interface, and you can switch between tasks without losing context. For a data team, that maps better to real work than one endless assistant thread.

Mission Control also adds attachments. You can attach images and PDFs to a task prompt, then reference them with @filename. Matillion says images can include diagrams, screenshots, mockups, and whiteboard photos, while PDFs can include specs, requirements documents, and reports. Text files are not supported as attachments because Maia can already read project text files.

Use that for pipeline triage. A screenshot of a broken canvas plus a pipeline execution crawler is exactly the kind of mixed context that a human engineer would ask for. The difference is Maia can now carry that context into a branch and produce work you can review.

Do not use Mission Control as a merge gate. Matillion says completing a task does not make changes visible on other branches. You still need to commit, push, and merge. Maia cannot merge changes, which is a good boundary. Keep code review, pipeline tests, and environment promotion in your normal process.

What would I do first in a real Matillion team?

Start with one graph per domain, not one graph per company. Finance, customer, product, and operations usually have different semantics and different blast radiuses. A public all-company graph sounds convenient until it contains restricted HR tables or ambiguous definitions that poison every prompt.

Then tune freshness by risk. Operational pipeline execution context can go stale quickly if you are actively refactoring. Warehouse schema metadata may not need hourly crawls if your governed marts change weekly. Context Engine supports any number of Warehouse data and Pipeline execution crawlers on a graph, so separate them by source and schedule rather than creating one giant crawler that nobody wants to touch.

Use PLAN mode as your default for the first few tasks. Ask Maia to propose pipeline structure, sources, joins, variable usage, and tests while grounded in the selected graph. Once the plans look sane, let a narrow ACT task implement on a feature branch.

Most of all, measure whether Context Engine reduces clarification loops. The win is not that Maia can produce more pipeline files. The win is fewer wrong assumptions about customer_id, fewer duplicated staging models, fewer prompts that repeat your business glossary, and fewer reviews that start with: who told the agent to use that table?

If Context Engine does that, it earns a place in the stack. If it becomes another metadata garden that nobody prunes, Maia will learn your mess at machine speed.