Data Sources
A data source is a named connection to live data: a REST or GraphQL API, a function or MCP tool you already have, a stream of platform events, a webhook, or a built-in query over your own project. The platform fetches the data, caches the result, and serves it to anything that reads the source:
- Agents read it with the
read_data_sourcetool instead of carrying their own fetch code. - Dashboard tiles bound to it show its latest result and update live when it refreshes. See Dashboards.
- Events. A source can publish a
datasource.refreshedevent every time it refreshes.
You manage data sources under Resources → Data Sources. The list shows each source's Name, Type (connector type, plus the event transport for streaming sources), Mode (refresh mode), Status and Last refreshed, with Refresh now, View latest data and Delete actions. Choosing a row opens its latest data as a table (or as JSON when the result is not a flat list of records).
What a data source is made of
| Part | What it decides |
|---|---|
| Connector type and connector config | What is fetched and how |
| Integrations | Which integrations supply credentials. Secrets are never stored on the data source |
| Refresh mode | When the data is fetched: on demand, on a schedule, or when something pushes to it |
| Cache TTL | How long a result is served before the next read fetches again. Default 300 seconds |
| Transform | Optional mapping that reshapes each row |
| Output schema | Optional declared shape; otherwise inferred from the rows |
| Scope | Who can see and read it |
| Emit event | Whether each refresh publishes a datasource.refreshed event |
Connector types
| Connector type | Value | Where it runs | Required config | Use for |
|---|---|---|---|---|
| REST API | rest | Platform | url | Any JSON HTTP API. Supports method, headers, query, body, result_path and pagination (none, page, cursor, link_header, up to 50 pages) |
| GraphQL | graphql | Platform | url, query | GraphQL APIs, with optional variables, headers and result_path |
| Function | function | The function's sandbox | function_id | Anything a declarative call cannot express: SDKs, several calls stitched together. Optional static args |
| MCP Tool | mcp | The MCP function's runtime | function_id, tool | Calling one tool of an mcp_stdio or mcp_http function. Optional args |
| Internal Query | internal_query | Platform | query | Read-only queries over your own project (below). Optional params |
| Event Stream | event_stream | Fed by an event source | An event transport | Live feeds; each event is folded into the cached rows |
| Webhook | webhook | Fed by a webhook | None | Provider pushes, delivered by a webhook linked to the source |
The Connector type list on the New Data Source dialog shows the connector types available in your project, each with a configuration form generated from the connector's own description.
Credentials and integration placeholders
REST and GraphQL configs reference secrets with placeholders, resolved on the server immediately before each request and never written back:
{{integration:<integration_id>:<SECRET_NAME>}}
A placeholder is only resolved for an integration that is attached to the data source (its integration_ids). A placeholder naming any other integration fails with Unresolved integration placeholder, so one source cannot read another integration's secrets. Function and MCP sources pass the attached integrations' credentials into the function's sandbox the same way a workflow node does.
Outbound requests to private, loopback and link-local addresses are always blocked.
Internal queries
| Query | Returns |
|---|---|
mission_counts | Number of missions by status |
workflow_count | Number of workflows in the project |
data_source_count | Number of data sources in the project |
element_counts | Dashboard tiles by element type |
token_usage | Total input, output and overall tokens, cost and mission count; optional period such as 7d, 30d or all |
active_sessions | Active chat sessions, with a breakdown by phase |
manifests_by_type | Every manifest of one type, one row per manifest (its data plus manifest_* identity fields); optional limit, default 500, maximum 2,000 |
manifest_counts | Manifests per type, or with type and group_by, counts of each value of a field within that type |
workspace_summary | Counts of dashboards, elements, data sources, workflows, missions and knowledge bases |
Push-fed sources: event streams and webhooks
Event-stream and webhook sources do not fetch; something sends rows to them. Each delivery is folded into the cached rows according to Accumulate:
| Accumulate | Effect |
|---|---|
| Append | Adds the new rows after the existing ones |
| Upsert (by key) | Replaces any existing row with the same Key field, appends the rest |
| Replace | Discards the existing rows |
Max rows keeps only the most recent rows (default 200). An event-stream source subscribes to an Event transport (an event source). A webhook source is fed by a webhook linked to it (the webhook's data_source_id, set through the API). Through the API you can also push a payload yourself with POST /data-sources/{id}/ingest (tool ingest_data_source_payload).
Where does my dashboard data come from?
Use the first row that fits. Each is cheaper to keep working than the one below it, because less of it is code you have to maintain.
| The data is... | Use | Why |
|---|---|---|
| The platform's own state: agents, missions, usage, spend | An internal_query source on an agent.* dataset, for example agent.top_agents | The platform reads it under the viewer's permissions. No code and no credential, so a dashboard built on it cannot go quiet because a function broke. Inside a function the same data is platform_api.call('agentapi', 'dataset_query', ...) with agentapi:dataset_query in the function's Allowed Operations. |
| In an external system that has a REST or GraphQL API | A rest or graphql source through an integration | The credential stays in the integration and the request is pinned to its allowed URLs. There is no code to sandbox. |
| Computed, joined across several calls, or in a protocol the declarative connectors cannot express | A function source whose code uses only the function SDK | The function runs in the sandbox and reaches other systems only through the broker. |
| Rows in the platform's own database | Not available to any function | Use the dataset in the first row. A function that imports app.dependencies, connects with psycopg2 or reads DATABASE_URL cannot run in the sandbox. |
A function source can use only what the sandbox provides. See What a Local Python function can and cannot do. When a function's code breaks that contract the platform says so when you create the source, and a source whose function fails in the sandbox for that reason shows unsupported_platform_access and what to use instead as its error, rather than "did not return usable data".
Refresh modes
| Mode | Value | What triggers a fetch |
|---|---|---|
| On demand | on_demand | A read that finds the cache empty or older than its TTL, or an explicit refresh |
| Scheduled | scheduled | The platform scheduler, every interval_seconds or on a cron expression in refresh_config |
| Streaming | streaming | Each event from the source's event transport |
| Webhook | webhook | Each delivery from a linked webhook |
Every read is served from the cache while it is fresh. Opening a dashboard also warms any bound source whose cache is empty or stale, within a short time budget, so tiles show live data rather than their sample content.
The New Data Source dialog creates on-demand sources (or streaming and webhook sources for those connector types). Create Data Source on a function sets On demand or Scheduled (interval). Any other refresh configuration, including cron schedules, is set through the API or by an agent.
Caching and limits
- Each distinct set of query parameters is cached separately, so two tiles that filter the same source differently are cached independently.
- A failed fetch is cached for about a minute, then retried on the next read. The last good result stays available to serve.
- A single fetch keeps at most 5,000 rows and 2 MB by default; anything beyond is truncated and the truncation is recorded in the cached result.
- Old cache entries are cleared after seven days. Disabling a source deletes its cached data.
Status
| Status | Meaning |
|---|---|
unknown | Never fetched |
ok | The last fetch succeeded |
degraded | The last fetch returned data, but the data reported its own errors (a top-level errors object). Partial data is served |
error | The last fetch failed; the error is recorded on the source |
Scope: who can see a data source
| Scope | Who can see and read it |
|---|---|
personal (default) | Only the person who created it |
project or shared | Everyone in the project |
Sources created on the Data Sources page and from a function are personal. Set a wider scope through the API or by asking an agent. A dashboard tile bound to a source you cannot read shows its sample content and a No access marker rather than someone else's data.
Transform and output schema
A transform maps output field names to JSONPath expressions evaluated against each row:
{ "title": "$.title", "author": "$.user.login", "opened": "$.created_at" }
The output schema tells agents and tiles what shape to expect. If you do not declare one, it is inferred from the rows of each fetch.
Agents and data sources
Three platform functions let an agent work with data sources. An agent can use them once they are assigned to it, like any other function:
| Tool | What it does |
|---|---|
list_data_sources | Lists the sources the agent can see: id, name, description, connector type, status, last refreshed, output schema |
read_data_source | Returns a source's rows, schema and status, serving the cache unless asked to force a refresh. Accepts query parameters |
create_data_source | Creates a source. Intended for assistant and orchestrator agents you trust to set up connections |
Templates
Data source templates are pre-configured connectors you instantiate into your project, supplying the integration that holds the credential. Templates are available through the API (/data-source-templates) and the platform tools (list_data_source_templates, instantiate_data_source_template); there is no template picker on the Data Sources page.
Events
With Emit a datasource.refreshed event on update ticked, every refresh (and every push into an event-stream or webhook source) publishes a datasource.refreshed platform event carrying the source's id, name, row count and status.
Related
- Creating Data Sources
- Binding Data Sources to Dashboard Elements
- Dashboards
- Integrations
- Choosing How to Add Knowledge: data source versus function, knowledge base or memory