Querying a data set
Reading a data set is a POST with the data set identifier in the URL:
Copy this and change the identifiers:
The response names its columns and gives you rows in that order:
Without a select clause, columns holds the identifier column first, then the defined columns in
declaration order.
The request body
Data set queries use the same clauses as any other search, with two differences worth memorising:
| Clause | In a data set query |
|---|---|
filter |
Standard operators, keyed by identifierColumn or a column name |
sort |
A string: the identifier of a sort declared in the data set — see below |
select |
An array of column names, not the object form used elsewhere |
limit |
start and size, as everywhere else |
group |
Does not exist for data sets |
The full comparison against the other query dialects is in Query dialects.
Asking for a sort
You do not compose an ordering in the request. You name one that already exists:
Valid values are the identifier of any sort in the data set definition, plus the automatically exposed
reverse of each one, so declaring an ascending sort gives you the descending direction too.
Omit sort and results come back sorted by the identifier column, ascending.
There is no fixed limit on how many sorts a definition can hold; the constraint is the optimization unit budget, described in Defining a data set.
Pagination and CSV
Data sets answer in JSON or CSV, and the format changes what an absent limit means:
| Body | JSON | CSV |
|---|---|---|
{"filter": {}, "limit": {"size": 500, "start": 1}} |
500 rows from row 1 | 500 rows from row 1 |
{"filter": {}} |
Configured default page | The complete data set |
{"filter": {}, "limit": {}} |
Configured default page | Error — an incomplete limit is rejected |
CSV retrieval turns sorting off on purpose: it is what makes large exports fast, and CSV output is usually consumed by something that will sort it anyway.
Complete retrieval is expensive
Omitting limit in CSV mode downloads the whole data set. Page it unless you truly want everything.
CSV formatting is customizable through HTTP header options — the quoting character (double quotes by
default), the escape character (a backslash by default), the end-of-line sequence (\n by default) and how
nulls are represented. You are responsible for the combination producing well-formed CSV. The exact header
names are not currently published, so ask your platform contact for them.
The other data set endpoints
Three more endpoints exist, and one of them is not what its URL suggests:
search/catalog/datasets lists the data sets available to you.
The other two are not a mirror of the .../data read above: they take a different request body. The
.../data endpoint uses the data set’s own dialect — sort as a declared identifier, select as an array
of column names, no group. These two take the generic Data Lake search body, with sort as the
{"parameters": [{"name": ..., "type": ...}]} object, select in its object form, and group available.
| Endpoint | Request body |
|---|---|
datasets/provision/.../{identifier}/data |
Data set dialect: sort is a declared sort identifier |
search/organizations/.../datasets/{datasetId} |
Generic Data Lake search body |
Use .../data unless you specifically need the generic clauses. Which of the two is intended to be the
long-term path has not been confirmed by the product team.