# Querying a data set

Reading a data set is a `POST` with the data set identifier in the URL:

```bash
POST /north/v80/datasets/provision/organizations/{organizationName}/{identifier}/data
```

Copy this and change the identifiers:

```bash
curl --request POST \
     --header "X-ApiKey: <your-api-key>" \
     --header "Content-Type: application/json" \
     --data '{"filter": {"eq": {"Prov Identifier": "MyDevice1"}}}' \
     https://api.opengate.es/north/v80/datasets/provision/organizations/{organizationName}/{identifier}/data
```

The response names its columns and gives you rows in that order:

```json
{
  "page": { "number": 26 },
  "columns": ["Coll Mobile ICC value", "Coll Mobile ICC date", "Prov Identifier", "Coll manufacturer", "Coll model", "Prov Mobile ICC value"],
  "data": [
    ["icc1", "2021-04-06T12:35:22.784Z", "MyDevice1", "OpenGate", "OpenGate", "icc1"],
    ["icc2", "2021-04-06T07:45:57.468Z", "MyDevice2", "OpenGate", "OpenGate", "icc2"]
  ]
}
```

Without a `select` clause, `columns` holds the identifier column first, then the defined columns in
declaration order.

## The request body

Data set queries use the same clauses as any other search, with two differences worth memorising:

| Clause | In a data set query |
|---|---|
| `filter` | Standard [operators](../../data_lake/features/filtering/), keyed by `identifierColumn` or a column name |
| `sort` | A **string**: the `identifier` of a sort declared in the data set — see below |
| `select` | **An array of column names**, not the object form used elsewhere |
| `limit` | `start` and `size`, as everywhere else |
| `group` | **Does not exist** for data sets |

The full comparison against the other query dialects is in [Query dialects](../../query_dialects/).

## Asking for a sort

You do not compose an ordering in the request. You name one that already exists:

```json
{ "filter": {}, "sort": "sortByDeviceAsc" }
```

Valid values are the `identifier` of any sort in the data set definition, **plus the automatically exposed
reverse of each one**, so declaring an ascending sort gives you the descending direction too.

**Omit `sort` and results come back sorted by the identifier column, ascending.**

There is no fixed limit on how many sorts a definition can hold; the constraint is the optimization unit
budget, described in [Defining a data set](../defining/#limits).

## Pagination and CSV

Data sets answer in JSON or CSV, and the format changes what an absent `limit` means:

| Body | JSON | CSV |
|---|---|---|
| `{"filter": {}, "limit": {"size": 500, "start": 1}}` | 500 rows from row 1 | 500 rows from row 1 |
| `{"filter": {}}` | Configured default page | **The complete data set** |
| `{"filter": {}, "limit": {}}` | Configured default page | **Error** — an incomplete `limit` is rejected |

CSV retrieval **turns sorting off** on purpose: it is what makes large exports fast, and CSV output is
usually consumed by something that will sort it anyway.

{{% notice style="warning" title="Complete retrieval is expensive" icon="triangle-exclamation" %}}
Omitting `limit` in CSV mode downloads the whole data set. Page it unless you truly want everything.
{{% /notice %}}

CSV formatting is customizable through HTTP header options — the quoting character (double quotes by
default), the escape character (a backslash by default), the end-of-line sequence (`\n` by default) and how
nulls are represented. You are responsible for the combination producing well-formed CSV. The exact header
names are not currently published, so ask your platform contact for them.

## The other data set endpoints

Three more endpoints exist, and one of them is **not** what its URL suggests:

```bash
POST /north/v80/search/catalog/datasets
POST /north/v80/search/organizations/{organizationName}/datasets/{datasetId}
POST /north/v80/search/organizations/{organizationName}/datasets/{datasetId}/summary
```

`search/catalog/datasets` lists the data sets available to you.

The other two are **not** a mirror of the `.../data` read above: they take a **different request body**. The
`.../data` endpoint uses the data set's own dialect — `sort` as a declared identifier, `select` as an array
of column names, no `group`. These two take the **generic Data Lake search body**, with `sort` as the
`{"parameters": [{"name": ..., "type": ...}]}` object, `select` in its object form, and `group` available.

| Endpoint | Request body |
|---|---|
| `datasets/provision/.../{identifier}/data` | Data set dialect: `sort` is a declared sort identifier |
| `search/organizations/.../datasets/{datasetId}` | Generic [Data Lake](../../data_lake/) search body |

Use `.../data` unless you specifically need the generic clauses. Which of the two is intended to be the
long-term path has not been confirmed by the product team.