# Defining a data set

Defining a data set means choosing which data streams become columns. This is administration work, done
once per data set.

## The identifier column

Every data set needs an `identifierColumn`. It maps to
`provision.administration.identifier._current.value`, with filtering enabled and sorting available, and it
identifies the device each row belongs to.

## Column paths

A column's `path` has three parts, and the third is only required when the data stream is not a primitive
value.

**1. The data stream identifier** — the `datastreamId` you want. If it contains `communicationModules[]`,
include the index of the module you mean:

```
device.communicationModules[0].subscription.mobile.imsi._current.value
```

**2. The data stream field** — one of:

`_current.value` · `_current.date` · `_current.at` · `_current.feedId` · `_current.source` ·
`_current.sourceInfo`

**3. The value path** — when the data stream holds an object or an array, a path down to a primitive value.

## What a column can be filtered by

Every column carries a `filter` value that decides how queries may use it:

| Value | Meaning |
|---|---|
| `NO` | Not filterable. The default |
| `YES` | Optional equality filter |
| `ALWAYS` | **Required** equality filter: every query must constrain this column |
| `RANGE` | Range filter, `>`, `<` and `BETWEEN`, as well as equality |

`RANGE` is only allowed on **numeric** columns, `integer` and `number`. Columns of type `date-time` are
always range-searchable whatever the value says.

## The `sorts` section

Sorting is **declared in the definition**, not composed at query time. The `sorts` section holds a list of
named sorts, each an ordered list of columns with a direction, and a query asks for one **by its
identifier**.

```json
"sorts": [
  {
    "identifier": "sortByDeviceAsc",
    "description": "Sort by device identifier ascending",
    "columns": [
      { "name": "Prov Identifier", "direction": "ASC" }
    ]
  }
]
```

| Field | Rules |
|---|---|
| `identifier` | Required, unique within the list. Letters, digits, spaces, `_` and `-`. Generated as a UUID if omitted, so name it |
| `description` | Optional free text |
| `columns` | Required, at least one. A column `name` from the `columns` section plus `ASC` or `DESC` |

At least one sort is mandatory, and the order inside `columns` is the sort precedence.

{{% notice style="primary" title="The reverse of every sort comes for free" icon="lightbulb" %}}
For each sort you declare, OpenGate also exposes its **reverse**, served by the same index through a reverse
scan. Those come back marked `derived: true`, which is read-only: the platform sets it and the web console
uses it. Never declare a derived sort yourself — flipping the direction of one you already have just spends
optimization units on an index you were given.
{{% /notice %}}

## Limits

There is **no fixed maximum** number of filterable columns or declared sorts. Each filterable column and
each declared sort consumes **optimization units**, and the data set has a budget of them — that budget is
the limit.

| Where to look | What it tells you |
|---|---|
| The `searchOptimizationInfo` of a data set | `usedSearchOptimizationUnits` and `freeSearchOptimizationUnits` |
| `POST .../optimizationPlan` | What a definition *would* consume, before committing to it |

```bash
POST /north/v80/datasets/provision/organizations/{organizationName}/optimizationPlan
```

## Creating

```bash
POST /north/v80/datasets/provision/organizations/{organizationName}
```

## Updating

```bash
PUT /north/v80/datasets/provision/organizations/{organizationName}/{identifier}
```

Updating can affect the data already stored or the structure holding it, which starts an adaptation process.
Until it completes, **dirty values** may be present.

These fields can be modified:

`Name` · `Description` · `IdentifierColumn` · `Columns` · `Sorts`

Rules for columns:

- **Names are unique.** You cannot add or rename a column to a name already in use.
- **`filter: ALWAYS` is immutable.** You cannot add or remove a column that has it, you cannot set it on an
  existing column, and you cannot change it away once set.
- **Paths cannot be edited.** Remove the column and create it again, which gets you the same result.

The optimization unit budget applies to updates as well, so run `optimizationPlan` before adding filterable
columns or sorts to a definition that is already close to it.

## Listing and deleting

```bash
GET    /north/v80/datasets/provision/organizations/{organizationName}
GET    /north/v80/datasets/provision/organizations/{organizationName}/{identifier}
DELETE /north/v80/datasets/provision/organizations/{organizationName}/{identifier}
```

Read the organization's data sets:

```bash
curl --request GET \
     --header "X-ApiKey: <your-api-key>" \
     https://api.opengate.es/north/v80/datasets/provision/organizations/{organizationName}
```