This feature is only available to root and super_admin_domain profiles. Ask your administrator for proper user role profiling.
What a data set is
A data set is a flat table over your devices: one row per device, one column per value you chose. You
pick the data streams that become columns, and the platform keeps the table current.
It is the answer to “give me a spreadsheet of my fleet” — the identifier, the model, the ICC, the last
reading — without writing a query that walks each device’s data streams and flattens the result.
Column values are limited to strings, numbers and booleans. If a data stream holds an object or an
array, the column definition has to include a path down to one of those primitive values. Devices with
communication modules need one column per module.
The two halves of the API
Defining a data set is administration: choose the columns, their paths, and which of them
are filterable, and declare the sorts a query may ask for. Done once.
Querying a data set is the daily work: POST a filter, read rows back as JSON or CSV.
Defining a data set means choosing which data streams become columns. This is administration work, done
once per data set.
The identifier column
Every data set needs an identifierColumn. It maps to
provision.administration.identifier._current.value, with filtering enabled and sorting available, and it
identifies the device each row belongs to.
Column paths
A column’s path has three parts, and the third is only required when the data stream is not a primitive
value.
1. The data stream identifier — the datastreamId you want. If it contains communicationModules[],
include the index of the module you mean:
3. The value path — when the data stream holds an object or an array, a path down to a primitive value.
What a column can be filtered by
Every column carries a filter value that decides how queries may use it:
Value
Meaning
NO
Not filterable. The default
YES
Optional equality filter
ALWAYS
Required equality filter: every query must constrain this column
RANGE
Range filter, >, < and BETWEEN, as well as equality
RANGE is only allowed on numeric columns, integer and number. Columns of type date-time are
always range-searchable whatever the value says.
The sorts section
Sorting is declared in the definition, not composed at query time. The sorts section holds a list of
named sorts, each an ordered list of columns with a direction, and a query asks for one by its
identifier.
Required, unique within the list. Letters, digits, spaces, _ and -. Generated as a UUID if omitted, so name it
description
Optional free text
columns
Required, at least one. A column name from the columns section plus ASC or DESC
At least one sort is mandatory, and the order inside columns is the sort precedence.
The reverse of every sort comes for free
For each sort you declare, OpenGate also exposes its reverse, served by the same index through a reverse
scan. Those come back marked derived: true, which is read-only: the platform sets it and the web console
uses it. Never declare a derived sort yourself — flipping the direction of one you already have just spends
optimization units on an index you were given.
Limits
There is no fixed maximum number of filterable columns or declared sorts. Each filterable column and
each declared sort consumes optimization units, and the data set has a budget of them — that budget is
the limit.
Where to look
What it tells you
The searchOptimizationInfo of a data set
usedSearchOptimizationUnits and freeSearchOptimizationUnits
POST .../optimizationPlan
What a definition would consume, before committing to it
POST /north/v80/datasets/provision/organizations/{organizationName}/optimizationPlan
Creating
POST /north/v80/datasets/provision/organizations/{organizationName}
Updating
PUT /north/v80/datasets/provision/organizations/{organizationName}/{identifier}
Updating can affect the data already stored or the structure holding it, which starts an adaptation process.
Until it completes, dirty values may be present.
These fields can be modified:
Name · Description · IdentifierColumn · Columns · Sorts
Rules for columns:
Names are unique. You cannot add or rename a column to a name already in use.
filter: ALWAYS is immutable. You cannot add or remove a column that has it, you cannot set it on an
existing column, and you cannot change it away once set.
Paths cannot be edited. Remove the column and create it again, which gets you the same result.
The optimization unit budget applies to updates as well, so run optimizationPlan before adding filterable
columns or sorts to a definition that is already close to it.
Listing and deleting
GET /north/v80/datasets/provision/organizations/{organizationName}GET /north/v80/datasets/provision/organizations/{organizationName}/{identifier}DELETE /north/v80/datasets/provision/organizations/{organizationName}/{identifier}
Read the organization’s data sets:
curl --request GET \
--header "X-ApiKey: <your-api-key>"\
https://api.opengate.es/north/v80/datasets/provision/organizations/{organizationName}
Querying a data set
Reading a data set is a POST with the data set identifier in the URL:
POST /north/v80/datasets/provision/organizations/{organizationName}/{identifier}/data
Without a select clause, columns holds the identifier column first, then the defined columns in
declaration order.
The request body
Data set queries use the same clauses as any other search, with two differences worth memorising:
Clause
In a data set query
filter
Standard operators, keyed by identifierColumn or a column name
sort
A string: the identifier of a sort declared in the data set — see below
select
An array of column names, not the object form used elsewhere
limit
start and size, as everywhere else
group
Does not exist for data sets
The full comparison against the other query dialects is in Query dialects.
Asking for a sort
You do not compose an ordering in the request. You name one that already exists:
{ "filter": {}, "sort": "sortByDeviceAsc" }
Valid values are the identifier of any sort in the data set definition, plus the automatically exposed
reverse of each one, so declaring an ascending sort gives you the descending direction too.
Omit sort and results come back sorted by the identifier column, ascending.
There is no fixed limit on how many sorts a definition can hold; the constraint is the optimization unit
budget, described in Defining a data set.
Pagination and CSV
Data sets answer in JSON or CSV, and the format changes what an absent limit means:
CSV retrieval turns sorting off on purpose: it is what makes large exports fast, and CSV output is
usually consumed by something that will sort it anyway.
Complete retrieval is expensive
Omitting limit in CSV mode downloads the whole data set. Page it unless you truly want everything.
CSV formatting is customizable through HTTP header options — the quoting character (double quotes by
default), the escape character (a backslash by default), the end-of-line sequence (\n by default) and how
nulls are represented. You are responsible for the combination producing well-formed CSV. The exact header
names are not currently published, so ask your platform contact for them.
The other data set endpoints
Three more endpoints exist, and one of them is not what its URL suggests:
POST /north/v80/search/catalog/datasets
POST /north/v80/search/organizations/{organizationName}/datasets/{datasetId}POST /north/v80/search/organizations/{organizationName}/datasets/{datasetId}/summary
search/catalog/datasets lists the data sets available to you.
The other two are not a mirror of the .../data read above: they take a different request body. The
.../data endpoint uses the data set’s own dialect — sort as a declared identifier, select as an array
of column names, no group. These two take the generic Data Lake search body, with sort as the
{"parameters": [{"name": ..., "type": ...}]} object, select in its object form, and group available.
Endpoint
Request body
datasets/provision/.../{identifier}/data
Data set dialect: sort is a declared sort identifier
Use .../data unless you specifically need the generic clauses. Which of the two is intended to be the
long-term path has not been confirmed by the product team.