Introduction

The Cognite API is the main entrypoint into the Cognite Data Platform (CDP). Here you can interact with the different components and data within the Cognite Data Platform. This page will serve as an API guide, introducing the various resources and how to work with them.

Cognite Data Platform

platform diagram

As an end user or application developer you wil access and retrieve data stored in the Contextualized Data Layer. This is data such as time series data, files, technical documents, relational data (assets, events, metadata, ERP/CRM data, etc) and so on.

Data access can be done directly by using our RESTful API, by using provided SDK’s, or by using any of the applications developed on top of the Cognite Data Platform.

Data added to the platform is written to the RAW data layer. In RAW we store the data as extracted from the source without any transformations applied. Contextualization and transformations are then applied on the data at any interval, preparing the data for access through the Contextualized data layer.

Data ingestion can be done directly by using our API, by using any of our data connectors, by using an existing ETL tool, or by developing your own connector.

Authentication

To use the Cognite API you must be authenticated, either by using an API key or retrieving a JWT through a configured active directory connection.

API Key

If you have an API key, you’ll authenticate requests by sending the API key in the Api-Key header.

You can check whether your API key is valid like this:

POST /login HTTP/1.1
Content-Type: application/json
Host: api.cognitedata.com
Content-Length: 31

{
  "apiKey": "DEFAULTAPIKEY"
}

Post body fields:

Path Type Description

apiKey

String

The API key to login with.

This should give a response similar to:

HTTP/1.1 200 OK
X-Request-Id: 82dfe96cccc67681
Content-Type: application/json

{
  "data": {
    "user": "defaultuser@test.com",
    "loggedIn": true
  }
}
Path Type Description

data

Object

Container object for the response.

data.user

String

Email of the user.

data.loggedIn

Boolean

Boolean indicating whether the user is logged in.

Json Web Token

If you have a Json Web Token, or JWT, (retrieved through a configured Active Directory connection), you’ll authenticate requests by sending the token in the Authorization header.

You can check whether your JWT is valid like this:

GET /login/token?token=ewogICJhY2NvdW50X3R5cGUiOiAic2VydmljZV9hY2NvdW50IiwKICAicHJvamVjdF9pZCI6IDEsCiAgImFjY291bnRfaWQiOiA0MiwKICAic2lnbmluZ19rZXkiOiAiN2ZiZjVjYjUtZGViMy00MDI4LWJmNjktMDcwZWQzYzgwY2NjIiwKICAiZXhwaXJlX3RpbWUiOiAxNTI3MDg0MjIxCn0%3D.YYH6rdIySp9O4FxxBWqUxuABPOxUo1nZbpO%2FPx%2FLvazkAcGWR6msIKuvqrlVGkJhrIE3EPdizNcBSB5PsDRQPu3NvvQnwCxUSLTQ5pN8bkcDO3L%2BjLFjGdFzRN7R%2Bo3NIS%2Fnw2uA0CiI0Ht%2B3gNEMPaWJQgEy01Gy%2FBXUC1uGGHobG8rEMEZAUXY4H0thNwWU6OxADbXC8KV2VXuuFP6GBs%2Faf%2BPdYYfznjVhzC0Un%2FCRRLPBAj%2FSd%2B7Yx6JPu8EBRUueOrFSyizI20W0azRQsdftXZslgnfyW6h%2F%2Bzm4%2FGmd0CROzKQRksaT0JYWizXAyuXjBHl88MOkLTswTM2Gw%3D%3D HTTP/1.1
Host: api.cognitedata.com

Query parameters:

Parameter Description

token

The token to check status for.

This should give a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "token": "ewogICJhY2NvdW50X3R5cGUiOiAic2VydmljZV9hY2NvdW50IiwKICAicHJvamVjdF9pZCI6IDEsCiAgImFjY291bnRfaWQiOiA0MiwKICAic2lnbmluZ19rZXkiOiAiN2ZiZjVjYjUtZGViMy00MDI4LWJmNjktMDcwZWQzYzgwY2NjIiwKICAiZXhwaXJlX3RpbWUiOiAxNTI3MDg0MjIxCn0\u003d.YYH6rdIySp9O4FxxBWqUxuABPOxUo1nZbpO/Px/LvazkAcGWR6msIKuvqrlVGkJhrIE3EPdizNcBSB5PsDRQPu3NvvQnwCxUSLTQ5pN8bkcDO3L+jLFjGdFzRN7R+o3NIS/nw2uA0CiI0Ht+3gNEMPaWJQgEy01Gy/BXUC1uGGHobG8rEMEZAUXY4H0thNwWU6OxADbXC8KV2VXuuFP6GBs/af+PdYYfznjVhzC0Un/CRRLPBAj/Sd+7Yx6JPu8EBRUueOrFSyizI20W0azRQsdftXZslgnfyW6h/+zm4/Gmd0CROzKQRksaT0JYWizXAyuXjBHl88MOkLTswTM2Gw\u003d\u003d",
    "valid": true,
    "expired": false
  }
}
Path Type Description

data

Object

Container object for the response.

data.token

String

The token.

data.valid

Boolean

Boolean indicating whether the token is valid.

data.expired

Boolean

Boolean indicating whether the token has expired.

For guidance on how to integrate with the Cognite API with an Active Directory connection, see the Cognite API Integration Guide.

HTTP Overview

This section explains how the Cognite API considers HTTP verbs, HTTP status codes and errors.

HTTP verbs

The Cognite API tries to adhere as closely as possible to standard HTTP and REST conventions in its use of HTTP verbs.

Verb Usage

GET

Used to retrieve a resource

POST

Used to create a new resource, or to delete many resources

PATCH

Used to update an existing resource, including partial updates

DELETE

Used to delete an existing resource

HTTP status codes

The Cognite API tries to adhere as closely as possible to standard HTTP and REST conventions in its use of HTTP status codes.

Status code Usage

200 OK

The request completed successfully

201 Created

A new resource has been created successfully. The resource’s URI is available from the response’s Location header

400 Bad Request

The request was malformed. The response body will include an error providing further information

401 Unauthorized

The request was not authorized. This usually means that no authentication information was provided.

403 Forbidden

The operation is not allowed given the authenticated user’s permissions. This error could appear if one attempts to access a restricted resource, for example.

404 Not Found

The requested resource did not exist

405 Method Not Allowed

The request method is not supported for the requested resource: for example, a GET request on a resource that only responds to POST requests.

Errors

Whenever an error response (status code >= 400) is returned, the body will contain a JSON object that describes the problem. The error object has the following structure:

Path Type Description

error

Object

The error object.

error.code

Number

Status code of the error.

error.message

String

Error message.

For example, a request that attempts to access a project that doesn’t exist will produce a 403 Forbidden response:

HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "error": {
    "code": 401,
    "message": "HTTP 401 Unauthorized"
  }
}

API responses

Data wrappers

When fetching data from the Cognite API, the results will be wrapped in one of two data types: Data or DataWithCursor.

This is an example of what a response wrapped in a Data object can look like:

GET /api/0.4/projects/test/timeseries/data/price?start=0&end=1500000000000 HTTP/1.1
Host: api.cognitedata.com
HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "tagId": "price",
        "datapoints": [
          {
            "timestamp": 1492732801096,
            "value": 799.6102905273438
          },
          {
            "timestamp": 1492732840095,
            "value": 805.0836181640625
          },
          {
            "timestamp": 1492732908096,
            "value": 804.9093017578125
          }
        ]
      }
    ]
  }
}

This is an example of what a response wrapped in a DataWithCursor object can look like:

GET /api/0.5/projects/myproject/assets?limit=3&cursor=middleCursor HTTP/1.1
Host: api.cognitedata.com
HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "previousCursor": "prevCursor",
    "nextCursor": "nextCursor",
    "items": [
      {
        "id": 4,
        "path": [
          4
        ],
        "name": "asset4",
        "description": "A particularly interesting asset."
      },
      {
        "id": 5,
        "path": [
          5
        ],
        "name": "asset5",
        "metadata": {
          "someKey": "someValue"
        }
      },
      {
        "id": 6,
        "path": [
          4,
          8
        ],
        "name": "asset6",
        "parentId": 4
      }
    ]
  }
}

The difference between Data and DataWithCursor is merely that the latter also has cursors that you can use to navigate through pages of results. The cursor is a random string that can be copied and sent with subsequent requests to navigate through pages of results.

Response types

The various response types the Cognite API returns are described in the API reference.

Software Development Kit

To make your experience working with the Cognite Data Platform easier, we offer several software development kits, or SDKs:

We also publish an OpenAPI specification of the Cognite API, which is available at http://doc.cognitedata.com/openapi.json. Using Swagger codegen with this will allow you to generate client libraries to the Cognite API in your language of choice, effectively creating a simple SDK.

Resources

Resources in the Cognite API are accessed under the https://api.cognitedata.com/api/{version}/projects/{project}/ URL. The {version} placeholder should be replaced with the version of the API, e.g. 0.4, and the {project} placeholder should be replaced with the project identifier.

All the example requests in this API guide have been exported to a Postman collection that is available here. Download this and open it in Postman to run the requests yourself.

Projects

The top-level entity in the Cognite Data Platform is the project. All other resources (assets, events, time series, etc.) are tied to a project.

Get project

A request to get a project can look like this:

GET /api/0.4/projects/zero HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "name": "Test",
        "urlName": "42",
        "authentication": {
          "type": "AzureAD",
          "protocol": "oauth2",
          "url": "login.example.com"
        }
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].name

String

The name of the project.

data.items.[].urlName

String

The urlName of the project.

data.items.[].authentication

Object

The description of the semantic type.

data.items.[].authentication.type

String

The type of authentication used in the project.

data.items.[].authentication.protocol

String

The protocol used for authentication in the project.

data.items.[].authentication.url

String

The url used for authentication in the project.

Assets

An asset can be any kind of physical equipment or some logical group of equipment. This could for example be a pump, a sensor on a pipe, a platform, a ship, etc.

An asset has the properties name, parentId, description, and metadata. When fetching assets from the API, an asset (wrapped in a data container) will look like this:

Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The ID of the asset.

data.items.[].name

String

The name of the asset.

data.items.[].parentId

Number

The ID of the parent asset.

data.items.[].description

String

Description of the asset.

data.items.[].path

Array

IDs of assets on the path to the asset.

data.items.[].metadata

Object

Any necessary metadata about the asset.

An asset hierarchy can look like this:

A
├── B
│   ├── C
│   └── D
├── E
│   └── F
├── G
└── H

Here the asset A is the root. A has four children: B, E, G, and H. B has two children: C and D, while E has one child: F.

If an asset represents a sensor, then a Time Series can refer to that asset. Events will also refer to an asset.

Assets can be managed at the {project}/assets url. What follows are a few examples of the methods available. See the API reference for a full overview.

Get assets

A request to get assets can look like this:

GET /api/0.5/projects/myproject/assets HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "id": 1,
        "path": [
          1
        ],
        "name": "Asset 1"
      },
      {
        "id": 2,
        "path": [
          2
        ],
        "name": "Asset 2"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The ID of the asset.

data.items.[].name

String

The name of the asset.

data.items.[].path

Array

IDs of assets on the path to the asset.

Get asset subtree using a cursor

A request to get a specific asset with its children, using a cursor, can look like this:

GET /api/0.5/projects/myproject/assets/1/subtree?depth=5&cursor=This+is+the+cursor HTTP/1.1
Host: api.cognitedata.com

Path parameters:

Table 1. /api/{version}/projects/{project}/assets/{assetId}/subtree
Parameter Description

assetId

ID of the asset.

Query parameters:

Parameter Description

depth

How far down in the tree to fetch assets.

cursor

Previously fetched cursor to start navigation from.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "id": 1,
        "path": [
          1
        ],
        "name": "Asset 1",
        "description": "Root asset.",
        "metadata": {
          "someKey": "someValue"
        }
      },
      {
        "id": 2,
        "path": [
          1,
          2
        ],
        "name": "Asset 2",
        "parentId": 1
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The ID of the asset.

data.items.[].name

String

The name of the asset.

data.items.[].parentId

Number

The ID of the parent asset.

data.items.[].description

String

Description of the asset.

data.items.[].path

Array

IDs of assets on the path to the asset.

data.items.[].metadata

Object

Any necessary metadata about the asset.

Events

An event is something that has happened which is of importance, and that you wish to store information about. This could be that a certain component wasn’t performing at an optimal rate for a certain period of time, that there was a leakage from a pipe, etc.

Events can be created from Event Transformers.

Events can be managed at the {project}/events url. What follows are a few examples of the methods available. See the API reference for a full overview.

Get events

A request to get events can look like this:

GET /api/0.4/projects/1/events HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "previousCursor": "previous",
    "nextCursor": "next",
    "items": [
      {
        "id": 0,
        "startTime": 123,
        "endTime": 234,
        "description": "description 0",
        "type": "type 0",
        "subtype": "subtype0",
        "tagIds": [
          "tag1",
          "tag2",
          "tag3",
          "tag4"
        ],
        "eventTransformerId": 1,
        "eventReferenceKey": "0"
      },
      {
        "id": 1,
        "startTime": 123,
        "endTime": 234,
        "description": "description 1",
        "type": "type 1",
        "subtype": "subtype1",
        "additionalData": {
          "id": "1",
          "link": "abc"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "1"
      },
      {
        "id": 2,
        "startTime": 123,
        "endTime": 234,
        "description": "description 2",
        "type": "type 2",
        "subtype": "subtype2",
        "tagIds": [
          "tag1",
          "tag2",
          "tag3",
          "tag4"
        ],
        "eventTransformerId": 1,
        "eventReferenceKey": "2"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.previousCursor

String

Cursor to go to the previous result.

data.nextCursor

String

Cursor to go to the next result.

data.items.[].id

Number

The ID of the event.

data.items.[].startTime

Number

Timestamp when the event started.

data.items.[].endTime

Number

Timestamp when the event ended.

data.items.[].description

String

Description of the event.

data.items.[].type

String

Type of the event.

data.items.[].subtype

String

Subtype of the event.

data.items.[].additionalData

Object

Any additional metadata about the event.

data.items.[].tagIds

Array

Tags associated with the event.

data.items.[].eventTransformerId

Number

Event transformer id.

data.items.[].eventReferenceKey

String

Event reference key.

Get events by reference

A request to get events by transformer id and reference keys can look like this:

POST /api/0.4/projects/1/events/byreference HTTP/1.1
Content-Type: application/json
Host: api.cognitedata.com
Content-Length: 62

{"transformerId":1,"eventReferenceKeys":["0","1","2","3","4"]}

Post body fields:

Path Type Description

transformerId

Number

The transformer id to get events for.

eventReferenceKeys

Array

The event reference keys to get events for.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "id": 0,
        "startTime": 1,
        "additionalData": {
          "id": "1",
          "order_type": "BK"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "0"
      },
      {
        "id": 1,
        "startTime": 1,
        "additionalData": {
          "id": "1",
          "order_type": "BK"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "1"
      },
      {
        "id": 2,
        "startTime": 1,
        "additionalData": {
          "id": "1",
          "order_type": "BK"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "2"
      },
      {
        "id": 3,
        "startTime": 1,
        "additionalData": {
          "id": "1",
          "order_type": "BK"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "3"
      },
      {
        "id": 4,
        "startTime": 1,
        "additionalData": {
          "id": "1",
          "order_type": "BK"
        },
        "eventTransformerId": 1,
        "eventReferenceKey": "4"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The ID of the event.

data.items.[].startTime

Number

Timestamp when the event started.

data.items.[].additionalData

Object

Any additional metadata about the event.

data.items.[].eventTransformerId

Number

Event transformer id.

data.items.[].eventReferenceKey

String

Event reference key.

Event transformers

An event transformer is a representative for a process, a pipeline, or some sort of entity that generates events from raw data. An event transformer could be AnomalyDetectionPipeline, that takes sensor data as input and generates events. The event transformers in the Cognite Data Platform are just names, referring to actual processes, pipeline, etc.

Event transformers can be managed at the {project}/eventtransformers url. What follows are a few examples of the methods available. See the API reference for a full overview.

Get event transformers

A request to get event transformers can look like this:

GET /api/0.4/projects/test/eventtransformers HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "id": 1,
        "name": "AnomalyDetection",
        "description": "desc1"
      },
      {
        "id": 2,
        "name": "OtherTransformer",
        "description": "desc2"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The id of the event transformer.

data.items.[].name

String

The name of the event transformer.

data.items.[].description

String

The description of the event transformer.

Semantic types

Semantic types are your descriptions of data. For example, if you want to store a STRING that represents a manufacturer, you’d might create a semantic type with name="manufacturer" and description="A string containing the manufacturer name".

Semantic phrases have a semantic type, which describes their meaning. An event can have a list of semantic phrases in it, containing data about the event.

Semantic types can be managed at the {project}/semantictypes url. What follows are a few examples of the methods available. See the API reference for a full overview.

Get semantic types

A request to get semantic types can look like this:

GET /api/0.4/projects/1/semantictypes HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "id": 1,
        "name": "Manufacturer",
        "description": "Name of the manufacturer."
      },
      {
        "id": 2,
        "name": "Origin",
        "description": "Name of the country of origin."
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].id

Number

The ID of the semantic type.

data.items.[].name

String

The name of the semantic type.

data.items.[].description

String

The description of the semantic type.

Storage

Uploaded files can be managed at the {project}/storage url. What follows are a few examples of the methods available. See the API reference for a full overview.

List files

A request to list files can look like this:

Unresolved directive in api-guide.adoc - include::/home/jenkins/workspace/github-builds_api_master-562G3MGSG52VYA74AAGXYQBJFJOIFZ6XNGOFZD6VSOXRR3OHGU7A/api/target/generated-snippets/storage/listFiles/http-request.adoc[]

This request should yield a response similar to:

Unresolved directive in api-guide.adoc - include::/home/jenkins/workspace/github-builds_api_master-562G3MGSG52VYA74AAGXYQBJFJOIFZ6XNGOFZD6VSOXRR3OHGU7A/api/target/generated-snippets/storage/listFiles/http-response.adoc[] Unresolved directive in api-guide.adoc - include::/home/jenkins/workspace/github-builds_api_master-562G3MGSG52VYA74AAGXYQBJFJOIFZ6XNGOFZD6VSOXRR3OHGU7A/api/target/generated-snippets/storage/listFiles/response-fields.adoc[]

Raw data storage

In addition to storing files, you can store raw data in a NoSQL-like manner in the form of key/value-pairs in the Cognite Data Platform by using the Raw API. This is useful if you need to "dump" raw data into our platform before running a data cleaning pipeline. In the raw API you can create databases, tables within databases, and rows of data.

Raw databases can be managed at the {project}/raw url. What follows are a few examples of the methods available. See the API reference for a full overview.

List databases

A request to list databases can look like this:

GET /api/0.4/projects/1/raw HTTP/1.1
Host: api.cognitedata.com

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "nextCursor": "thisIsAPossibleCursor",
    "items": [
      {
        "dbName": "db1"
      },
      {
        "dbName": "db2"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.nextCursor

String

Cursor to go to the next result.

data.items.[].dbName

String

Name of the database.

List tables in a database

A request to list tables in a database can look like this:

GET /api/0.4/projects/1/raw/db1 HTTP/1.1
Host: api.cognitedata.com

Path parameters:

Table 2. /api/{version}/projects/{project}/raw/{database}
Parameter Description

database

The database to get tables for.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "nextCursor": "thisIsAPossibleCursor",
    "items": [
      {
        "tableName": "table1"
      },
      {
        "tableName": "table2"
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.nextCursor

String

Cursor to go to the next result.

data.items.[].tableName

String

Name of the table.

List rows in a table

A request to list rows in a table can look like this:

GET /api/0.4/projects/1/raw/db1/table1 HTTP/1.1
Accept: application/json
Host: api.cognitedata.com

Path parameters:

Table 3. /api/{version}/projects/{project}/raw/{database}/{table}
Parameter Description

database

The database to get rows from.

table

The table to get rows for.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "nextCursor": "thisIsAPossibleCursor",
    "items": [
      {
        "key": "event1",
        "columns": {
          "type": "party",
          "price": 20,
          "participants": [
            1,
            2
          ],
          "duration": {
            "hours": 2.3
          }
        }
      },
      {
        "key": "event2",
        "columns": {
          "type": "meal",
          "price": 5
        }
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.nextCursor

String

Cursor to go to the next result.

data.items.[].key

String

The key of the row entry.

data.items.[].columns

Object

The values (columns) in the row entry.

Time series

Time series can be managed at the {project}/timeseries url. What follows are a few examples of the methods available. See the API reference for a full overview.

List time series

A request to get the time series in a project can look like this:

GET /api/0.4/projects/test/timeseries?includeMetadata=true&description=a HTTP/1.1
Host: api.cognitedata.com

Query parameters:

Parameter Description

includeMetadata

Boolean indicating whether to include metadata in the response.

description

Search for a substring of the description.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "tagId": "another.test",
        "isString": false,
        "isStep": false,
        "description": "an insignificant metric"
      },
      {
        "tagId": "simen.sine",
        "isString": false,
        "isStep": false,
        "description": "another metric of no interest whatsoever"
      },
      {
        "tagId": "test",
        "isString": true,
        "isStep": true,
        "description": "are you really interested in these metrics?"
      },
      {
        "tagId": "test.test",
        "isString": false,
        "metadata": {
          "test type": "very testy test"
        },
        "isStep": false,
        "description": "this description contains two a\u0027s."
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].tagId

String

The ID of the tag.

data.items.[].isString

Boolean

Boolean indicating whether the value stored in the time series is a string.

data.items.[].description

String

Description of the timeseries.

data.items.[].metadata

Object

Any stored metadata.

data.items.[].isStep

Boolean

Whether the time series is a step series or not.

Create new time series

You can create a post request to create new time series, like this:

POST /api/0.4/projects/test/timeseries HTTP/1.1
Content-Type: application/json
Host: api.cognitedata.com
Content-Length: 107

{
  "items":[
    {"tagId":"stabburtest", "metadata":{"unit":"kilograndiosa"}},
    {"tagId":"test2"}
  ]
}

Post body fields:

Path Type Description

items

Array

The list of time series to create.

items.[].tagId

String

tagId for the new time series.

items.[].metadata

Object

Any metadata.

A successful request will return the empty response {} with status code 200 OK.

Get data points in a time series

A request to get data points in a time series can look like this:

GET /api/0.4/projects/test/timeseries/data/price?start=0&end=1500000000000 HTTP/1.1
Host: api.cognitedata.com

Path parameters:

Table 4. /api/{version}/projects/{project}/timeseries/data/{tagId}
Parameter Description

tagId

The id of the tag to get data points from.

Query parameters:

Parameter Description

start

Start timestamp for the values.

end

End timestamp for the values.

This request should yield a response similar to:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "tagId": "price",
        "datapoints": [
          {
            "timestamp": 1492732801096,
            "value": 799.6102905273438
          },
          {
            "timestamp": 1492732840095,
            "value": 805.0836181640625
          },
          {
            "timestamp": 1492732908096,
            "value": 804.9093017578125
          }
        ]
      }
    ]
  }
}
Path Type Description

data

Object

Container object for the response. This can contain a list of items in addition to cursor for navigation.

data.items.[].tagId

String

The ID of the tag.

data.items.[].datapoints

Array

The data points.

data.items.[].datapoints.[].timestamp

Number

The timestamp of the recorded value.

data.items.[].datapoints.[].value

Number

The recorded value.

Insert data points into a time series

You can create a post request to insert data into a time series, like this:

POST /api/0.4/projects/test/timeseries/data/price HTTP/1.1
Content-Type: application/json
Host: api.cognitedata.com
Content-Length: 195

{
  "items":[
    {"timestamp":1492732801096,"value":799.6102905273438},
    {"timestamp":1492732840095,"value":805.0836181640625},
    {"timestamp":1492732908096,"value":804.9093017578125}
  ]
}

Path parameters:

Table 5. /api/{version}/projects/{project}/timeseries/data/{tagId}
Parameter Description

tagId

The id of the tag to get data points from.

Post body fields:

Path Type Description

items

Array

The list of data points to insert into a time series.

items.[].timestamp

Number

Timestamp for when the value was recorded.

items.[].value

Number

The recorded value.

A successful request will return the empty response {} with status code 200 OK.

Workflows

This section contains examples of common tasks, or workflows, performed in the Cognite API.

Get datapoints from a time series given an asset ID

Objective: Given the asset ID 124, find a related time series and get some data points from that time series.

  1. Find time series related to the asset ID

GET /api/0.4/projects/test/timeseries?assetId=124 HTTP/1.1
Host: api.cognitedata.com

Query parameters:

Parameter Description

assetId

The asset ID to find related time series for.

This gives us the response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "tagId": "price",
        "isString": false,
        "unit": "US dollars",
        "assetId": 124,
        "isStep": false
      }
    ]
  }
}

From this response we see that there’s one time series related to our asset ID, that is the one with tagId=price. We’ll use this in the next step.

  1. Get data points in the time series

Now that we know the tagId, we can get data points from the time series:

GET /api/0.4/projects/test/timeseries/data/price?start=0&end=1500000000000 HTTP/1.1
Host: api.cognitedata.com

Query parameters:

Parameter Description

start

Start timestamp for the values.

end

End timestamp for the values.

This gives us the response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "data": {
    "items": [
      {
        "tagId": "price",
        "datapoints": [
          {
            "timestamp": 1492732801096,
            "value": 799.6102905273438
          },
          {
            "timestamp": 1492732840095,
            "value": 805.0836181640625
          },
          {
            "timestamp": 1492732908096,
            "value": 804.9093017578125
          }
        ]
      }
    ]
  }
}

Machine learning

This section contains information about the machine learning capabilites in the Cognite API.

Packaging a TensorFlow trainer for Cognite compatibility

Google ML Engine

We use Google ML Engine to host TensorFlow models, this means two things:

  • We enable you to train machine learning models at scale by running TensorFlow training applications in the cloud.

  • We host those trained models for you in the cloud so that you can use them to get predictions about new data.

TensorFlow trainers should therefore be packaged to work with this framework. For more information see: https://cloud.google.com/ml-engine/docs/packaging-trainer.

Cognite Model Hosting Environment (CMHE)

As a user you will not interact with Google ML Engine on production level - this part of the model hosting is abstracted away. We simply let users train and predict on models through our api using trainer and data specifications.

For a training job the user specifies a trainer (uploaded by the user or provided by our ml toolkit) and which data to train on. Our hosting environment will then fetch the appropriate data from the platform, perform preprocessing and initiate the training job. When the training job completes, the user will be able to perform predictions on the model through an endpoint in our public API.

For a prediction job (batch or online) the user specifies a model id and which data to train on.

Preprocessing

Seeing as CMHE fetches the specified data on behalf of the user performing the prediction requests, it will also be performing necessary preprocessing steps to prepare the data for your model. We are very flexible here as we allow you to write arbitrary preprocessing code and store necessary objects such as label encoders or data scalers as pickled objects for us to retrieve upon preprocessing prediction data. In order to achieve this we do impose certain requirements on these data processing methods in your project. They must follow strict naming conventions, receive specific parameters and store data processing objects in locations provided by CMHE. This is described in detail below and is shown in the examples given in the last section.

Project structure

Requirements imposed to ensure that your trainer is compatible with the Cognite Model Hosting Environment are described here. The following structure is used in many Cloud ML Engine samples and by the in-house data scientists at Cognite, and having your project’s organization be similar to the this can make your life easier.

tf trainer proj structure

The following parts of this structure are a prerequisite in order for your trainer to be compatible with Cognites model hosting environment:

  • Use a subdirectory named 'trainer' to store your main application module.

  • 'trainer' must contain the trainer application module task.py which contains the trainer logic that manages the job.

    • This script must take the following arguments for CMHE to fill:

      • --train-files which contains the Google Cloud Storage location of the training files

      • --eval-files which contains the Google Cloud Storage location of the test files

      • --job-dir which contains the Google Cloud Storage location of the job directory

      • --pp-dir which contains the Google Cloud Storage location for storing and retrieving data processing objects.

  • 'trainer' must contain processing.py which contains methods used to preprocess training and prediction data as well as doing inverse transformations on prediction results.

    • The method which is used to preprocess prediction data must fulfill these requirements:

      • Must be named 'preprocess_prediction_data'

      • Must take a dataframe named 'df' and a directory named 'pp_dir' (containing objects to be used in the processing steps) as input

      • Must return a tuple containing input and output features as numpy arrays.

    • The method which is used to inverse transform prediction results must fulfill the following requirements:

      • Must be named 'inverse_transform_prediciton'.

      • Must take a prediction result named 'prediction' (as returned from Google ML Engines online predict functionality) and a directory named 'pp_dir' (containing objects to be used in the processing steps) as input.

      • Must return the prediction result with the same shape as the input.

The following parts of the project structure are recommended but not required:

  • Using a main project directory, containing your setup.py file.

  • model.py which contains the TensorFlow graph code—the logic of the model.

  • Whatever other subdirectories in your main project directory that you need to implement your application

Remember to create an init.py file in every subdirectory. These files are used by Setuptools to identify directories with code to package, and may be empty.

Examples

Here is an example Feed-Forward Neural network written using Keras which is compatible with Google ML Engine: https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census/keras.

Here is an example LSTM trainer package written using Keras which is fully compatible with both Google ML Engine and the Cognite Model Hosting Environment: https://github.com/cognitedata/ml-hosting-examples/tree/master/lstm.