Worker Manager API

Overview

Worker manager is a service which runs alongside every qserv worker or czar and manages many common operations:

database operations (table creating, data loading, etc.)
certain xrootd (plug-in) operations
management of other services’ lifetime

Primary motivation for implementing wmgr service was to facilitate data loading in integration tests. It is likely that some of the wmgr functions may be changed in the future once we implement production-level services for data loading and distribution. For more details on wmgr design consult pages:

This document describes wmgr HTTP-basedAPI in details. Note that there is also a Python client library implemented on top of HTTP API.

General API description

Wmgr API is implemented on top of HTTP protocol following RESTful principles (where possible). All communication with wmgr is performed over HTTP without encryption (implementing SSL/TLS is feasible but needs solution for certificate management).

Wmgr can be configured to require one of few authentication mechanisms - none, basic, or digest. Lack of SSL/TLS dictates use digest authentication for production services.

Whole wmgr API is split into three groups - database, xrootd, and process API. Below is description of individual API methods (actions).

Return codes and content types

Responses generated by wmgr service use regular HTTP status codes to indicate success or error. Codes in range 200-299 are used for success, 400-499 for errors (range 500-599 typically means service failure).

Normally responses generated by wmgr service include data in JSON format and have their Content-Type header set to application/json. This applies to both successful completion and error conditions. In some cases when error condition is returned by the transport/framework layer it will have different content type.

See also document which defines general structure of JSON objects for returned response data: https://confluence.lsstcorp.org/display/DM/REST+API+General+Guidelines

Database API

This section contains description of actions related to database operations - creating/deleting databases and tables, loading table data, etc.

`GET /dbs`

Return the list of all existing database names.
Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

500 - for database errors

Response body (for successful completion):

JSON object, “results” property is a list of database objects with keys:

name: database name

uri: URL for database operations

Request/response example:
GET /dbs HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "qservMeta",
      "uri": "/dbs/qservMeta"
    },
    {
      "name": "qservResult",
      "uri": "/dbs/qservResult"
    }
  ]
}

`POST /dbs`

Create new database. In addition to creating database itself this method also grants all privileges on this database to regular non-privileged account.
Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

db: Database name

Response headers:

Content-Type: application/json

Status Codes:

201 - if database was successfully created

400 - if database name is missing

409 - if database already exists

500 - for other database errors

Response body (for successful completion):

JSON object, “result” property is a database object with keys:

name: database name

uri: URL for database operations

Request/response example:
POST /dbs HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------bb306714c15713c2

--------------------------bb306714c15713c2
Content-Disposition: form-data; name="db"

newDB
--------------------------bb306714c15713c2--
HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/dbs/newDB"
  }
}

`DELETE /dbs/<dbName>`

Deletes database.
Parameters:

dbName: database name

Response headers:

Content-Type: application/json

Status Codes:

200 - if database was successfully deleted

400 - if parameters have invalid format

404 - if database does not exist

500 - for other database errors

Response body (for successful completion):

JSON object, “result” property is a database object with keys:

name: database name

uri: URL for database operations

Request/response example:
DELETE /dbs/newDB HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/dbs/newDB"
  }
}

`GET /dbs/<dbName>/tables`

Returns the list of tables in a database.
Parameters:

dbName: database name

Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

400 - if parameters have invalid format

404 - if database does not exist

500 - for database errors

Response body (for successful completion):

JSON object, “results” property is a list of table objects with keys:

name: table name

uri: URL for database operations

Request/response example:
GET /dbs/qservMeta/tables HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "QCzar",
      "uri": "/dbs/qservMeta/tables/QCzar"
    },
    {
      "name": "QInfo",
      "uri": "/dbs/qservMeta/tables/QInfo"
    },
    ...
  ]
}

`POST /dbs/<dbName>/tables`

Create new table.

If schemaSource (see below) is “request” then request must include schema parameter which is an SQL DDL statement starting with ‘CREATE TABLE TableName …’.

If schemaSource is “css” then table parameter must be specified. Table schema will be extracted from CSS in this case, schemaSource must not be given.
Parameters:

dbName: database name

Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

table: Table name

schemaSource: source for schema name, possible values: “request”, “css”, (default: “request”)

schema: complete “CREATE TABLE …” statement (optional)

chunkColumns: boolean flag, false by default, accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’. If set to true then delete columns “_chunkId”, “_subChunkId” from table (if they exist) and add columns “chunkId”, “subChunkId” (if they don’t exist)

Response headers:

Content-Type: application/json

Status Codes:

201 - if table was successfully created

400 - if parameters have invalid format or if form parameters are missing or conflicting

409 - if table already exists

500 - if table is not defined in CSS or other database errors

Response body (for successful completion):

JSON object, “result” property is a table object with keys:

name: database name

uri: URL for database operations

Request/response example:
POST /dbs/newDB/tables HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------c5c44964f0f9add0

--------------------------c5c44964f0f9add0
Content-Disposition: form-data; name="schema"

CREATE TABLE newTable (I INT)
--------------------------c5c44964f0f9add0
Content-Disposition: form-data; name="table"

newTable
--------------------------c5c44964f0f9add0--
HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "name": "newTable",
    "uri": "/dbs/newDB/tables/newTable"
  }
}

`DELETE /dbs/<dbName>/tables/<tblName>`

Drop a table and optionally all chunk/overlap tables.
Parameters:

dbName: database name

tblName: table name

Query Parameters:

dropChunks: boolean flag, false by default, accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’

Response headers:

Content-Type: application/json

Status Codes:

200 - if table was successfully deleted

400 - if parameters have invalid format

404 - if table does not exist

500 - for other database errors

Response body (for successful completion):

JSON object, “result” property is a table object with keys:

name: database name

uri: URL for database operations

Request/response example:
DELETE /dbs/newDB/tables/newTable HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newTable",
    "uri": "/dbs/newDB/tables/newTable"
  }
}

`GET /dbs/<dbName>/tables/<tblName>/schema`

Return result of SHOW CREATE TABLE statement for given table.
Parameters:

dbName: database name

tblName: table name

Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

400 - if parameters have invalid format

404 - if table does not exist

500 - for database errors

Response body (for successful completion):

JSON object, “result” property is a string with resulting schema.

name: table name

uri: URL for database operations

Request/response example:
GET /dbs/newDB/tables/newTable/schema HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": "CREATE TABLE `newTable` (\n  `I` int(11) DEFAULT NULL\n) ENGINE=MyISAM DEFAULT CHARSET=latin1"
}

`GET /dbs/<dbName>/tables/<tblName>/columns`

Return result of SHOW COLUMNS statement for given table.
Parameters:

dbName: database name

tblName: table name

Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

400 - if parameters have invalid format

404 - if table does not exist

500 - for database errors

Response body (for successful completion):
JSON object, “results” property is a list of column objects with keys: name, type, key, default, null

Request/response example:
GET /dbs/newDB/tables/newTable/columns HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "default": null,
      "key": "",
      "name": "I",
      "null": "YES",
      "type": "int(11)"
    }
  ]
}

`GET /dbs/<dbName>/tables/<tblName>/chunks`

Return the list of chunks in a table. For non-chunked table empty list is returned.
Parameters:

dbName: database name

tblName: table name

Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

400 - if parameters have invalid format

404 - if table does not exist

500 - for database errors

Response body (for successful completion):

JSON object, “results” property is a list of chunk objects with keys:

chunkId: chunk number (integer)

chunkTable: true if chunk has regular chunk table (boolean)

overlapTable: true if chunk has overlap table (boolean)

uri: URL for chunk operations

Request/response example:
GET /dbs/qservTest_case01_qserv/tables/Object/chunks HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "chunkId": 7648,
      "chunkTable": true,
      "overlapTable": true,
      "uri": "/dbs/qservTest_case01_qserv/tables/Object/chunks/7648"
    },
    ...
  ]
}

`POST /dbs/<dbName>/tables/<tblName>/chunks`

Create new chunk.
Parameters:

dbName: database name

tblName: table name

Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

chunkId: chunk ID, non-negative integer

overlapFlag: if true then create overlap table too (default is true), accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’

Response headers:

Content-Type: application/json

Status Codes:

201 - if chunk tables were successfully created

400 - if parameters have invalid format or if form parameters are missing or conflicting

404 - if table is missing

409 - if chunk table already exists

500 - if table is not defined in CSS or other database errors

Response body (for successful completion):

JSON object, “result” property is a chunk object with keys:

chunkId: chunk number (integer)

chunkTable: true if chunk has regular chunk table (boolean)

overlapTable: true if chunk has overlap table (boolean)

uri: URL for chunk operations

Request/response example:
POST /dbs/newDB/tables/newTable/chunks HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------df029da2ec8387ce

--------------------------df029da2ec8387ce
Content-Disposition: form-data; name="chunkId"

1000
--------------------------df029da2ec8387ce--
HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "chunkId": 1000,
    "chunkTable": true,
    "overlapTable": true,
    "uri": "/dbs/newDB/tables/newTable/chunks/1000"
  }
}

`DELETE /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>`

Delete chunk from a table, both chunk data and overlap data is dropped.
Parameters:

dbName: database name

tblName: table name

chunkId: chunk number, non-negative integer

Response headers:

Content-Type: application/json

Status Codes:

200 - if table was successfully deleted

400 - if parameters have invalid format

404 - if table does not exist

500 - for other database errors

Response body (for successful completion):

JSON object, “result” property is a chunk object with keys:

chunkId: chunk number (integer)

chunkTable: true if chunk has regular chunk table (boolean)

overlapTable: true if chunk has overlap table (boolean)

uri: URL for chunk operations

Request/response example:
DELETE /dbs/newDB/tables/newTable/chunks/1000 HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "chunkId": 1000,
    "chunkTable": true,
    "overlapTable": true,
    "uri": "/dbs/newDB/tables/newTable/chunks/1000"
  }
}

`POST /<dbName>/tables/<tblName>/data`

Upload data into a table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.
Parameters:

dbName: database name

tblName: table name

Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

table-data: the data come in original LOAD DATA format with binary/octet-stream content type and binary encoding, and it may be compressed with gzip.

load-options: set of options encoded with usual application/x-www-form-urlencoded content type, options are: - delimiter - defaults to TAB - enclose - defaults to empty string (strings are not enclosed) - escape - defaults to backslash - terminate - defaults to newline - compressed - “0” or “1”, by default is guessed from file extension (.gz)

Response headers:

Content-Type: application/json

Status Codes:

201 - if chunk tables were successfully created

400 - if parameters have invalid format or if form parameters are missing or conflicting

404 - if table is missing

409 - if chunk table already exists

500 - if table is not defined in CSS or other database errors

Response body (for successful completion):

JSON object, “result” property is an object with keys:

status: string “OK”

count: count of rows added to a table

Request/response example:
POST /dbs/newDB/tables/newTable/data HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------345ad77805210ac6

--------------------------345ad77805210ac6
Content-Disposition: form-data; name="table-data"; filename="table.dat.gz"
Content-Type: application/octet-stream

.....<.U..table.dat.3.2400.2.bS..;.......

--------------------------345ad77805210ac6
Content-Disposition: form-data; name="load-options"

compressed=1&delimiter=%2C
--------------------------345ad77805210ac6--
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "count": 4,
    "status": "OK"
  }
}

`POST /<dbName>/tables/<tblName>/chunks/<chunkId>/data`

Upload data into a chunk table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.

This method works exactly as previous one taking the same form parameter but it loads data into a chunk and has additional URL parameter specifying chunk number.

`POST /<dbName>/tables/<tblName>/chunks/<chunkId>/overlap`

Upload data into overlap table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.

This method works exactly as previous one taking the same form parameter but it loads data into an overlap table and has additional URL parameter specifying chunk number.

`GET /dbs/<dbName>/tables/<tblName>/index`

Return index data (array of (objectId, chunkId, subChunkId) triplets).
Parameters:

dbName: database name

tblName: table name

chunkId: chunk number (non-negative integer)

Query Parameters:

columns: specifies comma-separated list of three column names. Default column names are “objectId”, “chunkId”, “subChunkId”. Result returns columns in the same order as they are specified in ‘columns’ argument.

Response headers:

Content-Type: application/json

Status Codes:

200 - for successful return

400 - if parameters have invalid format

404 - if table does not exist

500 - for other database errors

Response body (for successful completion):

JSON object, “result” property is an object with keys:

description: array of three objects describing columns, each with keys “name” (column name) and “type” (MySQL type name)

rows: array of arrays of integers

Request/response example:
GET /dbs/qservTest_case01_qserv/tables/Object/index HTTP/1.1
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "description": [
      {
        "name": "objectId",
        "type": "LONGLONG"
      },
      {
        "name": "chunkId",
        "type": "LONG"
      },
      {
        "name": "subChunkId",
        "type": "LONG"
      }
    ],
    "rows": [
      [
        386937898687249,
        6630,
        897
      ],
      [
        386942193651348,
        6630,
        660
      ],
      ...
    ]
  }
}

`GET /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>/index`

Return index data (array of (objectId, chunkId, subChunkId) triplets) for single chunk.

Does the same as previous method but for one chunk from partitioned table. Useful when index for whole table may be too big.

Request/response example:

GET /dbs/qservTest_case01_qserv/tables/Object/chunks/7648/index HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "description": [
      {
        "name": "objectId",
        "type": "LONGLONG"
      },
      {
        "name": "chunkId",
        "type": "LONG"
      },
      {
        "name": "subChunkId",
        "type": "LONG"
      }
    ],
    "rows": [
      [
        433306365599363,
        7648,
        5
      ],
      [
        433314955527561,
        7648,
        10
      ],
      ...
    ]
  }
}

Xrootd API

This section contains description of actions related to xrootd operations - e.g. publishing database via xrootd.

`GET /xrootd/dbs`

Return the list of databases known to xrootd.
Response headers:

Content-Type: application/json

Status Codes:

200 - for success

500 - for other database errors

Response body (for successful completion):

JSON object, “results” property is a list of database objects with keys:

name: database name

uri: URL for xrootd database operations

Request/response example:
GET /xrootd/dbs HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "qservTest_case01_qserv",
      "uri": "/xrootd/dbs/qservTest_case01_qserv"
    }
  ]
}

`POST /xrootd/dbs`

Register new database in xrootd chunk inventory.
Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

db: database name (required)

xrootdRestart: if set to ‘no’ then do not restart xrootd (defaults to yes)

Response headers:

Content-Type: application/json

Status Codes:

201 - if database was successfully registered

400 - if parameters are missing or have invalid format

409 - if database is already registered

500 - on other database errors

Response body (for successful completion):

JSON object, “results” property is a database object with keys:

name: database name

uri: URL for xrootd database operations

Request/response example:
POST /xrootd/dbs HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------370e6e4d60b7499e

--------------------------370e6e4d60b7499e
Content-Disposition: form-data; name="db"

newDB
--------------------------370e6e4d60b7499e
Content-Disposition: form-data; name="xrootdRestart"

no
--------------------------370e6e4d60b7499e--
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/xrootd/dbs/newDB"
  }
}

`DELETE /xrootd/dbs/<dbName>`

Unregister database from xrootd chunk inventory.
Parameters:

dbName: database name

Query Parameters:

xrootdRestart: if set to ‘no’ then do not restart xrootd (defaults to yes)

Response headers:

Content-Type: application/json

Status Codes:

200 - for success

400 - if parameters have invalid format

409 - if database is not registered

500 - for other database errors

Response body (for successful completion):

JSON object, “results” property is a database object with keys:

name: database name

uri: URL for xrootd database operations

Request/response example:
DELETE /xrootd/dbs/newDB?xrootdRestart=no HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/xrootd/dbs/newDB"
  }
}

`GET /xrootd/dbs/<dbName>`

Return the list of chunk IDs in a database as known to xrootd.

Note

Not implemented yet.

Services API

This section contains description of actions related to operations on services - e.g. stopping and starting processes.

`GET /services`

Return the list of services.
Response headers:

Content-Type: application/json

Status Codes:

200 - for success

Response body (for successful completion):

JSON object, “results” property is a list of service objects with keys:

name: service name

uri: URL for service operations

Request/response example:
GET /services HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "xrootd",
      "uri": "/services/xrootd"
    },
    {
      "name": "mysqld",
      "uri": "/services/mysqld"
    }
  ]
}

`GET /services/<service>`

Return service state.
Parameters:

service: service name

Response headers:

Content-Type: application/json

Status Codes:

200 - for success

404 - for invalid service name

Response body (for successful completion):

JSON object, “result” property is a service object with keys:

name: service name

state: one of “active” or “stopped”

uri: URL for service operations

Request/response example:
GET /services/mysqld HTTP/1.0
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "mysqld",
    "state": "active",
    "uri": "/services/mysqld"
  }
}

`PUT /services/<service>`

Execute some action on service, like “stop” or “restart”.
Parameters:

service: service name

Request headers:

Content-Type: required as multipart/form-data

Form Parameters:

action: action: one of ‘stop’, ‘start’, ‘restart’ (required)

Response headers:

Content-Type: application/json

Status Codes:

200 - for success

400 - if parameters are missing or have invalid format

409 - if action has failed

Response body (for successful completion):

JSON object, “result” property is a service object with keys:

name: service name

state: state of service after action, one of “active” or “stopped”

uri: URL for service operations

Request/response example:
PUT /services/mysqld HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------48169e483bc7d12e

--------------------------48169e483bc7d12e
Content-Disposition: form-data; name="action"

restart
--------------------------48169e483bc7d12e--
HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "mysqld",
    "state": "active",
    "uri": "/services/mysqld"
  }
}

Worker Manager API

Overview

General API description

Return codes and content types

Database API

GET /dbs

POST /dbs

DELETE /dbs/<dbName>

GET /dbs/<dbName>/tables

POST /dbs/<dbName>/tables

DELETE /dbs/<dbName>/tables/<tblName>

GET /dbs/<dbName>/tables/<tblName>/schema

GET /dbs/<dbName>/tables/<tblName>/columns

GET /dbs/<dbName>/tables/<tblName>/chunks

POST /dbs/<dbName>/tables/<tblName>/chunks

DELETE /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>

POST /<dbName>/tables/<tblName>/data

POST /<dbName>/tables/<tblName>/chunks/<chunkId>/data

POST /<dbName>/tables/<tblName>/chunks/<chunkId>/overlap

GET /dbs/<dbName>/tables/<tblName>/index

GET /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>/index