The Query Interface

Submitting queries

Synchronous interface

The following REST service implements the synchronous interface:

POST

/query

To specify the query to be executed, a client must include the following JSON object in the request body:

{   "query" :           <string>,
    "database" :        <string>,
    "binary_encoding" : <string>
}

Where:

querystring

The required query text in which the default database may be missing. In the latter case, a client needs to provide the name in a separate parameter.

databasestring = ""

The optional name of the default database for queries where no such name was provided explicitly.

binary_encodingstring = hex

The optional binary encoding of the binary data in the table. For further details see:

• Binary encoding of the data in JSON (REST)

A call to this service will block a client application until one of the following events occurs:

• The query is successfully processed, and its result is returned to the caller.

• The query fails, and the corresponding error is returned to the caller.

• The frontend becomes unavailable (due to a crash, restart, or networking problem, etc.), and the network connection is lost.

If the request is successfully completed, the service will return a result set in the JSON object explained in the section Result sets.

Asynchronous interface

The following REST service implements the asynchronous interface:

POST	/query-async

A client is required to pass the JSON object in the request body to specify what query needs to be executed. The object has the following schema:

{   "query" :    <string>,
    "database" : <string>
}

Where:

querystring

The required query text in which the default database may be missing. In the latter case, a client needs to provide the name in a separate parameter.

databasestring = ""

The optional name of the default database for queries where no such name was provided explicitly.

A call to this service will normally block a client application for a short period until one of the following will happen:

• The query will be successfully analyzed and queued for asynchronous processing by Qserv. In this case, a response object with the unique query identifier will be returned to a caller.

• The query will fail and the corresponding error will be returned to the caller.

• The frontend will become unavailable (due to a crash, restart, or networking problem, etc.) and a network connection will be lost.

In case of the successful completion of the request, the service will return the following JSON object:

{   "queryId" : <number>
}

The number reported in the object should be further used for making the following requests explained in the dedicated subsections below:

• checking the status of the query to see when it’s finished

• requesting a result set of the query

• or, canceling the query if needed

Checking the status of the ongoing query

This service also allows checking the status of queries submitted via the synchronous interface, provided the unique identifier of such query is known to the user:

GET	/query-async/status/:queryId

Where:

queryIdnumber

The required unique identifier of the previously submitted query.

If the query identifier is not valid, the service will report an error in the response object. For example, consider the following request:

curl -k 'https://localhost:4041/query-async/status/123?version=39' -X GET

It might result in the following response:

{ "success" : 0,
  "error" :   "failed to obtain info for queryId=123,
               ex: Czar::getQueryInfo Unknown user query, err=,
               sql=SELECT status,messageTable,resultQuery FROM QInfo WHERE queryId=123",
  "error_ext" : {},
}

If the query identifier is valid then the following object will be returned:

{   "success" : 1,
    ...
    "status" : {
        "queryId" :         <number>,
        "status" :          <string>,
        "totalChunks" :     <number>,
        "completedChunks" : <number>,
        "queryBeginEpoch" : <number>,
        "lastUpdateEpoch" : <number>
    }
}

Where the status is an object that has following attributes:

queryIdnumber

The unique identifier of the previously submitted query.

statusstring

The current status of the query can have one of the following values:

• EXECUTING - The query processing is still in progress.

• COMPLETED - The query has been completed.

• FAILED - The query failed.

• ABORTED - The query was aborted:

  • explicitly by a user using the query cancellation REST service explained in the document.

  • or, implicitly by Qserv if the intermediate result set of the query exceeds the large result limit (which is configured by the Qserv administrators).

  • or, implicitly when the query processing service was restarted due to some failure or by Qserv administrators.

totalChunksnumber

The total number of so-called “chunks” (spatial shards used for splitting the large tables in Qserv into smaller sub-tables to be distributed across worker nodes of Qserv).

completedChunksnumber

The number of chunks that have been processed by Qserv so far. The value of this parameter varies from 0 to the maximum number reported in the attribute totalChunks.

queryBeginEpochnumber

The 32-bit number representing the start time of the query expressed in seconds since the UNIX Epoch.

lastUpdateEpochnumber

The 32-bit number represents the last time when the query status was recorded/updated by the Qserv internal monitoring system. The timestamp is the number of seconds since the UNIX Epoch. The service returns a value of 0 if either of the following is true:

• the query processing didn’t start

• the requst wasn’t inspected by the monitoring system

Here is an example of the status inquiry request that succeeded:

{   "success" : 1,
    "status" : {
        "queryId" :         310554,
        "status" :          "EXECUTING",
        "totalChunks" :     1477,
        "completedChunks" : 112,
        "queryBeginEpoch" : 1708141345,
        "lastUpdateEpoch" : 1708141359
    }
}

Users can use the status service to estimate when the query will finish. Typically, client applications should wait until the query status is “COMPLETED” before fetching the result set by calling the next service explained below.

Requesting result sets

The query results are retrieved by calling the following service:

method	service	query parameters
GET	/query-async/result/:queryId	binary_encoding=<encoding>

Where:

queryIdnumber

The required unique identifier of the previously submitted query.

binary_encodingstring = hex

The optional format for encoding the binary data into JSON:

• hex - for serializing each byte into the hexadecimal format of 2 ASCII characters per each byte of the binary data, where the encoded characters will be in a range of 0 .. F. In this case, the encoded value will be packaged into the JSON string.

• b64 - for serializing bytes into a string using the Base64 algorithm with padding (to ensure 4-byte alignment).

• array - for serializing bytes into the JSON array of numbers in a range of 0 … 255.

Here is an example of the same sequence of 4-bytes encoded into the hexadecimal format:

"0A11FFD2"

The array representation of the same binary sequence would look like this:

[10,17,255,210]

Like in the case of the status inquiry request, if the query identifier is not valid then the service will report an error in the response object. Otherwise, a JSON object explained in the section Result sets will be returned.

Result sets

Both flavors of the query submission services will return the following JSON object in case of the successful completion of the queries (Note: comments // used in this example are not allowed in JSON):

{   "schema" : [

      // Col 0
      { "table" :     <string>,
        "column" :    <string>,
        "type" :      <string>,
        "is_binary" : <number>
      },

      // Col 1
      { "table" :     <string>,
        "column" :    <string>,
        "type" :      <string>,
        "is_binary" : <number>
      },

      ...

      // Col (NUM_COLUMNS-1)
      { "table" :     <string>,
        "column" :    <string>,
        "type" :      <string>,
        "is_binary" : <number>
      }
    ],

    "rows" : [

      // Col 0     Col 1         Col (NUM_COLUMNS-1)
      // --------  --------      ------------------
      [  <string>, <string>, ... <string> ],         // Result row 0
      [  <string>, <string>, ... <string> ],         // Result row 1
      ...
      [  <string>, <string>, ... <string> ]          // Result row (NUM_ROWS-1)
    ]
}

Where:

schemaarray

A collection of rows, in which each row is a dictionary representing a definition of the corresponding column of the result set:

tablestring

The name of the table the column belongs to.

columnstring

The name of the column.

typestring

The MySQL type of the column as in the MySQL statement:

CREATE TABLE ...

is_binarynumber

The flag indicating if the column type represents the binary type. A value that is not 0 indicates the binary type. The MySQL binary types are documented in the corresponding sections of the MySQL Reference Manual:

• The BINARY and VARBINARY Types

• The BLOB and TEXT Types

• Bit-Value Type - BIT

Attention: Binary values need to be processed according to a format specified in the optional attribute “binary_encoding” in:

• Processing responses of query requests submnitted via the Synchronous interface

• Requesting result sets of queries submitted via the asynchronous interface

rowsarray

A collection of the result rows, where each row is a row of strings representing values at positions of the corresponding columns (see schema attribute above).

For example, consider the following query submission request:

curl -k 'https://localhost:4041/query' -X POST-H 'Content-Type: application/json' \
     -d'{"version":39,"query":"SELECT objectId,coord_ra,coord_dec FROM dp02_dc2_catalogs.Object LIMIT 5"}'

The query could return:

{ "schema":[
    { "column" : "objectId", "table" : "", "type" : "BIGINT", "is_binary" : 0 },
    { "column" : "coord_ra", "table" : "", "type" : "DOUBLE", "is_binary" : 0 },
    { "column" : "coord_dec","table" : "", "type" : "DOUBLE", "is_binary" : 0 }],
  "rows":[
    [ "1248640588874548987", "51.5508603", "-44.5061095" ],
    [ "1248640588874548975", "51.5626104", "-44.5061529" ],
    [ "1248640588874548976", "51.5625138", "-44.5052961" ],
    [ "1248640588874548977", "51.3780995", "-44.5072101" ],
    [ "1248640588874548978", "51.374245",  "-44.5071616" ]],
  "success" :   1,
  "warning" :   "",
  "error" :     "",
  "error_ext" : {}
}

Canceling queries

Note

This service can be used for terminating queries submitted via the synchronous or asynchronous interfaces, provided the unique identifier of such query is known to a user.

The status of the query can be checked using:

DELETE

/query-async/<queryId>

Where:

queryIdnumber

The required unique identifier of the previously submitted query.

If the query identifier is not valid, the service will report an error in the response object. For example, consider the following request:

curl -k 'https://localhost:4041/query-async/123?version=39' -X DELETE

It might result in the following response:

{ "success": 0,
   "error" : "failed to obtain info for queryId=123,
              ex: Czar::getQueryInfo Unknown user query, err=,
              sql=SELECT status,messageTable,resultQuery FROM QInfo WHERE queryId=123",
}

If the query identifier is valid then the following object will be returned:

{   "success" : 1
}