Options for making contribution requests

The API provides a variety of options for making contribution requests. The choice of the most suitable method depends on the specific needs of the client application. The following list outlines the available ingest modes:

• pull: Tell the ingest service to pull the data from the Web server:

  • Ingesting contributions by reference (WORKER)

• read: Tell the ingest service to read the data directly from the locally mounted filesystem that is accessible to the worker:

  • Ingesting contributions by reference (WORKER)

• push: Send the data over the proprietary binary protocol or http protocol to the ingest service:

  • qserv-replica-file (TOOLS)

  • Ingesting contributions by value (WORKER)

  The workflow can either read the data from the local filesystem or access the data directly from memory.

All methods support the CSV data format. Additionally, the push mode over the http protocol also supports the JSON format. For more details, refer to the documentation of the respective service.

The pull and read modes also support both synchronous and asynchronous data ingestion approaches.

The following diagrams illustrate the three modes of making contribution requests:

Pulling data from the Web server

Reading data from the local filesystem of the worker

Pushing data from the workflow to the service