Main concepts

Hint

This section of the document begins with the high-level overview of the Qserv ingest API. Please read this section carefully to learn about the main concepts of the API and a sequence of operations for ingesting catalogs into Qserv.

After completing the overview, a reader has two options for what to read next:

• Study the core concepts of the API in depth by visiting subsections:

  • Types of tables in Qserv

  • Transactions

  • Publishing databases and tables

  • etc.

• Go straight to the practical example of a simple workflow presented at:

  • Simple Workflow Example

• Overview of the ingest workflow
  • Plan the ingest
  • Prepare the input data
  • Register or un-publish a database
  • Register tables
  • Configure the Ingest service
  • Start transactions
  • Figure out locations of tables and chunks
  • Send the data to the workers
  • Monitor the progress of the ingest activities
  • Commit/abort the transactions
  • Publish the database
  • Verify the ingested data products
  • Perform the optional post-ingest data management operation on the ingested tables
• Types of tables in Qserv
• Transactions
  • Isolation and parallelism
  • Row tagging
  • Checkpointing
  • Transaction status and state
  • Contexts
  • Implementation Details
• Table contributions
  • Request atomicity and retries
  • Multiple contributions
  • Ingest methods
  • Status of the contribution requests
• Publishing databases and tables
  • Databases
  • Tables
• Database families