Full table of contents

• Introduction
• Administrator’s Guide
  • Run Qserv on top of Kubernetes
    • Deployment
    • Administration
      • Access to Qserv dashboard
  • Row counters optimization
    • Introduction
    • Building and deploying
    • Deleting
    • Disabling the optimization at run-time
    • Inspecting
  • Data Table Indexes
    • Introduction
      • Things to consider before creating indexes
      • Error reporting
    • Creating
    • Deleting
    • Inspecting
  • Director Index
    • Notes on the MySQL table engine configuration for the index
  • Instructins for upgrading Qserv to newer releases
    • Upgrading Qserv to MariaDB 11.4.4
• User’s Guide
  • Qserv SQL Dialect
    • Introduction
    • Selected Design Aspects
      • Partitioning and Sharding
      • Director Table
      • The Director Index
    • Extensions
      • Spatial Constraints
    • Restrictions
      • Spatial constraints should be expressed through our qserv_areaspec_* functions
      • Spatial constraints must appear at the beginning of WHERE
      • Only one spatial constraint is allowed per query
      • Arguments passed to spatial constraints functions must be simple literals
      • OR is not allowed after qserv_areaspec_* constraint
      • The director index constraint must be expressed through =, IN, or BETWEEN
      • Column(s) used in ORDER BY or GROUP BY must appear in SELECT
      • Expressions/functions in ORDER BY clauses are not allowed
      • Sub-queries are NOT supported
      • Commands that modify tables are disallowed
      • Outer joins are not supported with near-neighbor queries
      • MySQL-specific syntax is not supported
      • Repeated column names through * are not supported
      • USE INDEX() is not supported
      • Variables are not supported
    • User Defined Functions
    • Example Queries Supported
      • Counts and simple selections
        • Count the number of rows in a table
        • Find rows with a particular id
        • Select rows in a given area
        • Select rows in a given area meeting certain criteria
      • Joins
        • Join two tables
        • Find near neighbors in a given region
      • LIMIT, ORDER BY
        • Limit results, sort results
    • Known Bugs
      • Selecting by objectId can miss a row
      • WHERE objectId BETWEEN fails
    • Notes of Performance
      • Use objectId when selecting sources
  • Asynchronous Query API
    • Submitting Queries
    • Checking Status
    • Retrieving Results
    • Cancellation
  • HTTP frontend of Qserv
    • General guidelines
      • JSON
      • Error reporting
      • Protocol Versioning
        • Pull mode
        • Push mode
    • The Query Interface
      • Submitting queries
        • Synchronous interface
        • Asynchronous interface
      • Checking the status of the ongoing query
      • Requesting result sets
        • Result sets
      • Canceling queries
    • The User Table Ingest Interface
      • Naming conventions
      • Ingesting tables
        • application/json
        • multipart/form-data
        • Schema specification
        • Index specification
      • Deleting tables
      • Deleting user databases
      • Possible extensions of the table ingest service
• Ingesting catalogs
  • Introduction
  • The Ingest Workflow Developer’s Guide
    • Introduction
    • Main concepts
      • Overview of the ingest workflow
        • Plan the ingest
        • Prepare the input data
        • Register or un-publish a database
        • Register tables
        • Configure the Ingest service
        • Start transactions
        • Figure out locations of tables and chunks
        • Send the data to the workers
        • Monitor the progress of the ingest activities
        • Commit/abort the transactions
        • Publish the database
        • Verify the ingested data products
        • Perform the optional post-ingest data management operation on the ingested tables
      • Types of tables in Qserv
      • Transactions
        • Isolation and parallelism
        • Row tagging
        • Checkpointing
        • Transaction status and state
        • Contexts
        • Implementation Details
      • Table contributions
        • Request atomicity and retries
        • Multiple contributions
        • Ingest methods
        • Status of the contribution requests
      • Publishing databases and tables
        • Databases
        • Tables
      • Database families
    • Simple Workflow Example
      • Test data
      • Qserv setup
      • Register the database and tables
      • Start the transaction
      • Get locations of the workers for ingesting the regular table Filter
      • Get locations of the chunks 7310 and 7480
      • Ingest the data
        • Ingest the data using the binary protocol
        • Push the data to workers via the HTTP protocol
      • Commit the transaction
      • Publish the database
      • Test the catalog
      • Post-ingest operations
    • Advanced Scenarios
      • Global configuration options
        • The number of workers
        • Worker-specific parameters
      • Character sets in contributions
        • Background
        • Global configuration
        • In the REST API
        • qserv-replica-file
      • Concurrency control when processing async requests
      • Ingesting tables into the published catalogs
      • Transaction management
        • Planning multiple transactions
          • What is a resonable number of transactions per catalog ingest?
        • Parallel transactions
          • Ingesting chunks in parallel within a single transaction
          • Ingesting chunks in parallel within dedicated transactions
          • Scattered ingest of chunk contributions within multiple transactions
        • Aborting transactions
          • Reasons to abort
            • Communication Failures
            • Ingest System Failures
          • What happens when a transaction is aborted?
          • What to do if a transaction cannot be aborted?
      • Optimizations in using the REST services
        • Batch mode for allocating chunks
      • Options for making contribution requests
        • Pulling data from the Web server
        • Reading data from the local filesystem of the worker
        • Pushing data from the workflow to the service
      • Databases with many director tables
      • Ingesting ref-match tables
      • Using MySQL warnings for the data quality control
        • The context
        • Using warnings and counters
          • Analyzing counters
          • Increasing the depth of the warnings queue
    • Post-Ingest Data Management Tasks
    • Ingest API Reference
      • REST Services
        • General guidelines
          • Request headers
          • Error reporting when calling the services
            • Optional warnings
          • Authorization and authentication
          • Protocol Versioning
            • Pull mode
            • Push mode
          • Binary encoding of the data in JSON
          • Base versus final table names
        • Master Replication Controller
          • Configuring parameters of the ingests
            • Setting configuration parameters
            • Retrieving configuration parameters
            • Global configuration parameters of workers
              • The general information on all workers
              • Worker-specific information
          • Database and table management
            • Finding existing databases and database families
            • Registering databases
            • Registering tables
            • Publishing databases
            • Un-publishing databases to allow adding more tables
            • Deleting databases and tables
          • Transaction management
            • Status of a transaction
              • Database transactions
              • Single transaction finder
            • Start a transaction
            • Commit or abort a transaction
            • Transaction descriptor
              • Shortest form
              • With a summary of contributions
              • With detailed info on contributions
            • Transaction states
          • Table location services
            • Locate regular tables
            • Allocate/locate chunks of the partitioned tables
              • Single chunk allocation
              • Multiple chunks allocation
            • Connection parameters of the workers
          • Information services
            • Chunk disposition
            • Status of the contribution request
          • Director Index Management
            • (Re-)building the Index
          • Row counters
            • Collecting row counters and deploying them at Qserv
            • Deleting row counters for a table
            • Inspecting rows counters of a table
        • Worker Ingest Server
          • Ingesting contributions by reference
          • Ingesting contributions by value
            • JSON object
            • CSV file
          • Status of requests
            • One request
            • All requests of a transaction
          • Retrying failed contributions
          • Cancelling async requests
            • One request
            • All requests of a transaction
          • Contribution descriptor
            • MySQL warnings
            • Retries
      • The Command Line Tools
        • Error reporting in the command-line tools
        • qserv-replica-file
  • The Kubernetes-based Ingest Workflow
    • Set up ingest input data
      • Prerequisites
      • Example
      • Input data
      • Metadata
    • The version history of the Ingest Workflow
      • Version numbers
      • The summary info on the versions
        • Version 12: Add mandatory version numbers in configuration files
    • Run Qserv ingest on a Kubernetes cluster
      • Prerequisites
      • Prepare and configure Qserv ingest
        • Get the project
        • Configuration
      • Launch Qserv ingest
      • Delete an existing database
      • Run interactively a workflow step
    • How to use replication service client
      • Pre-requisites
      • Procedure
    • About integration test cases
      • Integration test datasets
      • How to add an integration test case to qserv-ingest
        • Pre-requisites
        • Procedure
  • Data preparation
    • Introduction
      • Data conversion
      • Partitioning
      • Post-processing
      • Handling the binary data
      • Restrictions for the variable-length column types
      • Staging
    • Partitioning
      • Overview
      • The Partitioning Scheme
      • Partition Identifiers
      • Overlap
      • Match Tables
      • Object movement
    • Partitioner
      • Summary
      • Requirements
      • Design
      • Usage
    • Partitioning data of the ref-match tables
      • Introduction
      • The spatial match within the given overlap radius
      • Partitioning using index maps
• Developer’s Guide
  • Quick start guide for developers
    • Bootstrap a development environment
    • Build an image for production
  • Generate documentation
  • sciSQL Developer Notes
    • Making a build container
    • Running the container
    • Building and testing sciSQL
    • Updating the HTML documentation