Advanced Scenarios

Hint

Read the following document first:

• Main concepts (CONCEPTS)

• Global configuration options
  • The number of workers
  • Worker-specific parameters
• Character sets in contributions
  • Background
  • Global configuration
  • In the REST API
  • qserv-replica-file
• Concurrency control when processing async requests
• Ingesting tables into the published catalogs
• Transaction management
  • Planning multiple transactions
    • What is a resonable number of transactions per catalog ingest?
  • Parallel transactions
    • Ingesting chunks in parallel within a single transaction
    • Ingesting chunks in parallel within dedicated transactions
    • Scattered ingest of chunk contributions within multiple transactions
  • Aborting transactions
    • Reasons to abort
      • Communication Failures
      • Ingest System Failures
    • What happens when a transaction is aborted?
    • What to do if a transaction cannot be aborted?
• Optimizations in using the REST services
  • Batch mode for allocating chunks
• Options for making contribution requests
  • Pulling data from the Web server
  • Reading data from the local filesystem of the worker
  • Pushing data from the workflow to the service
• Databases with many director tables
• Ingesting ref-match tables
• Using MySQL warnings for the data quality control
  • The context
  • Using warnings and counters
    • Analyzing counters
    • Increasing the depth of the warnings queue
• Generated columns
  • Registering tables with generated columns
  • Preparing CSV files with generated columns
    • Translating parquet files into CSV files
    • Partitioning parquet files into CSV files