Skip to main content
Version: 0.28

Apache Airflow

Apache Airflow is a powerful open-source platform for developing, scheduling, and monitoring batch-oriented workflows. FastBCP can be seamlessly integrated into Airflow DAGs using the Docker operator and its official Docker image.

Docker Image​

FastBCP provides an official Docker image available on Docker Hub:

Image: arpeio/fastbcp:latest
Repository: https://hub.docker.com/repository/docker/arpeio/fastbcp/general

Prerequisites​

Install Docker Provider​

To use FastBCP with Airflow, you need to install the Docker provider package:

pip install apache-airflow-providers-docker

Or add it to your Airflow requirements:

apache-airflow-providers-docker

Configure Airflow Connections​

Airflow connections allow you to securely store and manage credentials for external systems. FastBCP integration requires connections for your source database and cloud storage.

A) Source Database Connection (MSSQL Example)​

Create an MSSQL connection for your source database:

  1. Navigate to Admin → Connections in the Airflow UI
  2. Click + to create a new connection
  3. In the Connection Type dropdown, select ODBC
  4. Configure the connection:

Connection Settings:

  • Connection Id: mssql_tpch10 (or your preferred name)
  • Connection Type: ODBC (select this from the dropdown)
  • Host: 192.168.65.254 (your database host)
  • Port: 11433 (your database port)
  • Login: Your database username (e.g., FastLogin)
  • Password: Your database password
  • Extra: {"database":"tpch10"}
tip

For ODBC connections, the database name should be specified in the Extra JSON field using {"database":"your_database_name"}.

B) AWS Connection (for S3 Export)​

Create an AWS connection to pass cloud credentials to the container:

  1. Navigate to Admin → Connections in the Airflow UI
  2. Click + to create a new connection
  3. In the Connection Type dropdown, select Amazon Web Services
  4. Configure the connection:

Connection Settings:

  • Connection Id: aws_fastbcp (or your preferred name)
  • Connection Type: Amazon Web Services (select this from the dropdown)
  • AWS Access Key ID: Your AWS access key (in the Login field)
  • AWS Secret Access Key: Your AWS secret key (in the Password field)
  • Extra: {"region_name":"eu-west-1"} (or your preferred region)
tip

Airflow AWS providers commonly use region_name in Extra JSON. Alternatively, you can use {"region":"eu-west-1"} or {"aws_default_region":"eu-west-1"}.

Integration with DockerOperator​

The DockerOperator from the Docker provider allows you to execute FastBCP commands in a Docker container directly from your Airflow DAGs.

Complete DAG Example​

Create a DAG file (e.g., fastbcp_orders_mssql_to_s3.py) in your Airflow DAGs folder:

from datetime import datetime

from airflow import DAG
from airflow.sdk.bases.hook import BaseHook
from airflow.providers.docker.operators.docker import DockerOperator


def _mssql_parts(conn_id: str):
    """
    Extract MSSQL connection details from Airflow connection.
    
    Expected Airflow connection (using ODBC):
      - Conn Id: mssql_tpch10 (conn_type: odbc)
      - Host: 192.168.65.254
      - Port: 11433
      - Login/Password: FastLogin / FastPassword
      - Extra JSON: {"database":"tpch10"}
    """
    c = BaseHook.get_connection(conn_id)
    host = c.host
    port = c.port
    user = c.login
    pwd = c.password

    # For ODBC MSSQL, the database is typically stored in Extra JSON
    # (because the UI may not expose a dedicated "Database" field).
    db = c.extra_dejson.get("database") or c.schema

    if not db:
        raise ValueError(
            f"Connection '{conn_id}': missing database. Put it in Extra JSON "
            f'{{"database":"tpch10"}} (recommended for ODBC).'
        )

    return host, port, user, pwd, db


def _aws_env(conn_id: str):
    """
    Extract AWS credentials from Airflow connection and map to container env vars.
    
    Expected Airflow AWS connection:
      - Conn Id: aws_fastbcp
      - Conn Type: Amazon Web Services
      - Login    -> AWS_ACCESS_KEY_ID
      - Password -> AWS_SECRET_ACCESS_KEY
      - Extra    -> {"region_name":"eu-west-1"} (or your region)

    Returns a dictionary with:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_DEFAULT_REGION
    """
    c = BaseHook.get_connection(conn_id)

    # Keys can be in login/password or in Extra JSON
    access_key = c.login or c.extra_dejson.get("aws_access_key_id")
    secret_key = c.password or c.extra_dejson.get("aws_secret_access_key")

    # Airflow AWS providers commonly use "region_name" in Extra JSON
    region = (
        c.extra_dejson.get("region_name")
        or c.extra_dejson.get("region")
        or c.extra_dejson.get("aws_default_region")
    )

    missing = [
        k
        for k, v in {
            "AWS_ACCESS_KEY_ID": access_key,
            "AWS_SECRET_ACCESS_KEY": secret_key,
            "AWS_DEFAULT_REGION": region,
        }.items()
        if not v
    ]

    if missing:
        raise ValueError(
            f"Connection '{conn_id}': missing AWS fields: {missing}. "
            f"Set login/password and add Extra.region_name (or region)."
        )

    return {
        "AWS_ACCESS_KEY_ID": access_key,
        "AWS_SECRET_ACCESS_KEY": secret_key,
        "AWS_DEFAULT_REGION": region,
    }


with DAG(
    dag_id="fastbcp_export_orders_parquet_to_s3",
    start_date=datetime(2025, 1, 1),
    schedule=None,
    catchup=False,
    tags=["fastbcp", "mssql", "s3", "docker"],
) as dag:

    mssql_host, mssql_port, mssql_user, mssql_pwd, mssql_db = _mssql_parts("mssql_tpch10")
    aws_env = _aws_env("aws_fastbcp")

    DockerOperator(
        task_id="export_orders_parquet_to_s3",
        image="arpeio/fastbcp:latest",
        docker_url="tcp://host.docker.internal:2375",
        api_version="auto",
        auto_remove="success",
        mount_tmp_dir=False,  # Avoids the remote-engine tmp mount warning
        do_xcom_push=False,
        environment=aws_env,
        command=[
            "--connectiontype", "mssql",
            "--server", f"{mssql_host},{mssql_port}",
            "--user", mssql_user,
            "--password", mssql_pwd,
            "--database", mssql_db,
            "--sourceschema", "dbo",
            "--sourcetable", "orders",
            "--fileoutput", "orders.parquet",
            "--directory", "s3://fastbcp-export/orders/parquet",
            "--parallelmethod", "Ntile",
            "--paralleldegree", "12",
            "--distributekeycolumn", "o_orderkey",
            "--merge", "false",
            "--nobanner",
        ],
        network_mode="bridge",
    )

Key Parameters:

  • image: Specifies the FastBCP Docker image
  • docker_url: Docker daemon endpoint
  • environment: Dictionary of environment variables (AWS credentials in this case)
  • command: List of FastBCP CLI arguments
  • auto_remove: Automatically remove the container after successful execution

Helper Functions Explained​

The DAG includes two utility functions:

_mssql_parts(conn_id)​

Extracts connection details from an Airflow ODBC connection. This function:

  • Retrieves host, port, username, and password
  • Extracts the database name from the Extra JSON field
  • Returns all components needed to build the FastBCP connection string

_aws_env(conn_id)​

Extracts AWS credentials from an Airflow AWS connection and maps them to the environment variables expected by FastBCP:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

This function supports multiple Extra JSON formats for flexibility.

Logging​

All FastBCP execution logs are captured by Airflow and can be viewed in the Airflow UI:

  1. Navigate to DAGs in the Airflow UI
  2. Click on your DAG (e.g., fastbcp_export_orders_parquet_to_s3)
  3. Click on the task execution (green/red square in the Grid view)
  4. Click the Logs tab

Example Log Output​

The logs provide detailed information about:

  • FastBCP version and configuration
  • Source and target connection details (with masked passwords)
  • Data transfer progress
  • Performance metrics (rows/second, total time)
  • Exit status
[2026-03-02 16:15:37] INFO - Settings file not found in the working directory nor in the executable directory
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.882 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- The "FastBCP_Settings.json" file does not exist. Using default settings. Console Only with loglevel=Information
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.892 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- FastBCP Trial – will expires on "2026‑03‑27" (24 day(s) left).
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.892 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Starting
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- FastBCP Version : "0.29.2.0" Architecture : "X64" - Framework : ".NET 8.0.24"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Internal Trace ID : "5ccea857-1538-4a28-8b1c-cfe5c51fbd43"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Internal Run ID : "5ccea857-1538-4a28-8b1c-cfe5c51fbd43"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- OS : "Debian GNU/Linux 13 (trixie)"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Connection Type : "mssql"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Source SqlInstance : "sql22,1433"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Source Database : "tpch"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Source Schema : "tpch10"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Source Table : "orders"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Trusted Connection : False
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- User : "migadmin"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.893 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Output File : "orders.parquet"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Output Directory : /data/parquet
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Degree : 12
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Distribute Method : "Ntile"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Distribute Column : "o_orderkey"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Decimal Separator : ","
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Delimiter : "|"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Use quotes : False
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Date format: "yyyy-MM-dd"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Encoding : "UTF-8"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- No Header : False
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Timestamped : False
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Boolean Format : "automatic"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Cloud Profile : ""
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.894 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Application Intent : "ReadOnly"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.906 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- file extension is ".parquet"
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.909 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Encoding used : Unicode (UTF-8) - 65001 - utf-8
[2026-03-02 16:15:37] INFO - 2026-03-02T15:15:37.965 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Connection String : "Data Source=sql22,1433;Initial Catalog=tpch;User ID=migadmin;Password=xxxxx;Connect Timeout=120;Encrypt=True;Trust Server Certificate=True;Application Name=FastBCP;Application Intent=ReadOnly"
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.086 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Source Database Version : Microsoft SQL Server 2022 (RTM-CU2) (KB5023127) - 16.0.4015.1 (X64)  	Feb 27 2023 15:40:01  	Copyright (C) 2022 Microsoft Corporation 	Developer Edition (64-bit) on Linux (Ubuntu 20.04.5 LTS) <X64>
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.110 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Get DataTypes with header end : Elapsed=6 ms
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Data Segments Computation Completed in 260 ms
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Distribute Column Type : "Int32"
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 0  : 1 : 1 - 5000001 or "o_orderkey" is null
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 1  : 2 : 5000002 - 10000002
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 2  : 3 : 10000003 - 15000003
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 3  : 4 : 15000004 - 20000004
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 4  : 5 : 20000005 - 25000005
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 5  : 6 : 25000006 - 30000006
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 6  : 7 : 30000007 - 35000007
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 7  : 8 : 35000032 - 40000032
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 8  : 9 : 40000033 - 45000033
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 9  : 10 : 45000034 - 50000034
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 10  : 11 : 50000035 - 55000035
[2026-03-02 16:15:38] INFO - 2026-03-02T15:15:38.372 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Ntile Block 11  : 12 : 55000036 - 60000000
[2026-03-02 16:15:46] INFO - 2026-03-02T15:15:46.611 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 002 for o_orderkey between 10000003 and 15000003 into /data/parquet/orders_chunk_002.parquet : 1250001 rows x 9 columns in 8234ms
[2026-03-02 16:15:46] INFO - 2026-03-02T15:15:46.611 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 011 for o_orderkey between 55000036 and 60000000 into /data/parquet/orders_chunk_011.parquet : 1249989 rows x 9 columns in 8234ms
[2026-03-02 16:15:46] INFO - 2026-03-02T15:15:46.790 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 003 for o_orderkey between 15000004 and 20000004 into /data/parquet/orders_chunk_003.parquet : 1250001 rows x 9 columns in 8413ms
[2026-03-02 16:15:46] INFO - 2026-03-02T15:15:46.797 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 005 for o_orderkey between 25000006 and 30000006 into /data/parquet/orders_chunk_005.parquet : 1250001 rows x 9 columns in 8420ms
[2026-03-02 16:15:46] INFO - 2026-03-02T15:15:46.936 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 008 for o_orderkey between 40000033 and 45000033 into /data/parquet/orders_chunk_008.parquet : 1250001 rows x 9 columns in 8559ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.139 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 010 for o_orderkey between 50000035 and 55000035 into /data/parquet/orders_chunk_010.parquet : 1250001 rows x 9 columns in 8763ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.154 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 000 for o_orderkey between 1 and 5000001 into /data/parquet/orders_chunk_000.parquet : 1250001 rows x 9 columns in 8777ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.247 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 009 for o_orderkey between 45000034 and 50000034 into /data/parquet/orders_chunk_009.parquet : 1250001 rows x 9 columns in 8870ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.308 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 006 for o_orderkey between 30000007 and 35000007 into /data/parquet/orders_chunk_006.parquet : 1250001 rows x 9 columns in 8931ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.309 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 001 for o_orderkey between 5000002 and 10000002 into /data/parquet/orders_chunk_001.parquet : 1250001 rows x 9 columns in 8932ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.363 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 007 for o_orderkey between 35000032 and 40000032 into /data/parquet/orders_chunk_007.parquet : 1250001 rows x 9 columns in 8986ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.366 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Load Completed for Query 004 for o_orderkey between 20000005 and 25000005 into /data/parquet/orders_chunk_004.parquet : 1250001 rows x 9 columns in 8989ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- files generation end : Elapsed=9257 ms - maxdop=12
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total data rows : 15000000
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total data columns : 9
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total cells : 135000000
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Throughput rows : 1568531 rows/s
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Throughput cells : 14116787 cells/s
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total time : Elapsed=9563 ms
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total data rows : 15000000
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total data columns : 9
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Total cells : 135000000
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Final Throughput rows : 1568482 rows/s
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Final Throughput cells : 14116346 cells/s
[2026-03-02 16:15:47] INFO - 2026-03-02T15:15:47.368 +00:00 -|- FastBCP -|- 5ccea857-1538-4a28-8b1c-cfe5c51fbd43 -|- INFORMATION -|- orders.parquet -|- Completed Load
[2026-03-02 16:15:47] INFO - Task instance in success state
[2026-03-02 16:15:47] INFO -  Previous state of the Task instance: TaskInstanceState.RUNNING
[2026-03-02 16:15:47] INFO - Task operator:<Task(DockerOperator): export_orders_parquet_to_s3>
Copyright © 2026 Architecture & Performance.