Skip to main content
Version: 1.4 (Preview)

Parallel Parameters

FastBCP can parallelize data export to significantly improve performance. This section covers the parameters that control parallel execution.

Parallel Method​

Use the -m or --parallelmethod parameter to specify how data will be split across parallel threads.

.\FastBCP.exe `
...
 --parallelmethod Ntile `
...

Syntax:

  • Short form: -m <method>
  • Long form: --parallelmethod <method>

Methods Comparison​

MethodParallelNeeds Distributed ColumnDatabase Source Type
NoneAny
RandomAny
DataDrivenAny
RangeIdAny
NtileAny
PhysicalPostgreSQL, Oracle, SQL Server, Netezza (see Physical)
Ctid (deprecated)PostgreSQL (pgsql/pgcopy)
Physloc (deprecated)SQL Server (mssql)
Rowid (deprecated)Oracle (oraodp/adbc_oracle)
NZDataSlice (deprecated)Netezza (nzsql/adbc_netezza)
DBPartitionSQL Server, PostgreSQL, Oracle, MySQL, Teradata (and their adbc_* counterparts)
TimepartitionAny

Available Methods​

None - No parallel processing. Data is exported sequentially.

DataDriven - Uses all values of a column (or a given list provided by --datadrivenquery) to split the export. If the number of values is greater than the degree of parallelism, throttling will be applied.

tip

You can use an expression in the distribute key column instead of a column name.
Example: YEAR(o_orderdate)

Ntile - Uses the distributed column field and ntile values to retrieve evenly distributed chunks of data. Each parallel thread exports a portion based on a range built using the distributed column values. The uniqueness of the distributed column is not mandatory.

RangeId - Uses the distributed column field and its min/max values to retrieve chunks of data. Each parallel thread exports a portion based on a range built using the distributed column values.

Random - Requires a distribution column that must be an integer/bigint with many values (at least as many as the degree of parallelism).

Physical (recommended) - Splits the data on the physical row location of the source engine. The strategy (Ctid, Rowid, Physloc or NZDataSlice) is picked from the connection type. See Physical Method below.

Deprecated engine-specific methods​

The four engine-specific values below are still accepted, but they are deprecated and will be removed in FastBCP 2.0. Use Physical instead: it picks the same strategy from the connection type. Their description is kept here for reference.

Ctid (deprecated) - Uses an internal hidden field to retrieve chunks of data. Each parallel thread exports a portion based on the CTID range.

Deprecated

Ctid is deprecated and will be removed in 2.0. Use --parallelmethod Physical, which resolves to Ctid on PostgreSQL (pgsql, pgcopy, adbc_pgsql).

note

PostgreSQL databases only (and some compatible PostgreSQL databases).

Physloc (deprecated) - Uses an internal hidden field (%%physloc%%) to retrieve chunks of data. Each parallel thread exports a portion based on the Physloc value.

Deprecated

Physloc is deprecated and will be removed in 2.0. Use --parallelmethod Physical, which resolves to Physloc on SQL Server (mssql, msoledbsql, adbc_mssql).

note

Physloc parallel method is for SQL Server databases only. Since FastBCP 1.4, it requires --sourceschema and --sourcetable (directly or through Physical).

  • On a rowstore table, the rows are distributed on the two low bytes of the page id, so runs of 256 pages go to each chunk.
  • On a clustered columnstore table, %%physloc%% is a row id, not a page address. FastBCP detects the clustered columnstore index and distributes the rows on the whole row id instead (logged as Physloc : <schema>.<table> is a clustered columnstore, splitting on the columnstore row id). If the check fails, FastBCP logs a warning and uses the rowstore key.

Rowid (deprecated) - Uses an internal hidden field to retrieve chunks of data. Each parallel thread exports a portion based on the ROWID range.

Deprecated

Rowid is deprecated and will be removed in 2.0. Use --parallelmethod Physical, which resolves to Rowid on Oracle (oraodp, adbc_oracle).

note

RowId parallel method is for Oracle databases only (oraodp and adbc_oracle connection types). Since FastBCP 1.4, it requires --sourceschema and --sourcetable (directly or through Physical); it used to crash when they were missing.

NZDataSlice (deprecated) - Uses the native Netezza data slices to retrieve chunks of data. Each parallel thread exports a portion based on the data slice distribution. Slices go through a shared work queue: an idle thread takes the next waiting slice, which keeps all threads busy when slices are uneven. After the first failure, no new slice is started.

Deprecated

NZDataSlice is deprecated and will be removed in 2.0. Use --parallelmethod Physical, which resolves to NZDataSlice on Netezza (nzsql, nzcopy, nzoledb, adbc_netezza).

note

NZDataSlice parallel method is for Netezza databases only (nzsql and adbc_netezza connection types).

When a deprecated value is used, FastBCP runs the export as before and logs a warning:

Parallel method Ctid is deprecated and will be removed in 2.0 ==> use --parallelmethod Physical

DBPartition - Runs one extraction per native partition / subpartition of the source table. The largest partitions are scheduled first, based on the catalog row estimates.

SourcePartitioning used
SQL Server (mssql, adbc_mssql)$PARTITION
PostgreSQL (pgsql, pgcopy, adbc_pgsql)Leaf tables
Oracle (oraodp, adbc_oracle)PARTITION (...)
MySQL (mysql)PARTITION (...)
Teradata (teradata, adbc_teradata)Row partitioning, single- and multi-level PPI
note

DBPartition requires --sourceschema and --sourcetable; it does not need a distribute key column. It falls back to None, with a warning, when the table is not partitioned, has a single partition, or the source is not supported.

tip

The output file layout is controlled by the FASTBCP_DBPARTITION_TARGET_MODE environment variable: numbered chunks (CHUNK, default) or Hive-style directories named after each partition (HIVE). --merge works in both modes.

Timepartition - Uses a date/datetime column to partition data into time-based chunks. Each parallel thread exports data for a specific time period (year, month, week, or day). The distributed key column must be specified in a special format: (datecolumn, year, month), (datecolumn, year, month, day), (datecolumn, year), or (datecolumn, year, week).

tip

This method is ideal for time-series data and allows efficient parallel export based on temporal ranges.

Physical Method​

--parallelmethod Physical is the recommended way to split a table on its physical row location, without a distribute key column. FastBCP picks the physical-location strategy of the source engine from the connection type (-C / --connectiontype):

SourceConnection typesResolved strategy
PostgreSQLpgsql, pgcopy, adbc_pgsqlCtid
Oracleoraodp, adbc_oracleRowid
SQL Servermssql, msoledbsql, adbc_mssqlPhysloc
Netezzanzsql, nzcopy, nzoledb, adbc_netezzaNZDataSlice

The resolved strategy behaves exactly like the deprecated engine-specific method of the same name (see Deprecated engine-specific methods).

Requirements:

  • --sourceschema and --sourcetable are required. --query can be added to filter the table (simple query with a WHERE clause). FastBCP stops with an error when one of them is missing:

    ERROR in command line parameters : With Physical (Ctid) Method you must define source schema (--sourceschema) and source table (--sourcetable) ==> With Physical (Ctid) Method --sourceschema and --sourcetable are both mandatory

  • --distributekeycolumn is not used.

Unsupported sources: any other connection type (odbc, oledb, mysql, clickhouse, hana, teradata, adbc_teradata) stops with an error:

ERROR in command line parameters : Physical method is not supported for connection type 'mysql' ==> supported sources are PostgreSQL, Oracle, SQL Server and Netezza

With odbc and oledb, FastBCP cannot know which engine is behind the driver, so the error asks for a native connection type:

ERROR in command line parameters : Physical method cannot identify the source engine behind connection type 'odbc' ==> use a native connection type (pgsql, pgcopy, oraodp, mssql, msoledbsql, nzsql, nzcopy, nzoledb or their adbc_* counterpart)

Logs: the resolved strategy is written to the log as the distribute method:

Distribute Method : Physical (Ctid)

The OpenTelemetry spans carry the resolved method in fastbcp.method. When Physical was resolved, fastbcp.method.requested holds Physical.

Physical Examples​

# PostgreSQL: resolved to Ctid
.\FastBCP.exe `
 --connectiontype "pgsql" `
 --server "localhost:5432" `
 --database "tpch" `
 --user "FastUser" `
 --password "FastPassword" `
 --sourceschema "tpch_10" `
 --sourcetable "orders" `
 --parallelmethod "Physical" `
 --paralleldegree 8 `
 --directory "D:\temp\orders" `
 --fileoutput "orders.parquet" `
 --merge "False"
# SQL Server: resolved to Physloc
.\FastBCP.exe `
 --connectiontype "mssql" `
 --server "localhost" `
 --database "tpch" `
 --trusted `
 --sourceschema "dbo" `
 --sourcetable "orders" `
 --parallelmethod "Physical" `
 --paralleldegree -2 `
 --directory "D:\temp\orders" `
 --fileoutput "orders.parquet" `
 --merge "False"

Distribute Key Column​

Use the -c or --distributekeycolumn parameter to define the column (or computation) on the data source that will be used to split the data into several parts.

FastBCP will use SQL queries that run in parallel against the source. Each query will have a WHERE clause that retrieves a part of the total data.

Information

This parameter is mandatory when using methods that require a distributed column: Random, DataDriven, RangeId, Ntile, or Timepartition. It is not used by Physical (nor by the deprecated Ctid, Rowid, Physloc and NZDataSlice) or DBPartition.

Timepartition Special Format

When using the Timepartition method, the distribute key column must be specified in a special format:

  • (datecolumn, year, month) - Partition by year and month
  • (datecolumn, year, month, day) - Partition by year, month, and day
  • (datecolumn, year) - Partition by year only
  • (datecolumn, year, week) - Partition by year and week

Example: --distributekeycolumn "(o_orderdate, year, month)"

# Using a column name
.\FastBCP.exe `
...
 --distributekeycolumn order_date `
...

# Using an expression
.\FastBCP.exe `
...
 --distributekeycolumn "YEAR(order_date)" `
...

Syntax:

  • Short form: -c <column_name>
  • Long form: --distributekeycolumn <column_name>

Degree of Parallelism​

Use the -p or --paralleldegree parameter to control how many parallel threads will be used for the export.

DOP Values​

Positive value (e.g., 4) - Uses exactly that number of parallel threads. If greater than the number of CPU cores/threads, it will be downscaled to match available cores.

0 - Automatically aligns with the number of cores (or threads if Hyper-Threading is enabled) on the machine.

Negative value (e.g., -2) - Computed as number of cores / abs(dop). For example, if you have 16 cores and set DOP to -2, the actual DOP will be 8.

# Use 8 parallel threads
.\FastBCP.exe `
...
 --paralleldegree 8 `
...

# Auto-detect based on CPU cores
.\FastBCP.exe `
...
 --paralleldegree 0 `
... 

# Use half of available cores
.\FastBCP.exe `
...
 --paralleldegree -2 `
...

Syntax:

  • Short form: -p <value>
  • Long form: --paralleldegree <value>

Default: -2

Data Driven Query​

Use the --datadrivenquery parameter when using the DataDriven method to provide a query that returns the list of values that will be used to split the data. This allows you to filter the values that will be exported and used to split the data.

.\FastBCP.exe `
...
 --datadrivenquery "SELECT tagname FROM tags" `
...
.\FastBCP.exe `
...
 --datadrivenquery "SELECT o_orderdate FROM dim_date where ref_date > getdate() - 30" `
...

Syntax:

  • Long form only: --datadrivenquery "<query>"

Merge (boolean)​

Use the -M or --merge true/false to specify if the files generated by the parallel export should be merged into a single output file (true) or kept as separate files (false).

Default false

warning

Current version allows valid merge for CSV and Parquet formats only.

warning

Merge is not available for cloud destinations: --merge true is forced to false (with a warning in the log) when --directory is a cloud URI.

warning

Merge is not supported with gzip compressed output (.csv.gz, .tsv.gz, .json.gz): FastBCP stops with an error.

# Merge files after parallel export
.\FastBCP.exe `
...
 --merge true `
...

# Keep distributed files (default)
.\FastBCP.exe `
...
 --merge false `
...

Syntax:

  • Short form: -M
  • Long form: --merge

Default: false

Complete Example​

Here's a complete example using parallel parameters with the DataDriven method:

.\FastBCP.exe `
 --connectiontype "pgcopy" `
 --server "localhost:15432" `
 --database "tpch" `
 --user "FastUser" `
 --password "FastPassword" `
 --query "with T1 AS (select *, to_char(o_orderdate, 'YYYYMM') o_ordermonth from tpch_10.orders) SELECT * FROM T1" `
 --directory "D:\temp\tpch\orders\" `
 --fileoutput "pgcopy_orders.parquet" `
 --method "DataDriven" `
 --distributekeycolumn "o_ordermonth" `
 --datadrivenquery "SELECT to_char(d, 'YYYYMM') AS month FROM generate_series(DATE '1998-01-01', DATE '1998-12-01',INTERVAL '1 month') AS d" `
 --paralleldegree 10 `
 --merge "False" `
 --runid "pgcopy_to_parquet_parallel12_DataDriven"

This example:

  • Connects to a PostgreSQL database on localhost:15432
  • Exports data from the dbo.orders table
  • Uses the DataDriven distribution method
  • Distributes work based on the month of the order date using an expression
  • Use a custom query to generate month values to extract from a light query (instead of a SELECT DISTINCT o_ordermonth from the source query)
  • Uses 10 parallel threads (even if there is 12 month to export)
  • Keeps distributed files without merging (--merge false)
  • Assigns a custom Run ID for tracking

Timepartition Example​

Here's an example using the Timepartition method to partition data based on a date column:

.\FastBCP.exe `
 --connectiontype "mssql" `
 --server "localhost" `
 --database "tpch_copy" `
 --user "FastUser" `
 --password "FastPassword" `
 --sourceschema "tpch_10" `
 --sourcetable "orders_date_sorted" `
 --directory "D:\temp\TestPartition" `
 --fileoutput "orders.csv" `
 --decimalseparator "," `
 --delimiter "|" `
 --dateformat "yyyy-MM-dd HH:mm:ss" `
 --encoding "UTF-8" `
 --method "Timepartition" `
 --distributekeycolumn "(o_orderdate,year,month)" `
 --paralleldegree 16 `
 --merge "False" `
 --runid "mssql_to_csv_parallel8_timepartition"

This example:

  • Connects to a SQL Server database on localhost
  • Exports data from the tpch_10.orders_date_sorted table
  • Uses the Timepartition distribution method
  • Partitions data by year and month based on the o_orderdate column
  • Uses 16 parallel threads
  • Keeps distributed files without merging
  • Assigns a custom Run ID for tracking

DBPartition Example​

Here's an example using the DBPartition method to export each partition of a partitioned table into a Hive-style directory:

$env:FASTBCP_DBPARTITION_TARGET_MODE = "HIVE"

.\FastBCP.exe `
 --connectiontype "pgsql" `
 --server "localhost:5432" `
 --database "tpch" `
 --user "FastUser" `
 --password "FastPassword" `
 --sourceschema "tpch_10" `
 --sourcetable "orders_partitioned" `
 --method "DBPartition" `
 --paralleldegree 8 `
 --directory "D:\temp\orders" `
 --fileoutput "orders.parquet" `
 --merge "False" `
 --runid "pgsql_to_parquet_dbpartition"

This example:

  • Connects to a PostgreSQL database on localhost:5432
  • Exports the partitioned table tpch_10.orders_partitioned, one extraction per leaf table
  • Starts with the largest partitions and runs up to 8 of them in parallel
  • Writes each partition to partition=<leaf table>/orders.parquet (HIVE mode)
  • Keeps distributed files without merging
Copyright © 2026 Architecture & Performance.