FASTBCP_DBPARTITION_TARGET_MODE

Overview

The FASTBCP_DBPARTITION_TARGET_MODE environment variable controls how the files produced by the DBPartition parallel method are named and laid out.

DBPartition runs one extraction per native partition / subpartition of the source table. Each extraction writes its own file, either as a numbered chunk (CHUNK) or in a Hive-style directory named after the partition (HIVE).

Syntax

Windows

$env:FASTBCP_DBPARTITION_TARGET_MODE = "HIVE"

Linux

export FASTBCP_DBPARTITION_TARGET_MODE=HIVE

Configuration

Property	Value
Type	CHUNK / HIVE
Default	CHUNK
Applies to	-m DBPartition
Since	FastBCP 1.3

Modes

CHUNK (default)

Each partition is written as a numbered chunk next to the target file:

<file>_chunk_NNN.<ext>

For example, with --fileoutput "orders.parquet":

orders_chunk_001.parquet
orders_chunk_002.parquet
orders_chunk_003.parquet

HIVE

Each partition is written into a Hive-style directory named after the partition, and the subpartition when there is one:

partition=<p>/[subpartition=<sp>/]<file>.<ext>

Some sources use their own directory naming:

Source	Directory layout
SQL Server	bounds=<lower>_<upper>/<file>.<ext>
Teradata	L1=<n>/[L2=<n>/]<file>.<ext>
PostgreSQL, Oracle, MySQL	partition=<p>/[subpartition=<sp>/]<file>.<ext>

For example, an Oracle table partitioned by month, with --fileoutput "orders.parquet":

partition=ORDERS_2024_01/orders.parquet
partition=ORDERS_2024_02/orders.parquet
partition=ORDERS_2024_03/orders.parquet

tip

The HIVE layout can be read directly as a partitioned dataset by engines such as Spark, DuckDB, Trino or Athena.

--merge works in both modes.

Example

Windows

$env:FASTBCP_DBPARTITION_TARGET_MODE = "HIVE"

.\FastBCP.exe `
  --connectiontype oraodp `
  --server "dbserver:1521/ORCLPDB" `
  --user "FastUser" `
  --password "FastPassword" `
  --sourceschema "SALES" `
  --sourcetable "ORDERS" `
  --directory "D:\export\orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod DBPartition `
  --paralleldegree 8 `
  --merge false

Linux

export FASTBCP_DBPARTITION_TARGET_MODE=HIVE

./FastBCP \
  --connectiontype oraodp \
  --server "dbserver:1521/ORCLPDB" \
  --user "FastUser" \
  --password "FastPassword" \
  --sourceschema "SALES" \
  --sourcetable "ORDERS" \
  --directory "/data/export/orders" \
  --fileoutput "orders.parquet" \
  --parallelmethod DBPartition \
  --paralleldegree 8 \
  --merge false