FASTBCP_ADBC_BATCHCELLS

Overview

The FASTBCP_ADBC_BATCHCELLS environment variable controls the size of the Arrow record batches fetched by the ADBC connectors (adbc_mssql, adbc_pgsql), expressed in cells (rows × columns) rather than rows alone.

Because a batch's memory footprint depends on both its row count and its column count, sizing by cell count keeps memory usage predictable across tables of very different widths: a narrow table fetches more rows per batch, a wide table fewer, for a comparable memory footprint.

Syntax

Windows

$env:FASTBCP_ADBC_BATCHCELLS = 1000000

Linux

export FASTBCP_ADBC_BATCHCELLS=1000000

Configuration

Property	Value
Type	Integer (> 0)
Default	500000 (500,000 cells)
Applies to	adbc_mssql and adbc_pgsql connections (Arpe.io ADBC FastDriver)
Since	FastBCP 1.2

How Batch Size Is Derived

For a given table, the effective number of rows per Arrow record batch is computed as:

rows per batch = FASTBCP_ADBC_BATCHCELLS / number of columns

For example, with the default 500000 cells:

Columns	Rows per batch
10	50,000
50	10,000
500	1,000

When to Change It

• Increase the value to reduce the number of round trips for very wide or very narrow tables when more memory is available, which can improve throughput.
• Decrease the value to lower peak memory usage in constrained environments (containers, small VMs) or when exporting extremely wide tables.

Examples

Increase batch size for higher throughput

Windows

$env:FASTBCP_ADBC_BATCHCELLS = 2000000

fastbcp -S myserver.database.windows.net `
    -d MyDatabase `
    -T dbo.MyTable `
    -C adbc_mssql `
    --fileoutput "output.parquet"

Linux

export FASTBCP_ADBC_BATCHCELLS=2000000

fastbcp -S myserver.database.windows.net \
    -d MyDatabase \
    -T dbo.MyTable \
    -C adbc_mssql \
    --fileoutput "output.parquet"

Reduce batch size to limit memory usage

Windows

$env:FASTBCP_ADBC_BATCHCELLS = 100000

fastbcp -S pg-server:5432 `
    -d MyDatabase `
    -Q "SELECT * FROM public.orders" `
    -C adbc_pgsql `
    --fileoutput "orders.parquet"

Linux

export FASTBCP_ADBC_BATCHCELLS=100000

fastbcp -S pg-server:5432 \
    -d MyDatabase \
    -Q "SELECT * FROM public.orders" \
    -C adbc_pgsql \
    --fileoutput "orders.parquet"

Platform availability

This variable only applies to the ADBC FastDriver connectors (adbc_mssql, adbc_pgsql), available on Windows x64 and Linux x64 only. See Connection Parameters for details.