FASTBCP_CLOUD_ISBUFFERED
Overview
The FASTBCP_CLOUD_ISBUFFERED environment variable controls whether FastBCP uses buffered uploads when writing to cloud storage.
Syntax
Windows
$env:FASTBCP_CLOUD_ISBUFFERED = "True" # or "False"
Linux/macOS
export FASTBCP_CLOUD_ISBUFFERED=True # or False
Configuration
| Property | Value |
|---|---|
| Type | Boolean (True / False) |
| Default | False |
| Applies to | Azure Data Lake Storage (ADLS), OneLake, AWS S3, Google Cloud Storage |
| Since | FastBCP 1.0 |
Behavior
When False (Default)
FastBCP uses the cloud SDK's OpenWriteAsync streaming upload:
- Data is written directly to the cloud storage in chunks
- Each chunk blocks the writer until acknowledged
- Simpler but can be slower for Azure storage
When True (Recommended for Azure)
FastBCP uses buffered uploads:
- Data is first written to a local temporary file (see
FASTBCP_TMPPATH) - When the stream is closed, the file is uploaded in parallel blocks (azcopy-like pattern)
- Significantly faster for Azure Data Lake Storage and OneLake
- Requires temporary disk space