Skip to main content
Version: 1.4 (Preview)

Azure Configuration

Complete guide for configuring Azure Blob Storage and Azure Data Lake Gen2 connectivity with FastBCP.

Supported Azure Storage Services​

ServiceProtocolBest For
Azure Blob Storageabs://General-purpose object storage
Azure Data Lake Gen2abfss://Big data analytics and hierarchical namespaces

Authentication Methods​

FastBCP supports multiple Azure authentication methods:

  1. Azure CLI (recommended for local development)
  2. Connection String (simple but less secure)
  3. Storage Account Key (via environment variables)
  4. Managed Identity (recommended for Azure infrastructure)
  5. SAS URL (scoped, time-limited access)

For direct commit to Azure Blob Storage, FastBCP uses AZURE_STORAGE_CONNECTION_STRING or AZURE_STORAGE_SAS_URL when one of them is set, and falls back to DefaultAzureCredential (Azure CLI, Managed Identity, environment credentials, …) otherwise. The selected authentication method is written to the log.

1. Azure CLI​

The simplest method for local development and testing.

Setup:

# Install Azure CLI
# Windows: Download from https://aka.ms/installazurecliwindows
# Linux: curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Login to Azure
az login

# List subscriptions
az account list --output table

# Set active subscription (if you have multiple)
az account set --subscription "My Subscription Name"
# or by ID
az account set --subscription "12345678-1234-1234-1234-123456789abc"

# Verify access to storage account
az storage account show --name mystorageaccount --resource-group myresourcegroup

FastBCP will automatically use the Azure CLI's authentication credentials.

Using with FastBCP:

# No additional configuration needed - just run FastBCP
./FastBCP \
...
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/exports" \
 --fileoutput "data.parquet" \
...

2. Connection String​

Useful for CI/CD pipelines and automated workflows.

Get Connection String:

  1. Navigate to your Storage Account
  2. Go to Security + networking → Access keys
  3. Copy the Connection string for key1 or key2

Set Connection String:

$env:AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=xxxxx;EndpointSuffix=core.windows.net"

# Run FastBCP
.\FastBCP.exe `
...
 --directory "abs://mystorageaccount.blob.core.windows.net/container/exports" `
 --fileoutput "data.parquet" `
...

3. Storage Account Key​

Alternative to connection string using separate environment variables.

Get Storage Account Key:

  1. Navigate to your Storage Account
  2. Go to Security + networking → Access keys
  3. Copy key1 or key2

Set Environment Variables:

$env:AZURE_CLIENT_ID="<ClientID>"
$env:AZURE_TENANT_ID="<TenantId>"
$env:AZURE_CLIENT_SECRET="<SecretID>"


# Run FastBCP
.\FastBCP.exe `
...
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontaner/exports" `
 --fileoutput "data.parquet" `
...

4. Managed Identity​

When FastBCP runs on Azure infrastructure, it can use Managed Identity automatically. No credentials configuration needed!

Supported Azure Services:

  • Azure Virtual Machines
  • Azure App Service
  • Azure Functions
  • Azure Container Instances
  • Azure Kubernetes Service (AKS)

Enable System-Assigned Managed Identity:

  1. Navigate to your Azure resource (VM, App Service, etc.)
  2. Go to Identity
  3. Enable System assigned identity
  4. Save changes

Grant Storage Access:

# Get the Managed Identity Object ID
PRINCIPAL_ID=$(az vm show --name myVM --resource-group myResourceGroup --query identity.principalId -o tsv)

# Assign Storage Blob Data Contributor role
az role assignment create \
  --assignee $PRINCIPAL_ID \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-account}"

5. SAS URL​

Since FastBCP 1.3, you can authenticate to Azure Blob Storage with a Shared Access Signature (SAS) URL, set in the AZURE_STORAGE_SAS_URL environment variable. A SAS grants scoped, time-limited access without sharing the account key.

Generate a container SAS URL:

  1. Navigate to your Storage Account → Containers → your container
  2. Go to Shared access tokens
  3. Select the Create and Write permissions and an expiry date
  4. Click Generate SAS token and URL and copy the Blob SAS URL

Set the SAS URL:

$env:AZURE_STORAGE_SAS_URL="https://mystorageaccount.blob.core.windows.net/mycontainer?sv=...&sig=..."

# Run FastBCP
.\FastBCP.exe `
...
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/exports" `
 --fileoutput "data.parquet" `
...
tip

Quote the SAS URL: it contains & characters that the shell would otherwise interpret.

Azure Blob Storage​

URI Format​

abs://storageaccount.blob.core.windows.net/container/path/

URI Examples​

# Root of container
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/"

# Folder in container
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/exports"

# Nested folders
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/exports/sales/2024"

# Date partitioning
 --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/yearmonth=202401/"

Complete Examples​

Basic Export:

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "SalesDB" `
  --trusted `
  --sourceschema "dbo" `
  --sourcetable "Customers" `
  --directory "abs://mystorageaccount.blob.core.windows.net/mycontainer/exports/customers" `
  --fileoutput "customers.parquet"

Parallel Export with Query:

.\FastBCP.exe `
  --connectiontype "pgsql" `
  --server "localhost" `
  --port "5432" `
  --database "ecommerce" `
  --user "postgres" `
  --password "postgres" `
  --query "SELECT * FROM orders WHERE order_date >= '2024-01-01'" `
  --directory "abs://mystorageaccount.blob.core.windows.net/container/exports/orders" `
  --fileoutput "orders_2024.parquet" `
  --parallelmethod "Random" `
  --distributekeycolumn "order_id" `
  --paralleldegree 8

Azure Data Lake Gen2​

Azure Data Lake Storage Gen2 is recommended for analytics workloads due to hierarchical namespace support and better performance.

URI Format​

abfss://container@storageaccount.dfs.core.windows.net/path/
Key Differences from Blob Storage
  • Protocol: abfss:// instead of abs://
  • Endpoint: .dfs.core.windows.net instead of .blob.core.windows.net
  • Features: Hierarchical namespace, directory-level operations, better performance

URI Examples​

# Root of filesystem
 --directory "abfss://mystorageaccount.dfs.core.windows.net/datalake"

# Folder in filesystem
 --directory "abfss://mystorageaccount.dfs.core.windows.net/datalake/raw/sales"

# Hive-style partitioning
 --directory "abfss://mystorageaccount.dfs.core.windows.net/datalake/sales/yearmonth=202401"

Complete Examples​

Basic Export:

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "DataWarehouse" `
  --trusted `
  --sourceschema "dbo" `
  --sourcetable "FactSales" `
  --directory "abfss://mystorageaccount.dfs.core.windows.net/datalake/raw/{sourcetable}/" `
  --fileoutput "FactSales.parquet"

Parallel Export with Partitioning:

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "Analytics" `
  --trusted `
  --query "SELECT * FROM Events" `
  --directory "abfss://mystorageaccount.dfs.core.windows.net/mycontainer/{sourcedatabase}/events/" `
  --fileoutput "events.parquet" `
  --parallelmethod "DataDriven" `
  --distributekeycolumn "EventDate" `
  --datadrivenquery "SELECT RefDate From Calendar WHERE RefDate between '2024-01-01' and '2024-03-01'" `
  --paralleldegree 12

Required Permissions​

Azure RBAC Roles​

Recommended Role:

  • Storage Blob Data Contributor - Full read/write access to blobs

Alternative Roles:

  • Storage Blob Data Owner - Full access including ACL management
  • Storage Blob Data Reader - Read-only access (for verification)

Container Access​

FastBCP creates the target container if it does not exist. When the container already exists, a missing permission to create it (HTTP 403 or 409 on the create call) is tolerated, so an identity or SAS that can only write blobs into an existing container is enough.

Troubleshooting 403 Errors​

A 403 Forbidden from Azure Storage usually comes from one of the following:

  • Firewall: the storage account's network rules do not allow the IP address or virtual network FastBCP runs from.
  • RBAC: the identity used by DefaultAzureCredential has no data-plane role (such as Storage Blob Data Contributor) on the account or container. Management roles such as Owner or Contributor are not enough. Role assignments can take a few minutes to apply.
  • SAS permissions: the SAS token is expired, is scoped to another container, or lacks the Create / Write permissions.

FastBCP logs the authentication method it used, which helps tell these cases apart.

Assign Role via Azure Portal​

  1. Navigate to your Storage Account
  2. Go to Access Control (IAM)
  3. Click Add → Add role assignment
  4. Select Storage Blob Data Contributor
  5. Select user, group, or managed identity
  6. Click Save

Assign Role via Azure CLI​

# For a user
az role assignment create \
  --assignee user@example.com \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-account}"

# For a managed identity
az role assignment create \
  --assignee {managed-identity-object-id} \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-account}"
Copyright © 2026 Architecture & Performance.