AWS S3 Configuration

Complete guide for configuring AWS S3 connectivity with FastBCP.

Authentication Methods

FastBCP supports multiple AWS authentication methods, listed in order of precedence:

1. AWS Credentials File (recommended for local development)
2. Environment Variables (recommended for CI/CD)
3. IAM Roles (recommended for AWS infrastructure)

1. AWS Credentials File

The most common method for local development and testing.

Location:

• Windows: C:\Users\YourName\.aws\credentials
• Linux: ~/.aws/credentials

Credentials File Format:

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[production]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

[development]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

Configuration File (Optional):

Location: ~/.aws/config

[default]
region = us-east-1
output = json

[profile production]
region = eu-west-1
output = json

[profile development]
region = us-west-2
output = json

Using Profiles with FastBCP:

# Use default profile
./FastBCP \
...
 --directory "s3://my-bucket/exports"  \
 --fileoutput "data.parquet" \
...

# Use production profile
./FastBCP \
...
 --directory "s3://prod-bucket/exports" \
 --cloudprofile "production" \
 --fileoutput "data.parquet" \
...

# Use development profile
./FastBCP \
...
 --directory "s3://dev-bucket/exports" \
 --cloudprofile "development" \
 --fileoutput "data.parquet" \
...

2. Environment Variables

Useful for CI/CD pipelines and containerized environments.

Windows (PowerShell)

# Set AWS credentials
$env:AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
$env:AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
$env:AWS_DEFAULT_REGION="us-east-1"

# Optional: Session token for temporary credentials
$env:AWS_SESSION_TOKEN="FwoGZXIvYXdzEBYaD..."

# Run FastBCP
.\FastBCP.exe `
...
 --directory "s3://my-bucket/exports" `
 --fileoutput "data.parquet" `
...

Linux

# Set AWS credentials
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_DEFAULT_REGION="us-east-1"

# Optional: Session token for temporary credentials
export AWS_SESSION_TOKEN="FwoGZXIvYXdzEBYaD..."

# Run FastBCP
./FastBCP \
...
 --directory "s3://my-bucket/exports" \
 --fileoutput "data.parquet" \
...

3. IAM Roles (EC2, ECS, Lambda)

When FastBCP runs on AWS infrastructure, it can use automatically the instance's IAM role. No credentials configuration needed!

Supported AWS Services:

• Amazon EC2 instances
• Amazon ECS tasks
• AWS Lambda functions
• AWS Batch jobs
• Amazon EKS pods (with IAM roles for service accounts)

Example IAM Policy for FastBCP:

{
  "Version": "2012-10-17",
  "Statement": [
	{
	  "Sid": "FastBCPRequired",
	  "Comment": "REQUIRED - actual operations performed by FastBCP (multipart upload)",
	  "Effect": "Allow",
	  "Action": [
		"s3:PutObject",
		"s3:AbortMultipartUpload"
	  ],
	  "Resource": "arn:aws:s3:::MY-BUCKET/exports/*"
	},
	{
	  "Sid": "FastBCPOptionalConvenience",
	  "Comment": "OPTIONAL - operational convenience, not required by the code",
	  "Effect": "Allow",
	  "Action": [
		"s3:GetObject",
		"s3:ListBucket"
	  ],
	  "Resource": [
		"arn:aws:s3:::MY-BUCKET/exports/*",
		"arn:aws:s3:::MY-BUCKET"
	  ]
	}
  ]
}

S3 URI Format

s3://bucket-name/path/to/directory/

URI Examples

# Root of bucket
 --directory "s3://my-bucket"

# Single folder level
 --directory "s3://my-bucket/exports"

# Multiple folder levels
 --directory "s3://my-bucket/exports/sales/2024"

# Hive-style partitioning
 --directory "s3://my-bucket/data/year=2024/month=01/day=15"

Complete Examples

Basic Export to S3

Windows

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "SalesDB" `
  --trusted `
  --sourceschema "dbo" `
  --sourcetable "orders" `
  --directory "s3://my-bucket/exports/{sourcetable}" `
  --fileoutput "{sourcetable}.parquet"

Linux

./FastBCP \
  --connectiontype "mssql" \
  --server "localhost" \
  --database "SalesDB" \
  --trusted \
  --sourceschema "dbo" \
  --sourcetable "orders" \
  --directory "s3://my-bucket/exports/{sourcetable}" \
  --fileoutput "{sourcetable}.parquet"

Parallel Export with Profile

Windows

.\FastBCP.exe `
  --connectiontype "mssql" `
  --server "prod-sql-01" `
  --database "SalesDB" `
  --trusted `
  --query "SELECT * FROM Sales.Orders WHERE OrderDate >= '2024-01-01'" `
  --directory "s3://prod-datalake/sales/orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod "RangeId" `
  --distributekeycolumn "OrderID" `
  --paralleldegree 8 `
  --cloudprofile "production"

Linux

./FastBCP \
  --connectiontype "mssql" \
  --server "prod-sql-01" \
  --database "SalesDB" \
  --trusted \
  --query "SELECT * FROM Sales.Orders WHERE OrderDate >= '2024-01-01'" \
  --directory "s3://prod-datalake/sales/orders" \
  --fileoutput "orders.parquet" \
  --parallelmethod "RangeId" \
  --distributekeycolumn "OrderID" \
  --paralleldegree 8 \
  --cloudprofile "production"

Export with Date Partitioning

Windows

.\FastBCP.exe `
  --connectiontype "pgsql" `
  --server "localhost" `
  --port "5432" `
  --database "analytics" `
  --user "postgres" `
  --password "postgres" `
  --sourceschema "public" `
  --sourcetable "events" `
  --directory "s3://analytics-bucket/events/" `
  --fileoutput "events.parquet" `
  --parallelmethod "DataDriven" `
  --distributekeycolumn "eventmonth" `
  --paralleldegree 10 

Linux

./FastBCP \
  --connectiontype "pgsql" \
  --server "localhost" \
  --port "5432" \
  --database "analytics" \
  --user "postgres" \
  --password "postgres" \
  --sourceschema "public" \
  --sourcetable "events" \
  --directory "s3://analytics-bucket/events/" \
  --fileoutput "events.parquet" \
  --parallelmethod "DataDriven" \
  --distributekeycolumn "eventmonth" \
  --paralleldegree 10

S3-Compatible Storage

FastBCP works with S3-compatible storage services like MinIO, DigitalOcean Spaces, Wasabi, and OVH Object Storage.

Configuration for S3-Compatible Services

Windows (PowerShell)

# Set endpoint URL for S3-compatible service
$env:AWS_ACCESS_KEY_ID="your-access-key"
$env:AWS_SECRET_ACCESS_KEY="your-secret-key"
$env:S3_ENDPOINT="https://s3.your-provider.com"

# For MinIO
$env:S3_ENDPOINT="http://localhost:9000"

Linux

# Set endpoint URL for S3-compatible service
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export S3_ENDPOINT="https://s3.your-provider.com"

# For MinIO
export S3_ENDPOINT="http://localhost:9000"

Required Permissions

Minimum Permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    }
  ]
}

Recommended Permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::my-bucket",
      "Condition": {
        "StringLike": {
          "s3:prefix": "exports/*"
        }
      }
    }
  ]
}

Server-Side Encryption with AWS KMS (SSE-KMS)

FastBCP can write to buckets encrypted with SSE-KMS. Encryption is done by S3 on the server side, so FastBCP never calls AWS KMS directly: S3 calls KMS on behalf of the identity that uploads the files.

Encryption can come from:

• Bucket default encryption: the bucket is configured with SSE-KMS as its default encryption, and S3 encrypts every new object with the configured key. Nothing needs to be set on the FastBCP side.
• App-driven encryption: the upload request itself asks for SSE-KMS (and optionally a specific KMS key). The required permissions are the same.

KMS Permissions

FastBCP uploads files with S3 multipart uploads. With SSE-KMS, a multipart upload needs both kms:GenerateDataKey and kms:Decrypt on the KMS key, in addition to the S3 permissions above.

Key type	What to configure
AWS managed key (aws/s3)	Nothing more: its key policy already lets any principal of the account use it through S3. It cannot be used across accounts.
Customer managed key	Allow kms:GenerateDataKey and kms:Decrypt in the IAM policy of the identity running FastBCP and make sure the key policy allows that identity (or delegates to IAM policies).

Example IAM policy for a customer managed key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FastBCPS3Upload",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::MY-BUCKET/exports/*"
    },
    {
      "Sid": "FastBCPKmsForSSE",
      "Effect": "Allow",
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "s3.eu-west-1.amazonaws.com"
        }
      }
    }
  ]
}

S3 Bucket Keys

Enabling S3 Bucket Keys on the bucket reduces the number of calls from S3 to KMS, and therefore KMS costs and throttling, which matters for parallel exports that write many files. With Bucket Keys, the KMS encryption context is the bucket ARN instead of the object ARN: update any kms:EncryptionContext condition in your policies accordingly.

Network / Egress

Since S3 calls KMS itself, FastBCP only needs network access to the S3 endpoint, not to the KMS endpoint. In a private VPC, an S3 gateway or interface endpoint is enough.