How to copy data from a 3rd-Party S3 bucket to an EU DataVault bucket using rclone

Overview

This guide provides step-by-step instructions for copying data from a third-party S3-compatible bucket, such as AWS S3, to an EU DataVault bucket using rclone.

Prerequisites

1. Source bucket permissions

Ensure that you have the following permissions on the source bucket (e.g., AWS S3 or another S3-compatible service):

•  
  • S3:ListBucket: Allows rclone to view the contents of the source bucket.
  • S3:GetObject: Allows rclone to access each file for transfer.

2. Target bucket permissions

Ensure that you have the following permissions in a policy on the target bucket (EU DataVault):

• s3:PutObject, s3:PutObjectTagging, s3:PutObjectAcl: Allows rclone to upload the objects to the target bucket. See more information about IAM permissions here.

3. Target bucket creation

If you have not yet created the target bucket on EU DataVault, please follow this guide to create a bucket on EU DataVault.

Step-by-Step Guide

Step 1: Install rclone

1. Visit the rclone download page and download rclone for your operating system (Windows, Linux, macOS).
2. Extract the rclone executable, (e.g. rclone.exe on Windows), from the archive.
3. Follow the installation documentation if needed.

Step 2: Configure rclone for source and target storage provider

1. Open a command prompt (Windows) or terminal (Linux) and browse to the directory with extracted rclone executable.

2. Run the command to start rclone configuration:

   rclone config
    
3. When prompted, choose n to create a new remote. Name the remote (e.g., AWS for the third-party S3 bucket) and hit Enter .
   
4. Configure the source (3rd-party S3 cloud provider):
   • Find Amazon S3 Compliant Storage Providers including AWS3... and enter its number in the list, e.g. 4:
     
     
   • When asked for provider, enter the appropriate number (e.g., 1 for Amazon S3).
   • Proceed further and once asked, input the necessary Access Key ID and Secret Access Key for your third-party S3 account.
   • Input the number of the correct region (e.g., 1 for us-east-1).
   • Accept the remaining default settings or adjust as needed. The configuration of the source of the data copy source is now complete.
     
     
5. Configure the target (EU DataVault).
   • Run rclone config once again.
   • When prompted, choose 'n' to create a new remote and name it, for instance "EU DataVault Target".
   • When prompted for storage, select Amazon S3 Compliant Storage Providers....
   • When prompted for S3 provider, select Any other S3 compatible provider   \ (Other).
   • Select Enter AWS credentials in the next step .
     
   • Enter the Access Key ID and Secret Access Key of  the root user or the IAM user which has access to the target bucket.
   • For the region, use v4 signatures and an empty region:
     

1.  
   • For the endpoint, use the proper EU DataVault endpoint URL according to your bucket's location. For instance:
     https://eu-central-2.storage.eudatavault.eu
     
   • Accept remaining defaults unless you need custom settings.
   • Configuration is now complete and you have two remotes: one for the data source ("AWS" in the example below)  and another for the target ("EU DataVault Target" in the example below):
     
   • Press q and quit rclone config.

Step 3: Sync data from source to target

1. Run the following command to initiate the data sync:
   rclone sync source_rclone_config_name:source-bucket-name target_rclone_config_name:target-bucket-name
    

   Explanation: The sync command only copies new or changed files, allowing you to resume transfers efficiently if interrupted.
   Note: if  your rclone configuration name contains spaces, enclose it in quotes like in the example below:
   

2. Progress and Completion: rclone will display progress in the terminal. When complete, verify that all data has been copied successfully (see verification steps below).

Resuming transfers if interrupted

If the transfer is interrupted for any reason (e.g., network disconnection or system shutdown), please rerun exactly the same rclone sync command above. rclone will skip files that have already been transferred, resuming from the last incomplete point.

Verify transfer

To confirm the transfer was successful:

1. Re-run the rclone sync command and add -v by the end to view the verbose output. If you see the message "There was nothing to transfer", then all your data has been synced to the target bucket:
   
   
2. (Optional) Log in to your EU DataVault account.
   1. Navigate to the target bucket and check that the number of files matches the source bucket.
   2. Spot-check a few files to ensure they were transferred correctly.