How to copy data from one EU DataVault bucket to another EU DataVault bucket using rclone

Overview

This guide provides step-by-step instructions for copying data from one EU DataVault bucket to another one using rclone.

Since the CopyObject S3 operation between buckets is not supported, you have to configure two remotes in the rclone configuration, even if the source and target buckets belong to the same storage account.

Prerequisites

1. Source bucket permissions

Ensure that you have the following permissions on the source bucket:

• S3:ListBucket: Allows rclone to view the contents of the source bucket.
• S3:GetObject: Allows rclone to access each file for transfer.

2. Target bucket permissions

Ensure that you have the following permissions in a policy on the target bucket (EU DataVault):

• s3:PutObject, s3:PutObjectTagging, s3:PutObjectAcl: Allows rclone to upload the objects to the target bucket. See more information about IAM permissions here.

3. Target bucket creation

If you have not yet created the target bucket on EU DataVault, please follow this guide to create a bucket on EU DataVault.

Step-by-Step Guide

Step 1: Install rclone

1. Visit the rclone download page and download rclone for your operating system (Windows, Linux, macOS).
2. Extract the rclone executable, (e.g. rclone.exe on Windows), from the archive.
3. Follow the installation documentation if needed.

Step 2: Configure rclone for source and target bucket

1. Open a command prompt (Windows) or terminal (Linux) and browse to the directory with extracted rclone executable.

2. Run the command to start rclone configuration:

   rclone config
    
3. When prompted, choose 'n' to create a new remote. Name the remote (e.g., "EU DataVault Source", and hit Enter .
   
4. Configure the source: EU DataVault Source.
   • Find Amazon S3 Compliant Storage Providers including AWS3... and enter its number in the list, e.g. 4:
     
     
   • When prompted for S3 provider, select Any other S3 compatible provider   \ (Other).

• Input 1 toEnter AWS credentials in the next step 
  
• Enter the Access Key ID and Secret Access Key of  the root user or the IAM user which has access to the source bucket.
• For the region, use v4 signatures and an empty region:
  
• For the endpoint, use the proper EU DataVault endpoint URL according to your bucket's location. For instance:
  https://eu-central-2.storage.eudatavault.eu
  
• Accept the remaining default settings or adjust as needed. The configuration of the source of data copy is now complete.

1. Configure the target: EU DataVault Target.
   • Run rclone config once again.
   • When prompted, choose n to create a new remote and name it, for instance "EU DataVault Target".
   • When prompted for storage, select Amazon S3 Compliant Storage Providers....
   • When prompted for S3 provider, select Any other S3 compatible provider   \ (Other).
   • Select Enter AWS credentials in the next step 
     
   • Enter the Access Key ID and Secret Access Key of  the root user or the IAM user which have access to the target bucket.
   • For the region, use v4 signatures and an empty region:
     
   • For the endpoint, use the proper EU DataVault endpoint URL accordingly to your bucket's location. For instance:
     https://eu-central-2.storage.eudatavault.eu
     
   • Accept remaining defaults unless you need custom settings.
   • Configuration is now complete and you have two remotes: one for the data source ("EU DataVault Source" in the example below)  and another for the target ("EU DataVault Target" in the example below):
     
   • Press q and quit rclone config.

Step 3: Sync data from source to target

1. Run the following command to initiate the data sync:
   rclone sync source_rclone_config_name:source-bucket-name target_rclone_config_name:target-bucket-name

   Explanation: The sync command only copies new or changed files, allowing you to resume transfers efficiently if interrupted.
   Note: if  your rclone configuration name contains spaces, enclose it in quotes like in the example below:
   

2. Progress and Completion: rclone will display progress in the terminal. When complete, verify that all data has been copied successfully (see verification steps below).

Resuming transfers if interrupted

If the transfer is interrupted for any reason (e.g., network disconnection or system shutdown), please rerun exactly the same rclone sync command above. rclone will skip files that have already been transferred, resuming from the last incomplete point.

Verify transfer

To confirm the transfer was successful:

1. Re-run the rclone sync command and add -v by the end to view the verbose output. If you see the message "There was nothing to transfer", then all your data (only the latest version for the versioned bucket case) has been synced to the target bucket:
   
   
2. (Optional) Log in to your EU DataVault account.
   1. Navigate to the target bucket and check that the number of files matches the source bucket.
   2. Spot-check a few files to ensure they were transferred correctly.