Skip to content

S3 Migration

There are several elements to think about before migrating an S3 bucket as it may be impossible to change important settings later.

For more details see also S3.

Basic Checklist for Bucket Migrations

Gather Source Bucket Information

Define which bucket to migrate.

Also gather the following information:

  • credentials: access and secret keys.
  • application that is using the bucket: e.g. Veeam or rclone.
  • application configuration: for example Veeam or rclone configuration (where the endpoint is defined).

Choose New Endpoint

The typical case is to use an HDD-based cluster. If more performance is needed (latency, throughput) or if the average object size is small (< 500kiB), use an SSD-based cluster instead.

If large amounts of space are needed (like hundreds of TiB), please contact us, as it may influence the decision.

Decide if Object Lock is Needed

Before you continue, clarify if object lock is needed for the new bucket.

  • If the bucket is used by a backup application, then object lock is most likely needed. Check the backup configuration and look for the relevant settings. E.g. it is called immutability in the case of Veeam. Otherwise, object lock is rarely used.
  • If the source bucket is already using object lock, then probably keep it for the new bucket. You can check the bucket-level object lock configuration.
  • Activate object lock only if needed as it requires more space and will have an impact on the performance and the bill.
  • If you plan to use object lock later: now may be a good opportunity to activate it because it cannot be added later to an existing bucket.
  • For more information about object lock see Object Lock
  • In doubt please ask.

Create New Bucket

Create the new bucket, using the Portal or some other tool.

Warning

If object lock is needed, activate it when creating the new bucket. It cannot be done later.

Copy

Depending on how the bucket is used, the copy procedure differs.

Copy for Veeam

Typical Veeam backup products include:

  • Veeam Backup And Replication (VBR)
  • Veeam Backup For Microsoft 365 (VB365)

Veeam recommends a migration procedure that usually should be followed. In short, the backup job is changed to point to the new bucket (or buckets). A full backup and subsequent incremental backups get written there. The old bucket can be connected again for restores of older data and becomes obsolete over time. Eventually, it can be deleted.

Warning

Veeam often uses some defaults that are absolutely not suitable for S3. Most importantly, the "Large blocks" setting must be used, otherwise there will be far too many small objects in the bucket.

See:

Copy for Other Backup Software

Other backup softwares like HYCU are probably handling migration similarly to Veeam. Otherwise, refer to the documentation of the backup software provider.

Copy for Other Cases

In the other cases, data typically needs to be copied manually.

Usually there is no versioning involved. Then e.g. rclone is a good choice as a copy tool. You can check the versioning status.

The copying needs some planning. Indeed, writes to the bucket need to be stopped during the copy, otherwise the data will not be consistent on the source and target buckets. Possibly a maintenance window has to be announced.

The following command may be used to copy the data from a source to a destination bucket. If large amounts of data need to be copied, the command may need to be tested and tuned. Additionally, it may be a good idea to run it in a persistent session like tmux or screen.

rclone copy \
  <source_s3_profile>:<source_bucket_name> \
  <destination_s3_profile>:<destination_bucket_name> \
  --log-file=... \
  --transfers 16 \
  --no-check-dest \
  --retries 1

Check

Check if the copying was successful.

Check in Backup Case

Use the specific backup application validation tool.

Check in Other Cases

Check the copy log.

A full comparison may also be run:

rclone check \
  <source_s3_profile>:<source_bucket_name> \
  <destination_s3_profile>:<destination_bucket_name> \
  --download \
  --log-file ...log \
  -v \
  --stats=2m \
  --stats-one-line \
  --stats-one-line-date \
  --transfers 16 \
  --retries 1 \
  --differ ...log \
  --error ...log \
  --missing-on-dst ...log \
  --missing-on-src ...log

Reconfigure Client Applications

Reconfigure all the S3 clients to point to the new bucket.

Delete Old Bucket

When the old bucket is no longer needed, it can be deleted. If there are still large amounts of data, send a ticket and they will do it, as it can be done more efficiently from the backend.

Possible Additional Steps

There may be some metadata in place that has to be migrated separately, e.g:

  • bucket policy (permissions)
  • CORS configuration
  • Life-Cycle configuration

Possible Changes

A bucket migration can be a good opportunity to change some data properties. Some examples:

  • If the bucket is currently used for backups without immutability, it may be interesting to activate it for the new backup.
  • S3 becomes inefficient with many small objects. A good target for average object size is at least 1 MiB (maybe 512 KiB). Which can be translated to a maximum of 1M objects per TiB of data.
  • S3 becomes inefficient with large buckets. You may want to migrate one source bucket to several smaller destination buckets. Or, in the backup case, reconfigure the backup job accordingly.
  • To get better performance, an SSD-based endpoint can be chosen for the destination bucket.

See:

Billing

During the transition from a bucket in SWITCHengines / Cloudian OSv2 clusters to Switch Cloud S3 clusters, there may be a period of time where the used space can up to double as the data is present in both the source and destination clusters. This is for example the case for buckets with object lock as locked objects cannot be deleted until they expire. For a limited transition time, users may ask for a credit note.

Contact

In case of questions or problems, do not hesitate to ask: cloud-support@switch.ch