Data in the Cloud: Ebbs and Flows
Part 1: Understanding transfer speeds and avoiding vendor lock-in
In the last few posts we spoke about a couple of the requests we had from our customers to migrate data from one cloud storage provider (CSP) to another and how we helped them accomplish those goals. After the initial concerns about maintaining security throughout the entire process (not an issue with Nasuni), the second question they asked is how fast their data could be moved from one CSP to another.
Underneath the questions regarding cloud transfer speed lies a deep anxiety about vendor lock-in. It is possible for these performance issues to go undetected for years as data trickles in small daily increments. However, if a customer wants to take their business elsewhere they may be shocked to discover that their provider cannot support an all out migration. At Nasuni, we constantly stress test the capabilities of selected backend providers. We know customer data is always growing and we never want their data getting stuck because the pipes fail to scale.
So, what how fast is fast enough when it comes to bulk migration of data in the cloud?
The answer, like many answers related to storage performance, is that it depends. This kind of bulk movement depends on many factors:
- The performance of the CSP you’re moving data out of (CSP read performance)
- The performance of the CSP you’re moving data into (CSP write performance)
- The raw speed of the network between the CSPs and the machines you’re using to move the data (CSPs cant move data to each other directly)
- The load on the network between the CSPs and the migration machines at the time of the migration
- The number of machines being used to do the migration
- The specifications of the machines being used to do the migration (memory, CPU speed, disk speed, etc)
- Quality and speed of the tools being used to do the migration
“It depends” isn't a popular answer, so we often used actual performance numbers from prior migrations to frame our answers, but the question begged for some more extensive testing and more data so we set out to get some numbers.
In part 2 of this article we will look at different use cases and benchmark data transfer speeds for the top public cloud storage providers.