By Rob Mason on January 20, 2011
We’ve built digital archives in the past, so it came as no surprise to us that one of the most frequent questions our customers ask is how to move data to the Nasuni Filer. Up until today we’ve suggested using standard tools such as Robocopy, rsync, and the tried and true drag-and-drop from a client. These methods work, but they weren’t necessarily created for migrating terabytes of data to the cloud, preserving your ACLs and permissions, and keeping a full audit trail.
Enter the Nasuni Data Migration Service, a robust data mover that works directly from the Filer. The new service simplifies bulk data migration, automatically adjusting the ingest rate to expedite data transfer to the cloud. It is available to new customers and trial users with the 2.4 Release of the Nasuni Filer and offered as a free upgrade for existing subscribers.
Here is how it works:
Customers will find a new menu item in the Filer under “shares” called “Share Migration.” This is where you’ll start.
The migration service supports multiple sources and targets for migrations. If you’re migrating from a number of sources, you can queue them up to run back to back. But let’s begin with the simplest case: a single migration. First, in the screenshot above, you’d click “Add Migration.”
Here you’ll receive a notification that you need to configure a migration source. Next, fill in a descriptive name for the migration job. Then, when you go to pick a source share, you’ll have the option of adding a source. From there you’ll enter the information for your migration source:
Your source can be a CIFS share from any file server with or without Active Directory. And yes, it could be a share from your Nasuni Filer (if, for example, you want to migrate data between two cloud providers).
After you select a source, you pick a source folder to migrate from. It can be the whole share (the root of the share) or any subfolder:
Next you choose one of your local cloud volumes on your Nasuni Filer. Note that you are not picking a share from your Nasuni Filer, but one of the volumes that is backed by the cloud storage provider you selected. Shares are for exporting the volumes or portions of the volumes to your users. Also, we recommend migrating to an unexported portion of your volume so users are not accessing/changing data as you’re migrating into that portion of your volume.
Now pick a destination folder within that volume. Here you can choose from one of your existing folders or the root, or create a new target folder. Once this step is complete, your migration is configured:
Press “Save” and your migration will start. Beware that migrations will overwrite any files/folders at the destination path.
The migration makes two passes of the file tree. The first makes sure it can access the tree and calculates how many files need to be copied. The second is the actual process of copying the files from the source to the destination. During the second pass the migration service intelligently manages the cache space such that it does not stop because the cache becomes full. The result is that you can migrate terabytes of data into a Filer with a very small cache in a single operation.
Another note: The migration service copies files. It does not move them, so your source is not modified during the migration process.
After the first pass is complete and the migration begins, the estimated time to complete will be updated. This is a very approximate guess of how long the process will take based on the number of files to migrate, how many have been migrated at each point, and how long it has taken to migrate those files. As the migration progresses, this estimate becomes more and more accurate, but it can also oscillate a bit if your file sizes vary greatly.
While the migration is underway you can add additional migrations that will run when the first one finishes. To keep things simple in this release, we’ve limited migrations to run one at a time.
In addition to the progress bar, you can also view the migration log at any time. To do so, you either click through the “log” link on the status screen or navigate to a text file within the target migration directory of your volume. There you’ll find a file named something like “migration-log-2010-12-29_11152.txt”. You can view it with Notepad, Word, or other tools, and it will look like this:
This file is in a CSV format if you need to work with it, and it contains the detailed record of the migration: the source, the target, and the status of each file and directory involved.
Eventually the migration(s) will finish:
And you’ll see new options: “remove” and “rerun.” The first, remove, forgets about the migration. Remove is appropriate for one-time migration events that you will not want to run again. Rerun is for re-running a migration. This may come in handy. If any errors happen during the first migration – if, for example, users change files on your source as you’re migrating – and we are not able to automatically recover, the log will indicate the errors. In this case, you might want to rerun the migration to try the copy again.
The second case where it may be helpful is if you’re periodically migrating data from one source to the Filer for backup/DR reasons. When you rerun a migration, the Filer intelligently migrates the data such that it will only copy files that have changed. It won’t start everything all over again. And it will also update any ACLs/permissions that have changed.
We have many plans to build upon this migration service. For instance, we’d like to offer scheduled migrations, concurrent migrations and other options. But we’d also love to hear from you about what you’d like to see in this area of the Filer in the future, so send us feedback.
We’re looking at 2011 as the year of delivering higher-level functionality in the Nasuni Filer. We want to further simplify IT management. The migration service is just the first of many advanced features that our customers have been asking for. We know you’re going to enjoy what’s coming this year!
Rob Mason has more than 20 years of operational, management and software development experience, all of it in storage. A meticulous builder and obsessive tester, with an eye for talented engineers, Rob produces rock-solid software, and, through his own example of hard work and ingenuity, inspires his teams to outdo themselves. His determination for thoroughness extends to financial and operational matters, and at Nasuni, he is a powerhouse behind the scenes, managing the company’s operations, in addition to its engineering team. As the VP of Engineering at Archivas from 2004 to acquisition, Rob oversaw all development and quality assurance. After the Hitachi acquisition, he continued in his role, as VP of HCAP Engineering, managing the integration of his team with Hitachi’s and supporting the rollout of HCAP. Before joining Archivas, he was a senior manager at storage giant EMC, where he was responsible for the API, support applications and partner development for EMC’s content-addressed storage product, Centera. In a previous stint at EMC, he was Manager and Principal Design Engineer for the elite Symmetrix Group, where he improved the speed and reliability of EMC’s flagship enterprise storage disk array. Between Centera and Symmetrix, Rob was the co-founder and VP of engineering at I/O Integrity, a storage-based startup developing a high-performance caching appliance. He has a bachelor of science from Rensselaer Polytechnic Institute and a master’s in business administration with honors from Rutgers University. Rob holds upwards of 30 patents.
Nasuni is Cloud NAS, a complete storage solution leveraging the cloud as a primary storage component built-in to a unified, high-performance storage system. See how it works in this short video.