The Challenge of Archiving Growing Amounts of File Based Content
We face a tremendous challenge – the amount of file based content continues to grow. This is fueled by higher resolutions, content that is captured directly to file based formats and the need to protect source footage, stock footage and finished projects.
LTO-5 data tape is emerging as the broadly accepted solution for archiving. This is on account of its cost and reliability benefits. However, an equally important decision is what software you should be using to archive to LTO-5.
When faced with the need to archive to LTO-5, end users look to the most commonly used IT solutions. While these solutions address the basic need to write to tape, there are a number of archiving challenges (scalability, manageability, access, search etc.) encountered in a rich media environment that are not addressed by current IT solutions.
DNA Evolution – LTO-5 Archiving Designed To Scale
DNA Evolution is a new LTO-5 archiving solution designed to help rich media environments deal with the inevitability of tremendous growth.
While you may be considering archiving a single SAN today, you will quickly be adding other SANs, workstations and locations to your archival environment. Additionally, the amount of content in your archives is also guaranteed to increase over the next few years making it difficult to find content. DNA Evolution is designed to help you start from a single archive server but grow to hundreds if needed – while providing you the ability to manage it all with one interface. DNA Evolution’s single interface also enables you to search your archives globally – across any number of servers and locations.
DNA Evolution addresses key scalability, size, searchability and access challenges as you grow, including:
- How do I archive multiple SANs, workstations to the same tape hardware?
- How does one archive systems that may be at another location?
- How does one manage separate archive servers, catalog files and tape hardware?
- How do I simplify my archiving process so an end user can perform their own archiving?
- How do I search and browse my archives across terabytes and possibly petabytes of content?
- How do I access my archived tapes?
- How do I exchange these tapes with a customer or partner who does not have the same software?
- How do I guarantee my tapes are accessible 30 years from now?
In the following sections we highlight how DNA Evolution addresses a number of the above key issues.
Shared, Network Solution
- DNA Evolution is built on a shared, networked architecture.
- In the above diagram, an archive server is any workstation connected to tape hardware and has DNA Evolution archive server software.
- In the above diagram, an archive client is any workstation with the DNA Evolution client software. An archive client can be a standalone workstation, a laptop or a SAN client and can be archived over a LAN or WAN to an archive server.
- Similar to a shared SAN, the DNA Evolution architecture turns your tape hardware into a shared resource - allowing you to archive any SAN or workstation in your organization – not just the workstation connected to your tape drives.
- Rather than attach a separate tape drive for every SAN or workstation you want to archive, DNA Evolution enables you to share tape resources and save on tape hardware costs.
Central Web Management
- The DNA Evolution controller is a central appliance that is part of every environment. It is responsible for presenting a web management interface.
- Rather than being limited to managing archiving workflows from the archive workstation, DNA Evolution enables workflow management from anywhere.
- The web interface gives you the flexibility to manage all aspects of archiving your content including the ability to create, monitor and edit your archive jobs.
- Additionally, user/group access control enables different users with varying access privileges to use the web interface.
Scalability & Global Management
- Growth happens unexpectedly. However, most current solutions are not built to help you manage growing environments and archival needs.
- With current solutions, as you need to archive more SANs or locations you can end up with multiple archive servers – where each archive server has to be managed independently – with its own interface and its own catalog files resulting in a fragmented archive.
- On the other hand, the DNA Evolution architecture is designed to help you manage growth.
- Rather than having to manage each archive server as a standalone entity, DNA Evolution allows you to add archive servers and archive clients while managing all the systems as a group.
- Not only can you manage archive servers and clients at one location but a single controller can scale to manage multiple locations – each with its own archive servers and clients.
Enables a Global Archive
The DNA Evolution centralized controller also acts as the central storage for all catalog information, metadata, tape information and proxy files. This enables a global archive of all your content, across all locations – made searchable from any location using a web browser.
On the other hand, if you add archive servers in standalone architectures, you can end up with a fragmented archive. This becomes a major challenge over time - because of no central catalog file, you have to search multiple separate systems to find content in your archive.
DNA Evolution enables workflows tailored to rich media environments. The summarized workflows below are explained in more detail over the next few pages.
DNA Evolution allows administrators to create pre-configured jobs. Once enabled, end users can follow a simple two step process for archiving content. DNA Evolution enables three archiving workflows:
- Content can be archived by directly selecting files/folders
- Or files can be archived by selecting an AAF export
- Or files can be archived by selecting an XML export
Search & Browse
DNA Evolution also makes it easy to find archived content. The central controller stores all meta-data and proxy information for all content archived across all archive servers and locations. This enables a global archive that can be searched via a web interface.
Tape Access & Restoring Content
Finally, DNA Evolution enables archived tapes to be accessed on any LTO-5 tape drive, just like a hard drive.
DNA Evolution enables you to setup continuous protection for any content you want to protect.
Disk, Tape and WAN
DNA Evolution integrates not only LTO tape archiving but also enables you to integrate disk and WAN based workflows.
File system Archiving
Step 1: Via the web interface, select workstation. Browse and pick files and folders. You can select one file or multiple folders based on how you want to group and tag your content.
Step 2: Add description, tags, run archive.
Avid AAF Archiving
Step 1: Via the web interface, select an AAF exported file. The exported file is a meta-data export.
Step 2: The AAF file is parsed and the corresponding media files are picked for archiving. Then add a description, add tags, and run.
Final Cut XML Archiving
Step 1: Via the web interface, select a XML exported file. The exported file is a meta-data export.
Step 2: The XML file is parsed and the corresponding media files are picked for archiving. Then add a description, add tags, and run.
Globally Browse and Search Your Archives
- DNA Evolution allows you to browse and search your archives across all your archive jobs, servers and locations.
- From the web interface, you can give end users, partners and even your customers the ability to:
- Browse archived file names, size, description, tags, tapes
- Search by tags, descriptions
- Play proxy files from the web
- Find what tapes your content is on
- DNA Evolution enables you to setup access controls so approval users are limited to search, browse and proxy play – without the ability to archive content.
- Once you find the content you are looking for, DNA Evolution enables open and disk-like access to your tapes – on any LTO-5 drive. The next section highlights tape access.
Open Exchange and Access Archived Tapes with the Flexibility of Disk
Step 1: Simply download the free DNA Evolution browser software. On any system connected to an LTO-5 drive, mount the tape using the DNA Evolution browser software.
Step 2: Access content like a hard drive. The DNA Evolution browser allows you to double click and open files, drag n’ drop one or more files directly to Finder/Explorer windows, drag n’ drop files directly into applications.
Step 1: Rather than archive files on-demand, you can select folders and setup an automated, continuous backup. Source folders are automatically monitored and new updates are moved to LTO-5 tape.
Step 2: In case you need to recover content, use any LTO-5 drive to mount the tape, browse the tape like disk and drag files to your disk based system.
Integrates Disk, Tape and WAN
Not only does DNA Evolution enable archiving to tape but also integrates disk based synchronization over LAN and WAN links. This capability allows you to integrate both tape and disk workflows. Some sample workflows include:
- Archiving in the field to LTO-5. At the same time, sync low res media over the WAN to a central location for editorial or dailies.
- Sync source footage to LTO-5 tape for archival while also syncing source content to your SANs for editorial purposes.
For more information on disk based and WAN workflows, please refer to DNA Sync product information.
Product Details and Pricing
NOTE: The following prices are only approximate to help you understand DNA Evolution pricing. Please refer to your local StorageDNA reseller for up to date pricing information.
DNA Evolution is priced by three components. This includes the DNA Evolution Archive Server, the DNA Evolution Archive Client and the DNA Evolution Archive Controller.
DNA Evolution Archive Server (Call (818) 303-3900 For Price):
The Archive Server is a software component that can be installed on a workstation connected to your tape hardware. The archive server can archive any data directly visible on the archive server and also enables network archiving of any system with the Archive Client installed. The Archive Server is priced per workstation and allows you to attach any number of tape drives.
DNA Evolution Archive Client (Call (818) 303-3900 For Price):
The Archive Client is a software component that can be installed on any system in your organization. It allows you to archive any content on that workstation to one or more archive servers in your environment. For example, you can install an archive client on one of your SAN client systems to archive the entire SAN. Archive clients save you the need to deploy tape hardware at every workstation you need to archive.
DNA Evolution Tape Browser (Free):
The DNA Evolution tape browser is a free software component that allows you to mount an LTO-5 tape with DNA Evolution archived content on any LTO-5 tape drive. The tape browser acts as a more efficient Finder/Explorer enabling disk like access to the mounted tape and its contents.
DNA Evolution Archive Controller Appliance (Call (818) 303-3900 For Price):
The Archive Controller is a one-time investment that enables a scalable archive environment. It can manage multiple archive servers, clients, users and locations. The Archive Controller acts as a central repository, policy controller and web manager. It provides centralized management, scalability, search and access features. Every DNA Archive Controller includes one Archive Server as part of its pricing. The Archive Controller is a hardware appliance that comes in multiple hardware configurations (Rackmount, Rackmount w/ redundant components).
StorageDNA Goes Down Under with DEEPSEA CHALLENGE
By Debra Kaufman - CreativeCOW.net (Click to view the full article)
National Geographic explorer-in-residence and director James Cameron launched the DEEPSEA CHALLENGE project -- a historic solo dive to the Mariana Trench in the western Pacific Ocean -- not just for the adventure but to document it for an upcoming feature film. The untitled 3D documentary, which is due out in 2014, will comprise footage from nine dives to various depths and showcase the lightweight compact 3D 4K cameras and rigs from Cameron Pace Group's R&D.
Behind the scenes, Cameron and his team partnered with Australian film/TV post production company Digital Pictures to deliver a range of post services. Given the massive amounts of 4K data that the shoot would produce, Digital Pictures aimed to create a digital post production pipeline that consisted of a high performance archive and retrieval workflow solution that was both efficient and cost effective. The solution needed to decrease time spent in post production, be able ingest/digitize film/tapes to DPX, reduce costs for stock media, SAN storage and nearline disks, minimize form factor and maximize performance, and ensure format flexibility.
"The existing post production workflow involved time-consuming manual processes with high stock media and SAN storage costs," says Digital Pictures Technical Director Nic Smith. "The producers wanted the ability to quickly back-up data, transcode to dailies and view everything right out of a camera negative while protecting the digital assets." The production also needed a solution for its RANGER Data Cart, a streamlined content management system designed to manage the data demands of a production in a mobile or remote location; the RANGER Data Cart would be on the ship from which the underwater dives would be launched. "This was a very challenging and complicated project because it was a multi-camera shoot with every kind of file format you can think of," adds Smith.
At the heart of the project was the effort of Digital Pictures, partnered with workflow/archive company StorageDNA, to turn LTO from a strictly archival medium into an online conforming medium. The results, thought Smith, would give Digital Pictures exactly what they needed to offer cost effective and efficient solutions to archive, restore and conform content in a tapeless workflow at very high speeds.
The solution -- and the relationship -- began, as these stories often do, at NAB. "Nic Smith approached me at NAB 2010," says StorageDNA CEO Tridib Chakravarthy. "They wanted to deploy file-based pipelines. There are a number of vendors out there that focus on the LTO archiving space, but he wanted to do something different."
"Nic asked if it was possible to turn the LTO into a medium that you can do conforms from," he continues. "We scratched our heads and thought, this could be possible." Smith worked with StorageDNA on specifying the conform process until, two years ago, Chakravarthy was able to give him a beta version of what he was looking for. "We've turned LTO into online storage," he says.
After using the product on several projects, says Chakravarthy, Smith came back with a bigger project -- the DEEPSEA CHALLENGE -- that was switching from tape to tapeless workflow. "Everyone wants to go tapeless," says Chakravarthy. "Everyone puts material on LTO. But, after you put your material on LTO, what is the most creative way of getting your content back?"
"Normally, they would have to keep the high resolution media live on their online storage," he says. "With our solution, an EDL of the low-res sequence finds the frames or sequences across all the LTO tapes -- no matter how many -- and brings only those back to for conform. It's a huge cost savings. You reduce your storage by one-third no having to use online media, and no one has to manually find the frames or sequences."
Digital Pictures customized the RANGER cart with StorageDNA's DNA Evolution for the archive workflow solution. "The DNA Evolution LTO-5 archive workflow has allowed us to save tremendous time and costs for our clients," says Smith, who notes that Digital Pictures has used DNA Evolution for several clients' productions since July 2011. "Since we're now working in a digital pipeline, the need to ingest has been eliminated and the conform process has been fully automated."
"The reason we chose this LTFS archive solution is we needed maximum performance and reliability since we often deploy in the field for our clients," he adds. "DNA Evolution met all of our rigid requirements."
The pipeline is applicable to any file-based acquisition, from RED Epic to GoPro cameras. The workflow for DEEPSEA CHALLENGE project presented very challenging parameters. "As raw footage came onto the ship, the first thing they'd do is off-put it to LTO tape and also make an editorial low-res copy to DNx145 or ProRes HQ that went to editorial," says Chakravarthy. "The editorial team edits in low-res and then feeds us the EDL. Using reel and shot number metadata, we find the right tapes and automatically pull out the correct media. Digital Pictures had 24 1.5 terabyte tapes online, so it functioned like near-line storage."
On the ship, the team actually made two LTO copies, one for conform and the second for its more traditional use -- disaster recovery or archive. Transfer between DNx145 and LTO tape was also fast: under nine minutes for an hour's worth of footage, says Chakravarthy. "We are trying to put all our innovation and transform something that ordinarily sits on the shelf for 50 years into something that's more active and near-line. The performance and price benefits are pretty amazing."
The system also scales. "There is no limit," he says. "I could have 5,000 LTO tapes sitting in a warehouse and you can give us an EDL and we can tell you what tapes you need, plug them into the auto robot and we can restore them to you. You can get 100 TB of LTO tape for under $4,000 so it's infinitely scalable and cost effective."
Chakravarthy says they're getting ready to deploy this LTO-based pipeline for reality TV and episodic TV, for different reasons than what made it ideal for the DEEPSEA CHALLENGE project. "It's not that they have a lot of content because of 4K cameras," he says. "It's because they deploy Panasonic P2 and Sony HDCAM and record a lot of material."
As content creation moves increasingly into a tapeless world, Chakravarthy adds, the issue of how to handle massive amounts of data will become even more crucial. "With tape, producers had storage media they could put on the shelf," he says. "Now they're making LTO copies, but think of it as archive. Nobody has stepped up to transform it into near-line. The pipeline is pretty messy."
"Anyone can use LTO for an archive," he adds. "But who can do intelligent restores and conforms? That's where we focused. If you're mandated to shoot tapeless, you still have a time crunch and a budget and vendors haven't come up with a solution. You only need one hard drive to fail to be convinced that keeping drives on a shelf is not a great idea."
Digital Pictures thought out of the box to provide Cameron with a solution to meet his extreme shooting conditions. In the process, they nudged StorageDNA to reinvent the use of LTO tape, and come up with one solution for handling data and metadata in today's often confusing tapeless workflows.
If post workflows are snowflakes, then StorageDNA won't be the only company to come up with a solution to challenges specific to today's TV/film productions. But they're a good example of how ingenuity paired with technology is capable of inventing new ways of working. Chakravarthy says Storage DNA is already planning their next line-up of innovations. You can be sure we'll cover those -- and other companies' innovations -- as they become available.
StorageDNA Video Gallery
BAMM.TV: LTO/LTFS Archive & Retrieval Workflow Solution
BAMM.TV: LTO/LTFS Archive & Retrieval Workflow Solution
Create a Basic Archive in DNA Evolution
Archiving Folders and Files in DNA Evolution
Archiving Camera Media in DNA Evolution
Restoring Using the Web in DNA Evolution
Avid: P2 Conform from LTO in DNA Evolution
FCP7: Archive via XML in DNA Evolution
FCP7: P2 Conform from LTO in DNA Evolution
FCP7: XML-based Restores in DNA Evolution
StorageDNA: Archiving Camera Masters
StorageDNA: Bin and Sequence Archiving and Restore
StorageDNA: Conform from LTO-5
StorageDNA: Performing a Web Restore