Simon “Thanks for raising the topic. Its something we should all be talking about far more often.
With each new technology, we seem to be going to media with shorter and shorter storage lives. From the age of film and vinyl records which had a very long life; to tape with still a fair life; to DVD/CD/BluRay with about 17 years; to hard disks with about 4 years.
I certainly remember when I learnt about the likely lifespans of hard drives, and then was a bit shocked to learn that they can have a much shorter life when they aren’t used regularly. Had been archiving everything to hard drives and placing them in storage at that stage.
Learning that and not having tapes to go back to, started me thinking long and hard about how to store all our assets for the long term.
Am testing LTO tape – long life but clunky process. In the meantime am working on the basis that we should have three copies of everything and at least one copy kept off site. Like Petter, a Mini and RAIDs works well as part of this.
None of this is quick and its proving reasonably expensive. Am looking forward to a better long term approach to archiving media assets.
Certainly something to talk more about.
HDW : Archive was easy 25 years ago, we had tape and although not without it’s problems it worked very well for archive purposes. Today most of us are stuck with 2-3 copies of important video files on various hard drives. Hard drives are the most common, cheap archiving for todays digital age, Solid State or SSD drives are far too dear and beyond 750GB cost a small fortune.
The boys in Japan have never quite conceived the problem with archive and digital media, Sony’s SxS and Panasonic’s P2 cards are a prime example, useless for archive due to cost.
Sony have unveiled their next-generation Optical Disc Archive storage system. The system, which provides guaranteed inter-generational compatibility and ultra-reliable long-term storage, is at the heart of Sony’s vision to create a complete archiving solution for broadcasters, motion picture companies and production houses.
Optical Disc Media
A variety of high capacity media cartridges are available for the Optical Disc Archive system, from 300GB to 1.5TB. Each cartridge allows random file access using the Universal Disc Format (UDF) system. Optimised for ultra-long term media archiving, the cartridges have an estimated 50 year lifespan and are built to withstand changes in temperature and humidity, and resist dust and water droplets. Each cartridge has a built-in RFID to check the contents of the media with mobile devices for efficient asset management, such as cartridge loaning, stocktaking and shelf archiving.
This system does not seem to be available yet but by all accounts it will be expensive, but how do you equate Expense v Lost Work
Duncan’s solution “Personally, I keep all my rushes on pairs on USB2 1.5TB hard drives, really cheap and easy to do.
The original footage might have been rewrapped to QT or optimised to ProRes, but I will only then archive the assets that I created (animations, AIFFs etc.) and the masters involved. And again this gets archived to pairs of USB drives.
If I need to recreate and edit I still have the camera originals, and all the assets it’s simply a case of reinstating them from the archive.
On a daily basic, I incrementally backup up my some of my scratch drive, a 2TB internal RAID to a 1.5TB external at the end of each session. Again very easy.My aim to minimise hard drive failure will be to replace the internal RAID every year, only £200 at the most. I might well use 2x 3TB drive once their a sensible price/GB and more reliable.
Of course, I’m not working with 200 hours of uncompressed HD over 5 simultaneous projects, But I don’t think you are either?”
I am just starting to archive a 500+ tape HDCAM library with Catdv and plan keep everything on two hard drives and LTO5 LTFS tape. New SxS and P2 cards will be archived to LTO5 LTFS tape before putting on 2 hard drives for logging and general use. LTO5 tapes will be kept offsite. BTW Catdv proxies will allow viewing of complete library of shots for making rough offlines and sales use.
Before someone jumps in and suggests to me that hard drives only have a short life span and I’m taking huge risks…
I started on the hard drive archiving ten years ago.
Back then I was using 120GB bare IDE drives, these sat on a firewire adaptor caddy. FW & USB enclosures were really expensive and there was little choice.
Eventually as the price/GB came down I moved and combined drive onto larger drives. I was still shooting on tape at this point however.
Once I got my EX1 is began archiving the BPAV folders onto pairs of 1TB and 1.5TB USB drives. I use a drag and drop approach. With a paper log of the clip names/number used for each client, blah blah. that log stays with the camera. I won’t go into too much detail here.
After copying to a new numerically incremented subfolder on the HDs I’ll use the compare function in Toast to CRC check the copy. I’m normally doing all this on a spare computer during the evening after the shoot and the next day if I have time.
I use CDFinder to catalogue all my hard drives.
These drives will get spun up for an hour or so at a time each time I’m been out shooting, they aren’t being pushed too hard and should last some time. If one failed I would be able to use the other version, and I’d be copying it all off ASAP to another pair of backups pronto.
Eventually 3TB USB3 drives will be £55 and I’ll move these onto larger drives, and so on.
But I won’t ever be putting all my eggs into two baskets… Something will come along eventually that doesn’t fail so fast, is a sensible price and reads/write quickly enough. Maybe it’ll be flash based?
The worst thing about all the different types of USB drives I have is the bloody power supplies, but gradually manufacturers are moving to a single type of 12V PSU with the same connector.
My old Lacies (almost all have dead controller boards), WDs, and Buffalos use multi-voltage multi-pin connectors and aren’t cross compatible.
And back to my original point, I had almost a hundred drive I suppose. Nad I think there’s been one date clicking failure I can remember. The other failures were burnt out controller boards on the bare IDE drives, when I used a third party 5v/12v PSU and it’s went titsup.
That’s my workflow as it’s evolved, YMMV as they say. If you’re out filming everyday ten you’ll need something quite different. But you’d probably need an assistant and have a lot more money that me to pay them!!
Cheers, Duncan.
Since moving to AVCHD camcorders I have a LTO3 tape backup of the native files as well as LTO3 tape backup of finished projects and also LTO3 tape backup of system files including the NLE project files. Also keep for quick access copies of finished projects on hard drive. I was always happy with just the video tapes and since LTO tapes are more reliable see no reason not to view them as my new video tapes for the original files !!! They are a LOT lower cost than video tapes too. My Quantum HH LTO3 drive cost about $1100Can when I bought it several years ago and I use Retrospect software. I am thinking of getting a RAID5 NAS as a better solution for all the bare drives I have around. Video is a serious retirement hobby for me but I still have 5 cameras and over the years manage to average out at about 200G a month of storage needs. PC has 8T internal storage, I have accumulated about 40 400G LTO3 tapes for my backup, source files and finished projects. I have a project hard drive to which I transfer all original files from the cameras using the Sony CMU or MBS, backup these files to LTO3 tape before i start editing. I use Edius to edit so edit using these native AVCHD files from this hard drive and when project is finished delete from this drive ready for the next project. Its a 2T drive so can run several projects at the same time. I have directories for the groups I video so these stay and I can then always restore from tape if I need to re-edit a project.
Ron Evans
Well, I have 2 1-T WD drives in a pretty fast Win/Intel PC. I have also just added this week a Synology 8-Terabyte Raid 5 to my system. I’m feeling pretty good about Raid-5 redundancy. as backup.
I have never had an internal HD go bad on me. Knock-On-Wood! But I must say I’m quite surprised at how many HD Ctrlr boards are claimed as burnt out. If that is truly the case than I would consider looking into how much cooling is provided to those HD’s. Keeping them running super cool is paramount to meeting and extending the MTBF hours. In my PC case/Mid-tower, I have 3 120mm fans, and I have 3 140mm fans, plus 2 40mm fans mounted on the top of the 3.5 1-T HD’s.
So far staying true to belief about limiting thermal breakdown has worked pretty well. At this time my Raid-5 is all I’m using for back-up, until 1-T SSD’s and larger become more affordable. SSD’s just plain rock!! I do have a 480-GB SSd as my Boot-OS-App drive, Love it! I also back up an image copy of the Boot drive to my Raid.
I think you will be surprised by the cost of Sony’s ODA system. Sony are trying to make the cartridges cost around the same as a good quality Hard Drive. The actual deck will not be cheap, maybe around £2k, but the cartridges should be very affordable and competitive with hard drives. Unlike LTO you can store ODA discs in almost any environment and they won’t degrade.
Southerndude – I’ve seen the aftermath of a production company that entrusted their archive to a single raid array only to have the main power supply fail and fry all the hard drives with a massive over voltage.
They did recover most of their data by getting new controllers fitted and calibrated for their drives, but it cost thousands to get back and took about 8 weeks.
ALISTER- I have never heard of that happening to any power supply, Failing yes, but burning up the hard drives, never. This raid was not cheap, and has dual cooling fans that alarm and notify you if any issues disrupt there operation, same for the power supply. Again, I will say that many problems that occur with power supplies, circuit boards or PC electronics in general come from thermal breakdown. You could also have a faulty unit, poorly assembled, or poor components. The MTBF on these drives is long,and will must likely upgrade to larger drives in the foreseeable future. I appreciate what your saying, but that has to be astronomical odds of that happening unless I bout the same HD’s and NAS Raid system they did.
Video for me is a serious retirement hobby but even I have several backups. I can’t imagine a business being silly enough to have just one backup, in one place.
Archiving… God, I love this topic. What I hate is the current state of solutions available for independent producers. I’m currently using blu rays for archiving (way too slow). Have been following the LTO developments for the media industry. I got exited when I heard about Long Term File System (LTFS) on the LTO5. Then got even more exited when I heard about the StoreOpen software from HP that would make the whole process more user friendly, allowing the use of a GUI instead of a command-line. So there is finally an “open”, more standardized solution at “affordable” prices (if you shop around). The only thing I don’t like about LTO is that it is “tape”, and I started hating tape long before the media cards came out. I always found tape to be a fragile format. That’s why, when Sony announced the ODA format I got exited again, and decided to put my LTO purchase on hold. I read some preliminary prices on the ODA drive and media, and thought they were kind of high when compared to LTO5. The other thing is that the media seen to be just cartridge with multiple blu ray discs inside. It reminds me of the early CD Changers. I think Sony can do better; maybe something like holographic disc or card.
Archiving has been a concern since the advent of the solid-state media; I’m surprise there are not more competition at the low-end of this market. The general public rely on hard drives and never worry about what will happen to that data in the future. They just say “I’ll just buy another hard-drive, you can get 3TB hard drives for pennies these days”. But we all know that the issue is not that simple. Just the other day I helped my sister, recover very valuable footage from her Hard Drive based camcorder. The camera had not been used for a while and when she turned it on for the first time in months, it was giving her an error message. The manufacturer recommendation was to send the camera to a Sony authorized service center. After a lot of research and downloads of data recovery programs I was able to “heal” the data and put it on a new hard drive. Then I formatted the hard drive inside the camcorder and she was able to use it again. It’s was a long process but the footage was “priceless”.
The message here is that, professionals and consumers alike, are affected by the same issues. The difference is that the general consumer, trust technology like something that never is going to fail; that is at least until it does. We, the professionals, have enough experience to know that we have to “prepare for the worst and hope for the best”.
I’m very grateful for this thread and hope that we can continue this conversation so many colleagues around the world benefit from this knowledge.
Retrospect is a very easy GUI driven program to use to manage backups with all sorts of media including the LOT3 tape that I use for my backups. It goes as fast as the hard drives can sustain, about 65MB/s in my case so backups are very fast and recovery is as quick. As a hobby I have NX5U, SR11, XR500 and CX700 cameras as well as HX30V compact camera, GoPro Hero 3 Black edition and a ContourHD. Of course I also backup my daughters cameras !!!! One has my old SR7 camera and both have HX10V Sony’s They at least are computer savvy and also backup on their own computers. I too have recovered peoples files from cards they formatted unintentionally but at least had the sense not to do anything else to the card before they gave it to me !!!
Ron Evans
I really appreciate the kind of topics you post here. Thanks for sharing information that is actually helpful.