Attack Of The Terabytes

There was a movie a couple of years ago called Cloverfield, which was one of the first to use the Sony F23 and a few other digital cameras, such as the Panasonic HVX200, that needed to be ‘data wrangled’. That DIT became one of the most important members of the crew as different departments looked to him to either draw their footage off cameras and then direct it in to the post pipeline. Also VFX supervisors needed to talk to him about tracking questions and to see certain scenes in full resolution. Not to mention the DoP wanting to review shots.

This position demands a number of disciplines in shooting and post production and on the movie District 9, the RED camera supervisor, Jonathan Smiles, more or less took on that role.

The movie was originally going to be shot maybe with a Sony F23 or F35 – it was then decided to go with producer Peter Jackson’s own RED cameras. But to provide the ‘videography scenes’ of the movie a few Sony EX1s were used with a Canon AVCHD camcorder providing the CCTV footage. Aerial footage was shot on the Cineflex to HDCAM SR tape. There was also an IR camera in the mix.

On set Jonathan had his own ‘data’ truck, “In my truck I looked after all the data formats, primarily the RED but then the EX1 had to be handled, that would come from multiple sources, so we had to copy that, spot check and make multiple copies.

“It’s all data coming in on CF cards, Sony SxS cards and SD cards. The SR tape from the Cineflex would go to the post house for cloning and then directly to editorial.

“My job on this movie was always going to be big because of the nature of the beast, visual effects heavy and of course keeping the REDs shooting. We had procedures for re-cycling media which had to be totally nailed down to prevent any data loss – we actually had zero data loss on set.

“I would always make sure there were multiple copies of the data. Two of the copies were transfer drives, so they were hard drives in a Peli case and one would be transported direct to editorial and one would go with the DoP as his rushes drive. What that means is that we have complete copies of all the data from all the cameras, and on those drives we also backed up the script supervisor notes and the sound. So we can lose the DIT vehicle but we still have another copy!

“On a busy day we would have about 500GB worth of data. On the RED we were shooting on 8GB cards which gave us four and half minutes per card. Having this constant traffic of data to and fro is actually quite a good thing because instead of having these large lumps of data coming at lunchtime and ‘wrap’ we’re getting this constant trickle which actually keeps the machines working very well. We would split rushes every lunchtime, so every lunchtime me and my assistant would send the transport drive for editorial back and at the end of the day there would be the DoP one. So editorial would get the morning’s material at lunchtime and so that could be processed and be with the editor in the evening. Then in the evening they get the afternoon’s rushes.

“The important part was getting the material in to the hands of the editor, back then it was slow to get the RED Code into the Avid, as the tools matured we got that down to two hours. So that’s one hour of material in front of the editor in two hours using RED Rushes.”

As well as the RED, the EX1 and the Canon AVC HD were working everyday with their own loaders. “The EX1 and Canon footage had to be transcoded into Avid DNXHD – one of the beauties of having a Mac is that Final Cut can take anything in and transcode it to DNx36 material, and we had procedures in place so that we were using Final Cut as the bridge to get the XDCAM HD material and the AVCHD material into the Avid. At the time Avid didn’t have AVCHD or XDCAM HD native. So we had to do something with it in such a way that when we had to do a conform we could go back to the original material – so making the DNX36 just an offline step.

“So I’m presenting editorial with a Firewire drive and on that drive will be a folder that will be ‘D’ for day and which day of the shoot it was, underscore then the date. In that folder there will be a folder for RED Cam A and Cam B and a folder for the EX1 and so on. So for each camera letter there will be a dated folder and for AM and PM as well. So you can go back and find say Tuesday morning’s shot and in there will be everything. All the camera original files, all the sound recordist files and script supervisor notes. On other productions we have done some of the transcoding in the truck, perhaps when we’re not running so many cameras and they’re not shooting super fast.

“For this movie we had to concentrate on getting the data safe and getting it back to the camera department so they can re-use it.

“We are the first port of call for the cards and they have to be passed to us in a procedure that is written down. If somebody brings an ‘exposed’ card and leaves it on a table we might not know so it has to be physically given to us. Immediately I get it I’m getting it copied and we clone the whole volume structure, we don’t just copy the files so we have the mag information as well.

HD Hub
HD HUB is part of the Visual Impact Group and provided the editorial, RED data backup and rushes services on the movie. James Middleton from HD Hub explains how the pipeline worked once they’d received the transport drives.

“We provided all the RED data handling equipment used on the shoot. On set the data was copied from the memory cards, to a RAID protected stage array and a copy taken to post/editorial on a transport drive.
RED files were copied on an Avid Unity – SAN – we had a 16TB unity on the movie, files were then backed up to multiple LTO 3 data tapes, using Time Navigator software, due to insurance conditions we had to have multiple files away from site within three days of shooting.

RED files were converted to new proxy files using ‘Red Alert’ and imported into AVID as AVID DNxHD 36, the Metadata was converted into an Avid ALE using Metacheater software

The other footage, like the EX, was converted using Final Cut Pro log and transfer and exported as Quicktime files and imported into AVID as AVID DNxHD 36, also AVCHD files from Canon consumer cameras were handled in the same way

“Aerial footage was shot on HDCAM SR tape and digitized directly into Avid from an SR deck, also a low cost protection copy was made of this footage at DNxHD175 in case of tape loss

“We produced approximately 120TB of data over the course of the film.

“There was so much footage, five to six hours a day on so many different formats, that no one thought it was possible at the time.

“At the time I remember people outside the project talking about how hard it was to deal with an hour of RED. We often dealt with five hours of footage a night every night for the duration of the shoot. Our data backup systems had to handle this large amount of data with fail safe redundancy and integrate directly with Park Road post in New Zealand, the post company owned by Peter Jackson.”

“RED footage was converted using Red Alert, editing was done on the Avid Media Composer on Avid Adrenaline HD systems on Macs, in the end we had 11 Mac systems on the movie.

“We built a mini post house at the production office with everything connected via fibre channel to Avid Unity SAN. We had four editorial machines and then data wrangling/render machines and 2K quality check stations on all connected together.

“Peter Jackson got his rushes on XDCAM HD disk, all other rushes were done on DVDs with each recipient name watermarked into the footage – security was intense on the movie.

“But only a year later and it’s all much easier now, the RED software now has better integration with Avid through RED Rushes. For current projects we use FilmLight’s BLT – Baselight Transfer, they made a tool set to do all the functionality required in real time such as converting files and one-light grades etc, so there is much better workflow now for these types of projects.”

District 9 shot so much footage that wasn’t used in the final film including huge sets of corridors inside the space ship – it should make for a good extras part of the DVD. All fitting on a tiny 4GB silver disc.