Digital camera post data workflow

For shoots on Arri Alexa, Red Epic, Sony F65, Canon 5D, Phantom, or any such digital camera, is there a preferred workflow for the post?

When the edit is complete and its time to extract files for promos, for VFX, or even for the final DI, most producers send the original hard disks (al 20, 30, 50 or more drives) to the DI facility. Places  like Prime Focus, Reliance, Prasad or wherever. 

In these studios there are a bunch of very over worked in-house data managers and conformists who use the hunt-and-copy technique to copy the selected files from original disk to their own Baselight or Resolve or whatever. 

Delays, wrong files, confusion, tempers… and other such dramas end up happening.

What’s the best way to do this? Is there some kind of data or file management in post? Can someone who can handle this workflow more elegantly?

Over the past few years, and after doing it for a dozen films, I’ve worked out a cleaner method.

1. First, I make a database of all your hard drives. Which should be anything between 10 to 20 to 50 individual drives. And data spanning anything between 10 Terabytes to 100 Terabytes. All this happens at the producer’s office, or they can send drives to my place. I save only the file names and paths, no pictures. Or sounds.
2. Then, as and when files are needed, I call for the Avid or FCP sequence. No need to make EDLs or XMLs or anything. 
3. Then, my software scripts go through the Avid or FCP (or even Premiere Pro) sequence extract needed files, and search my database for the needed files. No need to access the hard drives yet. So all this can happen at my place and you can simply email me the Avid or FCP sequence.
4. Then I build scripts to automatically copy the necessary shots to another hard disk for sending to Promos, VFX, DI or wherever.

The result is that instead of carting all your tens of hard disks to the DI place, you only send out a small drive containing only the used shots and nothing else. And, to save space and time at both ends, I can even ‘trim’ the original shots to the required duration plus handles.

Sending me a reference mov helps me to confirm that I have the right shots. But that’s optional. If the original conversion from camera files to FCP or Avid files was done accurately by the data manager on location, then my systems and software usually pick the shots fine.

As an example, for a movie that released last week, the original F65 data was spread over about 15 drives, about 30 Terabytes. The sorting took just about 2 days. And the sorted reels were ready at about 4-5 hour intervals. So, 4 hours after I got the edit, the DI could begin.

And the total sorted data with 1 sec handles was just 4 TB. And this is the archive the production needs to keep a backup of after the film is released.

Conversion, transcoding, dailies

After the shoot one needs shooting files need to be converted (or transcoded) to something before editing can begin. That something can be Apple ProRes for FCP or DNxHD for Avid. 

This conversion is usually done by the on-location data manager. But, with the sheer volume of shooting, multiple cameras being used, time to backup being huge with older laptops, data managers often find the task daunting. So, they sometimes end up with errors like not passing the original timecode, reel names, or even at times messing up file names.

If I or my systems do the original conversion on a daily basis, then the database of your drives is built as we go. So, in the end, shot spotting and copying is much faster and more accurate. So, productions send me one copy of the shooting drive on a daily basis after packup each day. And I do conversions overnight and you have editable dailies the next morning. For outstation shoots, they cargo the drive to me every few days and I do the conversions.

So, digital shooting need not be a nightmare in post and DI. Some planning, some competence in data handling, and its a breeze.

DIT in India

For a while now I've heard from DoPs and cinematographers in India shooting movies on digital cameras - Alexa, Red, Sony Canon etc - requesting the services of a DIT "just like in Hollywood". Which basically means what they read about on the net, in forums, and in American Cinematographer.

I run a small company that provides data management and workflow guidance services for feature films in India - called Post BlackBox. And we are often asked to provide DIT services in addition to data wrangling and conversion that we do. We would love to.

I've worked on one movie with a 'real DIT' and have shot in the US and conversed with DITs there, so I've examined this in relation to how we make movies and the kind of post services available here for digital workflows.

Incidentally, for an explanation on dailies, video assist, and the 'look' in digital movie-making, check out an earlier post I wrote on this.

What's a DIT

For a definition of a DIT, check wikipedia…

http://en.wikipedia.org/wiki/Digital_imaging_technician

and many debates like these…

http://www.arridigital.com/forum/index.php?PHPSESSID=p6rird78fdt6ndj1kodk52h4b5&topic=6845.0

and a very informative article at the Creative Cow…

http://magazine.creativecow.net/article/a-dit-tells-all

What a DIT does

So let me first set up what a DIT typically does. And how that fits in the general post-production workflow.

On a movie set where a feature (or TV show, docu, ad film etc) is being shot on a digital camera, a DIT…

1. Copies camera footage from camera cards/disks to multiple disks before the card/disk is erased and reused.
2. Verifies that these copies are fine compared to the original which will be erased.
3. Shows the DoP what his material will look like in post. So the DIT carries a good calibrated monitor or can calibrate the monitor that came with the camera.
4. Makes color adjustments to the camera picture either off a live video feed, or on shot material. This is 'setting a look' similar to doing one-light rushes in the film days.
5. Has access to and is knowledgeable about LUTs and conversion of a variety of log an other color spaces to 'normal' color.
6. Has instruments (waveform/vectro/histogram) that can analyze the camera image and hence advice on aperture and exposure setting. The Cinematographer takes this advice seriously.
7. Sets up timecode and sync between camera and sound. 
8. Is knowledgable on menu settings and controls of most digital cameras. Or at least has access to camera manuals and can refer and find a fix if needed. Good DITs do refer to manuals, even in the field.
9. Can do conversions of camera raw formats to editable formats for FCP, Avid, or any other editing platform.

So a DIT is knowledgeable is on digital formats, and also has some sense of how film used to behave in different lighting situations. A DIT also has a good sense of post-production and editing and is often from an editing or post background. 

So, the DIT is also a first step to post-production. So, in that sense he (or she) takes up some of the jobs that the post house would do. This is an area of debate with post houses trying to reclaim some of the tasks.

Cost of DIT services

In Hollywood, a good DIT cart with equipment for all of the above DIT functions, rents out for about $ 1000 per day (equal to about Rs 56,000 per day). And a good DIT charges between $ 500 to $1000 per day (equal to about Rs 28,000 to Rs 56,000 per day) excluding gear.

So, we are looking at about Rs 80,000 to Rs 1 lakh per day for DIT services by Hollywood standards. On a 50 day shoot, this amounts to probably more than what some of the senior members of the crew make in the average budget Hindi movie. 

Clearly this is not affordable in India.

Check out what fully equipped DIT carts look like…

http://bigfootmobilecarts.com/BigFoot%20Mobile%20Models.htm

This one is from Maxx Digital. A complete mobile Scratch solution

Cost, about Rs 27 lakhs. 

The cost of the equipment that goes into a DIT cart if obtained in India, would be in the range of anything between Rs 15-30 lakhs. Which makes it cost about as much as a good digital camera like the Alexa or Red Epic. So, even if one makes up a homegrown DIT system one would have to charge a rental of at least Rs 20,000 per day. 

A trained DIT, of a cinematography or post background, would charge at least Rs 10,000 per day. So that makes the total, DIT services and gear, to at least Rs 30,000 per day. Or, Rs 15 lakhs for a 50 day shoot. Plus assistants, flights, hotel, local travel, hard disks.

So, is even this affordable?

DIT vs post house

In Hollywood and even Europe/UK, post houses charge for post services before edit. Data conversions, file management, backup etc - all before the edit starts. And this charge is pretty hefty. So the cost of a DIT in Hollywood is in comparison to these costs.

In India, the cost of DI is so small in comparison, and competition so fierce between DI studios, that some of the services offered in Hollywood are often thrown for free if the DI is done with them. So, Rs 30,000 ($ 500) per day for a DIT is also probably unaffordable. When the entire DI process costs as less as Rs 5 lakhs ($ 8000) to 'as much' as Rs 15 lakhs ($ 25,000). 

Just $8,000- $25,000 for Spirit 4k scanning, conform, Baselight8 grading, Arrilaser film out, SD and HD-SR tape deliverables... everything, taking up to 3 months of post work. If you push it, then the data conversions are thrown in for free too.

Storage costs

In addition to DIT services, disk storage is borne by the production in Hollywood. This cost is about the same in India. Working out to anything between Rs 5 to 20 lakhs for good quality hard disks, and RAID drives which the production has to bear anyway. 

In India, productions then bring down the services of the data management team. So, for basic data management services, I've found even Rs 8,000 ($ 130) per day is often not accepted. DIT services are nowhere near affordable for these kind of rates.

Workflow

The ideal DIT workflow requires time to set up the cameras, monitors, scopes and timecode. Directors often get impatient as now, apart from lighting and actor delays there are these 'set the look' delays. So they prefer to postpone this to the DI stage. 

DIT in India

But there's still hope for DIT services. Particularly for smaller budget movies where getting the look right in the shoot stage is more crucial as they don't have the time for reshoots, or for an extended DI. Nor do they have time and budget for elaborate lighting setups. So in these situations having some rudimentary DIT services can affect (positively) how the movie turns out.

So, while a full blown DIT cart and fully trained DIT is not affordable (someone please prove me wrong), a more basic setup with capabilities nearly as in a 'proper DIT' can be made possible. If the budget can go up to at least Rs 15,000 per day, a DIT can make a difference. 

He or she can work closely with the editor and colorist so this can be a end-to-end service which integrates DIT, editing, and color grading - the entire post cycle. Then it could make sense, economically and aesthetically. If that makes it look interesting, then one could take a closer look at providing these services.