The age of the visualisers
Posted on Jan 10, 2020
You are a builder of worlds, you sit astride multi-cinematic disciplines with the imagination to foresee complicated but extraordinary movie sequences: you are a visualiser
Words Julian Mitchell / Pictures The Third Floor, HBO and ITV
Previsualising, or previs, is hardly new. It was originally a modern way to animate storyboards, which itself was a traditional way of mocking up scenes that needed some extra help to imagine. The shower scene in Hitchcock’s Psycho featured over 100 cuts so had to be carefully pre-planned. In 2002 David Fincher’s Panic Room had beautifully previs’d moves that allowed the camera to navigate from sleeping homeowner upstairs to intruders looking through the kitchen windows downstairs in one sweeping, hypnotic shot from one to the other. There are hundreds of other examples but these two stand out in the evolution of the art.
The Third Floor
Most of our guys come from an animation background; rarely do they come specifically from a VFX background as you might assume
15 years ago a new company was formed out of Lucasfilm; it was called The Third Floor (TTF), after where the founders worked in Skywalker Ranch. They were six guys who were working on Star Wars: Episode III and were pioneering the ‘visualisation’ process and pipeline at the same time. Alex Webster, MD of The Third Floor UK, explains the further origins of the company.
“It was the first time that using CG animation allowed director and DOP to really plan what it was they were going to shoot on a particular day, what would be CG and what would be live action. That would allow them to make early decisions thereby saving time, money and resource on set.”
The company has grown from that point from its core business of feature films, crossing over to work on projects including television, games, VR and themed attractions. Webster brings us up to date. “About eight years ago Framestore in London was embarking on early development on the film Gravity with Alfonso Cuaron. They were using a creative process, a very technical one which would allow the filmmakers to really define how they were going to make this so-called impossible-to-make movie.
“William Sargent, who is the CEO of Framestore, knew that they needed some kind of extra support in that process by bolstering the previs part of the crew. He got hold of Chris at TTF in LA and they deployed a handful of artists to come over to London to work alongside the guys at Framestore in the early pre-production phases. From that, the relationship grew and Third Floor London is now eight years old and is a joint venture between Framestore and The Third Floor LA.
“We’re now a team of about 100 people and our core business is with the major studios – Disney, Warners, Marvel, Lucas Films – as well as doing a lot more work now with the likes of Netflix and Amazon on the streaming platforms.”
Bridging role
The Third Floor sits somewhere between traditional pre-production, early concepts, art department, scripts, storyboarders and the live action and VFX teams. “Paradoxically we work underneath visual effect supervisors but most of our work is with the production team,” says Webster. “We work closely with DOPs, directors, art directors, production designers and stunt coordinators to inform the creative choices around shot composition, editorial structure and camera language as well as helping solve technical problems as to how complex sequences or single shots are going to be captured on the day and in CG.
“We provide an essential sandbox of tools which allows those department heads to play and make creative decisions in a highly inexpensive, disposable way. This allows them to iterate, change, to consider different options in a way that is ultimately throwaway, meaning that you are making those decisions before you get into the really expensive part of the process, which is on-set and in a VFX pipeline.”
Within our systems we have every piece of kit you can imagine, from Technocranes and wire cameras to dollys and lens packages
A call will go out to The Third Floor from a director, VFX supervisor or producer early in the process of making a film, when they’ve got a script and possibly a storyboard. Webster explains the continuing process. “The filmmaker will look to us to start designing shots and sequences. In order to do that we build proxy assets, low poly (polygon) assets with which we can construct basic layouts and composition of what a scene may be. We have a pipeline and tools at our disposal that have been tuned over 15 years to be able to work incredibly quickly, so we can compose entire scenes in a rudimentary way.
Images: A finished scene from the Beyond The Wall episode in Season 7 of Game Of Thrones (top); and a virtual camera in the previs (bottom)
“Within those assets we can animate and articulate characters, creatures, VFX etc, but we can also do it in a way which is authentic and constrained by real world physics. Within our systems we have every piece of kit you can imagine, from Technocranes and wire cameras to dollys and all the camera and lens packages you can think of. So a director can add different things in, and really start seeing what those decisions will look like in a rudimentary but fundamental way.
“We really try to author the more complex sequences so you can see what they’re going to look like and how they’re going to play out.”
Tech-vis
Tech-vis allows a sort of blueprint for the filmmakers by extrapolating all the data from those sequences to be played back on set. “They’ll then have a very good comprehension of time of day, camera position relative to the sun, the lenses being used, the camera height, frame rate, what size of green you’ll need, the number of extras you’re going to need,” says Webster. “All that really key information just helps in finding efficiencies on the day and takes away a lot of the risk that is inherent on a large film production.
“One of our areas of development is in how we can incorporate real-time game-engine technology into what we do. This makes that creative process slightly more interactive. We do this in a lot of TV series – for instance in Game Of Thrones, we built entire virtual sets and we were working with the art department and production design so that we knew what the set was going to be and look like. We had the measurements and the design so we could recreate that in CG.
“That allows us to use real-world physics to help inform creative forces. If you’re putting that environment into a VR headset, then as a director or a DOP you can get into that environment and start exploring with a virtual camera how you’re going to lens a lot of this set-up. Again, relative to the real-world physics of lighting conditions, position of sun and all those kinds of variables. That’s the stuff that you would be discovering on the day in a very expensive fashion, but if you know exactly what you’re going to be shooting and from what angle, this informs what scale or size of each part of your practical set needs to be built.”
Images: For Gemini Man, previs tech-vis was used to map action ahead of filming, also informing work in locations from Cartagena to Budapest and shooting logistics with a 4k camera. The previs process also allowed shots to be developed and blocked with representations of Will Smith in both his older and younger forms
World building for GOT
As an option for all the production teams in the infamous Season 8 of Game of Thrones, virtual reality previsualisation was encouraged in order to design sets and shooting scenarios virtually before any sets were built or finalised; TTF helped with this. DOP Jonathan Freeman enjoyed this new style of prep, others were less impressed. “This new way of working was introduced by our brilliant production designer Deborah Riley,” says Freeman. “She had seen this type of work before and was very inspired by it, so suggested that we tried it for Season 8. I had a couple of critical scenes that required set extensions and set recreations, and with this technique myself and Dan and David (DB Weiss and David Benioff, the directors of the finale) were able to step in to a virtual set before it was even built.
“Working virtually in this way allowed the directors and cinematographers to literally walk in to existing space in virtual reality. So we could walk around in this virtual space and then take screenshots as storyboards that we’d eventually use to implement our previs. But it also allowed us to suggest changes in the design of the set, such as to extend a wall or push a wall back or structurally change things to make the shot better; that works for practical set building and VFX set design extension. The system allowed us to be very specific in the design of the sets and also provided us a great opportunity to imagine what a sequence would look like.
“Normally what you get are blueprints and elevations, maybe some concept art, which are there to inspire you to figure out what your shots are. Then you use a storyboard artist and get it ‘kind of’ right. But in the end it would never be exactly the same. Now, with the VR, you can essentially create a photo-board which can eliminate the need for a storyboard. Storyboarding can take anywhere from three to six weeks and in some cases we were able to skip this process using VR. Also the VR had a direct link to the previs tools so it made it much quicker for the previs to be put together. They had references or markers in the model that had camera information, so that every single shot that we photo-boarded, for example, would have been recorded in the previs. You could then recreate the shot within half the time and then continue with the animation. It was really, really effective and it not only saved us a lot of time but was also extremely inspiring.”
I think where there’s pushback from crews is where there’s an assumption that the computer nerds are trying to take over
Images: Previs for The Bells episode of GoT Season 8
New skill sets
Webster from TTF likes to describe the skills of these new visualisers generally as a mixture of creativity and technology. “Most of our guys come from an animation background, rarely do they come specifically from a VFX background as you might assume. They are people who have experience in narrative storytelling, they probably understand more about the filmmaking process coming from editorial backgrounds, and they also understand cinematography. So it’s more about working with directors and understanding their vision, then translating that cinematically and editorially into a basic layout for the film.”
The visualisers can also be brought on to the production in the early discovery phase when plans for the movie or TV series are scarce or not entirely fleshed out. “When they come to us early on in the process they are asking us to help inform the seminal design phase and to really help author some of those crucial decisions,” says Webster. “We have to be malleable enough to really understand what the questions are that we are being asked to help solve, if that makes sense. Sometimes it’s about exploring story with the director to help create the basic framework of a film and other times it’s much more technical. We have a team of supervisors who have replete skill sets but specialise in some specific areas.”
Above: For the inaugural Disney series, The Mandalorian, The Third Floor helped visualise the story and shot structure end-to-end across each episode in previs. “Every scene was mapped out shot-for-shot with sound effects and dialogue,” says visualisation supervisor Chris Williams. “We visualised more than 4000 shots, also helping inform the filming of in-camera shots captured using 4k projections in a large-scale capture volume.”
Non-VFX content
The more you understand the practical help that a company like The Third Floor can offer a production, the more the technology is demystified and seen as a stabilising influence on the machine of making content that can take in any type of narrative; including coming away from VFX. Webster sees TTF as offering these fundamental new services to even independent movies, even though the company has cut its teeth and flourished over the last 15 years in the superhero genre.
“On lower-budget productions we can help solve broader narrative problems and can be more of a support crutch helping the creative team in that way. Just as much as when you are working on a big action sequence where you are helping to design very complicated, sophisticated, choreographed sequences. That’s stimulating in one sense but we can also be deployed to work with a different range of clients; we do a lot of work with games developers and publishers in helping them realise the story arcs within their own cut scenes or cinematics. We also work with independent filmmakers to help shape the structure, and with commercials directors to, in a very short-form way, design a 30-second TV spot that crams a lot of story into a very short time frame.”
Above: Gravity’s previs was the starting point for The Third Floor UK’s operation which is in association with Framestore
Previs pushback
Similar to when digital image technicians (DITs) started helping DOPs wrangle digital data, the new visualisers have experienced resistance from the image capture crowd. “We have been welcomed by many but reviled by others, which is fine,” says Webster. “Some directors that we work with are great adopters of technology, the Robert Zemeckises of this world, Steven Spielbergs, George Lucases. People who recognise that visualisation is an integral part of their process. Other directors feel it’s anathema to them. It’s just not in their own creative process to be spending time in the computer when they want to get on set. They want to make the decisions there and in that context.
“I think where there’s pushback from crews is where there’s an assumption that the computer nerds are trying to take over what they do. Actually, all we’re here to do is help support them in their decision making and to provide a suite of tools and a creative foil with our supervisors that can help them realise some of those creative decisions. Why not experiment before you’re thrust into a high-pressure environment of being on location or being in a studio or in the confines of an inflexible VFX pipeline?”
Prepping the engine
Above: Artists from The Third Floor, headed by Andrew Honacker, stepped into the moors for Disney sequel Maleficent: Mistress of Evil. Working under the direction of Joachim Ronning, the visualisation team created previs, tech-vis and post-vis shots which enabled the exploration of various story beats
Game engines were identified as potential previs engines nearly 20 years ago with their ability to work efficiently with polygons for instant game play. There are a few engines out there but by far the most popular are Unreal and Unity. These are TTF’s playgrounds and where the magic happens: but what’s on offer and what has to be authored? “We do a lot of work with Unreal, we have a great relationship with them that’s been built up over the last few years but especially with our collaborative work on The Mandalorian, the Star Wars TV series, where we have worked with them to help author a new pipeline.
What real-time technology allows us to do is not work faster but potentially create a better-looking product
“In London we’re in constant discussions with them about the work we’re doing, certainly in the areas of virtual production or more specifically interactive previs, and getting a live link between Maya and Unreal flowing. In fact, we’re in conversations with them as we speak about how we can best support each other in some of these areas.
“What real-time technology allows us to do is not work faster, but potentially create a better-looking product. It’s more about the interactivity of it and a process where you can have the key decision makers in one environment acting in a kind of collegiate way to facilitate that decision-making process. I think this process is going to be something of a paradigm shift not just for us but the wider VFX industry.”
Narrative visualisation for the rest of us
The experience of working with companies like The Third Floor is at the moment limited to big movies and their budgets but this type of virtual visualisation is slowly filtering down to episodic TV, commercials and video games. For the majority of network TV drama however, the reality of the ‘script to screen’ journey is less about technology and more to do with the personalities of the heads of departments, the showrunner and the writer – sometimes the last two being the same person. That collaboration should be able to raise the potential of the script to a much higher level and push it nearer to production.
Visualisation of how the narrative plays out on screen is, however, fundamental in the writer’s room and can evolve into subconscious shot selection the nearer the script gets to the final draft. We spoke to writer Kate Brooke who has written the Bancroft detective series – Series 2 has just aired on ITV.
Brooke currently also has two new pilot scripts commissioned by US networks and her other UK TV credits include A Discovery of Witches, Mr Selfridge, Ice Cream Girls and The Making of a Lady amongst many others.
Above: Bancroft season 2 has just aired on ITV Drama in the UK with Kate Brooke as showrunner
Writing freedom
We asked Brooke whether she visualises the production as she writes the script. “Because I’m a showrunner and I’ve EP’d a lot of stuff I’m very aware of production constraints. But I don’t think you want them to be in your head at the top, I think you want to give yourself a pretty broad canvas and then gradually work inwards, really. I think if you start worrying too early about things like too many night shots, for instance, you end up somewhere that’s not where you want to be to start with.
“There is a journey from that first draft into production and there are compromises to make and people to listen to. There are often solutions when you get into production that you don’t expect to be there. Really for me I enjoy the writing bit, because you can just have total freedom, which you should as much as possible. When you get into production that sort of changes a little bit and it’s all about listening.
“I started off in theatre, I was a theatre director, so I have a quite clear sense of production. I like the production bit where quite a lot of writers don’t. I love it when you begin to see the locations and the costumes, but it’s about coming up with creative solutions with your heads of departments. I personally think that writers have to be part of that equation and in the UK we don’t let that happen enough. I’m very lucky that I have done a lot of shows, I did Mr Selfridge for three seasons and A Discovery of Witches. I was really embedded in those productions, and when you have those production constraints you’re much more likely to come up with good collaborative results when you have the writer in those discussions, and even better leading those discussions with the HoDs and director.
“So many British productions don’t have the writer there which is totally different in America. There, if you’re showrunning you’re absolutely embedded in it, you’re making all those decisions. Your producers work for you, your line producer works for you, everyone works for you. Constraints can be a good thing as we all know but the writer has to be in that conversation and I think over here a lot of the time they aren’t.”
Shot decisions
We wanted to know when production constraints might impact on the writing of scripts and Brooke acknowledged that as those constraints start coming down the line it might affect the route of the narrative. “In the first draft of a scene for instance it might feel a bit soggy, then you might think ‘how will this be shot?’. Then it does help, it’s an extra tool in the box being able to say, ‘what’s a sexy way to shoot this?’.
Above: Writer and showrunner Kate Brooke
Brooke was all over the production side of both Bancroft seasons and as she would say, that’s the way it should be. “I don’t go on set unless I feel I have to; I don’t actually find myself particularly useful on set. I had a very good relationship with John Hayes who was director of Bancroft Season 1, and I think I was even more of a creative force on Season 2. Once it’s your own as you created it, people do turn to you, saying things like ‘what colour dress would she be wearing?’, that kind of thing. For example, The Discovery of Witches, which was an adaptation, and Bancroft which was self-created, in terms of production what was fascinating was there was nobody else to turn to. Production does look to you, HoDs do say ‘you’ll know’, as it was in your head.”
Storyboards
As her shows near production Brooke does get to see storyboards; this is as near a previs as you’ll get. “I very much see storyboards, we have tone meetings, I think these are essential. It does depend on the director and some storyboard more than others. But you have lots of conversations with directors – Lawrence Gough who directed Season 2 is a very visual director who brought so many solutions to the table, we collaborated very well together and he added a huge visual element to the show which I hadn’t maybe realised. This relationship can be so positive, but you have be allowed access to each other.
“But because it’s my show and I’m relatively high status as a writer, I’m lucky enough to have directors come and listen to me; but I would hope that I listen back as well. They’ll turn to me often about character, they might not understand why a scene is a happening in this way, for example. At the end of the day it’s about time, you must put in the time to have those conversations. I have been in the position when someone is about to shoot something and I’ve said, ‘What are they doing?’ You can’t quite believe that somehow they’ve got to that stage.”
