Avatar: The Way Of Water VFX Supervisors On Expanding Pandora And High Frame Rate [Exclusive Interview]

Director James Cameron has the pioneering spirit of an explorer. That spirit has been on full display throughout his career; each of his movies attempt to reach higher, climb faster, and break new ground in building worlds with which to tell emotionally resonant stories.

In the case of "Avatar" and its new sequel, "Avatar: The Way of Water," Cameron's mandate was to almost literally build a new world: a moon known as Pandora, which is the home to the Na'vi. The original 2009 film achieved that task, and the bar was set incredibly high for "The Way of Water" and its forthcoming sequels.

Fortunately, the visual effects teams are more than up to the challenge. Leading the charge are Weta FX's senior VFX supervisor, Joe Letteri, and Lightstorm Entertainment's VFX supervisor, Richie Baneham. Between the two of them, they've won six Academy Awards, and their resumés include some of the biggest and most groundbreaking effects films of all time.

I was fortunate to get the opportunity to speak to Letteri and Baneham recently, and they revealed quite a bit about their approach to "The Way of Water," the new techniques developed for the film, and just how they're going about developing and enriching the world of Pandora.

Note: This interview has been lightly edited for clarity and brevity.

One of the things I was struck by was watching the re-release of the first film a few months ago and watching "The Way of Water" the other night. It feels like they were made back to back, especially since I hadn't seen the first "Avatar" since it came out —

Baneham: Nobody had in theaters. It's a theater experience.

It is. So when you were making "Way of Water," were you attempting to try and match up both films as close as you could? How much of it was, "Oh, we're making a sequel, let's try and keep these things tied together as much as we can"?

Letteri: That was a big part of it. We referenced the designs for Jake and Neytiri, for example. We pulled out the old models, but we updated them both for detail but also for aging, because story-wise they're 15 years later. But we wanted to capture the spirit and the look and feel of what we had, but bring to it what we could now, but not do something completely different, right? Because we're in the same place. We didn't want to break that.

Baneham: And thematically as well, the movies are in a lot of ways a continuation. There is that break, but again, we want that continuity. We want the audience to go back to a familiar place, and the structure of the movie is we take you back to where we left off. Then obviously through events, we take you to a new place again. So had we jumped in when the Sullys had moved on, that would've been a much more abrupt change, but to just have the continuity over the two movies and really bring it back in that soft, subtle way into the jungle — and again, we open with a lot of beauty and, of course, it changes as it goes.

It's rare to have a science-fiction/fantasy series that has a sequel take place just a little bit further down the map instead of a whole different realm or planet. Was such a mandate harder to follow in that way, in terms of these environments being just around the corner rather than light years away?

Baneham: I think there's a lot of terrestrial references and we talked about it, which is that if you look at a movie that's made here on Earth, you can literally be in Iceland, and that have a very specific look and feel, and then be tropical by Hawaii. That is not that far really, geographically, so, similarly, that's the way we approached it.

Letteri: You know what was difficult, though — that question came about, "Would you change anything?" The scenes we were doing early on that had the full jungle, we are much more realistic now with our rendering and our tools and everything than we could be on the first film. But if you do a realistic rendering of a jungle, it's kinda boring. It's not Pandora. So, we had to actually figure out how to recapture that feeling that we had of being in the Pandoran jungle, but pushing these new tools into that, because the first ones were designed to give us exactly that. So it was kind of interesting. We spent a lot more time on that jungle than I thought we were going to, because of trying to capture the feel. That was more important than being a hundred percent realistic.

Baneham: Echoing the first movie is a healthy and hardy way to bring your audience back in. I think feeling like you come back to that same jungle is important.

Letteri: And actually for the artists. Because it grounds everything. It's like, "Okay, now we know the world we're in, and we take it from there."

Are you guys constantly looking at real-world references for the creatures, the people...?

Baneham: Everything has a terrestrial reference. Because we need to build things up from the inside out — not just creatures, not just from a kinematic and physical structure, but literally down to each and every thing you see. Joe will attest to light bounces and transmission are big parts of making something feel photographic.

Joe, did you start on "The Abyss?"

Letteri: No. That's a bit of a misnomer because Jim, when he did the director's cut, I had joined ILM afterwards, and I did a couple of shots for him on the director's cut, but I was not there for the original.

Still, I was so reminded of "The Abyss" during the sequence where the Sullys have to escape from the interior of the sinking Sea Dragon. Was that a deliberate reference? Were there other references to other films that you used in the movie at all?

Letteri: I think we had that in the back of our minds, but because Jim was so clear about that one, we kinda just knew where to go with it.

Baneham: And the third act structure and the underwater scene inside the Sea Dragon, as the Sea Dragon turns over, there's a lot of constructs to how that's told. Again, Jim, not just as a director but as an editor, I think along with Steve [Rivkin] and the editorial team, really do a great job of setting that as a clock that's ticking. I think that's reminiscent as well of "The Abyss," which is there's this breathiness to it that makes you think you really go on a journey. And I think there's a lot of aspects there about "The Abyss," I think that hold true structurally, that are reflected in the third act here.

That third act has some of the best use of the varying frame rate in the film, for me. How much of that frame rate manipulation were you guys thinking of when you were creating the effects?

Letteri: Well, the live-action [footage] was all shot at 48 [frames per second], so you had that bed to work with. But whether or not a shot went into the release as 48 really depended on the action. Where 48 serves the best purpose is if you've got fast-moving action in stereo [3D]. Because your eyes in the real world ... motion blur happens outside your field of view. Your eyes track a fast-moving object, so your eyes don't see motion blur because they're not paying attention to anything that you're not focused on. And so stereo gives you that same effect, and we use 48 to help get you that focused look to guide you where the action is. But if you've got a dialogue scene where that's not important, you want to let it breathe, you drop back to 24.

Baneham: And it's not necessarily for us about, even from inception, identifying the shots immediately on first run. It's about identifying a rule set. Like Joe said, when things move quickly, when 48 services the shot. And ultimately, it's a tool. It's another tool in our belt.

Letteri: We also used it for the underwater, the scene where you first go underwater, because the other thing it does is it gives your eye a little bit more detail to land on when you roam.

Baneham: A hyper focus.

Letteri: Yeah, exactly.

It doesn't feel plastered over as an effect, it feels deliberate. As one would use a camera lens, or something —

Baneham: Exactly. You're arming filmmakers with tools. Again, if technology doesn't empower art, it's not useful in this form.

If James Cameron came to me tomorrow and asked, "Hey, could you do the effects for me on the next one?" I think I'd faint and die. It would be so overwhelming. How do you as supervisors approach even just something as basic as organization and communication?

Letteri: I think we're very lucky because we've all been working together for so long. Even before Lightstorm, Richie and I worked together back on "[The Lord of the] Rings."

Baneham: We started the same week on "Lord."

Letteri: Exactly. And all of our team has been such long term with all of us. We were working with the team at Lightstorm, who's been there since the first film, a good chunk of them. And we've had this collaboration going on that communication is like ... you'll see we're pretty non-formal with communication. A lot of producers will balk at that. But really, I just have a lot of faith that if I talk to somebody like Wayne [Stables] or Pavani [Bodapatti], "This is what we need," they know what to do.

Baneham: Yeah. It's a shorthand that you build up by knowing each other for a very long time. And we have a hierarchical structure that's very, very sharp and flat, which is a strange thing. Jim talks to us and we try to allow him to talk directly to the artists whenever we can. And that empowers it right there. And then for myself and Joe and the crew, it's pretty much a flat structure. You're able to talk to anybody anytime. So it's great. It's not going through multiple layers of communication. You need to be able to get your hands dirty, get in and work shots. I've seen Joe do it at desks and I'm doing it.

Letteri: Absolutely.

Baneham: It's an important thing. And I think it brings a continuity to the show if everybody feels like they're aiming at one target.

Letteri: I think facial's probably the best example of that, right?

Baneham: Yep.

Letteri: Trying to develop a language to express what an emotion is, [what] it consists of. When you watch a performance and try to understand, "Okay, this is what we think the actor is doing. Here's what we need to be doing to convey that." You have to develop a language for it. And that language has evolved over the years. We started with Gollum doing facial and talking about it and trying to understand what motivates what. What reaction comes from what things you're seeing on the face. And yeah, we can talk about that pretty much all day long now, and we do, when you're going through these shots, because there's a lot of detail work to make that happen. But it's that vocabulary that —

Baneham: The evolution of that rig, that we literally go back to — I don't even know what year it was, but the first –

Letteri: 2002.

Baneham: Yeah, 2002. So on "Two Towers," evolving the facial rig for Gollum, that lived through the Weta pipeline in principal ideas all the way through "Alita." And this time, it was a true tear-down. Again, the team at Weta have evolved [what is] I think the most sophisticated, the most realistic replication of the human physiognomy from the inside out. I think that gives us a huge advantage to be able to emote outwardly and to our audience.

I love that you're speaking about that, going all the way back to Gollum. Because I have an acting background, I was studying acting at the time that film released, and Gollum was that flashpoint, that watershed moment for actors, realizing that, "Oh, the performer can still come through." And I think Weta's really carried that through, from Alita to the Na'vi and all the other digital—

Baneham: A digital prosthetic.

Letteri: Richie will tell you: When you're doing an animated character, animators will often refer to themselves, because it's the closest reference they have. And we started doing Gollum, it's like — every animator traditionally has a mirror at their desk. And every shot that we looked at for review, you could see who the animator was.

Baneham: We can still tell you which animator –

Letteri: We can still tell which animator did it, exactly. And at some point we had to say, "Stop. We have Andy [Serkis]. Why don't we just use him?"

Baneham: I can even define the moment, which is the soliloquy sequence where Andy is playing two characters. We're like, "Ah, we got it down to a small group of people." I'm like, "I don't want any of Mike. I don't want any of Atsushi. Get Andy." And we were able to [pull it off].

But I do think that's a building block for all the way to today, which is value the performances the actors give you, and protect them. It's [the animator's] job to shepherd them to screen. Again, not unlike the editorial process. They have to try to bring everything the actor has given us to screen in a cohesive performance. It's exactly the same thing for us.

Is it the same pipeline for every shot in the film, basically? Or does it vary shot by shot?

Baneham: There's a fundamental idea behind most shots. They fall into different structures. But again, and this is for us, and we have to evolve it to a point where we hand it off to Weta. The first time around, we actually rebuilt the shots at Weta. This time we were able to hand off a consolidated file that lived in the Weta pipeline, which is a massive step up and a huge help for us to be using a Weta-based system on stage, and that allows it to be full circle. We could literally take anything that the animators would have done, or if there were updates to shaders or costume that were done where we could incorporate it back on the stage where we were shooting live, which I don't believe any other movie has that advantage.

Letteri: "Pipeline" is a bit of a generic term. It basically means how you're going to flow the work through the different people that need to touch it. It kind of breaks down which departments are going to handle which things. But it's unique to each studio. Because we base our departments and our tools around the key artists that are doing the work, we're set up differently than another studio might be, or a different way of doing it. Again, characters being a good example. Traditional animation, a lot of times you'll have a lead animator on each character. We don't work that way. We look at everything as a scene and expect everyone working on the scene to understand all the aspects that are going into it. So the animation team is responsible for the characters in their scene, no matter who those characters are.

And Dan [Barrett], Richie, and myself will oversee that, to make sure we've got everything going the way we want, it's working, it's consistent. But they need to know those characters. And it doesn't matter who the characters are. That works for us, because as we go across the board, for example, the animation team needs to work with the effects team.

It's very easy to do something in animation where you've got a really cool move and something's moving really fast. But then you realize when you go to simulate it that, "Okay, this character's arm is moving at hypersonic speeds and the water's just exploding." It's never going to look correct when we try to do that. So they have to go back to the animation team and talk it through and figure out how far you can push it. So that's why we do everything as a sequence, shot-based approach. So you're looking at the story really that you're trying to tell as opposed to just the effect. The artists are responsible for the things that they're experts at, but they also have to be responsible for making the whole thing work.

It feels to me from an outsider's perspective that there's a push-pull in effects, where there's the pioneer idea of, "Oh, we've got to do this thing that no one's ever been able to do before" versus this traditional filmmaking mindset of "Let's be as invisible as possible, let's just tell the story." Is that something you're dealing with on a daily basis?

Baneham: I can't speak for Joe, but we would never want to break new ground for the sake of breaking new ground. It should be a part of the storytelling. If the story demands it, then you need to do it. Again, that's layered. You don't want to be novelty-based. Spectacle for spectacle's sake has no real narrative value. For us, it has to be driven from the narrative of "Does this further the story?" And that's it. And with Jim at the helm, you tend to have no choice but to break new ground!

Letteri: Yeah, exactly. We're fortunate that way as we get to do both, because we have the story that supports it. Look, Jim, throughout the whole history of visual effects, has known exactly where to push. Right? Starting with "Terminator," but then "Terminator 2," it's like, "Okay, we can do a chrome shader on a guy that can change shape." And it's like, "There's the story. Make that work." Every step of the way, he knew what to do.

I remember getting the call on the first "Avatar" as we were finishing "King Kong." Jon [Landau] called me and we had about six weeks left to go, and you know, "How's it going?" It's like, "Good. We're in the usual crunch time trying to get it done." He goes, "Is everything kind of working?" Because we had met before we started with the characters and the jungle and all that. And I said, "Yeah, I think we got it going pretty well." He goes, "Good. Jim has this script he wants you to read." It's like, "Okay!" And that's it. It's like the right place at the right time and knowing how to make it work. But Jim's also very collaborative that way. We talk about, "Here's what we can do, here's what we know now that we didn't know before, and here's what we can do." And he'll say, "Okay, here's what I'm thinking we need to do." And we'll just toss it together and put it together.

Baneham: Yeah, it's a value spend. It's not the least effort, but the most effort for the best result. Again, and it's never trying to solve a single problem for the sake of solving it. It's about furthering the art form. If we can get wind in our sails in certain scenes so that we can spend that time and energy somewhere else, we'll take it. Because not everything has to be groundbreaking. It just needs to sing on screen and connect with the audience.

Since y'all know a little bit about what's coming for the future "Avatar" adventures, can you maybe give us a hint of what you're excited to tackle next?

Baneham: It's a tricky one. We're headlong into ["Avatar"] 3 and we have plenty challenges ahead of us, but I'm excited to see where we go as far as the story is concerned. But I think from a technical standpoint, we have a few things on the broiler that are waiting to be solved.

Letteri: I mean, that's the thing about this whole process. It took a long time, because as Jon was saying, Jim spent a long time working out these stories and we sat down and read 2, 3, 4 and 5, and the stories just work. They just sing. So yeah, we're just ready to do the next one.

"Avatar: The Way of Water" is in theaters now.