Crafting Immersive Audio in Cyberpunk 2077 and Next-Gen Open Worlds with Colin Walder CD PROJEKT RED

The very worst case for streaming in Cyberpunk is you're driving and having a conversation at the same time. Because it means you're driving through the world. It means that you're you're streaming the textures in, but worse than the textures not streaming in, if the physics doesn't stream in, you'll fall through the world. If the gameplay doesn't stream in, then anything can start breaking.

And streaming is is one of those those cases where when it goes wrong, it just falls off a cliff and anything can happen at that point. Right. It could be t posers, it could be bug faces, it could be the game crashing, it could be getting stuck in some infinite loop or like, who knows? Like, all bets are off if this goes too much into pot.

Welcome to in development, where we explore the technical realities of creating today's most ambitious digital experiences through conversations with the problem solvers who build, maintain, and optimize the tools, pipelines, and workflows behind games, animation, VFX, real time visualization, and more. Each episode features in-depth discussions with technical directors, pipeline specialists, and digital creators who share their approaches to overcoming real life production challenges. Not just what they accomplished, but what went wrong along the way and how they approached solving those problems.

My favorite part of this job is getting to learn from so many amazing creators and to share my own knowledge with others. I learned so much from my guests on every episode and I hope that you do too. And with that, I am so excited to get into this episode with Colin Walder from CD Projekt RED. I met Colin when he was a panelist on a panel discussion at the game developer conference twenty twenty five that I moderated, and he just had so many great stories and so much passion for what he does at CD Projekt RED from his background in audio to working on pipeline tools for the whole dev services division within CD Projekt Red.

I'm sure many of us have seen the tech demos and trailers for The Witcher four, which should be coming out soon. They've been working with Epic Games to add features into Unreal Engine specifically for The Witcher four, and then sharing that so that others can use them as well. And over the years, they've had to build a lot of special tools to overcome the challenges that come with creating such large open world games like what CD Projekt RED is known for.

In today's discussion, we're getting into managing assets for audio as well as performance and streaming when it comes to games. I think that there's a lot here for everybody and especially if you're interested in the audio behind some of the games, definitely stick around for this episode. And with that, I'm excited to get into the interview with Colin.

So, Colin, thank you so much for joining me today.

Hey. My pleasure.

Can you tell us a little bit about what it is that you do at CD Projekt RED?

And also maybe what you did at the time when we're talking about this, because I know your your position's changed a little bit.

Yeah. Yeah. Yeah. Exactly. Currently, I'm, one of the engineering directors at CD Projekt, and I'm I'm responsible for what's called shared dev services. So this is a few of the tech teams who are working across multiple games, and that includes the audio, the localization, the DevOps, and the asset pipelines.

And and we wanna build some tech that's that's gonna be foundation for all of our games, but my my my history is most strongly in the audio aspect. So this is this is where I I started in the industry as an audio programmer back in two thousand and six. And then, yeah, through my entire career, I've been working on on audio programming, so I still have a a lot of love for that. And and even I do a bit of composing on the side now as my my side projects.

My original interest was composing as well. Like, that was my degree. And then I kinda shifted from there into visual effects and programming.

So I think I think maybe we share a love for audio stuff.

Absolutely. I think you have even the more musical room than me. I mean, I've got I've got I've got a couple of instruments lying around, but yours is is pretty impressive.

Thanks. So to start out, I'm curious, what is the toolset that you use? And and I guess for that question, maybe focus more on the audio side because I'm curious, what is a modern day toolset for an audio department on a game?

Sure. So it's it's a it's quite a diverse group that we have. We think about the whole pipeline here for audio. So from the from the point that we recorded to to getting in, it's gonna involve a whole chain of people.

So on on the one hand, we have our our content creators. So this will be our our composers and our sound designers. And actually, it can be quite varied. So they will do everything from recording the sounds to then designing them.

Yeah. So this is one of the recorders, my personal recorder mix pre six that I used to record some of the ambisonics. So I borrowed a a sound field mic microphone from Sennheiser on one of my many trips to Japan and recorded a a whole a whole bunch of sounds that we then processed and put into to the game. So this will be the first stop here, something like this with various different recorders, maybe a a Sony PCM, which I don't know.

I have I have that line around somewhere. And then we'll take those recordings to the sound design into software called Reaper. So this is very popular digital audio workstation in in games. It's quite well suited for the kind of challenges that we face.

So when you're working many different files, making many different versions of those of assets and you need to export them, like do batch exports and custom names, it's very handy for those things as well as being quite flexible to be adapted to a lot of our sound designers. They're creating their own scripts, their own processes for how to speed up the workflows. So they'll design the sounds in there, and then we take the sounds into a software that's called Wise or Wise actually. Usually, pronounce it Wise for short, but full name is Wise.

And this is a software that we use to orchestrate the interactive flow of of the audio. So you can take those recordings, you can take those sounds, and it still has some similarities with, let's say, the digital audio workstation paradigm, but it lets you arrange those into containers and say, okay, when this event happens, I want to, for example, trigger a random sound, or I want to trigger a sequence of sounds, or I want to blend between some sounds based on some parameters. So it's really the point where we can start to tie in some idea of interactivity that's coming from the game with those audio assets.

For the composer, similarly, actually, will their stuff will end up in Wise, so there's an interactive music system in Wise where they can design. Okay. When this happens in the game, when some trigger happens, transition from this piece of music to this piece of music, or maybe blend some stems. The composers mostly are working in in Logic.

So this is a Mac based digital audio workstation, which has some great features for composing especially. So this is they will both come from either Reaper or Logic, bringing the assets into Wise, and then this is where we create the the interactive bed.

And then from Wise, this is where it starts to enter our tool chain, our pipeline for having it in the game. So we build what we call called sound banks. This is the output of Wise, the thing that we can load in the game that says, okay, here's the wave data, here are the the audio assets, plus some data to say how they need to be played. And then this is where it starts to transition a little bit into what we call technical sound design or or also onto the code side, onto the audio programming side.

So once we have the assets from Wise, they will need to be implemented in the game. So we have to hook up these things when something happens in the game. Someone shoots a gun or a footstep happens or a scene starts, then there needs to be some connection that says, this is the point in the game, trigger this event or set this parameter in Wise that will control the behavior. It might be done by the sound designers.

It might be done by the audio coders, or it might be done by a technical sound designer. So actually, this is fairly new role for us, but you see sometimes in different places in in the industry, someone who's mostly focused on this kind of interaction between the middleware, wise side, and the engine. And so the tools that we're looking at with Witcher four and Cyberpunk two and the following games, this is Unreal Engine. It's the editor, the pipeline where these things will be hooked up.

Previously, where we think about Cyberpunk, think think about Witcher three, this is Red Engine four, Red Engine three, respectively, which have some similar kind of tools and pipelines. Yeah.

There's an editor, and in that editor, you Right.

Maybe have some timelines that you can tag things, or on the entity, you can set different properties. And of course, this whole this whole section now, we're now touching the editor, now starting to look at the game, this really is also the domain of where we work from the audio programming side. So we will be using a lot of Visual Studio, sometimes Rider, to work in c plus plus and doing things as varied as, okay, we need to build the pipeline that will import those sound banks. We're doing the work that will do the streaming at runtime, but also building workflows and and tools for the sound designers so they can set things up in in the editor or or sometimes creating procedural systems.

So for example, in the Witcher four tech demo, which we showed at Unreal Fest, this is using procedural footsteps. So all of the footsteps, think, apart from the the Manticore, which is like a custom scene, but all of the rest of footsteps, including the Kelpie, the horse, and all of the NPCs and the Siri, these are being procedurally driven. So this is taking a huge weight off the sound designers where previously they would need to go through the timeline on the animation and say, footstep here, footstep here, footstep here. It's it's not creative work.

Not not very creative work at least Right. Most of the time. Maybe except except in the scenes, but when we think about gameplay animation or the vast majority is not creative work, it's just time consuming and vulnerable to, okay, the animation's got changed. We now need to go through and retag anything.

Right. So so it was a huge pain point on previous games, especially Witcher three. Cyberpunk, we did have some, like, let's say partial automation, but we have on Witcher four the much more thorough procedural footsteps or building some procedural systems in in the ambiance as well. We're working with the sound designers, and we do work very closely with them.

So every sprint, some of the sound designers who are working most closely with the technology that we're building in this three weeks, they will actually come and join the co team. So they will sit in on the meetings. They will take part. They will be giving feedback.

They will be having this very close loop of working with them to make sure that we're we're delivering the technology that we really will help them there.

Yeah. That that's something I was curious about is kinda how that handoff goes. So it sounds like for you having them be very closely involved, how much does it go from a creative sense of we want these sorts of sounds, let's go get those, put them in Wwise, package them up, and then now we need to program hooks for them? And how often is it the other side where it's like, hey, you know what?

This thing happens in the game. It'd be great to hook that to a sound. Can you get that? Like, how much does it go one way or the other?

Yeah. Of course, both parts are happening. The majority for audio often is something that is driven by the rest of the game. So we often thought of what's called like a post team, meaning that the game is there and then we will be checking the game. We'll do maybe what we call spotting sessions, which is where you look at the the game or the designs and you say, okay, we think it will need sounds for this, this and this. So a great example of this would be routine stuff.

If we know that the player can wear different types of clothes, then we will want to design sounds for those different types of clothes. If there's a scene and something is happening in a scene, then we'll say, okay, what's happening in the scene? Because we will need to put some some stuff in there. Or we're going into combat.

Okay. We need to change the mood because it's going to combat. So so there is a lot of, let's say, the workflows of the the folks that are are about responding to what's there in the game and flushing up. That said, there is also this other side of audio is something that's very good for delivering emotional context, emotional instruction to the players, especially music, but also the sound design.

So the context of the game, the larger story, the kind of atmosphere that we want to build, what things can we do on our side, what kind of mixing effects can we apply or especially with maybe with the music, how can we orchestrate this, obviously, orchestrating both in terms of the composers are gonna write the music and record it with an orchestra as as much as the Like a literal orchestration. Yeah. A literal orchestration.

Okay.

But also in terms of, okay, Now, we have all of these interactive sections of music, how will we play them back in such a way that will create this narrative? What can we do to create some organic arcs when we have to toss some narrative, even in those Let's say seem seemingly random, desperate moments. How can we start to tie everything together? Some of it would would be like in the ambient sounds, we will often be putting sounds into the world which they fit the mood, they fit the atmosphere, but there's not something perhaps specific that they're responding to. There's not something that that's like definitely says, now you have to play here. And maybe a good comparison of this would be, if we think about the sound of adverts in cyberpunk, we have TVs throughout the world, and the TVs have adverts playing on them. And so when the adverts play, they need to play sound.

You there. Yes. I'm talking to you. Are you getting what you want?

This is obviously the the clear case where the audio is gonna fit the visuals. Yeah. It's gonna be what we might call diegetic in the world sound. We can then have on the other side, maybe in the music some themes, and this is score music, and this would we would call non diegetic.

So this is purely emotional sound. You could have some sound effects sometimes used like this, but a lot of this will be music that is used to influence the the emotions of the player and very this is very cinematic approach. It's like, instead of what we use to this kind of sound all of the time. Right.

And and then there's this this this ground in the middle where you have the sounds of adverts, but you have them where there's no TVs. So when you're walking through the streets of Night City, you actually often hear these adverts coming, bouncing off the walls. And this is part of creating the atmosphere, part of creating vibe. But actually, this is just the sound designers have gone through, and they've used some technology that we've created to to make it easy to have this, like, dense feeling of adverts.

But it's not that there's specific adverts that are being played. It's these have been crafted to create this emotion. And maybe in some districts, it's gonna play the sounds that are more glitchy because this is a more rundown district. And maybe in a different way, it'll be playing adverts for more high end gear and you'll have the more polished sound.

And this is something the sound designers are are just doing. Right? They're just putting into here and saying, okay, we understand what the emotion, the environmental narrative is in this area, and then what can we do to put this in? When we are creating, like, crowds of Waller Sands, so Waller is a a sound of people talking where you can't listen.

Can't actually sit tell what they're saying. Yeah.

And and the name sounds funny. And actually, it's it's it's interesting because it's different in different languages. I think in Japanese, it's like, gabba gabba gabba.

And Okay.

It comes from the film side of audio, actually. It was when they wanted this kind of sound right inside of a crowd, they just had people saying, wallow wallow wallow wallow wallow wallow wallow And and when you have all of those people talking over, it's all mergers together and you can't make the distinct voices. And so when we're doing this Yeah. This is not synchronized to the actual NPCs or the characters.

And usually, we will have the sound of more people, more characters than we have actual people. You know, we're not we're not like counting the number of people and saying exactly this. We do count the number of people, of NPCs, because we generally want to increase it. So if we all if we have, like, a bunch of NPCs over there, then we will increase it more.

But probably we'll use the sound to give this feeling of it being busier than it actually is.

Right.

There's more that you can't see that are Yeah.

Exactly. All of these things. All of the games is like smoke and mirrors. Yeah. Like, this is like the most effective thing you can do is understand what you're presenting to the player and then be very clever about how you're doing it.

But we're not trying to simulate a world. It's not it's not actually the matrix. It's not the the the Right. Creating.

Yeah?

It's a Just the illusion of it.

Yeah. The just the illusion. It is about creating those emotions and and portraying that. And I think this this is very interesting for us as programmers as well, because we're working very closely with the sound team.

So so typically, what we see from working in audio programming definitely, but I I think in all game programming where you have this dynamic, yes, you're working on technology, but you also have this creative aspect in mind. You're also trying to think how is my technology serving the creative goals of the projects. On on this hazy line, maybe even between where the where the creativity ends and the techno technology begins, it becomes intertwined. Games is the, like, the collision, the meeting point of technology and art.

And, yeah, for me, that's, like, one of the really interesting things about it. Yeah. One of the interesting place to place to spend time.

Yeah. Yeah. Absolutely. So to go back to the tool set a little bit, just to kind of paint this full picture of like how many different pieces of software there are all involved in day to day operations.

Yeah. Yeah. So we can also talk about how we manage assets, how we choreograph all of our builds. So we we have our own build system that we've created, and now this incorporates also Hoard, at least for the the Unreal projects.

Oh, so you have actually integrated it with Hoard specifically?

Yeah. We use Hoard for the UBA. So the UBA is really, really powerful build distribution tool. We've had really great experience with it.

It's very fast, very solid. And this has also gone hand in hand with us moving into the cloud for our builds. We have our build split now between on prem and cloud to get this acceleration. And we're taking a lot of care to try try and not not spend too much money in the cloud because I spoke with many Yeah.

That's always the challenge.

Many many developers who had some experiences with that.

But so so far, it's been pretty good. I think our DevOps team and our IT team have been really smart about the way that they've implemented it, the way to use those cloud resources, and we do see the the good acceleration coming from there. And so this is our own CICD solution that we built for Cyberpunk and now brought across to to our new projects. And this includes some tools that, like, gather together various different tools.

So if someone wants a build, we have a a tool called Toolkit, which is a place you can see the status of the builds. You can grab the packages. You can maybe install it to a console. You can get a save from our SaveDB, which is another tool that we built, so a bit space for our saves.

As people are playing through the game, this is logging off these saves. So you can say, okay. I want a save at this part in the quest with these parameters like like, I don't know, female v, this quest with this build, what saves on there that exist, and it'll just we'll find out from all the people who are playing, okay, here's the save, and then you can say, okay, I want to launch this save on the console. You can automate a lot of this.

So I think having this one tool something actually iterated over years. We there was a tool called RedLauncher, I think, back in the day. And then there was another thing called RedLauncher, which is actually the thing that our players have when they launch their games on the on PC.

Oh, right. Right. Okay.

Yeah. Yeah. We were was somehow similar. Right? It was a way that you could launch builds or or maybe DCZ tools and stuff from single place.

Have some single configuration, which is important if we have a lot of a lot of people using like Maya or or Motion Builder. And if they all have like different settings or different setups or different versions, then this can become really hard to manage. You can, something something's not working.

Why is it not working? Did did they do every step that was listed on some obscure Confluence somewhere? Yeah. Maybe not.

It's hard to do. So we have one place that can standardize that and make it easy for people to access this or to or to run the game or grab different builds. This is another one of those tools that we built up there. And then, of course, we we're using Perforce for our asset storage.

We also use git actually, not for the game code, but for our our DevOps codes or asset asset pipelines.

Oh, okay. Like your internal tools.

Yeah. Some of the internal tools. When we think about a game code and and game assets, like places where we have binary stuff, then there's Perforce behind that. Although sometimes now people have less and less exposure directly to the Perforce interface because some of that can be done from within Unreal Engine or Wyze has its own integrations, so you don't have to go to the tool to do things or we have our or maybe our own tools where we have a CL tracker tool which can be used to submit over and test or track things.

Kind of like your own version of Unreal game sync?

Yeah. I actually the version that we have on because because again, we have these different projects here. So the stuff that's running on the back end of the CP is quite different from what's running on Unreal, but the front end is the same. We have this one tool that can serve both of these.

For Unreal, this is using the the back end of UGS, of Unreal Game Sync. It's just presenting it Yeah. As part of this one tool. Try and unify things and tidy things up so that there's not so many places that people have to to look to do the thing that they want to do.

I think this is a worthwhile thing in terms of streamlining those workflows.

Yeah. I feel like since we're all using so many different tools now, like anything we can do pipeline wise to minimize that, to keep it from being yet another five tools you have to learn. Submit this here. Go over here to update someone.

Put it on network storage. Update it in a different database. All of that just to streamline. Yeah.

Make it less less human error, think, and also more enjoyable for humans to work with.

Yeah. Exactly. It's a little bit similar to, you know, what we were doing with the procedural footsteps. Obviously, very different problems in place, but we really want people to be spending their time being creative and solving technical problems or creative problems, not trying to find the documentation for the specific command line that needs to run this one piece of software, or like update to the right version of this or learn yet another interface. This is not something that's really pushing our our game game forward.

Let's move on to talking about the kind of challenge that you had to overcome when it came to getting all of the audio into cyberpunk. So we talked about this a little bit beforehand, but when you have a world that has so many different things in it, like the TVs with the adverts on them, as well as radios, there's your own personal radios in your car and on your person versus all the ones out in the world. Maybe start by describing what is the the challenge that you ran into with that?

Yeah. So we can break this down into maybe a couple of sides of of the challenge. Because the the first one would be, like, how do we actually create all of this audio? How do we actually author so much? If we think about, for example, like, the voiceovers, we localize into I think I think it was it was like eleven or shortly after launch, it was eleven VO languages.

Each language has eighty, ninety thousand lines now of of dialogue. So it's it's probably probably a million lines of dialogue across all localized, languages.

That's wild.

A huge number of assets. And when we were doing this on Witcher three, this was actually all all of this was handled just by, network drives. So people were copying Right. Copying files manually into folders. And, of course, we had the the expert team of localization, but, you know, there were times where someone copied a language into the wrong folder. And now half of English is Japanese, and someone has to go through because there's no versioning on on there Yeah. On on that.

On the network store.

Yeah. Some hairy situations there. And then on Cyberpunk, our approach to try and improve this was to put a lot of this through Perforce. In some ways, works. Right? It gave us a lot of control over the versions and tracking and automating the pipelines, but we weren't prepared for the volume of the assets. So this is submitted in Perforce now.

There's millions of files.

Yeah. Millions of files and then every version of those millions of files sitting somewhere. And whenever we made a change, sometimes we were making changes that was like, okay, let's let's change the way we normalize the files.

That's all of the files get redone every single Yeah.

That makes sense.

So and not just those million files, but different stages of those files. The way we did it, we had the raw file, the processed file, and then we had the mastered file. Some of those could then have additional processing done by the sound designers. So we had all of these different stages, which was kind of useful for us to access and say, okay, what was the file like at this stage? Because we're sure what's going on there. We won't we'll do a custom version. But it became kind of tangled to do, and yet, it's at the point where you cannot download this folder yet.

If anyone downloads this, this is annihilation for any SSD that's For their hard drive.

Maybe the newer ones are bigger. Yeah. Have a full time SSD.

Yeah. When we talked about this beforehand, I've been on a big kick lately of trying to teach more people about virtual streams in Perforce. It's like, that's a perfect use case there. Yeah.

You could have this this depot, this, you know, mainstream that has terabytes of files on it. But if each individual person only needs maybe the one region that they're working on plus a few shared files, I think that's such a good example of how you can do that. Yeah. Each can pull what they need, and they don't have to have all of that.

Yeah. And we see this paradigm push to work, and there's more people working remote. Even after COVID now, we still have a lot of people work remote and Yeah. And the Internet connections you know, in the office, we have ten ten gig gigs, so it's like downloading anything is is pretty fast.

Okay. You you can make something that's big enough to download slow, but I know I have I have an audio coder working from Italy or or another coder working in Ireland. If you're downloading over the Internet there, some very big package or all of the assets, it can be a lot. So we see this in in Unreal Engine now with the virtual assets can be like a huge win for those people in terms of speeding up the workflows.

Or or sometimes we're having workflows like, Mostly they will work on a computer they have in their room, but there's an option to dial into a shared computer on-site to do some stuff and deploy a build to a console that's also on-site, and then those processes can be like blinding fast by comparison.

I mean, it's a different of hours of Yeah.

Of iteration time difference. Not cumulative, each operation. Yeah. Yeah.

Yeah. When you're at that volume of content for sure.

So this is one side of it, like like dealing with generating that amount of content. Part of it is just like the the pure doggedness and hard work of the sound designers to build this much content of the composers to write and record this much music. I think it helps a lot that we have the the large in house team on both sides. It's it's not that normal, I think, in the industry to have, for example, a team of composers in house. But we do have Right.

Yeah. I don't feel like I see that very often.

We have in house and we also do some outsource as well. But having that that constant team, that means that we can do quite a lot over the lifetime of the game. We can be iterating from the beginning and have the opportunity to generate a lot. The second part is, like you're saying, where we're in the game, how then do we deal with all of these sounds that need to be played, especially in a swear like Night City or almost everything that that can be there can make a sound potentially, and there's only so many so many resources to to play them.

But also, we want to take a kind of I mentioned the word cinematic before, and I think it's really relevant because it's quite mature discipline, in in cinema, in terms of understanding, like, what people can hear and make sense of and how to use this creatively. So if we just play all of the sounds all of the time actually, you won't hear all of the sounds anyway. This is the nature of sound. It has masking.

It will it will do that. Maybe it will even be, like, slightly more messy because we don't have the the full dynamic range of natural sound that we're shipping with, so everything's slightly compressed even even if we're not, like, heavily compressing our assets, we want kind of clean sound there. It's still not the same as you have in the world. So you don't want to play everything.

So it means that you then need to start to choose which things to play and which things not to play. Even with this, we we want generally to have a large amount of sounds, I think. And whenever I talk with audio kinetics and I say, oh, we're we're having some problems with this many sounds, they're like, why why are you doing so many sounds? No one does this many sounds.

Okay. We just Yeah. Go that way, I guess.

Well, you wanna create the whole world.

That's that's kinda CD Projekt's whole thing, right, is creating these big worlds and so Yeah.

All of the world and all of the detail with all of these all of these layers. And and this just start to add up, you know, even if you're doing cinematic combined sounds like the wallow or ambient sounds like a quad. So let's imagine you're like a quad of ambience, and you want the quad so that as the player turns, you have this bed of ambience and it rotates around the You get this sensation. Yeah.

Even though it's not real diegetic things in the world, it's still relevant to Will and he wants to be feels like it's tied to the world. So if we think about it before maybe like Witcher three before we we started working on audio code, and I joined in twenty fifteen as the first audio coder there. So before this, maybe we'd have the a stereo sound for the ambience and then some spots in the world. We a traditional way to do things is, I think, it's quite good.

You can have this cinematic feeling in from the spots. Rather than just the stereo bed being static, actually having it rotate around the player ties in this immersion much more. Great. Gives this cinematic design and interactive connection into that is very powerful.

So, okay. So you have a quad bed, four things. But maybe you want this to be dynamically shifting based on the number of people you have in some direction or objects. Sure.

Two or three of those quad beds. Okay. Now, this this is time to multiply and have some more. And maybe you want some different variations of those.

And so you can see how these these things just for that starts to amplify. You have the wallow, your wind, maybe some traffic noise, maybe some some different kinds of of of where the where the sounds starts to add up, starts to add up. So we needed to handle the impact of really three things if we want to have this kind of dense experience. We must be able to handle the CPU load, the memory, and then the streaming.

So the streaming and the memory go hand in hand. In order to do this, we built some custom systems around the streaming that meant we were able to go from in which the three a case where we we used a traditional approach in the sound bank approach where you you're in some area and you say, let's load in all of the sounds related to this area or we're in this quest. Let's load in all of the sounds to this quest so that they're in memory. We can have them obviously compressed in in memory.

We we don't have the raw PCM. This is too big. But this requires this requires a lot of work from the sound designers. They need to organize their sounds into these blocks so that they can be loaded in.

And then, at runtime, you have to be careful about how you're loading the blocks, because it obviously takes more and more memory the more things you have. And if you have the open world game, then it's very easy to start creating situations where, okay, I have all of the quests open and I'm running between areas and and then do a fast travel and all all of this stuff can start to stack up. So with Cyberpunk, what I think is becoming more popular now, in, for example, in Unreal Audio, just taking the same approach of, okay, we will basically stream everything. We will have everything streaming off the off the disc, which means that the very low memory hit by default, because you will only have in memory really the sounds that are playing.

If you compare this to before, it's a tiny fraction what you would have in memory from sound banks if you're streaming.

However, then you hit this next problem. Right? The streaming impact that you're gonna gonna have. Because we were always streaming large files like music, maybe some ambience. Just having them in memory was too much. But we, of course, have to share this streaming with other systems and with the rest of the game to come in.

And what would you say are the main competitors for that streaming?

Is it textures mostly that you're competing with in audio or So the thing about audio streaming is that when you want it, you need it fast.

Any disruption in the audio is felt very keenly. If you have a cutout or glitch at some point, it's it's very, very obvious. More obvious than a small like, small glitch in audio is very obvious compared with a small glitch in graphics. Of course, it's not that you want to have glitch in graphics, but Yeah.

Typically, we're more comfortable with it probably because we blink. So our whole lives, we're used to having these very short disruptions of the visual process. It's why you can very comfortably have a camera jump between locations and is not not really not really a problem. Even a very small disruption in the audio, however, from the moment we're born, in fact probably even from before we're born, we never have a single discontinuity in audio.

Right? It's like just constant.

There's no Right.

Eyelids on on audio.

Maybe if you go to an accessory deprivation chamber, you can start to have this weird experience and and start to hallucinate because And that's probably why.

Yeah.

Because we've never experienced it.

Our brains just aren't aren't used to having that. So, we have to be very careful about it. That means that when you need audio, you need to have it very quickly. Especially if you're streaming something like like VO. Like, some audio, you could at least have some delay. Right? You could wait a little bit.

If it's ambient audio Imagine music could have a little delay.

Yeah. If it's if it's if it's like about when you're gonna start the ambience or most of the time when you're gonna start the music. The music has to be strongly on the beat, but you can get like quite a chunk of up upfront. But if we think about a VO, a voice, if if it's slightly out of sync with the the lips, you feel it very keenly, very sharp to the sensors.

So from this point of view, we're gonna have high priority on the audio streaming most of the time. And we did make some distinct differences between if we were in the heavy streaming situation. So let's say we're just driving through the world. So this is like the worst case for streaming.

The very worst case for streaming in Cyberpunk is you're driving and having a conversation at the same time. Because it means you're driving through the world, it means that you're you're streaming the textures in, but worse than the textures not streaming in, if the physics doesn't stream in, you'll fall through the world. If the gameplay doesn't stream in, then anything can start breaking. And streaming is is one of those those cases where when it goes wrong, when stuff starts coming late, too late you say, there's there's some tolerance it can have, but after something, it just falls off a cliff and anything can happen at that point.

Right. It could be t posers, it could be blob faces, it could be the game crashing, it could be getting stuck in some infinite loop, or like, who knows? Like, all bets are off if this goes too much into pot. So we had to to to put some work in to a, just to cash things up front, so that things which really needed to be there as soon as possible.

When we're in the cutscenes, we're looking ahead through the cutscene and we already start streaming the voice lines before you get to the voice lines. So as soon as as soon as we see this Right. Scene coming, we're procedurally just start to load those. When the player loads their weapon up, then we we say, okay, any sound that can be triggered from this weapon, let's just pull those pull those in, start to to pull them.

When you get in a car, we're loading all of the collision sounds. It was it was it was quite common during development that you'd be driving along, and because it's just streaming intensive time, it builds up a long queue of things you've streamed. You would crash something.

Crash sound.

Like, sometimes sometimes it would it would would come in.

I would imagine with we'll start with the weapon thing you were talking about. I'd imagine that you're also having to procedurally check for all the weapons around because you wouldn't want to not hear gunfire from nearby.

Yeah. So whenever anyone's equipping the weapon, we can load this in. There's a little more tolerance on the MPC sounds because they're not as diverse as the player sounds.

Right. So we can be a little more tolerant there. But if if someone if especially if someone presses a button, they you're experiencing fire. If an NPC has to wait a fraction of a second before they shoot because waiting for the audio to load in, generally, that's okay for the game play. There's there's there's maybe some Right.

That makes sense.

Specific cases where it's it needs to be already there, but you can do this preload manually. But most of the times, the player needs to have this very responsive thing. There's cases that you have, not in Cyberpunk, because single player, but if you have the multiplayer game, there's a a middle a middle situation that you can have with other players' sound weapon sounds. Because typically, the local player, the client is gonna have the priority and the most important sounds, the biggest sounds.

But actually, other players in this case, if it's competitive especially, it's very important that you hear them. Because if you don't hear a sound and then you die, you're like, woah, woah, woah, what happened? Yeah. I didn't have a chance to dodge.

I didn't know what it was was happening, I didn't have this feeling. So in this case, also, they need to be very acutely measured. So so to help with this, we built a custom streaming system. It was built on top of Wise.

And to the sound designers, it looks like everything is is streamed. And actually, to them, it looks like everything is streamed PCM data, so uncompressed. This was to help us also with the CPU side of it. We had a part of the pipeline after the sound designers had done their their work.

They would take all the files and compress them to Opus, which is a slightly more expensive codec for audio compared with, for example, Vorbis, which which is very common. It's it actually uses partly Vorbis in its compression, but it's very effective for for speech.

And is that you said it's more expensive, but still but still affordable, I guess we could say.

It's it's so I'll I'll get I'll get to this part. I'll get to this. There was a reason that we went there. And a slight, like, hint towards that is that the consoles of this generation, so the the PS five, the Xbox series, they typically will be using Opus for their sounds.

Maybe not all the sounds, but a lot of the sounds. And on Switch as well, actually. Okay. Which actually I just bought my my Switch too, so I could I could play cyberpunk on it.

And and the so the advantage of Opus is very high quality. So if you compare like like to like, Vorbis, Vorbis is very good, but there's an extra clarity even when you compress quite heavily the Opus. It's it's very resistant to losing quality at low bit rates. So you can you can squeeze the files down more and more, which then helps with the memory.

Potentially with the streaming, if you do this one cool trick, I'll mention. Because the thing with the sound files is that have some sound files that are big, like a track of music. This is big enough that we all want to only stream parts of the time. But many of the sound files are gonna be way smaller when compressed, you know, like like a few k.

Maybe maybe like tens of k, something like this.

Right.

And if you stream individually each of those, you you're starting to lose some some of that bandwidth from the streaming because there's there's a minimum size. On SSDs, it's for sure smaller. But on the hard disk, which we shipped to Cyberpunk originally on the on the previous gen Yeah.

PS four and Right.

Xbox one. If you if you seek to the audio portion of the disk, where you have the spinning disk, there's a huge cost to do this.

And and if you go in there and taking like a few k, then coming back Right.

Going back and forth.

Yeah. This sucks. This is no way to optimize the the bandwidth. So what we did was after the sound scientists have done the the stuff, we took the loose files and we packed them together based on the order of the events in Wise.

So depending how the sound designers are structured by whatever organization they thought of, we would group them together and put them into one meg blocks. So we would make one meg, we we call them opus packs. So it's like a one meg ish block of opus data, and then this is what we streamed in. The idea here is that whenever we do a stream for audio, we would pull in not just one file, but we'd pull pull in that file and then approximately one meg of data that surrounds this file.

And because this was very likely the other files from the event, or the files that maybe would be triggered the next time that someone shoots the same weapon, maybe it's gonna choose a different random sound from this thing, we would already just have them cached in memory. And so we put a very large cache, like forty meg cache, just for holding these blocks. So we're constantly streaming them in and out, but we're very often taking from the cache instead of having to stream from the disk. And so this is where we start to put some optimization for the for the streaming, because it means that we can be playing many, many sounds, but if it's something we played recently, maybe it worked like for materials as well, if you walk on a certain material, now you have all of the sounds for that material in memory.

And if you're working on a material, very likely that you're gonna have some other sounds related to that Right. To to to that material and they will just happen to be to be in memory.

What's jumping out to me is that by basing this off of how the sound designers organize their projects, it's almost as if you set up a procedural way of grouping these. So, like, rather than saying, oh, we had to come up with this system and figure out what would be together, you're just like, you know what? People already were doing this. Let's just leverage the organization they already used and then take advantage of that in the game.

Exactly.

That to me is very cool to think about that procedural approach to pipeline In a way.

Yeah. It's it's probably worth saying that this is not this is not the optimal way to load sounds. If you really wanted to be optimum about it, then you would have discrete chunks that were exactly what needs to be loaded. I know, if you have a very deterministic linear game, then you could do it like this where you go, I know exactly what's needed in memory, I will load exactly the things that I need in memory at this time, and this will get you the smallest possible size.

But the the idea is it doesn't have to be perfect, it is good enough to get the job done. And so we want to have that balance of workflow and efficiency that will get in there. And for us, was okay. It was a tight squeeze for sure, especially on the previous gen to get it in.

But if we look if we compare, like, Witcher three with with Cyberpunk, we reduced the amount of audio memory we were using between Witcher three and Cyberpunk, despite the fact that Cyberpunk is a much bigger game. Yeah. It's denser. You know, the the amount of sounds, I think it's probably a doubling amount of dialogue and four times the amount of music and and generally the denser world.

Had so many things going on in this.

More playing at once.

Yeah. More certainly more than most people. So more more materials, more weapons. You have guns, you have vehicles, you have all of these different vehicles, you have all of these things.

So we have, like, the much denser world and use cases, but still we managed to shrink that by, I think it was like thirty megs or something. This is this is like, you know, fifteen percent reduction in in size of memory. And at the same time, this was now including on Cyberpunk the acoustic graph that we added, so the procedural acoustics. It was including the localization, all of the code structures which didn't used to be, includes part of that number.

So memory wise, a huge win for us and memory was very tight to manage to get that get that in. Without even this would have been really, really tough to make happen. But then we get to the CPU part that you mentioned as well. Yeah.

So we have all of this Opus data in memory, and we have the calls from Wyse. The sound designers in Wyse set it up to be PCM. Wyse is asking for PCM data. Now, normally, you would give it the compressed data and WISE would handle the compression.

But what we did was that we did the compression ourselves after WISE had done his part of the pipeline, and then we did the decoding ourselves of the Opus. And what this meant was that instead of doing the decoding in line on the the normal Wyze job, which is very time sensitive here, as I mentioned, the audio has to be streamed and especially processed very regularly, or you'll get the clicks, you'll get the cuts and the disruptions.

We decoded it ourselves, but we decoded it async. So we off the wise job, and actually used a separate core. The rest of the game wasn't really using. There's a a core that is, let's say, shared with the system, and so it's not typically used for real time work.

You can't put audio on it. Actually, well, when I first came to see the project, audio was running on this call, and then when we did the wise update, all of the audio broke, because you're not allowed to have audio running on this call because of the I see. I discovered the hard way when I broke all of the audio in the game by updating it was only on the consoles. That was a lesson lesson there.

But so you can't run audio on that core, but you can run compression decompression?

Yes. You can't run the wise update on this core because it's not reliable enough to do But actually, because the audio needs to be continuous, it can often tolerate latency. So if you're already tolerating some latency for streaming it in, maybe you can tolerate a little little extra latency to do the decode. Because of this, it meant that we could do the Opus away from the game and have this higher quality audio.

Again, even on the previous gen where this was not very commonly used level of quality audio because of the CPU cost, we were able to do this. And this helped us then get the memory down in app because it's a you can compress it more while having the the higher quality. So this was part of our strategy. In addition, because we took control of doing decoding, it meant that we could then take advantage of the streaming cache in So the same way that we were caching our streaming, Wise does this natively.

Right? It has its own stream cache to do exactly the same thing for exactly the same reasons.

But now, in this situation, why the streaming caching wise is the PCM cache. Right. So the decompressed audio, much bigger than normal streaming cache, so we have to give it a healthy chunk, another forty megs of of memory to be meaningful.

But because it's now caching the uncompressed data, it means if we have the same sound played on repeat, we will only pay the decode once for this. Come on. Tell me what's happening.

Come on. So when we have those physics sounds that are playing around or some machine gun or or something like this, we don't clear the cost for every sound, only this one sound and then the PCM cache will be getting hit in wires. So this also is then saving on the CPU cost. In specific situations, yes, so it's not every situation that it saves in.

But this is how you Right. You do it. Right? You chip away at those things, and that's where you're happy.

Do you feel that because Opus and these codecs are becoming more common, that's gonna be built into more of the silicon on chips?

This is So that that Yeah. Is faster to real time?

This is this is exactly why they're becoming more common because on the modern consoles, they have hardware for decoding Opus. So we were maybe a little ahead of the wave there, but now you have hardware for Opus. And I think this was the response to the strong voices from game devs saying, we really want to have Opus decoding. Whenever people would ask, okay, what audio features do you want?

Sound design would go, we we want Opus decoding so we can have this higher quality assets because, you know, it's it's nice to have cool, flashy new features, but often, the thing that you really need is just being able to do that basic thing better. Yeah. Just raising the quality of the most basic thing can have this very wide effect. And we did do AB tests in cyberpunk.

Even when we came back to Phantom Liberty, we wanted to check again and say, okay. It's been a while since we were running in in in Opus. Like, how does it compare with Vorbis? And so we made some a version which was using Vorbis rather than Agropus, and we compared them, especially in those dense fight scenes when you have lots going on.

It's not like Vorbis is is like crackling or very heavily distorted.

Right. Not that bad.

It was it was quite good itself. I recommend it. With with the music we actually do in in Vorbis, it's maybe a little technical detail, but under the hood, Opus is is like a combination of Vorbis plus Silk voice codec. And and this is where it gets its efficiency by combining like like, techno sound and half decoding in these two. And ordered sound like music might as well just be done with Vobis.

There's not there's not really advantage for putting it into It's not taking advantage of the No.

You don't get extra compression like you do with sounds like or especially with voices. If you compress voice, Vobis is very effective at that.

And that's why Discord uses Opus now. Voice chat is all Opus.

Yeah. Yeah. Yeah. So yeah, you get the great compression and the great quality. It's like win win.

Add some CPU cost, but for something like Discord, I think the CPU cost is not Right.

You're only streaming a few channels at once.

Yeah. It's not it's not trying to do like a couple hundred voices. Yeah. I mean, maybe there are cases where you do it.

But even so, it's probably not not okay to have that. I don't know. Maybe there's some cases. I've not come across them in my personal Discord use at least.

Really? Me neither.

But the clarity was clear. You can just hear more detail in all of that sound. It means that if you're gonna make that choice, everything will sound better. Right? The whole the whole game, the whole audio. Right. And it's not something that someone's gonna point at and say, oh, yes.

This was definitely this codec's been being used because I can hear that. No. It doesn't it doesn't work like this. It will just be overall better quality that the people will experience.

Yeah. Absolutely. I think that's always one of the funny things with audio is maybe because it's always going on in our heads all the time, we can't turn it off. It's something that I think compared to visuals we take for granted most of the time.

Like, it has a big effect on us, but we're not as able to identify why it is that we're having more of a reaction or why it is that we find this harder to navigate. Unless we are maybe someone like you who's really tuned into audio and, like, you're you know what to listen for and you know what you're trying to what you're trying to get, right? Yeah. To get that feeling of both immersion but also the emotional impact that you want.

Yeah. Yeah. I mean, even for me, honestly, like, this would be hard for me to check without the AB. If someone presented me, like, cold and said, is this opus or is this fullness?

I wouldn't do it. If you give me both of them, I will tell you which one is better. Yeah. What we showed in this experiment that we did wasn't only the people who, let's say, had a lot of experience working with those things.

Everyone could notice the difference. Like Okay. Anyone we showed it to was like, yeah, this one clearly sounds better than than the other one. So it's not just something nice for the sound designers to have, it does have this genuine impact and effect for the players too.

I wanna take a moment to give a shout out to people that are newer in the industry or who are wanting to get into it.

And I'm curious for you as an experienced pro, what workflow essentials would you prioritize for small teams or indie developers that have limited resources? They might think, oh gosh. There's no way we could do all these custom tools and custom CICD build stuff. What would you find? Like, the one or two things that would be the biggest impact if you're kinda low on resources like that?

So I have a little experience with this now with my with my side projects. I'm composing for indie games. And I get to see, let's say, both sides of the industry, both some of the biggest games being worked on where it's at scale with huge teams and also the very, very small team, scrappy little little team trying to Yeah. To make the most of of everything.

I think there's maybe a bit of a sound bite, but I think there's not been a better time from the creative perspective. The industry, in terms of jumping into the industry right now, there's certainly challenges like getting investment, things like this is certainly something that we struggle with. But in terms of the workflows and the creative power, you can download Unreal Engine and you'll be using many of the same tools that we use at CD Projekt because that's that's that's the engine that we're using to make the They're all built into the engine. Yeah.

And and even, you know, the collaboration we've done with Epic on The Witcher four tech demo. From there, the pipelines, the streaming, the foliage, the animation stuff. This is all going back to Unreal two. So you have at your fingertips incredibly powerful technology, and you see a lot of work being done to make it easier to get that high quality.

Also, in the keynote, there was this demonstration from the Metahuman. They had the webcam. So Hazzanadu Blue was performing live on stage. The webcam converting his expressions, performance capture live into the I I guess, you can say it's a VTuber now.

Correct.

So it looks the same as the stuff that he was doing when he was captured and done offline, at least in my eyes. These are very powerful tools that you can have access to. I really recently was impressed by Claire Obscure, Expedition thirty three. I'm playing through this. Yeah. I haven't finished it yet.

No spoilers, but I'm just about to start it.

Yeah.

I also won't throw any any spoilers there. But I think this this is a great example of a team that's for sure on the the smaller side and using like every ounce of possibility from the tools that are there, you know, using the assets from the marketplace. And in interviews, they were talking about using the the the free assets and and and making making use of those, having the really solid creative vision of what they want, an artistic vision, and then also a weird world where they can experiment and go wild, but this whole feels pulled together because of that clear artistic theme.

Generally, I think they just made great use of the tools that were there, you know. So so They only had a few programs, I think, the end, four programmers. So it's not like they were redesigning workflows. It's not like they were reimplementing whole whole tool pipelines, but from from what I gathered, from what I read from from them, they really have this focus of, okay, what can it do well?

Let's use that.

Let's lean into those things.

And lean into it. Exactly. And the game is fantastic. I think it's really inspiration, to be honest, to see the team is squeezing so much juice of what's there.

And I think every everyone can have this approach to that. You can just go and download Unreal Engine for free. But I think it is cool what they've done there. You don't pay anything until you've made a million dollars, you know.

That that would be cool to get to If your game hits a million Yeah.

Great. Good for you. I'll be glad. Bring my side project routes to that milestone of having to Yeah.

Having to pay.

Exactly.

One last thing I wanted to check is if you were to travel back in time to yourself at the beginning of the challenge that we talked about, right, where you're trying to figure out how are we gonna stream all this in, how are we gonna manage all this, is there anything that you'd say, go about this differently? Do you feel like you learned anything from this that wasn't just the solution, but kinda an approach or a mindset?

Yeah. There's probably some technical details that I would I would go into. I would maybe like to have from the beginning some automated testing, especially when we think about the container system, the streaming system we did. It is complex.

It is hard to understand, and not not just everyone can go in and fix bugs with it. So having some thorough automated testing from the beginning and even going over content files and checking this this parity between what we expect, what we have. I think and maybe I could put some some specific things to write for yourself about, okay, you need use some atomics in this place and not this place. Be aware of how this thread is going to work or or here's these calls back from wires.

You should use the these ones in this location, all of this kind of small technical detail. But I think I think the approach, I'm pretty happy with. Maybe slightly different in the future. We live in a a different world now with the SSDs.

I hear it probably wasn't an option, but if it was possible to go back and put SSDs in in every console like a generation ago Right. Then Perfect. For sure that would have made my life easier. That takes some powers beyond time travel, I think.

Maybe I travel back even further then start investing in the stock market at the beginning and then by the time that I can afford to know. There you go.

I mean, we start doing time travel, why not go all the way?

Yeah.

Well, thank you so much for taking the time today to share your experience and share what's been going on with this.

I'm super excited to check out Witcher four, obviously, when that finally drops. I don't know. I don't have a date yet, but I'm excited for it. And, of course, I wanna go back and check out Cyberpunk and listen to more of these audio things that you're talking about and kinda get more of a sense for that. So thank you so much for joining us, Colin.

My pleasure. It's been great.