Sharkmob's Journey to 10,000 Builds a Week: Infrastructure Scaling

It's funny to say, like, from an infrastructure standpoint, one thing that's really important is community, but I think community is super important to develop that at work. And then if you wanna take any serious moves or serious discussions and stuff, take a meeting outside of Slack, Have a physical meeting. Talk about the future. Talk about what does that look like. I think that's, like, the important thing as a whole. Just talk about these things because it's gonna build that community. It's gonna develop the community, But it also gets you a little bit more constructive on thinking, well, think out of the box a little bit.

Yeah. Welcome to In Development. The podcast where we explore the technical realities behind building today's most ambitious real time projects and digital experiences.

The tools, the pipelines, and the people solving real production challenges. I'm your host, Jace Lindgren. Today, I'm joined by Steven Rodriguez, studio infrastructure engineer at SharkMob. Basically, person responsible for keeping everything from build farms to cloud systems to the Perforce server running smoothly. Steven shares how the studio grew from humble beginnings to pushing ten thousand builds per week, The hidden cost traps of moving development to the cloud and why they ended up building a data center inside an actual bank vault. We're also getting into the modern toolkit powering triple a studios, how working groups keep teams aligned, and Stephen's practical advice for smaller studios trying to scale without breaking themselves along the way. And with that, let's get into it.

So Steven Rodriguez, thank you so much for joining us today.

Thank thank you. It's good to be here.

Yeah. So as I teased at the beginning, you are a studio infrastructure engineer at SharkMob. Can you give our listeners a sense of what does that actually mean if they haven't heard that title before or aren't quite sure what that means in your context?

Yeah. So it is pretty much new of a title, something we've been talking about and working for for SharpMob. But studio infrastructure engineers have really kind of this heightened power user at work who, helps manages a lot of our core technology that we recognize at work as being essential for the development of our current projects and also future projects. So for most people in the games industry, you might have heard of things like Perforce or build farm technology, cloud infrastructure, on-site infrastructure, data centers. I mean, it kinda runs the gamut. So my role is multifaceted for managing all of that.

Right. So when I hear infrastructure engineer, my mind goes to, like, physical networking and servers and network storage and stuff like that. But it sounds like at SharkMob, this title incorporates also kind of a pipeline aspect too?

Yeah. A little bit. So kinda supporting the base pipelines for the projects and stuff. That could be the build pipelines and such, whatever those teams might need for having faster builds or supporting larger environments, but also just infrastructure as a whole because these things are all interconnected. It's something we've realized over the past couple of years that as making games becomes harder and also more expensive, the amount of infrastructure you need isn't just on-site, but it's also external or in the cloud and so on. Right. So how these things communicate with each other is quite essential to kind of the backbone of how well a studio can perform.

Yeah. I feel like that's something that for a lot of really small indie studios just starting out, it's usually the last thing they think about. Right? It's just let's focus on making our game, getting our team together.

And it's usually not until you realize, oh, shoot. We we really need to get this infrastructure thing figured out and more solid that then you have to kinda backtrack and figure that out. So I'm curious about the journey of SharkMob because SharkMob started, what? Eight, nine years ago?

Quite some time ago. I think it was around two thousand sixteen, two thousand Yeah. I'm correct.

And so what was the size at first and what was the speed of growth like?

Yeah. So I joined around the time we were about a hundred and thirty people, something like that. And some of the friends that I worked with back then were as early in the studio's history as when we were, like, five or ten people. And honestly, it was a kind of story of, like, humble beginnings, you know, very small space, one floor, big ambitions, big projects in mind and such. And around the time that I joined, we expanded quite quickly. I think in that first year that I joined, we went from about a hundred and thirty people to close to, I think, two hundred or so, two fifty. We really fast.

Yeah. All within one year. Wow.

Yeah. Yeah.

Of course, when you have that many people being hired and a lot of different disciplines and stuff, you're talking about all that equipment, all that resource infrastructure. You need to have a lot of good people in IT to support that because that's a lot of new computers coming in, and, of course, space as well. So in that first year, we I think we ended up getting two more floors of space.

And for anyone in the games industry when you're kinda seeing that initial expansion of new building space, new office space, it's so exciting to be a part of that.

Yeah. The at the VFX studio where I worked, I was one of those. I think I was employee number, you know, twelve or fifteen or something like that. And we moved buildings a few times as we got to a hundred and then, you know, a hundred fifty or something like that kept moving buildings. And it was exciting to kind of be like, we're really growing up. We're becoming a big studio now.

It's a good feeling. You get all the branding placed out and stuff. Right. And it's just Yeah.

It really comes together. And you start to see a theme as well for the studio and the place you work at. There's a lot of camaraderie that grows as well and passion. Yeah.

I think very quickly on, we there was the whole feeling of black and red. Those are our colors. And we also changed the lighting in the office to match the game that we're constantly working on.

Okay. Nice.

So nowadays, when you come to the office, I think it's like this kind of yellow from the Exoborn logo Right. And bluish black or navy blue for everything.

Nice. And so what? When you were working on the Vampire the Masquerade game before, was it just all red all the time?

A lot of red. Okay. A lot of red all throughout the studio. You almost would think that the walls would bleed red at that point. No.

Yeah. That's awesome. So going back to the infrastructure engineer side of it, tell us a little bit about that and kinda where you came into the studio at this time where they're growing really rapidly.

And it sounds like taking this need for infrastructure and kind of these essential central services in the pipeline that clearly the studio was taking those seriously enough to create a position for it and and bring you on to help with that. But kinda what what was that process like? And specifically, I'm interested in what were some of the ways that you made decisions and some of the challenges that you ran into in that process of trying to figure out how do we even do this? How do we get this to be really stable and solid?

It's a really good point. Yeah. I mean, of course, during the lifetime of a development of a triple a project, you come across all sorts of situations that come up. Not only was it that the development of the project and hitting certain milestones and targets were challenges for our build team.

Because that's where I was first brought in. I was brought in as a DevOps engineer and working with our build team and then slowly kinda work through the gaps. But over those years, we started to realize, like, there are things where we need to think a bit bigger about our build infrastructure. We need to think about how do we scale our on-site infrastructure a lot more, how we see that we're getting a lot of great offers with cloud providers and it's a great relationship.

How do we continue to develop that? All these things are being considered and thought of because these are on the scales of the project that's being made, which was Bloodhunt at the time. But there's also the needs of the studio as well. How do we suddenly take something as small at the time per force working for, like, fifty or a hundred people to suddenly scale that out to three hundred or even, like, seven hundred people?

And having that kind of plan and mindset without sacrificing the amount of effort on all these different fronts was an interesting set of unique challenges that I think regardless if you're in the triple a industry or even if you're in the indie industry, it's it comes on your plate at some point.

I mean, so what's what's an example of some of those challenges you ran into that you think are common between more indie studios as well as growing into more of a triple a kind of size?

Well, one that really kinda hit us quite hard early on was the number of builds we were doing. I think the the very common loop that we saw or or what I've seen early on, and I haven't really worked in the games industry before SharkMob, the common thing I saw was that we needed to build the game every day.

This wasn't just a a progress meter, right, where we needed to iteratively build the game and see where our progress was each day. It was also for our QA team. So the QA team needed to do their analysis, do the review. Are we breaking features or are we progressing on our features? And also the playtesting that would happen nearly every day at that point. So building the game every night, not just development builds, but also what we call shipping builds.

All of that was quite a necessity.

Those shipping builds you're talking about, does that include, like, console builds for your testers?

So no. The shipping builds is more of, a higher grade of resource factor.

Okay.

For I'm really simplifying here. Sure. Yeah. It's a higher grade of resource factor for what the game looks like.

So if you think of it like when you do a dev build, it's kinda loosely packed together. It's still the full game, but it's not of the high fidelity factor that's necessary to get shipped out. Shipping is like, if we wanted to launch the game like today, this is that final product. It's almost there.

Just Sure. Not all the bells and whistles. Yeah.

Got it. Okay. So sorry. You were saying around a hundred builds per year, per month. What what were we looking at at first?

So those early days, I would like to say ballpark, we're about a hundred builds a day or so.

Okay.

How do we calculate that? It's everyone on their machine building the game each day. So doing normal on device client compilations or, like, local things. Then we had our build farm that was doing custom build. Because when you wanted to build the game in a shipping mode, that one you wouldn't want to do on your personal workstation. That's gonna take several hours. You want to send it over to the build farm, which has a lot more powerful resources to do that kind of work and at a lot faster rate.

So talking about that build, what was the issue there? You had that build farm problem solved. Right?

Somewhat.

Those early days were really rough. And if anyone has actually gone to one of the presentations we've given from SharkMob at GDC or at our local event, Nordic Game Week, we've shown pictures of what our early kind of build farm looked like. And it really was a bunch of custom PCs slapped together and connected on cables. Right. We would hope it works.

And, you know, as a growing studio, you run into all sorts of situations like power outages, power fluctuations, not planning, networking infrastructure as well to handle the load. We ran into all sorts of issues.

And during those early days, riskiest part is that when those nightly builds would not run and QA wouldn't get their work the next day

Because they they they work, like, one day behind. Devs do their work. QA does the review the next day, so on and so on. So if QA wasn't getting those builds, they're not making a lot of progress, and it's like a a race at that point.

Yeah. It seems like you basically set that whole department behind if that didn't go through. Right?

Exactly. Exactly. And then at the same time, the the playtesting team as well. Because we had a even during those early days at Sharkmo, even to this day as well, we have a reef testing lab on our bottom floor.

And in there, we run playtests. We invite people externally to come try and test our game. We're always constantly looking for playtesters, and they also feed off of the QA team and knowing, well, this is a build we wanna test. This is what's okay.

Right.

You guys can work with this. So, again, during those early days, really hard because once a build failed, well, then we're racing in the morning, the whole build team. You know, we need to get a new build prepared.

Can we get this done in time by lunch and so on?

Right.

Yeah. Because that's the time frame we were looking at. I was like, okay. Well, it's gonna be another, you know, four hours or so to run this build. Let's get it going as fast as we can.

Right. Right. So in your talk at GDC and at Nordic Games, you talked about basically leveraging infrastructure that you already had using Jenkins To do that on premise. Right? Yeah. That that was kind of the the solution for you. But when we were talking before this, you mentioned that there was also a time when you were thinking about what if we move this all to the cloud?

Yeah. There was a time.

I think especially for the kind of the studios that are very rapidly growing, that's where cloud comes in and says, hey, we're built for this. This is where we come in and solve this problem. Tell us about what you ran into with that.

It's a good one. And I I wanna phrase the answer to this one in the scope of like, well, if you're a smaller studio and you're listening to this and you're having the same kind of problems, what are the things that maybe you should consider that we realized a little too late?

And I and I'm not saying that any solution is wrong or good or whatever. It's all circumstantial. Right? You have to look at all the conditions and requirements that you have.

But the hardest thing we realized was that when we were testing things in the cloud, it really much was a situation that you have to have everything in the cloud. You have to have your entire Perforce instance. You have to have your entire build farm there. And then maybe if you're also technically knowledgeable enough, your workstations Where you actually open like the Unreal Engine or Unity Engine and everything and do your development work, all of that in there as well.

And the reason why is because the cost of moving the data between Perforce and the build machines and then Perforce and the workstations to develop the game, that ingress egress cost was the most expensive thing.

And I don't think early on, we were like, you know, that sounds like we wanna do because that's a lot of trust in putting it in there, and it might not actually have the bandwidth that we were expecting. And some tests early testing kinda showed that. That's not to say, like, that's the only option. The option we ended up sticking around with was to just have everything on-site. So we invested heavily in buying physical infrastructure and also hiring the right kind of staff on-site to effectively create a data center in our own office to host all this infrastructure, servers, everything, all of it.

I mean, I think that piece though of having to hire someone who understands the physical infrastructure and setting that up is a big area where people can go wrong. Right? If they think, oh, we'll just do it on premise. But then if you don't have the infrastructure for that, if you don't have the right cooling and, you know, electrical setup and networking setup and everything.

I think the cooling is like the craziest thing. I remember when we put it together, we built our entire data center inside of a bank vault.

Wow. Like an actual bank vault.

Like an actual bank vault. Yeah. Because the building that we're in used to be a very old bank from back in the day. So the funny thing is when you come join for a playtest, you're going into one of the bank vaults, the biggest one.

It'll lock you in here until you can beat the game or something like that.

I mean, thankfully enough, the door is bolted to the wall. But Okay. You know, if it wasn't, the other bank vault, which I would kind of think is like where the security deposit boxes were. That one is where the data center is. And Wow. The hard thing with that from the cooling side was drilling through two meters of concrete for cooling pipes.

Woah. I didn't even think about that.

Yeah. Put that into a factor. Like, that's not something maybe we didn't consider early on. Hopefully, we did.

Perfect. I wasn't part of those discussions early on, and that just adds and adds and adds to cost. So any smaller studio thinking, like, can we just do this all on-site? Do factor that in.

Right? Because there's that cost, the cooling alone, where to put it, the amount of energy being sucked into there. It all kinda just boils up.

Yeah. If you were approaching that now with what you know, what would be some of the key factors that you would take into account if you're debating between do we just start supplementing what we have now with cloud versus do we start buying more physical hardware? What do you think would be the key reasons why you might lean one direction or the other? And probably end up doing some of both, I would assume.

I would say, like, still at the end of the day, do both. Like, if you've got room if you got budget to work with, it doesn't really do that much harm. You could have just a really powerful workstation that's sitting at the office, and you have that running as a Perforce Edge or so.

Or you work with great partners like IncrediBuild who can help you do build farms with, like, just existing workstation setups.

And that's, like, a great way, like, early on in the journey as a game studio to maximize and get those, like, really kind of massive benefits at scale. And then, of course, using the cloud as much as you can, maybe put your per first commit server over there in the beginning, because one of the things that I realized now, but I wished I realized early on enough, is that, let's say the development of your games is going well, and you really get a lot of developers on it, and the rate of change is sporadic. It's it's really up there. You're gonna need to increase your resources all of a sudden.

Right.

And if you have physical on-site infrastructure, the scariest thing is telling all your developers, hey.

I need to turn off Right.

Perforce Right.

Because I need to add in a couple hard drives. I can promise you, your producers, your director, you know, the c suite, all of it. They're gonna say, no. No.

No. No. No. You can't do that. And then you ask them for a window, and they're like, well, the closest one I can give you is, like, in six weeks.

Right. For you as, like, the IT guy or whatever, you're like, but we need it now. Like, now now.

Yeah. Yeah. But we're maxed out on what we can handle right now.

Exactly. Exactly. But, of course, if you have it in the cloud, it's actually quite easy to do an expansion like that. Most cloud providers these days give you the feasibility to do snap disk growth expansions if you do the technology right.

Yeah. And most cloud providers these days provide depending if you have a really good relationship or not, which I recommend you should have a good relationship in the beginning, like have a key account manager. They don't cost extra. And in fact, they help you use their technologies even better that way.

But if you don't know how to set up the right kind of infrastructure in the cloud, you just ask them. They'll give you a solution architect, and you guys will work together. You'll get direct feedback. Oftentimes, the person that you're paired with to do that kind of, like, initial infrastructure planning and discussions is someone who's already done this for other studios as well, And more often than not, other studios within your own region, you get a lot of that relevant knowledge that you want to hear.

Yeah. Yeah. How how does someone go about getting a CSM if they don't have one already?

So if you don't have a CSM already, the first thing to do is talk to the sales manager and say, hey. You know, as part of the sales contract and signing things forward, you you wanna do two things. First is you wanna switch to invoicing. Right?

Just so you have a better view of your billing, if you have, like, an invoice team or procurement team. And then the second thing you do is you ask your sales guy, hey. Can we get a CSM? And and we'll also kinda poke a little hard and say, can we get a CSM within your own region?

Like, unless you're in, like, the most remote part of the world, you should be able to get a more localized CSM within your own time zone as well.

Right. The time zone thing is is key. Yeah.

It is. It is. I used to work with CSMs who were in vastly different time zones, and I'm like, well, this doesn't work that well. Right. I want someone that's a little bit more responsive and quick. And, of course, as your expenditure grows, which naturally would, if you're really happy with the solution, the CSM is gonna work with you more to give you discounts, better offers.

That makes it so that your job goes better with your higher ups and your directors Correct. And saying like, hey. I saved us some money here and this and that.

Yeah. That does look good on your reviews for sure.

Exactly. Like, that's what I feel like a lot of my job is as a studio infrastructure engineer. It's not necessarily about spending the money the right way, but it's spending the money in the most effective way.

Right.

Sometimes that really boils down to where do we cut the cost while keeping or improving our performance and our reliability of our systems. But, yeah, I would say that's one great way to just talk to your sales manager or if you already have an account relationship, someone you report to for that cloud provider, just give them a poke and say that.

Nice. So one question that I like to ask everyone who comes on the show is what is your day to day toolkit? And I actually think that because of the position you're in, you kind of have a unique perspective where you probably see every piece of software that anybody uses because at some point, that's gonna hit your infrastructure and hit your pipeline. So could you give us a sense of, you know, what is the toolkit that you use and what does your studio use on a daily basis?

Yeah.

So I think I can go a little bit more let's go broad, and then we go specific.

Yeah.

So studio wise, we're a Unreal Engine game engine studio. So we work with Unreal Engine. We do have a custom engine team. So we make some modifications to the engine to get the right kind of effect that we're looking for for developing the game.

We also have a a tools team that makes custom solutions and software to help support the developers. On top of that, I mean, we do all the normal things. We use Maya. We use Photoshop here and there, like all the Adobe tools and stuff.

Right.

We do the UI and UX design using tools like Figma as well or mirror boards.

Right.

Right. Which is really great. And Perforce, of course, that is the the bread and butter. But from a Perforce standpoint, we do Perforce. We do Swarm for our reviewing structure.

Okay.

So you do code reviews.

Yeah. So we do code reviews. It's quite active actually for our swarm usage. I think I I go in and I refresh, and it's constantly a new review popping in every second. So from a swarm standpoint or, you know, management of the branching structure and stuff, we have a kinda what I would call like a working group that helps define the branch strategy and stream strategy for the entire studio. From time to time, we branch out. We we do new streams when you do new things and stuff.

And we just kinda put together a new solution for that as well. Previously, we do a lot of triggers. We use something called robo merge.

I don't know if Right.

Epic's You've heard of that?

Custom tool. Yeah.

Yeah. We were a little hesitant in the beginning to use it, but I can promise like, lot of people, if you read the documentation and you join a good community for working with RoboMerge, it actually is an absolutely amazing piece of technology that works just so well and seamlessly.

I feel like recently, I've been hearing that from more and more people. I haven't tried it out myself, so I'm a little bit like, what do you mean? How do like, robo sounds scary when it comes to merging. It's like, it sounds like, oh, it's gonna automatically do stuff. I don't know if that's what I want.

Exactly. And that was kind of up here in the beginning because it's like, how do you just trust an application to automatically copy things over between streams and stuff? Right. But the real real simplified thing for anyone listening on the podcast who hasn't worked at VrboMerge before.

The real simple thing is, like, let's say you have two milestones. You have the current milestone that's coming up, and then you have the future milestone. And stuff you're doing in the upcoming milestone also feeds into the later milestone. So you want the stuff to feed into both streams.

Some of your developers aren't working on the upcoming milestone. Some of them are just working on the next one. So you wanna set up a system that can copy the commits over between both streams, but you don't wanna tell all your developers like, hey, do your work twice as much. That's where you let robo merge take the work from there.

You just say, alright. You know, whatever's happening in this one, auto copy it over, keep the ball rolling forward. And then from a developer standpoint, they don't even have to think about it. So that's kind of the benefit here.

And for us, our branching structure, I wish I had a diagram for it. Holy hell. It is extensive nonetheless because of course, every new milestone, every new release, you know, you may backtrack as well, you know, you fix something from a long time ago. That all needs to feed upwards.

So the tool side, just rolling back on that. Yeah. Let's get a little more specific like my set of tools because I do have a lot more specific tools to do my job. And since it's a lot of infrastructure, I do really lean heavily on some tools like Terraform and Ansible Yeah.

To do a lot of that automation structure. On any given day, I manage anywhere from fifty to a hundred different servers. These are everything ranging from build servers to cloud infrastructure to Kubernetes, namespaces and pods and deployments and stuff. So having a infrastructure as code type solution that can help me manage things on mass with just writing code is something essential to do my job.

Right.

And so you mentioned two different systems. Right? You said Terraform and Ansible. Why both? Why not just one?

So the thing is there's a lot of things Terraform can do. Right? Terraform is quite great. It's fantastic.

I would say Terraform is very state driven and whatnot. But Ansible is kinda like a little bit more simplistic. Sometimes they need to do custom scripting or sometimes we need to just manage in house infrastructure using Ansible and focus on having a similar baseline of how things work. So we use Ansible for those kind of situations.

Could be like Windows boxes or Windows servers and things.

Okay.

And then Terraform is more on the cloud side of stuff.

Got it. Okay. That makes sense.

And at the same time too, you know, we go one step further. We use stuff like Semaphore, which helps automate our playbook run throughs from Ansible. And then Versus Code has been the godsend of the industry, not just for gaming, but for everyone. It's absolutely amazing. It's a great piece, a great editor, great tooling.

I don't want to have to open Visual Studio every time, but I'd rather open code Yeah.

It's nice to work those those quick things where you don't need the full integrated ID.

And then on top of all these things, because since it's lot of infrastructure that's being managed, things like monitoring solutions are super important. So we have a great relationship with Datadog. We run an EOK stack, and we work with another company that we just put a contractor with called Auvik. It's called a u v I k.

Auvik. Okay.

Yeah. And what they do is we put together, like, bastion containers that run all throughout our ecosystem. And they monitor everything from device performance to the flowing of network packets to everything, honestly. Wow. And the benefit of us having that on the system is that it gives a lot more deeper insight into how our entire infrastructure is working, how the I call it like the ley lines are tied between all different systems and whatnot so that we can quickly determine, well, where's the bottleneck happening or where's issues happening and stuff.

It's interesting because I was just having some conversations recently with people from Epic. It was talking about a very similar thing, but just from the performance of the game or the software or whatever that's being developed in Unreal and kinda how you can set up monitoring in advance so that you can keep an eye on that and see what changes. And so you're kinda talking about that, but where the the game in this case is the whole organization, like your whole network.

Exactly. It's like the developers are working on making the game, but I'm working on, well, how do I keep the studio functioning? Right. Right.

Yeah. A very high level. And that's the thing. Because, sometimes as part of my role, we may have developers or, power users, as I would say, doing fantastically great things.

Because when you work in the games industry, you hire great talent from time to time. And these people will put together fantastic solutions. Right? That's what you kind of expect.

But, of course, they might not be aware of where all the bells and whistles and the bottlenecks might be. So sometimes you might get hit and it can hurt a lot of people or take a system down.

I see, like, putting a bunch of load on a system all of a sudden to get something done faster, but it actually takes everything down. Yeah. That makes sense.

Exactly. Exactly. And that's kind of the thing that happened with us because we were talking earlier and saying how we in the beginning, it was making like a hundred builds a day. But now, and just not too long ago, we hit a target that we've been trying to hit for quite some time now. And that is that we hit ten thousand builds now per week. So it's a ten x improvement in how many builds we can handle with just on-site infrastructure. It's it's been a long time coming to get to that.

Nice. Yeah. That's great. And so what about task management and organizing the studio? What tools do you use for that?

For task management and such, I think around when I first joined, this was close to five years ago, we had a lot of discussions about it internally, and the discussion was centered around, let's use Slack, which is a great workplace communication software. I will, like, stand on my grave on. Slack is terrible at times for communication, but it works.

Yeah.

Right. Yeah. It's an industry standard. Everyone likes to use it. Yep. But sometimes you just gotta take your head out of Slack and have a physical meeting Or or a Zoom call or something. It makes such a difference.

Yeah.

But we use Slack. We use Jira. We use the entire Atlassian toolkit. So Jira and Confluence mostly. And then on top of all that, we in Slack, we set something up that I kind of picked off from Spotify over in Stockholm. They use something called tribes.

So taking people from different departments and putting them into the same Slack channels or or groups or so

And trying to solve bigger problems or bigger things at hand with a cross departmental effort.

Interesting. And so that's something that you've started doing at SharkMod?

We've been doing it for quite some time now, actually. Almost five years now. I call them working groups. Some people call them task forces.

But it's this effort where when we are a bunch of power users or really deeply knowledgeable people about a certain topic or about a certain thing, let's just make a group on it and work together on that with the same objective in mind. There's still gonna be someone who's like a leading force in that. For me, that's per force. I'm the one in charge of that, a shark mob.

But for some of these other efforts and stuff, there might be several different people who are leading those efforts and such. The thing is is these working groups, these task forces, these are non game related, so to say. Right? These are studio related efforts and stuff for like long term goals and long term objectives.

I think the closest relation that maybe a lot of people might have experienced before is just when you're in Slack and there's a lot of passion about a new game that's come out or a new cool thing to see. Okay. Maybe you got a lot of, you know, cinema buffs and film people or music people. Right.

You create a music channel. You create a film channel. But for us, we we said, alright. Well, let's make a working group for Perforce stuff or for BuildFarm technology and so on and so on.

Yeah.

Because I was gonna ask what are some concrete examples of those task forces or working groups.

And so it's more just that the thing in common isn't we need to make a decision about some particular thing, but more, hey. We're all the people who are passionate about this. Let's make a group to talk about it. Almost like a Yeah. Like a developer community might be online, but within your own company.

Exactly. Because we're a lot of people in the company at this point. If you include all the co dev studios and that we work with, there's a lot of impassionate people in there. Yeah. And the big ones, of course, is obviously our Perforce group. It's gonna be our Build Farm group as well.

We have one for gaming.

So, of course, just, you know, talking about the latest, greatest Right.

Funnest game to come out. Yeah. Let's talk about it. We even have a local group as well.

So, like, all the local events happening in our city, it's a nice way to help build community and make some friends. And I think that's one of the quite important things. It's funny to say, like, from an infrastructure standpoint, one thing that's really important is community, but I think community is super important to develop that at work, that it takes effort. Yeah.

One really good one And I think this is one that I think a lot of game studios should do, is a Linux group.

Oh, sure.

Yeah. So, like, actually, you know, get your Linux power users together. Just talk about Linux. Just talk about it. Like, Proton is a thing. Proton's fantastic from Steam, and just discuss these things in specific, you know, what can be done in Linux Right. What kind of performance improvements, more importantly, kind of cost savings you can get from using Linux as opposed to Windows, things like that.

Yeah. Yeah. Absolutely. In the VFX industry, that was always a big dividing line between the studios that had gone as fully Linux as they could and the ones like the one where I worked where we'd been more resistant to it. Because we relied on just enough Adobe stuff and they don't support Linux, so we couldn't really make that transition at the time. It is an interesting thing though that then the other studios who did use Linux did have a lot of advantages and some flexibility that we didn't. So that that is cool to have a group within your company that can just kinda brainstorm and talk about those things.

Yeah. It's really good to just have those discussions, at least with a lighter air and stuff. And then if you wanna take any serious moves or serious discussions and stuff, take a meeting outside of Slack, have a physical meeting, talk about the future, talk about what does that look like. There's some groups that we have that meet on, a monthly basis or, like, a biweekly basis at work to talk about these things, and not at just, like, within their own departments, but also, like, within the studio as a whole. I think that's, like, the important thing as a whole. Just talk about these things because it's gonna build that community. It's gonna develop the community, but it also gets you a little bit more constructive on thinking, well, think out of the box a little bit.

Yeah. Awesome. So as we're coming toward the end here, I wanna spend a little bit of time talking to the beginner. And so what I mean by that is someone who's either starting up a studio or just starting to grow past that five to ten people and they're starting to expand past that. Kinda what would be a few of your takeaways you found from your experience for them, as well as what might be some advice you would give to your past self with the knowledge you have now to kinda help you, have a better time with some of the transitions and challenges?

I I remember having a conversation about this recently. And one of the big things honestly is, like, especially as a early on studio trying to grow and, you know, develop your games and stuff, is be a little more specific on some of the roles that you hire for.

So hiring game developers and and three d modelers and artists and things like that. I mean, those are necessities. Right? But that's more on the game side. But I'm thinking in my role, like studio side. You you need certain kinds of people.

And I think a lot of people here are, you need an IT team. For a long, long time, we had just one IT guy. Right. And we made that work. But was it a good idea? Not necessarily. We should've had a lot more.

Right. Right.

And it's not just having, like, your IT guy, but you're gonna wanna also have, like, someone who's like a Linux administrator. Early on, you're gonna wanna look into cloud technology. So you're gonna wanna look at roles like a cloud architect or like a full stack engineer. Someone who is knowledgeable, also like a DevOps engineer as well. But people who are knowledgeable enough to design architecture, write entire applications from client facing to back ends with ease, and really kinda help you think a lot more broadly on, like, what does your infrastructure need to start looking like In order for you to build towards something a little bit more robust. And then later on, get some more specific people to help with things.

Now, here here's a question for you. And this is something that came up also in the VFX world when I started taking over our pipeline team. I'm curious for your thoughts is if you had to kind of ballpark thinking about a percentage of your total company employees who are in some kind of IT support type role like those ones you were describing. What you think that would be?

It's it's very small. It's very small. And I think that's I might not have the numbers right, but I think it's somewhere around like five percent or less. Right.

Okay.

I think the better metric to say is like two to four people for, like, the first hundred or so, and then maybe you wanna ramp up a little more after that.

Right. Right.

But, of course, the reason why I say keep it so small, I think that's the key thing when going for here, is that you don't wanna have too many chefs in the pot. Right? These roles that I said specifically, like cloud architects, DevOps engineers, full stack engineers, these are really, really talented and highly specific roles. They don't come cheap, but you don't need that many people for that. I think for a long time, like, when I joined, I was one of two DevOps engineers, two or three, and we stayed like that for a couple years actually. So we went up to, like, almost three hundred people with only, like, two or three people being DevOps engineers.

Right? Plus some other IT as well. Right?

Yeah. Plus some other IT, and then we started to create an SRE team as well, but that was a whole separate thing, more project focused. Altogether, I mean, that's the kind of thing. Like, you don't wanna have too many chefs in the pot because they're working with such large robust systems. There isn't a lot of handover that needs to happen in the sense of, like, working as a team. But these guys, they can do a lot of automated work.

That's interesting. So the number that that I had heard before in the VFX industry is a the goal number is nine percent.

Nine percent. Okay.

Yeah. And I was working at a studio where, you know, we had a hundred people and we had two of us, I think, and it was definitely not enough. Like, we needed more.

And so Yeah.

Yeah.

And so I'm I'm always interested to what people think is is the balance. Yeah.

Yeah. I would say if you include IT as as a whole department of things that need to be Yeah, it should be ten percent because you also have to think of, like, what you're growing as a studio. You have a lot of people who are coming in. You're talking about monitors, PCs, keyboards, mice, all that kind of stuff.

You have to bring all that equipment in. You're either developing it in house or you're having it custom made elsewhere. Right. And tickets issues.

Right. People saying, my mouse doesn't work. Like, what do I do?

Yeah. Yeah. Sure.

Right. So I think that's where the ten percent comes in. Right? You need to have someone who answers those tickets, makes people feel like the office environment is being developed and is being worked on.

Right. That it's not just staying afloat, but it's progressing, that it's that it's advancing and improving.

Exactly. And I think the great thing I've realized over my my time at SharkMob is that almost as a necessity, like, have to have a relationship with the office team. Having an office team, especially if you have a large enough office, is quite a necessity. Working with the office management team is quite important to talk about the bigger plans and bigger structures because like we've talked earlier on, infrastructure is not necessarily just the physical hardware. It's also where it kinda sits to sometimes the software. So it's good to talk with your office management team, and then through them, talk with your landlord if you don't actually physically own the property and everything Right. About those bigger plans and bigger things involved.

Yeah. What else do you have for beginners or people just starting on this journey?

Big one, honestly, is make a budget and think big. Right? If you only have a budget of a couple million or something like that and and I say million because we we talk Swedish crowns. So a million is like more like a hundred thousand euro or something like that or maybe less.

Got it. Right? But no matter what size kind of budget you're working with, just think about, well, how much of that budget are you gonna put towards your technical infrastructure? And we're not just talking like buying workstations and desk setups for people.

We're also talking about servers, high end PCs to build the games and stuff, cloud infrastructure as well.

And And way more storage than you think you'll need.

Yes. Exactly. That was gonna be that was gonna be a later point too. Like but of course, with all these things, we talked about CSMs before and cloud relationship managers and things like that. Talk with them. Because a lot of these cloud providers these days are providing great benefits for new studios, people new in the games industry, because they want to see people using their products more. They wanna see themselves bake into this new area of the industry that maybe they haven't really delved too deeply into.

Right.

What that translates to is a lot of free credits and good discounts.

Yeah. Yeah. So Definitely.

So even if your budget looks like this, it can be a lot bigger if you have the right kind of conversations. So, yeah, making a budget is super important and setting things aside for tech infrastructure because if you don't plan for it, it's not gonna happen. But then we talked about storage. Yeah.

Storage. That's expensive. Some situations that we came across early days was like, all of a sudden, you know, you're working with a codevelopment studio to make a trailer for your game. Right?

And they're like, hey, by the way, we're done. And they're like, k. We can send the files over. And, you know, you ask them, like, how big are those files?

And they're like, a terabyte. Right. And then it's like, oh, okay. Well, how do you download a half a terabyte file when your milestone delivery is in, like, one day?

Sure. Sure. Right?

Yeah. That's the scary part. That's where the previous point is invest in the right kind of infrastructure for that. Sometimes you don't need to have that on-site. Sometimes you have to have that in the cloud.

Right.

So having a CSM helps because you can then talk to them and say, hey. I have this big problem. Can you help me?

Right.

And also, the key thing is is sometimes don't always, and I think this is a good thing to understand, don't always trust your gut. Just because in the past, you might have trusted your gut and thought tools like Dropbox or Box or OneDrive or something might be okay to do this, always have that additional discussion with a CSM or a Linux administrator or some technical person and stuff. Talk with them about it because there might be a better solution on hand. I I've had situations like this before where someone just drops a big file on some other system or software, and then all of a sudden, I see, like, a really large bill come in, I'm like, why did we do that? Yeah.

And I'm not just talking like what I've seen in the games industry from what I've heard from other people, but this is just what I've seen in my history in tech as a whole. Right. Storage. If you don't plan for it, it's not gonna happen, and you need to have super reliable storage.

You know? Don't go to the store and just buy a NAS because someone said, this is gonna work for you. The problem is is the guy at the store is gonna say it's gonna work for you and only you. But how do you make that suddenly work for not just you, but a hundred and fifty other people?

Yeah. There's a big jump when you go from that five, ten person team to something that can support your hundred person team. Yeah. That's a big big jump.

And I think that's the next thing to give as a tip for beginners and stuff or smaller studios, but partnership relationships. I swear, it doesn't matter how small of a studio you are, just consider those relationships. Consider those partnerships. And we talked a lot about cloud partnerships, but we're also talking about equipment partnerships.

For us in Sweden, that's companies like Duston. Outside of Sweden, that might be companies like just talking to Dell or HP or whomever that might be, whoever provides your IT resources and things like that. Right. And then also partnerships with, like, service contracts.

You know, who is your build farm provider? We use IncrediBuild. We have a great relationship with them. They respond, like, within five, ten minutes if I wanted to talk to them.

Nice. And managing those partnerships and talking with them and just being open, you know. Sign the NDAs, like, that's the key thing. Sign those NDAs, talk more openly with each other, have those discussions because I can promise you, you're not the first ones to come across whatever you're going through.

Yeah. Someone else did.

Yeah. I've I've absolutely had that experience on the, I guess, CSM side of things working for Perforce where, you know, when someone comes and talks to me, often it's at a conference or something like that. We'll get into a conversation and I'll be able to bring up other ways I've seen it done and some solutions. And oftentimes people are like, dang it. I wish I had talked to you like a year ago before I spent all this time doing this more complicated way. And so, yeah, having that relationship could definitely save you a lot of time.

Exactly. So, of course, I mean, I think that's the big one that I I can close on with all these, like, pieces of advice. Those partnerships and stuff, and also between studios as well. It's something I'm trying to add in more to my role as well, but going to these different conferences and events and having a bigger presence to talk with people because Nice.

There's a lot of learnings I have, but there's also a lot of learnings that other people have. And having those moments to just have a lunch, have a dinner, reach out, talk about these kind of stories and situations really kind of opens the eyes a lot more. And I'm more than happy to connect people and say like, you know, hey, the IncrediBuild or other partnerships and stuff, here you go. These are great people.

Talk with them and stuff, and maybe it helps ease that burden a little bit that you've been shouldering for a little too long.

Yeah. I love that. And I'll second that too. As someone who goes to a lot of conferences now, I always learn so much as well as being able to help a lot of people, and it's just such a cool thing to do.

Yeah. Just earlier this year, I was at Unreal Fest in Orlando as well as also Bali. And then later this year, I'm going to the one in Tokyo. But, like, I always learned so much at all of those conferences that yeah.

Can't recommend it enough.

They're amazing, honestly. And and I actually have not gone to Unreal Fest before, but the more I hear about it, the more I'm like, I need to make some moves that way.

Yeah. Add it onto your list.

Yeah. Exactly. Exactly.

GDC, Nordic Games, Unreal Fest. Yeah.

So, of course, I mean, I encourage talk between studios and stuff. If it makes it easier, sign an NDA just so you guys feel a little safer to talk about things. I'm pretty sure most studios these days have a default NDA. Yeah. So you can just, like, ask them for it, grab it, hand it over, and then talk. Yep. You you would be surprised the kinds of fantastic conversations and learnings you can pick up in just a small amount.

Yeah. It's great. Well, thank you so much, Steven, for talking with me today. I learned a lot during this, and I hope that our listeners have as well. So thank you so much for sharing with us.

I appreciate it. It's been a fantastic time. Always happy to talk about all the wonderful stories. If anyone wants to reach out on, LinkedIn or social media, I mean, by all means, I appreciate it. I think I keep also an open booking. So if people wanna book a time slot or whatnot, I can find some time.

Awesome. That's great. Well, thank you so much again. I appreciate it.

Thank you. Thank you.

If you enjoy this kind of in-depth conversation with leaders in the media, gaming, and visualization world, be sure to subscribe so you get new episodes as they come out.

Subscribing and giving a review or rating is the only way that we know you enjoy this content so that we can keep making more of it. Reach out to me via email or LinkedIn with feedback, guest suggestions, or to connect if you'd like to be a guest yourself. I'm always looking for great stories and I love learning about the many ways game and real time technologies are being used today. Links for that are in the episode description. Be sure to check out perforce dot com for information about our full p four platform based on the industry standard Perforce p four version control. Special thanks to our production team, Ella Reiswig, Kaylee Torres, Luisa Puhala, and Chris Perez. I'm Jace Lindgren, and I will see you next time on in development.