About the Episode
How do you improve workflows when developers don't realize there's a problem? In this episode, Joe Deuchar, Technical Art Director, joins Jase Lindgren to explore how telemetry can help studios uncover hidden friction in their tools and processes. Joe explains how collecting even simple usage data can reveal performance degradation, workflow bottlenecks, and opportunities for automation long before they become major issues. Telemetry isn't just for web-dev teams anymore and it's easier to set up than you think!
The discussion covers practical approaches to setting up telemetry systems, deciding what data to track, and using quantitative insights alongside direct user feedback. Joe also shares lessons from years of building tools for game teams, including ways to validate workflows, measure the business impact of pipeline improvements, and communicate the value of technical initiatives to leadership.
In this episode:
- How telemetry helps identify workflow issues before they’re reported
- What data studios should track and what can be ignored
- Why quantitative and qualitative feedback work best together
- How validator systems improve pipeline quality and user adoption
- Measuring ROI for internal tools and automation projects
- Using data to communicate pipeline value to leadership
- Practical telemetry advice for indie studios and small teams
- Why creativity should matter more than metrics alone
FEATURING
Jase Lindgren
Senior P4 User Advocate
linkedin.com/in/jaselindgren
Joe Deuchar
Technical Art Director
linkedin.com/in/josephdeuchar/
Check Out More Episodes...
Ready or Not, Here Comes CI/CD: Build Systems for an Indie Sensation
Stephen Post, Technical Director
VOID Interactive
From Maya to Unreal: The Story Behind Nickelodeon's Max and the Midknights
Sica von Medicus, CG Supervisor
Nickelodeon Animation Studios
Fail Fast, Fix Faster: Pipeline Wisdom from Halo Infinite to Virtual Production
Luis Placid, Engineering Director
ICVR & Shiftstorm Entertainment
Full Transcription
If you're starting to build something and if you're starting a company for the first time, you wanna set up like just a very simple telemetry service somewhere just to give you a counter on what tools have been used, what pipelines have been executed. Because if you get that in early, just super superficial data, very kind of broad open to interpretation in a couple of different ways. If you start doing that early capture of data, you can then go and start looking at changes you make over the life cycle of your projects and the actual impact it's having on people. So for example, we realized that there was a problem of like navigation meshes in the game we're working on and users were having to manually rebake the nav mesh every time. Unfortunately, this project doesn't have like other telemetry service set up for tracking this particular flow. But if we had, having that telemetry there would have allowed us to kind of discover these moments where the workflow isn't being followed.
Welcome to In Development. The show where we explore the real engineering challenges behind building and shipping digital experiences like games, animation, VFX, anything that involves real time three d. I'm your host, Jace Lingrid. Imagine that you've created a tool to help automate something. You made this for your team and you're very excited about it, only to discover months or even a year later that no one was using it.
Or maybe you've made something where you had a button that you could click to manually do something if the automation doesn't work. And you discover later that everyone's clicking that button constantly every day instead of letting the automation work. They've made a workaround around your tool rather than using it, but you didn't even realize it. How would you know if you weren't sitting there watching them?
If you can't sit there and watch them, how would you know? And to answer that question, I'm joined by Joe Dushar today, a technical artist who specializes in tools and pipelines. And he has some very cool and interesting ways of using telemetry, that thing that you normally think about being for end users or maybe for website analytics. What if you use that in your own tools to get better data?
And if you have that data, how do you actually use it? How do you make sense of it? How do you make that useful for how you develop your tools and how you talk to your teams? Today, we're going to look at the unreasonable amount of power in getting data that's actually useful and helpful for you so that you can catch problems before they happen and you can implement better solutions so that you and your team can do your work faster, more efficiently, with fewer errors, and focus more on the creative part that you actually love.
And with that, let's get into it.
Well, Joe, thank you so much for joining me today.
Thanks thanks for having me.
So can you tell us a little bit about what is what is it that you do? What would you say you do here?
Okay. So my job is something called technical art. And, what I do is I work in the games industry and I help bridge the gap between content creatives, people working in three d s max Maya, three d modeling applications or Photoshop, or people working of audio. And generally, just content creators wanting to kind of express themselves creatively and then integrate their work into an engine or editor which are ultimately building, like, that gaming experience we wanna provide to our customers.
So when you talk about technical art, I feel like there's a little bit of a confusion in the industry where someone might be a technical artist and that means they do, like, shader compiling kind of work. Like, there's some very technical thing related to art. Yeah. But it sounds like you're describing more like tools or pipeline development. How do you see those two things being related or not?
Yeah. Okay. That's a really good question. So I mean, I'm lucky enough to kind of be more focused on tools and pipelines. But over my career as technical artist, I've worked with a lot of other people with different specializations. Technical art, I generally kind of see as a Venn diagram of three overlapping circles, one of which been tools and pipelines, the other one being graphics rendering and performance, and the third part, the triumvirate, it's been rigging and animation.
Okay. Yeah. Maybe maybe just as a former audio person myself, I might slip in like, you know Yep. Audio specialization or something like that, a little more technical side of how audio is rendered.
Definitely feels quite a big overlap between normally the animators and the audio engineers. Like one common thing, you know, we'll do if you're gonna see is look at ways of automating, like trigger effects for footfalls or sounds.
Yeah.
It tends to be TA and audio work a lot together.
Yeah. Yeah. That makes sense. Okay. So you've worked in technical art. And tell us a little bit before we get into our main topic, you know, how did you get here? What was your trajectory to that?
When I was young, I really like video games and board games in general. So that's where it begins. I decided I wanted to study this at university. At the time I was studying, there was only really two kind of games related courses been offered.
One of which was like a gameplay programming, a little on gameplay design. As part of the design course, they taught a broad kind of range of skills, not just like writing GDBs, but also through to how you'd actually build content, how you'd like organize a group of people to collaborate. I originally wanted to be a character artist, but unfortunately, my artistic skills are not really that good. So I spent a bit of time after university trying different roles.
I worked in networking, IT administration, networking, as well as architecture before I got approached about a job for a company that is based in Nottingham, my home city, because monumental games that used to make massively multiplayer online experiences and MotoGP and things like this.
Nice.
Very fond memories of that company. And then, yeah, unfortunately, as with many game studios, they eventually went under. Yeah. And I spent more time kind of bouncing around different studios in The UK.
And were you doing this kind of technical art tools stuff that whole time or like when did that switch happen?
So funny story that when I actually went for my actual interview of this company that ultimately gave me the first job, I went super prepared, well over prepared, did the forbidden thing of showing up in a suit and tie, like all the cringe you can imagine. And I almost immediately freaked out when about thirty minutes after the interview was meant to start, the guy who became my manager came in and was like, sorry, I completely forgot we had an interview today. I haven't had time to prepare. And I'm like, oh, ****, I've been overly worried about this. So I've spent time writing a list of things that I'd expect you to ask me. And he literally just asked me to give my notepad so he could interview prepared.
And yeah, the very end of interview, actually, I don't actually know what this role or job title is. And he said, oh, it's for a junior technical artist. And yeah, basically, that guy was my mentor for a couple of years and monumental and sent me on the path that I'm on now.
Nice. Nice. Very cool. And so you've worked across multiple different types of games, right? From Yeah. MMO type PC games to mobile games to consoles as well, all of it?
Yep. Pretty much. Though I will say, mean, I think MMOs was actually a very good place to start this kind of career because you kind of only really get to launch once. Sure.
There's a lot of emphasis on kind of forward thinking on how foundation should be established, how you can scale things up over the long term. And even as I've tried working in different fields, like console or mobile, I'm always kind of coming back to how to scale things over the long term, like the full life cycle of a game rather than just how we might solve things for the needs of the now.
Yeah. That makes a lot of sense. And so looking at today's game market, I feel like we don't just have MMOs, but we have a lot of live service games that also need to be set up for this. It gets released once like a Fortnite or a League of Legends or something like that that then constantly is being updated, constantly being, you know, adapted and changed. But like you said, that foundation is there and it's really hard to change that after the fact.
Yep. Absolutely.
Yeah. But I feel like that MMO experience is probably even more relevant today since there's so many more live service games now.
Yeah. Absolutely. I mean, this is one of the main kind of, I think, changes we're seeing in how gaming's evolved over the last couple of decades. Know, before a lot of people competing for getting a product out, getting into people's eyes, and then that being an experience they play for a while. Whereas now everything's more tailored towards like a game ecosystem, tries to keep people more engaged with that experience over the long haul.
Right. Exactly.
People I think play a narrower variety of games these days.
It does seem to be the case. It's definitely not true for me. I'm, you know, one of those people with like, you know, 2,000 things in my Steam library and I'm just always wanting to play Oh, new things to to see what people are doing. Right?
And when I say I've got those 2,000 games in my library, people are like, sure, but how many have you played? I've played more than a thousand of them because I just love seeing what people are doing. Like, what...
You know, I don't finish very many games, but I wanna see, like, how did you set this up? What did you do? What new thing are you trying? How are you innovating here?
Oh, absolutely agree on that. I'm probably in a similar boat of thousands of games in the library but yeah, I tend to gravitate towards a big chunky four x colonies in organization...
Those real time sync. Yeah.
Yeah. It's a bit weird to choose to kind of finish doing that for the day and then want to do it in the evening as well.
No. My team that did Pipeline in VFX, the three of us that worked in that department in the evenings would play, you know, games like Satisfactory or Factorio or things where we're building pipelines after hours too. It's just something about that got us.
Scratch is an itch.
Yep. Yep. Exactly. Okay. So so let me bring us back to our topic for today.
So you're working in tools design and something that we've talked about on this show in the past is when it comes to making automations or pipeline tools, there's always this push pull between, you know, how much time and resources am I putting into developing this tool versus how much benefit am I getting from it. Right? Like, how much benefit is this giving our artists or our developers? And then how do you measure that benefit?
Is it saving time? Is it making things more accurate? Is it just quality of life? You know, how do you measure those things?
And so I feel like there's two sides to that. One is how do you decide what to even start working on in the first place? And then the second part is, you know, how do you evaluate if it worked? Did you accomplish the goal that you set out to do?
Yeah.
And that's the part where when you and I were talking beforehand, you approached this in a way that I hadn't heard anyone else talk about before.
So making decisions in where it's gonna commit time and labor into like automation or improvements in any organization, especially if you're just starting out of this process. So you're gonna spend a lot of time trying to connect with end users, like listen to people, engage them directly based on their perception of what they say. They might go into other activities, we do retros and aggregate information together and make some decisions based on on your retro. All of that subjective experience is really important because it's a big driver of how people feel when engaging with tools or technology.
But you might find in a small company where you can kind of reach out and interact with most of the...
Most of your other colleagues directly. Cool. You know, no problem. You got a really, really close connection.
People are quite transparent in how they communicate. But as you scale up into larger organizations, like more specialization, more like cross functional teams, it can get quite hard to kind of keep track of number one who key stakeholders are, and also how accurate their reporting is. And one thing I've noticed a lot of companies I work at, they put a lot of a lot of investment into their kind of data tracking telemetry services primarily for understanding how their product is used. What they're selling...
Like how the end users are interacting.
Exactly.
And all this infrastructure is very rarely kind of made available for development teams inside the company to use. And this was couple of years ago I saw on some GDC talks and I was advocating for the if you're starting to build something and if you're starting a company for the first time, you wanna set up like just a very simple telemetry service somewhere just to give you a counter on what tools have been used, what pipelines have been executed. Because if you get that in early, just super superficial data, very kind of broad open to interpretation in a couple of different ways. If you start doing that early capture of data, you can then go and start looking at changes you make over the life cycle of your projects and the actual impact it's having on people.
So for example, example from today, we realized that there was a problem of like navigation meshes in the game we're working on and users were having to manually rebake the nav mesh every time. Unfortunately, this project doesn't have like a telemetry service set up for tracking this particular flow. But if we had, we would have seen, you know, why is this debug button been pressed 50 times a day? Right.
But this should be pressed.
Never. Yeah.
Well, like, just once or twice by developer testing something.
But like Right.
Having that telemetry there would have allowed us to discover these moments where the workflow isn't being followed. But also the users themselves didn't know that this was the undesirable behavior. This is just what they thought they had to do.
Oh, man. Yes. Working in pipeline, there have been so many times where, you know, someone finally complains about something and the answer is, oh, did you know you could just click over here and that would do that for you? And they're like, what? I... Like I didn't know that that was an easier way to do it. And so you're right. If I knew they'd been doing that all along, could have proactively talked to them about it or changed something about that to make it more obvious.
And I just never had that that, like, visibility into what they're doing and if it's... If they're actually using the tools like we designed them or if we need to redesign them.
Yeah. And being able to tap into that kind of treasure trove of kind of knowledge of what people are doing over time when you come to have a design problem is awesome because it's there retroactively. You can go back and then mine it to understand how people have been using things, how that usage pattern changes over time.
Yeah. And so I guess what I'm thinking about is when I was working in pipeline, one of the things that we set up was using Sentry to track when things would go wrong. Right? And so I was able to, you know, contact people and say, hey, I saw you just ran into an error using this publish tool or something like that.
And they would often be like, oh, I hadn't even noticed that I had an error yet. That's cool that you saw that right away. But I didn't ever set up any of the just proactive, you know, tracking of, oh, here's what they clicked. I was just looking for errors Yeah.
Which was still helpful. That let us be a lot more responsive. Yeah. But it leads me to the question of, if I were setting that up, what to track?
Because there's so much. Right? Is it every click? Is it how long things take? Like, how would you guide that?
I think depending on the context you're in, you're gonna tackle this in a different way. So I think, you know, if you're in a really, really small organization, you know, maybe just you and 50 other people, you don't do anything fancy. You're probably gonna make it bespoke. You know the events that you're interested in knowing about.
Is the project opened? Is the project closed? How long does it take to open the scene? You know, things like this.
You just can do custom measurements for questions you already have in your mind. But working in larger organizations where you don't actually kind of know what data you need until the question is confronting you.
Right.
And at that moment, you want the legacy data. I think in those contexts, it's actually kind of fine to go quite broad. I think again, if you're in a large organization, generally someone would like, okay, I want our tools to have like a consistent UI UX coherence to them. So you probably got somewhere a UI library of buttons that are stylized, widgets, spinners, everything else.
Right.
So you can get into that and just tag on really generic events. This was clicks. And then, you know, what scene was I put at that time for that tool. Then you're adding it at like a layer where it's non intrusive for the user.
And also it's non intrusive for you as a developer because you kind of just extending your base class somewhere. And again, inside a company, like when you're on the internal Internet, the extra network throughput, the actual volume of data you're putting through is pretty disposable. It's volatile anyway. You're only gonna probably retain it for like sixty to ninety days.
So, yeah, and it might seem like it's a ridiculous amount of volume.
Guess that was my concern. Yeah. That was just storing a bunch of garbage, but I guess I guess it is valuable.
I mean, in my experience, I've used the influx DB open source version and coupled it with Grafana normally, which I think there's also an open source version of. Yeah. So even if you've got, like, rudimentary knowledge of web services and micro containers, you can very quickly set up, you know, a Docker container of influx DB and Grafana. That gives you your template that can handle, you know, millions of records even on super simple light hardware.
Right.
You don't run into the scaling problems until you're dealing with millions of connections per second. And that's when you talk to your data science team and get them to help you.
Right. That that's where you're at the size of a company where you have a data science team.
Yeah, I don't. Yeah. And for most people, that's not gonna be the case. So just running a little tiny web server somewhere, just pipe data to it's super handy. And again, if it drops, if it dies, it just means you get small blips in the data. Because again, if you're using diagnostic or problem discovery, you don't necessarily need like a... The sheer volume and redundancy factor you might feel for your users and players.
Right. Right. Yeah. You don't need as fancy and, you know, horizontal scaling of a system.
It's just...
Yeah. Ping this server with what's happening. Now, that's cool. I'll have to look into InfluxDB.
I've used Grafana some, and I have found that recently my Grafana usage has gotten better because AI is good at writing queries for it. Because that was always the part where I'm like, how do I put together a useful graph or useful theory of what I'm looking for? And that's definitely changed that for me.
Now I'm thinking about it. I actually think, yeah, Grafana even has their own time series database instead.
In the influx DB bit might not be You might not know anymore.
Okay. Use pure Grafana stack, I think.
Yeah.
In last month's episode of this podcast, I was talking with Robin Jan Storm, and he was talking about the value he got from just standing behind someone watching them work and noticing, like, oh, they're having to do this, like, annoying clicking and dragging and rotating the camera every single time they're moving an object when that was not something they ever complained about because they were just so used to it. Right? They didn't even notice that. And what I think is really interesting about what you're talking about is it gives you a way to do that if you're not in a situation where you can actually stand over their shoulder or, like, tell them to screen share with you or something if you're remote, where you're kind of getting this larger set of data and finding things they might not even know to complain about because it just feels natural to them.
Yeah. I follow.
There's two answers I want to give to this. So first of all, that kind of example you gave of the the value of standing behind someone. Also, that's that's qualitative feedback. Incredible.
Whereas the way in which you might design kind of telemetry is more quantitative. On the qualitative front, I did actually have a colleague a number of years ago who had made this really, cool tool that basically just asked the user after every hour, hey, do you mind if I take a screenshot of your desktop in this And it was used for pretty much exactly that reason of wanting to kind of get a rough idea what people are doing mid workflow, focus, without having to stand behind them because maybe they're a different location, maybe they're a different site. It was something that we allowed people to kind of opt into or out of the day and never, you know, took screenshots about the user approving the screenshot.
But it still let us get a lot of like qualitative information on on how that user might be working. And whether we approached different users to get qualitative feedback, we could actually decide based on our quantitative data on what tools have been used the most by which people, you know, someone who uses a tool 20 times a day, you probably want to value their feedback over someone who might use it once or twice.
Right. Good point.
But like getting to the point where you can do that kind of intervention and ask people for qualitative feedback. I mentioned before, you know, the need to really think about how you can get generic data that you can then use to answer questions later. The process we generally followed for this was we sat with people, other engineers or product owners who had questions they wanted to answer. We just wrote the questions in raw form.
Like, oh, I wanna know if this tool is used. I wanna know which office users are coming from. And what you find when you start writing yourself just a list of chaotic questions, trying to answer specific things, broad things, whatever it is. When you then come back and look at that listing of the day, you can kind of go for every single question, what metadata do you need to answer that question?
So for example, a previous company, we realized that to infer useful things after the fact, generally, we would want to know the host name of the device. I would want to know the office location is based in.
We would want to know the like projects that user is working using the tool with. So we can detect via git which version of unity or the game engine we're using and then include that in the measurement data.
Right.
So yeah, basically you get generic measurements by asking a bunch of specific questions and then trying to work backwards to get your...
What data would I need to answer that question?
Exactly. And taking that approach, it's awesome because when you then do have like an incident, you've got a very clear historical record. You can see when problems and destabilization get introduced into your your projects.
Yeah. And I think for a lot of people, you know, when you're setting up collecting your telemetry, you're in the granular part of, like, what things should I include in here? And it's easy to just get caught up in those details instead of first zooming out to say Exactly. What are some things I might want to know and then come in.
Because I've also had the case where we've tracked data that we never once looked at. Right? Where it just didn't it didn't answer any questions that we cared about. And so it just was there for nothing.
Definitely the value if you ask your questions and work backwards.
Yeah. Yeah. That's great. So what are some examples of some things that you've discovered with this?
Over the years, this approach has given a bunch of different kind of discoveries. It's helped us identify things like just slow degradation to our editor's performance.
The kind of little things you're just adding a couple of like computational frames, extra overheads, slowly over months. You see that in the graphs. You see like, you know, 10, 20% degradation in Unity performance.
And you're talking about in the editor, not just in the product. Because I feel like we're tend to be more tuned to performance in the product and less so in our internal tools.
I mean, if you're doing it, for example, large environment generation Unity, you might start adding hundreds of game objects if you're not thinking about how you're structuring data well. And that does increasingly lead to extra load time, opening the empty scenes, just general latency because the empty is hogging more memory. So you get a much better way to intervene if you've got the historical record of which pull requests are introducing these little tiny cumulative delays because your users will not complain until it's breaking point.
Right. Till they can't work anymore, then they'll complain. Yeah.
So that'll be kind of one general kind of pipeline example. Another one might be, you know, if you're seeing a tool be overused or more than you might expect, it might indicate that someone's discovered a creative workaround. Hello? You might want to intervene to understand why they're using that thing more than you might expect. Sometime it might be for benign reasons, like just changes in workflow. And sometimes, yeah, it might be that someone's found out if I do this combination of things that I shouldn't do, then I get the result I want.
Interesting. Interesting.
But even this kind of work you can use to guide all kinds of decision making. And then so we obviously use Perforce in most of the companies I've worked out. It's used primarily for our binary assets. But p four v the desktop client, it's it's not the most friendly thing to people who are more used to working web interfaces.
Right.
And we realized that most of our kind of concept artists and a couple of different companies, they were just not versioning their stuff or they were relying on, you know, the hardware versioning coming from the IT team. When we spoke with them, we kind of realized the reason they're not showing in our telemetry data is just because they get other people to check-in the assets they want.
Oh, interesting.
Okay.
After sitting with them, okay. Yeah. If we provide you a web view of doing this, do you think it'll be more suitable for them? Which we did.
We adopted Perfult stem, made it run on top of the existing infrastructure. And now we see in telemetry that pretty much everyone creating content is universally using that product, which is good for us. Data's versions Yeah. Also good for them because they can choose the entry point that suits the user profile.
More engineering people like myself were fine getting into people via direct and manipulating stuff. But for people who are more focused on kind of creative, they just want to use a web view because it's what they use. Like Adobe Creative Bridge and like all these other tools, that's what they're expecting.
Right. That's the flow that they're used to. Yeah. Yeah. Something that I actually really like about DAM when we introduced that is that it also introduces a REST API that can be used for submitting things.
And so that's also something that I've talked to people about is like making a custom submitter plug in inside of Yes. Maya or three d s max or blender or whatever. Yeah. Exactly.
Submit for you. Yeah.
That's really good things to have tooling and telemetry because obviously of that you can... In your tool, you can actually point out that this commit or whatever coming from an automated process, and then you can map that against your telemetry server for for your pipeline.
Right. And so how much do you find that you're running around kind of putting out fires that the telemetry has shown you of like, oh, this thing's getting slow. They're doing this wrong. Or how much is it looking to be like, okay, where could we save everyone a few minutes every day or that kind of more investigative?
That is a good question. I would say it's it's varied wildly depending on the organization I've worked up.
Sure.
Again, companies has a lot more kind of direct intervention based on data and trying to achieve like specific benefits to the user. Whereas I think in larger organizations, it's generally been used as like an early warning sign that something could be imminently going wrong.
Thing I've got with the current version is like a a pass the Unity editor logs to figure out unity load times for domain reloads, package refreshing, things like this. And yeah, about a year ago, I was able to identify that we were hammering our auto factory server, what the unity remote package server but posted internally. Right. Yeah, we were just overloading that with requests.
But again, it was like preemptive. It hadn't really surfaced until we had a lot more packages been used, which then meant a longer resolution time. But it's kind of exponentially worse resolution time. And yeah, you've saw the first inklings and data, so we we were already kind of on it before it turned into an agent.
Nice.
I mean, that's awesome. That's the dream for a pipeline team to to prevent those sort of things.
Although... Okay. Maybe this is a little too, like, meta strategic of a question here. But...
They're the good ones. Yeah.
So I find a constant challenge if you're a tools developer or someone who works in pipeline is if you do your job too well, no one ever notices it. And they only notice when something goes wrong and it can be hard and I I could see a situation where if that did go wrong and you all panicked and pulled over nighters and fixed it, then maybe you're celebrated for that. But if you fixed it before it went wrong, no one notices. So is there any way that telemetry can also help prove the value of your team?
Definitely.
Although, I would definitely say if you're a pure technologist, and it's definitely a learning I've had to like, if you're more focused on the tech and the usage of the tech, sometimes it can kinda cloud your vision a little bit to the actual needs kind of your product or business directors. They weren't necessarily infer the same story from the data you're looking at. So I definitely think having this absolutely, it's a good way of communicating to kind of like higher up executives the value of your pipeline team. That you definitely do want to spend the extra bit of time to think about how you're going to present that data.
Because to you, everything being green, numbers going down is generally a good thing for someone caring about, you know, maintainability, predictability. You you don't want big spikes and error rates. Yeah. That graph going down, that's not seen as a good thing to more business savvy people.
Okay. Yeah. You're... A good point. Yeah.
It's a sweeping generalization, but it is still how you choose to represent data. You really do need to know your audience. So if you're presenting to other engineers, yeah, you wanna format things a certain way. If you're trying to present more to executive function, you probably wanna focus less on things like downtime and more on money saved effectively or time saved. That's always gonna be a line that goes up via automation.
Right. Right. That makes sense.
Subtle thing but very important and it did sink me once in my career by not focusing on that distinction.
But just not presenting it in a way that Yeah.
Was too focused on being a, you know, tech enthusiast and really into what I was doing and didn't really think about necessarily the narrative that was readable on the other side.
Right. And I think part of that too comes down to when you're looking at what things you're gonna try to solve or what workflows you want to change for the users of, like, what is the benefit of it? Is this saving, you know, one person five seconds every day? Or is this saving ten seconds, but for a 100 people, 20 times a day? Right? Like, the payoff for those is very different.
Yeah. Absolutely. Payoff is different. But, like, there is the whole kind of map is not the territory aspect to this.
What I mean by that is day is only ever gonna give you like a way of gauging how something is used because again, it's quantitative. It's not qualitative. So you you can only know that at some point this button was pressed or this computer flipped this bit. You can't really know the intent behind why that happened. So saving one person five minutes a day, if that person you kind of is in a very important role or is doing composition in the engine or whatever, then their time is actually highly valuable and keeping their frustration level down is good because if not their frustrations propagate onto everyone else they're working with.
Good point. Yeah. Yeah.
Whereas the inverse, like a fuel saving five seconds off a process, but that process is only benefiting 20 people, but it's taking you two days. Yes, probably not worth the time. But like if it's for 500 people. Yeah, absolutely.
It's worth the time again. Again, think it always comes back to the kind of original questions that you've got. What are you trying to solve? Because you will find how much to tip the balance in favor of one party or the other.
But ultimately, the data is just there as a tool. This is what I mean by, you know, it's not the territory. It's just a a lens through which you can look at the reality of kind of your organization, how they're working.
Now, that makes a lot of sense. So so far, we've been looking at, someone doing something wrong or like they're clicking a button to fix something that should be automated or load times, things like that. But something that was a constant challenge when I worked in pipeline at a VFX studio was just like getting users to do the thing you wanted them to do, like to submit in the right way or to include the right information, things like that. And we always have this sort of push pull between how much is it making the option attractive and nice and, like, selling them on the value of doing it this way, or how much is it making it so they can't do it any other way.
Right? Like That's a good one. You know.
Okay. Yeah. Yeah. Yeah.
And I'm curious about that, how that shows up for you, especially when we come to actually looking at telemetry of how this affects user behavior.
That is a great question and a meat...
A meaty one. So before I talk about the telemetry aspect of this, effectively what we're talking about is data validators. These can take many forms. I've seen people, you know, do as part of their continuous integration, certain checks, and if checks don't pass, then build fails and get back to the drawing boards.
I've seen people do, like, little tools that live at various stages along the pipeline that validate the scene state and then tell the user if things going wrong. Maybe they provide also fixes as well. I'd really recommend the validator approach. I think it's probably the best way of bridging the need for engineers to have things in a clean organized way and not taking away user agency, you know, their ability to read their message and go warning and go, actually, that's intentional in this case for it.
I made a mistake but the fact the helpful robots just told me I've made a mistake. I don't want to embarrass myself. I'll click the auto fix button and check. I get what I want.
Right. So it's still giving them a way to override it and say like, no, this has to be this way. Okay.
Yeah. Think generally problems just have two broad categories. You've got problems that are predictable and the solution is predictable. And then you've got more general, no human needs to intervene here to Right. Unpack or solve the problem.
Where telemetry can fit into this is if you do build yourself a small telemetry server and you build yourself a validator, you start tracking the frequency of the different failures you get.
Right.
You start tracking the amount of times that someone actually uses the autofix associated with a particular failed check. And what you get from kind of monitoring that information is number one, if you start seeing the same problem showing up over and over and over again in validation reports, it's an opportunity to provide training. It's pretty clear that hey, so the users are either relying on the tool to certainly check their problems or there is a problem and how they're working, potentially a training issue, potentially an opportunity to kind of coach and help people better themselves.
Similarly, if you see certain checks failing frequently and the auto fix not been used, that's a sign that maybe something's wrong with the auto fix or you're getting some sort of false positive and you need to kind of get in there and update workflow documentation and a process of things like this. I definitely think, you know, if you're intending to explore telemetry, data validate is this way you can get the most bang for your buck very quickly. And again, especially if you take the mentality that problem should only happen twice. And then the second time it happens, you add a check.
Right. Okay.
Yeah. You can get very chaotic pipelines under control quite quickly with that mentality.
I love that. Yeah. That's a good rule of thumb. Problem can only happen once. The second time you have to fix it. Yeah.
Or at least automate the discovery of the the problem.
Sure. That makes sense. Yeah. Also, when we think about telemetry for, you know, a game or a software product or something for the end user.
Yeah. Part of that is also looking at things like, oh, we made this button, you know, blue instead of red, and it gets clicked more or less. Yeah. Is that is that... Would you even take it to that level for an internal tool of, like, making it more user friendly to encourage the right behavior?
Personally, probably not. Do you think that there is value in having, you know, an agreed style sheet for tools. Generally, I've seen when people are doing tooling for their business, you're gonna be building different bits of technology for different parts of your pipeline. And if you can agree on a visual kind of standards, it improves the user experience.
People know specifically what icons are gonna do. They buried up some usage of text or color indicates. It guides them in It a also just feels more professional as well. It's really nice to be part of a team when you actually get, you know, you you start your company, you start getting people together, people are working all the immediate fires.
And then at some point, like, cool. We can now start to be more cohesive. And the day that you get a good, like, visual kind of style and iconography down, it just feels good. Even as an engineer, it feels good to have things that just look nice.
Yeah. I definitely wouldn't use the data science side for internal tools like UX, but obviously, large organizations will definitely be looking at how players interact and, like, there's little differences in color and and everything else related to marketing, really.
I I have found that one that I often try to understand is, like, the wording of messages to Yeah. Be like, is is this clearly communicating to the user succinctly, right, in as few words as possible, what this actually does? And that is one where I'm thinking now doing some, like, a b testing internally would even be interesting. I never had a system to do that, but I I like that idea.
I was reviewing actually some feedback from the the projects I'm currently working on. I realized, yeah, pretty much all... Like, none of our users are really checking the errors and warnings in Unity because they don't they don't indicate what's required. So it's like, oh, that's a problem for an engineer. Was gonna ignore that. Yeah. So I am starting to think that, yeah, we actually really should get into the habit of actually number one, make a knowledge base and then have errors and warnings reference a knowledge base article. Just allow people to kind of self serve a little bit more.
If I were to do that and if I would put telemetry on it, would obviously monitor the click through rates to see if the knowledge base are Are they using it?
Used. Yeah. Exactly.
Good point. Good point. Yeah.
And maybe if they're not clicking it, you realize You've wasted your time.
I'm wasting my time. Or maybe I need to even just, like, surface the key parts of that knowledge base right in the message instead of having to click or something like that.
Exactly.
Yeah. Alright. So this is fascinating to me. I love data and so this obviously excites me about how we can apply data for good. But we're getting toward the end here, and so I want to ask you a couple questions. So the first is, if you could travel back in time to yourself at the beginning of getting into tools design, what advice would you give your past self?
I mean, definitely number one, yeah, take an interest in in how you can keep track of stuff, even if it's just a simple clicker, a counter somewhere. Also, I would definitely say that if you're if you're not like a pure engineer, if you are coming from more of an artistic background, learning how the services work will unlock a lot of knowledge for you, you know, to learn how to actually containerize and run like a small telemetry server and this associated, like, visualization platform. You will learn a bit about Kubernetes. You'll learn a bit about Docker.
You'll learn a bit about orchestration. It'll give you a wealth of knowledge and kind of socket based communication, and that can lead to you making better tools. And obviously, right now, we're in the middle of this AI revolution. A lot of companies are switching from like these MCP servers just go, oh, why don't we just use a simple REST API?
Right.
And so yeah, getting exposed to telemetry is a really good way to learn microservices and microservices are a really good way to start making applications talk to each other and make your pipeline better. Yeah. And you gotta monitor it being used. Right.
Right. Yeah. You get to see that it's being used. I love that. And so now the question... I'm gonna rephrase this one a little from how I normally ask it. So this is Yeah. What advice would you give to someone who's...
You know, has some kind of interest in pipeline tools or this kind of automation integration, but they're just starting out today and they're in, like, a smaller indie studio? Like, where would they start? Because I feel like this is the kind of category of thing that most indie studios I talk to assume that's only for big studios. Right?
This Yeah. We we can't we can't do this. Like, we can't make custom tools. That's not really worth it until we hit a certain size.
So what what would be your advice to someone in that position?
That's a good question again. I'm I'm not sure I have a clear answer to it because I'm I'm actually still wrestling with this problem a little bit myself. Depending on whether you're working in an indie studio or even in a large corporate organization or small prototyping team, you're gonna be very resource constrained time time poverty is gonna be extreme. You really want to make sure that you're kind of applying the best use of your time.
And there's different mechanisms obviously for for navigating that. But again, having just a clear counter on what is being used or having the ability to then retroactively go to kind of data archaeology to understand things. It's one of these things you don't appreciate the value until you need it. You know, really really...
Even if you're doing indeed doing a playtest, you you just focus. I'm gonna live my gaming experience, I don't go get set up with like a basic Google Analytics platform or whatever. Then you're blind when you put your thing out there. You're relying on perhaps your Discord community or relying on kind of nondeterministic measurements.
And you do want that, but you do want the synergy of both. You wanna be able to take people's quality of experience and then compare it against the quantitative kind of data you get from from what you're building.
I think that makes a lot of sense. And also because I've noticed that when it comes to people actually taking time out of their day to give you their opinions, like on a Discord server or something or emailing you or whatever, that they they have to care enough to take that extra time and effort to send that to you. But also because of that, you're getting opinions from like this very small passionate niche of people that something one wants might be the opposite of what the 99% of users that don't take the time to write in actually want. Yeah. So that telemetry helps you to kind of, you know, validate those things, you know. Is this is this something I should address or not?
Absolutely. And again, that's one of the really good kind of use cases because like exactly as you say, you already know what people are telling you.
You don't know Yeah.
What's not been reported. But at least if you're able to compare what people are telling you with like the historical record, You can find gaps. You can find when maybe something's been misrepresented or it's just not been represented at all.
Yeah. Yeah. Yeah. Maybe maybe helps guide the questions to ask your community as well.
I know that community engagement is so important for indie games and that's often what people like about indie games is they... Their feedback actually gets listened to in a way that it doesn't feel like it does in, you know, a big triple a game in the same way.
As as someone who loves games, I will also say though for any indie people listening that, you know, don't kind of sacrifice your kind of creative ideals on the the altar of data. Because what the masses want might not actually translate to a fun original experience.
So that's okay.
That's good advice right there.
Yeah. Commit to your dream and see if your community finds you.
Don't don't bend entirely over to To just use your data.
Yeah. Yeah. Yeah. Exactly. Oh. You send it with something average that doesn't kind of cut through it in the same way.
Yeah. No. That's that's a good point. Yeah. I'd say that's the also the caution about letting AI get involved too much in your design decisions cause it's always gonna gravitate to the average of everything.
I was literally about to say, I mean Yeah.
Personal opinion, we're probably gonna see, you know, huge volume of new games appearing on the market, but we're all gonna be kind of mid. Yeah.
The... That's... I'm... It's already happening. I'm already seeing it.
Yeah. Exactly. And the things that are gonna rise to the top are gonna be things that have been made like with passion and with quality and like, you know, of level of sincerity. And true, AI might be part of creating that experience, but it's gonna be the people who are like, this is the experience I wanna provide. This is the story I wanna tell. That'll rise to the top.
Yeah. Even just the the...
This is the feel I want the game to have. I've found that so often it's hard to even find good games just from trailers because you're like, I don't know what it feels like to play this game. Right? If it's a faster paced like a bullet hell game or something, it's like all feel.
Right? Or like a platform. It's all feel.
It's been interesting to see like demos return a bit. Yeah. As a means of getting your products out there.
Yeah. Absolutely.
So going back to the the question of, you know, the advice for someone starting out in an indie studio, making custom tools, things like that. I'm curious your thoughts on this, but my thing I've landed on recently that I'm trying to spread the word about more is taking the time to get your CI set up.
And what I mean by that is even just if it's a a batch script, you you just run that builds your game so that you have like a consistent really quick easy way to build it. More ideally, it's like every single time someone submits to Perforce, have it automatically build the game so that if it breaks it, you know right away instead of not knowing till a week later when you're trying to do a demo for someone. But if, like, get that from day one is my sort of subclass lately. Yeah.
Would you agree on that one?
I'm in absolute agreement of that one.
Yeah.
Again, if you're tiny one person, maybe it seems like a necessary overhead. Like if you're one person, you're relying on your computer's setup. And so literally just by setting up even a basic CI, okay, you're at least removing your computer and your computer state from BIM any way impacting what's getting built.
Right. Yeah. That's a good point. Yeah. Yeah. Like I have a...
Like an old old laptop that I've kept around that I have just kind of as a little Yeah. Build server where I can have it do things like that for me. And it takes a little time to set up, but also once you do it once for any game jam you do, any new project you start, it's there. Yeah.
Yeah.
Again, thing. If you do do the telemetry thing as well, you plug your CI into the telemetry server and you start profiling your build times over long periods of time. You start noticing...
That's an interesting point. Yeah.
Start noticing when complications in your build pipeline start leading to longer turnaround times. And again, that stuff's really important, especially if you're working in, like, a more distributed team with a dedicated QA, like actual build speed is one of the most important metrics for avoid avoiding problems of concurrent people working.
Yeah. I'm I'm also a big fan of testing early. And so having the automated build means literally every single day, you have a build that works. Even if it's not the very latest submit, like maybe that one broke, but you've got the one before that. So anytime you have a person that you can rope into testing your game, you've got a build that works that they can run.
And it's also nice. Same thing, you can bisect that way. You get the two timestamps between when an instant or a problem emerges and you just bisect until you find the specific commit that introduced it.
Oh, yeah. I love that. That's great. Awesome. Well, thank you so much, Joe. I feel like I could talk about tools and data forever.
Obviously, that's why I do this podcast.
Yeah.
That's why I do this job.
But I'm gonna bring it to a close here. So thank you so much for taking the time to share all this with me, Joe. I really appreciate it, and I think that our listeners will as well.
Thank you. It's been a real pleasure to go to come and talk about this. It's definitely one of things I'm most passionate about.
Awesome. Yeah. And if people want to reach out to you or or get in touch about anything, how can they do that?
You can contact me by email or you can find me on LinkedIn and I'll pass you those bits of information to include in the video description.
Awesome. Great. Well, thank you so much. If you enjoy this kind of in-depth conversation with leaders in the media, gaming, and visualization world, be sure to subscribe so that you get new episodes as soon as they come out.
I work hard to find amazing guests like this that are willing to share their journeys and their knowledge with all of you. And taking a moment to leave a comment, give a good rating, that's the best way to let us know that you enjoy this content so that we can keep making more of it. Reach out to me via email or LinkedIn with feedback or guest suggestions or to connect if you would like to be a guest yourself. I'm always looking for great stories and I love learning about the many ways that game and real time technologies are being used today.
Links for that are in the episode description. Be sure to check out perforce.com for information about our full p four platform based on the industry standard Perforce p four version control. Special thanks to our production team, Ella Reiswig, Emily Matlak, Kaylee Torres, and Chris Perez.
I'm Jace Lindgren, and I will see you next time on In Development.