October 31, 2014

Build Your Own Monster in the Perforce Workshop

MERGE User Conference

Build your own monster in the Perforce Workshop

It's October 31st, All Hallows' Eve.

Tonight I'm going to listen to my favorite witch Stevie Nicks sing Rhiannon and skulk where the boundaries between the worlds of the living and the dead overlap. Not only a place where the spirits of P4Win and P4Web wander around looking for bodies to inhabit, but a place where the coolest and latest new Perforce tools are being actively developed. I'm talking about the Perforce Workshop where I can get the right tools and inspiration to build my own Perforce Monster.

In early Summer 2014, I was cool as a cucumber when I learned that my manager Matt Attaway volunteered me to do a talk at the MERGE 2014 conference. Prepping for a talk is a lot of work and it's even worse when you don't know anything about the topic. Matt volunteered me to talk about the Perforce Workshop. I had never even seen it, let alone written a line of open source code. I felt like Br'er Rabbit. There was only one way out of this. Get Matt to throw me in the briar patch. See, some of my biggest strengths are my directness, my willingness to ask questions and my tenacity. So I told Matt I would gladly stand on that stage if he stood up there with me and answered all of my questions about the Perforce Workshop. He agreed.

When all was done and said, I ended up with something like 23 questions. Matt did a great job answering the questions about all of the "technology and stuff" in the Perforce Workshop. My questions covered the logistics of my development environment options, my development tool options, the Perforce APIs, open source licensing, which cool tools were already there for me to use, and the future of Perforce and open source.

If you'd like to read the transcript of the talk, see below. If you want to listen to it or get the slides, click here.

If you want to see what the Workshop is and just wander around at your pleasure, click here.


Transcript of MERGE 2014 session: Build Your Own Monster in the Perforce Workshop

Steph: Hi my name is Steph Turner and I'm an Open Source Developer in the Perforce community. If you use Perforce tools or you write your own or you want to see what the coolest kids are writing against Perforce, then you're going to want to watch this video on building your own monster in the workshop.

Gabe: From the midst of this darkness a sudden light broke in upon them. A light so brilliant and wondrous, and yet so simple.

Matt: That's right. Sound effects. This is serious business.

Steph: Scream, everyone scream! This way they think we're having a great time over here.

Gabe: Change the source code from plus to minus and from propriety to open? They succeeded in discovering the secret of bestowing life. They became capable of bestowing animation upon lifeless matter.

Steph: Now the lights come on. The lights were off before.

Matt: Yes, pretend.

Steph: Pretend.

Matt: Hi everyone. My name's Matt Attaway.

Steph: And I'm Steph Turner.

Matt: Welcome to our talk on the Perforce Workshop. We're real excited to have you here today.

Steph: Some of you guys might recognize me from last year. I did a talk on the Holy Grail. My manager came to watch this talk and it was standing room only, so you guys can stand and we can recreate that. I saw him come in and my heart started racing. I'm going to lead the way and put my phone down. I think you all should do that too. My manager was standing there in the corner and then I saw him in the middle of the talk walk out. I was like "that is so wrong." This year, I have my manager standing on the stage with me and if he walks off, he is in trouble. Matt, how long have you been here at Perforce anyway?

Matt: I have been here for about 14 years at Perforce.

Steph: 14 years? That's like half my life.

Matt: Man, that's shocking, isn't it?

Steph: Matt has worked in every group. Because he's so good, they want him everywhere. I've worked in every group because they're trying to punt me off and turf me off to the other groups. Matt finally said "I'll take her."

Matt: Steph, how long have you been here now?

Steph: I've been here 11 years. I missed my line and Matt just picked me up. He's awesome. I've been here 11 years. I have a couple go-to people when I need help, which is all the time, and Matt is one of them because he's been in QA and Engineering and Product Development, and now he's actually leading the community group. I'm going to look at my line. Gabe, line?

Gabe: Tricks, tricks!

Steph: Tricks! Matt is my go-to guy and he knows so much about writing against the server and all that stuff. You have a lot of tricks up your sleeve.

Matt: Definitely. I've built a number of products over the years at Perforce and certainly built a number of open source tools. Perforce is my life at this point.

Steph: I am proud to say I spent nine years writing folder diff. So if you've ever seen it, if you've ever used it. Nine years to write that product. I worked on P4, the Frankenstein if you've seen the movie Young Frankenstein. We call it Frankenstein. What am I saying? I was thinking that now that I'm working for Matt in the Community group, I'm going to have a chance to work on something other than P4V. I'm very excited about that.

Matt: Right on. You should do it out in the Perforce Workshop.

Steph: You're kidding, right? Isn't that a place where we put all the expired products?

Matt: It double flashed.

Steph: We're going the wrong way. We're on the wrong talk. This is called Steph sweating out. It's odd that I'm freezing but sweating at the same time.

Gabe: You're talking about the ravings of a nonsensical mind! Dead is dead.

Matt: There's no way. The workshop is absolutely rad. We're in fact doing a lot of our new development internal to Perforce, actually out in the workshop so you can follow us and see what we're doing.

Steph: In June they were talking about who wants to do a talk? And I said, "the conference in September" and I got my kids going back to school, and I'm taking a sabbatical. I've been at Perforce for 10 years. They give me a month off. I'm going to have all of August off. I don't want the stress of doing a talk or anything. So I just lay quietly and I went away on vacation and came back, read my email and there was an email from this person I didn't know. It said "Thank you for your submission to the Merge talk. Your topic is blah blah blah." I thought "this must be a mistake because I'm not doing a talk this year." Matt was on vacation so when he got back, I was like "Matt, I got this weird e-mail and I have to do this talk." And he said "Oh yes, I forgot to tell you. You're doing a talk." And I said "Really? On what?" And he said "The Workshop." I said, "Matt, I've never even written a single line of open source code. I don't know anything about the workshop." And he said "Well, go to it and start learning." So I came up with 23 questions to ask Matt. And I said "If I'm going to do this, you're doing it too." These are probably the first 23 questions that you guys would ask about the workshop.

Matt: Alright, let's do it.

Gabe: Throw the third switch!

Matt: Let's talk a little about the workshop and its architecture so you can get a sense of what we've built out there. At the end of the day, it's just a basic Perforce server that you can stick whatever content you want in it. Open source projects. This is actually something that we've had around for years and years. It was called the Public Depot before. It's probably one of the oldest Perforce servers out there. It just had its 16th year of operations about a week ago. It's a very old Perforce server. It needed a big face-lift though. Perforce server alone is not going to get you a long way when it comes to being able to communicate and collaborate on products. Conveniently, when I was looking to start up the Community team and really build a real Open Source portal, the Swarm team came around and made a thing for communicating and collaborating on code. So I said "Okay, I'll take that." We use a customized instance of Swarm that's been extended to suit the needs of an Open Source community. Swarm is fantastic but it wasn't really built to have a whole bunch of random people show up and create accounts and start hacking on things. We've had to harden it in interesting ways and create account management tools, and a number of other widgets to make our lives better. Of course you can use whatever Perforce clients you want with the Perforce server itself. You can use P4, P4V or Eclipse, or whatever you like. In the next couple weeks we should have Get Fusion support as well. We're just waiting on our SSL certs. Yay for Zig in the back.

Steph: Yay. Zig, stand up. You don't want to stand up? I worked with Zig for a really long time. Actually, when he first came to Perforce, I started working with him and I would stand and he would kneel next to me and we'd be eye to eye. Is the Perforce Workshop only for Perforce tools?

Matt: No. Anyone's welcome to come and put stuff inside of the workshop. We have some people put their music in there recently, that they're working on. I actually know a person from the Community team who does robots in his spare time so he puts his diagrams, 3D models, and instructions all inside of the workshop so he can share it out with other people. It's really cool because Swarm actually has a built-in 3D model viewer so you can see the models in the browser and rotate them and see the lighting on them. Anyone is welcome if they want to use Perforce for their projects.

Steph: We used to have the Perforce Depot. Are there any other hosted Perforce opensource depots out there?

Matt: Not really. There's a new Helix project is coming along and there's Assembla out there but they're really targeting closed source development. You're getting your own private server. The Workshop is probably the best place to do open source Perforce space development.

Steph: I have 22 questions because he just answered one.

Matt: We're going to see how many I actually [inaudible 00:08:32]. The red button is right where my finger is.

Steph: The other question was "What's the difference between Helix and Assembla?" And I guess you're saying that it's . . . . Matt just did the slide. You guys aren't supposed to see this yet.

Matt: Me and the clicker are having some issues. We're going to work through it.

Steph: It's going to be okay. I am writing an open source tool now. I'm writing P4 Bisect. I don't know if any of you guys know what Bisect is, but it's a tool that I wanted for a really long time, so now that I'm in the open source community, I get to write this tool. [inaudible question from the audience 00:09:11]

Steph: I don't know. What's [inaudible 00:09:16]? No, [inaudible 00:09:19] is just putting them together, right? No, Bisect is

Gabe: Talk about it later.

Steph: What did you say?

Gabe: Let's parking lot that one. You can talk about it later.

Steph: You know what? We're going to parking lot that one for after the talk.

SPEAKER: As the minuteness of their parts formed a hindrance to our speed, we resolved therefore to make the creature of a gigantic stature.

Steph: I have a bunch of questions about development and a build environment. Are there any rules about OS or platform that I need to know about?

Matt: Certainly not. You're welcome to contribute code from any type of OS or platform. Whatever you're working on, we'll be happy to host it.

Steph: That's great for everybody that likes those odd platforms like Vax or something like that.

Matt: Right. VMS. Any VMS fans in the house?

Steph: I'm going to go out on a limb and say there's probably no rules for my development tools? I use Xcode5 right now but I've used Eclipse and my favorite, of course, VI.

Matt: No, of course not. Any tool chain that you need to use, you're welcome to use. You can suggest to users in your ReadMe what they might want to use.

Steph: Is there any support for automated builds and testing?

Matt: When you create your Swarm project, you have a couple of fields. If you've ever used Swarm before, in the project field, there's a place to say "anytime a review comes in, please kick off a set of tests at this service." You can easily hook it into something like CloudBees or Ship.IO. You could have your project have its test runs anytime someone contributes to it. Many hosting companies have free open source hosting. They're happy to have your builds there for free.

Steph: This next one's a little bit tough question to ask. I didn't know how to phrase it. Size always matters, right? I was wondering, is there any kind of size limit to my project?

Matt: No, not at all. It's Perforce, right? If we had size limits, that would be pretty sad. Certainly if you started showing up with a terabyte virtual machine images on a regular basis, we'd have to have a conversation. We actually have people who are storing their base boxes for Vagrant all up in the cloud. Actually the SAS team does that. We use that to store a number of our virtual machines and it works great. Let's be honest, in the grand scheme of things hard drives are cheap.

Steph: So there is no size limit?

Matt: No, again as long as no one gets too crazy. As long as it's part of your project and you need it to succeed, then certainly we want to support everyone.

Steph: So I can put my music in there?

Matt: Probably not.

Steph: Can I put my art?

Matt: If you're making music, sure. We'd love to host your music. If you're open sourcing it, creative commons stuff. Bring it on.

Steph: For my P4 Bisect tool, what else do you have in that Workshop for me?

Matt: What else do we have for you? Certainly we have the APIs that you're going to need to be able to talk to Perforce. The API situation is a little interesting and I'm probably stomping on another of Steph's questions here. If you're looking for the supported builds of any of the APIs like C++API or P4Java, you're going to want to go to either FTP.Perforce.com or the Perforce download site. On www.perforce.com we have the Downloads page. That's where all the supported binaries are, and builds. If you're looking to see our source code, and soon we're going to have the source code for all of our APIs up there, including P4Java which I know a lot of people will be excited about. It's all available there, so you can grab it, make changes, and even contribute them back, depending on the project.

Steph: Did you hear them clapping? Can you guys clap louder so they can hear us clapping? It's so great that I get to have so many people in one room indulging my narcissistic issues.

Matt: It's really your dream come true, isn't it. [inaudible audience comment 00:13:18]

Steph: Thank you guys. I do. I will. Free beer from here. Those APIs are really going to help me a lot because that's what my line says. What would be even better. . . has anybody here seen Young Frankenstein?

Matt: Okay, that's good.

Steph: I want the enormous Schwanzstucker.

Gabe: Isn't that my line? Build a monster with an enormous Schwanzstucker!

Steph: I just stole Gabe's line. It's like we're passing it ahead.

Matt: It's an improv show. Conveniently, if you want to get a sense of how we're using all these APIs, we are really trying to open source as much as we can. We have tons of internal tools we've built over the years that we're trying to get out there, and obviously some classics. P4Win and P4Web. All of the source code is available out there today and people have been hacking on them and working on them. Someone's actually brought in Stream support for P4Win and other fun stuff that need to pull into the main line. Piper, which is a tool that looks a whole lot like Desi, wink wink. You can find that up in the workshop. All the full source code is there as well as a build of it for the Mac. P4Search, which is a fantastic tool that Piper relies on to do its tag searching and all that jazz. We use it inside of Commons and other places. The full source code for that is available there, was also developed out there.

One of the things that I'm most excited about is the SDP. If you're not familiar with it, the SDP is the Standard Deployment Package. It was a set of scripts that we've had for years that make running a Perforce server really really easy. We use it in the workshop and I'm a little suspicious. I know Perforce. I don't need your fancy scripts. No, I need their fancy scripts. They're really fantastic. They do offline checkpoints and backups for you. They keep your downtime to almost nothing. I highly recommend checking out. It used to be a consulting only package. I'm really happy to say it's now completely open source and we're going to be doing future development on that in the workshop as well. You can follow that, keep up to date, see the changes that are happening, and even contribute back if you see changes that need to be made. All the contrived APIs are out there. And Chronicle, which was our CMS system. It uses P4PHP and it's entirely available out there. We're hoping to get much, much more. The goal as I understand it right now is we want to open source every client we have. It's going to take some time. Most likely P4Java and P4Eclipse will be the next things to show up out there. We also have a great tool that Steph is working to clean up and package up which is Pixar's Linkatron. If you've seen any of their talks, there's a good chance you've heard them talk about it before. It's something that makes all of the archive files on the Perforce server available through an NFS share automatically. It's a really nice piece of kit and that will be available as well.

Steph: We just got two more people in the room. Three. Can we just clap one more time? Just to let you know, be careful because if you walk out of those doors before I'm done, I'm not even going to tell you where I'm going to go with that one. I don't know about you guys, but I love ripping off code and I'm pretty sure I'm going to rip off code from some of these apps here. Because I wrote some of the code. Is it considered ripping it off when you've written it yourself?

Matt: I don't think so. We also license everything very liberally. Everything is under a BSD2 clause license. Please rip us off. Christopher always jokes that if someone can make more money off of our code than we can, then go for it. We're happy to have it.

Steph: Chris is actually here. This is the first talk he's ever come to mine. We could clap again but we don't have time. I am going to use a lot of other open source code like QT and Boost and I'm doing web development and I actually found a way to do it with C++. I'm one of those rare people who uses VI and C++ on web stuff. It's called Witty. It's a pretty cool library. We do have to check those into my third party. . . ?

Matt: You could if you wanted. I certainly will do it for stuff I think is hard to find. But for the most part these sites are all active and they're getting bug fixes on a regular basis, so I like to redirect people to their proper sites. Also the documentation's available there. It's a good thing to go in your ReadMe. Obviously if you're using a language like Ruby where you have Ruby gems, then by all means, have a Gem file so they can all suck that in and be nice and versioned.

Gabe: Life. Do you hear me? Give my creation life!

Steph: Matt, where do I actually put my code?

Matt: Totally. When you create an account for the workshop, we take your username and we give you a space in the depot which is Slash-Slash-Guest, your name here. And you are the only person who is allowed to write to that space. Full control.

Steph: When do I put my code in there? From the first line of development, or after I've got it all working? Because I don't know if I want people to see my code while I'm working on it.

Matt: I tend to like to put my stuff in early on so people can see all the warts and see how it evolved. Sometimes when you're trying to rip off someone's code later on, it's easier to see why they made the choices they made. But certainly if you're the kind of person that likes to make a nice, clean thing, by all means. If you're going to share, share. However makes you feel comfortable.

Steph: How do I actually get my code into this Workshop. Is there a way through the Swarm interface?

Matt: Not right now, although we do have plans in the future to have drag-and-drop submits right from the web interface but for now what you'll need to do is set up a Perforce client. We actually have a trigger in the back that automatically creates the client view for you in the right spot and you're off to the races to submit.

Steph: I create my dev branches in my guest area and then I can integrate them to where my main project is. Are people going to be able to see my dev spaces too?

Matt: They can. One thing to know about the workshop is that you can see everything. That's actually one of the reasons we haven't rolled out Commons. Commons has the concept of a private space and I never want anyone to think that they have privacy out there. There's no privacy. This is just like a Github as far as open projects go. It's all open, all visible.

Steph: I did actually get on the website and look at the stuff, the workshop because it's my team doing it. I saw that everyone has these project pages. "My project is about blah blah blah." Do you have to have one of those? How do you do it?

Matt: It's certainly not required. At some point we'd like to make that process very easy to do so you'd want to no matter what every time. Generally if you're sharing code, you want people to find it. That's the whole point of publishing it in the first place, so going through that creation process is really good to help service it and as we add new functionality, we'll mostly attach it to projects. That gives us a nice bucket to hang everything off of. I highly recommend creating a project.

Steph: I have my project in there. What's the expectation for me to have to maintain this and support this project once it's in there?

Matt: In the perfect world, we'd all be able to support everything we shared. But a lot of times we make something cool and we want other people to be able to take advantage of it but we just don't have the time and energy to take care of support questions for stuff that we're sharing for free. It's just hard to do. We have day jobs. It's always good, if you really want a project to grow, to curate it and interact with people who are talking about it. But if you can't, that's life. That's open source development. The great thing is that they have the source code. They can do what they need to, hopefully.

Steph: So here's the sensitive question. I want to get paid for my open source code.

Matt: Don't we all?

Steph: Sometimes when I download open source stuff, they're like "Hey, put your donation here." Can I ask for donations on the Perforce site?

Matt: Of course. You're welcome to have a GitTip link. All the cool open source kids do it. It would be really weird if we were like "No, I'm sorry. We're above money." You're welcome to have a GitTip link. Don't expect to make much. But you might get a beer a year.

Steph: Here's another question. Chris is here, maybe he'll answer this. If you pay me to write an open source tool for Perforce, can I put a GitTip up to ask for donations for myself?

Matt: No you can't.

Steph: I can't do that.

Matt: I don't think your boss would approve of that.

Steph: Okay. That all sounds pretty cool but I'm very, very anal about my code. What if somebody dumps some bad code into my project?

Gabe: Stand back for the love of God he's got a rotten brain. Are you saying that I put an abnormal brain into a seven and half foot long, 54 inch wide gorilla?

Steph: Laugh!

Matt: Obviously, since you're the only one who has write-access to your space, no one can mess with your stuff. They can certainly branch it and do horrible things to it that might make you cringe, but no one can touch the code that's in your guest branch, that's your space.

Steph: Then how do they actually get it into my project?

Matt: Great question. I've seen two workflows that are fairly common in the community so far. So if the people are just making a few line change, like a quick patch, what they'll do is just sync the code from the other person's space, map it into their workspace and then they'll make the change and they'll shelve it for review. Then Swarm will notify that person that "hey, someone's got a patch here. You want to take care of it?" Then you can look at it, talk about it, and then right from the Swarm interface, you can click and accept it, and push it right into the project. It's a sweet workflow. For larger changes, we have this great guy named Dan Tran who's been completely rewriting the P4Maven plugin, it's beautiful. He's branched the code off into his own space and he'll work on it for days or weeks and then we'll take contributions back into the main branch when he's ready. It's really up to you. Generally it's nice to be able to pull it all in and give yourself the space to work on it. It'll be interesting to see how that changes when we have the distributed functionality into the server. The workflows might change a little bit.

Steph: I own the code and that means I have complete control over what I take in or take out? I'm thinking, this is going to be really good for me. I don't have to listen to my Project Manager or anything like that because I have control over this stuff?

Matt: That's right. That's the great thing about open source.

Steph: I have complete control over this.

Matt: You do.

Steph: If I own the project and the code, what about licensing? I use a bunch of open source code. Who here uses open source code? Be honest about this, how many of you guys really understand the licensing? Or do you think, "I'll just use it and if somebody comes after me later, I'll figure that out." I'll be the first one. I always think maybe I'll make a million dollars and by the time they come after me, I can pay them. What happens with licensing?

Matt: It's your code. You own the copyright. You can license it however you need to. We use a really permissive license. It's important to us at Perforce to let people use our code however they want to. We've certainly been on the other end where we've seen some really nice code and were like "GPL. Oh well, I guess we'll write something similar." The BSD licenses, the Apache license, and the MIT licenses are the most permissive ones. GPL is a great license as well but it's got the copy-left thing so any changes you make to the source need to be shared. LGPL's kind of a weird intermediate beast where you can link in the code and not have to worry about having it affect your code at all. But if you make any changes to the library, you're going to have to share those back.

Steph: What is in it for everyone here? Why is Perforce doing this?

Matt: For us, we really want to give the community a central place to come together to share their knowledge. I guarantee everyone of you have written the same script at least once. Guarantee it. There was some sort of thing where there was a client trigger to morph the client views or some sort of cool thing to parse logs in the way you like. We all have really fantastic tools that are easily shareable and other people could take advantage of. It's almost criminal to me that we all have to reinvent the wheel, so I want us to have one place where we can come together and share these great tools, help make them more visible, and give a place for people to share their Perforce knowledge. We have administrators, we have users, we have a lot of different knowledge levels in the Perforce community, and it's nice to bring that all together.

Steph: Can I ask a question? How many of you guys out there would come to the community to look for some tools that other people have written? That's pretty cool. Look at that. They all raised their hands. Clap, clap, clap. Thunder. We need thunder. Now here's the hard question. How many of you guys would put tools into the community for others to use? That's pretty good.

Matt: I'm monitoring all of you. There's a camera back here taking all of your pictures.

Steph: We expect to see 15 new accounts on the workshop.

Matt: Of course with Perforce, the thing that's important to me is it just brings everyone together. In the closed source world sometimes it's hard to build a community and get people together and in the open source world it's easy. They have this big thing and they're building it together. In closed source we don't always have that but I think it's important, especially when we're talking about development tools like Perforce. I'm just happy that we can have one big place where we can all come together. I'm really excited that we're going to be doing more of our development out there. Because you can keep an eye on us and see exactly what we're doing and stay up to date with the changes we're making.

Steph: This is off script, completely. I'm going to put my code out there and you guys can all look at it. The reason why I don't usually contribute to open source is because it gets a little nasty out there, Matt. Are we mean and nasty to each other? Are people going to tell me "You're so stupid. How could you have been at Perforce for 11 years and not know this?" How embarrassing would that be. It's out in the public.

Matt: It's true. It's one of the things I don't worry about with the Perforce community. You see events like we throw here and they are always positive and they're always energetic. Even when people have criticism for each other, I think they're very respectful about it. I believe that that's going to come through in the workshop, too.

Steph: This is one more off question. I hate when I go off. If I put something in there and I'm really, really embarrassed about it, can I beg you to obliterate it?

Matt: Yes. You can. And if something were to get out there accidentally that is proprietary. "Oh crap, I didn't mean to submit that." Let us know. We can obliterate it. I'd actually like to surface that functionality for everyone to use for your own space. And so at last, hopefully. Sweet mystery of life, we've found you. We've made our monster. Or at least we know how. We've sadly run the clock out. We're down to 21 seconds. Steph and I will both be at the Meet the Speakers thing in just a little bit.

Steph: They can ask one question. Does anybody have a question? [inaudible audience remark 00:28:18]

Steph: Nobody has a question.

Matt: They're afraid you're going to make them clap again. That's the problem.

Steph: You have a question about the .NET and the 32 and 64 bit and don't leave me hanging here. I bought you extra time, Yuri.

Yuri: Right. You owe me a minute. We at DBmaestro we integrate with Perforce jobs and we use the P4 .NET API. What we've found a little difficult for us is that in the website there is P4.NET for 32 bit and P4.NET 64 bit. And we expected to find one P4.NET API which compiled for any CPU, any platform.

Matt: Totally. We can take that back to the .NET team and see why there's a split and if we can fix that. That's not a problem.

Yuri: Thanks.

Matt: Yes, no problem. Alright. Thank you very much everyone.

Steph: Now everybody clap again.

Gabe: Let's give them a round of applause, folks. Thank you Steph and Matt.