The Apache way: community over code

Roman Shaposhnik: I must say Baruch is an extremely tough act to follow. So I might not just even try, but I will try anyway. So my name is Roman, Roman Chaposnik. By day, I'm director of open source strategy at this company called Pivotal, which how many of you do know Pivotal? Awesome.

So I don't actually have to explain. But to me, it's one of the most quintessential sort of developer oriented companies. And if you want to talk to me about that, catch me after the presentation. But by night, I'm actually an Apache Software Foundation guy. These days, I'm also a Linux Foundation guy.

But Apache Software Foundation people are my peeps, you know, so that's what the talk is all about. Not to get all meta on you, but I'm actually a little bit of an outsider to this community. I don't have a title of developer advocate or community manager or even evangelist, as Birch was talking about evangelists. I'm a developer still, although I sort of crossed to the dark side. But I've been mostly hanging around big data communities, which means I have a lot of t shirts with yellow elephants on them.

And Apache is probably the place that you associate with Hadoop and big data and all of these projects. But what Apache is really known for is not producing these awesome projects, not even powering billion dollar industries, but actually producing awesome communities. So we, as a foundation, are actually in the business of producing awesome communities. And we expect the awesome communities to do whatever they do. And most of the time, whatever they do happens to be product or project or some code.

But it's totally fine for them to produce documentation or something else. Now the question that I typically ask anybody who is trying to approach Apache Software Foundation to actually start a project because it's easy. All you have to do is just talk to us. So I used to be a VP of the site of the foundation that is known as Intubator. That's how you start new projects in the foundation.

You just go approach the current VP and you pitch your project. And most of the time when the project gets pitched, it's not about how awesome the technology is, but how good a community does the project expect to have around this particular set of technology. And it's actually a very tough question to answer because what is the community? Amazingly enough, if you go to a dictionary definition, it doesn't actually do you any good. The best definition, amazingly enough, is to me at least found in the Urban Dictionary, which a lot of times is the best dictionary of them all.

But let's actually try to run the definition, which kind of talks about cohesiveness and common set of goals and sort of identifying with each other. Let's try to run the definition given this picture. So do you guys think, like, how many of you do think that this represents a community? Like all these poor souls on the galley, how many of you do think this is the community, right? Well, good.

No one. And the trouble is that actually tough for me to pinpoint why not. But I think I've come to a realization of the highest level order, which is, again, aside from the horrible slavery thing that is depicted here, the biggest problem with this picture is this guy with a whip. And it's not even that he is whipping the people. It's without this guy, all of these other people will disappear.

There is no community without that guy. No, seriously. That guy is what makes the community go. So let's actually try to de escalate. Let's try to get rid of the slavery and all the other horrible bits.

So is this a community? Well, maybe. How many of you do think this is a community? Okay. At least a few hands went up.

Okay. And this actually may be a community. I mean, problem, of course, is again, this is the same guy. Now that guy doesn't have a whip anymore. That guy basically just tells other people what to do.

And he typically is a coach. You know, he helps sort of these set of people achieve common set of goals. And he tries to get them all marching sync. But he's still there. And this is actually a better picture in a way because even if this guy disappears, there's still a chance that those people would kind of try to find a new coach because there's a common set of interests that binds them all.

I mean, want to win at a particular championship or whatever it is that they're training for. So this guy to me is actually a quintessential problem. And that's where me being an outsider to your community comes in. A lot of times when I talk to community evangelists, developer relationship people, community orchestrators, they feel that they are doing this. This is basically sort of their goal in life, to make sure that a community of people that is entrusted to them behaves in the most positive way and achieves the goals that it feels like they all share.

Now the trouble is that I've been in the open source long enough to sort of realize that most of the time this is what happens, right? So we all have good intentions, right? We all feel like we're the ones to lead, right? But it's tough. It really is kind of tough to do that.

So what I'm actually going to talk to you about is how do you get rid of this guy completely? Because it's not even a problem of good guy versus a bad guy, a guy with a whip, with no whip, like, you know, Linus Torvald versus, you know, Richard Stallman, bless his name. Bless his It's actually about this. So how do you put a set of principles in place so that without anybody you actually get an emergent behavior that is beneficial to your community? Because look, these guys, you know, this flock of birds, they have no conductor.

They have no guy with a whip. And yet they demonstrate a very cohesive behavior that is super beneficial to them as a species in a number of different ways. You know, the predator avoidance, feeding patterns, like you name it, flocking helps them. And as the ultimate sort of behavior in flocking, this is how I explain Apache Software Foundation to the people who are unfamiliar with it. This is what we do at ASF.

And when I give this picture, everybody goes like, but this guy at the very tip of the V formation, like isn't it the same guy as with the whip and the trainer and the conductor? But it's not. And here's the biggest difference. Because this is an emergent behavior. The guy at the tip formation is there because he is leading right at that moment.

And if the behavior patterns change, there will be yet another guy at the tip of the V formation who is most qualified at that very moment in time to be at the tip of the V formation. That is what differentiates algorithmic behavior from a very prescriptive behavior. This v formation will survive until there's at least three birds in the v formation. There's no voting going on. There is no process of elimination going on.

It just flies. And that to me, what is Apache Software Foundation. So what is it that binds us all at ASF? Before I actually go there, let me talk a little bit about the history of the foundation itself. So not a lot of people realize that Apache can claim its lineage to this little organization known as IETF, which is kind of like the folks who brought you the internet.

And IETF still exists. So it's not like IETF turned into Apache software foundation. IETF still exists. It does a lot of useful things. But I think the ethos that actually helped us get the internet and all these other amazing things, which was very sort of succinctly put together as rough consensus and running code.

Because remember what internet was all about. You basically had to produce an RFC, and you had to produce a working implementation. And ideally, you had to produce two independent working implementations for that RFC to be adopted. Right? That's how the internet developed into what we know it today.

So that rough consensus on running code was what the IETF was known about. And that was the same principle that a group of people back in 'ninety five, who were essentially eight members hacking on an abandoned HTTP server code base, kind of put in principle because that was the community they grew out of. There was that ethos of IETF still present. So Apache Group formed in 'ninety five, and the foundation itself got incorporated in 'ninety nine. So we are now a formal nonprofit foundation with roughly three fifty projects, translation communities, and about 16,000 commits per month going into the repositories that ASF manages.

And the rest of my presentation will be about the principles that ASF is built upon. And my point, sort of my call to action to you, is not that you should all just run and give all of your software to ASF, although of course you should consider it, but that you should think about whether those principles could actually apply in your communities as well. Because some of them are principles that sustained us even since before 'ninety five. And since before 'ninety five to today, we actually get from essentially one HTTPD to three fifty projects, I mean, maybe there's a few things that we're doing right. So let's talk about those things.

And that will be the introduction into the Apache Way. So Apache Way has this mantra, community over code. So again, we are putting community as the basic guiding principles and expect code to just be a byproduct of a good functioning community. So the first principle and again, Apache Ways is of like Xen. It's kind of like explaining Xen to people.

What is it? You actually have to practice it a little bit to get into the principle. But there's a few things that I think we can all agree on. So the first one, rough consensus on working code. As I said, it's what gave us the internet.

And it's still a very valid principle for developing not just code, but the working communities. Because you can replace that running code with functioning docs or some kind of an output that community is expected to produce. The point being that you don't have to spend hours and hours on end agreeing on something upfront, then sort of trying to implement it and figuring out that it doesn't work. In fact, I guess IETF and Apache Foundation stumbled upon this agile principle methodology way before it was even cool, right? So this constant feedback loop of actually trying stuff out is what makes it pretty special.

Small reversible steps. Now that applies to the code, but that also applies to your bylaws. So don't try to implement big changes. Don't try to suggest big changes. Chunk it up in smaller bits and pieces.

I still remember for the first time somebody from the, let's say, industry sent me a patch that my editor choked on. And I'm like, no, dude. Like, you actually have to split it into smaller reversible chunks. The following principles are actually interesting. That's how we sort of get the meritocracy that a lot of people talk about at Apache.

And it's sort of like in the following four principles. First of all, any constructive contribution earns you merit. It doesn't mean what is code. It doesn't mean like it has to be documentation. It can be a presentation.

It can be evangelism. It can be anything that is constructive and sort of gets community going further along. So make sure that you recognize that type of contributions and you associate merit building with those types of contributions. Second principle, which is actually super important for Apache, and I think it may be even important for what's known as single vendor open source projects, try to get as far away from corporate affiliation and seniority as possible. Try to completely sort of check out your corporate hat at the door when you're participating in the open source community.

And make it a principle. Basically say like we are providing a level playing field for all of these corporations to collaborate with each other. But by definition, that means that we do not recognize you as a senior developer or an architect from IBM or Microsoft or any other big corporation. We recognize you as a person, you know, we recognize you as Roman Shaposhnik, and your merit is your merit. That actually has a flip side that I will be talking about, that the merit now is associated with an individual rather than a company.

And the company is actually an allege into that. So all of those committer wars who controls the biggest amount of committers in the big data industry I have a few t shirts around that subject that's actually the function of that. They need those people on staff because all of a sudden it's not, oh, we need to wait for the great new thing from Apple. It's, oh, we need to wait for the great new thing from one of the committers on the project. Merit doesn't expire.

And that's actually a very controversial sort of point in Apache because in a lot of other communities if you disassociate yourself from the community, there are some kind of timers built in that would basically degrade your seniority and level of merit within that community. I think it's a fundamental principle that merit should not expire. And here's why. So the biggest reason for me is that it provides that cohesiveness. It provides that sense of belonging that these are your peeps, right?

And as a community member, it's super important for you to basically have people who might not have contributed a single line of code, but who are traveling on a flight. And somebody turns to them and like, who are you? Oh, am an Apache Software Foundation member. That sense is actually super important, much more important than real contribution to the code base. Finally, merit, like I said, doesn't buy you authority, but it gets you access.

And what's interesting is all of these principles I've actually been much maligned because of this book, The Rise of Meritocracy, that came at about 47. And I actually read it. And it's kind of like with all of the influential books. Not a lot of people actually read it, but they all have an opinion about it. My opinion about this is thus.

The only problem that I have with this book, it's sort of unbearable the fact that it's unbearably rooted in British culture. And not even culture, but British sense of empire. Other than that, it's actually a very useful book for all of you here because it talks to you about the people who are organized not by any other principle, but by the principles that I sort of outlined here. And what happens to the society when you allow those principles to rule? Finally, no vetoes.

Typically in Apache try to build consensus first. And that means no forced voting as well. So what I suggest and try to implement it in your communities as much as possible. Give consensus a chance, right? It's like a lot of times somebody, especially who feels that they're senior enough, just puts a proposal and says like, vote on it because I think it's good and we all need to vote on it.

No, even for a senior guy, you actually have to build a consensus first. And by the end of it, you might find that even voting is not useful anymore because everybody agrees anyway. You can just go execute and do whatever it is that you were trying to agree to. And speaking of which, progress is basically made by arguing with code, not pros. You don't write lengthy emails, although it's tempting.

You basically throw code over the wall and you attach explanation to that code and that's how you make progress. Like I said, building rough consensus is super important. But these three principles is, I think, overlooked in our reliance on over processing ourselves. So bylaws, I'm actually a big believer in not having the bylaws at the level of the local communities. The foundation itself in our case has a set of bylaws because we're a legal entity and we kind of have to.

But the local communities don't. And that again sort of helps you instill the sense of self management within the community, right? You know, self reliance. Because without the bylaws, everybody is now in business of essentially being a little bit of self policing. Because no longer you can say like, oh, it's not my job to basically tell that person not to be an asshole because we have a bylaws.

Somebody must be enforcing them. It's not my job anymore. No, it is your job. It is your job as a community member. Metrics, again, I understand why it's useful to you guys because, again, that's how you convince your superiors and your bosses that they have to keep investing in what you and to a certain extent all of us do here.

But again, for the community, metrics actually tend to be a little bit counterproductive most of the time because they essentially highlight the behavior that are useful for the outside entities and not necessarily for the community itself. And like I said, I mean constant voting or videos that I've seen in some of the open source communities just rub me the wrong way. And I think Apache is exactly right for trying to de escalate that. And rely on individuals. I mean individuals still, not groups of people, just individuals, are the ones, the engines of your community.

They are the ones who are making things happen. They are the ones who write code, write presentations, give presentations, you know, what have you. So a healthy community so I will not go through this slide. But for anybody who wants to sort of get a full definition of what a healthy community looks like, look through this set of principles. That is actually brought to you by Brian Benelhorf, who was one of the people behind the original five who incorporated the Apache Software Foundation.

And I think that is actually kind of like the most effective in your face set of basically laws of what constitute a healthy community. He actually has a great presentation on that. So look it up, Brian Vanderhoff at Con giving sort of the presentation on what makes a healthy community. When a healthy community is no longer a healthy community, that's where the corporate part of the ASF kicks in. We're still a corporation, which means we can act as one.

And as part of the corporate structure, we basically have the membership as sort of the shareholders of the company. But then the bureaucracy is very minimal. So we basically have a set of nine directors. And we have pretty much 10 officers who are basically overseeing the business of the foundation. And the foundation is literally in business of making sure that the communities thrive and trying to help when they don't.

But if it all fails, we're actually in the business of stopping the project and moving the project into the attic. So in that sense, we're super useful in giving you a sense of which communities are kind of making it and which communities are unfortunately no longer viable. But that having said, ASF is trying to provide very minimum governance at the level of the ASF board. So as was roughly sort of explained, ASF board is a sledgehammer, not a scalpel. It's not a fine tuned instrument to help you in all these sort of glorious ways.

And by ASF the doesn't pay for contribution. So don't approach ASF board of directors with donations and grants for specific developments. We will gladly take those, but we will not take them for the specific feature development. So that's basically what ASF is. And Baruch is telling me that I'm out of time.

So I would like to close this presentation with just a little bit of a reflection. And this is what I think is going on. So a lot of people are looking at it and saying like, well, there's this transfer of merit and people are now empowered and developers are now sort of these rock stars. But we've been through these ones before. This is exactly sort of how the Hollywood, the movie industry evolved.

Because if you think about it, it started as a set of big corporations essentially having this very sort of slavery like contracts with developers of the time, which were movie stars, right? And at some point it just became completely idiotic. So the first sign of the industry realizing how idiotic it is is basically Charlie Chaplin and a whole bunch of other actors actually starting up which look exactly like Apache Software Foundation, to basically trying to get empowered by themselves. And on the right, you see sort of the title page from the PC Magazine about the Apache Software Foundation formation. Aren't the two kind of the same?

It's almost like we're looking at the same kind of people. It's just in different time intervals, right? So I think we're kind of going through the same trajectory. And where we're getting right now is that with Apache Software Foundation principles on merit, we're essentially getting sort of these ideas of individuals driving the industry much more than studios. So in a way, studios are now paying for the benefit of being associated with the name of one of these guys.

Because these are the guys you go to see the movies of. Right? You know, you do not necessarily go to see the movie of, you know, Warner Brothers or, you know, Disney. These are the guys. So I think it's a very healthy development.

I think that's actually what's empowering us as developers. And like I said, I'm still a developer so, like, that's super useful to me. My only concern is I don't want us to turn into this. So this is the cautionary tale. And I guess I can put a lot of booze and Coke pictures from the movie industry here as well.

But to me, the Silicon Valley sort of HBO show is so funny because it is documentary. With this empowerment comes great responsibility. We really have to not be assholes. And now I'm actually talking about my peeps, you know. Like a lot of times we are, and I can only hope that the community spirit is what can save us from becoming the fodder for the HBO Silicon Valley show.

And with that, thank you so much.