Scaling developer evangelism

Rob Spectre: Well, thanks a lot for the introduction, y'all. So I just got into London on Sunday night, and apparently, I'm staying in the right neighborhood because there was a riot, which I was really excited about. Did you guys hear about this? There was a riot just down the street. Apparently, there was a class war uprising against a cafe that was selling breakfast cereal as its main service.

So there was, like, a lot of classmates. I was reading the paper and apparently the owner, after the riot occurred, looked blindly into the reporter and said, cheerio. Oh, man. That was a quality joke. Was a quality joke.

Folks, it's really really great to be here and thanks to Matthew and the whole gang for putting on this event. To my knowledge, this is the first time any collection of people have gotten together to talk in a focused way about the craft of developer evangelism. And I'm really, really stoked to see all you guys here. Off the jump, because this is such special occasion, let's make sure that this isn't presentation. Let's make this a conversation.

So we got a few a b bits to illustrate some of the stuff we're gonna be talking about today in terms of developer outreach. If you guys have questions or you have a comment, I would love for you guys to just share it now, and we'll we'll engage in the dialogue together as opposed to making this yet another talk that you have to sit through, which I know in this line of work can be somewhat tedious. Does that sound good to you guys? Yeah. Cool.

So I wanted to set the stage to this thing called evangelism, mostly because it being a fairly new practice. There's some question about what the business case is for a developer outreach program. And in my personal point of view, this is something that is quite critical and it has a lot to do with the unlimited reach that the developers in 2015 have. So to talk a little bit about macroecon that drives developer evangelism, it is indeed a developing world. And if you take a look at the number of software developers that are in The US, it is dwarfed by the number of software developers that are on the world.

There's about 18,200,000 on the entire planet. When you think about those 18,200,000 developers and who they touch through the software that they create, the force multiplier in terms of that headcount is absolutely crazy. So you compare the number of developers that are out there against the number of smartphone users. There are about 3,000,000,000 out there in the world today. You can see there's a huge disproportion between the number of people who are making software for these mini computers that are running around in our front pockets and the people who are using it.

It's worth even still if you consider the number of Internet users, the folks that are using the World Wide Web, which comes in around 4,000,000,000. And then if you're using a service like Twilio that does interact with the phone at a more fundamental level over the PSTN, like with telephone calls and with SMS, that number is even higher. Right? Look at that look at that proportion. That is absolutely crazy.

The the number of people who are building the stuff that all these other people build is disproportionate. Right? In a very clear way, welcome to the 1%. Right? Or more specifically, the point 008% that have twenty first century economy worker lives their lives.

Alright? Only 18,200,000 in the whole world and the growth is massively flat. Alright? 6% growth year over year in the world. That number is actually worse in the in the Western world, which is crazy when you consider how many jobs are getting created for each of these developers.

Right? In The US, the Bureau of Labor Statistics suggests that the number of developer jobs will increase between 2832% year over year for the next decade. Whereas the number of developers that that same economy produces is only gonna be growing by between 23%. Right? Same story internationally with the exception of three countries, China, India, and Russia, which are the only three countries that are showing double digit growth in terms of the number of developers year over year.

As a matter of fact, India will overtake all nations in the world by 2017, that is the same in two years in terms of total number of developers. Now the reason why this growth is so flat and everywhere everywhere that every education system that is not taking a brute force approach to teaching computer science has has a lot of has a lot of hurts. Right? Here at Middlesex University, there's the suggestion that as many as 60% of computer science fresh men that are taking their first course fail at the first test. Right?

And that is what is producing the dramatic language gap that every technical hire is trying to fill right now. How many of you folks are hiring for technical roles right now? How easy is that right now? There are a couple of people in the back that are looking even more white than when they walked in. It's very it's very, very tragic.

Right? And it's particularly tragic when you look at the skills that we actually need. Like like, the gap in terms of the number of developers for the stuff that people are likely to use right now is even more stark than the gap that exists for stuff that was in use twenty years ago. It's even worse when you start to take a look at the infrastructure that is responsible for delivering these services over the Internet. Right?

And this is in stark contra position from the growth of capital that is available to hire these developers in the first place. This is a report from Bain Capital that they refer to as a world awash with money. And this disproportion between the amount of money that's available to hire developers and the amount wealth that they produce is almost on the order of a decimal point. That's almost in the order of magnitude. Right?

The wealth in terms of financial capital capital is gonna grow at 10 times the rate of the gross domestic product internationally. And one of the quotes that was pulled out of the survey, I think, is particularly salient for anybody that is considering developer outreach as a career. Power will shift from people who own capital, from people who own wealth, to owners of good ideas. And in the twenty first century, a good idea isn't worth anything unless you have a developer to bring it to life. And so my suggestion is that the story point, that is the hour of developer time, will be the most valuable resource of the twenty first century.

It's going to eclipse the dollar, the pound, and the euro. That hour of developer time is going to be the most valuable resource that a twenty first century business has. And that future is pretty frightening. A lot not a lot of people know what to do with it. Right?

And if you think about the value of a developer's time in those stark terms, in those stark economic terms, as being the most valuable resource that a company has, the line of work that you and I are engaged in has become incredibly important, more important than ever before. And the way that we approach this job is going to look very different than anybody has ever tried to approach it before. It's gonna take a new class of hero in order to serve this constituency, which is going to be increasingly more important to the function of the twenty first century business. So my name is Ross Becker. I work for a company called Twilio.

How many of y'all heard of Twilio before? Awesome. You You guys are on the Christmas card list. So we're a cloud communications company based in San Francisco, and I live and work in the greatest city in the world, New York. And today, I'd like to share a little bit about how we approach developer outreach through the context of its critical business importance for the twenty first century business.

We're gonna be talking about a few things. First, we're gonna be talking about a bit about what the organization looks like at Twilio and some of the moderate amount of scale we've been able to achieve there. We're gonna talk about some of the important tools that brought us to that scale. We're gonna talk about some of the common problems that you're likely to encounter in your own developer outreach programs, and then we're gonna talk about future trends. And I swear to God, if we don't have a question or comment soon, I will tell another joke.

Okay. Scaling with Twilio. So the way that we view how many of you guys met developer evangelist from Twilio on the field? Well, the experience we hope you had if we're doing our best work is that you felt that they served as the red carpet to the Twilio community. This was your your welcome wagon into using Twilio for for the first time.

And their mission is to catalyze the network of software people that will change communications forever. Now, we choose those words very specifically. Right? Our mission as a company is to change communications forever. Our sense is if we give developers the capability of creating their own communications experience and the applications they're already building and the languages that they already know, some crazy radical stuff is gonna happen.

And that that crazy radical stuff is gonna be for the benefit of all of us who try to communicate with other people. Alright? And we feel the people that are gonna do that are software people. They are developers, people who are making software for a living. And if we get a large enough network of folks who have that that capability together, it will create a defensible network effect around the business.

And our sense is that we don't have to build this network. We don't have to create this network. It's not something that we're gonna affect it to be by virtue of, you know, giving a bunch of talks and giving a bunch of demos. We have to catalyze this capability inside every software developer right now. Our sense is that everybody, regardless of their skill level or regardless of the language that they use or how they look at their jobs or the application they're building, has the capability of changing communications forever right now.

And our job on the developer network team is to bring that potential out now. Does that make sense? It's cool. So far, we've achieved a fair amount of success for around three quarters of a million developers right now with the kind of hockey stick growth that always certainly always makes my CFO happy. A lot of what drove that is a lot of hard work by a lot of impossibly dedicated people, many of which are in the graph today.

A lot of these developers that you see on the graph were the direct work of folks like Phil Nash and Michael Hawker who was with us for a long time. And, obviously, that includes a lot of roadwork in producing a lot of stories with good that people can read. Alright. And when folks take a look at those inputs and those outputs and you're wearing your your marketing hat, which I'm not even sure what a marketing hat looks like. I guess maybe it's maybe it's a ball cap, like like, put to the back.

I guess that would be the only appropriate, like, marketing app that exists. They take a look at this developer outreach in terms of the funnel. Right? There's a number of people who are aware of what this thing called Twilio is. There are fewer still who understand what it can do.

A few that have given it a try. And then hopefully a small amount can become active customers that continue to use Twilio long term. And like and like many of you who are operating on a free game model, this is more or less what our our own funnel looks like. Right? But on the developer network crew, we obviously take a look at it a little differently.

And a lot of that is applying some of the network theory that many of you may have seen last night at Don's talk at the evangelist meetup. If you haven't had the opportunity to check out that talk yet, she's running around and has some neat tools for you to take a look at. But a lot of what she demonstrated resonated with us on the developer network crew because that's the kind of theory applies that we apply to our developer average every day. A lot of it has to do with this huge constellation of developers that we feel are out there. And there's this big idea called Twilio in the middle of it.

This company, this product, this community that we would like to introduce them to. And we divide responsibility for that introduction into three organizations that are in different levels of this network. And on the periphery are the folks that I hope you guys have had a great experience with out in the field, and those are the developer evangelism team. And their responsibility is to inspire and equip developers to change communications forever. So they go out there to developers where they live online and offline and make an introduction to themselves and an introduction to Twilio, hopefully creating a moment where they do something that they didn't know they could do.

Right? And they go to developers where they live. Right? And that includes the geography that immediately surrounds them and the language community that they ultimately serve. And hopefully, by the time they're done, we have a network of software people that can change communications forever.

Now the developer evangelism team obviously bears the responsibility of the surface area of this network graph that we're trying to build, but it also is reflective of the strengths and weaknesses of that particular graph. So if there's an area where we're particularly strong, like the East Coast Of The United States or the Python and JavaScript communities like we are right now, the network is going to be very strong. If there's an area of weakness like, you know, Eastern Europe where we don't have any developer evangelists now or a language community like Java, which we're not doing enough work in right now, the network is also going to be weak. Right? The network effect that you create is a direct result of the people that you have on your developer evangelism team.

Now we break that adoption down a little further in order to increase and accelerate the network effect that we're trying to build. And the first component of that we introduced middle of last year with developer education team. So if you think of developer evangelists, those psychic folks, started with Twilio, closer to the center of this network we're trying to build are a set of developer educators who are trying to challenge and empower developers to change communications at their business forever. So if developer evangelists are getting people started with Twilio, hopefully our developer educators are getting people finished with Twilio. They're creating these tight educational bonds between the individual developer and Twilio by helping them get something into production that can meaningfully have an effect on the way their users communicate today.

Does that make sense? And hopefully, if we do that job to well, the center of the network, which will then create a deeper and wider moat around the business that we're trying to build. Does that make sense? There's one missing component to all this, and that's the connections that these developers have with each other. And that's the responsibility of our community team, which we spun up at the beginning of the year.

And if the evangelism and education teams are focused on increasing the frequency and fidelity of connections that developers have with Twilio, Our community team is responsible for increasing the frequency and fidelity of connections that these developers have with each other. Their mission is to propel developers to change communications together. Right? Find folks who are using develop using Twilio out in the field and make sure that all these folks know each other and what they're working on and how they help. Alright?

And, hopefully, if all three of these functions are working in concert and everyone is producing their best work, alright, the network that our developers create will be far wider and far deeper than the one that we could ever create ourselves. Does that make sense? Kinda looks like a bunch of spaghetti, doesn't it? Yeah. I'm not super great at JavaScript, so that's that's definitely a reflective here.

But when I take a look at what the end product would ultimately be, yes, it looks a little bit chaotic, And, yes, it looks a little bit different than what you would expect if you have been a member of the marketing organization for most of your career. But to me, it looks very similar to something else that's very important to me. Right? It looks very similar to a cloud. It looks very similar to the distributed architecture that runs Twilio.

Right? And our hope is by decentralizing, right, the ownership of the Twilio brand into the community of the folks that we serve. They will feel the same way about their telephony API that they do about their iPhone. Right? They will have the same personal connection and investment to the people in their community for their APIs that they'll have with their native programming language.

Cool. Any questions on what our org looks like? Carla? I'm not Well, it certainly was around for a long time, but it wasn't focused in the same way that our education team was. And when you think about this stuff, it's kind of the same sort of specialization that you would get at any engineering organization as you reach scale.

Right? At first, everybody's a full stack developer. You're working on everything from the front end to the back end. But as organization matures, you'd want to have to get some level of specificity around the roles in order to ensure that they're a 100% effective. Right?

And there is no possible way, believe me, because we have tried it over and over again, for a single person to maintain helper libraries in six different languages, work 20 different cities, produce 10 blog posts a month, use 30 Stack Overflow questions, and be responsive on Twitter and Facebook at the same time. Right? Like like, it's just impossible. How many folks are are are just starting their developer outreach program right now? There's a lot of folks who who are new to the group.

So to share a little bit about our journey and how we came to this network model, we first started with developer evangelists who are focused on a particular language and focused on a particular geography. Alright? So the way to think about approaching this is the same way that you would approach producing a large software project. Right? Your objective is to get to an MVP or a vertical slice.

Right? Right? One constrained set of functionality that evidences that the product is a good idea. Alright? The same approach can be applied to your developer outreach.

Alright? Pick three cities, pick one language community, and find someone who's enthusiastic about serving both at a level of excellence. Alright? And start there and compare the results in that community and in that geography to the other geographies and demographies that you intend to serve and see if you can produce a notable difference. Right?

There's gonna be a lot iteration occurring. You're gonna try a lot of different things. You're gonna try going to a lot of different events and try a lot of different online executions. Alright? And by starting with that one evangelism core that's serving that one community well for for Twilio, it was PHP, San Francisco, Seattle, and Los Angeles when it when it first started.

You'll find out whether or not you can own the space or whether or not the product is compelling. And then from there, you can make some determinations on how you like to grow your team. Does that sound that answer the question? What's that? Maybe there's some insight in the number of people who are the missing team.

We have six we have 16 folks on the crew right now. We have 10 I feel like I probably owe you a pint at the end of this because you set up the next slides perfectly. We'd love to talk more about how measurement is a very important piece. And in fact, if any of you are interested in keeping your team from being co opted by other elements inside the organization that perhaps don't have the developers best interests at heart. I feel measurement is is the primary way that you do that.

But were there any other questions around measurement related before? I think it's a little more aggressive than that. Right? My sense, if you take a look at the history of developer tools that have been very, very popular and have dominated that particular market, it has always been by virtue of a defensible network effect. So when you think about the world of network engineering in the old adage that no one ever got fired for buying Cisco, that is an example of a network effect because there is an activated set of network engineers, right, through their huge certification program, the tons of events that they put on every year, the amount of education they do around their product.

Right? And the reason why no one gets fired for for using Cisco is because everyone knows how to use Cisco. Right? And once books are being written about it, once degrees can be attained in using it, you have the understanding that you have become a de facto industry standard for the product category that you're operating in. Alright?

In my sense, call me crazy, is at the end of this whole desktop and the whole communication to API space, there's probably gonna be one winner. Right? And if we do our jobs well, I suspect that's likely to be us. Right? So when I talk about the dependable network effect, it's more than just taking a burden off the support.

It's it's about winning in a capital w way. Make sense? How much time we got left now? Twelve minutes. Twelve minutes.

We're gonna we're gonna cruise. So when talking about measurement, usually, the first time, or at least in my experience with a a lot of folks who have programs out there in the field, the first time they come up to me and talk about measuring their program is when they're getting this space from their boss. That is to say there's suddenly a need to justify those massive expense accounts and the bar tests that they're ranking up on the field. Right? Which is I hope that's not the only way that you're engaging developers, when you're when you're starting to get this space, it's probably too late to have a meaningful effect on where your program is going to go in terms of service of the business.

Alright? Measurement is what gives your program wings. Right? If you have the capability of measuring what you are doing, you have the capability of adjusting your execution for your product and for the developers that you were going to serve in a meaningful way. And, obviously, there are a lot of challenges to that.

Like, I don't think there's anybody that's running a developer outreach program that's like, no. I don't want any metrics. And I don't know why I just use that voice. It's usually because there's some challenges to getting out that data. First, getting to the data is really hard.

Right? It's like cracking a safe in a big way. Especially if you have an API platform that has been fortunate enough to get a lot of growth really really fast. Most of your underlying back end infrastructure is gonna be put together with duct tape and baling wire, which is very difficult for you to get that in a meaningful way with a single tool. Right?

Another problem is the propensity for folks in our line of work to focus on vanity metrics. I tragically showed you some vanity metrics just a little while ago when I was talking about the number of events and the number of cities and the number of tutorials that we were doing. Right? A lot of us do rely on our inputs rather than our outputs naturally. And this seems to be a problem that afflicts the entire field of marketing discipline.

It's not something that's necessarily constrained to just developer evangelism. And then finally, getting a grip on these two factors can drive someone so insane that they choose to focus on other things instead. So there are a few solutions that I can suggest that will help you measure the effectiveness of your program. First, you gotta be a pirate when it comes to the data. Right?

You gotta beg, borrow, and steal whatever directional information you can get on the performance of your program. The distance that you can go with Google Analytics alone is astonishing. Right? If you know the right way to measure it. And a lot of that has to do with picking the right lens with which to view your work on the field.

So I'll give an example that borrows from the world of advertising. When we first started being serious about developer evangelism at Twilio, this is the story that we would always see when we would take a look at Google Analytics. How many folks use Google Analytics to take a look at your thing? And you guys know what I'm talking about. Right?

You get these, like, flat little plateaus that correspond with the number one, and you're like, oh my god. That that was a total crap, dude. It was totally bad. Alright? When you're taking a look at the effectiveness of your program through the lens of a ZIP code, this is what it's always gonna look like.

Mostly because the volume that's coming to your site is gonna be so small because the number of developers is so small. Right? That the folks that you activate at the end of moment at the particular event is not gonna be very strong. Right? However, if you take a step back and instead look at the entire metro area, like if we look at Los Angeles instead of just looking at Santa Monica, the story becomes a lot better.

Right? You get a peak when the event actually occurs, and a few days later, you usually get one more. Right? When people get back to their on the next business day, when they get back to their desks and they remember the presentation and the demonstration that they got before. Alright?

So when I talk about peg stealing and borrowing, right, I'm talking about scrapping together the right lines to give you some directional information on what's going on. A lot of that has to do with looking at the metro area that you're trying to serve as opposed to the ZIP code you're trying to serve. Now in The United States, this is very easy because we have from the world of advertising a unit of measurement called the designated media area, and it's used mostly for television advertising. So we can use that to cobble together what we feel is an accurate view of a particular metropolitan area. You know, you know the cities that are in the New York Metropolitan Area as opposed to those that are in the New Haven or in the Philadelphia Metro Area.

Right? However, this is a little bit more difficult in in Europe, and we use that we just do it by hand. We pick the series of cities that we feel represent London as a whole, and then we use that to gain some more insight onto what's going on with this demographic that happens to skew highwaysuburban. Does that make sense? Is that useful?

Pushing layers up to the edge. Obviously, this data is not useful unless it's in the hands of every developer evangelist that's working there. Right? One of the mistakes I I made and I suspect a lot of you have made in serving your programs in leadership capacity is for the first little bit, we would get together every month and then we would go over the numbers. I would feed the numbers to the team and the team would basically feel like they were getting a report card every month on on how they did as opposed to getting a regular update on the impact that they're having on the field.

How effective do you think that approach was? Not very effective at all. It was not very good at all. Unlocking this data from the safe and making it available to every person on your team is a critical part of reaching scale, right, and reaching a shared vision over the impact that your program is having. We have couple different tools that we use in order to do that.

First is a trip report. Tonight, after we're done celebrating the awesome day of conversation that we've had around developer evangelism here at Deborah O'Conn, we're gonna head back to my hotel and share some of the data that we were able to collect around this event and how effective it was and what worked and didn't work in a qualitative sense around this particular event. So this gives the folks that are in the field a micro view on each execution that they do. Right? Then on the macro view, every quarter, each developer evangelist on the crew gives a presentation on the metro areas that they serve.

Right? The primary metro area and their secondary metro area. And this gives them the opportunity to see the forest from the trees. Right? Step back and look at what the overall growth for the two border metro area is.

Here's an example from Berlin, and we publish these to our internal Wiki every every quarter. It includes statistics around the traffic that's getting pushed, the sign ups, the upgrades, how much usage is going on in the area, and critically for any salesperson that is visiting the area, the beer index, which is the price in every local currency of how much a beer will cost in that particular metro area. You can thank Michael Barbara for that innovation by the way. Cool. So that's a little bit about ownership.

Last bit I'll say, Denver Mortgage Store is is great to open. You should definitely use it in as many places as possible. This is a really, really good question and easy to administer to any audience. As a matter of fact, I'm gonna administer it to all of you here in about three minutes when we wrap up with the last questions. Alright?

Nepromorph score works very simply. You ask the audience one question, one question only. Would you recommend this execution to a friend or colleague? And you ask them to rate it on a scale of zero to 10. You take the percentage of people that said nine to 10, you subtract the number of people that said anything less than six, and you get a value that usually is a quantitative measurement on how successful the event was on on getting folks excited about your product.

This tool is amazingly useful for wrapping your hands around in a quantitative sense how strong the quality of the executions you have are in the particular field. Little pro tip for all you folks that are working in Europe. You need to take the six and move it down to a five in order to be statistically accurate accurate for your particular demographic. Folks, particularly in Western Europe, tend to be a little more pessimistic about the scale than in The US. So definitely use that to get better sense of what's going on.

Cool. I feel like probably there are a lot of questions about measurement. Is that fair to say? Maybe not. Maybe not.

Let's finish up talking about content, and then we'll open up for questions. So I'd like to share a story. GIF. This is GIF. It's a pomeranian that belongs to one of our developer evangelists that's working out in New York, Ricky Robinette.

GIF is so amazing and so excited like every Pomeranian is that, you know, she's got her own parody Twitter account as administered by the team indicating how stoked this particular dog is. The dog is so stoked that Ricky was concerned that it was having trouble sleeping. That is to say the dog was so excited, it was forgetting to fall asleep. Naturally, being an engineer, he wanted to quantify the degree to which his stokedness was interfering with his sleep. So using a Tesla, created a device that he put into a little doggy sweatshirt that would track when GIF was sleeping and when GIF was not sleeping and send him a SMS update every day with the amount of time that the dog actually slept.

Pretty amazing hack. Right? It gets folks excited about using Twilio with an Internet of Things device. He presented this at Brooklyn JS, one of the best meetups that's in New York, and ended up getting a fair amount of people to show up in an execution that produced a fairly strong result. Now a lot of folks in a lot of different programs would have taken this result, called it a day, and and went ahead and went home patting themselves on the back.

But one of the little tidbits tidbits of value that I would share that has really worked for our crew in the past is taking these talks and turning them into online content, stories with code that people can follow step by step in order to recreate the same experience with their own super stoked dogs with their own hardware. Right? And the difference was super stark and not necessarily what you would expect. Right? So you hop back.

We had about 70 people come to the event, and a ton of folks were really stoked. Score an NPS higher than 50 is upper deck home run or whatever success means in cricket. Like, it's it's a real real real good result. Right? Getting an idea for some of the attendees, absolutely moron.

Right? However, when Ricky published this blog post, which took, you know, about eight hours total to make, The result was stark. On the peak day when we released, it got close to 1,600 folks who viewed it and the average time on page was greater than six minutes, which is almost unheard of on the Internet. Well, what was amazing about this execution is not what happened the day the content was released. It's what happened in the months after that content was released.

For the next three months for the next three months, another 100 visitors came to see that tutorial at a rate that was greater than six minutes for their average time on page. So for reference, just by taking the eight hours to take that one talk and turn it into online technical content, he effectively went to that same meetup for the three months afterwards afterwards without actually having to show up and bring t shirts. That's the real key to scale with your developer evangelism work. Take the content that you're presenting in front of people and bring it to the Internet. So we'll finish up with two minutes of questions, it sounds like.

This is the second time I've ever given this talk, so if you guys wouldn't mind whipping out your phones and sharing some feedback on this particular event. The question to you is how likely are you to recommend this talk to a friend or colleague. If you can go ahead and text a 1 to 10 to the number here on the screen, I'd be deeply obliged. And we'll go ahead and take take maybe maybe three questions and call it back.

Speaker 2: How would you have turned that into a corporate strategy rather than an extended global strategy?

Rob Spectre: You know, that is a really great question. Right? And it kinda highlights one of the slides I was gonna cover. I don't know why everyone thinks corporate developers are aliens. I don't understand it.

I really don't understand it at all. Like, everyone thinks that these folks that work at large corporations are somehow like mutant creatures from outer space that are different. If there is one thing that I have learned in the four years that I've had the privilege of serving at Twilio, it's developers regardless of their skill level, regardless of where they work in the entire world, regardless and of the company that they work at, they're all turned on about the same stuff. Alright? They're all modified technologists that are genuinely enthusiastic about learning new things.

Alright? And whether they work at IBM, right, or they work at Indiegogo, the result is gonna be the same if you bring something to them as a technical peer with a genuine feeling of enthusiasm. Honestly, one of the best trade shows ever had, okay, was one where we brought a device that Ray Boggess, our developer of Angels in Chicago, that allowed his dog to take selfies and text them to him via MMS. Alright? We bring that to a hyper corporate trade show in the middle of Chicago, which is I think you would agree is a hyper corporate city for developers and people want to dance.

The same stuff works. It's just different audiences. We got time for one more. Gotcha. Gotcha.

And and usually, this is within the context of sales engineering. Right? Like, a lot of folks have talked about where developer evangelist job begins and ends. It has to do with, like, well, know, when are they a sales engineer when they're evangelist? Is that fair to say?

Fair enough. A lot a lot of good questions. Love to talk to you more. Broadly, I would say the way that we define developer evangelism organizationally in terms of boundaries is by funnel and by engagement. Right?

So in terms of funnel, our objective in the developer outreach group as being part of the marketing organization is trying to get the developer as close as we can to get to their putting in their credit card, right, which is the the upgrade portion, which hopefully is an indication that they're about ready to bring their application into production. We do have a dramatic effect on revenue overall, and a substantial chunk of the revenue that we do receive comes from developer average. But in terms of, like, area of responsibility, that's the demarcation point between marketing and sales for our organization. Right? The second bit is engagement.

Developer outreach is a one to many activity. What we focus on with our folks out in the field as distinct from sales engineering is making sure that all their engagements are one to many, meaning that they are in front of a large number of developers. Right? As opposed to working with the developer one on one in kind of a presale support capacity. Does that make sense?

Alright. Thanks. Cool. Last one. So that's Yeah.

It's a very important component and something that we we definitely involved a lot over time. I'll say right now, we do a few different things to provide that feedback to the community. So we do summits every quarter with each member of the developer network team where we get together for two days to talk about our craft and plan our strategy for the next quarter. We alternate between New York and San Francisco for these events mostly to help folks from Europe not have to endure that fourteen hour flight to San Francisco. And everyone that we have in San Francisco are usually happy with meetings of the product owners of particular areas and that is what gives us the opportunity to provide synchronous feedback.

In addition to that, we do the same thing everyone else does in terms of managing Jira backlog and performing escalations on a biweekly basis to the product management teams. An important step, I think, that distinguishes the Twilio product from a lot of other ones that are out there is performing an internal hack day for which the entire developer network team participates for every new product product that goes out the that has a certain level of consequence. Right? And it's through these internal hack days that we can help discovery of a lot of bugs and a lot of corners and edges that exist in many different languages and many different frameworks. Guys, thank you so much for your time.

Really, really appreciate your time.