49. Building Effective Distributed Teams

Juan Pablo: Yes, so I've been chief technical husband for about, I think it's seven years now. Melanie started a fashion startup called Elizabeth & Clarke. It's a subscription service workwear for women she invented and unsustainable shirt and I've had the fortune of being able to support a lot of the engineering development in the marketplace, the subscription service, the inventory, a lot of that stuff.

Juan Pablo: In the early days and more recently we have a tiny engineering team that supports it now.

Anthony: Very cool. Two very different animals, Splice and Elizabeth & Clarke, which is definitely a really great background context for the idea of effective distributed teams just by virtue of the fact that those two businesses, both their industry and their scale are substantially different.

Anthony: Before we dive into some of those maybe media specifics, why don't we talk generally about your take on the role of engineering within a business that isn't purely a tech company?

Juan Pablo: Yeah. I think my understanding of the role of engineers has evolved a ton as I've been building teams and I've been getting deeper into tech and tech-enabled or supported companies.

Juan Pablo: More and more today I understand the role of engineers to be levers, business levers, and our job is to use our skillset to build technological solutions, whether it's through spreadsheets and sometimes by writing code to enable the business to accomplish its goals.

Juan Pablo: Especially when you're a startup, it's helping the company learn faster than the market before you run out of money.

Anthony: One of the things in a leadership position that I think is a key responsibility is helping your engineering teams find that connectedness back to the business, both in terms of having a voice into it, but also understanding their role and supporting that higher level initiative.

Anthony: I'm curious if you've had any techniques or tools or processes for just helping your engineers stay more connected with that part of the engineering responsibility?

Juan Pablo: Yeah. I think the biggest or the most important thing that I've been able to do is be extremely transparent around the state of the business and helping the organization understand what levers we are able to move and how they work.

Juan Pablo: I think there is a big portion of our industry that idealizes the role of just building software. Our job is to build software no matter what and we... Look it's a really great thing that we're fortunate in many cases of getting paid to do things we enjoy very much, which is writing code or building things, but sometimes we over-index in the building things or in the learning, "Oh look, I really want to join this company because they use X technology-

Anthony: Yeah.

Juan Pablo: ... Not because they're building X or they're driving... I just like working with X and it doesn't matter, I'll do whatever. I just want to do this."

Juan Pablo: It makes it a little challenging when you're trying to drive an organization, if you're not connecting the folks who are responsible for building the business to the actual business. Let me take a little bit of a step back. What I do really try to do is help the organization understand the entire funnel, get them in touch with our customer, Splice artists.

Juan Pablo: We support both artists in our music marketplace on both ends. People who use the samples that are in our collection and also artists, sound designers, producers who make the samples that we sell.

Juan Pablo: We have a plugin business line which is a little bit more of a rent to own model. We enable artists who don't always have a ton of money to acquire very expensive piece of software in a rental model and creating that empathy for like, "Look, this is why it's important that we make software accessible and we don't make it more expensive and we actually deliver the service."

Juan Pablo: Giving that to the engineers who are responsible for building that is critical for me as an engineering leader. Once we are able to present the objectives or the goals of the business, with their skillset, they can determine what the right solutions are for some of the problems.

Juan Pablo: Of course in conjunction with design and product management, but what the solutions are going to be and how to start experimenting and iterating towards whether it's increasing your revenue, acquire more customers, making a specific piece of our product more accessible or more functional.

Anthony: Yeah. I also think that something that you touched on there that I think is really ironically empowering to engineers is once they understand the business context, the benefit of constraints of knowing where revenue is or what the health of the company is or what are our business objectives are, helps them arrive at an effective solution ironically a bit faster by virtue of knowing whether it's the urgency or the initiative the end-user experience versus to your original point, the greenfield just build for building sake.

Anthony: The crippling openness of that can be a bit interesting. It's cool for sure. One of the things that I think is particularly interesting about your situation is having the experience of getting to work with Splice and see and help along Elizabeth & Clarke, 40-ish engineers at Splice, a couple at Elizabeth & Clarke, it's a very different ballgame. Can you talk to me a little bit about the share overlap of those types of engineering teams at that scale and then also the most stark differences in you opinion.

Juan Pablo: I think I've been very lucky to have been a part of both at the same time because I can contrast what may work in one, what may work in another, and also I have the context of tiny team in a lot of software that I wrote versus a pretty decently sized team, an autonomous software I didn't write.

Juan Pablo: Both our businesses and both to some extent are subscription based-businesses. We do... We are trying to make money and not just grow infinitely. The constraints on Elizabeth & Clarke has always been, it's a bootstrap business. I at some point I've also been VC.

Juan Pablo: Whereas Splice has raised significant amount of money for the purpose of growth and for the purpose of grabbing a market much faster. The key learning has been how the delivery pipeline has such a big impact in making the team effective.

Juan Pablo: When you are a one-person team, and when I was writing the first version of Elizabeth & Clarke, we had to ship the first version of Elizabeth & Clarke, and I think it was in three week we had a lunch.

Juan Pablo: Melanie had had some challenges with her original developing outsourcing team and so I doubled down on automation for our entire platform and I doubled down on domain-driven design for designing our software or our business on top of the software and then optimizing the delivery pipeline.

Juan Pablo: I really had to make sure that we were continuously delivering, that automation was running and that CSED had to work really well. Splice had a different scale and I got a chance to take something that worked for five people, but then when we grew it broke down.

Juan Pablo: We had a staging environment that was a single staging environment that a lot of the things that we had to test on were there and when multiple people start sharing a staging environment, which is like an extension of their development environment, people start stepping on each other toes and like, "Look, staging is blocked. You can't really ship anything. We're testing this."

Juan Pablo: It starts at scale. These little paper cuts are a lot more noticeable. On one end, to ensure that the tool engineers who right now work at Elizabeth & Clarke are extremely effective, we continue to invest on those tools in different ways.

Juan Pablo: In Splice, we do the same thing. We have a production engineering team that continuously invests and their primary goal is to drive cycle time down. Whereas in Elizabeth & Clarke, we have Heroku, and it's not... this is not a... it's possible but we have built the least amount of software ourselves and effectively outsourced many of these things to other companies and would rather pay them that use our own money or at least our own time to build things that are already built.

Juan Pablo: We use CircleCI, we use Heroku, we use a database of primary stories on Compass, there's a lot of software as a service that we use, because we cannot afford engineers to do that.

Juan Pablo: At Splice we can, and it is a lot more effective and it is a lot more... the return on investment is also much better. I've been able to contrast these two patterns and recognize that neither of them are better, they're different and in some cases work depending on the context.

Anthony: One of the things that we touched on earlier, obviously is the notion of a distributed teams, so you've got 40 or so at Splice, you have a handful of Elizabeth & Clarke and that's another what's say around the effectiveness of a team of factor for sure, both in terms of your ability to effectively hire, build and scale an engineering team, but then also in enabling different modes of work for them.

Anthony: I'm curious for both sides, both Spice and Elizabeth & Clarke, how was your experience been with the role of locality and remoteness and how that's played out for both of those organizations?

Juan Pablo: I've been fortunate enough to have built or run teams with a distributed component for the majority of my management career for about nine years, eight years.

Anthony: Awesome.

Juan Pablo: I've been working with distributed teams. It got to a point where I was enlightened by the fact, okay look, I have one person who is not in the office, we are a distributed team. That's it.

Anthony: Mm-hmm (affirmative).

Juan Pablo: We have to act like it. In the past, two teams I've had the chance to build the past three teams, have been delivered about it. Before Splice, I was at a company called Ride.com, which was ridesharing or carpooling.

Juan Pablo: That's the first distributed team I built from scratch and I learned a ton. That team was primarily in Latin America. Then the second major region was North America and we had a few people are across all their time zones in Europe and in other places.

Juan Pablo: The first learning there was, for me, constraints and time zone as you are scaling is a big challenge in some cases, so I started constraining at least an overlap. Why? Because I don't like getting into people's lives.

Juan Pablo: No matter what, if you have a really broad time zone overlap, some business is going to come in the way and someone's dinner is going to be affected or someone's taking their child to school and they're going to get a call and I don't like that.

Juan Pablo: At Spice, we have constrained the first version or the first phase of our team to a time zone. People who live within three hours of New York. The majority of the Splice organization is in North America, the United States, there's a few folks in Columbia.

Juan Pablo: I am biased. I am Columbian and I had a group of folks who had worked with me at Ride who came along to Splice. We have an engineer in Uruguay as well. But yeah, the America's right now is our focus.

Juan Pablo: What this has allowed me to do is to be extremely deliberate around communication. What I've found, at least in my experience, is that when you have a distributed team in here, I'll make a side note, I believe all teams eventually become distributed because you can't fit all in a room forever.

Juan Pablo: At some point you're going to have a different floor or you have a different office or different anything and you're distributed. If you get distributed processes to work sooner, you can scale or you can be ready for these things faster.

Juan Pablo: Very much like when we were talking about the staging environment, if you plan a little bit ahead, then the pain will be much less. What we've done at Splice and to a minor degree at Elizabeth & Clarke is, there's very specific channels or channels, and I don't mean them by attaching but mediums or media by which we communicate.

Juan Pablo: We communicate about code being our source control mechanism. We use GitHub. That's anything related to our software goes there. Anything around our iteration planning goes through Clubhouse. Anything that is related about coordinating, yeah, we use the chat, Slack back and forth, but at least for engineering chat can be considered only for gifs.

Juan Pablo: It's like for jokes and for things, it's not nothing mission-critical should only be relayed in chat. We distribute important information in email. Then as a leader, I assume that everyone will be eventually consistent over a 24-hour period.

Juan Pablo: Some people will open their email in the morning, some people will end their day with email and everyone will at least have seen what we've communicated, what is important broad communication.

Juan Pablo: That also gives me the ability to plan and have a cadence. If there's emergencies, we have protocols. There's an on-call rotation that uses PagerDuty for that and if there's anything critical, people know I will text them.

Juan Pablo: Everyone has their phone number on their Slack profile. I think I've texted people three times or four times in the last three years. But that way people can organize their time. We don't expect to constantly be available and it makes it a lot more scalable and a lot more reliable.

Anthony: You mentioned the effectiveness of bringing those things to your engineering teams. Obviously communication's a huge part of helping engineering teams be effective. I could make the argument that's probably the most important one.

Anthony: There's a whole bunch of other stuff that goes into optimizing an engineering team and helping them to produce strong and valuable products. I'm curious, I know there's the notion of time, there's the notion of their tool change and developer experience, just internal processes, I'm curious what are some of your thoughts on how to help an engineering team be its most effective?

Juan Pablo: Software, at least from my perspective, and I'm stealing this from... this line I'm stealing it from Kellan, who is the CTO at Etsy, is a process and not an end-product.

Juan Pablo: You codify, whatever you know about the business, into software, knowing that tomorrow, whatever you know is going to be outdated. It is an iterative process of you learning about the business, learning about the problem that you're trying to solve, codifying it, releasing it to actual users, learning about all the ways that you were wrong in the assumptions of why you thought this was going to be the right solution and then doing that over and over and over again.

Juan Pablo: What you strive to become as an engineering organization, is a learning organization. An organization that with every line of code or that was deployed or to lead it, you're aiming to help the company learn something about its market. Faster, I don't know if we've said this in startups, faster than you run out of money.

Juan Pablo: At Splice, we have a mission. The engineering mission is to enable our company to learn faster than the market by deploying or delivering high-quality production software at a fast tempo.

Juan Pablo: The tempo or the rhythm at which we deliver software is critical. The faster we can do it, the better that you're going to... or at least a faster the organization can learn and the better we can stay ahead of the market or at least knowing how to move forward.

Juan Pablo: I had two huge problems at Spice at least since since I've been here. The first one was building the team. I had to solve that problem, so going from five to I think we went 55 engineers in the first 18 months, that was a huge huge thing to figure out and build a recruiting pipeline and attract people and build a culture and build... all these things were really, really challenging to do.

Juan Pablo: We got there, in the process of doing that, I also broke down the delivery of software. It stopped working. The processes stagnated, the staging thing was broken, our quality process was broken, how we deployed, and there's still many broken things, but when I started looking into why everything was so slow and why we couldn't deliver value in time or at all, in some cases, I was able to start to look for how our industry was looking at this problem.

Juan Pablo: It took me a while to find the right at least mindset or the right research, but I stumbled into at least a few important resources. The first one was the state of DevOps report, which led me to to a book called Accelerate.

Juan Pablo: Accelerate summarizes a lot of the learnings of the state of DevOps report on what makes engineering organizations high-performing and how high-performing engineering organizations are really great indicators of high-performing businesses in no matter what industry and then they break down 20 capabilities that you should strive to get in your organization in order to become a high-performing organization.

Juan Pablo: They also outlined metrics that they found were really good indicators of whether an organization was high-performing or not, which is cycle time, mean time to restore... Cycle time is the time it takes you to get software out of customers.

Juan Pablo: You can measure it from... At least at Slice, we measured from the first commit of an engineer to when that pull request was merged because then we assume that pull request is automatically deployed behind a feature flag or something, so it's already out to some customers.

Juan Pablo: Mean time to restore or how long it takes you to recover from an outage in production, change fail rate or how often often you're introducing errors or defects into your software and deploy frequency, how often you're deploying software into production.

Juan Pablo: Those four metrics, at least the top three, so cycle time, mean time to restore and deploy frequency, helped us at Splice start thinking in a very scientific manner about our own delivery practice.

Juan Pablo: We were able to understand look, engineering delivery is our own responsibility, how fast we can get software out of customers is something that we have a lot of... well a hundred percent ownership on, but also the ability to have impact through whether we speed up code reviews, we deploy in smaller batches, we break down features smaller, we reduce a lot of the perhaps bureaucracy and how do you get some stuff out, you decentralize staging.

Juan Pablo: One of the coolest things we did was we created feature branches. It's every PR gets its own... gets deployed at least into S3. We use a single page app so it can be deployed into S3.

Juan Pablo: You have infinite staging environments. That unlocked a huge part. I think we've reduced... When we started doing this work, which was a year and a half ago almost, our cycle time was an average 200 hours. It took us 200 hours to get code out to customers.

Juan Pablo: Right now I believe we're on an average of 40 hours, so this was done by deliberately setting metrics, setting a baseline, measuring and understanding, "Okay, look what can we move?"

Juan Pablo: Those solutions came from engineers.

Anthony: Right.

Juan Pablo: Like, "Look, here's the direction. How can we drive this down? What ideas do you have?" Everyone started little by little contributing to, "Oh, we should not do so many reviews per person. Okay, let's watch what impact that has on the quality of our code."

Juan Pablo: But there's all these ideas that started coming together. We built a production engineering team dedicated to watching and driving cycle time down, and it had a huge impact on how we can deliver business value faster.

Anthony: Yeah. 240 hours is a tremendous reduction in the amount of time it takes you to get from first line of code out to customers. I think one of the things I'd be curious to hear a little bit more about is how an engineering organization like Splice, where you've got 40 plus engineers versus Elizabeth & Clarke where you've got just a handful. Is there any obvious difference in scope or ability to tackle that sort of stuff? Or is there a different way that maybe you seek out some of these optimizations or approach helping your engineers find which optimizations make sense?

Juan Pablo: Yeah. The resources available in the two different companies are different. By resources I do not mean people, resources I mean money and time. We have a significantly different... an order or two orders of magnitude difference in the two companies.

Juan Pablo: The way to approach the problem has always been different. Even though I wasn't so conscious about the importance of the delivery pipeline in processes, when I started to set up Elizabeth & Clarke, I was a little bit more experienced than say when I joined Splice, but Splice had already three years-

Anthony: Sure.

Juan Pablo: ... of many, many, many experiments built up. As a startup, Splice was for many years, for three years, I think it was 14, 15 employees tops that there was a cap on people that we were willing to be. Because we were in survival mode, we need to find a product that sticks.

Juan Pablo: Elizabeth & Clarke was in a similar position but we had not raised any money. The agreement that I did have with Melanie was we were going to... We went out to market, we are going to keep the site surviving but we are going to spend some time and invest in our infrastructure and in our architecture, so that then we can scale much better.

Juan Pablo: The first year I built a site and I built a bunch of stuff and then with Simon who I hired in Columbia, he came in and he helped me rewrite the site and take over some of the stuff that I had already put in place of our delivery pipeline and make it better.

Juan Pablo: Here it just took a while rewriting our site, took almost a year and while Melanie could focus on improving her product and improving many things of like the shirts and launching a couple of kick-starters and stuff, we couldn't really iterate on the website for a while.

Juan Pablo: This was just technical debt repayment. We couldn't do parallel trucks for a while. At Splice, it's different. When I joined, yes we had a ton of technical debt but that was not a problem because we had found product market fit to some extent. We had found a product or sample library that people were subscribing to or giving us their money and that gave us the opportunity.

Juan Pablo: We had also found the plugin business as well. But it gave us the opportunity to start paralyzing and yes, we went to-

Anthony: Sure.

Juan Pablo: ... investors like, "Look, we want to do many, many more things at the same time for this we need capital. Can we take some?" What I get is a lot more engineering time to invest in paying down many, many of the decisions that we had to make in the past because we had less information.

Juan Pablo: But those were the right decisions at the time, when you were five engineers and trying to build all this stuff in a couple of weeks and just ship it out and see what happens. You will make the decisions you'll need to make with whatever tools you have hoping that when you have money and when you've survived and someone else is going to come in and clean it up.

Juan Pablo: We have... Look, we've upgraded our search infrastructure, we've upgraded our entire infrastructure, we've upgraded our payments system, we've upgraded our desktop application, we've relaunched our mobile app. It's like we have had at least six or seven successful migration projects because we have had the space to do so and to think so.

Juan Pablo: We are not a huge team either, but we've been very deliberate around and very strategic around which bets we take to move the business forward into the future. There's a big difference in the resourcing, how you use time.

Juan Pablo: At Splice, let's say that for example, an hour of engineering costs the company $100 an hour, the way I help engineers think about their time is "Look, if you're going to spend more than an hour on something, see if you can buy it."

Anthony: Right.

Juan Pablo: This is the same thing I tell Nicole, Nicole is right now though the lead software engineer at Elizabeth & Clarke and I tell her, "You should not be spending your time because you're still spending company money. If you need to spin up more servers in Heroku, do so, you have this up until this amount where you don't even need to ask me about the money you spend."

Juan Pablo: When from our perspective the cost gets close to a yearly salary of an engineer and with benefits and the management overhead, because we're all overhead when this whole thing adds up and it makes sense then we can look at bringing those things in-house.

Juan Pablo: But an example is feature flags. We don't need to build our own feature flag system. Let's look it out what's out there. "Oh LaunchDarkly does some really cool stuff. Let's use it." We don't need to build everything because Splice is not in the business of building software.

Juan Pablo: Splice is in the business of helping musicians make more and better music, helping them get paid. All of our energy, all of our in-house energy has to go to that. We're here for that.

Juan Pablo: The same for Elizabeth & Clarke.

Anthony: Nice. I guess I'm curious like if you had any high level CG words of wisdom or advice for either existing engineering leaders or engineers looking to get into like more of a leadership or management type role. Yeah. What would be some of the bigger high level points that you'd like to share?

Juan Pablo: I think the first thing I like to usually say in this is that all advice sucks because everyone's context is different. Or the advice I'd even give myself as the chief technical husband versus the VP of engineering of a venture backed startup is completely different.

Juan Pablo: Just anything I say and anything you read about leadership or management contextualize, try to understand what context this worked at. Even like, look, I am an immigrant man and who is a little punk so doesn't care about who he pisses off.

Juan Pablo: That has enabled me to be a certain kind of leader. This may not for other people. It applies to engineering advice as well. What works for Google or Amazon or Netflix does not work for your five-person startup. Yes, chaos engineering is a wonderful idea, the idea of having a robot going around, tearing everything down is wonderful, but maybe in the first six months when you have zero users, you shouldn't spend your time there.

Juan Pablo: Contextualize it, because as you progress, as your company progresses, as you reach milestones, you will earn the privilege of investing and learning about other more interesting problems once you have a business that is solid.

Juan Pablo: The advice that I would like to give all engineers is aim to learn as much of the business as much as you try to learn in engineering. If you spent three hours learning about the latest front-end framework, spend three hours understanding what is cause of acquisition, what its relationship to lifetime value is, why its ratio is important, how you can drive higher or lower cause of acquisition in a business when it's with growth marketing or organic search. If you are at a startup that's trying to build a business and you'd know a lot more about, what is it, React hooks, then how your business acquires customers, then you're not really being there for your company.

Juan Pablo: Because you really do not have the foundations to solve the problems for the business that you're trying to support. That would be my advice. Whether you're a manager or an engineer. Being an engineer who really understands the business, gives you super powers.

Anthony: Could not agree more on that front for sure. Also, well, I think that's... I could probably carry this conversation for another real long time to be honest with you, but we should probably give you some years back.

Anthony: I really appreciate your time on this was a super fun stretch of time to spend with you.

Juan Pablo: Anthony. Thank you very much for having me. If people are interested in following my stupid tweets, they can find me on Twitter @buritica, B-U-R-I-T-I-C-A. I'm sorry in advance though. I just say a lot of stuffs.

Anthony: Awesome. Thanks Juan.