Making the Internet faster at Netflix

In our fourth episode of Breaking 404, we caught up with Sergey Fedorov, Director of Engineering, Netflix to understand how one of the world’s biggest and most famous Over-The-Top (OTT) media service provider, Netflix, handles its content delivery and network acceleration to provide uninterrupted services to its users globally.

Subscribe: Spotify| iTunes | Stitcher | SoundCloud | TuneIn

Sachin: Hello everyone and welcome to the 04th episode of Breaking 404, a podcast by HackerEarth for all engineering enthusiasts and professionals to learn from top influencers in the tech world. This is your host Sachin and today I have with me Sergey Fedorov, The Director of Engineering at Netflix. As you all know, Netflix is a media services provider and a production company that most of us have been binge-watching content on for while now. Welcome, Sergey! We’re delighted to have you as a guest on our podcast today.

Sergey: Thanks for having me, Sachin!

Sachin: So to begin with, can you tell the audience a little bit about yourself, a quick introduction about what’s been your professional journey over the years?

Sergey: Yeah, sure. So originally I’m from Russia, from the city of Nizhny Novgorod, which is more of a province town, not very well known. And that’s where I got my education. I went to college from a very good, but also not very well known university and that’s where I had my first dream team back in 2009 when I was in third grade in college. I teamed up with my friends and some super-smart folks to compete in a competition by Microsoft, which is a kind of student contest where you go and create software products. In that year we were supposed to solve one of the big United Nations problems and what we did, we were building a system to monitor and contain the spread of pandemic diseases. Hopefully, that sounds familiar, but it’s what it was in 2009. And as a result, we had unexpected and very exciting success. We happen to take second place in the worldwide competition in the final in Egypt. And that was really exciting to be near the top amongst the 300,000 competing students. And it was really the first pivotal point in my career which really opened the world to me because the internship at Intel quickly followed and it was kind of the R & D scoped, focused on computer graphics and distributed computing. And a year after I was lucky to be one of the few students from Europe to fly, to Redmond, to be a summer intern at Microsoft. It followed with a full-time offer to relocate to the US upon graduation from college in 2011. At Microsoft, I worked in the Bing team helping to scale and optimize the developer ecosystem, particularly the massive continuous deployment and build system for the Bing product that Microsoft. That was a really exciting journey, but the relatively short one, because quickly after an unexpected, the referral happened to me with an invitation to interview for the content delivery team at Netflix, that was just kind of getting started and to help them build the platform and to link and services for the content delivery infrastructure. And quite frankly, I don’t expect that I’ll make it, but I couldn’t pass the opportunity at least to interview. But somehow I made it, very early in my career. I was 23 years old with just a few years of practical experience and it was quite stressful to join the company. I was on an H1B visa. I lacked confidence. I lacked a lot of, kind of relevant to and can experience in that area. Yet I gave it a shot, and I joined a team of world-renowned experts in internet delivery. And, um, I stayed there ever since. I will say that that decision and that risk that I took was the second big milestone in my career. Because from there it allowed me to grow extremely quickly and it allowed me to be truly on the frontier of technology and shape my mindset working for one of the top kinds of leading companies in the Silicon Valley, I’ve been here for about eight years. I initialized, I stayed on the platform and tooling side. I built a monitoring system, a number of data analysis tools. The overall mission of the team is to build the content delivery infrastructure, to support the streaming for Netflix. And over time, we added some extra services on top of pure video delivery. And a few years ago, that’s the group that I joined still staying within the same org, working on some of their extra advanced CDN like functionality, specifically developing some of the ways to accelerate the network interactions between clients and the server, uh, helping to better balance the network traffic, the traffic between clients and the multiple regions in the cloud. And I also worked a little bit on the public-facing tool. So I built the speed task called fast.com, which is one of the most popular internet testing services today powered by open connect CDN. And as of today, I’m a hands-on engineering leader. I don’t really manage the team. Instead, I work extremely cross-functionally with partners and folks across the Netflix engineering group. And I help to kind of drive major engineering initiatives in areas related to client-server network interactions. And I have to improve and evolve different bits and pieces of Netflix infrastructure stack. 

Sachin: Thanks so much for that and it’s an amazing journey. You know, it’s really inspiring to see. Um, would it be fair to say that, you know, you kind of didn’t, it’s been serendipitous for you in some sense, did you plan to be here in the US and you know, be working in an organization like this or it all just happened back when in school, when you decided to participate in the Imagine cup challenge?

Sergey: Well, I wouldn’t say that I didn’t want to do that, but I definitely didn’t expect to, and I definitely didn’t expect to be in a place where I am today. I would say that my whole career was a very unexpected sequence of very fortunate events. I guess, in any case, I was sort of seeking those opportunities and I was not afraid to take a risk and jump on them.

Sachin: Yeah, that’s super inspiring for our audience and, like you correctly said, you got to seek those opportunities, and of course you need a little bit of luck, but if you’re willing to take those risks, doors do open. So, definitely very inspiring. Uh, so a fun question for you. What was the first programming language you, you ever recorded in and you still use that?

Sergey: Yeah, that’s a really interesting question. Um, the first language that I used was Pascal. And, uh, it was when I was 14 years old. So I started my journey with computers relatively late. And so it was kind of in the high school at this point. And the first lines of code that I wrote were actually on paper and I was attending The Sunday boot camp, led by one of the tutors who was preparing some of the folks to compete with ACM style competitions, where you compete on different algorithmic challenges. And he did it for free just for folks to come in. And someone mentioned that to me. I was like, Ooh, that’s interesting. Let me see what it’s about. And for the first few months, I was just doing things like discussing different bits and pieces about programming and all I had was a paper to write different things on. Later on, I of course had a computer and the first few years of Pascal was the primary entry for me to programming. And it was primarily around CLI and some of the algorithmic challenges. It’s only a couple of years ago when I discovered the ID and the graphic interfaces, and it really opened the world of what they could do. Uh, so yeah for me the first programming language is Pascal. And no, I don’t use it, but still have very warm memories of that because I think it’s a really, really good language to start with.

Sachin: Writing your first piece of code on paper. That’s an amazing thing. The folks who are getting into computer science today, they get all these IDEs, autocomplete, you know, all the infrastructure right upfront. Uh, but I think there is some merit in doing things the hard way. It prepares you for challenges and that’s my personal opinion. 

Sergey: Yeah, I definitely agree with that. I’m not sure whether the fact that they had to go through that is an advantage or disadvantage for me, because I really had to understand the very basics and fundamentals. And I was super lucky with a tutor for that. He really didn’t go to the advanced concepts until I really nailed down the fundamentals. And I think having to really painfully go through that, if you’re kind of using a pen and sheets of paper, I think it really forces you to really get it.

Sachin: Right. Makes sense. So Netflix is one of the companies that has been growing massively over the last few years and acquiring millions of users. What are some of those key design and architecture philosophies that engineers at Netflix follow to handle such a scale in terms of network acceleration, as well as content delivery?

Sergey: Yeah, that’s an excellent question. In my case, as I mentioned, I’ve been here for quite a while and I had a lot of fun and enjoyed watching Netflix grow and be part of the amazing engineering teams behind it. But quite frankly, it’s really hard for me to summarize the base concept like use cases, there are so many different aspects of Netflix engineering and challenges, and that there are so many different, amazing things that have happened. So I’ll probably focus a little bit more on some of the bits and pieces that I had on the opportunity to touch. And for me, the big part of the success of growth was actually a step above the pure engineering architecture. It’s firstly rooted in the engineering culture because the first Netflix employees are great people. But second and most importantly, it really enables them to do the best work and gives them a lot of opportunities and freedom to do so. And with that empowerment and freedom to implement the best and to do the best work, I think the engineers are truly opening themselves up for the best possible solutions that really advance the whole architecture and the whole kind of service domain. On the technical side, in my experience, what I think was fundamental to effectively scale infrastructure is the balance that we have had between innovation and risk. And in our case, many fundamental components of our engineering infrastructure are designed to be extremely resilient to different failures and to reduce the blast radius, to contain the scope of different issues and errors. With that’s really embedded like this thinking about errors, thinking about failures, it’s really embedded in the mindset and that leads some of the solutions and some of the implementations to be really robust and really resilient to some of the huge challenges and lots of unexpected demands. And in that aspect is that many systems I designed and thought of to scale 10 X from the current state. So that’s often when we think about the design, we don’t think about today. We think about the 10 X scalability challenge, and that includes both architecture discussions and some of the practical things like performing the skill exercises constantly and stress testing our system, both existing and proposed solutions and constantly making sure that things can scale. So in case, we have unexpected growth, we have confidence that we can manage it. And I think as a result of that, we are not only getting an architecture, that’s stable and scalable. But we also get an architecture that’s safe to innovate on, because we can do the changes with more confidence that we can roll back things. We have confidence in our testing and tooling and with that confidence, I think it’s much as much easier to apply and do your best. 

Sachin: Interesting. So you spoke about designing for innovation as well as being resilient and then kind of designing for a 10X scale in the very beginning. So typically, and this is my experience and I may be wrong here, but when we were younger in our journey as a software engineer, right, we tend to get biased towards building out the solution very quickly and, do not have that discipline to kind of think about the long term scale and all of those challenges, because that is very deliberately put that in place. Right. So, so has there, like, how did your journey kind of evolve in that? Are there any tools, techniques that you use to kind of force yourself to come up with the right architecture? Could you talk a little bit about that? 

Sergey: Well, so I think you were what you touched upon a really great point, but it’s, I would say it’s a slightly different dimension, a bit more of a trade-off between the pace of innovation and sort of the technical debt, the quality of code, so to speak. And I think this is an extremely broad topic, uh, with where I would say their answer would really depend on their application domain. For example, I would give you one answer if you were working on some medical or military services, versus some ways like a social network, consumer and product entertainment sort of services because the risk of failure and the mistake is completely different in that case. And I think another factor comes from the understanding of the problem. There is, I think, a big difference in designing the system for the problem that you understand really well, and you have a pretty good idea that it’s there to stay for quite a while versus more of an exploration where you’re not exactly sure whether this would work or not. You are still trying to kind of get a hand at it. And, uh, quite often you start with a second, with a latter option, and that’s what made you start to do. And I would say that in that case, uh, in my personal experience, I think it’s much more productive to focus on the piece of innovation. And, uh, maybe in some cases build some of the technical debts, maybe in some cases to compromise some of the aspects of the best practices but being able to get things out and get some kind of bits and pieces really quickly and learn from it. And since you are relatively lightweight, it’s much easier to pivot and change direction. At the same time, it doesn’t mean that we all have to be Cowboys and break things here and there. There is a balanced approach. You can still invest in the core principles and the core architecture that allows all those things innovations to happen safely. And I think at Netflix, that’s what really we excelled at. We have some of the core components, some of the core tools that are available for most of the engineers. That’s allowed to make things, uh, and innovate safely while not being overly burdened by some of the hard rules and, uh, some of the complicated principles and gain that experience. And I would say this is sort of a natural process. You have something that’s done relatively quickly. Then you were at this kind of crossroads. Whether now you know, this is a real thing and you’ll have to scale it. And then you would likely apply a different way of thinking or maybe it doesn’t work and well you save a bunch of work by not overcommitting to something really big before confirming that this is useful. And at this point when you were on the road to actually build it for the long term, it might be the proper solution to rebuild what you’ve designed in the past. And it might sound like you were wasting a lot of time. Like you’re doing the double effort. But the way I see it, there’s actually, you’ve saved a lot of time because you were able to relatively cheaply test a bunch of lightweight solutions. You got the confidence, what really works. And now you’re only investing a lot of resources on building the long term for the one thing, and essentially you’ve saved all the time by not doing that for all other ideas that you’ve had. Um, I have them all, it’s sort of a 20, 80 rule that takes 20% of the time to build a working prototype and it takes 80% of the time to productize that and make it resilient and scalable. Um, in many aspects of innovation, it makes sense to start with the 20 and only go for the 80% over time. Yeah, but as I mentioned, it doesn’t mean that everything has to be all or nothing. There are still major principles and it definitely makes sense, especially as you get larger to invest in the main building blocks to enable those things to happen safely. There are always some of the quantum principles that are cheaper and easier to follow in all scenarios. I think one of my favorite books that I was lucky to read early on is the Code Complete by Steve McConnell, which goes into the lots of fundamentals about just writing good and maintainable code, which in most cases doesn’t take more time to write. I just need to follow some relatively simple guidelines. 

Sachin: Gotcha. That’s a very interesting perspective. If I were to summarize it, you were saying that, uh, architecture design is context-dependent. You got to know what the problem is and what you’re optimizing for. And sometimes you’ll go for something lightweight and then optimize it later on because the speed of innovation is also important, but there are always certain principles that one can use without really increasing the development time, certain strong arteries that can help in building robust code. So that’s, you know, definitely interesting. Uh, another fun question. Do you get time to watch any shows, movies on Netflix, and if so, which one’s your personal favorite? 

Sergey: Yeah. Well, while often I don’t have a ton of time to watch I definitely love to have an opportunity to relax and enjoy a good show and Netflix is naturally my go-to place for doing that. And, I’m in a losing battle to keep up with all the great shows that I would like to watch. And, um, it’s quite hard for me to choose one favorite. So I think I’ll cheat and I’ll choose a few instead of just one. So I hope you’re fine with that. I think one thing is I’m a fan of sci-fi as a genre and I really enjoyed Altered Carbon, especially the first season. And over-time I’m also learning that I’m affectionately a fan of bigger shows that I have no idea about. And the one title that I really enjoyed was ‘The End of the F***in world’, which is a dark comedy-drama. It follows the adventures of two teenagers. It’s a really kind of unique piece of content and I truly enjoyed every episode of it. I’m really glad that as a company, we really invest in more and more international content, not just coming from the American or the British world. And the latest favorite for me was ‘The Unorthodox’, which is a German American show with most of the dialogues actually in Yiddish, which is a part of the Orthodox Jewish culture. I enjoyed both the personal story and I also learned a lot about it because I had no idea about this part of the cultural experience for some of the folks. I was both enjoying the ways, done the story behind it, and it had a huge educational component.

Sachin: Thanks for sharing that. So moving back to the technical discussion. So you worked at multiple organizations, you know, Intel, Microsoft, while having the bulk of your time you have spent at Netflix. If you were to look back and think about one or two major technical challenges that you faced and is there something that you would like to talk about and more so along the line of how did you overcome it?

Sergey: Sure. So I think I’ll probably choose one of my favorites. And I think that’s the biggest challenge that I can recall probably by far. And that was my first major project when I joined Netflix. So the task was to build the monitoring seal system for the new CDN infrastructure. And, that was really quick as the task quickly forwards after I joined the CDN group at Netflix. As I mentioned, I was relatively early in my career. I was relatively inexperienced. I know very little about this domain and there’s a huge infrastructure that’s about to like, is being built and we are migrating a lot of video traffic on it. And this is a huge amount of traffic. At that point, Netflix was about one-third of all downstream traffic in North America. So like a third of the internet is there. And here I am like a new employee, that’s not like, Hey, let’s go see some that will tell us how we do like that. We’ll monitor the main state of the system. Like you will, you’ll have to design the main metrics. And really design the system end-to-end on both the backend and the front end, that of UI. And in the true Netflix culture was given the full authority to make its own tactical decisions on product design and implementation. So it was just a full-on like, here’s the problem context, please go and figure it out and we are sure you’re, you’re going to agree. And The biggest challenge of all of that is that many aspects of the system were new and quite unique. And even the folks who were working on this history for a long time, they were quite upfront that we are learning as we go in many ways. So we cannot really give you the precise technical requirements, but we actually wanted to look at. And overall we wanted to keep the whole system and the approach to the monitoring as hands-off as possible, just to make sure that the system reflects some of the architectural components, which reflect some of those principles like a self-healing system that’s resilient to individual failures. So I had to fully understand the engineering solution. I had to model it and there, in terms of the services and the kind of data layer. I had to look at and partner really closely with the operations team to learn a lot about how the system performs, what metrics we should look at, what’s noisy, what’s not. And it’s been quite a ride but especially remembering that was an extremely fun challenge. And I think some of the things that were fun like: a) That I was very unexpected, given the huge responsibility on a pretty critical piece of Netflix infrastructure stack and I was given full control of what I’m using for that. And I could either choose something that I’m comfortable with or something that’s completely new to me. There were really fun interactions with various folks, even though some of my teammates were not necessarily experts in building cloud services or building UIs. There were many other folks at the company who were extremely open and helpful to get me up to speed. I think some of the things that have allowed me to where success is that system is still used today with lots of components still the same as they were built many years ago. I think I made the right decision to focus on very quick iteration. As a matter of fact, the first version of the system fully ready for production and actually used by the on-call by the operations team was done in about two months. And that with me learning how to deploy ADA services in the cloud. I chose Python as a framework, and I knew very little about it before I learned the new UI framework and kind of built the front end in the browser for it. But focusing on the initial core critical components and getting something working was a huge help because it allowed me to build a full feedback loop with the users and started to start learning about the system. And then that calibration of the stakeholders allowed it to iteratively evolve it over time. And even though I didn’t know a lot of different things early on, I was extremely flexible and adaptable. I think some of the key things that were critical for my success to get it done is my ability to wear my mistakes, to be very upfront about mistakes, and actively seek help. And I think that’s one thing that I often notice, different people are not doing for various reasons. They think that it’s not the key to make mistakes, or they are somewhat unskilled or unqualified if they ask for help. For me, it’s been always the opposite. No one, nobody knows everything. Nobody’s perfect. Everyone, everyone makes mistakes. And, uh, the sooner you realize it and the more upfront and open you are around those aspects. The better you’ll be able to find the ideal solution and the faster you’ll be able to learn over time. 

Sachin: Right. So it would have been a lot of confidence for you back in that time. Like you said, you were early in your career and the organization just said, Hey, this is your project. You have complete authority to just go out and do. And when we know, we’re sure you do the right thing, it must have also given you a lot of confidence, right? 

Sergey: Well, quite honestly, initially it didn’t. Initially, it freaked me out because I was especially after companies like Intel or Microsoft, where their approach is very different. And I only had a few years of experience and I was not a well-known expert. That was very unusual. It was very scary. I would say the confidence really came months later when I was starting to see that the key is something that’s been built, that’s been used, I’m getting good feedback. And people are thanking me for working on that. They are giving some constructive feedback. They make suggestions, and I’m becoming the person who actually knows how to do it. Then in some of the domains, I’m becoming the most knowledgeable person, which is natural when you’ve worked on that. I would say confidence really came at this point, which was many months after that I would say probably a year or so. Maybe even after that. 

Sachin: Got it. That makes sense. So, moving on to the next question, do you believe engineers should be specialists or generalists and how does this really impact career growth in the mid to long term?

Sergey: Yeah, that’s a great question. And personally, I don’t think there is one right style. To me, it’s like comparing what is more important, front end or backend. I think any effective team requires both types of personalities. And for nearly any major project, you need to rely on those because if you think about it, if you have a team of only specialists, you’ll have really well done individual pieces of the system, but it will be really hard to connect them together. Similarly, if you only have generalists, you may have liked a lot of breaths, but it would be really hard to actually build truly innovative aspects of the products because that’s the point of focusing on the one area that you have to give a compromise and not know something else. I think ultimately for effective teams, you need both times and you really need to have effective and efficient communication between both groups of them. You need them to be able to work together as a very well-aligned team. Uh, so yeah, I think for me personally, like what type of engineer to be is more of a personal choice. And also in my experience, there have been many opportunities to change the preference. You don’t have to necessarily pick ones and stick to that. You can mix it as you can go into one area or another. In my case I’ve been a specialist at some point and actually in the early stages of my career, I was probably the most specialized. When I was at Intel, it was a heavily dedicated area focused on computer graphics. I was optimizing some of the retracing algorithms and methodologies, what specific types of the network of Intel hardware. So it was all of low-level C, assembly, and some of the specific Intel instructions for, to get the most out of it. At Microsoft, I worked on search and some of the developer experience, then I switched to network and networking. So it’s, it’s sort of a mix. So I think I was becoming more of a generalist over time. On the tactical stuff, but still, I’m specializing in which area on the larger area. But this is also a personal choice and the industry and the technology is moving so fast that even if you were the expert in one area, very specialized today, in fact, years, you might, if you’re not keeping up, you might be off-site or that area is not everything. And you don’t have to stay there. You may find the passion somewhere else and switch to it. Or you can always stay as a generalist and just explore and move alongside technology growth. 

Sachin: Yeah. So if I, if I were to summarize that, uh, you’re saying teams eventually need both kinds of engineers, and it really boils down to a personal choice, whether you want to be a specialist or a generalist, but, you know, given the current pace at which like you said, technology is evolving, it’s really hard to just be narrow jacketed into one thing, you know, because things around you would just constantly change and then you’ll have to adapt to them.

Sergey: Well, I think it’s on the latter point, I would say, I would say really depends. There are some of the areas that remain relevant, uh, for quite a while, for example, talking about the networking area, we’re still using TCP and that’s the technology from the 1980s. And there is still a lot of really interesting research and developments going on. And if anything, in recent times, the pace of development has accelerated. And yet, someone who specialized in that in the nineties would be still very relevant today. So in some of the areas you can still, you can specialize and you’ll be growing your influence. You’re growing your impact over time, but there’s no guarantee and it’s really hard to predict those areas. So I think, well, if you’re really passionate about it, it makes sense to stay. But I would say you should always be ready to pivot go and dig into something else. 

Sachin: That makes sense. So another fun question, which software framework or tool do you admire the most?

Sergey: I think my answer will be probably quite boring at that. I’m pragmatic, I don’t have a favorite intentionally. I tend to follow the principle that there is always the right tool for the job. And as that principal and trying to avoid any sort of absolute beliefs or absolute favorites. Having said that, uh, the very few frameworks that I personally like and they’ve helped me quite a bit. I like Python quite a bit for its simplicity, its flexibility. From personal experience, it’s one language I was able to deliver a fully usable work in projects that are being consistently used for several years after in just two weeks. And before those two weeks, I barely knew Python. So I think that shows the extreme power of the language, how easy it is to pick up and do something actually practically useful. Related to Python, I like pandas quite a bit, which is a statistical library with some of the ways to do time serious or data frame analysis. From the network world, I should mention Wireshark, which is a general tool and it’s fantastic and definitely go-to for me to understand all that happens on the network communications at an insane level of detail. In terms of overall impact, I should mention the Hive, which is a big data framework. While it’s becoming sort of obsolete technology right now replaced by Spark and all of the following innovations. I think it’s really created a revolution in many ways. In its own time, creating, making it possible to access enormous amounts of data, very easily using the very familiar SQL like language. And for me, I happen to use it around the time and it really had a massive impact on a number of insights into things I was able to do. 

Sachin: Interesting. I agree with you on the Python bit. I myself learned Python very quickly and saw the power of the framework and the versatility in terms of the things that allow you to do, like there’s hardly any industry domain, where, where you can’t use Python to very quickly prototype. Right? So in that sense, it’s a very powerful and versatile framework. Thanks for that. Let’s move on to the next one. You know, given the current scenario around COVID-19 everybody working from home, what’s your take on remote engineering teams? Personally, what do you feel about remote work and you mentioned that your work involves a lot of cross-team collaboration? So how has that been impacted positively or negatively in recent months? 

Sergey: Yeah, so I think for the first question for remote work in general, the group that I’m in the content delivery group at Netflix, we were remote from the ground up. So our teammates, they are all scattered around the globe all the way from Latin America, to the US, to Europe, to Asia and all the way to Australia. In terms of working remotely we’ve figured out the way to do it very efficiently, but what’s challenging is that now we are a hundred percent remote because what you’ve done in the past, like some of the folks that are in the office, like in Los Gatos in California, some of the folks that are working from home and we effectively collaborate with each other, but every quarter we will do what we call the group of sites where everyone would get together in the same place. We will have a number of meetings and discussions, both formal and informal, where you’ll be able to sort of put the actual person to their image that you see on the screen. And you’ll be able to really know those persons, those folks, your teammates outside of their direct work domain. In my experience, that’s hugely impactful in terms of affecting your future interactions and building a relationship and working together as efficiently as possible. And with today’s COVID-19 world, we are losing that. So we are 100% remote and even though it hasn’t been a hugely long period of time, based on some estimates, it might take a while for us to work the way. And, it’s a challenge not to have some of that context and to lose some of this nonverbal thesis of communication. To your question, it’s also much harder to build new relationships. I would say it’s still possible to sustain some of the relationships that you’ve built from the past based on previous work together, previous interactions. But when you have to meet a new partner or when there is a new person joining the team, it’s extremely hard to find the common commonalities or find the same language, when you only have a chance to interact via chat or VC. I would say we are definitely trying different things to fix that. We haven’t found the perfect solution. We hope to find it. I would say we also call that you won’t have to find it for the longterm. Hopefully, the COVID-19 situation will be addressed as quickly as possible. But yeah, that’s the very few things that I would say that’s becoming even more critical. First is extremely clear and efficient communication. It becomes paramount and the sharing of the context, and especially from the leadership side, it becomes extremely important to make sure that everyone is on the same page. And that you really need to double down on all of the context sharing in that sense. And, uh, in terms of the partners, I think it’s extremely important to make sure that folks feel safe when they work that way. Because as part of not having a chance to talk face to face, it’s a great environment too, uh, for some sort of or kind of fear and paranoia to build up. Um, it’s harder to make sure like how you’re doing, how things are going, especially when there’s lots of stress happening on the personal side as well and there is lots of research that shows that we are not productive when we are experiencing high levels of stress. And, uh, I would say that’s on the individual side. It’s really critical to make sure that both yourself and all the partners around you are feeling safe and in the right state of mind primarily. And then it comes down to where something that’s really difficult, which is building trust between each other to do the best work. Even in the case, when you are very far away from each other, you really need to make sure that once you share it’s all the context about the problems, about the solutions, about the ideas. You have the full trust in others to do the best work to address some of the things and help you with some of the things or ask you for help as well.

Sachin: Got it. That makes sense. I completely agree with you on the fact that. Having a shared conversation in person is definitely different from having it over video and the kind of relationships that get built subconsciously is very, very hard to replicate that on video and, and I’m with you that hopefully, we can safely return back to work at some point in time sooner, rather than later.

Sergey: In the meantime, but one sort of thing that we are doing is that we are making sure that we still communicate informally. One thing that we do as a team, we have three times a week, we have a virtual breakfast. If someone can’t make it that’s okay. But otherwise, folks just have an informal breakfast together. And we tried to talk about things unrelated to work, uh, just any subject, basically something that you would have as a conversation if you went for the team lunch outside. 

Sachin: That’s interesting. And is that working out well, like, do you see people interacting and joining these discussions?

Sergey: In my opinion, yes. I think personally I feel much more connected after those things. When I have an opportunity to hear and see folks discussing aspects outside of the specific tactical work domain. I think it’s useful for others. It’s good for morality. And I’m seeing that many other teams experimenting with different ideas along the same lines.

Sachin: Nice. So, onto the next question, you know the tech interview process is talked about a lot. People have their different opinions. What’s your take on given the current norms around tech assessments and interviews? What do you think is unoptimized today or what in your opinion should be changed?

Sergey: Cool. Would you mind clarifying, are you asking specifically about the current, highly remote situation or interviewing in general? 

Sachin: Tech interviewing in general, the process that, you know, that is there. I’m assuming Netflix, other than the cultural aspects, maybe from a talking perspective and your previous organizations have had similar methods or processes. So do you think there’s something that we could do better? Not in the context of COVID-19 per se, but in general. 

Sergey: All right, got it. I think it’s generally, I think there are lots of challenges with a typical interview process. And if you think about it, the typical interview experience where we have someone coming in for 30-40 minutes, solving some of the specific problems on the whiteboard, or sometimes on the shared screen, it’s not exactly what we experience in the day to day life. Quite often the problems are not very well defined, but you very rarely have specific constraints on time to solve it. Most of the time or I hope almost all of the time, there is much less stress in the typical work environment and you’re relating the person to something that they might not have the subtle experience in the workplace. At Netflix, many teams do try different – different approaches. We don’t have a single right way that everyone has to follow. Depending on the team, depending on the application domain, often depending on the candidate, folks will try to adjust the interview process. In our case, what we have tried and what we genuinely try to do, we’re avoiding very typical whiteboard questions. We try to focus on some of the problems that are much closer to real life. We try to lean on some of the homework, take-home assessments if possible. If the candidate has time to perform that and a general, I think this gives a much better read of the candidate skills because they can take it in the environment that they’re used to. There is no stress. There is not someone looking over the shoulder. And you can assess a much broader range of skills, not just a specific, like, I know how to solve it the way I don’t know how to solve it, but how do you write code? How do you document that? How do you structure it? And in some cases like even how do you deploy it? And those operational aspects of coding is a big part of engineering life, which are extremely important to assess as well. And I would say generally it’s a huge benefit if a candidate has something to share in the open-source and the open environment. If they have a project that someone can just follow or can take a look at the code, I would say that’s one of the best assessments of the skills it has just working, that’s been used, and that has been produced. It still doesn’t cover all aspects of it. It’s really hard to assess the qualities like teamwork or some of the compatibilities with the teammates. Um, those areas tend to be quite freaky. Um, and honestly, I don’t think I have any ideal solutions for that other than to make sure that as many partners for the new hire as possible are actively participating in the interview process. They have the ability to chat a little bit more and get an idea of whether they can work with a specific person and achieve strategies to do that depending on the team size or particular situation. 

Sachin: Got it. So if I were to summarize this, if the interviewing process can be as much as possible, close to the actual work that you’ll be doing, while eliminating or reducing the stress that one goes through in the interview process, that should bring out a more fair assessment of the candidate.

Sergey: I would say, yeah, at least that’s the general strategy that in my experience, in the interview processes, I tend to follow. 

Sachin: Interesting. So, another fun question, if not engineering, what alternate profession you would have seen yourself excel in?

Sergey: I would say it really depends on the time when you would ask me. I happen to get excited very easily and my immediate passions change quite frequently. As of recently, I would say I could easily find myself having a microbrewery or running like a barbecue-style restaurant. So those are the two things that I found interesting and I’m doing quite consistently for the last few years. I homebrew in my garage. I also have a few kegs of homebrew on top. And I have three grills in my backyard and those things complement each other very nicely and they bring lots of joy to myself and my friends as well. 

Sachin: That’s really nice to know that you have a home brewery and you said you’ve been doing it for two years now.

Sergey: Uh, well, I would say more about five years.

Sachin: That’s an interesting hobby. Uh, so, you know, with that we are almost towards the end of our podcast. The final question today: So if there was like one tip that you could give to your peers, people who are at a similar role and even to those people who want to step up and, you know, come to a role where you are today, what would that be?

Sergey: I think I would respond with sort of a catchy phrase from our Netflix culture deck. And I think that defines the leadership style that the company tends to follow and that I personally strive for, which is leading with context and not control. And what that means is that as a leader, learning to gather, summarize, and effectively communicate the most critical goals and challenges that the business, you, your group faces and effectively share it with the team but trust the individual contributors and your partners to find the most optimal solution and execute it and not trying to do both at the same time, which is really hard to do it, but that’s, that’s what often happens. Because I think that empowering the folks with the proper knowledge and the kind of context around the problem, encourages folks to fully own it and better understand it and they become much more committed to that. And that has a much higher chance to provide the best optimal solution versus the situation when someone just tells you what to do like ABC. And that you’ll get more commitments. I think it inspires folks to grow much more. And I think overall it makes the person who is able to foster such an environment a much better leader, which is also extremely challenging to do. You’ve asked me for advice like for the managers, directors. I’m not sure I’m qualified to give that advice. Uh, it’s more of some things that I’m working on to prove myself and, as someone who is relatively new to their engineering leadership role, I’m finding lots of challenges and struggles, and also those things where you feel like, uh, you might know various aspects of the solution, but you don’t really have to be actively involved in every bits and piece of it and balancing those things is a huge challenge. And personally, as I progress on those, I see that I’m becoming more efficient and more useful for the group and for the company. And I think it’s a kind of ideal and useful goal to live by.

Sachin: So it’s more about empowering people so that they can find their own solutions. And then certain times you may even have the right solution in your hand, but you don’t want to do it because you want the people to fight their own battles. And maybe they come up with something completely different that you might not have imagined. So fostering that innovation is important. 

Sergey: Yeah. I would say empowering with the context around the solution and empowering down with the trust for them to execute on it and fully own the implementation.

Sachin: Makes so much sense. And I think you’ve gone through the same in your journey at Netflix. From the early days, you got the context and you got full control.

Sergey: Absolutely. Yes, I experienced that and the full power of it as an individual contributor. And now I’m actively trying to get better at doing that for others as well. 

Sachin: Yep. That makes sense. Sergey, it was a pleasure having you today as part of this episode, I really appreciate you taking your time. It was informative and insightful, and I definitely enjoyed listening. I hope our listeners also have a great time listening to you. 

Sergey: Thanks a lot, Sachin! session. It’s been a pleasure to have a chance to share my story. 

Sachin: Thank you. So, this brings us to the end of today’s episode of Breaking 404. Stay tuned for more such awesome enlightening episodes. Don’t forget to subscribe to our channel ‘Breaking 404 by HackerEarth’ on Itunes, Spotify, Google Podcasts, SoundCloud and TuneIn. This is Sachin, your host signing off until next time. Thank you so much, everyone!

About Sergey Fedorov
Sergey Fedorov is a hands-on engineering leader at Netflix. After working on computer graphics at Intel, and developer tools at Microsoft, he was an early engineer in the Open Connect — team that runs Netflix’s content delivery infrastructure delivering 13% of the world Internet traffic. Sergey spent years building monitoring and data analysis systems for video streaming and now focuses on improving interactive client-server communications to achieve better performance, reliability, and control over Netflix network traffic. He is also the author and maintainer of FAST.com — one of the most popular Internet speed tests. Sergey is a strong advocate of an observable approach to engineering and making data-driven decisions to improve and evolve end-to-end system architectures.

Sergey holds a BS and MS degrees from the Nizhny Novgorod State University in Russia.

Finding actionable signals in loosely controlled environments is what keeps Sergey awake, much better than caffeine. This might also explain why outside of work he can be seen playing ice hockey, brewing beer, or exploring exotic travel destinations (which are lately much closer to his home in Los Gatos, California, but nevertheless just as adventurous).

Links:
Twitter: @sfedov
Website: sfedov.com