How Facebook Manages 500+ Engineers

Want to know how Facebook manages daily code pushes from over 500 software engineers? In the company’s latest tech talk they described exactly that.

For software engineers working anywhere, the talk was extremely insightful for learning how the company manages to avoid too many catastrophic errors and how the company is always realeasing updates. Given that the company is known for their continuous pushes (many of which break functionality of the site), it’s interesting to hear about the various technologies are used. In addition to describing how developers check in their code, Chuck Rossi explained the various tools used by the company including IRC Bots, test automation, shadow branch, error tracking, gatekeeper, and more.

One of the interesting IRC Bots developed by Facebook helps to protect which code gets pushed based on which engineers are around when the code is pushed. If you aren’t there, your code won’t be pushed (as any errors will need to be supported by the developer who wrote the code). The company’s test automation suite provides a lot of details about errors taking place on the site to assist engineers in tracking down bugs. Possibly the most interesting tool is the Gatekeeper tool which controls who gets to see which code pushes (or features).

Developers can go in and select to test code on a subsegment of the user base including filters based on country, age, ip address, and journalist filtering tools. You can also use the gate keeper to gradually introduce features in order to test them and see what works and what fails. Another interesting feature is push karma which shows how each engineer is performing. Low performing engineers are more risky to have their code pushed and as such the code push team will scrutinize their code more than others.

Thanks to a modified version of BitTorrent, Facebook is also able to push code to all of Facebook in approximately 15 minutes. All of the juicy technical details are included in the video below so check it out!