Skip to content
Search

Zero Downtime Release

lastmod 2020-09-04created 2020-09-042 edits

Zero Downtime Release: Disruption-free Load Balancing of a Multi-Billion User Website

A paper on Facebook's method for releasing without downtime.

Summary

  • Servers are restarted often : upgrades, bug fixes, security patches
  • With the adoption of Continuous Release
    • At Facebook, deployments went from once a week in 2007 to more than 100 times a week
    • Millions of servers restart every day
    • The same goes for AWS, Atlassian, Yelp, Ebay, and Uber
    • Health checks fail intermittently
  • Traditional deployment methods
    • Blue/Green deployment (AWS CodeDeploy, Kubernetes) : split into two groups of machines, and the load balancer moves traffic to the side that was updated first
      • Expensive. Unsuitable for the Edge, which has a huge number of machines
    • Rolling Updates (Envoy, NGINX, Apache, Kubernetes, AWS) : update one machine at a time gradually while the load balancer adjusts traffic
      • Lower CPU utilization during the update period, and slow iteration.
    • Hot Restart (HAProxy, Envoy) : start a new-version process on the same machine, and once the old process's traffic gradually drains, the new process takes the traffic (SO_REUSEPORT, CMSG, SCM_RIGHTS)
      • Only works for TCP; UDP can be misrouted

"How can we roll out release updates without downtime, with fast iteration, and without disruption?"

Facebook's Traffic Infrastructure

Edge PoP (L7 LB, ProxyGen) - Data Center(L7 LB, ProxyGen) - App. Server(HHVM,MQTT)

  • Restart tier by tier to prevent disruption
  • Machines in the Edge and the Data Center each start a new ProxyGen and do a "Socket TakeOver"
  • "Downstream Connection Reuse" between the Edge and the Data Center
  • "Partial Post Replay" for connections between the Data Center and App Servers

Socket Takeover

Start a new process and hand over the TCP Listening and UDP VIP sockets via SCM_RIGHTS and CMSG

  • SCM_RIGHTS : a socket type that lets a process receive another process's File Descriptor
  • CMSG : lets local processes send control messages to each other via sendmsg()
  • For QUIC, a UDP-based connection, the new process asks the old process for the QUIC ConnectionID of existing connections and forwards those packets to the old process

Partial Post Replay ( App server restart )

  • App servers have two kinds of workloads : short API calls and long HTTP POST calls (Upload)
  • These app servers have no spare resources, so starting a new instance and handing over sockets as Socket Takeover does is impossible, and long upload times are also a problem
  • If an app server is updated in the middle of a long upload, since the proxy doesn't hold all of that content, the server aborts the POST and returns the data received so far to the proxy along with code 379
  • The proxy combines the data received from the old app server with the remaining data and tries to resend it to a new machine

Advantages of Zero Downtime Release

  • About 20% higher CPU utilization compared to Rolling Update
  • Almost no misrouting, compared to Hot Restart, which misrouted up to 20K QUIC packets
  • At Facebook, Socket TakeOver has been in use since 2013 and the rest since 2015

See also

Favorite site

author Zer0 sitemap