At GitHub, we're our own biggest customer—we test changes internally before rolling them out to users. But this creates a tricky circular dependency: if GitHub goes down, we can't access our own source code to fix it. This post explores how we leverage eBPF (extended Berkeley Packet Filter) to break these cycles and improve deployment safety. Below are 10 key insights into our approach, from understanding the problem to practical implementation.
1. The Circular Dependency Conundrum
GitHub hosts its own code on github.com. While this ensures we experience what our users do, it also means an outage can prevent us from deploying fixes. This circular dependency—needing GitHub to fix GitHub—is the core challenge. We mitigate this by maintaining a mirror of our code and pre-built assets for rollbacks. However, this only addresses one layer; deployment scripts themselves can introduce new circular dependencies, which is where eBPF comes in.
2. Three Types of Circular Dependencies
During an outage, deployment scripts can encounter three types of circular dependencies. Direct dependencies occur when a script tries to pull a tool from GitHub itself, which is unavailable. Hidden dependencies arise when a local tool checks for updates online, causing hangs if GitHub is down. Transient dependencies happen when a script calls an internal service that, in turn, fetches a binary from GitHub. Each type can block a fix during critical incidents.
3. The MySQL Outage Scenario
Imagine a MySQL outage that prevents GitHub from serving release data. To resolve it, engineers need to deploy a configuration change to MySQL nodes. The deploy script might attempt to pull an open-source tool from GitHub—a direct dependency that fails because GitHub is down. Alternatively, a local servicing tool might check for updates (hidden dependency), or the script could call a migrations service that fetches a new binary (transient dependency). Any of these can stall recovery.
4. Traditional Mitigation: Manual Reviews
Historically, each team owning stateful hosts had to manually review their deployment scripts for circular dependencies. This process is error-prone and time-consuming. Dependencies are often not obvious—they may be buried in third-party tools or indirect calls. Teams had to identify all potential points of failure, from direct network calls to background updates. As GitHub's infrastructure grew, this approach became unsustainable and unreliable.
5. The Need for Automated Enforcement
To solve the scalability problem, GitHub needed a way to automatically detect and block circular dependencies during deployments. The solution must be lightweight, low-overhead, and able to inspect system calls in real time. It should allow fine-grained control—blocking specific call patterns without disrupting other operations. This is where eBPF excels, offering a safe, programmable way to monitor kernel-level events.
6. What is eBPF?
eBPF is a technology that allows you to run sandboxed programs inside the Linux kernel without modifying kernel source code or loading modules. It provides a safe interface for tracing, monitoring, and filtering system calls, network events, and more. For deployment safety, eBPF lets us attach small programs to syscalls like connect() or open() to inspect and potentially block unwanted behavior—all with minimal performance impact.
7. Using eBPF to Monitor Deployment Scripts
GitHub's host-based deployment system uses eBPF to selectively monitor and block calls made by deployment scripts. An eBPF program is attached to relevant syscalls—for example, intercepting network connections to GitHub or file accesses to update-checking binaries. When a script tries to make a call that could create a circular dependency, eBPF can log it, alert, or block it entirely. This prevents hidden failures before they happen.
8. Blocking Direct Dependencies
For direct dependencies, eBPF can block deployment scripts from making outbound HTTP requests to GitHub's own servers. By inspecting the destination IP address or domain in the connect syscall, we can disallow connections to github.com during a deployment. This ensures scripts don't attempt to pull code or tools from the very service being fixed. The eBPF program runs with minimal overhead and can be toggled based on deployment context.
9. Preventing Hidden and Transient Dependencies
Hidden dependencies (e.g., a local tool checking for updates) are trickier because the call originates from a process the script launched. eBPF can trace process hierarchies and monitor all descendant processes. If any child process attempts to contact GitHub or perform other restricted operations, it is flagged. For transient dependencies, eBPF intercepts inter-service API calls that might chain to GitHub, cutting the cycle early. This comprehensive coverage ensures no slip-throughs.
10. Getting Started with eBPF for Your Deployments
You don't need GitHub's scale to benefit from eBPF. Simple tools like bcc or bpftrace allow you to write small eBPF programs that monitor specific syscalls. Start by identifying your own circular dependency risks—what external services does your deployment script rely on that might be down? Then write eBPF hooks to block or log those calls. GitHub's experience shows that eBPF is a powerful, low-risk addition to deployment safety toolkits.
Conclusion: Circular dependencies are a silent threat to deployment reliability, especially when your own platform hosts your code. By leveraging eBPF, GitHub has automated the detection and prevention of these dependencies, reducing reliance on manual reviews. Whether you're a small startup or a large enterprise, eBPF can help you deploy safer and recover faster from outages.