Debugging in Production and Distributed SystemsLesson 6.3
How to use feature flags to safely test fixes in production
feature flags, gradual rollout, kill switch, canary deployment, flag-based debugging, targeted user testing
Feature Flags as a Debugging Tool
Feature flags let you deploy code to production without activating it for all users. This means you can test a bug fix in production -- with real data and real infrastructure -- against a small percentage of traffic before rolling it out fully. If the fix causes new problems, you turn off the flag instantly.
Implementing a Simple Flag
// Simple flag check backed by a config service
const flags = {
useNewDiscountEngine: process.env.NEW_DISCOUNT_ENGINE === 'true'
};
function calculateDiscount(order) {
if (flags.useNewDiscountEngine) {
return newDiscountEngine(order); // new fix, active for test users
}
return legacyDiscountEngine(order); // current code, all other users
}
// With a flag service (LaunchDarkly, Unleash, etc.)
const enabled = await flagClient.isEnabled('new-discount-engine', {
userId: user.id,
rolloutPercentage: 5 // enable for 5% of users
});
The Kill Switch Pattern
Every feature flag for a bug fix should also function as a kill switch -- a way to instantly revert to old behavior without redeployment. Deploy the fix behind a flag. Ramp up slowly: 1%, 5%, 25%, 50%, 100%. Monitor error rates and key metrics at each step. If errors spike, set the flag to 0% in seconds.
