TryBugHunt

The Half-Open Thundering Stampede

hardpython

Circuit breaker floods recovering downstream dependency during half-open trial state

0:00
Step 1: Test Traffic & Expose Flaw
"""
Circuit Breaker State Machine
States: CLOSED (normal), OPEN (tripped), HALF_OPEN (canary testing recovery).
"""

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_timeout=5.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = "CLOSED"
        self.failure_count = 0
        self.last_state_change = 0.0
        self.trial_in_flight = False

    def allow_request(self, now):
        if self.state == "CLOSED":
            return True

        if self.state == "OPEN":
            if now - self.last_state_change >= self.recovery_timeout:
                self.state = "HALF_OPEN"
                self.last_state_change = now
                # AI Attempt: Transition to HALF_OPEN and allow request
                # BUG: Does NOT limit concurrency! All concurrent requests enter HALF_OPEN!
                return True
            return False

        if self.state == "HALF_OPEN":
            # BUG: Unconditionally returns True, stampeding the recovering upstream
            return True

        return False

    def record_success(self):
        self.state = "CLOSED"
        self.failure_count = 0
        self.trial_in_flight = False

    def record_failure(self, now):
        self.failure_count += 1
        self.trial_in_flight = False
        if self.state == "HALF_OPEN" or self.failure_count >= self.failure_threshold:
            self.state = "OPEN"
            self.last_state_change = now
Step 1: Test Traffic & Expose Flaw

Simulate downstream recovery where 30 concurrent threads storm the recovering service in HALF_OPEN state, crashing it.

Choose Traffic Scenario to TestClick to simulate
Architectural Guide & Interview Notes
↑ Jump to Interactive Studio

Deep Dive: The Half-Open Thundering Stampede

An engineering analysis of why this AI-generated Pull Request passed isolated unit tests, how multi-tenant traffic exposes the architectural flaw, and how interviewers score your response.

The Root Cause Anti-Pattern

The Interviewer's Scoring Rubric

Candidates are assessed on whether they jump to trial-and-error editing or follow a hypothesis-driven approach. A "Strong Hire" candidate restates the data flow between modules, explains why single-client unit tests passed, crafts an adversarial probe showing cross-tenant leakage, and implements a surgical 1-line patch without breaking existing contracts.

Candidate Strategy for the Code Comprehension Round

  1. 1Map Module ContractsTrace how the gateway invokes the underlying service. Note what data is passed by reference vs. by value.
  2. 2Break It With An Adversarial ProbeSimulate concurrent or sequential traffic across multiple identities to force shared state mutation.
  3. 3Deliver a Surgical PatchMake the minimal required diff to ensure proper per-client state isolation without refactoring unnecessary files.