Implementing Chaos Engineering in CI/CD with Gremlin

Shifting Chaos Left

Historically, chaos engineering was restricted to staging or production environments often right after a cascading failure ruined someone's weekend. That is no longer the case. Discovering that a downstream payment service crashes your entire checkout pipeline at 2 AM on a Sunday is an expensive and painful way to discover missing timeout configurations. Shifting left is logically imperative.

The core idea is simple: Gremlin provides an API alongside lightweight agents that allow you to automate failure injection directly within pull requests. You need to determine whether your microservices gracefully handle unexpected performance degradation such as an upstream API taking an extra 500ms to respond or if they freeze and crash. Do not wait for your users to report these issues.

Designing the Automated Chaos Experiment

Eliminate guesswork. This pipeline stage spins up an ephemeral test environment, triggers a targeted latency attack against a selected dependency, executes your integration test suite during the simulated disruption, and cleans up the attack afterward. If your application cannot withstand a slight network delay without failing, the build fails immediately.

name: Resilience Validation
on: [push]
jobs:
 chaos-test:
 runs-on: ubuntu-latest
 steps:
 - name: Deploy ephemeral environment
 run: ./scripts/deploy_test_env.sh

name: Trigger Gremlin Latency Attack id: gremlin_attack env: GREMLIN_API_KEY: ${{ secrets.GREMLIN_API_KEY }} GREMLIN_TEAM_ID: ${{ secrets.GREMLIN_TEAM_ID }} run: | # Inject 500ms latency to all traffic destined for the Payment Service API ATTACK_ID=$(curl -s -X POST https://api.gremlin.com/v1/attacks/new \ -H "Authorization: Key $GREMLIN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "command": { "type": "latency", "args": ["-l", "500"] }, "target": { "type": "Exact", "exact": { "tags": { "service": ["checkout-api"], "env": ["ci-test-env"] } } } }' | jq -r '.uid')

echo "Started Attack ID: $ATTACK_ID"
 echo "ATTACK_ID=$ATTACK_ID" >> $GITHUB_ENV
# Wait for attack to reach 'Running' state
 sleep 15

name: Run Integration Tests (Resilience Validation) run: | # Run tests expecting degraded performance but NOT failure. # The application MUST handle the latency via retries/circuit breakers. pytest tests/integration/test_checkout_flow.py --timeout=30


name: Halt Gremlin Attack (Always run) if: always() env: GREMLIN_API_KEY: ${{ secrets.GREMLIN_API_KEY }} run: | if [ -n "$ATTACK_ID" ]; then echo "Halting attack: $ATTACK_ID" curl -s -X DELETE https://api.gremlin.com/v1/attacks/$ATTACK_ID \ -H "Authorization: Key $GREMLIN_API_KEY" fi

How the Workflow Works

Here is a breakdown of the key steps:

• POST /v1/attacks/new: Directly invokes Gremlin's REST endpoint to programmatically launch the attack from within the check runner, eliminating manual web UI interactions.
• "type": "latency", "args": ["-l", "500"]: Introduces a 500ms latency delay to network packets, effectively simulating a slow downstream API or database connection without severing network connectivity.
• "target": { "tags": ... }: Restricts the attack scope. Gremlin agents only target containers labeled with service=checkout-api within the ci-test-env environment, preventing unintended side effects on shared infrastructure.
• pytest ...: Executes the test suite while latency is actively injected. This validates that circuit breakers and retry policies respond properly; if the application crashes or exceeds the 30-second timeout, pytest exits with an error and blocks the pull request.
• if: always() ... DELETE: Guarantees cleanup. Even if the integration tests fail, GitHub Actions executes this teardown block to send a DELETE request to Gremlin, stopping the active attack before destroying the environment.

Building Continuous Resilience

Executing this workflow on every commit provides immediate feedback on your application's resilience prior to staging deployments. It transforms chaos engineering from a complex quarterly exercise into a continuous, automated verification step ensuring circuit breakers, retries, and fallback mechanisms actually work as designed when subjected to stress.