Final Chapter: Jenkins EFS Problem Solved - From 100% to 0% Throughput Usage

TL;DR

After three articles tracking down a Jenkins EFS performance issue, enabling Shared Library cache reduced throughput usage from 100% spikes to near 0%. This article covers the final results and the complete SRE process from emergency response to permanent fix.

Previous Episodes (Quick Recap)

This is the final article in a 4-part series:

• 
  

  Episode 1: How Git Temp Files Killed Our Jenkins Performance

  
  
  
  
  
  • Problem: Jenkins slowed down, Git clone failures, 504 errors
  
  
  • Discovery: EFS metadata IOPS exhaustion
  
  
  • Culprit: ~15GB of tmp_pack_* files accumulating
  
  
  • Emergency fix: Provisioned throughput 300 MiB/s + cleanup job
  
  
  
  
  
  



• 
  
  

  Episode 2: How I Spent $69 in 26 Hours (and How to Avoid It)

  
  
  
  
  
  • Cost: $69 in 26 hours
  
  
  • Lesson: Didn't know about Elastic Throughput (1/20th the cost)
  
  
  • Learning: Decision process was sound, but not the optimal solution
  
  
  
  
  
  



• 
  
  

  Episode 3: How Jenkins Slowly Drained Our EFS Burst Credits Over 2 Weeks

  
  
  
  
  
  • Key finding: Symptom appeared on 1/26, but root cause started on 1/13
  
  
  • Multiple factors:
  
  
  • Shared Library cache was disabled (existed before)
  
  
  • Changed to disposable agent approach (1/13) → metadata IOPS increased
  
  
  • Development accelerated in new year → more builds
  
  
  • Root fix: Enabled Shared Library cache, set Refresh time to 180 minutes
  
  
  
  
  
  

The Result: Dramatic Improvement

After enabling the Shared Library cache setting (Cache fetched versions on controller for quick retrieval) and setting Refresh time to 180 minutes, the EFS throughput usage changed dramatically.

Before the fix (left side, 06:00-12:00):

• Throughput usage spiking to 100% frequently


• Almost constantly under high load


• Far exceeding the 75% warning zone

After the fix (after 12:00):

• Throughput usage dropped dramatically and stabilized


• Baseline near 0%


• Regular small spikes every 3 hours (max ~60%)

The 3-hour spikes are from the Shared Library cache refresh checks (Refresh time: 180 minutes). In other words, it's working exactly as expected.

Honestly, I didn't expect the effect to be this clean.

The Complete Timeline: 5 Throughput Modes

Over the course of this incident, we went through 5 different EFS throughput modes:

Mode Progression:

Bursting (original)

    ↓ 1/27 emergency response

Provisioned 300 MiB/s (26 hours)

    ↓ 1/28 cost reduction

Elastic Throughput (1 day)

    ↓ 1/29 verification

Provisioned 10 MiB/s (current)

    ↓ planned

Bursting (return to original)

Cost Comparison:

Why We Changed From Elastic Throughput

Elastic Throughput turned out to be "surprisingly costly":

• Daily cost: ~$8


• Monthly estimate: ~$240 (~¥35,000)

In contrast, Provisioned 10 MiB/s costs ~$72/month (~¥10,000). Given our current usage pattern (average throughput a few %, max ~60%), 10 MiB/s is sufficient.

However, this is just for the verification period. We plan to eventually return to Bursting mode.

Hypothesis Verification

Let's verify the hypothesis from Episode 3.

Initial Hypothesis

Did the change to disposable agent approach (1/13) cause the metadata IOPS spike?

Answer: Partially correct, but not the main culprit.

The Real Culprit

Shared Library cache was disabled (existed before the incident)

Just enabling the cache brought throughput usage down to near 0%. This means Shared Library's full fetch on every build was consuming the overwhelming majority of metadata IOPS.

Impact of Disposable Agent Approach

So was the disposable agent approach irrelevant?

Not quite. I believe the change to disposable agents was one factor that accelerated Burst Credit depletion.

The combination of three factors:

• Shared Library cache disabled (pre-existing) → Controller-side metadata IOPS on every build


• Disposable agent approach (from 1/13) → Agent-side metadata IOPS on every build


• Development acceleration in new year → increased build frequency

These three factors together caused rapid Burst Credit depletion from 1/13, with symptoms appearing 2 weeks later on 1/26.

The SRE Process: From Detection to Resolution

Looking at the timeline of our response, you can see a clear SRE process:

The Complete SRE Workflow:

1. 
   

   Problem Detection (1/27 morning)

   
   
   
   
   
   • Symptoms: Jenkins slow, Git clone failures, 504 errors
   
   
   • Metrics check: EFS throughput usage at 100%
   
   
   • Time: ~30 minutes
   
   
   
   


2. 
   

   Emergency Response (1/27 morning)

   
   
   
   
   
   • Decision: Change to Provisioned throughput 300 MiB/s (executed next day)
   
   
   • Goal: Ensure investigation could continue
   
   
   • Tradeoff: High cost vs. continued development
   
   
   
   


3. 
   

   Impact Mitigation (1/27 afternoon)

   
   
   
   
   
   • Created cleanup job
   
   
   • Planned tmp_pack_* deletion
   
   
   • Implemented recurrence prevention
   
   
   
   


4. 
   

   Root Cause Investigation (1/27-1/30)

   
   
   
   
   
   • Stage 1: Found tmp_pack_* accumulation
   
   
   • Stage 2: Burst Credit Balance graph analysis revealed 1/13 as starting point
   
   
   • Stage 3: Discovered Shared Library cache was disabled
   
   
   
   

To be honest, I got stuck here. When I found tmp_pack_*, I thought "this is it," but it was actually just part of the symptom. Reviewing the graphs chronologically led me to the true root cause.

1. 
   

   Permanent Fix (1/30)

   
   
   
   
   
   • Enabled Shared Library cache
   
   
   • Set Refresh time: 180 minutes
   
   
   • Optimized throughput mode
   
   
   
   


2. 
   

   Effect Measurement (1/30 onward)

   
   
   
   
   
   • Confirmed dramatic improvement in throughput usage
   
   
   • 3-hour spikes are as expected
   
   
   • Continuing to monitor
   
   
   
   


3. 
   

   Reflection & Knowledge Sharing (this article)

   
   
   
   
   
   • Reflected on cost decisions (didn't know about Elastic Throughput)
   
   
   • Understood the multiple contributing factors
   
   
   • Sharing knowledge with the organization
   
   
   
   

This last part is surprisingly crucial. It's not just about solving the problem, but articulating "why it happened" and "how we decided" to apply to future situations.

Outstanding Issues and Next Steps

Short-term Tasks

1. Return to Bursting Mode

We're currently running on Provisioned 10 MiB/s, but plan to return to Bursting mode eventually.

What to check before switching back:

• Has Burst Credit Balance recovered sufficiently?


• Are new tmp_pack_* files being generated?


• Is the cleanup job working correctly?

2. Strengthen Monitoring

This incident could have been caught earlier with proper monitoring.

Alerts to set up:

• EFS throughput usage > 75%


• Burst Credit Balance < threshold (TBD)


• Abnormal storage capacity increase

Long-term Considerations

Reconsidering the Disposable Agent Approach

We're continuing with the disposable agent approach, but its impact on metadata IOPS can't be ignored.

Options to consider:

• Extend agent lifecycle slightly to reuse across multiple jobs


• Share Git cache on EFS across all agents


• Place cache in S3 and sync on startup

Finding the right balance between cost and performance is the next challenge.

Key Takeaways

Looking back at the timeline, here's what I learned.

Technical Lessons

1. 
   

   EFS Metadata IOPS Characteristics

   
   
   
   
   
   • Massive small file operations are deadly
   
   
   • File count matters more than storage capacity
   
   
   • Burst Credit management is key
   
   
   
   


2. 
   

   Jenkins Caching Mechanisms

   
   
   
   
   
   • Importance of Shared Library cache
   
   
   • Setting the right Refresh time balance
   
   
   • Hidden costs of disabled caching
   
   
   
   


3. 
   

   Throughput Mode Selection

   
   
   
   
   
   • Elastic Throughput isn't a silver bullet
   
   
   • Optimization based on usage patterns
   
   
   • Importance of cost estimation
   
   
   
   

Process Lessons

But more importantly, it's about "how we decided."

Emergency Decision Making:

• Make decisions without perfect information


• Prioritize avoiding worst-case scenarios


• Clarify tradeoffs

Investigation Approach:

• Look at graphs chronologically, not just symptoms


• Form hypotheses, test them, move to next hypothesis if wrong


• Acknowledge when you're stuck

Accountability:

• Costs can be explained after the fact


• Articulate the decision process


• Share both successes and failures

I regret not knowing about Elastic Throughput, but I don't regret the decision to "ensure investigation could continue."

The $69 tuition might have been expensive, but I think I got more than that in learning.

Related Articles

This is the final article in the series:

• 
  Episode 1: How Git Temp Files Killed Our Jenkins Performance
  


• 
  Episode 2: How I Spent $69 in 26 Hours
  


• 
  Episode 3: How Jenkins Slowly Drained Our EFS Burst Credits
  

I write more about SRE decision-making processes and the thinking behind technical choices on my blog.

Check it out: https://tielec.blog/en/tech/sre/jenkins-efs-final-report