Peloton's engineering team makes the case for test in production
Peloton cut its performance environment and saved 40% on infrastructure costs without disruptions. How? Essential prerequisites and key metrics protect the user-facing environment.
Peloton had a problem: Manage peak traffic events -- greater than 450,000 requests per second -- while controlling its AWS infrastructure costs. In 2024, Peloton's engineering team made a bold decision to reduce costs and improve performance. It eliminated the performance environment entirely and chose to test in production instead.
"We questioned why [the performance environment] is actually even needed, and that's what led us to this approach of being able to test in production," said Taq Karim, senior director of engineering at Peloton.
Peloton, known for its interactive fitness equipment and tech platform, determined that its performance environment wasn't worth the ROI as a testing stage. The change saved an estimated 30% to 40% on infrastructure spending. Additionally, there have been no major incidents, including during highest-stakes events such as the annual Turkey Burn workout.
To adopt testing in production, the company shifted engineering to design for failure, named cost as an engineering metric alongside latency and uptime and used a mix of custom-built and off-the-shelf management tools to support the member experience.
The performance environment problem
A performance environment is commonly used to prevent outages and slowdowns. It simulates real-world workloads on dedicated systems that replicate the hardware and network traffic of production. Performance environments enable engineers to perform stress testing and identify bottlenecks before releasing updates to actual users.
A performance environment can be costly. Peloton encountered various issues, such as:
High duplication costs.
Dissimilarity to production.
High complexity and maintenance.
Slower engineer velocity.
Attempting to duplicate a production environment can significantly raise infrastructure costs, especially when paired with imprecise rightsizing. Additionally, Peloton's performance environment was dissimilar to the actual production environment due to numerous distributed systems and complex technologies, which are difficult to precisely duplicate. Even slight differences between environments created "mixed signals," according to Alex Niderberg, director of site reliability engineering at Peloton.
"I think the [difference between environments] is often not only the complexity of like queues, databases and the way services interact, but also the data," Niderberg said. Without exact data replication, decisions are based on somewhat inaccurate information, which can, in turn, affect performance.
The engineering team can now properly re-replicate member behavior without encoding any assumptions, which is another big risk in the duplicate environment, according to Karim.
The cost of replication isn't just about the infrastructure; it's also about the engineering effort required to ensure the two environments are identical. Maintaining such precision adds to the overall costs, as a company continues to evolve its systems and features. It also takes up valuable engineering time that a company could spend more productively, such as enhancing member features.
Without the performance environment, engineers now develop locally and use a high-fidelity staging environment that mimics production as their final gate before deployment.
Treat cost as a first-class metric
Peloton elevated cost optimization from a finance concern to a core engineering discipline, treating cost observability as a first-class metric, the same as critical performance metrics like latency and uptime. It decided to tightly correlate cost efficiency and engineering efficiency. Cost optimization became an engineering problem to be solved, analyzed and iterated upon.
Taq Karim
Rather than closing off or hiding data, Peloton makes cost and component information highly visible so everyone in the engineering organization can see it, learn from it and make informed decisions.
"From a cultural perspective, I think what we strive to do is build an information furnace instead of an information fridge," Karim said. With the furnace method, data is highly visible and continuously updated, rather than hidden in a "fridge" (e.g., a centralized location) requiring team members to hunt for it. This fosters a well-informed and collaborative team.
All on-call engineers gather weekly to review their team's service performance and cost-effectiveness metrics. These reviews are open to leadership and the entire engineering team, creating an open forum where people can learn together how to make systems more efficient and effective. Teams share success stories, which creates a positive reinforcement loop.
Bridge the tool gap to preserve performance
Turkey Burn is a live class event Peloton holds on Thanksgiving Day. What started as a community-driven event has snowballed into the highest-traffic day and the largest traffic surge of the year on the Peloton platform. In 2025, about 34,500 people worked out on the same ride -- at the same time.
Alex Niderberg
"If you're not scaled to a certain level, you'll end up with a time delay of, say, three to eight minutes [when users attempt to join a class]. And that's absolutely unacceptable for us," Niderberg said.
Peloton's safeguard against saturating network connections or available capacity is to pre-scale the infrastructure to the targeted level. The move requires an internal tool, called Auto Left, that works in tandem with Karpenter, an open source, high-performance Kubernetes cluster autoscaler. These two tools scale up AWS EC2 instances in less than two minutes if the cluster's available capacity is depleted due to rising traffic.
Peloton also created Gatekeeper, a tool designed to prevent database queries from running too long and saturating the database. This tool set keeps the system more efficient and less expensive, especially since engineers can integrate custom tooling with off-the-shelf observability tools.
"We don't build what [we] can buy. But we have to build what [we] can't," Karim said.
When they can, they opt to use open source options or cloud provider offerings. Peloton considers it an 80/20 problem: 80% standard, with Peloton innovation and engineering teams inventing 20% to bridge the gap.
Design for failure
Without a performance environment, testing happens in production. Therefore, Peloton's engineers do not design for how a service will perform when everything goes right. Instead, they design for how the system will perform if anything goes wrong.
Designing for failure is built into Peloton's engineering ethos; every team thinks about what happens if there is a lack of availability or if the service gets saturated, according to Karim.
Designing for failure across a system is hard. It takes time, it takes a lot of investigation.
Taq KarimSenior director of engineering at Peloton
"Designing for failure across a system is hard. It takes time, it takes a lot of investigation," Karim said. "It's worth the work, and it ensures that if anything goes wrong, your system is up and available. The hard part is to define and understand ‘what does that failure mean?’"
Peloton strives to ensure its systems are available to its members whenever they want to work out, anywhere, anytime. Certain member features must always be available; these are referred to as Tier-0. To handle failures without disrupting the user, Peloton implements local fallbacks -- such as caching workouts directly on the device -- and graceful degradation.
Peloton's approach to testing in production is a form of chaos engineering. Teams apply loads strategically at less critical times -- often 2.5 times the expected capacity -- to understand the breaking point. When they find anomalies, they shut down the test, fix the anomaly and then continue testing.
Additionally, because engineers design for failure, they do not feel the need to overprovision services in case of performance issues. Teams can find a closer match between actual system volume requirements and provisioned resources.
Are you ready for testing in production?
Karim noted five non-negotiables before a company attempts to remove the performance environment:
An established observability system. A company needs to see latency, error rates and saturation per service in near real time, because that's what tells you when to stop the test.
Automated kill switches. If stopping a test means a human paging another human, you're not ready.
Tier-0 services. Know the services you need to protect while deliberately stressing your entire system.
Isolated test traffic data. At a minimum, engineers need a reliable way to exclude activity by synthetic users from member metrics, billing and analytics.
Blameless postmortems. None of this works without a blameless postmortem culture. If an incident becomes a blame game, you lose the opportunity to learn from anomalies.
"There are absolutely scenarios where I'd say don't do it. If your failure mode is catastrophic and irreversible, production is not the place to experiment," Niderberg said.
What metrics should I track if I move forward?
To determine if the removal is a success, Niderberg suggests to track the following metrics:
Cost per peak event, year over year.
Mean time to detection on test-induced anomalies.
The number of scaling-related bottlenecks discovered by load tests.
On-call quality of life.
Time from bottleneck found to fix shipped.
Cloud cost as a percentage of revenue.
Internal tool reuse rate.
"[For internal tool reuse rate], if capabilities built for one team aren't being inherited by others, you're building one-offs and the maintenance debt will catch up with you," Niderberg said.
Lessons learned
When Karim began at Peloton, he felt that he over-indexed on load testing. When the engineering teams shut down the performance environment, they still decided to load test in production -- which he feels was the right call. However, he still faced the issue that a load test will only tell you how your system will perform under the assumption that everything goes exactly as expected.
Looking back, Karim realized the importance of starting small when testing in production. He also wishes he had known ahead of time to implement a kill switch and anomaly detection. Peloton now performs monthly tests to ensure its systems are prepared for interruptions.
"Honestly, the infrastructure work was never the story; it's the means," Karim said. "The story is what the architecture now lets the product and our member experience teams do that they couldn't before. The platform's job is to be invisible, and the more invisible it gets, the more our teams can focus on building what members actually came for."
Kathleen Casey is an award-winning writer and editor covering various IT infrastructure topics, including cloud computing, networking and emerging technologies, for Informa TechTarget
Dig Deeper on Cloud infrastructure design and management
Taq Karim
Alex Niderberg
