How YellowDog Optimises GPU Selection for Cost Efficient, Continuous LLM Retraining
Erasmus.AI is one of the world’s most experienced AI research organisations, with 18 years of planetary-scale data powering 40 NLP engines and over 200 LLMs across five GPU generations. Its flagship product, ClimateGPT was launched at COP28 and the World Economic Forum, is the first foundational AI model built to address climate change.
ClimateGPT requires continuous retraining to stay current as the climate landscape evolves. The core challenge is ensuring the right GPU resources are available at the right time and cost, while minimising the engineering overhead of adapting training pipelines to deploy on new hardware architectures. Erasmus.AI had been training ClimateGPT on Nvidia H100s, but increasing scarcity of this hardware was preventing training cycles from running on schedule, directly impacting the team’s ability to keep the model current.
YellowDog solved this by identifying AWS Trainium as the optimal and most cost-efficient hardware for ClimateGPT training and delivering a 36% reduction in training cost leading to 1.5× more training runs for the same budget. YellowDog’s ability to orchestrate a wide range of GPU architectures gives engineering teams the flexibility to evaluate and adopt new hardware without operational risk. Teams can focus on pipeline engineering, knowing that once validated, the hardware is immediately available for production workloads at scale, and that YellowDog will manage the orchestration regardless of which architecture is chosen.
Erasmus.AI used this model to better optimise their training pipeline to run across both H100s and Trainium GPUs. Giving GenAI engineering teams better access and visibility to available GPU and CPU instances removes the uncertainty surrounding hardware scarcity, allowing teams to make confident infrastructure decisions, commit to pipeline engineering work, and focus on what matters most: advancing the model.
GPU Scarcity Is Holding Back GenAI Innovation
Reliable access to GPU hardware has created a severe supply/demand imbalance for high-performance compute.
With ClimateGPT already outperforming ChatGPT 3.5 on climate domain benchmarks, the pressure to retrain quickly and cost-effectively was critical. Erasmus.AI’s data pipeline processes over a quarter-billion curated URLs per day yet accessing the GPU capacity on demand remained unpredictable.
Erasmus.AI had been training ClimateGPT on Nvidia H100s, however as demand for H100 capacity intensified across the industry, both the cost and availability of that hardware became increasingly unreliable. Continued uncertainty around hardware availability made the economics of continuous retraining on premium GPU hardware unsustainable at the pace ClimateGPT’s development required.
With H100 capacity increasingly scarce and expensive, adopting a new hardware architecture was essential. The challenge was not simply cost. It was knowing which hardware was available, without committing weeks of engineering effort to identify and evaluate potential candidates. Extending ClimateGPT’s training pipeline to run on a new architecture needed to happen with certainty that the target hardware was available at the cost and performance required to sustain continuous retraining at scale. Investing in pipeline re-engineering for an unproven alternative was a significant decision. It demanded high confidence in both hardware availability and production-grade performance before pipeline engineering work was committed.
YellowDog Intelligence: Orchestrating The Full Training Lifecycle
YellowDog identified AWS Trainium as a viable and significantly more cost-efficient alternative to the H100s. Before Erasmus.AI could commit to re-engineering its pipelines, commercial and operational assurance was needed to confirm that Trainium was the right hardware choice. To support this decision, YellowDog implemented a structured five-step process to validate the new hardware.
YellowDog Insights was used to provide the Erasmus.AI team with a regional and operational analysis of Trainium availability on AWS. YellowDog then produced a benchmarking analysis indicating a 36% reduction in total training cost. This led to a technical validated exercise which demonstrated that using YellowDog as the primary orchestration layer introduced no additional infrastructure overhead and no compromise on model quality. With capacity availability confirmed across preferred AWS regions and Availability Zones, YellowDog presented a clear return on investment model, giving Erasmus.AI the confidence to fully commit to its Trainium pipeline engineering plan.
With the optimal hardware identified and validated for both cost-efficiency and production-scale orchestration on AWS, YellowDog managed the entire training workflow from provisioning to completion. When hardware availability or pricing shifted, YellowDog automatically moved workloads to where capacity was available. The training code remained unchanged.
The YellowDog Platform provisioned compute across multiple AWS Availability Zones simultaneously, drawing from the most cost-efficient capacity available at any given time. It scaled seamlessly, intelligently orchestrating instances to ensure training pipelines completed reliably and without engineering intervention.
For ClimateGPT training cycles, this translated directly into operational resilience. Deployed within Erasmus.AI’s secure AWS Virtual Private Cloud, YellowDog orchestrated training workloads across Availability Zones, scaling infrastructure precisely when needed. Training pipelines completed reliably on schedule, giving the team confidence to commit to continuous retraining as a core part of their model development process.
YellowDog Insights: Eliminating The Guesswork
Choosing compute for LLM training isn’t a one-time procurement decision, it’s a moving engineering target shaped by new instance types, shifting capacity, and changing prices. Many GenAI teams make these choices with incomplete data, defaulting to using the hardware they are most familiar with.
YellowDog Insights removes that uncertainty. It provides a comprehensive index of available instances across all major cloud providers, regions and chipsets, delivering real-time benchmarking across 10,000+ instance types using QuantLib and SysBench, ranked by price, performance, and price/performance ratio.
This proved decisive for Erasmus.AI. Instead of time-consuming manual GPU benchmarking, Insights helped identify AWS Trainium as the most cost-effective option for ClimateGPT training. The reduced costs effectively funded the next run, enabling faster iteration without increasing budget.
Insights makes hardware selection an automated, data-driven decision process, and is integrated directly into the YellowDog Scheduler to provision the optimal instance every time.
What GenAI Performance Looks Like in Production
YellowDog Gives GenAI Teams The Compute Foundation To Innovate
GPU scarcity, opaque pricing, and infrastructure lock-in are the invisible ceiling on GenAI innovation. YellowDog’s intelligent orchestration, real-time hardware intelligence, and portable workload execution give any GenAI team the compute foundation to train faster, at lower cost, and without being constrained by any single hardware architecture or cloud provider.
When GenAI training workloads need to scale, your next model iteration shouldn’t be constrained by infrastructure or limited by scarce hardware. YellowDog maps your training needs to a fully orchestrated compute solution that delivers the best price/performance outcome, so innovation never suffers. Contact us here.
