In its ‘State of Cloud Costs 2024’ report, Datadog, a global platform for cloud application monitoring and security, has disclosed a noteworthy 40% rise in GPU instance spending by enterprises during the previous year. Large language models (LLMs) and artificial intelligence (AI) are two areas of growing experimentation that are thought to be responsible for this rise.
GPUs, known for their parallel processing capabilities, are essential for training LLMs and executing other AI workloads, offering more than 200% speed advantage over traditional CPUs.
Yrieix Garnier, VP of Product at Datadog, noted that the most commonly used GPU instances are currently the least expensive, indicating that many companies are still in the early stages of AI adoption, focusing on adaptive AI, machine learning inference, and small-scale training. As these organizations advance their AI initiatives and transition to production, it is expected that their cloud compute budgets will increasingly be allocated to more costly GPU instances.
The report also highlights significant inefficiencies in cloud spending, particularly concerning the use of containers. It was found that 83% of container costs were linked to idle resources, with 54% attributed to cluster idle due to overprovisioning of infrastructure and 29% to workload idle from overestimated resource requests. This inefficiency has been exacerbated by an increase in the allocation of EC2 compute resources to containers, rising to 35% from 30% the previous year, according to Datadog’s study.
Additional insights from the report include the prevalent use of outdated technologies. Despite newer, more cost-effective AWS infrastructure offerings, 83% of organizations still allocate an average of 17% of their EC2 budgets to previous-generation technologies. Moreover, fewer organizations are taking advantage of commitment-based discounts offered by cloud service providers, with only 67% participating in such programs, down from 72% last year.
Adoption of Green Technology
The report also indicates a rise in the adoption of green technology. Organizations using Arm-based instances now spend 18% of their EC2 compute budget on these instances, double the amount from the previous year. Arm-based instances are known for using up to 60% less energy than comparable EC2s, often delivering better performance at a lower cost, according to Datadog’s report
To compile this report, Datadog analyzed AWS cloud cost data from hundreds of organizations, examining their usage of both emerging and legacy technologies, patterns of cloud resource utilization, and participation in AWS discount programs. The findings would underscore the complexities and evolving nature of cloud cost management in the face of rapid technological advancements and increasing reliance on AI and machine learning.
