The environmental impact of LLMs vs. SLMs

Comparing LLMs vs. SLMs

There is no universally agreed line separating LLMs from SLMs. The distinction lives on a continuum, and the boundary has shifted as technology has advanced.

That said, practical rules have emerged from research and deployment experience that can anchor decision-making, even if the definitions remain somewhat fluid.

SLMs

SLMs are generally understood to be models with fewer than 10 billion parameters. In practice, the most commonly deployed SLMs cluster around 1, 3 and 7 billion parameters.

Examples include the following:

• Microsoft's Phi-3 Mini, at 3.8 billion parameters.
• Meta's Llama 3.2, available at 1 billion and 3 billion parameters.
• Mistral 7B, at 7.3 billion parameters.
• TinyLlama, at 1.1 billion parameters.

Many of these models can run on consumer-grade hardware, such as a single GPU or a high-end laptop. Fine-tuning them for a specific domain can cost in the low hundreds of dollars and take days rather than months.

LLMs

By contrast, LLMs sit at the other end of the scale: typically 70 billion parameters and above, with frontier models reaching into the hundreds of billions or beyond.

GPT-4 has been estimated at around 1.76 trillion parameters. Models like Claude, Gemini Ultra and Llama 3.1 405B occupy this tier. Running them requires specialized GPU clusters, and training or fine-tuning from scratch is out of reach for most organizations.

A middle tier, loosely 10 to 70 billion parameters, spans models like Llama 3.1 70B that offer more capabilities than a compact SLM without the full infrastructure burden of a frontier LLM.

These parameters also reveal more about compute, memory and energy at every stage of the model lifecycle. Well-designed SLMs trained on high-quality data, particularly those distilled from larger models, have closed much of the quality gap for bounded, structured tasks.

Compute-optimal training research from Cornell University shows that smaller models trained on more tokens can outperform larger, undertrained models at the same compute budget. This weakens the assumption that bigger is always better and invites a more nuanced analysis of which model suits a given workload.

The environmental cost breakdown

The environmental impacts of LLMs vs SLMs differ by model type and by each phase of the process.

Training phase

Training is a one-time cost, but it is a large one. Published training footprints vary significantly depending on architecture, grid carbon intensity and infrastructure overhead.

The Stanford AI Index puts Grok 4's estimated training emissions at 72,816 tons of CO2 equivalent, the equivalent of 17,000 cars driven for a year. That figure illustrates the scale of AI's carbon footprint at the frontier, even before a single inference is served.

Most enterprises do not pretrain frontier models from scratch, but they are still implicated in training emissions through procurement choices, fine-tuning decisions and hosting arrangements. Fine-tuning energy for typical enterprise tasks is small compared to pretraining, but it is not negligible, particularly when hardware idle overhead is included.

Embodied carbon -- the emissions baked into manufacturing GPUs and other AI chips -- is also part of the training-phase picture, especially when refresh cycles are short, or server clusters are duplicated for data sovereignty.

SLM training and fine-tuning carry a fraction of these costs. With the right optimization techniques, fine-tuning a 7B model can run on a single consumer-grade GPU and cost less than $1,000 in compute.

Still, training is a one-time event. At the organizational scale, amortized training energy per request is often lower than inference energy per request, making the operational phase the more consequential sustainability lever.

Usage phase

A single LLM query can consume between 0.3 and 1 watt-hour of energy. That range may sound modest, but multiply it by millions of daily requests across an organization's AI systems, and it becomes a significant budget line item for both cost and emissions.

SLMs, running on a fraction of the hardware, consume correspondingly less per request. But raw per-request energy is not the right metric to compare these models. Utilization matters enormously.

A frontier LLM serving thousands of concurrent requests in a well-optimized, high-utilization environment can be more efficient per useful output than an SLM sitting idle in a lightly used, always-on deployment. Idle fleets consume energy without producing value, eroding any efficiency advantage of smaller models entirely.

Token economics compound the picture. Long prompts and outputs significantly increase inference energy, because self-attention -- the technique used in transformers -- has a non-linear computational cost with respect to sequence length. Agentic workflows, where models make repeated tool calls across extended contexts, can turn a seemingly efficient system into an environmentally expensive one once total token volume is counted.

The right unit of comparison is not energy per raw request but energy per successful work item -- or the output that achieved the business objective, accounting for retries, escalations and human corrections.

Water consumption

Water consumption is another dimension of AI's environmental impact. Annual inference water use -- the water used to cool data servers or run them from hydroelectric sources -- for GPT-4o alone may exceed the drinking water needs of 12 million people, according to Stanford's research. That figure captures both on-site operational water and the indirect water embedded in electricity generation.

Water use efficiency (WUE) is the key metric for operational water consumption in data centers. It varies significantly by cooling design and location.

Facilities that reduce water use through air-side economization or other methods often do so at the cost of higher energy consumption, creating an explicit energy-water tradeoff that governance frameworks must consider. SLMs running on edge hardware or in smaller regional facilities can sidestep some of this burden, but only when utilization is high and cooling infrastructure is modern.

Overall, the water footprint, like the energy footprint, is determined by an interaction between workload, geography, cooling design, utilization and governance, not by model size alone.

Where LLMs vs. SLMs excel

LLMs are best in situations where the cost of a wrong answer is high, as they are often more accurate than SLMs. These include the following scenarios:

• Regulated settings where errors carry legal or financial consequences.
• Complex multi-step tasks where failure cascades into expensive human remediation.
• Advanced coding and reasoning work that requires broad contextual understanding.
• Tier-2 and tier-3 support levels, with costly rework and retry loops.

LLMs often have a lower environmental cost per successful task in high-stakes contexts, even when the per-request energy is higher.

SLMs dominate sustainability where the task is well-scoped, measurable and retrieval-supported. They support high-volume bounded work, including the following:

• Customer support triage.
• Document classification.
• Sentiment analysis.
• Entity extraction.
• Short-form content generation.
• Tier-1 customer support.
• Enterprise search.
• Document processing.
• Coding autocomplete.

Edge deployment and domain-specific applications are also a strong fit. When data cannot leave a facility or latency requirements demand local inference, SLMs are often the only practical option regardless of sustainability considerations.

The potential for hybrid deployments

The most sustainable enterprise architecture is not a choice between LLMs and SLMs. It is a hybrid, tiered system that routes work to the smallest effective model for each task, escalating to larger models only due to complexity or risk.

Dynamic routing is the core mechanism: Start every request at the SLM tier, classify its difficulty and route to the LLM tier only when the SLM's confidence or success rate falls below the threshold required for the business outcome.

A hybrid strategy also means using specialized SLMs for specific functions rather than a single, general-purpose LLM. Some examples include the following:

A coding assistant could split between a compact autocomplete model and a larger reasoning model for architecture-level questions.

A customer service platform could handle FAQ resolution at the SLM tier and escalate complaints and complex claims to an LLM.

An IT ops tool could use a small model for alert triage and a larger one for root cause analysis.

The sustainability rationale is clear, as the energy savings from routing even a moderate share of traffic to smaller models are substantial.

Edge computing extends this logic further. Where data sovereignty, latency or connectivity requirements make cloud-based inference impractical, SLMs running on local hardware reduce the burden on data centers. Serving-stack optimizations, such as continuous batching, quantization, request shaping and phase-aware power management, can also reduce per-request energy across an enterprise deployment.

The business case for sustainable AI

Sustainable AI is sometimes framed as a tradeoff between environmental responsibility and operational performance. But the same design choices that reduce AI's carbon footprint, such as better utilization, smarter routing, controlled output length and optimized serving stacks, also reduce infrastructure waste, lower operational costs and improve system reliability. Green AI strategies, in practice, are often good engineering discipline.

However, sustainable AI has rebound effects and potential risks. Efficiency gains can trigger higher usage. When inference becomes cheaper per request, more tasks get automated, agent call volumes increase, context windows expand and more always-on deployments get stood up.

Demand management matters as much as technical optimization. Without it, volume growth absorbs model-level efficiency improvements, leaving total emissions flat or rising even as per-request energy falls.

Procurement and measurement discipline are the mitigating levers. Organizations can require vendors to disclose energy per 1,000 requests, PUE, WUE and carbon accounting methodology. They should also track kWh per 1,000 inferences, kilograms of CO2e per 1,000 inferences, and GPU utilization as standard sustainability KPIs alongside cost and latency.

The regulatory picture is also moving in this direction. The White House's 2027 budget proposes $202 million toward EPA AI and data center oversight. The European AI Act, SEC climate disclosure rules and sector-specific ESG requirements create governance obligations that extend to AI infrastructure decisions. Organizations that build measurement and reporting infrastructure now will be better positioned when mandatory disclosure arrives.

The executive implications are clear. Inference dominates lifecycle energy at the organizational scale. Serving architecture and utilization often matter more than the choice between adjacent model sizes. And the most sustainable posture is hybrid-by-default, with success per unit of energy as the metric that governs routing, procurement and investment.

LLMs and SLMs are not competitors in a zero-sum race. They are complementary tools in a tiered architecture, and enterprises that deploy them this way will find that energy-efficient AI deployment and high-quality AI deployments are often two sides of the same coin.

Kashyap Kompella, founder of RPA2AI Research, is an AI industry analyst and advisor to leading companies across the U.S., Europe and the Asia-Pacific region. Kashyap is the co-author of three books, Practical Artificial Intelligence, Artificial Intelligence for Lawyers and AI Governance and Regulation.