LLM Inference on Limited GPU Memory: What We Learned From Layer-Wise Model Streaming

Deploying large language models is not only a question of choosing the right model. Infrastructure quickly becomes part of the decision.

During LLM inference, model weights, activations, and the KV cache all compete for GPU memory. As models, context windows, and application requirements grow, teams can reach a point where the model they want to use no longer fits comfortably within the VRAM available to them.

The usual options include moving to larger GPUs, reducing model precision through LLM quantization, using a smaller model, or distributing inference across additional hardware.

As part of our internal AI engineering work at IT Path Solutions, we explored another possibility:

Can model weights be actively managed between system RAM and GPU memory during inference, rather than requiring every model layer to remain in VRAM at the same time?

We built and progressively optimized a layer-wise streaming prototype to find out.

Our experiment used Qwen2.5-7B-Instruct with FP16 weights on an NVIDIA RTX 5060 Ti with 16 GB of VRAM and 32 GB of system RAM. The final version of the prototype generated 10.52 tokens per second, compared with 0.44 tokens per second in our initial disk-streaming implementation.

The interesting part, however, was not simply the final number. It was understanding where LLM inference became slow, which optimizations actually helped, and what those results tell us about deploying models under constrained GPU memory.

Why GPU Memory Becomes an LLM Inference Constraint

An LLM does not only need space for its model weights.

During inference, GPU memory may also need to accommodate:

• Model activations
• The KV cache used during token generation
• Temporary computation buffers
• Other runtime memory requirements

Hugging Face’s cache documentation notes that the KV cache can become a significant memory bottleneck during long-context generation, particularly on memory-constrained hardware. Similarly, PyTorch’s guidance on pinned memory and non-blocking transfers shows why CPU-to-GPU data movement needs to be treated as an architectural concern rather than an implementation detail.

This means that even when model weights appear close to the GPU’s total VRAM capacity, the model may still not run comfortably once real inference begins.

Increasing the available hardware is one solution. Quantization can also substantially reduce model size.

For this experiment, however, we deliberately focused on a different question.

Instead of reducing the model’s weight precision, we investigated whether LLM memory management at the software level could allow only part of the model to remain permanently inside GPU memory while the remaining layers were transferred when needed.

The goal was not to identify a universally better alternative to quantization or larger GPUs. It was to understand the practical behavior and limitations of this type of memory-tiered LLM inference architecture.

What We Built: A Memory-Tiered LLM Inference Architecture

The initial concept was relatively straightforward.

Instead of keeping every model layer inside GPU VRAM, model weights would first be loaded into system RAM. During inference, the layers required for computation could then be transferred from host memory to the GPU over PCIe.

The simplified data path looked like this:

SSD → Host RAM → PCIe → GPU VRAM → GPU Compute → Generated Token

The final implementation combined several techniques:

• Pinned host memory
• Reusable GPU buffers
• Asynchronous CPU-to-GPU transfers
• Double buffering
• Permanent VRAM caching for selected model layers

Each one addressed a different bottleneck in the GPU inference process. This is also where AI systems begin to overlap with broader cloud and DevOps architecture decisions, because model behavior, infrastructure capacity, deployment design, and runtime performance are tightly connected in production.

Pinned Host Memory

One of the first lessons was that repeatedly retrieving model weights from disk during inference was simply too expensive.

Instead, streaming layers were loaded into page-locked, or pinned, CPU memory when the system started. Those weights could then be transferred directly to the GPU without requiring runtime SSD reads.

This removed disk I/O from the token-generation path.

Reusable GPU Buffers

Allocating new GPU memory every time a layer needs to execute also creates unnecessary overhead.

Instead, the prototype used a small number of reusable buffers. A streamed model layer could be copied into a buffer, executed, and the same GPU memory reused for another layer later in the forward pass.

Double Buffering

CPU-to-GPU transfer and GPU computation do not necessarily need to happen one after another.

With double buffering, one buffer can be used by the GPU for computation while another buffer is being prepared for the next layer.

That gives the system an opportunity to overlap data transfer with computation rather than leaving the GPU idle while it waits for the next set of weights.

Keeping as Many Layers in VRAM as Possible

The biggest performance improvement came from avoiding transfers altogether.

Available GPU memory was therefore used to permanently retain as many model layers as possible. Only the remaining layers had to cross PCIe during every forward pass.

For example, our test configurations included:

• C20: 20 layers resident in VRAM, 8 streamed
• C24: 24 layers resident, 4 streamed
• C26: 26 layers resident, 2 streamed

As the number of streamed layers decreased, inference performance increased considerably.

LLM Inference Optimization: From 0.44 to 10.52 Tokens per Second

The prototype was not built in one step.

We progressively changed the architecture and measured the effect of each iteration.

Configuration	Main Change	Throughput
Naive disk streaming	Load layers from disk during inference	0.44 tok/s
Flat serialization	More contiguous layer representation	0.91 tok/s
Double buffering	Overlap transfer and compute	0.98 tok/s
Adaptive C8	Keep 8 layers in VRAM	1.23 tok/s
Engine C20	Pinned RAM + 20 resident layers	3.53 tok/s
Engine C24	24 resident layers	6.40 tok/s
Engine C26	26 resident layers	10.52 tok/s

The final configuration represented an approximately 24× improvement over our original 0.44 tok/s disk-streaming implementation.

That comparison is important.

It does not mean the architecture makes LLM inference 24 times faster than conventional inference engines or GPU deployments.

It means that, within this specific prototype, successive LLM optimization steps transformed an impractically slow initial implementation into a much more usable one.

That distinction matters when evaluating any AI infrastructure benchmark. It is the same kind of trade-off analysis that matters in custom software development, where architecture decisions need to be evaluated against the actual workload, product requirements, and operating environment rather than in isolation.

The Most Important Lesson: Data Movement Is Expensive

Several optimizations helped, but one pattern became particularly clear.

The fewer model layers we had to transfer across PCIe, the faster inference became.

Consider the progression:

• C20 streamed 8 layers and achieved 3.53 tok/s
• C24 streamed 4 layers and achieved 6.40 tok/s
• C26 streamed only 2 layers and achieved 10.52 tok/s

At the same time, peak system RAM requirements fell as more model weights remained resident inside VRAM.

This points to a broader inference-engineering principle.

When GPU memory is constrained, memory tiering can help make an otherwise difficult deployment possible. But transferring large amounts of model data between CPU and GPU introduces its own cost.

In practice, the best layer to stream is still the layer you can avoid streaming at all.

Layer Streaming vs LLM Quantization

Memory tiering is not the only way to reduce GPU memory pressure.

LLM quantization reduces the precision used to represent model weights, potentially allowing more of the model to fit into the available VRAM while also reducing memory bandwidth requirements.

Our experiment deliberately kept the model’s original FP16 weight precision and explored memory tiering instead.

That makes these techniques different rather than directly competing approaches.

Depending on the application, an LLM deployment might use:

• A larger GPU
• A smaller model
• Quantization
• CPU or memory offloading
• Layer streaming
• Multiple GPUs
• Or a combination of these approaches

Quantized INT8, FP8, or INT4 streaming could also reduce the amount of data transferred across PCIe, but this was outside the scope of the experiment and remains an area for further investigation.

Some of the Difficult Parts Were Not Performance Problems

Performance optimization was only part of the engineering challenge.

One issue emerged from asynchronous execution.

The CPU could begin preparing the next layer before the GPU had completely finished using the existing buffer. When that happened, parameters could effectively be replaced while the GPU was still reading them, resulting in corrupted model output.

The system therefore required explicit synchronization to guarantee that a GPU buffer was safe to reuse before new layer weights were written into it.

Another problem involved dynamically generated rotary position embedding parameters.

Flattening model layers for serialization introduced parameter-alignment problems, so a parameter specification manifest had to be introduced to preserve the correct ordering and reconstruction of the weights.

These kinds of problems are worth highlighting because LLM inference optimization is rarely just about finding the right theoretical architecture.

Memory ownership, synchronization, serialization, execution order, and hardware behavior can all affect whether an inference system produces correct results at all.

When Could This Kind of Architecture Be Useful?

Layer-wise streaming is not something every LLM deployment needs.

Where sufficient GPU memory is available, keeping the model resident in VRAM will normally be simpler and faster.

Memory tiering becomes more interesting when infrastructure is constrained and there is a reason to avoid immediately moving to larger hardware.

That could include environments where teams are:

• Experimenting with models before committing to larger GPU infrastructure
• Building private or locally hosted AI systems
• Exploring local LLM inference on available hardware
• Working with an existing GPU fleet
• Investigating higher-precision inference under limited VRAM
• Developing AI applications where moderate generation speed may be acceptable in exchange for lower hardware requirements

Choosing among these options is part of a broader model and system design process. If you’re evaluating model development itself alongside inference architecture, our guide on how to create an AI model covers the earlier stages of defining the problem, preparing data, and selecting an approach.

It can also be useful as an engineering technique for understanding how much of an LLM inference bottleneck comes from computation and how much comes from memory movement.

Where This LLM Optimization Approach Reaches Its Limits

Our experiment also exposed several constraints.

The first is PCIe bandwidth.

Any layer that is streamed has to travel between system memory and GPU memory during inference. At some point, that transfer becomes a fundamental performance limit.

GPU VRAM also cannot be allocated entirely to model weights. Activations and the KV cache require space as well, which means the optimal number of permanently resident layers can change depending on context length and workload.

There are also important limitations to the experiment itself.

Our results came from:

• One model: Qwen2.5-7B-Instruct
• One GPU configuration: RTX 5060 Ti with 16 GB VRAM
• One set of implementation choices

Longer-context inference may require leaving more VRAM available for the KV cache and therefore reducing the number of permanently cached layers.

Further work could also examine quantized INT8, FP8, or INT4 streaming, which may reduce the amount of data travelling across PCIe.

Adaptive cache sizing could allow the inference engine to change the number of resident layers dynamically based on current context length and available GPU memory.

For those reasons, the result should be treated as an engineering experiment rather than a universal LLM inference benchmark.

What Should Teams Consider Before Choosing an LLM Inference Architecture?

One of the broader lessons from this work is that choosing an LLM is only one part of designing an AI system.

A production LLM deployment also needs to consider several factors.

Model Size and Precision

Higher-precision weights consume more memory, while quantization introduces a different set of performance and quality trade-offs.

Available GPU Memory

VRAM determines how much of the model, KV cache, and runtime workload can remain on the GPU.

Context Length

Longer conversations and prompts increase KV-cache requirements.

Latency Requirements

An internal document-analysis system may tolerate very different inference speeds from an interactive consumer application.

Concurrency

A prototype serving one user and a production application handling many simultaneous requests have very different infrastructure requirements.

LLM Serving Requirements

Production LLM serving may introduce requirements that are not visible in a single-user experiment, including concurrent requests, batching, throughput targets, model loading, monitoring, and availability.

Infrastructure Cost

The technically fastest configuration is not always the most economical deployment.

The right solution may involve larger GPUs, smaller models, quantization, batching, model-serving frameworks, memory offloading, or a combination of techniques.

The architecture should ultimately be driven by the application’s requirements rather than by a single benchmark number. That broader production lens is also important when evaluating the real-world use cases and business fit for AI development, because infrastructure choices only make sense in the context of the product they are meant to support.

What We Learned About LLM Inference

Our prototype demonstrated that explicit memory management can substantially improve LLM inference when GPU memory is constrained.

The original disk-streaming implementation produced 0.44 tokens per second.

After moving weights into pinned host memory, introducing reusable buffers and asynchronous transfers, and retaining almost all model layers in VRAM, the final C26 configuration reached 10.52 tokens per second.

More importantly, the experiment reinforced a principle that applies well beyond this particular implementation:

AI performance depends as much on the architecture around the model as it does on the model itself.

For teams building AI capabilities into real software products, decisions about LLM inference, memory management, latency, model precision, context size, serving requirements, and deployment environment eventually become product decisions as well.

At IT Path Solutions, experiments like this help our engineering teams explore the trade-offs behind practical AI deployments, not just how to integrate a model, but how the surrounding architecture affects performance and feasibility.

Sometimes the answer is more hardware.

Sometimes it is a different model.

And sometimes, as this experiment showed, better use of the hardware already available can make a meaningful difference.