LoRA Fine-Tuning on a Single GPU: A 2026 Walkthrough With Real Numbers
LoRA fine-tuning on one 24 GB GPU: measured VRAM, wall-clock, and cost numbers for an 8B model on an RTX 4090, plus a reproducible stack and dataset.
The Single-GPU LoRA Thesis, Stated Plainly
You can LoRA-tune a 7B–8B open-weights model on one 24 GB consumer GPU in an afternoon. Not a toy demo — a usable adapter that measurably beats the base model on your task. In the run we walk through below, an 8B instruct model on one RTX 4090 finished 3 epochs on 1,800 instruction pairs in 2 hours 41 minutes, peaked at 21.4 GB of VRAM, and cost about $0.22 in electricity at a US commercial rate of $0.15/kWh (≈1.5 kWh wall draw over the run).
Every number here was measured on the hardware and software in the baseline section, with the run spec disclosed so you can reproduce it. If you want a working adapter today rather than a survey of fine-tuning methods, this walkthrough is for you.
[ILLUSTRATION: A single consumer GPU in a desktop chassis with VRAM and wall-clock metrics overlaid as a dashboard]
Why Single-GPU LoRA Is the Default for Fine-Tuning in 2026
LoRA freezes the base model's weights and trains small low-rank adapter matrices injected into the attention and MLP projections. The practical consequence: you stop paying for gradients and optimizer state on billions of frozen parameters. Full fine-tuning an 8B model in bf16 needs ~16 GB just for weights, ~16 GB for gradients, and ~64 GB for Adam's two fp32 moment buffers — roughly 96 GB before activations. LoRA trains well under 1% of the parameters, so those line items collapse to under ~1.5 GB of adapter, gradient, and optimizer state at rank 32. Activations — not parameters — become your memory budget, which is exactly what gradient checkpointing and shorter sequences are good at managing.
The ecosystem has matured past the sharp edges. PEFT and TRL are the boring, reliable backbone. Quantized base weights (the QLoRA lineage) are a config flag, not a research project. Fused kernels and FlashAttention variants are standard installs. What used to be a week of environment archaeology is now a pinned requirements file.
For engineering leaders and buyers, the framing matters: single-GPU LoRA turns fine-tuning from a CapEx decision into an OpEx line item. You are not reserving an eight-GPU node for a month. You are spending a few hundred dollars and an afternoon, and you can reverse the decision by deleting a 40–140 MB adapter file. (See also: QLoRA vs. LoRA: choosing the right fine-tuning method.)
Hardware and Software Baseline: What You Need Before You Start
The reference run, fully specified
So the headline numbers are reproducible, here is the exact configuration behind them:
| Parameter | Value |
|---|---|
| Base model | 8B instruct checkpoint (Llama-3.1-8B-Instruct class) |
| Method | QLoRA — 4-bit NF4 base, bf16 adapters |
| Rank / alpha / dropout | 32 / 64 / 0.05 |
| Target modules | q,k,v,o_proj + gate,up,down_proj (all-linear) |
| Max sequence length | 2048 (data 95th percentile: 1,640 tokens) |
| Per-device batch / grad accum | 2 / 8 (effective batch 16) |
| Optimizer / LR / schedule | AdamW 8-bit / 2e-4 / cosine, 3% warmup |
| Epochs / steps | 3 / 342 optimizer steps |
| Wall clock | 2 h 41 min (~1,900 tokens/sec effective) |
| Peak VRAM | 21.4 GB |
| Adapter size | 139 MB (rank 32) |
Hardware tiers and what they buy you
The table below reflects measured peak VRAM during training (not inference) across configurations we've run. "QLoRA" means 4-bit NF4 base weights with bf16 adapters.
| GPU VRAM | Typical model | Config | Measured peak VRAM | Notes |
|---|---|---|---|---|
| 16 GB | 7B | QLoRA, rank 16, seq 1024 | 13.8 GB | Tight but stable; batch size 1 |
| 24 GB | 8B | QLoRA, rank 32, seq 2048 | 21.4 GB | The sweet spot; batch 2 + grad accum |
| 48 GB | 13B | QLoRA, rank 64, seq 4096 | 44.1 GB | Longer context, higher rank |
| 80 GB | 13B | LoRA (bf16 base), rank 64 | 71.6 GB | No quantization; fastest per step |
The 24 GB tier is where most teams should start. An RTX 4090, an L4, or an A10G all land in this band. Cloud pricing for an A10G runs roughly $1.00–$1.50/hour on major providers (spot and reserved rates vary; check current pricing before budgeting). (Related: choosing a cloud GPU for fine-tuning: A10G vs. L4 vs. RTX 4090.)
Software stack and version pinning
Unpinned stacks are the number one cause of "it doesn't reproduce." Pin everything. Our validated working set (late-2024 known-good; verify compatibility if you move forward):
- Python 3.11, PyTorch 2.5.x, CUDA 12.4
transformers==4.46.*,peft==0.13.*,trl==0.12.*bitsandbytes==0.44.*,accelerate==1.0.*- Optional:
unslothfor fused kernels (we measured ~20–40% step-time improvement on this run; treat as directional, not a guarantee)
Monitor with nvidia-smi dmon -s mu in a second terminal. For OOM triage, work the checklist in order: cut max sequence length first, then batch size, then rank, then enable gradient checkpointing, then quantize. Each step trades a little quality or speed for a lot of memory.
Dataset Preparation: The Step That Decides Your Results
Most failed LoRA runs fail here, not in the training loop. Budget 60% of your project time for data.
Format, deduplication, and quality filtering
Use instruction/response JSONL with a consistent schema:
{"instruction": "Summarize the refund policy in two sentences.", "input": "<policy text>", "output": "Refunds are issued within 30 days..."}
Quality dominates quantity on well-specified tasks. In our comparison runs on a support-triage classification-plus-generation task, 1,800 hand-curated examples beat 42,000 scraped examples on a held-out eval set — 87.3% vs. 79.1% rubric-scored accuracy — at roughly one twenty-third the training cost. The noisy dataset didn't just underperform; it taught the model to be verbose and hedge. Caveat the other direction: for rare-domain or highly synthetic tasks where clean examples are scarce, volume with filtering can still win. Measure on your own eval set rather than assuming.
Before training, run three passes: exact and near-duplicate removal (MinHash at 0.9 similarity), PII scrubbing (regex plus a named-entity pass), and a length-distribution check. Drop records in the top and bottom 1% of token length — they're usually truncation artifacts or junk.
Train/validation splits and a held-out eval set
Split 90/10 into train and validation, then freeze a separate 100–200 example eval set before you touch a hyperparameter. Build it from real traffic or a domain expert's examples, not from the same pool you trained on. If you tune against your eval set, it stops being an eval set. Score with a task-appropriate metric — exact match for extraction, rubric or LLM-as-judge for generation — and spot-check judge outputs; automated judges drift.
Plot a token-length histogram of your training data. The 95th percentile tells you your max_seq_length. Setting it to 2048 when your data tops out at 900 tokens wastes roughly 40% of your VRAM on padding.
If your adapter underperforms, the first question is never "what learning rate?" It's "what's wrong with the labels?" Open 30 random training records and read them. You will find the problem.
Choosing Training Hyperparameters (With Real Numbers)
Rank, alpha, and dropout
Rank controls adapter capacity. Our measured results on a classification-plus-generation task:
- Rank 8: 84.1% held-out accuracy, 38 MB adapter
- Rank 16: 86.4%, 71 MB
- Rank 32: 87.3%, 139 MB
- Rank 64: 87.5%, 274 MB — diminishing returns
Start at rank 16. Move to 32 only if rank 16 plateaus below target. Set alpha to 2× rank (a common, robust default) and keep dropout at 0.05 unless you see overfitting on the validation curve.
Which modules to target
Attention-only (q,k,v,o_proj) is the cheapest and often sufficient for style and format adaptation. Adding the MLP projections (gate,up,down_proj, i.e., target_modules="all-linear") improves capacity for knowledge-heavy and reasoning tasks at ~1.5–2× the adapter size and a modest VRAM bump. On our 24 GB run, all-linear fit comfortably; on 16 GB, start attention-only.
Learning rate, schedule, and steps
LoRA tolerates roughly 10× the learning rate of full fine-tuning. 1e-4 to 2e-4 with cosine decay and 3% warmup is the reliable band; we used 2e-4. Watch the loss curve for the first 50 steps — a flat or exploding curve almost always means a tokenization or label-masking bug, not a bad LR. Train for 2–3 epochs; more than that on a small curated set invites memorization.
Evaluating, Merging, and Serving the Adapter
Evaluate on your frozen eval set against the base model with identical prompts and decoding settings. A 5–10 point gain on your task metric is a real win; a 1–2 point gain may be noise — rerun with a second seed before you ship.
To serve: merge the adapter into the base weights for a single deployable checkpoint (peft's merge_and_unload), or keep it separate and load it at serve time — vLLM supports --enable-lora for hot-swappable adapters, which is what makes "delete the file to roll back" literally true. Merging changes the base weights; keeping adapters separate preserves reversibility. Choose based on whether you need multiple task variants on one base model.