A GTX 1650 with 4 GB of VRAM, no tensor cores, and a driver five years out of date. It now runs speech models faster than my CPU by a factor of four. Everything that stood in the way, in the order it stood there.
A GTX 1650: 4 GB of VRAM, Turing generation, compute capability 7.5, and no tensor cores at all. It is the cut-down Turing part, without the hardware that makes newer cards fast at low precision. Every "run models locally" guide assumes something better.
I kept using it anyway, because the interesting shift of the last two years is not that models got bigger. It is that the useful small ones got small enough to matter. A 0.6-billion-parameter speech model occupies 2.2 GB and does real work. That is a size this card can hold.
Everything below is what stood between the card and that work. Almost none of it was performance. It was plumbing.
Mine was version 462.30, shipped February 2021. Five years stale, and it caps you at the CUDA 11.2 runtime, which no current PyTorch wheel targets. Every downstream thing I tried would have failed against it for reasons that had nothing to do with the real problem.
Check this before anything else:
nvidia-smi # Driver Version: 462.30 CUDA Version: 11.2 <- five years old # Driver Version: 616.56 CUDA Version: 13.4 <- after updating
One practical note if your machine still has GeForce Experience on it: that client was retired, and its driver-download backend no longer serves the old builds. Express Install will sit there doing nothing forever and tell you nothing. Download the driver directly instead, take the Studio branch, and choose the custom install so you can decline the extra background services on a memory-constrained machine.
You do not need the CUDA Toolkit. Modern PyTorch wheels bundle their own runtime, and on a machine with 8.5 GB of RAM the toolkit is three gigabytes you will never use.
Install almost any library that depends on torch and pip will resolve a CPU-only build. Silently. It has now cost me an afternoon twice, so it is the first thing I check after any install that touches torch.
python -c "import torch; print(torch.__version__, torch.version.cuda)" # 2.13.0+cpu None <- broken, and nothing told you pip install --force-reinstall torch==2.13.0 torchaudio \ --index-url https://download.pytorch.org/whl/cu126
Then confirm your card's architecture is actually compiled into the build. This is the check specific to old hardware, and it takes two seconds:
python -c "import torch; print(torch.cuda.get_arch_list())" # ['sm_50','sm_60','sm_61','sm_70','sm_75','sm_80','sm_86','sm_90'] # ^^^^^^ Turing
Turing is sm_75. CUDA 13 dropped several older architectures, so on a pre-Ampere card I pin the cu126 index rather than taking whatever is newest. Newer is not better when the newer thing has removed support for your hardware.
I spent longer on this than on everything else combined, and the fix is two lines.
The pip nvidia-* wheels put their DLLs in site-packages/nvidia/*/bin. The documented way to make those visible on Windows is os.add_dll_directory(). That is not sufficient. onnxruntime loads its provider DLL through a route that searches the real PATH, so the provider fails to load and falls back to CPU. No exception. No warning at the default log level.
Your job just runs four times slower than it should, and you conclude that this is what the hardware does. I concluded exactly that, and I was wrong for about a day.
root = Path(importlib.util.find_spec("nvidia").origin).parent for sub in sorted(root.rglob("bin")): if sub.is_dir(): os.environ["PATH"] = str(sub) + os.pathsep + os.environ["PATH"] os.add_dll_directory(str(sub)) # helps, insufficient alone
Match the CUDA major version to your onnxruntime release while you are here. Version 1.29 wants CUDA 13 and looks for cublasLt64_13.dll; 1.22 wants CUDA 12. Install the wrong nvidia-*-cu1x wheels and you get a missing-DLL error naming a file you have never heard of. On Windows the CUDA 13 wheels had no builds available when I tried, so I pinned onnxruntime-gpu==1.22.0 and used the CUDA 12 set.
Let an auto setting degrade quietly. Make anything explicit fail loudly. If --device cuda silently falls back to CPU, you find out an hour into a long run, and you will blame the wrong thing.
active = session.get_providers()[0] if device == "cuda" and "CUDA" not in active: sys.exit("Asked for CUDA but got %s" % active)
Before CUDA worked I tried DirectML, because it needs no CUDA toolkit and runs on any DirectX 12 GPU. On a Windows laptop it is the obviously attractive option.
onnxruntime-directml advertises the provider, claims the node, then dies inside the model's F0 encoder on ConvTranspose with error 80070057. Because it has already claimed the node there is no CPU fallback. Nothing renders at all. It also installs over onnxruntime, since both provide the same module, so testing the alternative means uninstalling one first.
I wrote the whole thing off as "GPU acceleration does not work on this machine" and moved on. That conclusion was wrong and it cost me a day. DirectML does not work with this model. CUDA does. Those are different claims and I collapsed them into one.
The model I was running ships in bfloat16. Turing has no native bf16, so PyTorch emulates it, and fp16 is the genuinely native path. Switching to fp16 for the speed is the obvious optimisation. It is also wrong, and the failure mode is nasty.
File "transformers/generation/utils.py", line 2787, in _sample torch.AcceleratorError: CUDA error: device-side assert triggered
bf16 has a much wider exponent range than fp16. In fp16 the logits overflow inside the sampling loop, sampling returns an out-of-range index, the embedding lookup asserts on device, and the process dies. Device-side asserts report asynchronously, so the traceback location is not trustworthy either. You get a crash pointing at a line that is fine.
Emulated bf16 is slower and correct. Take correct. The related setting: attn_implementation="sdpa", because FlashAttention-2 requires Ampere or newer and will not build against this card at all.
Autoregressive decode is memory-bandwidth bound, so batching normally pays for itself. On a small card it pays, then abruptly stops paying, and the clock alone will not tell you where.
| Batch | Speed | Peak VRAM | Verdict |
|---|---|---|---|
| 1 | 4.4x real time | 3.38 GB | fits |
| 2 | 2.7x real time | 3.77 GB | 1.6x faster, still fits |
| 4 | 2.8x real time | 5.07 GB | over the card, spills to system RAM |
Batch 4 exceeds 4 GB, spills over PCIe, and hands back everything it gained. The clock said batch 4 was fine. The memory counter said why it was not. On a card with headroom you would never notice; on this one it is the whole game.
The same principle explains a much larger result. I first tried a 0.5B model whose weights took 3.21 GB, leaving 0.79 GB of headroom. It peaked at 4.30 GB against a 4.00 GB card and paged constantly, and its cost grew with utterance length: 11.6x real time at four seconds, 24.2x at eighteen. The replacement occupies 2.2 GB and holds a flat 4.5x regardless of length, because it never leaves the card.
Half a gigabyte of weights was the difference between a twelve-hour job and a three-hour one. On this class of hardware, model size is not a quality dial. It is a cliff.
My first long run died after 40 minutes with no Python traceback. The Windows event log had the answer:
Id 153 nvlddmkm Error
That is a GPU engine reset. A CUDA kernel ran past the display driver's two-second watchdog, Windows reset the graphics engine, and the CUDA context died taking the process with it. The underlying cause was the same paging described above: when kernels wait on PCIe transfers they stretch past the timeout.
The usual advice is to raise the watchdog through TdrDelay in the registry. I would try the actual fix first. Reducing the work per kernel, by shortening chunks or lowering batch size, addresses the cause; raising the timeout hides it and costs you a sixty-second frozen screen whenever the GPU genuinely hangs. After cutting chunk length the reset never recurred.
Across five interruptions I had five different causes: a GPU reset, a charger that was not seated, sleep on battery, a background Windows update under memory pressure, and a task timeout. Only the first was mine to fix in code. Without the event log I would have kept tuning the model.
A three-hour unattended job on a laptop will be interrupted. Design for it rather than hoping.
The highest-value change is caching every unit of work to disk as it completes, keyed on its inputs. Write to a temporary name and rename into place, so a crash mid-write cannot leave a truncated file that poisons the next run.
tmp = path.with_suffix(".part") with open(tmp, "wb") as fh: # np.save appends ".npy" to a bare np.save(fh, samples) # name, so hand it a handle instead tmp.replace(path)
That comment is a bug I shipped and hit immediately. np.save appends .npy to any path lacking it, so writing to chunk.part produced chunk.part.npy and the rename looked for a file that did not exist. It died on the first unit of an eighteen-file run. Passing an open handle keeps the name you chose.
Across five interruptions, that cache cost me a few seconds of redone work in total.
Then check the machine will not put itself to sleep. The idle timer counts user input, not GPU load, so a long render does not keep it awake. My hibernate default was three hours, which is exactly long enough to kill a three-hour job at the worst possible moment.
powercfg /change standby-timeout-ac 0 powercfg /change hibernate-timeout-ac 0
And watch system RAM, not just VRAM. On a machine with 8.5 GB total, one render stalled for 53 minutes on work its neighbours did in two and a half, purely because memory ran short. If a unit of work takes disproportionately long, it is paging, not thinking.
After all of that, the same speech model on the same machine:
| Execution provider | Audio | Compute | Speed |
|---|---|---|---|
| CPU | 118.5 s | 63.4 s | 0.53x |
| CUDA | 118.5 s | 14.7 s | 0.12x |
A 4.3x speedup on a card that was mid-range in 2019 and is not sold any more. Forty-five minutes of narration in six minutes rather than twenty-four.
The card was never the problem. A stale driver, a CPU-only wheel, a missing PATH entry and a dtype mismatch were the problem, and every one of them failed quietly rather than loudly. That is the pattern worth taking away: on old hardware the errors you get are rarely about the hardware, and the hardware gets blamed anyway.
A 4 GB card from 2019 runs a 0.6B model at four times CPU speed. Getting there was almost entirely plumbing, and almost every failure was silent.
If you have an old GPU sitting in a laptop, it is probably more capable than the last time you checked, because the models came down to meet it.
torch.version.cuda after every install that touches torch.get_arch_list(). Newer CUDA drops old cards.add_dll_directory.