NVIDIA L4
Data center GPU · Ada Lovelace
Summary
The NVIDIA's L4 is a low-power, efficient GPU which works well for video transcoding workloads. It is also capable of inference on very small LLMs, but the larger and more powerful NVIDIA L40s is generally a better fit for inference workloads.
Launched
Q2 2023
VRAM
24 GB
Mem. bandwidth
300 GB/s
On-demand from
Tech specs
VRAM 24 GB GDDR6
Memory bandwidth 300 GB/s
Interface PCIe
CUDA cores 7,424
Tensor cores 240 (Gen 4)
TDP 72 W
Supported data types
FP64FP32TF32FP16BF16FP8INT8
Cloud rental prices
8 providers · available in 18 countries · lowest price per GPU, per hour
On-demand
$0.44 – $1.67 /GPU/h
Reserved
$0.37 – $1.09 /GPU/h
Spot
$0.29 /GPU/h
| Provider | On-demand | Reserved | Spot |
|---|---|---|---|
 AceCloud | — | $0.60 | — |
 AWS | $0.80 | $0.37 | — |
| | $0.44 | $0.40 | $0.29 |
 OVHcloud | $0.87 | — | — |
 Runpod | $0.49 | — | — |
| | $0.91 | — | — |
 Seeweb | $0.44 | $0.37 | — |
| | $1.05 | — | — |
Models that fit in VRAM
Open-weights models that fit in VRAM. Estimated using 🤗 accelerate, plus approximation for up to 8K context.
Media
Peak theoretical performance
FP8 Tensor Core 242 TFLOPS
INT8 Tensor Core 242 TOPS
BF16 Tensor Core 121 TFLOPS
FP16 Tensor Core 121 TFLOPS
TF32 Tensor Core 60 TFLOPS
FP32 30.3 TFLOPS
Performance figures assume no sparsity; in cases where only sparse performance figures are published by the manufacturer, these are halved to give approximate dense performance.
Resources
Detailed documentation from the manufacturer.