NVIDIA TensorRT™ is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high-throughput for deep learning inference applications.



TensorRT-based applications perform up to 40x faster than CPU-only platforms during inference. With TensorRT, you can optimize neural network models trained in all major frameworks, calibrate for lower precision with high accuracy, and finally deploy to hyperscale data centers, embedded, or automotive product platforms.

TensorRT is built on CUDA, NVIDIA’s parallel programming model, and enables you to optimize inference for all deep learning frameworks leveraging libraries, development tools and technologies in CUDA-X for artificial intelligence, autonomous machines, high-performance computing, and graphics.

TensorRT provides INT8 and FP16 optimizations for production deployments of deep learning inference applications such as video streaming, speech recognition, recommendation and natural language processing. Reduced precision inference significantly reduces application latency, which is a requirement for many real-time services, auto and embedded applications.

Rocket Fuel

(Click to Zoom)


You can import trained models from every deep learning framework into TensorRT. After applying optimizations, TensorRT selects platform specific kernels to maximize performance on Tesla GPUs in the data center, Jetson embedded platforms, and NVIDIA DRIVE autonomous driving platforms.

With TensorRT developers can focus on creating novel AI-powered applications rather than performance tuning for inference deployment.



TensorRT Optimizations and Performance



Weight & Activation Precision Calibration

Maximizes throughput by quantizing models to INT8 while preserving accuracy


Layer & Tensor Fusion

Optimizes use of GPU memory and bandwidth by fusing nodes in a kernel


Kernel Auto-Tuning

Selects best data layers and algorithms based on target GPU platform


Dynamic Tensor Memory

Minimizes memory footprint and re-uses memory for tensors efficiently


Multi-Stream Execution

Scalable design to process multiple input streams in parallel


TensorRT dramatically accelerates deep learning inference performance on NVIDIA GPUs. See how it can power your inference needs across multiple networks with high throughput and ultra-low latency.




Widely Adopted






Integrated with All Major Frameworks



NVIDIA works closely with deep learning framework developers to achieve optimized performance for inference on AI platforms using TensorRT. If your training models are in the ONNX format or other popular frameworks such as TensorFlow and MATLAB, there are easy ways for you to import models into TensorRT for inference. Below are few integrations with information on how to get started.

Tensor logo


TensorRT and TensorFlow are tightly integrated so you get the flexibility of TensorFlow with the powerful optimizations of TensorRT. Learn more in the blog post.

Mathworks logo

MATLAB is integrated with TensorRT through GPU Coder so that engineers and scientists using MATLAB can automatically generate high-performant inference engines for Jetson, DRIVE and Tesla platforms. Learn more in this .

ONNX logo

TensorRT provides an ONNX parser so you can easily import ONNX models from frameworks such as Caffe 2, Chainer, Microsoft Cognitive Toolkit, MxNet and PyTorch into TensorRT. Learn more about ONNX support in TensorRT .

TensorRT is also integrated with ONNX Runtime, providing an easy way to achieve high-performance inference for machine learning models in the ONNX format. Learn more about ONNX Runtime - TensorRT integration .

If you are performing deep learning training in a proprietary or custom framework, use the TensorRT C++ API to import and accelerate your models. Read more in the .


“In our evaluation of TensorRT running our deep learning-based recommendation application on NVIDIA Tesla V100 GPUs, we experienced a 45x increase in inference speed and throughput compared with a CPU-based platform. We believe TensorRT could dramatically improve productivity for our enterprise customers.”

— Markus Noga, Head of Machine Learning at SAP SAP logo

“By using tensor cores on the V100, the most recently optimized CUDA libraries and the TF-TRT backend we were able to speed up our already fast DL network by a factor of 4x”



— Kris Bhaskar, KLA Senior Fellow, VP AI initiatives, KLA KLA logo

“Criteo uses Nvidia's TensorRT over T4 cards to optimize its deep-learning models for faster inference on GPUs. Now, removing inappropriate images over billions of them is 4 times faster. It also consumes half less energy.”


— Suju Rajan, SVP Research, Criteo Criteo logo