How to convert a PyTorch model to TensorRT and speed up inference (2023)

The life of a machine learning engineer consists of long periods of frustration and a few moments of joy!

First, fight for your model to perform well with your training data. You look at your training data, clean it up and train again. You read about itbias variance compensationin machine learning to approach the training process systematically.

One fine day, your PyTorch model will be perfectly trained and ready for production.

This is pure joy!

You pride yourself on accuracy, mark your task as complete on your project tracker, and let your CTO know the model is complete.

She shakes her head disapprovingly and announces that the model is not yet ready for series production. Training a model is not enough. You need to change the model to be efficient at run time (aka inference).

You don't know how to proceed. Your friendly CTO recommends reading this post on TensorRT on LearnOpenCV.com. So here it is to indulge in a different learning experience.

This post will show you how to use it quickly and easilyTensorRTfor implementation if you already have the networkPyTorch.

We will use the following steps.

1. Train a model with PyTorch
2. Convert the model to ONNX format
3. Use NVIDIA TensorRT for inference

In this tutorial, we'll just use a pre-trained model, so we'll skip step 1. Now, let's understand what ONNX and TensorRT are.

What is ONX?

There are many frameworks for training a deep learning model. The most popular are Tensorflow and PyTorch. However, a model trained by Tensorflow cannot be used with PyTorch and vice versa.

ONNX stands for Open Neural Network Exchange. It is an open format created to represent machine learning models.

You can train your model in any framework of your choice and then convert it to ONNX format.

The great advantage of a common format is that the software or hardware that loads your model at runtime only has to be ONNX compatible.

ONNX is to machine learning models what JPEG is to images or MPEG to video.

What is TensorRT?

NVIDIA TensorRT is a high performance deep learning inference SDK.

It provides APIs to infer pre-trained models and builds runtime engines optimized for your platform.

There are a variety of ways in which this optimization is achieved. For example, TensorRT allows us to use INT8 (8-bit integer) or FP16 (16-bit floating point) arithmetic instead of the usual FP32. This reduction in accuracy can significantly speed up inference with a small reduction in accuracy.

Other types of optimizations include minimizing the memory footprint of the GPU through memory reuse, merging layers and tensors, choosing the appropriate data layers based on hardware, etc.

Setting up the environment for TensorRT

To reproduce the experiments mentioned in this article, you need aNvidiaGraphic card. Any architecture newer than Maxwell that has a compute capacity of 5.0 will do. You can find your GPU processing power in the table here:https://developer.nvidia.com/cuda-gpus#compute. Don't forget to install properlydriver.

Install PyTorch, ONNX and OpenCV

To installPython 3.6or later and run

python3 -m pip install -r requisitos.txt

Requirements.txtContents:

Torch==1.2.0torchvision==0.4.0albumentations==0.4.5onnx==1.4.1opencv-python==4.2.0.34

The code has been tested on specific versions. But it's okay to try running it on other versions if you already have some of those components installed.

Install TensorRT

1. Download and installNVIDIA CUDA 10.0or later according to official instructions:shortcut
2. download and extractCuDNNLibrary for your version of CUDA (login required):shortcut
3. Download and extract NVIDIATensorRTLibrary for your version of CUDA (login required):shortcut. The minimum required version is 6.0.1.5. Please follow theinstallation Guidefor your system and don't forget to install itPython part
4. Add absolute path to CUDA, TensorRT, CuDNN libraries to environment variableAWAYÖLD_LIBRARY_PATH
5. To installPyCUDA

Now we are ready for our experiment.

How to convert a PyTorch model to TensorRT

Let's walk through the steps required to convert a model from PyTorch to TensorRT.

1. Load and run a pre-trained model with PyTorch

First, let's implement a simple classifier with a pre-trained network in PyTorch. For example we will takeResnet50but you can choose what you want. More information and explanations about working with PyTorch can be found here:# PyTorch for beginners: image classification with pre-trained models

Torchvisionmodel import models = models.resnet50 (pretrained = true)

Next important step:preprocessorthe input image. We need to know what transformations were performed during training in order to replicate them for inference. We recommend the following modules for the preprocessing step:albuminationsjCV2(OpenCV).

The model was trained on 224×224 images. The input data was then normalized (divide the pixel values ​​by 255, subtract the mean and divide by the standard deviation).

import cv2import torquefrom albumentations import Resize, Composefrom albumentations.pytorch.transforms import ToTensorfrom albumentations.augmentations.transforms import Normalizedef preprocess_image(img_path): # transformaciones para los datas de enterda transforms = Compose([ Resize(224, 224, interpolation=cv2.INTER_NEAREST ) , Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ToTensor(), ]) # leeres Bild vom Eingang input_img = cv2.imread(img_path) # hacer transformaciones input_data = transforms(image =entrada_img)["imagen"]

Prepare batch for submission to network. In our case, there is only one image in the stack. Please note that we load input data onto the GPU to run the program faster and to keep our comparison with TensorRT honest.

batch_data = antorcha.unsqueeze(input_data, 0) return lote_datosput = preprocess_image("turkish_coffee.jpg").cuda()

Now we can draw the conclusion. Don't forget to switch the model to evaluation mode and also copy it to the GPU. As a result, we get tensor[1, 1000] with certainty about which class the object belongs to.

model.eval() model.cuda() output = model(input)

To get human-readable results, we need the post-processing step. The names of the classes can be found inimagenet_classes.txt. calculationweichmaxto get percentages for each class and print out the best classes provided by the network.

def postprocess(output_data): # class names with open("imagenet_classes.txt") as f:classes = [line.strip() for line in f.readlines()] # compute human-readable value with softmax trusts = Torch . nn.functional.softmax(output_data, dim=1)[0] * 100 # finds the predicted top classes _, indices = Torch.sort(output_data, descending=True) i = 0 # returns the top classes predicted by the model off while confidences[indexes[0][i]] > 0.5: classidx = indexes[0][i] print( "class:", classes[classidx], ", confidences:", confidences[ class idx].Element( ), " %, index:", class_idx.item(), ) i += 1postprocess(output)

It's time to test our script! Our input picture:

And results:

Class: Cup Confidence: 92.430747% Index: 968 Class: Espresso Confidence: 6.138075% Index: 967 Class: Coffee Cup Confidence: 0.728557% Index: 504

2. Convert the PyTorch model to ONNX format

To convert the resulting model, you only need one statementTorch.onnx.export, which took the following arguments: the pretrained model itself, tensor of the same size as the input data, ONNX filename, input and output names.

ONNX_FILE_PATH = 'resnet50.onnx'torch.onnx.export(modelo, enterda, ONNX_FILE_PATH, input_names=['input'], output_names=['output'], export_params=True)

Call to check if the model converted wellonnx.checker.check_modelo:

onnx_model = onnx.load(ONNX_FILE_PATH)onnx.checker.check_model(onnx_model)

3. View the ONNX model

Now let's visualize our ONNX chart withNeutron. To install this, start:

python3 -m pip installs netron

WriteNeutronon the command line and openhttp://localhost:8080/in your browser. You see the complete network diagram. Make sure the input and output are the expected size.

4. Initialize the model in TensorRT

Now it's time to parse the ONNX model and initialize TensorRTcontextjMotor. To do this, we need to create an instance ofconstructor. The builder can createThe networkand generateMotor(which would be optimized for your platform/hardware) from that network. than we createdThe networkWe can define the structure of the network through flags, but in our case it is sufficient to use the default flag, which means that all tensors would have an implicit batch dimension. WithThe networkDefinition of which we can create an instanceAnalyzerand finally analyze our ONNX file.

import pycuda.driver as cudaimport pycuda.autoinitimport numpy as npimport tensorrt as trt# logger to capture errors, warnings, and other information during the build and inference phasesTRT_LOGGER = trt.Logger()def build_engine(onnx_file_path): # initialize the TensorRT engine and parse ONNX model builder = trt.Builder(TRT_LOGGER) network = builder.create_network() parser = trt.OnnxParser(network, TRT_LOGGER) # parse ONNX mit open(onnx_file_path, 'rb') als model: print('Starting parsing of ONNX files' ) parser.parse(model.read()) print('Vollständige Analyse der ONNX-Datei')

It is possible to configure some engine parameters, e.g. B. the maximum memory allowed by the TensorRT engine or the FP16 mode. We also need to specify the lot size.

# allow TensorRT to use up to 1GB GPU memory for tactical selection builder.max_workspace_size = 1 << 30 # we only have one image in batch builder.max_batch_size = 1 # use FP16 mode if possible if builder.platform_has_fast_fp16 : builder.fp16_mode = True

After that we can generate itMotorand build the executablecontext. The engine takes input data, performs inference, and outputs inference results.

# Building the TensorRT engine optimized for the target platform print('Building an engine...') engine = builder.build_cuda_engine(network) context = engine.create_execution_context() print("Engine build complete") return engine, context

Tips:Initialization can take a long time as TensorRT tries to find the best and fastest way to create your network on your platform. To just do it once and then use the already built engine you canserializeYour Engine Serialized engines are not portable between different GPU models, platforms, or versions of TensorRT. Engines are specific to the exact hardware and software they are built on. Here you will find more information:https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#serial_model_c.

5. Main pipe

So what would the full pipeline for inference in TensorRT look like? Let's take a look at thoseHeadmasterFunction. First, let's analyze the model and initialize the engine and context:

def main(): # Initialize TensorRT engine and parse ONNX model engine, context = build_engine(ONNX_FILE_PATH)

Once we initialize the engine, we can figure out the input and output dimensions in our program. To know that we can allocate the required memory for the input data and the output data. In normal cases a model can have many inputs and outputs, but in our case we know that we only have one input and one output.

# get the input and output sizes and allocate the memory required for the input and output data to bind in the engine: if engine.binding_is_input(binding): # expect only one input input_shape = engine.get_binding_shape(binding) input_size = trt .volume (input_shape) * engine.max_batch_size * np.dtype(np.float32).itemsize # in bytes device_input = cuda.mem_alloc(input_size) else: # and create an output output_shape = engine.get_binding_shape(binding) # per page locked memory buffers ( i.e. not switched to disk) host_output = cuda.pagelocked_empty(trt.volume(output_shape) * engine.max_batch_size, dtype=np.float32) device_output = cuda.mem_alloc(host_output.nbytes)

CUDA functions can be called asynchronouslystreams, scripts that are executed in order. All instructions in a thread are executed sequentially, but different threads can execute their instructions at the same time or out of order. If you run asynchronous CUDA commands without specifying a stream, the runtime uses the default null stream. In our simple script we create only one stream and that would be enough. For example, in more complicated cases, you can use different streams to process different images at the same time.

# Create a sequence where inputs/outputs are copied and inferences are made. stream = cuda.Stream()

To get the same result in TensorRT as in PyTorch, we would prepare the data for inference and repeat any preprocessing steps we did previously. The main advantage of the Python API for TensorRT is that data pre-processing and post-processing can be reused from the PyTorch side. The only thing we need to do additionally is place datacoherentand use paged locked memory whenever possible. We can then copy this data to the GPU and use it for inference.

# Vorverarbeitungsdaten vom Eingang host_input = np.array(preprocess_image("turkish_coffee.jpg").numpy(), dtype=np.float32, order='C') cuda.memcpy_htod_async(device_input, host_input, stream)

Infer and copy the output from the device to the host:

# Ejecutar-Inferenz context.execute_async(bindings=[int(device_input), int(device_output)], stream_handle=stream.handle) cuda.memcpy_dtoh_async(host_output, device_output, stream) stream.synchronize()

The result is saved inoutput_hostas an array with one dimension. So before we use the post-processing part of PyTorch to get human-readable values, we need to refactor it.

# Verarbeitungsergebnisse output_data = Torch.Tensor(host_output).reshape(engine.max_batch_size, output_shape[0]) postprocess(output_data)

That's all! Now you can run and test your script.

6. Accuracy test

We ran some ad hoc tests, summarized in the table below.

As we can see, the predicted classes agree. Confidence is almost the same in FP32 mode (bug smaller than 1e-05). In FP16 mode the error is larger (~0.003) but still enough to get correct predictions.

Note that there is no guarantee that testing with different hardware, software, or even an input image will result in the same error. The error may depend on the initial reference decision and may vary with different cards. We get these results with the following configuration:

Ubuntu 18.04.4, octa-core ×16 AMD® Ryzen 7 2700x processor, GeForce RTX 2070 SUPER, TensorRT 6.0.1.5, CUDA 10.0

7. Accelerate with TensorRT

To compare the time in PyTorch and TensorRT, we would not measure the model initialization time since we only initialize the model once. So we will compare the inference time. When first started, CUDA initializes and saves some data, so the first call to a CUDA function is slower than usual. To account for this, we run the inference multiple times and get an average time. And what we have:

In our example, we achieved 4x to 6x speedup in FP16 mode and 2x to 3x speedup in FP32 mode.