🔒 Leveraging TensorFlow-TensorRT integration for Low latency Inference | 🎥 Künstliche Intelligenz Videos

Posted by Jonathan Dekhtiar (NVIDIA), Bixia Zheng (Google), Shashank Verma (NVIDIA), Chetan Tekur (NVIDIA)

that leverages inference optimization on NVIDIA GPUs within the TensorFlow ecosystem. It provides a simple API that delivers substantial performance gains on NVIDIA GPUs with minimal effort. The integration allows for leveraging of the optimizations that are possible in TensorRT while providing a fallback to native TensorFlow when it encounters segments of the model that are not supported by TensorRT.

In our previous is enabled, in the first step, the trained model is parsed in order to partition the graph into Figure 1: An example of graph partitioning and building TRT engine in TF-TRT

Workflow

In this section, we will take a look at the typical TF-TRT workflow using an example.

format contains all the information required to share or deploy a trained model. In native TensorFlow, the workflow typically involves loading the saved model and running inference using TensorFlow runtime. In TF-TRT, there are a few additional steps involved, including applying TensorRT optimizations to the TensorRT supported subgraphs of the model, and optionally pre-building the TensorRT engines.

First, we create an object to hold the conversion parameters, including a precision mode. The precision mode is used to indicate the minimum precision (for example FP32, FP16 or INT8) that TF-TRT can use to implement the TensorFlow operations. Then we create a converter object which takes the conversion parameters and input from a saved model. Note that in TensorFlow 2.x, TF-TRT only supports models saved in the TensorFlow SavedModel format.

Next, when we call the converter convert() method, TF-TRT will convert the graph by replacing TensorRT compatible portions of the graph with TRTEngineOps. For better performance at runtime, the converter method. The code corresponding to the workflow steps mentioned in this section are shown in the codeblock below:

PYTHON

from tensorflow.python.compiler.tensorrt import trt_convert as trt

# Conversion Parameters 
conversion_params = trt.TrtConversionParams(
    precision_mode=trt.TrtPrecisionMode.<FP32 or FP16>)

converter = trt.TrtGraphConverterV2(
    input_saved_model_dir=input_saved_model_dir,
    conversion_params=conversion_params)

# Converter method used to partition and optimize TensorRT compatible segments
converter.convert()

# Optionally, build TensorRT engines before deployment to save time at runtime
# Note that this is GPU specific, and as a rule of thumb, we recommend building at runtime
converter.build(input_fn=my_input_fn)

# Save the model to the disk 
converter.save(output_saved_model_dir)

As can be seen from the code example above, the build() method requires an input function corresponding to the shape of the input data. An example of an input function is shown below:

PYTHON

# input_fn: a generator function that yields input data as a list or tuple,
# which will be used to execute the converted signature to generate TensorRT
# engines. Example:
def my_input_fn():
    # Let's assume a network with 2 input tensors. We generate 3 sets
    # of dummy input data:
    input_shapes = [[(1, 16), (2, 16)], # min and max range for 1st input list
                    [(2, 32), (4, 32)], # min and max range for 2nd list of two tensors
                    [(4, 32), (8, 32)]] # 3rd input list
    for shapes in input_shapes:
        # return a list of input tensors
        yield [np.zeros(x).astype(np.float32) for x in shapes]

Support for INT8

Compared to FP32 and FP16, INT8 requires additional calibration data to determine the best quantization thresholds. When the precision mode in the conversion parameter is INT8, we need to provide an input function to the convert() method call. This input function is similar to the input function provided to the build() method. In addition, the calibration data generated by the input function passed to the convert() method should generate data that are statistically similar to the actual data seen during inference.

PYTHON

from tensorflow.python.compiler.tensorrt import trt_convert as trt

conversion_params = trt.TrtConversionParams(
    precision_mode=trt.TrtPrecisionMode.INT8)

converter = trt.TrtGraphConverterV2(
    input_saved_model_dir=input_saved_model_dir,
    conversion_params=conversion_params)

# requires some data for calibration
converter.convert(calibration_input_fn=my_input_fn)

# Optionally build TensorRT engines before deployment.
# Note that this is GPU specific, and as a rule of thumb we recommend building at runtime
converter.build(input_fn=my_input_fn)

converter.save(output_saved_model_dir)

Example: ResNet-50

The rest of this blog will show the workflow of taking a TensorFlow 2.x ResNet-50 model, training it, saving it, optimizing it with TF-TRT and finally deploying it for inference. We will also compare inference throughputs using TensorFlow native vs TF-TRT in three precision modes, FP32, FP16, and INT8.

Prerequisites for the example :

Ubuntu OS
Docker::
- docker pull tensorflow/tensorflow:latest-gpu
NVIDIA Container Toolkit: installed on the host machine (at the time of writing, check the requirements of the latest tensorflow container). You can check which version is currently installed on your machine by running: nvidia-smi | grep "Driver Version:"

Training ResNet-50 using the TensorFlow 2.x container:

First, the latest release of the ResNet-50 model needs to be downloaded from the TensorFlow github repository:

PYTHON

# Adding the git remote and fetch the existing branches
$ git clone --depth 1  https://github.com/tensorflow/models.git .

# List the files and directories present in our working directory
$ ls -al

rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:05 2020  ./
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:30:45 2020  ../
rw-rw-r--  user user   337 B    Wed Sep 30 15:31:05 2020  AUTHORS
rw-rw-r--  user user  1015 B    Wed Sep 30 15:31:05 2020  CODEOWNERS
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:05 2020  community/
rw-rw-r--  user user   390 B    Wed Sep 30 15:31:05 2020  CONTRIBUTING.md
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:15 2020  .git/
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:05 2020  .github/
rw-rw-r--  user user     1 KiB  Wed Sep 30 15:31:05 2020  .gitignore
rw-rw-r--  user user     1 KiB  Wed Sep 30 15:31:05 2020  ISSUES.md
rw-rw-r--  user user    11 KiB  Wed Sep 30 15:31:05 2020  LICENSE
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:05 2020  official/
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:05 2020  orbit/
rw-rw-r--  user user     3 KiB  Wed Sep 30 15:31:05 2020  README.md
rwxrwxr-x  user user     4 KiB  Wed Sep 30 15:31:06 2020  research/

As noted in the earlier section, for this example we will be using the latest TensorFlow container available in the Docker repository. The user does not need any additional installation steps as TensorRT integration is already included in the container. The steps to pull the container and launch it are as follows:

PYTHON

$ docker pull tensorflow/tensorflow:latest-gpu

# Please ensure that the  Nvidia Container Toolkit is installed  before running the following command
$ docker run -it --rm \
   --gpus="all" \
   --shm-size=2g --ulimit memlock=-1 --ulimit stack=67108864 \
   --workdir /workspace/ \
   -v "$(pwd):/workspace/" \
   -v "</path/to/save/data/>:/data/" \  # This is the path that will hold the training data
   tensorflow/tensorflow:latest-gpu

From inside the container, we can then verify that we have access to the relevant files and the Nvidia GPU we would like to target:

PYTHON

# Let's first test that we can access the ResNet-50 code that we previously downloaded
$ ls -al
drwxrwxr-x  8 1000 1000  4096 Sep 30 22:31 .git
drwxrwxr-x  3 1000 1000  4096 Sep 30 22:31 .github
-rw-rw-r--  1 1000 1000  1104 Sep 30 22:31 .gitignore
-rw-rw-r--  1 1000 1000   337 Sep 30 22:31 AUTHORS
-rw-rw-r--  1 1000 1000  1015 Sep 30 22:31 CODEOWNERS
-rw-rw-r--  1 1000 1000   390 Sep 30 22:31 CONTRIBUTING.md
-rw-rw-r--  1 1000 1000  1115 Sep 30 22:31 ISSUES.md
-rw-rw-r--  1 1000 1000 11405 Sep 30 22:31 LICENSE
-rw-rw-r--  1 1000 1000  3668 Sep 30 22:31 README.md
drwxrwxr-x  2 1000 1000  4096 Sep 30 22:31 community
drwxrwxr-x 12 1000 1000  4096 Sep 30 22:31 official
drwxrwxr-x  3 1000 1000  4096 Sep 30 22:31 orbit
drwxrwxr-x 23 1000 1000  4096 Sep 30 22:31 research

# Let's verify we can see our GPUs:
$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.XX.XX    Driver Version: 450.XX.XX    CUDA Version: 11.X     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:1A:00.0 Off |                  Off |
| 38%   52C    P8     14W / 70W |      1MiB / 16127MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

We can now start training ResNet-50. To avoid spending hours training a deep learning model, this article will use the smaller MNIST dataset. However, the workflow will not change with a more state-of-the-art dataset like ImageNet.

PYTHON

# Install dependencies
$ pip install tensorflow_datasets tensorflow_model_optimization

# Download MNIST data and Train
$ python -m "official.vision.image_classification.mnist_main" \
  --model_dir=./checkpoints \
  --data_dir=/data \
  --train_epochs=10 \
  --distribution_strategy=one_device \
  --num_gpus=1 \
  --download

# Let’s verify that we have the trained model saved on our machine.
$ ls -al checkpoints/

-rw-r--r-- 1 root root      87 Sep 30 22:34 checkpoint
-rw-r--r-- 1 root root 6574829 Sep 30 22:34 model.ckpt-0001.data-00000-of-00001
-rw-r--r-- 1 root root     819 Sep 30 22:34 model.ckpt-0001.index
[...]
-rw-r--r-- 1 root root 6574829 Sep 30 22:34 model.ckpt-0010.data-00000-of-00001
-rw-r--r-- 1 root root     819 Sep 30 22:34 model.ckpt-0010.index
drwxr-xr-x 4 root root    4096 Sep 30 22:34 saved_model
drwxr-xr-x 3 root root    4096 Sep 30 22:34 train
drwxr-xr-x 2 root root    4096 Sep 30 22:34 validation

Obtaining a SavedModel to be used by TF-TRT

After training, Google’s ResNet-50 code exports the model in the SavedModel format at the following path: checkpoints/saved_model/.

The following sample SavedModel has been properly saved, we can proceed with loading it with TF-TRT for inference.

Inference

ResNet-50 Inference using TF-TRT

In this section, we will go over the steps for deploying the saved ResNet-50 model on the NVIDIA GPU using TF-TRT. As previously described, we first convert a SavedModel into a TF-TRT model using the convert method and then load the model.

PYTHON

# Convert the SavedModel
converter = trt.TrtGraphConverterV2(input_saved_model_dir=path)
converter.convert()

# Save the converted model
converter.save(converted_model_path)

# Load converted model and infer
model = tf.saved_model.load(converted_model_path)
func = root.signatures['serving_default']
output = func(input_tensor)

For simplicity, we will use a script to perform inference ( and put it in the working directory “/workspace/” of the same docker container as before. After this, we can execute the script:

PYTHON

$ wget https://raw.githubusercontent.com/tensorflow/tensorrt/master/tftrt/blog_posts/Leveraging%20TensorFlow-TensorRT%20integration%20for%20Low%20latency%20Inference/tf2_inference.py

$ ls
AUTHORS     CONTRIBUTING.md  LICENSE    checkpoints  data      orbit     tf2_inference.py
CODEOWNERS  ISSUES.md        README.md  community    official  research

$ python tf2_inference.py --use_tftrt_model --precision fp16

=========================================
Inference using: TF-TRT …
Batch size: 512
Precision:  fp16
=========================================

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
TrtConversionParams(rewriter_config_template=None, max_workspace_size_bytes=8589934592, precision_mode='FP16', minimum_segment_size=3, is_dynamic_op=True, maximum_cached_engines=100, use_calibration=True, max_batch_size=512, allow_build_at_runtime=True)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Processing step: 0100 ...
Processing step: 0200 ...
[...]
Processing step: 9900 ...
Processing step: 10000 ...

Average step time: 2.1 msec
Average throughput: 244248 samples/sec

Similarly, we can run inference for INT8, and FP32

PYTHON

$ python tf2_inference.py --use_tftrt_model --precision int8

$ python tf2_inference.py --use_tftrt_model --precision fp32

Inference using native TensorFlow (GPU) FP32

You can also run the unmodified SavedModel without any TF-TRT acceleration.

PYTHON

$ python tf2_inference.py --use_native_tensorflow

=========================================
Inference using: Native TensorFlow …
Batch size: 512
=========================================

Processing step: 0100 ...
Processing step: 0200 ...
[...]
Processing step: 9900 ...
Processing step: 10000 ...

Average step time: 4.1 msec
Average throughput: 126328 samples/sec

This run was executed with a NVIDIA T4 GPU. The same workflow will work on any NVIDIA GPU.

Comparing Native Tensorflow 2.x performance vs TF-TRT for Inference

Making minimal code changes to take advantage of TF-TRT can result in a significant performance boost. For example, using the inference script in this blog, with a batch-size of 512 on an NVIDIA T4 GPU, we observe almost 2x speedup with TF-TRT FP16, and a 2.4x speedup with TF-TRT INT8 over native TensorFlow. The amount of speedup obtained may differ depending on various factors like the model used, the batch size, the size and format of images in the dataset, and any CPU bottlenecks.

In conclusion, in this blog we show the acceleration provided by TF-TRT. Additionally, with TF-TRT we can use the full TensorFlow Python API and interactive environments like Jupyter Notebooks or Google Colab.

Supported Operators

The TF-TRT user guide lists .

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf blog.tensorflow.org.

↗ Original-Artikel auf blog.tensorflow.org lesen

Wie bewertest du diesen Beitrag?

1 Klick Feedback

Teilen mit Netzwerk & Team:

Hat Ihnen dieser Tipp / Anleitung geholfen?

Community-Analysen & Experten-Meinungen 0

Verfasse deine eigene Analyse, teile Workarounds oder diskutiere diesen Vorfall im Blog.

Noch keine Community-Analyse verfasst. Markiere einen Textabschnitt oder klicke oben auf „ Eigene Analyse verfassen“!

Community Pulse: Relevanz-Einschätzung

1 Klick Experten-Votum

Auch interessante Nachrichten Leveraging TensorFlow-TensorRT integration for Low latency Inference

Thematisch verwandte Begriffe: Leveraging, TensorFlowTensorRT, integration, latency · 6 Treffer

🔧 AI Nachrichten MacDailyNews

Apple accuses OpenAI of destroying evidence as trade-secrets fight intensifies

vor 1 Tag 📆 01.09.2026 um 01:44 Uhr

Weiterlesen ↗

🔧 AI Nachrichten MacDailyNews

Major AI platforms go down in unprecedented simultaneous outage

vor 1 Tag 📆 03.09.2026 um 17:34 Uhr

Weiterlesen ↗

🔧 AI Nachrichten The Mac Observer

GPT-6 Astra Release Today? OpenAI’s Next Major AI Model Is Almost Here

vor 1 Tag 📆 03.09.2026 um 19:08 Uhr

Weiterlesen ↗

🔧 AI Nachrichten The Mac Observer

ChatGPT, Claude, and Grok Down? Users Report Widespread Outages

vor 1 Tag 📆 03.09.2026 um 19:14 Uhr

Weiterlesen ↗

🔧 AI Nachrichten The Mac Observer

OpenAI Launches GPT-6 Astra, Says We May Have Entered the AGI Era

vor 1 Tag 📆 03.09.2026 um 22:08 Uhr

Weiterlesen ↗

🔧 AI Nachrichten The Mac Observer

Claude Comes to CarPlay as Fifth Major AI Chatbot App

vor 1 Tag 📆 05.09.2026 um 05:31 Uhr

Weiterlesen ↗

Videos werden geladen ...

Beiträge werden geladen ...

Videos werden geladen ...

Beiträge werden geladen ...

Videos werden geladen ...

Beiträge werden geladen ...

Videos werden geladen ...

Beiträge werden geladen ...

Videos werden geladen ...

🚀 Auf tsecurity.de entdecken

Community Diskussion & Feedback

💬 Social News Network

Community Debatte 0 Beiträge

Tags:Noch keine Tags.

Noch keine Diskussion eröffnet

Sei der Erste, der seine Einschätzung oder PoC zu diesem Thema teilt.

🚀 Auf tsecurity.de entdecken

🔖 Gespeicherte Artikel

📂 Keine gespeicherten Artikel vorhanden.

Alle Lesezeichen verwalten ➔

📂 News ⏱️ 3 Min vor 10 Min

Artikeldaten werden geladen...

Im Vollbild öffnen ↗

Zum Aktualisieren ziehen

TTS Reader • tsecurity.de Voice

tsecurity.de App

Offline-Lesen, Eilmeldungen & 0ms Ladezeit

Installiere tsecurity.de direkt auf deinen Home-Bildschirm für das ultimative Vollbild-Magazinerlebnis ohne Browser-Leisten.

Nächster Beitrag

Dein Cluster: Security Explorer

👥 Match:

lädt…

Verbindung zum Community-Stream wird aufgebaut...

📡 Aktivitäten deiner Analysten

lädt…

💡 Neues Thema oder Eilmeldung einreichen

Reiche interessante Links, Zero-Days oder Debatten ein. Die Community entscheidet per Upvote über die Veröffentlichung.

🔥 Heiß diskutierte Einreichungen

🔥 TRENDING VOTE

vor 12s

tsecurity.de Live Threat Radar

Aktuelle Bedrohungslagen & Sicherheits-Analysen

Jetzt ansehen

0-Day CVE Alerting

Geräteübergreifende Merkliste

Experten-Diskussionen

Karma & Leaderboard

Benutzername / Alias

E-Mail-Adresse (für Benachrichtigungen)

Passwort (mind. 8 Zeichen)

Deine Interessensgebiete (optional):

Zero-Days ☁️ Cloud Security 🎯 Pentesting 🐧 Linux & Kernel 🤖 AI Security

Cyberbriefing abonnieren: Wöchentliche Top-CVEs, 0-Day Intelligence & Incident-Reports kostenlos per E-Mail.

Ich stimme der Datenschutzerklärung und den AGB zu (DSGVO-konform, Double-Opt-In).

tsecurity.de LIVE

tsecurity.de Live Threat Radar

Leveraging TensorFlow-TensorRT integration for Low latency Inference

Workflow

Support for INT8

Example: ResNet-50

Prerequisites for the example :

Training ResNet-50 using the TensorFlow 2.x container:

Obtaining a SavedModel to be used by TF-TRT

Inference

ResNet-50 Inference using TF-TRT

Inference using native TensorFlow (GPU) FP32

Comparing Native Tensorflow 2.x performance vs TF-TRT for Inference

Supported Operators

Community-Analysen & Experten-Meinungen 0

Verwandte Story-Cluster & Quellen (Vektor-KI)

Auch interessante Nachrichten Leveraging TensorFlow-TensorRT integration for Low latency Inference

Apple accuses OpenAI of destroying evidence as trade-secrets fight intensifies

Major AI platforms go down in unprecedented simultaneous outage

GPT-6 Astra Release Today? OpenAI’s Next Major AI Model Is Almost Here

ChatGPT, Claude, and Grok Down? Users Report Widespread Outages

OpenAI Launches GPT-6 Astra, Says We May Have Entered the AGI Era

Claude Comes to CarPlay as Fifth Major AI Chatbot App

🔖 Gespeicherte Artikel

tsecurity.de App

📡 Aktivitäten deiner Analysten

💡 Neues Thema oder Eilmeldung einreichen

🔥 Heiß diskutierte Einreichungen

tsecurity.de LIVE

Themen-Radar & Intelligence Matrix

tsecurity.de Live Threat Radar

Leveraging TensorFlow-TensorRT integration for Low latency Inference

Workflow

Support for INT8

Example: ResNet-50

Prerequisites for the example :

Training ResNet-50 using the TensorFlow 2.x container:

Obtaining a SavedModel to be used by TF-TRT

Inference

ResNet-50 Inference using TF-TRT

Inference using native TensorFlow (GPU) FP32

Comparing Native Tensorflow 2.x performance vs TF-TRT for Inference

Supported Operators

Community-Analysen & Experten-Meinungen 0

Verwandte Story-Cluster & Quellen (Vektor-KI)

Auch interessante Nachrichten Leveraging TensorFlow-TensorRT integration for Low latency Inference

Apple accuses OpenAI of destroying evidence as trade-secrets fight intensifies

Major AI platforms go down in unprecedented simultaneous outage

GPT-6 Astra Release Today? OpenAI’s Next Major AI Model Is Almost Here

ChatGPT, Claude, and Grok Down? Users Report Widespread Outages

OpenAI Launches GPT-6 Astra, Says We May Have Entered the AGI Era

Claude Comes to CarPlay as Fifth Major AI Chatbot App

Bleibe Cyber-Bedrohungen einen Schritt voraus

🔖 Gespeicherte Artikel

tsecurity.de App

Community Radar & Live Chat

📡 Aktivitäten deiner Analysten

💡 Neues Thema oder Eilmeldung einreichen

🔥 Heiß diskutierte Einreichungen 🔄 Aktualisieren

🔥 Heiß diskutierte Einreichungen