Skip to content
IFAI

TensorRT-LLM

Product2 stories

  1. AI toolsReach 54

    NVIDIA brings custom AI to your local PC

    NVIDIA just released Chat with RTX, a free tech demo that lets you run customized chatbots completely offline on Windows. By connecting local files and YouTube links to open-source models like Mistral and Llama 2 via TensorRT-LLM, you can query notes and documents without sending data to the cloud. You will need a GeForce RTX 30 Series GPU or higher with at least 8GB of VRAM.

  2. Open sourceReach 48

    NVIDIA open-sources TensorRT-LLM for faster H100 inference

    NVIDIA just introduced TensorRT-LLM, an open-source library built to accelerate large language model inference on modern GPUs. It brings in-flight batching and FP8 quantization, delivering up to an 8x throughput boost on H100 GPUs and running Llama 2 4.6x faster than A100 systems. Why it matters: teams can now tap peak GPU parallelism via a modular Python API without writing low-level CUDA, slashing data center running costs.