ninfer

High-performance single-GPU inference for selected model checkpoints and GPUs.

NInfer for a 10 GB RTX 3080 on native Windows: MSVC and CUDA 13 build fixes, a built-in RTX 3080 device profile, and ninfer-switch. Based on iamwavecut/ninfer-3090.

Lightweight Windows GUI, tray controller and model manager for NInfer

A cross-platform desktop app that tails the NInfer server request log in real time and displays the core operational details of a running NInfer inference server in a live GUI dashboard:

Port NInfer, a single-GPU CUDA inference engine, to the NVIDIA L20 (Ada sm_89, 92 SMs, 48 GB): patch set, build tooling, and measured results

Measuring proxy + throughput dashboard for local LLM engines (llama.cpp, NInfer): live tok/s, cache-hit rate, TTFT, and history charts. Single-file panel, stdlib-only proxy, MIT.

Qwen3.8-27B on RTX 4090 D (48GB): production deployment of NInfer with MTP7 + E8 KV + NVMe disk cache, 195 tok/s decode, crash forensics for WDDM desktop GPUs

Rust CLI to run local LLMs through rootless Podman Compose