Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

A tool for examining GPU scheduling behavior.

PipelineScheduler optimizes workload distribution between servers and edge devices, setting optimal batch sizes to maximize throughput and minimize latency amid content dynamics and network instability. It also addresses resource contention with spatiotemporal inference scheduling to reduce co-location interference.

The GPU Optimizer for ML Models enhances GPU performance for machine learning. It offers advanced scheduling, real-time monitoring, and efficient resource management through a user-friendly web interface and robust API, integrating big data technologies for seamless data processing and model optimization. @NVIDIA

Design of a GPU Dynamic LLM Inference Task Scheduling Architecture Based on KubeAI

HPC research toolkit infrastructure for interfacing & analyzing LLMs (Kit is composed of: API gateway service, GPU scheduler, model servicer, and web interface)