NVIDIA H100 SXM5 Cluster Latency & Throughput Benchmarks
Walker Direct-Answer Invariant
What empirical TTFT latency and throughput does Elesium achieve on 8x NVIDIA H100 SXM5?
Audited benchmark telemetry across 8x NVIDIA H100 80GB SXM5 clusters connected via 900 GB/s NVLink v4 confirms a median Time-To-First-Token (TTFT) of 148ms and sustained generation throughput of 1,403 tokens/second per node across FP8 quantized weights.
AUDITED PERFORMANCE TELEMETRY | NVIDIA H100 SXM5 CLUSTER
All benchmark measurements were conducted inside an air-gapped VPC located in Mumbai (ap-south-1) on bare-metal 8x NVIDIA H100 80GB SXM5 instances connected via 900 GB/s bidirectional NVLink v4. Tests were executed across 100,000 continuous benchmark cycles.
Audited Latency & Throughput Matrix
| Model Architecture | Quantization | Batch Size | Time-To-First-Token (TTFT) | Inter-Token Latency (ITL) | Throughput (Tokens/s) |
|---|---|---|---|---|---|
| Elesium-Indic-70B | Native FP8 (W8A8) | 1 (Real-time Stream) | 148 ms | 8.2 ms | 121.9 tok/s |
| Elesium-Indic-70B | Native FP8 (W8A8) | 16 (Concurrent Batch) | 182 ms | 11.4 ms | 1,403.5 tok/s |
| Elesium-Sovereign-8B | Bfloat16 (Uncompressed) | 1 (Real-time Stream) | 38 ms | 3.1 ms | 322.5 tok/s |
| Elesium-Sovereign-8B | Bfloat16 (Uncompressed) | 32 (Concurrent Batch) | 56 ms | 4.8 ms | 6,666.6 tok/s |
Reproducibility Command
To reproduce these benchmarks on your private cluster, use our open-source benchmark utility:
BASHTerminal
python3 -m automation.benchmark_serving --backend vllm --host 10.240.16.4 --port 8443 --model elesium-indic-sovereign-70b-fp8 --num-prompts 1000 --input-len 512 --output-len 256