AMD-Pro-BenchCommunity

13,812 benchmarks in 32 GPU combinations documented

Llama-3-70B-Instruct on W7900

19 results · 2 sources · 19 measurements

Explore all results

Grouped by reported model identity. Exact weights, quantizations and software can differ. How results are reviewed

Documented results

19 selected tests
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py——
7.03 tok/sGeneration · per request
1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
3968 / 128 tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructGPTQ (bit width unreported)Model ref. ↗Token speed——
~17 tok/sGeneration · per request
1 × W7900Single GPU · 48 GB VRAM/GPU
Configuration
Memory
48 GB VRAM/GPU
OS
Linux
Engine
vLLM (AMD fork)
Backend
ROCm
Harness
Not reported
Prompt / output
? / ? tokens
Context
Not reported
Concurrency
Not reported
Date
2024-07-11
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py——
4.51 tok/sGeneration · per request
1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
8064 / 128 tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py——
6.98 tok/sGeneration · per request
1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
3968 / 128 tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
154.06 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
128 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
269.56 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
256 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
358.51 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
384 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
359.84 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
512 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
365.2 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
640 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
429.57 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
768 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
426.6 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
896 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
430.63 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
1024 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
416.85 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
2048 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
394.76 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
3072 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
363.34 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
4096 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Token speedtest_inference.py—
283.31 tok/sPrefill · per request
—1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
8192 / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Memorytest_inference.py1 concurrent request
~39 GiBVRAM usage · Whole system
——1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
? / ? tokens
Context
4096
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Latencytest_inference.py1 concurrent request
2,802,517.2 msModel load time · Whole system
——1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
? / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
View
ReportGeneric LLM		Llama-3-70B-InstructEXL2 4.0bpw h6Weights ↗Latencytest_inference.py1 concurrent request
3,402,622.2 msModel load time · Whole system
——1 × W7900Single GPU
Configuration
Memory
Not reported
OS
Ubuntu 22.04 LTS HWE
Engine
ExLlamaV2
Backend
ROCm 6.0
Harness
Not reported
Prompt / output
? / ? tokens
Context
Not reported
Concurrency
1
Date
Not reported
Full record →
Quantizations and software

Quantizations: EXL2 4.0bpw h6, GPTQ (bit width unreported)
Engines: ExLlamaV2, vLLM (AMD fork)

Explore all 19 results →

This is a selection across sources, not a ranking or a matched experiment. Reading benchmark reports

GPU benchmark pages: W7900