Optimizing DeepSeek-V4-Flash-0731 for Full Precision Lossless Performance with Dual 7900XTX GPUs and 128GB RAM
In the quest to push the boundaries of large language model (LLM) deployment, achieving full-precision, lossless performance presents unique challenges and opportunities. Recently, I undertook an experiment to run DeepSeek-V4-Flash-0731 in full precision mode, leveraging a high-performance hardware setup comprising two AMD Radeon RX 7900 XTX GPUs and 128GB of system RAM. The goal: to determine how to fit the model comfortably within available memory while maintaining robust throughput and stability.
Model Size and Hardware Capacity
The full-precision version of DeepSeek-V4-Flash-0731 measures approximately 162 GB. Combined with my system’s total memory and GPU VRAM (which together total around 176 GB), I aimed to accommodate the entire model and its associated processes without resorting to significant offloading or complexity. This open-ended exploration raises the question: what does an optimized setup for such a large, lossless model look like?
Performance Metrics
The per-request throughput observed during testing was approximately 52 tokens per second during prefill, and around 10.5 tokens per second during generation. These figures reflect a single user working intensively on a complex coding problem—traversing a graph of related issues over several hours—essentially a proof-of-concept scenario rather than a production-ready environment. Nonetheless, the results showcase the potential for sustained, high-quality problem-solving with full-precision models in a dedicated hardware context.
Model Configuration and Deployment Details
The following is a overview of the core configuration parameters used to run the model:
bash
llama-server \
--model /ds4/models/DeepSeek-V4-Flash-0731-Q8/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
--alias deepseek-v4-flash-0731 \
--host 0.0.0.0 \
--port 8080 \
--jinja \
--metrics \
--perf \
--log-timestamps \
--device ROCm0,ROCm1 \
--split-mode layer \
--n-gpu-layers 44 \
--tensor-split 7,37 \
--fit off \
--fit-target 512,512 \
--ctx-size 131072 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 512 \
--threads 12 \
--threads-batch 12 \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-ram 256 \
--load-mode none \
--spec-type draft-dspark \
--spec-draft-model /ds4/models/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K-Q8_0-pr25784.gguf \
--spec-draft-device ROCm1 \
--spec-draft-ngl all \
--spec-draft-n-max 5 \
--spec-draft-n-min 1 \
--spec-draft-p-min 0.7 \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0 \
--override-tensor 'blk.6.ffn_(gate|up|gate_up|down).* = ROCm1, blk.(7|8).ffn_(up|down|gate_up|gate)_(ch|) exps=CPU, blk.9.ffn_(down|gate).* = CPU, blk.(10-43).ffn_(up|down|gate|exps)=CPU'
Key Notes on Configuration
-
Tensor Split Strategy: Splitting the model into layers across the GPUs—ROCm0 handling seven larger blocks (0-6) containing MoE expert tensors, which remain resident in VRAM; ROCm1 managing 37 thinner blocks (7-43) with most experts evicted to RAM and CPU, focusing on attention layers and the drafter.
-
Override Tensor Settings: By selectively moving parts of block 9 and other components onto RAM or CPU via
--override-tensor, I optimized memory usage to fit the entire model within the available hardware. -
Batching and Threads: A modest
--ubatch-size(512) and multiple threads help balance throughput and memory constraints, though larger batch sizes may lead to OOMs on limited hardware.
Supporting Tools and Resources
For drafting and model management, I used the DeepSpark Drafter in Q2_K-Q8_0 mode, which can be found here:
https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF
Using a lower-bit precision drafter enabled the model to fit into RAM without sacrificing the final quality of the full-precision version, effectively doubling throughput potential in this test.
Additional Recommendations
For anyone exploring similar setups, I strongly recommend integrating monitoring tools such as Grafana and Prometheus. These tools greatly simplify the process of diagnosing performance bottlenecks and crashes, providing detailed insights into GPU and system resource utilization.
Final Thoughts
While this experiment was primarily a proof of concept rather than a production-grade deployment, it demonstrates that with careful configuration, high-precision, lossless models like DeepSeek-V4-Flash-0731 can be operated effectively on robust dual GPU systems. The approach balances model fidelity with hardware constraints, paving the way for more ambitious applications such as in-depth code analysis, research, and complex problem solving.
Your questions and comments are welcome. As this is my first blog post on this topic, I appreciate your kindness and feedback. Happy experimenting!
