介绍

Dynamo是英伟达开源的一款高吞吐、低延时的推理框架，支持在多节点多GPU分布式环境中为生成式AI提供推理服务。在多节点，多GPU的分布式环境中，如何协调这些分片、路由请求并共享 KV 缓存，才能让人感觉就像使用了一台加速器一样快？NVIDIA Dynamo 应运而生！

接下来，我将介绍在NVIDIA的RTX 4090 部署Dynamo，并在本地体验了一下LLM。

测试环境

 CPU ：25 核心
 内存：90 GB
 GPU ：NVIDIA GeForce RTX 4090, 1

部署安装

• 参考地址：https://github.com/ai-dynamo/dynamo

• 安装命令：

 apt-get update
 DEBIAN_FRONTEND=noninteractive apt-get install -yq python3-dev python3-pip python3-venv libucx0
 python3 -m venv venv
 source venv/bin/activate
 ​
 pip install "ai-dynamo[all]"

• 运行过程：

 安装成功：
 Successfully installed MarkupSafe-3.0.2 a2wsgi-1.10.10 ai-dynamo-0.3.0 ai-dynamo-runtime-0.3.0 ai-dynamo-vllm-0.8.4.post2 aiohappyeyeballs-2.6.1 aiohttp-3.12.13 aiosignal-1.3.2 aiosqlite-0.21.0 airportsdata-20250523 annotated-types-0.7.0 anyio-4.9.0 appdirs-1.4.4 asgiref-3.8.1 astor-0.8.1 attrs-25.3.0 bentoml-1.4.8 blake3-1.0.5 cachetools-5.5.2 cattrs-23.1.2 certifi-2025.6.15 charset_normalizer-3.4.2 circus-0.19.0 click-8.1.8 click-option-group-0.5.7 cloudpickle-3.1.1 compressed-tensors-0.9.3 cupy-cuda12x-13.4.1 deprecated-1.2.18 depyf-0.18.0 dill-0.4.0 diskcache-5.6.3 distro-1.9.0 dnspython-2.7.0 durationpy-0.10 einops-0.8.1 email-validator-2.2.0 fastapi-0.115.6 fastapi-cli-0.0.7 fastrlock-0.8.3 filelock-3.18.0 frozenlist-1.7.0 fs-2.4.16 fsspec-2025.5.1 gguf-0.17.1 google-auth-2.40.3 googleapis-common-protos-1.70.0 grpcio-1.73.0 h11-0.16.0 hf-xet-1.1.4 httpcore-1.0.9 httptools-0.6.4 httpx-0.28.1 httpx-ws-0.7.2 huggingface-hub-0.33.0 idna-3.10 importlib_metadata-8.0.0 iniconfig-2.1.0 interegular-0.3.3 jinja2-3.1.6 jiter-0.10.0 jsonschema-4.24.0 jsonschema-specifications-2025.4.1 kantoku-0.18.3 kubernetes-32.0.1 lark-1.2.2 llguidance-0.7.29 llvmlite-0.44.0 lm-format-enforcer-0.10.11 markdown-it-py-3.0.0 mdurl-0.1.2 mistral_common-1.6.2 mpmath-1.3.0 msgpack-1.1.1 msgspec-0.19.0 multidict-6.5.0 nats-py-2.10.0 nest_asyncio-1.6.0 networkx-3.5 ninja-1.11.1.4 nixl-0.3.0 numba-0.61.2 numpy-2.2.6 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-cusparselt-cu12-0.6.2 nvidia-ml-py-12.575.51 nvidia-nccl-cu12-2.21.5 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.4.127 oauthlib-3.3.1 openai-1.88.0 opencv-python-headless-4.11.0.86 opentelemetry-api-1.26.0 opentelemetry-exporter-otlp-1.26.0 opentelemetry-exporter-otlp-proto-common-1.26.0 opentelemetry-exporter-otlp-proto-grpc-1.26.0 opentelemetry-exporter-otlp-proto-http-1.26.0 opentelemetry-instrumentation-0.47b0 opentelemetry-instrumentation-aiohttp-client-0.47b0 opentelemetry-instrumentation-asgi-0.47b0 opentelemetry-proto-1.26.0 opentelemetry-sdk-1.26.0 opentelemetry-semantic-conventions-0.47b0 opentelemetry-semantic-conventions-ai-0.4.9 opentelemetry-util-http-0.47b0 outlines-0.1.11 outlines_core-0.1.26 packaging-25.0 partial-json-parser-0.2.1.1.post5 pathspec-0.12.1 pillow-11.2.1 pip-requirements-parser-32.0.1 pluggy-1.6.0 prometheus-client-0.22.1 prometheus-fastapi-instrumentator-7.1.0 prompt_toolkit-3.0.51 propcache-0.3.2 protobuf-4.25.8 psutil-7.0.0 py-cpuinfo-9.0.0 pyasn1-0.6.1 pyasn1-modules-0.4.2 pycountry-24.6.1 pydantic-2.10.6 pydantic-core-2.27.2 pygments-2.19.1 pyparsing-3.2.3 pytest-8.4.1 python-dateutil-2.9.0.post0 python-dotenv-1.1.0 python-json-logger-3.3.0 python-multipart-0.0.20 pyyaml-6.0.2 pyzmq-27.0.0 questionary-2.1.0 ray-2.47.1 referencing-0.36.2 regex-2024.11.6 requests-2.32.4 requests-oauthlib-2.0.0 rich-14.0.0 rich-toolkit-0.14.7 rpds-py-0.25.1 rsa-4.9.1 safetensors-0.5.3 schema-0.7.7 scipy-1.15.3 sentencepiece-0.2.0 shellingham-1.5.4 simple-di-0.1.5 six-1.17.0 sniffio-1.3.1 starlette-0.41.3 sympy-1.13.1 tiktoken-0.9.0 tokenizers-0.21.1 tomli-w-1.2.0 torch-2.6.0 torchaudio-2.6.0 torchvision-0.21.0 tornado-6.5.1 tqdm-4.67.1 transformers-4.52.4 triton-3.2.0 typer-0.16.0 types-psutil-7.0.0.20250218 typing-extensions-4.14.0 urllib3-2.5.0 uvicorn-0.34.3 uvloop-0.21.0 watchfiles-1.1.0 wcwidth-0.2.13 websocket-client-1.8.0 websockets-15.0.1 wrapt-1.17.2 wsproto-1.2.0 xformers-0.0.29.post2 xgrammar-0.1.18 yarl-1.20.1 zipp-3.23.0
 ​
 [notice] A new release of pip is available: 24.0 -> 25.1.1
 [notice] To update, run: pip install --upgrade pip

在本地运行推理模型

• 要在本地运行模型，可以使用dynamo run 运行一个hugging face托管的模型。dynamo run 运行mistralrs、sglang、vllm 和 tensorrtllm。

• 比如运行DeepSeek-R1-Distill-Qwen-1.5B

  • 首先使用hfd脚本下载模型，比如

     hfd deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --hf_username your_hr_name --hf_token your_hf_token

  • 使用llama.cpp工具生成guff文件

     python convert_hf_to_gguf.py /root/models/DeepSeek-R1-Distill-Qwen-1.5B

  • 最后，使用dynamo 运行guff文件

     dynamo run DeepSeek-R1-Distill-Qwen-1.5B-F16.gguf

测试

• 运行

   # dynamo run DeepSeek-R1-Distill-Qwen-1.5B-F16.gguf 
   2025-06-20T07:04:15.981Z  INFO dynamo_run: Using default engine: mistralrs. Use out=<engine> to specify one of echo_core, echo_full, mistralrs, llamacpp, sglang, trtllm, vllm
   2025-06-20T07:04:15.981Z  INFO dynamo_run: CPU mode. Rebuild with `--features cuda|metal|vulkan` for better performance
   2025-06-20T07:04:17.009Z  INFO dynamo_llm::gguf::gguf_tokenizer: GGUF tokenizer model is `gpt2`, kind: `Bpe`, num tokens: 151936, num added tokens: 0, num merges: 151387, num scores: 0
   general.architecture: qwen2
   general.basename: DeepSeek-R1-Distill-Qwen
   general.file_type: 1
   general.license: mit
   general.name: DeepSeek R1 Distill Qwen 1.5B
   general.quantization_version: 2
   general.size_label: 1.5B
   general.type: model
   qwen2.attention.head_count: 12
   qwen2.attention.head_count_kv: 2
   qwen2.attention.layer_norm_rms_epsilon: 0.000001
   qwen2.block_count: 28
   qwen2.context_length: 131072
   qwen2.embedding_length: 1536
   qwen2.feed_forward_length: 8960
   qwen2.rope.freq_base: 10000
   2025-06-20T07:04:33.525Z  WARN toktrie_hf_tokenizers: error: missing char: ｜ for "<｜User｜>"    
   2025-06-20T07:04:33.525Z  WARN toktrie_hf_tokenizers: error: missing char: ｜ for "<｜Assistant｜>"
   ? User › 

• 测试

   ? User › Who are you?
   Hmm, the user just asked “Who are you?” - a simple but fundamental question.
   ​
   Okay, this seems like a first-time user or someone testing my capabilities. The query is open-ended but friendly in tone. They might be curious about my identity, functionality, or just initiating conversation....
   ...
   ​

写在最后

本次分享主要是部署Dynamo，并进行简单的测试，对于使用docker、k8s部署，可以参考给的docs文档：Welcome to NVIDIA Dynamo — Dynamo 以及Prefill、Decode、KV等技术还需要进一步的研究。