当前位置: 首页 > news >正文

K8S部署Openwebui 服务(Nvidia版)

K8S部署BOBAI 服务(Nvidia版)

目录

  • 一、GPU 节点部署 Driver && CUDA部署
    • 1、前提准备
      • 检查机器上面有支持CUDA的NVIDIA GPU
      • 查看自己的系统是否支持
      • 验证系统是否有GCC编译环境
      • 验证系统是否安装了正确的内核头文件和开发
    • 2、开始安装
      • 禁用nouveau
      • 开始安装驱动NVIDIA Driver (也可跳过,直接去安装CUDA)
      • 开始安装CUDA Toolkit
  • 二、容器环境(Docker or Containerd)
    • 1、安装 nvidia-container-toolkit
      • 说明:
      • With dnf: RHEL/CentOS, Fedora, Amazon Linux
    • 2、配置Runtime 为NVIDIA
      • Docker
      • Containerd
    • 三、K8S调用GPU
      • 说明:
      • 部署Plugin
  • 三、部署服务
    • 1、部署Deekseek-v3
      • 示例yaml文件如下,仅供参考
    • 2、qwen-embedding模型部署
      • 示例yaml文件如下,仅供参考
    • 3、Tika部署
      • 示例yaml文件如下,仅供参考
    • 4、部署ASR
      • 示例yaml文件如下,仅供参考

一、GPU 节点部署 Driver && CUDA部署

官方安装文档

1、前提准备

检查机器上面有支持CUDA的NVIDIA GPU

 lspci | grep -i nvidia

查看自己的系统是否支持

CUDA Installation Guide for Linux — Installation Guide for Linux 13.0 documentation The installation instructions for the CUDA Toolkit on Linux. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#system-requirements

验证系统是否有GCC编译环境

gcc -v

验证系统是否安装了正确的内核头文件和开发

sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r)

2、开始安装

禁用nouveau

nouveau是一个第三方开源的Nvidia驱动,一般Linux安装的时候默认会安装这个驱动。 这个驱动会与Nvidia官方的驱动冲突,在安装Nvidia驱动和和CUDA之前应先禁用nouveau。

# 查看系统是否正在使用nouveau
lsmod | grep nouveau# 如果显示内容,则禁用。以下是centos7的禁用方法#新建一个配置文件
sudo vim /etc/modprobe.d/blacklist-nouveau.conf
#写入以下内容
blacklist nouveau
options nouveau modeset=0
#保存并退出
:wq
#备份当前的镜像
sudo mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
#建立新的镜像
sudo dracut /boot/initramfs-$(uname -r).img $(uname -r)
#重启
sudo reboot
#最后输入上面的命令验证
lsmod | grep nouveau

开始安装驱动NVIDIA Driver (也可跳过,直接去安装CUDA)

📌一定要确认NVIDIA Driver的版本适合自己的显卡

  • 下载NVIDIA Driver

    首先到 NVIDIA 驱动下载 下载对应的显卡驱动:

    在下载前确认Driver是否支持自己的显卡

  • 安装NVIDIA Driver

    rpm -ivh nvidia-driver-local-repo-rhel9-580.82.07-1.0-1.x86_64.rpm
    
  • 验证驱动是否安装成功

    # 执行如下命令
    root@GPU1:~ nvidia-smi
    +---------------------------------------------------------------------------------------+
    | NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
    |-----------------------------------------+----------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
    |                                         |                      |               MIG M. |
    |=========================================+======================+======================|
    |   0  Tesla T4                       On  | 00000000:3B:00.0 Off |                    0 |
    | N/A   51C    P0              29W /  70W |  12233MiB / 15360MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+
    |   1  Tesla T4                       On  | 00000000:86:00.0 Off |                    0 |
    | N/A   49C    P0              30W /  70W |   6017MiB / 15360MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------++---------------------------------------------------------------------------------------+
    | Processes:                                                                            |
    |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
    |        ID   ID                                                             Usage      |
    |=======================================================================================|
    |   
    +---------------------------------------------------------------------------------------
    

    到这里我们的GPU的驱动就安装好了,系统也可以正常的识别到GPU了。这里显示的CUDA Version指的是当前驱动最大支持的CUDA版本。

开始安装CUDA Toolkit

  • 下载CUDA Toolkit

    首先下载CUDA Toolkit

    选择好自己的系统和版本

    建议下载.run文件

  • 安装CUDA Toolkit

    wget https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda_13.0.0_580.65.06_linux.run
    sudo sh cuda_13.0.0_580.65.06_linux.run# 安装成功的日志示例
    ===========
    = Summary =
    ===========Driver:   Installed
    Toolkit:  Installed in /usr/local/cuda-13.0/Please make sure that-   PATH includes /usr/local/cuda-13.0/bin-   LD_LIBRARY_PATH includes /usr/local/cuda-13.0/lib64, or, add /usr/local/cuda-13.0/lib64 to /etc/ld.so.conf and run ldconfig as rootTo uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-13.0/bin
    To uninstall the NVIDIA Driver, run nvidia-uninstall
    Logfile is /var/log/cuda-installer.log

    下载并安装即可,其实如果你没有安装NVIDIA Driver这一步会帮你安装好适合你的NVIDIA Driver,比如这个会安装580.65.06版本的Driver。

  • 配置环境变量

    vim /etc/profile.d/cuda.sh# 编辑一个新文件,内容如下:# 添加 CUDA 13.0 到 PATH
    export PATH=/usr/local/cuda-13.0/bin:$PATH# 添加 CUDA 13.0的 lib64 到 LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:$LD_LIBRARY_PATH
    

    保存,刷新配置文件

    source /etc/profile.d/cuda.sh

    
    检查是否部署成功
    ```markdown 
    # 如果输出版本号即为成功
    (base) root@Colourdata-GPU:~# nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0

二、容器环境(Docker or Containerd)

官方地址

1、安装 nvidia-container-toolkit

说明:

NVIDIA Container Toolkit 的主要作用是将 NVIDIA GPU 设备挂载到容器中。兼容docker、containerd、cri-o等。

With dnf: RHEL/CentOS, Fedora, Amazon Linux

# 配置生产存储库
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo# 安装NVIDIA Container Toolkit 软件包
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1sudo dnf install -y \nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}

2、配置Runtime 为NVIDIA

Docker

# 配置runtime=doker
sudo nvidia-ctk runtime configure --runtime=docker# 建议在 /etc/docker/daemon.json 里面检查一下,并将默认runtime也修改为nvidia
(base) root@Colourdata-GPU:~# vim  /etc/docker/daemon.json
{"registry-mirrors": ["https://ihsxva0f.mirror.aliyuncs.com","https://docker.m.daocloud.io","https://registry.docker-cn.com"],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "10m","max-file": "3"},"storage-driver": "overlay2","default-runtime": "nvidia","runtimes": {"nvidia": {"args": [],"path": "nvidia-container-runtime"}}
}systemctl daemon-relod 
systemctl restart docker

Containerd

# 配置runtime=containerd
sudo nvidia-ctk runtime configure --runtime=containerd# 建议在 /etc/containerd/config.toml 里面检查一下,并将默认runtime也修改为nvidia
# 修改好后重启containerd
sudo systemctl restart containerd

以上部署完成,就可以配置K8S调用GPU了。


三、K8S调用GPU

说明:

device-plugin由NVIDIA提供,官网文档。

部署Plugin

  • 建议先给GPU节点打上标签 gpu=true

  • 部署服务

    # 下载地址,建议选择新版本
    https://github.com/NVIDIA/k8s-device-plugin/blob/main/deployments/static/nvidia-device-plugin.yml# 发布服务
    root@test:~# kubectl apply -f nvidia-device-plugin.ymlroot@test:~# kubectl get po -l app=gpu -n bobai
    NAME                                   READY   STATUS    RESTARTS   AGE
    nvidia-device-plugin-daemonset-7nkjw   1/1     Running   0          10m
    
  • 检查服务是否部署成功

    # 如果可以看到nvidia gpu说明服务已经部署成功了
    root@test:~# kubectl describe node GPU | grep nvidia.com/gpunvidia.com/gpu:     2

以上部署完成后,你的K8S集群就可以调用GPU了。

三、部署服务

1、部署Deekseek-v3

示例yaml文件如下,仅供参考

apiVersion: apps/v1
kind: Deployment
metadata:name: deepseek-v3namespace: bobai
spec:replicas: 1selector:matchLabels:app: deepseek-v3template:metadata:labels:app: deepseek-v3spec:containers:- command:- sh- -c- vllm serve  --port 8000 --trust-remote-code --served-model-name deepseek-v3 --dtype=fp8  --max-model-len 65536 --gpu-memory-utilization 0.95 /models/DeepSeek-V3  name: deepseek-v3image: registry.cn-shanghai.aliyuncs.com/colourdata/bobai-dependency:vllmimagePullPolicy: IfNotPresentports:- containerPort: 8000volumeMounts:- name: model-volumemountPath: /modelsresources:requests:nvidia.com/gpu: 8memory: "16Gi"cpu: "8"limits:nvidia.com/gpu: 8memory: "32Gi"cpu: "16"livenessProbe:tcpSocket:port: 8000initialDelaySeconds: 300periodSeconds: 10failureThreshold: 3readinessProbe:tcpSocket:port: 8000initialDelaySeconds: 300periodSeconds: 10failureThreshold: 3volumes:- name: model-volumehostPath:path: /modelstype: Directory

2、qwen-embedding模型部署

示例yaml文件如下,仅供参考

apiVersion: apps/v1
kind: Deployment
metadata:name: vllm-embeddingnamespace: bobai
spec:replicas: 1selector:matchLabels:app: vllm-embeddingtemplate:metadata:labels:app: vllm-embeddingspec:containers:- command:- sh- -c- vllm serve  --port 8000 --trust-remote-code --served-model-name vllm-embedding  --max-model-len 4096 --gpu-memory-utilization 0.85 /models/Qwen3-Embedding-0.6Bname: vllm-embeddingimage: registry.cn-shanghai.aliyuncs.com/colourdata/bobai-dependency:vllmimagePullPolicy: IfNotPresentports:- containerPort: 8000volumeMounts:- name: model-volumemountPath: /modelsresources:limits:nvidia.com/gpu: 1requests:memory: "8Gi"cpu: "4"limits:memory: "16Gi"cpu: "8"livenessProbe:tcpSocket:port: 8000initialDelaySeconds: 30periodSeconds: 10failureThreshold: 3readinessProbe:tcpSocket:port: 8000initialDelaySeconds: 10periodSeconds: 5failureThreshold: 3volumes:- name: model-volumenfs:server: 192.168.2.250path: /data/bobai/modelsrestartPolicy: Always---
apiVersion: v1
kind: Service
metadata:name: vllm-embeddingnamespace: bobai
spec:type: ClusterIPports:- port: 8000protocol: TCPtargetPort: 8000selector:app: vllm-embedding

3、Tika部署

示例yaml文件如下,仅供参考

apiVersion: apps/v1
kind: Deployment
metadata:name: tikanamespace: bobai
spec:replicas: 1selector:matchLabels:app: tikatemplate:metadata:labels:app: tikaspec:containers:- name: tikaimage: tika-ocr-cn:v1imagePullPolicy: IfNotPresentports:- containerPort: 9998resources:requests:memory: "1Gi"cpu: "500m"limits:memory: "2Gi"cpu: "1"livenessProbe:tcpSocket:port: 9998initialDelaySeconds: 30periodSeconds: 10 failureThreshold: 3readinessProbe:tcpSocket:        port: 9998initialDelaySeconds: 10periodSeconds: 5failureThreshold: 3restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:name: tika-servicenamespace: bobai
spec:type: ClusterIPselector:app: tikaports:- protocol: TCPport: 9998targetPort: 9998

4、部署ASR

示例yaml文件如下,仅供参考

apiVersion: apps/v1
kind: Deployment
metadata:name: openai-edge-ttsnamespace: bobai
spec:replicas: 1selector:matchLabels:app: openai-edge-ttstemplate:metadata:labels:app: openai-edge-ttsspec:containers:- name: openai-edge-ttsimage: travisvn/openai-edge-tts:latestimagePullPolicy: IfNotPresentports:- containerPort: 5050env:- name: API_KEYvalue: "Colourdata1234@"- name: PORTvalue: "5050"- name: DEFAULT_VOICEvalue: "en-US-AvaNeural"- name: DEFAULT_RESPONSE_FORMATvalue: "mp3"- name: DEFAULT_SPEEDvalue: "1.0"- name: DEFAULT_LANGUAGEvalue: "en-US"- name: REQUIRE_API_KEYvalue: "True"- name: REMOVE_FILTERvalue: "False"- name: EXPAND_APIvalue: "True"resources:requests:memory: "512Mi"cpu: "500m"limits:memory: "1Gi"cpu: "1"livenessProbe:tcpSocket:port: 5050initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 3readinessProbe:tcpSocket:port: 5050initialDelaySeconds: 10 periodSeconds: 5  failureThreshold: 3     restartPolicy: Always---
apiVersion: v1
kind: Service
metadata:name: openai-edge-tts-servicenamespace: bobai
spec:type: ClusterIPselector:app: openai-edge-ttsports:- protocol: TCPport: 5050targetPort: 5050      
http://www.hskmm.com/?act=detail&tid=19448

相关文章:

  • 传统AI对话:悟空也辛苦(ai元人文)
  • 理解大语言模型中的 Token
  • 软件工程第一次团队作业
  • 实验1作业
  • 苍穹外卖-day01(软件开发整体介绍,苍穹外卖项目介绍,开发环境搭建,导入接口文档,Swagger) - a
  • 9.27动手动脑及课后实验
  • idea必备插件
  • 新学期每日总结(第6天)
  • 第六天
  • Combinatorics
  • 绘制倒杨辉三角形
  • ABC425 总结
  • 解决方案 | 无需安装任何插件,chrome如何快速搜索书签
  • 订单模块逐字稿
  • 课后作业小结
  • 课后3
  • 尝试决定
  • 竞赛第一步----进实验室
  • Java语法基础课程动手动脑与实验问题深度解析
  • lc1038-从二叉搜索树到更大和树
  • 课程中的问题
  • 课后2
  • Java语法基础课程“动手动脑”问题与实验整理
  • 课后感想
  • mysql的单表如何仅保留半年的数据
  • Java基础核心问题 链接版
  • java作业
  • Insightly存储型XSS漏洞分析:通过链接名称注入恶意脚本
  • H3C交换机的配置学习-01
  • Python脚本生成包含标准的#ifndef保护宏的头文件