一、tensorflow与cuda对应关系表
https://tensorflow.google.cn/install/source?hl=en#tested_build_configurations
参考：https://blog.csdn.net/FL1768317420/article/details/134840200 、
https://www.weiyeji.com/2025/wsl-tensorflow-gpu/
TensorFlow Version
Python Version
Compiler
Build Tools
cuDNN
CUDA
2.15.0
3.9-3.11
Clang 16.0.0
Bazel 6.1.0
8.8
12.2
2.14.0
3.9-3.11
Clang 16.0.0
Bazel 6.1.0
8.7
11.8
2.13.0
3.8-3.11
Clang 16.0.0
Bazel 5.3.0
8.6
11.8
2.12.0
3.8-3.11
GCC 9.3.1
Bazel 5.3.0
8.6
11.8
2.11.0
3.7-3.10
GCC 9.3.1
Bazel 5.3.0
8.1
11.2
2.10.0
3.7-3.10
GCC 9.3.1
Bazel 5.1.1
8.1
11.2
2.9.0
3.7-3.10
GCC 9.3.1
Bazel 5.0.0
8.1
11.2
2.8.0
3.7-3.10
GCC 7.3.1
Bazel 4.2.1
8.1
11.2
2.7.0
3.7-3.9
GCC 7.3.1
Bazel 3.7.2
8.1
11.2
2.6.0
3.6-3.9
GCC 7.3.1
Bazel 3.7.2
8.1
11.2
2.5.0
3.6-3.9
GCC 7.3.1
Bazel 3.7.2
8.1
11.2
2.4.0
3.6-3.8
GCC 7.3.1
Bazel 3.1.0
8.0
11.0
2.3.0
3.5-3.8
GCC 7.3.1
Bazel 3.1.0
7.6
10.1
二、在原镜像容器中做修改
以175.6.40.93:8196/volcanosh/ubuntu18-dist-mnist-tf-example:v3镜像为基础，做如下操作。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 # 创建容器test01 # 进入容器内
以下都是在容器test01内操作：
1 2 3 4 # 安装pip3
1 2 # 卸载旧版本TensorFlow（如果有）
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 # 安装cuda11.8 # 加载/root/.bashrc # 验证cuda版本
1 2 3 4 # 使用anaconda安装虚拟python3.9环境
1 2 # 安装TensorFlow 2.14.0（支持CUDA 11.8）
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 # 安装libcudnn # 安装cudnn # 解压下载的文件（下载地址：https://developer.nvidia.com/cudnn-archive） # 具体是 cuDNN 8.x - 1.x (December 2023 - August 2014) # 复制文件到 CUDA 安装目录 # 检查 cuDNN 版本 # define CUDNN_MAJOR 8 # define CUDNN_MINOR 7 # define CUDNN_PATCHLEVEL 0 # define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # 验证CUDA和cuDNN兼容性 # #
三、单个容器内验证与测试
最终的文件：gpu_dist_mnist-v6.py， 进行tensorflow分布式训练
在容器test01中，启动PS服务器：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 (py3.9) root@59cbbed5325b:/opt/tensorflow2.14.0# export TF_CONFIG='{"cluster": {"ps": ["localhost:2222"], "worker": ["localhost:2223"]}, "task": {"type": "ps", "index": 0}}' # 运行结果的最后几行：
同时在容器test01中的另一个窗口中启动Worker：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 (py3.9) root@59cbbed5325b:/opt/tensorflow2.14.0# export TF_CONFIG='{"cluster": {"ps": ["localhost:2222"], "worker": ["localhost:2223"]}, "task": {"type": "worker", "index": 0}}' # 运行结果的最后几行：
在运行过程中，可以在宿主机或容器test01中查看nvidia
GPU的使用情况，如下是在宿主机上查看：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # 可以看到有两个进程在使用NVIDIA GPU
四、将容器保存成镜像
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # 将容器test01保存成镜像175.6.40.93:8196/volcanosh/ubuntu18-dist-mnist-tf-example:v4 # 上述命令中-m参数的值写错了，其实是从tensorflow1.8.0升级到tensorflow2.14.0 # 可以看到v4镜像比v3镜像大了很多，主要是增加的文件在/usr目录与/root/anaconda3目录下。前者是CUDA、cuDNN相关文件，后者主要是由于安装虚拟python3.9环境、tensorflow2.14.0增加的文件 # 推送到自己的harbor仓库（需要用harbor仓库户名与密码） # 保存到本地 # 查看文件
五、使用volcano1.10.0进行任务调度
5.1 只使用CPU
tf-dist-mnist-example-cpu-v2-2000times-4presentation.yaml：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tensorflow-dist-mnist-cpu spec: minAvailable: 2 schedulerName: volcano plugins: env: []svc: []policies: - event: PodEvicted action: RestartJob tasks: - replicas: 1 name: ps template: spec: containers: - command: - sh - -c - | PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; PYTHONUNBUFFERED=1 python /var/tf_dist_mnist/dist_mnist.py --num_gpus=0 image: 175.6 .40 .93 :8196/volcanosh/volcanosh/dist-mnist-tf-example:0.0.5 name: tensorflow ports: - containerPort: 2222 name: tfjob-port resources: {}restartPolicy: Never - replicas: 1 name: worker policies: - event: TaskCompleted action: CompleteJob template: spec: containers: - command: - sh - -c - | PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export TF_CONFIG={\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}; python /var/tf_dist_mnist/dist_mnist.py --num_gpus=0 --train_steps=2000 --batch_size=10000 image: 175.6 .40 .93 :8196/volcanosh/volcanosh/dist-mnist-tf-example:0.0.5 name: tensorflow ports: - containerPort: 2222 name: tfjob-port resources: {}restartPolicy: Never
1 2 3 4 5 6 7 8 9 # 执行任务： # 查看pod
5.2 使用NVIDIA GPU
以下文件中使用到的镜像很大，有23G。
tf-dist-mnist-example-vgpu-v2-2000times-4presentation.yaml：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tensorflow-dist-mnist-vgpu-v2 spec: minAvailable: 2 schedulerName: volcano plugins: env: []svc: []policies: - event: PodEvicted action: RestartJob tasks: - replicas: 1 name: ps template: spec: containers: - command: - bash - -c - | source /root/.bashrc && __conda_setup="$('/root/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" && eval "$__conda_setup" &&conda activate py3.9; export PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export TF_CONFIG="{\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"ps\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}"; echo $PS_HOST; echo $WORKER_HOST; echo $TF_CONFIG; python /opt/tensorflow2.14.0/gpu_dist_mnist-v6.py image: 175.6 .40 .93 :8196/volcanosh/ubuntu18-dist-mnist-tf-example:v4 imagePullPolicy: IfNotPresent name: tensorflow ports: - containerPort: 2222 name: tfjob-port resources: limits: volcano.sh/gpu-number: 1 volcano.sh/gpu-memory: 3072 restartPolicy: Never tolerations: - key: "node.kubernetes.io/disk-pressure" operator: "Exists" effect: "NoSchedule" - replicas: 1 name: worker policies: - event: TaskCompleted action: CompleteJob template: spec: containers: - command: - bash - -c - | source /root/.bashrc && __conda_setup="$('/root/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" && eval "$__conda_setup" && conda activate py3.9; export PS_HOST=`cat /etc/volcano/ps.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export WORKER_HOST=`cat /etc/volcano/worker.host | sed 's/$/&:2222/g' | sed 's/^/"/;s/$/"/' | tr "\n" ","`; export TF_CONFIG="{\"cluster\":{\"ps\":[${PS_HOST}],\"worker\":[${WORKER_HOST}]},\"task\":{\"type\":\"worker\",\"index\":${VK_TASK_INDEX}},\"environment\":\"cloud\"}"; echo $PS_HOST; echo $WORKER_HOST; echo $TF_CONFIG; python /opt/tensorflow2.14.0/gpu_dist_mnist-v6.py --train_steps=2000 --batch_size=10000 image: 175.6 .40 .93 :8196/volcanosh/ubuntu18-dist-mnist-tf-example:v4 imagePullPolicy: IfNotPresent name: tensorflow ports: - containerPort: 2222 name: tfjob-port resources: limits: volcano.sh/gpu-number: 1 volcano.sh/gpu-memory: 3072 restartPolicy: Never tolerations: - key: "node.kubernetes.io/disk-pressure" operator: "Exists" effect: "NoSchedule"
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 # 执行任务： # 查看pod # 在pod运行的k8s节点上，都可以看到一个使用NVIDIA GPU的进程