最近因为ai的需求在研究本地部署ds，所以就有了告诉互联的需求, 高速互联最主要的就是Infiniband技术。配置麻烦特此记录
内核驱动
Infiniband依赖Rdma和verbs组件，debian 13已经默认自带
sudo tee /etc/modules-load.d/infiniband-qlogic.conf > /dev/null <<EOF
iw_cm
ib_qib
rdmavt
ib_umad
ib_uverbs
ib_ipoib
rdma_ucm
rdma_cm
ib_cm
ib_core
EOF
sudo systemctl restart systemd-modules-load.service # 重载OpenSM
Infiniband要求必须有Subnet Manager来管理拓扑结构
sudo apt install opensm
sudo systemctl enable opensm
sudo systemctl start opensm管理工具
Debain 13的stable源工具过于老旧会出现无法开发udma（注意看自己的os是不是出现了大规模升级）
ps: 这是最大的坑
sudo tee /etc/apt/sources.list.d/sid.sources >/dev/null <<'EOF'
Types: deb
URIs: https://deb.debian.org/debian
Suites: sid
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
EOF
sudo tee /etc/apt/preferences.d/sid.pref >/dev/null <<'EOF'
Package: *
Pin: release n=sid
Pin-Priority: 50
EOF
sudo apt update
sudo apt-get install -t sid \
rdma-core \
infiniband-diags \
perftest \
iperf3 \
iproute2 \
iputils-ping \
sysstat \
libibumad3 \
libibmad5使用
sudo ibstat
CA 'ibp24s0' # 管理名
CA type: InfiniPath_QLE7340
Number of ports: 1
Firmware version:
Hardware version: 2
Node GUID: 0x00117500006ebf54
System image GUID: 0x00117500006ebf54
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 4660
LMC: 0
SM lid: 4660
Capability mask: 0x0761086a
Port GUID: 0x00117500006ebf54
Link layer: InfiniBand
sudo ibportstate -C ibp24s0 -P 1 -D 0 1 enable # 改成你自己的 CA *通过正常的网络方案想管理普通以太网一样管理端口就好了
测速
# 一侧
ib_write_bw -d you_ca_name -i 1 -s 1048576 -D 30
# 另一侧
ib_write_bw -d you_ca_name -i 1 -s 1048576 -D 30 you_ip
案例数据
最好不要使用iperf3之类的，性能拉胯跑不满
hanxi-cat@debian:~$ ib_write_bw -d ibp24s0 -i 1 -s 1048576 -D 30
WARNING: BW peak won't be measured in this run.
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : ibp24s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : OFF Using DDP : OFF
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1234 QPN 0x0005 PSN 0x21269a RKey 0x000100 VAddr 0x007fac41220000
remote address: LID 0x01 QPN 0x0005 PSN 0x5131fa RKey 0x000100 VAddr 0x007f87c9055000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
1048576 48076 0.00 3004.76 0.003005
---------------------------------------------------------------------------------------
hanxi-cat@MiWiFi-RD01-srv-2:~$ ib_write_bw -d ibp11s0 -i 1 -s 1048576 -D 30 192.168.4.1
WARNING: BW peak won't be measured in this run.
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : ibp11s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
WARNING: CPU is not PCIe relaxed ordering compliant.
WARNING: You should disable PCIe RO with `--disable_pcie_relaxed` for both server and client.
ibv_wr* API : OFF Using DDP : OFF
TX depth : 128
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x01 QPN 0x0005 PSN 0x5131fa RKey 0x000100 VAddr 0x007f87c9055000
remote address: LID 0x1234 QPN 0x0005 PSN 0x21269a RKey 0x000100 VAddr 0x007fac41220000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1200.000000 != 3396.129000. CPU Frequency is not max.
1048576 48076 0.00 3004.76 0.003005
---------------------------------------------------------------------------------------
hanxi-cat@MiWiFi-RD01-srv-2:~$
双端禁用cpu
hanxi-cat@debian:~$ ib_write_bw -d ibp24s0 -i 1 -s 1048576 -D 30
WARNING: BW peak won't be measured in this run.
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : ibp24s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : OFF Using DDP : OFF
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1234 QPN 0x0007 PSN 0x4fabb6 RKey 0x010200 VAddr 0x007f6ec0c6d000
remote address: LID 0x01 QPN 0x0007 PSN 0x24afac RKey 0x010200 VAddr 0x007fad54fd1000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
1048576 42001 0.00 2625.05 0.002625
---------------------------------------------------------------------------------------
hanxi-cat@debian:~$
hanxi-cat@MiWiFi-RD01-srv-2:~$ ib_write_bw -d ibp11s0 -i 1 -s 1048576 -D 30 --disable_pcie_relaxed 192.168.4.1
WARNING: BW peak won't be measured in this run.
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : ibp11s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF Lock-free : OFF
ibv_wr* API : OFF Using DDP : OFF
TX depth : 128
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x01 QPN 0x0007 PSN 0x24afac RKey 0x010200 VAddr 0x007fad54fd1000
remote address: LID 0x1234 QPN 0x0007 PSN 0x4fabb6 RKey 0x010200 VAddr 0x007f6ec0c6d000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1200.000000 != 2300.000000. CPU Frequency is not max.
1048576 42001 0.00 2625.05 0.002625
---------------------------------------------------------------------------------------
hanxi-cat@MiWiFi-RD01-srv-2:~$
目前没有出现 CPU frequency 冲突警告，但 PCIe relax order 仍然是 ON。没完成禁用 relaxed ordering 的对照测试。