how to test RoCE driver
-v0.1 2018..27 Sherlock init
This document shares how to do a sanity test for a RoCE driver. If you are a
new guy for RoCE, it is for you.
Here is what I had done to test RoCE driver:
I had a ARM64 based machine installed a Redhat system in it.
download perftest, this is a open source RoCE test cases:
git clone https://github.com/lsgunth/perftest.git
download RoCE’s user space library, rdma-core:
git clone https://github.com/linux-rdma/rdma-core.git
download kernel which includes the RoCE driver to test.
build perftest tools followed by the README.
Here in a Redhat(CentOS) system, when you did ./configure, it maybe show
that you lack ibverbs head files. you can install rdma-core-devel to solve it:yum install rdma-core-devel.aarch64
if it is successful, you will get ib_send_bw …. tools in perftest.
build rdma-core followed by its README. In a Redhat(CentOS) system, maybe you
need:yum install libnl-devel.aarch64
if it is successful, you will get user space library under rdma-core/build/lib
Copy the libraries under rdma-core/build/lib/* to /lib64 in your system.
(I also did the test in ubuntu, you should copy the libraries to /lib)build your kernel together with your RoCE drivers.
In my case, RoCE driver will be built into some kernel modules.install modules: make modules_install
install kernel image: make install
(before build kernel, you can add a localversion file to help to tell your kernel)reboot system, then boot up using your built kernel.
in my case, RoCE kernel can not be loaded automatically, so I need to do:
modprobe hns-roce-hw-v2
then you can find your roce device in /sys/class/infiniband. Or use the tools
in rdma-core/build/bin: ibv_devices, ibv_devinfo to see your RoCE devices.up your RoCE’s networking interface, and keep its link status as: yes
you can find RoCE’s networking interface by searching, e.g.:/sys/class/infiniband/hns_0/ports/1/gid_attrs/ndevs
in a redhat system, you need do:
echo “driver hns” > /etc/libibverbs.d/hns.driver
but in a ubuntu system, you need do:
echo “driver hns” > /usr/local/etc/libibverbs.d/hns.driver
finally you can use ib_send_bw, ib_read_bw, ib_write_bw… to do the sanity
test. Every time you do the test, you should create a RoCE server and create
a RoCE client to access and send date to the server.you can do as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171[root@localhost perftest]# ./ib_write_bw -n 5 -d hns_0 &
[1] 10352
[root@localhost perftest]#
************************************
* Waiting for client to connect... *
************************************
[root@localhost perftest]# ./ib_write_bw -n 5 -d hns_0 192.168.2.198
---------------------------------------------------------------------------------------
RDMA_Write BW Test
---------------------------------------------------------------------------------------
Dual-port : OFF Device : hns_0
RDMA_Write BW Test
Number of qps : 1 Transport type : IB
Dual-port : OFF Device : hns_0
Connection type : RC Using SRQ : OFF
Number of qps : 1 Transport type : IB
CQ Moderation : 5
Connection type : RC Using SRQ : OFF
Mtu : 1024[B]
TX depth : 5
Link type : Ethernet
CQ Moderation : 5
Gid index : 0
Mtu : 1024[B]
Max inline data : 0[B]
Link type : Ethernet
rdma_cm QPs : OFF
Gid index : 0
Data ex. method : Ethernet
Max inline data : 0[B]
---------------------------------------------------------------------------------------
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
hr_qp->port_num= 0x1
hr_qp->port_num= 0x1
local address: LID 0000 QPN 0x0022 PSN 0x16566e RKey 0x000300 VAddr 0x00ffff93c9b000
local address: LID 0000 QPN 0x0023 PSN 0x1fd9e RKey 0x000400 VAddr 0x00ffffbe24d000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
remote address: LID 0000 QPN 0x0023 PSN 0x1fd9e RKey 0x000400 VAddr 0x00ffffbe24d000
remote address: LID 0000 QPN 0x0022 PSN 0x16566e RKey 0x000300 VAddr 0x00ffff93c9b000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 5 8169.77 8161.23 0.130580
---------------------------------------------------------------------------------------
65536 5 8169.77 8161.23 0.130580
---------------------------------------------------------------------------------------
[1]+ Done ./ib_write_bw -n 5 -d hns_0
[root@localhost perftest]# ./ib_read_bw -n 5 -d hns_0 &
[1] 10339
[root@localhost perftest]# ---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
************************************
* Waiting for client to connect... *
************************************
[root@localhost perftest]# ./ib_read_bw -n 5 -d hns_0 192.168.2.198
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
RDMA_Read BW Test
RDMA_Read BW Test
Dual-port : OFF Device : hns_0
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
Connection type : RC Using SRQ : OFF
CQ Moderation : 5
TX depth : 5
Mtu : 1024[B]
CQ Moderation : 5
Link type : Ethernet
Mtu : 1024[B]
Gid index : 0
Link type : Ethernet
Outstand reads : 128
Gid index : 0
rdma_cm QPs : OFF
Outstand reads : 128
Data ex. method : Ethernet
rdma_cm QPs : OFF
---------------------------------------------------------------------------------------
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
hr_qp->port_num= 0x1
hr_qp->port_num= 0x1
local address: LID 0000 QPN 0x0020 PSN 0xe6d890 OUT 0x80 RKey 0x000300 VAddr 0x00ffffb0250000
local address: LID 0000 QPN 0x0021 PSN 0x87bda4 OUT 0x80 RKey 0x000400 VAddr 0x00ffff9dbe8000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
remote address: LID 0000 QPN 0x0020 PSN 0xe6d890 OUT 0x80 RKey 0x000300 VAddr 0x00ffffb0250000
remote address: LID 0000 QPN 0x0021 PSN 0x87bda4 OUT 0x80 RKey 0x000400 VAddr 0x00ffff9dbe8000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 5 8002.35 8002.35 0.128038
---------------------------------------------------------------------------------------
65536 5 8002.35 8002.35 0.128038
---------------------------------------------------------------------------------------
[1]+ Done ./ib_read_bw -n 5 -d hns_0
[root@localhost perftest]# ./ib_send_bw -n 5 -d hns_0 &
[1] 10300
[root@localhost perftest]#
************************************
* Waiting for client to connect... *
************************************
[root@localhost perftest]# ./ib_send_bw -n 5 -d hns_0 192.168.2.198
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
Send BW Test
Send BW Test
Dual-port : OFF Device : hns_0
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
Connection type : RC Using SRQ : OFF
RX depth : 6
TX depth : 5
CQ Moderation : 5
CQ Moderation : 5
Mtu : 1024[B]
Mtu : 1024[B]
Link type : Ethernet
Link type : Ethernet
Gid index : 0
Gid index : 0
Max inline data : 0[B]
Max inline data : 0[B]
rdma_cm QPs : OFF
rdma_cm QPs : OFF
Data ex. method : Ethernet
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
hr_qp->port_num= 0x1
hr_qp->port_num= 0x1
local address: LID 0000 QPN 0x001e PSN 0xab3cfa
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
local address: LID 0000 QPN 0x001f PSN 0xdefae7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
remote address: LID 0000 QPN 0x001f PSN 0xdefae7
remote address: LID 0000 QPN 0x001e PSN 0xab3cfa
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:02:198
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 5 8054.00 8051.92 0.128831
---------------------------------------------------------------------------------------
65536 5 0.00 15340.76 0.245452
---------------------------------------------------------------------------------------
[1]+ Done ./ib_send_bw -n 5 -d hns_0
