perftest是一组基于uverbs编写的测试程序，是RDMA性能相关的benchmark。可用于软硬件调优以及功能测试。

这个工具在此前的笔记中有记录并简单使用过，在 https://sulao.cn/post/979 这篇笔记记录，方便后续继续测试的时候能够深入了解其参数和使用特性，所以这次专门开了一篇来记录此工具集使用的详细操作。

perftest安装比较简单，在centos和ubuntu上均可直接在线安装

yum install perftest #centos
apt install perftest #ubuntu

perftest测试集包含以下各个测试工具

ib_send_lat：发送转换延迟测试
ib_send_bw：带宽测试（含发送转换）
ib_write_lat：使用 RDMA 写事务进行延迟测试
ib_write_bw：使用 RDMA 写事务进行的带宽测试
ib_read_lat：使用 RDMA 读取事务进行的延迟测试
ib_read_bw：使用 RDMA 读取事务进行的带宽测试
ib_atomic_lat：带有原子事务的延迟测试
ib_atomic_bw：使用原子事务进行带宽测试

测试需要开一个server，然后使用client进行测试，测试中IP均换成自己的IP地址。

接着我们就来进行相关的内容进行测试，注意如果是RoCE网络我们在下面测试中都需要添加-x参数指定gid的index，同时client端测试的连接IP，可以在server端使用ibvdev2netdev命令查询绑定的网络接口IP地址。

1.双向发送带宽测试

ib_send_bw -h
Usage:
ib_send_bw start a server and wait for connection
ib_send_bw <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-b, --bidirectional Measure bidirectional bandwidth (default unidirectional)
-c, --connection=<RC/XRC/UC/UD/DC/SRD> Connection type RC/XRC/UC/UD/DC/SRD (default RC)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-e, --events Sleep on CQ events (default poll)
-X, --vector=<completion vector> Set <completion vector> used for events
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-g, --mcg Send messages to multicast group with 1 QP attached to it.
-h, --help Show this help screen.
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-I, --inline_size=<size> Max size of message to be sent in inline
-l, --post_list=<list size>

server端我们执行

ib_send_bw -a -b -d mlx5_1 -i 1

client端执行

ib_send_bw -a -b -d mlx5_1 -i 1 10.10.10.2

我们可以看到测试结果在23G多

2.发送延迟测试

ib_send_lat -h
Usage:
ib_send_lat start a server and wait for connection
ib_send_lat <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-c, --connection=<RC/XRC/UC/UD/DC/SRD> Connection type RC/XRC/UC/UD/DC/SRD (default RC)
-C, --report-cycles report times in cpu cycle units (default microseconds)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-e, --events Sleep on CQ events (default poll)
-X, --vector=<completion vector> Set <completion vector> used for events
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-g, --mcg Send messages to multicast group with 1 QP attached to it.
-h, --help Show this help screen.
-H, --report-histogram Print out all results (default print summary only)
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-I, --inline_size=<size> Max size of message to be sent in inline
-L, --hop_limit=<hop_limit> Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64)
-m, --mtu=<mtu> MTU size : 256 - 4096 (default port mtu)
-M, --MGID=<multicast_gid> In multicast, uses <multicast_gid> as the group MGID.
-n, --iters=<iters> Number of exchanges (at least 5, default 1000)
-p, --port=<port> Listen on/connect to port <port> (default 18515)
-r, --rx-depth=<dep> Rx queue size (default 512). If using srq, rx-depth controls max-wr size of the srq
-R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs
-s, --size=<size> Size of message to exchange (default 2)
-S, --sl=<sl> SL (default 0)
-T, --tos=<tos value> Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off)
-u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14
-U, --report-unsorted (implies -H) print out unsorted results (default sorted)
-V, --version Display version number
-W, --report-counters=<list of counter names> Report performance counter change (example: "counters/port_xmit_data,hw_counters/out_of_buffer")
-x, --gid-index=<index> Test uses GID with GID index (Default : IB - no gid . ETH - 0)
-z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs

在server端我们执行

ib_send_lat -a -d mlx5_1 -i 1

在client端执行

ib_send_lat -a -d mlx5_1 -i 1 10.10.10.2

3.双向写入带宽测试

ib_write_bw -h
Usage:
ib_write_bw start a server and wait for connection
ib_write_bw <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-b, --bidirectional Measure bidirectional bandwidth (default unidirectional)
-c, --connection=<RC/XRC/UC/DC> Connection type RC/XRC/UC/DC (default RC)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-h, --help Show this help screen.
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-I, --inline_size=<size> Max size of message to be sent in inline
-l, --post_list=<list size>
Post list of send WQEs of <list size> size (instead of single post)
--recv_post_list=<list size> Post list of receive WQEs of <list size> size (instead of single post)
-L, --hop_limit=<hop_limit> Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64)
-m, --mtu=<mtu> MTU size : 256 - 4096 (default port mtu)
-n, --iters=<iters> Number of exchanges (at least 5, default 5000)
-N, --noPeak Cancel peak-bw calculation (default with peak up to iters=20000)
-O, --dualport Run test in dual-port mode.
-p, --port=<port> Listen on/connect to port <port> (default 18515)
-q, --qp=<num of qp's> Num of qp's(default 1)
-Q, --cq-mod Generate Cqe only after <--cq-mod> completion
-R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs
-s, --size=<size> Size of message to exchange (default 65536)
-S, --sl=<sl> SL (default 0)
-t, --tx-depth=<dep> Size of tx queue (default 128)
-T, --tos=<tos value> Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off)
-u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14
-V, --version Display version number
-w, --limit_bw=<value> Set verifier limit for bandwidth
-W, --report-counters=<list of counter names> Report performance counter change (example: "counters/port_xmit_data,hw_counters/out_of_buffer")
-x, --gid-index=<index> Test uses GID with GID index (Default : IB - no gid . ETH - 0)
-y, --limit_msgrate=<value> Set verifier limit for Msg Rate
-z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs

双向写带宽命令参数和ib_send_bw一致，这里我就不在一一测试了。

4.写入延迟测试

ib_write_lat -h
Usage:
ib_write_lat start a server and wait for connection
ib_write_lat <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-c, --connection=<RC/XRC/UC/DC> Connection type RC/XRC/UC/DC (default RC)
-C, --report-cycles report times in cpu cycle units (default microseconds)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-h, --help Show this help screen.
-H, --report-histogram Print out all results (default print summary only)
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-I, --inline_size=<size> Max size of message to be sent in inline
-L, --hop_limit=<hop_limit> Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64)
-m, --mtu=<mtu> MTU size : 256 - 4096 (default port mtu)
-n, --iters=<iters> Number of exchanges (at least 5, default 1000)
-p, --port=<port> Listen on/connect to port <port> (default 18515)
-R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs
-s, --size=<size> Size of message to exchange (default 2)
-S, --sl=<sl> SL (default 0)
-T, --tos=<tos value> Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off)
-u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14
-U, --report-unsorted (implies -H) print out unsorted results (default sorted)
-V, --version Display version number
-W, --report-counters=<list of counter names> Report performance counter change (example: "counters/port_xmit_data,hw_counters/out_of_buffer")
-x, --gid-index=<index> Test uses GID with GID index (Default : IB - no gid . ETH - 0)
-z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs

写延迟命令参数和ib_send_lat基本一致，这里就不再进行一一测试了。

5.双向读取带宽测试

ib_read_bw -h
Usage:
ib_read_bw start a server and wait for connection
ib_read_bw <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-b, --bidirectional Measure bidirectional bandwidth (default unidirectional)
-c, --connection=<RC/XRC/DC> Connection type RC/XRC/DC (default RC)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-e, --events Sleep on CQ events (default poll)
-X, --vector=<completion vector> Set <completion vector> used for events
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-h, --help Show this help screen.
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-l, --post_list=<list size>
Post list of send WQEs of <list size> size (instead of single post)
--recv_post_list=<list size> Post list of receive WQEs of <list size> size (instead of single post)
-L, --hop_limit=<hop_limit> Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64)
-m, --mtu=<mtu> MTU size : 256 - 4096 (default port mtu)
-n, --iters=<iters> Number of exchanges (at least 5, default 1000)
-N, --noPeak Cancel peak-bw calculation (default with peak up to iters=20000)
-o, --outs=<num> num of outstanding read/atom(default max of device)
-O, --dualport Run test in dual-port mode.
-p, --port=<port> Listen on/connect to port <port> (default 18515)
-q, --qp=<num of qp's> Num of qp's(default 1)
-Q, --cq-mod Generate Cqe only after <--cq-mod> completion
-R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs
-s, --size=<size> Size of message to exchange (default 65536)
-S, --sl=<sl> SL (default 0)
-t, --tx-depth=<dep> Size of tx queue (default 128)
-T, --tos=<tos value> Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off)
-u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14
-V, --version Display version number
-w, --limit_bw=<value> Set verifier limit for bandwidth
-W, --report-counters=<list of counter names> Report performance counter change (example: "counters/port_xmit_data,hw_counters/out_of_buffer")
-x, --gid-index=<index> Test uses GID with GID index (Default : IB - no gid . ETH - 0)
-y, --limit_msgrate=<value> Set verifier limit for Msg Rate
-z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs

测试命令参数也和ib_send_bw一致

6.读取延迟测试

ib_read_lat -h
Usage:
ib_read_lat start a server and wait for connection
ib_read_lat <host> connect to server at <host>

Options:
-a, --all Run sizes from 2 till 2^23
-c, --connection=<RC/XRC/DC> Connection type RC/XRC/DC (default RC)
-C, --report-cycles report times in cpu cycle units (default microseconds)
-d, --ib-dev=<dev> Use IB device <dev> (default first device found)
-D, --duration Run test for a customized period of seconds.
-e, --events Sleep on CQ events (default poll)
-X, --vector=<completion vector> Set <completion vector> used for events
-f, --margin measure results within margins. (default=2sec)
-F, --CPU-freq Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-h, --help Show this help screen.
-H, --report-histogram Print out all results (default print summary only)
-i, --ib-port=<port> Use port <port> of IB device (default 1)
-L, --hop_limit=<hop_limit> Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64)
-m, --mtu=<mtu> MTU size : 256 - 4096 (default port mtu)
-n, --iters=<iters> Number of exchanges (at least 5, default 1000)
-o, --outs=<num> num of outstanding read/atom(default max of device)
-p, --port=<port> Listen on/connect to port <port> (default 18515)
-R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs
-s, --size=<size> Size of message to exchange (default 2)
-S, --sl=<sl> SL (default 0)
-T, --tos=<tos value> Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off)
-u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14
-U, --report-unsorted (implies -H) print out unsorted results (default sorted)
-V, --version Display version number
-W, --report-counters=<list of counter names> Report performance counter change (example: "counters/port_xmit_data,hw_counters/out_of_buffer")
-x, --gid-index=<index> Test uses GID with GID index (Default : IB - no gid . ETH - 0)
-z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs

测试命令参数也是ib_send_lat一致。