Fix rdmap message to be splited into two segments on receiver #45
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
I'm not convinced that this is the correct fix. The mtu value that this function is using should already take the header length into account. |
ZiYang-oyxy
force-pushed
the
fix-rdma-write
branch
from
6d64f51 to
e4b55cf
Compare
【BUG DESCRIPTION】
Testcase `ib_write_lat -d urdma_0 -i 1 -x 0 -R -n 5 -s 976` will hang if
run with 82599 NIC.
On the receiver, `ddp_place_tagged_data` can only handle one mbuf
segment, but the rdmap message will be splited into two segments by
the 82599 port.
Dump tx mbuf on sender:
```txt
USER1: <dev=0 qp=2> [TX]
dump mbuf at 0x7002e7ecd800, iova=1d9d8cd8c0, buf_len=1654
pkt_len=1042, ol_flags=f0000000000000, nb_segs=2, in_port=65535
segment at 0x7002e7ecd800, data=0x7002e7ecd916, data_len=52
Dump data at [0x7002e7ecd916], len=52
00000000: 70 10 6F AE CD B5 70 10 6F AE CD 9D 08 00 45 00 | p.o...p.o.....E.
00000010: 04 04 00 00 00 00 40 11 00 00 C0 A7 01 03 C0 A7 | ......@.........
00000020: 01 04 F8 44 AF DD 03 F0 87 57 00 00 00 01 00 00 | ...D.....W......
00000030: 00 00 00 00 | | | | | | | | | | | | | ....
segment at 0x7002e7ecdfc0, data=0x7002e7ecd172, data_len=990
Dump data at [0x7002e7ecd172], len=990
00000000: C1 40 00 65 49 DD 00 00 00 00 00 69 44 00 00 00 | .@.eI......iD...
00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
.
.
.
000003B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
000003C0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
000003D0: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 | | | ..............
```
Dump rx mbuf on receiver:
```txt
USER1: <dev=0 qp=2> [RX]
dump mbuf at 0x7002e9004500, iova=1f2d204580, buf_len=1654
pkt_len=1042, ol_flags=180, nb_segs=2, in_port=0
segment at 0x7002e9004500, data=0x7002e9004600, data_len=1024
Dump data at [0x7002e9004600], len=1024
00000000: 70 10 6F AE CD B5 70 10 6F AE CD 9D 08 00 45 00 | p.o...p.o.....E.
00000010: 04 04 00 00 00 00 40 11 F3 93 C0 A7 01 03 C0 A7 | ......@.........
00000020: 01 04 F8 44 AF DD 03 F0 7C A7 00 00 00 01 00 00 | ...D....|.......
00000030: 00 00 00 00 C1 40 00 65 49 DD 00 00 00 00 00 69 | .....@.eI......i
00000040: 44 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | D...............
00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
.
.
.
000003D0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
000003E0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
000003F0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
segment at 0x7002e9004c40, data=0x7002e9004d40, data_len=18
Dump data at [0x7002e9004d40], len=18
00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00000010: 00 01 | | | | | | | | | | | | | | | ..
```
【REASON】
The `mbuf_size` will less than 2K if the MTU is 1500, than the ixgbe PMD
will set the rx buffer to 1KB. Refer to the code below:
```c
ixgbe_dev_rx_init(struct rte_eth_dev *dev)
{
...
/*
* Configure the RX buffer size in the BSIZEPACKET field of
* the SRRCTL register of the queue.
* The value is in 1 KB resolution. Valid values can be from
* 1 KB to 16 KB.
*/
buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) -
RTE_PKTMBUF_HEADROOM);
srrctl |= ((buf_size >> IXGBE_SRRCTL_BSIZEPKT_SHIFT) &
IXGBE_SRRCTL_BSIZEPKT_MASK);
IXGBE_WRITE_REG(hw, IXGBE_SRRCTL(rxq->reg_idx), srrctl);
...
}
```
【FIX】
Refer to the comment of `RTE_MBUF_DEFAULT_BUF_SIZE`, it's the recommended
minimal buffer length.
```c
/**
* Some NICs need at least 2KB buffer to RX standard Ethernet frame without
* splitting it into multiple segments.
* So, for mbufs that planned to be involved into RX/TX, the recommended
* minimal buffer length is 2KB + RTE_PKTMBUF_HEADROOM.
*/
(RTE_MBUF_DEFAULT_DATAROOM + RTE_PKTMBUF_HEADROOM)
```
Signed-off-by: ziyang <hh123okbb@gmail.com>
ZiYang-oyxy
force-pushed
the
fix-rdma-write
branch
from
e4b55cf to
bbaa3ab
Compare
ZiYang-oyxy
changed the title
Contributor
Author
|
The first commit is not correct, I have pushed a new commit to overwrite the old one |
patrickmacarthur
approved these changes
Aug 24, 2018
Collaborator
patrickmacarthur
left a comment
patrickmacarthur
left a comment
There was a problem hiding this comment.
Looks good to me.
Collaborator
|
@BernardMetzler @PepperJo Can you one of you give your approval on this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: ziyang hh123okbb@gmail.com