修改senflow中每个QP对传输数据量后程序无法结束 #224
Description
SIMAI在SendFlow中在_QPS_PER_CONNECTION_个QP间更接近于均分，我尝试自定义了一下每个QP上的分配比例，保证了全部分完且不存在QP分配数据量为0.但还是会出现程序无法结束问题，打印数据后发现waiting_to_sent_callback.size()最后停留在106，而waiting_to_notify_receiver.size()已经为0。
以下是我的自定义拓扑，一个简单的spine-leaf网络，和分配比例文件（主机级，同一个server上的n个GPU使用同一个分配比例），dp=8，测试512MB的alltoall通信。
168 8 16 24 384 H100
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
160 161 162 163 164 165 166 167
0 128 2400Gbps 0.000025ms 0
0 144 400Gbps 0.0005ms 0
1 128 2400Gbps 0.000025ms 0
1 144 400Gbps 0.0005ms 0
2 128 2400Gbps 0.000025ms 0
2 144 400Gbps 0.0005ms 0
3 128 2400Gbps 0.000025ms 0
3 144 400Gbps 0.0005ms 0
4 128 2400Gbps 0.000025ms 0
4 144 400Gbps 0.0005ms 0
5 128 2400Gbps 0.000025ms 0
5 144 400Gbps 0.0005ms 0
6 128 2400Gbps 0.000025ms 0
6 144 400Gbps 0.0005ms 0
7 128 2400Gbps 0.000025ms 0
7 144 400Gbps 0.0005ms 0
8 129 2400Gbps 0.000025ms 0
8 145 400Gbps 0.0005ms 0
9 129 2400Gbps 0.000025ms 0
9 145 400Gbps 0.0005ms 0
10 129 2400Gbps 0.000025ms 0
10 145 400Gbps 0.0005ms 0
11 129 2400Gbps 0.000025ms 0
11 145 400Gbps 0.0005ms 0
12 129 2400Gbps 0.000025ms 0
12 145 400Gbps 0.0005ms 0
13 129 2400Gbps 0.000025ms 0
13 145 400Gbps 0.0005ms 0
14 129 2400Gbps 0.000025ms 0
14 145 400Gbps 0.0005ms 0
15 129 2400Gbps 0.000025ms 0
15 145 400Gbps 0.0005ms 0
16 130 2400Gbps 0.000025ms 0
16 146 400Gbps 0.0005ms 0
17 130 2400Gbps 0.000025ms 0
17 146 400Gbps 0.0005ms 0
18 130 2400Gbps 0.000025ms 0
18 146 400Gbps 0.0005ms 0
19 130 2400Gbps 0.000025ms 0
19 146 400Gbps 0.0005ms 0
20 130 2400Gbps 0.000025ms 0
20 146 400Gbps 0.0005ms 0
21 130 2400Gbps 0.000025ms 0
21 146 400Gbps 0.0005ms 0
22 130 2400Gbps 0.000025ms 0
22 146 400Gbps 0.0005ms 0
23 130 2400Gbps 0.000025ms 0
23 146 400Gbps 0.0005ms 0
24 131 2400Gbps 0.000025ms 0
24 147 400Gbps 0.0005ms 0
25 131 2400Gbps 0.000025ms 0
25 147 400Gbps 0.0005ms 0
26 131 2400Gbps 0.000025ms 0
26 147 400Gbps 0.0005ms 0
27 131 2400Gbps 0.000025ms 0
27 147 400Gbps 0.0005ms 0
28 131 2400Gbps 0.000025ms 0
28 147 400Gbps 0.0005ms 0
29 131 2400Gbps 0.000025ms 0
29 147 400Gbps 0.0005ms 0
30 131 2400Gbps 0.000025ms 0
30 147 400Gbps 0.0005ms 0
31 131 2400Gbps 0.000025ms 0
31 147 400Gbps 0.0005ms 0
32 132 2400Gbps 0.000025ms 0
32 148 400Gbps 0.0005ms 0
33 132 2400Gbps 0.000025ms 0
33 148 400Gbps 0.0005ms 0
34 132 2400Gbps 0.000025ms 0
34 148 400Gbps 0.0005ms 0
35 132 2400Gbps 0.000025ms 0
35 148 400Gbps 0.0005ms 0
36 132 2400Gbps 0.000025ms 0
36 148 400Gbps 0.0005ms 0
37 132 2400Gbps 0.000025ms 0
37 148 400Gbps 0.0005ms 0
38 132 2400Gbps 0.000025ms 0
38 148 400Gbps 0.0005ms 0
39 132 2400Gbps 0.000025ms 0
39 148 400Gbps 0.0005ms 0
40 133 2400Gbps 0.000025ms 0
40 149 400Gbps 0.0005ms 0
41 133 2400Gbps 0.000025ms 0
41 149 400Gbps 0.0005ms 0
42 133 2400Gbps 0.000025ms 0
42 149 400Gbps 0.0005ms 0
43 133 2400Gbps 0.000025ms 0
43 149 400Gbps 0.0005ms 0
44 133 2400Gbps 0.000025ms 0
44 149 400Gbps 0.0005ms 0
45 133 2400Gbps 0.000025ms 0
45 149 400Gbps 0.0005ms 0
46 133 2400Gbps 0.000025ms 0
46 149 400Gbps 0.0005ms 0
47 133 2400Gbps 0.000025ms 0
47 149 400Gbps 0.0005ms 0
48 134 2400Gbps 0.000025ms 0
48 150 400Gbps 0.0005ms 0
49 134 2400Gbps 0.000025ms 0
49 150 400Gbps 0.0005ms 0
50 134 2400Gbps 0.000025ms 0
50 150 400Gbps 0.0005ms 0
51 134 2400Gbps 0.000025ms 0
51 150 400Gbps 0.0005ms 0
52 134 2400Gbps 0.000025ms 0
52 150 400Gbps 0.0005ms 0
53 134 2400Gbps 0.000025ms 0
53 150 400Gbps 0.0005ms 0
54 134 2400Gbps 0.000025ms 0
54 150 400Gbps 0.0005ms 0
55 134 2400Gbps 0.000025ms 0
55 150 400Gbps 0.0005ms 0
56 135 2400Gbps 0.000025ms 0
56 151 400Gbps 0.0005ms 0
57 135 2400Gbps 0.000025ms 0
57 151 400Gbps 0.0005ms 0
58 135 2400Gbps 0.000025ms 0
58 151 400Gbps 0.0005ms 0
59 135 2400Gbps 0.000025ms 0
59 151 400Gbps 0.0005ms 0
60 135 2400Gbps 0.000025ms 0
60 151 400Gbps 0.0005ms 0
61 135 2400Gbps 0.000025ms 0
61 151 400Gbps 0.0005ms 0
62 135 2400Gbps 0.000025ms 0
62 151 400Gbps 0.0005ms 0
63 135 2400Gbps 0.000025ms 0
63 151 400Gbps 0.0005ms 0
64 136 2400Gbps 0.000025ms 0
64 152 400Gbps 0.0005ms 0
65 136 2400Gbps 0.000025ms 0
65 152 400Gbps 0.0005ms 0
66 136 2400Gbps 0.000025ms 0
66 152 400Gbps 0.0005ms 0
67 136 2400Gbps 0.000025ms 0
67 152 400Gbps 0.0005ms 0
68 136 2400Gbps 0.000025ms 0
68 152 400Gbps 0.0005ms 0
69 136 2400Gbps 0.000025ms 0
69 152 400Gbps 0.0005ms 0
70 136 2400Gbps 0.000025ms 0
70 152 400Gbps 0.0005ms 0
71 136 2400Gbps 0.000025ms 0
71 152 400Gbps 0.0005ms 0
72 137 2400Gbps 0.000025ms 0
72 153 400Gbps 0.0005ms 0
73 137 2400Gbps 0.000025ms 0
73 153 400Gbps 0.0005ms 0
74 137 2400Gbps 0.000025ms 0
74 153 400Gbps 0.0005ms 0
75 137 2400Gbps 0.000025ms 0
75 153 400Gbps 0.0005ms 0
76 137 2400Gbps 0.000025ms 0
76 153 400Gbps 0.0005ms 0
77 137 2400Gbps 0.000025ms 0
77 153 400Gbps 0.0005ms 0
78 137 2400Gbps 0.000025ms 0
78 153 400Gbps 0.0005ms 0
79 137 2400Gbps 0.000025ms 0
79 153 400Gbps 0.0005ms 0
80 138 2400Gbps 0.000025ms 0
80 154 400Gbps 0.0005ms 0
81 138 2400Gbps 0.000025ms 0
81 154 400Gbps 0.0005ms 0
82 138 2400Gbps 0.000025ms 0
82 154 400Gbps 0.0005ms 0
83 138 2400Gbps 0.000025ms 0
83 154 400Gbps 0.0005ms 0
84 138 2400Gbps 0.000025ms 0
84 154 400Gbps 0.0005ms 0
85 138 2400Gbps 0.000025ms 0
85 154 400Gbps 0.0005ms 0
86 138 2400Gbps 0.000025ms 0
86 154 400Gbps 0.0005ms 0
87 138 2400Gbps 0.000025ms 0
87 154 400Gbps 0.0005ms 0
88 139 2400Gbps 0.000025ms 0
88 155 400Gbps 0.0005ms 0
89 139 2400Gbps 0.000025ms 0
89 155 400Gbps 0.0005ms 0
90 139 2400Gbps 0.000025ms 0
90 155 400Gbps 0.0005ms 0
91 139 2400Gbps 0.000025ms 0
91 155 400Gbps 0.0005ms 0
92 139 2400Gbps 0.000025ms 0
92 155 400Gbps 0.0005ms 0
93 139 2400Gbps 0.000025ms 0
93 155 400Gbps 0.0005ms 0
94 139 2400Gbps 0.000025ms 0
94 155 400Gbps 0.0005ms 0
95 139 2400Gbps 0.000025ms 0
95 155 400Gbps 0.0005ms 0
96 140 2400Gbps 0.000025ms 0
96 156 400Gbps 0.0005ms 0
97 140 2400Gbps 0.000025ms 0
97 156 400Gbps 0.0005ms 0
98 140 2400Gbps 0.000025ms 0
98 156 400Gbps 0.0005ms 0
99 140 2400Gbps 0.000025ms 0
99 156 400Gbps 0.0005ms 0
100 140 2400Gbps 0.000025ms 0
100 156 400Gbps 0.0005ms 0
101 140 2400Gbps 0.000025ms 0
101 156 400Gbps 0.0005ms 0
102 140 2400Gbps 0.000025ms 0
102 156 400Gbps 0.0005ms 0
103 140 2400Gbps 0.000025ms 0
103 156 400Gbps 0.0005ms 0
104 141 2400Gbps 0.000025ms 0
104 157 400Gbps 0.0005ms 0
105 141 2400Gbps 0.000025ms 0
105 157 400Gbps 0.0005ms 0
106 141 2400Gbps 0.000025ms 0
106 157 400Gbps 0.0005ms 0
107 141 2400Gbps 0.000025ms 0
107 157 400Gbps 0.0005ms 0
108 141 2400Gbps 0.000025ms 0
108 157 400Gbps 0.0005ms 0
109 141 2400Gbps 0.000025ms 0
109 157 400Gbps 0.0005ms 0
110 141 2400Gbps 0.000025ms 0
110 157 400Gbps 0.0005ms 0
111 141 2400Gbps 0.000025ms 0
111 157 400Gbps 0.0005ms 0
112 142 2400Gbps 0.000025ms 0
112 158 400Gbps 0.0005ms 0
113 142 2400Gbps 0.000025ms 0
113 158 400Gbps 0.0005ms 0
114 142 2400Gbps 0.000025ms 0
114 158 400Gbps 0.0005ms 0
115 142 2400Gbps 0.000025ms 0
115 158 400Gbps 0.0005ms 0
116 142 2400Gbps 0.000025ms 0
116 158 400Gbps 0.0005ms 0
117 142 2400Gbps 0.000025ms 0
117 158 400Gbps 0.0005ms 0
118 142 2400Gbps 0.000025ms 0
118 158 400Gbps 0.0005ms 0
119 142 2400Gbps 0.000025ms 0
119 158 400Gbps 0.0005ms 0
120 143 2400Gbps 0.000025ms 0
120 159 400Gbps 0.0005ms 0
121 143 2400Gbps 0.000025ms 0
121 159 400Gbps 0.0005ms 0
122 143 2400Gbps 0.000025ms 0
122 159 400Gbps 0.0005ms 0
123 143 2400Gbps 0.000025ms 0
123 159 400Gbps 0.0005ms 0
124 143 2400Gbps 0.000025ms 0
124 159 400Gbps 0.0005ms 0
125 143 2400Gbps 0.000025ms 0
125 159 400Gbps 0.0005ms 0
126 143 2400Gbps 0.000025ms 0
126 159 400Gbps 0.0005ms 0
127 143 2400Gbps 0.000025ms 0
127 159 400Gbps 0.0005ms 0
144 160 200Gbps 0.0005ms 0
144 161 400Gbps 0.0005ms 0
144 162 200Gbps 0.0005ms 0
144 163 400Gbps 0.0005ms 0
144 164 200Gbps 0.0005ms 0
144 165 400Gbps 0.0005ms 0
144 166 200Gbps 0.0005ms 0
144 167 400Gbps 0.0005ms 0
145 160 200Gbps 0.0005ms 0
145 161 400Gbps 0.0005ms 0
145 162 200Gbps 0.0005ms 0
145 163 400Gbps 0.0005ms 0
145 164 200Gbps 0.0005ms 0
145 165 400Gbps 0.0005ms 0
145 166 200Gbps 0.0005ms 0
145 167 400Gbps 0.0005ms 0
146 160 200Gbps 0.0005ms 0
146 161 400Gbps 0.0005ms 0
146 162 200Gbps 0.0005ms 0
146 163 400Gbps 0.0005ms 0
146 164 200Gbps 0.0005ms 0
146 165 400Gbps 0.0005ms 0
146 166 200Gbps 0.0005ms 0
146 167 400Gbps 0.0005ms 0
147 160 200Gbps 0.0005ms 0
147 161 400Gbps 0.0005ms 0
147 162 200Gbps 0.0005ms 0
147 163 400Gbps 0.0005ms 0
147 164 200Gbps 0.0005ms 0
147 165 400Gbps 0.0005ms 0
147 166 200Gbps 0.0005ms 0
147 167 400Gbps 0.0005ms 0
148 160 200Gbps 0.0005ms 0
148 161 400Gbps 0.0005ms 0
148 162 200Gbps 0.0005ms 0
148 163 400Gbps 0.0005ms 0
148 164 200Gbps 0.0005ms 0
148 165 400Gbps 0.0005ms 0
148 166 200Gbps 0.0005ms 0
148 167 400Gbps 0.0005ms 0
149 160 200Gbps 0.0005ms 0
149 161 400Gbps 0.0005ms 0
149 162 200Gbps 0.0005ms 0
149 163 400Gbps 0.0005ms 0
149 164 200Gbps 0.0005ms 0
149 165 400Gbps 0.0005ms 0
149 166 200Gbps 0.0005ms 0
149 167 400Gbps 0.0005ms 0
150 160 200Gbps 0.0005ms 0
150 161 400Gbps 0.0005ms 0
150 162 200Gbps 0.0005ms 0
150 163 400Gbps 0.0005ms 0
150 164 200Gbps 0.0005ms 0
150 165 400Gbps 0.0005ms 0
150 166 200Gbps 0.0005ms 0
150 167 400Gbps 0.0005ms 0
151 160 200Gbps 0.0005ms 0
151 161 400Gbps 0.0005ms 0
151 162 200Gbps 0.0005ms 0
151 163 400Gbps 0.0005ms 0
151 164 200Gbps 0.0005ms 0
151 165 400Gbps 0.0005ms 0
151 166 200Gbps 0.0005ms 0
151 167 400Gbps 0.0005ms 0
152 160 200Gbps 0.0005ms 0
152 161 400Gbps 0.0005ms 0
152 162 200Gbps 0.0005ms 0
152 163 400Gbps 0.0005ms 0
152 164 200Gbps 0.0005ms 0
152 165 400Gbps 0.0005ms 0
152 166 200Gbps 0.0005ms 0
152 167 400Gbps 0.0005ms 0
153 160 200Gbps 0.0005ms 0
153 161 400Gbps 0.0005ms 0
153 162 200Gbps 0.0005ms 0
153 163 400Gbps 0.0005ms 0
153 164 200Gbps 0.0005ms 0
153 165 400Gbps 0.0005ms 0
153 166 200Gbps 0.0005ms 0
153 167 400Gbps 0.0005ms 0
154 160 200Gbps 0.0005ms 0
154 161 400Gbps 0.0005ms 0
154 162 200Gbps 0.0005ms 0
154 163 400Gbps 0.0005ms 0
154 164 200Gbps 0.0005ms 0
154 165 400Gbps 0.0005ms 0
154 166 200Gbps 0.0005ms 0
154 167 400Gbps 0.0005ms 0
155 160 200Gbps 0.0005ms 0
155 161 400Gbps 0.0005ms 0
155 162 200Gbps 0.0005ms 0
155 163 400Gbps 0.0005ms 0
155 164 200Gbps 0.0005ms 0
155 165 400Gbps 0.0005ms 0
155 166 200Gbps 0.0005ms 0
155 167 400Gbps 0.0005ms 0
156 160 200Gbps 0.0005ms 0
156 161 400Gbps 0.0005ms 0
156 162 200Gbps 0.0005ms 0
156 163 400Gbps 0.0005ms 0
156 164 200Gbps 0.0005ms 0
156 165 400Gbps 0.0005ms 0
156 166 200Gbps 0.0005ms 0
156 167 400Gbps 0.0005ms 0
157 160 200Gbps 0.0005ms 0
157 161 400Gbps 0.0005ms 0
157 162 200Gbps 0.0005ms 0
157 163 400Gbps 0.0005ms 0
157 164 200Gbps 0.0005ms 0
157 165 400Gbps 0.0005ms 0
157 166 200Gbps 0.0005ms 0
157 167 400Gbps 0.0005ms 0
158 160 200Gbps 0.0005ms 0
158 161 400Gbps 0.0005ms 0
158 162 200Gbps 0.0005ms 0
158 163 400Gbps 0.0005ms 0
158 164 200Gbps 0.0005ms 0
158 165 400Gbps 0.0005ms 0
158 166 200Gbps 0.0005ms 0
158 167 400Gbps 0.0005ms 0
159 160 200Gbps 0.0005ms 0
159 161 400Gbps 0.0005ms 0
159 162 200Gbps 0.0005ms 0
159 163 400Gbps 0.0005ms 0
159 164 200Gbps 0.0005ms 0
159 165 400Gbps 0.0005ms 0
159 166 200Gbps 0.0005ms 0
159 167 400Gbps 0.0005ms 0
src,dst,path1_ratio,path2_ratio,path3_ratio,path4_ratio,path5_ratio,path6_ratio,path7_ratio,path8_ratio
00,1,0.054,0.118,0.083,0.127,0.064,0.303,0.049,0.202
0,2,0.097,0.098,0.089,0.144,0.105,0.251,0.100,0.116
0,3,0.066,0.298,0.070,0.245,0.089,0.088,0.079,0.065
0,4,0.099,0.088,0.037,0.217,0.103,0.168,0.019,0.269
0,5,0.051,0.099,0.092,0.064,0.094,0.078,0.103,0.419
0,6,0.068,0.229,0.061,0.231,0.074,0.123,0.098,0.116
0,7,0.084,0.251,0.088,0.246,0.044,0.102,0.091,0.094
0,8,0.068,0.108,0.184,0.211,0.024,0.032,0.110,0.263
0,9,0.064,0.270,0.102,0.068,0.105,0.206,0.078,0.107
0,10,0.081,0.066,0.069,0.166,0.082,0.296,0.079,0.161
0,11,0.111,0.013,0.077,0.088,0.046,0.120,0.167,0.378
0,12,0.081,0.232,0.121,0.148,0.145,0.150,0.084,0.039
0,13,0.092,0.149,0.048,0.067,0.098,0.419,0.079,0.048
0,14,0.065,0.215,0.057,0.338,0.041,0.079,0.041,0.164
0,15,0.150,0.267,0.045,0.171,0.093,0.140,0.047,0.087
1,0,0.065,0.160,0.046,0.118,0.081,0.367,0.084,0.079
1,2,0.059,0.142,0.122,0.234,0.068,0.216,0.063,0.096
1,3,0.081,0.060,0.047,0.157,0.091,0.113,0.043,0.408
1,4,0.078,0.346,0.087,0.123,0.087,0.121,0.047,0.111
1,5,0.062,0.132,0.090,0.248,0.126,0.154,0.124,0.064
1,6,0.092,0.350,0.070,0.125,0.064,0.126,0.050,0.123
1,7,0.101,0.357,0.074,0.131,0.060,0.110,0.033,0.134
1,8,0.097,0.068,0.085,0.202,0.106,0.082,0.197,0.163
1,9,0.104,0.208,0.066,0.183,0.101,0.053,0.068,0.217
1,10,0.116,0.082,0.171,0.156,0.107,0.134,0.126,0.108
1,11,0.054,0.202,0.057,0.436,0.027,0.054,0.022,0.148
1,12,0.093,0.105,0.086,0.048,0.076,0.246,0.099,0.247
1,13,0.107,0.111,0.070,0.157,0.080,0.067,0.072,0.336
1,14,0.066,0.117,0.060,0.056,0.099,0.510,0.037,0.055
1,15,0.067,0.061,0.108,0.112,0.063,0.181,0.158,0.250
2,0,0.080,0.044,0.112,0.079,0.083,0.162,0.073,0.367
2,1,0.104,0.127,0.142,0.095,0.097,0.136,0.147,0.152
2,3,0.086,0.159,0.056,0.056,0.049,0.285,0.190,0.119
2,4,0.088,0.107,0.085,0.249,0.049,0.068,0.103,0.251
2,5,0.076,0.069,0.089,0.297,0.119,0.082,0.060,0.208
2,6,0.118,0.199,0.061,0.176,0.082,0.220,0.062,0.082
2,7,0.056,0.355,0.078,0.145,0.087,0.046,0.022,0.211
2,8,0.078,0.251,0.035,0.064,0.125,0.299,0.068,0.080
2,9,0.041,0.096,0.091,0.075,0.114,0.415,0.076,0.092
2,10,0.100,0.174,0.105,0.175,0.103,0.069,0.085,0.189
2,11,0.086,0.477,0.044,0.071,0.031,0.066,0.044,0.181
2,12,0.083,0.179,0.073,0.191,0.089,0.213,0.052,0.120
2,13,0.080,0.089,0.086,0.261,0.059,0.058,0.129,0.238
2,14,0.066,0.085,0.092,0.494,0.025,0.043,0.056,0.139
2,15,0.066,0.130,0.079,0.147,0.112,0.319,0.048,0.099
3,0,0.071,0.241,0.090,0.138,0.067,0.223,0.083,0.087
3,1,0.090,0.133,0.090,0.153,0.116,0.098,0.090,0.230
3,2,0.102,0.220,0.088,0.136,0.065,0.142,0.087,0.160
3,4,0.107,0.125,0.120,0.100,0.114,0.171,0.183,0.080
3,5,0.113,0.391,0.078,0.085,0.063,0.127,0.066,0.077
3,6,0.051,0.096,0.054,0.098,0.099,0.174,0.043,0.385
3,7,0.062,0.098,0.073,0.460,0.059,0.089,0.039,0.120
3,8,0.074,0.079,0.114,0.118,0.136,0.224,0.040,0.215
3,9,0.134,0.088,0.065,0.245,0.058,0.102,0.126,0.182
3,10,0.103,0.114,0.072,0.187,0.082,0.134,0.111,0.197
3,11,0.064,0.195,0.143,0.078,0.123,0.304,0.033,0.060
3,12,0.051,0.148,0.063,0.190,0.054,0.075,0.052,0.367
3,13,0.058,0.409,0.055,0.140,0.053,0.045,0.081,0.159
3,14,0.113,0.024,0.073,0.019,0.066,0.526,0.134,0.045
3,15,0.039,0.116,0.064,0.379,0.070,0.079,0.073,0.180
4,0,0.135,0.108,0.090,0.153,0.137,0.135,0.098,0.144
4,1,0.066,0.086,0.086,0.345,0.045,0.132,0.047,0.193
4,2,0.041,0.078,0.079,0.114,0.225,0.243,0.046,0.174
4,3,0.122,0.111,0.133,0.121,0.076,0.203,0.029,0.205
4,5,0.093,0.161,0.093,0.138,0.104,0.150,0.117,0.144
4,6,0.042,0.132,0.051,0.094,0.058,0.451,0.110,0.062
4,7,0.062,0.265,0.059,0.155,0.058,0.180,0.103,0.118
4,8,0.061,0.203,0.023,0.134,0.046,0.254,0.083,0.196
4,9,0.071,0.644,0.028,0.083,0.058,0.032,0.026,0.058
4,10,0.096,0.081,0.083,0.100,0.093,0.203,0.115,0.229
4,11,0.067,0.066,0.131,0.318,0.059,0.068,0.116,0.175
4,12,0.054,0.106,0.094,0.407,0.053,0.097,0.085,0.104
4,13,0.103,0.162,0.072,0.143,0.078,0.130,0.063,0.249
4,14,0.140,0.050,0.095,0.114,0.103,0.108,0.114,0.276
4,15,0.084,0.249,0.109,0.111,0.041,0.092,0.083,0.231
5,0,0.078,0.109,0.044,0.167,0.121,0.068,0.104,0.309
5,1,0.056,0.114,0.088,0.133,0.112,0.384,0.026,0.087
5,2,0.107,0.282,0.079,0.116,0.068,0.107,0.064,0.177
5,3,0.096,0.179,0.074,0.127,0.061,0.101,0.058,0.304
5,4,0.081,0.272,0.102,0.081,0.073,0.036,0.145,0.210
5,6,0.079,0.204,0.087,0.221,0.063,0.097,0.061,0.188
5,7,0.158,0.161,0.025,0.139,0.032,0.292,0.099,0.094
5,8,0.092,0.053,0.188,0.052,0.167,0.087,0.143,0.218
5,9,0.056,0.114,0.079,0.318,0.082,0.072,0.114,0.165
5,10,0.075,0.102,0.066,0.352,0.092,0.095,0.068,0.150
5,11,0.079,0.245,0.087,0.182,0.105,0.124,0.063,0.115
5,12,0.050,0.089,0.043,0.060,0.068,0.408,0.182,0.100
5,13,0.043,0.060,0.040,0.330,0.034,0.383,0.011,0.099
5,14,0.071,0.358,0.108,0.145,0.052,0.053,0.023,0.190
5,15,0.097,0.135,0.066,0.124,0.107,0.229,0.076,0.166
6,0,0.062,0.059,0.105,0.210,0.123,0.212,0.138,0.091
6,1,0.086,0.046,0.114,0.095,0.054,0.152,0.126,0.327
6,2,0.096,0.240,0.057,0.127,0.064,0.144,0.082,0.190
6,3,0.046,0.179,0.095,0.249,0.067,0.186,0.057,0.121
6,4,0.054,0.192,0.114,0.198,0.047,0.267,0.042,0.086
6,5,0.146,0.034,0.053,0.072,0.011,0.503,0.111,0.070
6,7,0.094,0.210,0.075,0.136,0.096,0.094,0.053,0.242
6,8,0.092,0.133,0.064,0.099,0.133,0.112,0.123,0.244
6,9,0.045,0.166,0.051,0.371,0.048,0.118,0.083,0.118
6,10,0.123,0.191,0.065,0.176,0.051,0.106,0.068,0.220
6,11,0.070,0.330,0.081,0.156,0.085,0.084,0.097,0.097
6,12,0.111,0.107,0.077,0.048,0.104,0.262,0.105,0.186
6,13,0.059,0.312,0.081,0.338,0.076,0.064,0.039,0.031
6,14,0.079,0.213,0.094,0.088,0.162,0.090,0.053,0.221
6,15,0.071,0.098,0.089,0.143,0.095,0.129,0.072,0.303
7,0,0.034,0.111,0.066,0.230,0.027,0.121,0.126,0.285
7,1,0.080,0.209,0.090,0.092,0.087,0.290,0.076,0.076
7,2,0.046,0.245,0.101,0.087,0.039,0.118,0.075,0.289
7,3,0.094,0.117,0.093,0.111,0.161,0.154,0.146,0.124
7,4,0.082,0.157,0.076,0.139,0.118,0.192,0.077,0.159
7,5,0.081,0.122,0.083,0.311,0.099,0.091,0.073,0.140
7,6,0.107,0.203,0.092,0.190,0.095,0.068,0.098,0.147
7,8,0.081,0.227,0.072,0.223,0.022,0.116,0.039,0.220
7,9,0.079,0.087,0.108,0.123,0.074,0.220,0.124,0.185
7,10,0.111,0.323,0.059,0.129,0.079,0.121,0.053,0.125
7,11,0.075,0.120,0.057,0.099,0.089,0.412,0.063,0.085
7,12,0.117,0.073,0.127,0.293,0.061,0.068,0.076,0.185
7,13,0.107,0.261,0.077,0.033,0.050,0.330,0.041,0.101
7,14,0.061,0.217,0.058,0.266,0.111,0.040,0.062,0.185
7,15,0.083,0.067,0.066,0.214,0.093,0.178,0.081,0.218
8,0,0.118,0.303,0.086,0.251,0.048,0.057,0.042,0.095
8,1,0.079,0.153,0.088,0.110,0.118,0.261,0.095,0.096
8,2,0.092,0.206,0.085,0.162,0.086,0.149,0.126,0.094
8,3,0.080,0.111,0.053,0.344,0.046,0.301,0.030,0.035
8,4,0.102,0.162,0.075,0.103,0.090,0.092,0.094,0.282
8,5,0.080,0.363,0.072,0.099,0.074,0.166,0.067,0.079
8,6,0.094,0.127,0.117,0.111,0.099,0.092,0.112,0.248
8,7,0.042,0.079,0.066,0.101,0.175,0.229,0.108,0.200
8,9,0.086,0.048,0.032,0.105,0.038,0.429,0.068,0.194
8,10,0.076,0.175,0.103,0.084,0.095,0.118,0.135,0.214
8,11,0.078,0.076,0.118,0.185,0.060,0.119,0.084,0.280
8,12,0.074,0.110,0.113,0.374,0.042,0.067,0.057,0.163
8,13,0.058,0.131,0.069,0.120,0.102,0.189,0.041,0.290
8,14,0.074,0.259,0.055,0.119,0.079,0.141,0.078,0.195
8,15,0.104,0.192,0.094,0.273,0.074,0.117,0.068,0.078
9,0,0.137,0.154,0.050,0.230,0.064,0.177,0.075,0.113
9,1,0.099,0.381,0.024,0.100,0.095,0.055,0.029,0.217
9,2,0.054,0.151,0.041,0.210,0.108,0.201,0.061,0.174
9,3,0.082,0.091,0.126,0.280,0.025,0.057,0.143,0.196
9,4,0.092,0.167,0.085,0.167,0.061,0.223,0.061,0.144
9,5,0.046,0.253,0.073,0.115,0.047,0.073,0.040,0.353
9,6,0.032,0.060,0.033,0.180,0.082,0.374,0.048,0.191
9,7,0.096,0.142,0.158,0.139,0.078,0.127,0.121,0.139
9,8,0.045,0.258,0.026,0.243,0.058,0.228,0.010,0.132
9,10,0.067,0.055,0.101,0.098,0.127,0.200,0.107,0.245
9,11,0.083,0.153,0.080,0.106,0.164,0.183,0.079,0.152
9,12,0.110,0.134,0.126,0.174,0.062,0.163,0.086,0.145
9,13,0.094,0.093,0.088,0.188,0.092,0.147,0.178,0.120
9,14,0.105,0.171,0.132,0.114,0.114,0.113,0.125,0.126
9,15,0.078,0.289,0.098,0.163,0.054,0.203,0.034,0.081
10,0,0.081,0.052,0.035,0.113,0.133,0.295,0.049,0.242
10,1,0.100,0.028,0.052,0.221,0.050,0.273,0.084,0.192
10,2,0.164,0.205,0.105,0.119,0.061,0.096,0.082,0.168
10,3,0.050,0.457,0.044,0.193,0.066,0.055,0.039,0.096
10,4,0.080,0.171,0.062,0.317,0.048,0.091,0.063,0.168
10,5,0.052,0.196,0.102,0.090,0.060,0.284,0.132,0.084
10,6,0.112,0.116,0.110,0.099,0.087,0.178,0.120,0.178
10,7,0.063,0.140,0.084,0.098,0.089,0.217,0.086,0.223
10,8,0.095,0.132,0.113,0.474,0.024,0.069,0.052,0.041
10,9,0.040,0.042,0.122,0.200,0.105,0.200,0.087,0.204
10,11,0.085,0.098,0.056,0.189,0.096,0.232,0.082,0.162
10,12,0.067,0.332,0.068,0.059,0.109,0.171,0.060,0.134
10,13,0.101,0.263,0.095,0.094,0.088,0.160,0.086,0.113
10,14,0.086,0.112,0.093,0.091,0.100,0.138,0.168,0.212
10,15,0.065,0.171,0.071,0.159,0.102,0.072,0.021,0.339
11,0,0.046,0.167,0.086,0.244,0.039,0.088,0.045,0.285
11,1,0.039,0.397,0.021,0.287,0.086,0.017,0.045,0.108
11,2,0.104,0.239,0.075,0.235,0.052,0.073,0.057,0.165
11,3,0.120,0.150,0.073,0.144,0.091,0.230,0.072,0.120
11,4,0.083,0.107,0.102,0.143,0.107,0.241,0.134,0.083
11,5,0.075,0.083,0.049,0.161,0.069,0.318,0.037,0.208
11,6,0.135,0.207,0.094,0.136,0.089,0.146,0.075,0.118
11,7,0.100,0.058,0.107,0.220,0.092,0.206,0.115,0.102
11,8,0.077,0.101,0.076,0.227,0.074,0.231,0.045,0.169
11,9,0.088,0.064,0.161,0.051,0.014,0.124,0.068,0.430
11,10,0.049,0.449,0.083,0.082,0.144,0.055,0.035,0.103
11,12,0.082,0.212,0.064,0.178,0.050,0.121,0.097,0.196
11,13,0.063,0.096,0.098,0.111,0.108,0.257,0.102,0.165
11,14,0.066,0.068,0.093,0.213,0.076,0.180,0.154,0.150
11,15,0.079,0.132,0.072,0.093,0.107,0.259,0.151,0.107
12,0,0.060,0.579,0.020,0.100,0.053,0.048,0.028,0.112
12,1,0.052,0.069,0.094,0.159,0.074,0.180,0.109,0.263
12,2,0.090,0.129,0.082,0.320,0.128,0.069,0.086,0.096
12,3,0.072,0.088,0.065,0.209,0.072,0.197,0.077,0.220
12,4,0.131,0.077,0.123,0.141,0.122,0.124,0.082,0.200
12,5,0.109,0.194,0.061,0.219,0.075,0.153,0.047,0.142
12,6,0.093,0.084,0.107,0.262,0.082,0.066,0.134,0.172
12,7,0.040,0.062,0.101,0.108,0.103,0.360,0.053,0.173
12,8,0.049,0.578,0.040,0.140,0.025,0.137,0.011,0.020
12,9,0.126,0.075,0.090,0.157,0.149,0.037,0.149,0.217
12,10,0.043,0.009,0.016,0.134,0.019,0.610,0.099,0.070
12,11,0.096,0.113,0.083,0.231,0.096,0.069,0.101,0.211
12,13,0.076,0.172,0.097,0.128,0.104,0.120,0.052,0.251
12,14,0.084,0.190,0.132,0.093,0.067,0.112,0.075,0.247
12,15,0.100,0.148,0.099,0.149,0.062,0.234,0.098,0.110
13,0,0.103,0.127,0.209,0.103,0.089,0.088,0.189,0.092
13,1,0.137,0.366,0.078,0.137,0.049,0.070,0.050,0.113
13,2,0.090,0.103,0.084,0.083,0.078,0.159,0.122,0.281
13,3,0.076,0.155,0.068,0.130,0.091,0.219,0.088,0.173
13,4,0.053,0.061,0.067,0.067,0.074,0.510,0.087,0.081
13,5,0.098,0.087,0.115,0.234,0.090,0.118,0.070,0.188
13,6,0.050,0.099,0.074,0.242,0.067,0.177,0.095,0.196
13,7,0.070,0.032,0.109,0.225,0.125,0.124,0.120,0.195
13,8,0.077,0.115,0.040,0.189,0.086,0.184,0.061,0.248
13,9,0.110,0.112,0.084,0.277,0.073,0.098,0.068,0.178
13,10,0.097,0.248,0.056,0.125,0.074,0.133,0.050,0.217
13,11,0.046,0.298,0.047,0.175,0.105,0.190,0.010,0.129
13,12,0.096,0.283,0.062,0.185,0.119,0.064,0.066,0.125
13,14,0.057,0.152,0.029,0.144,0.050,0.307,0.052,0.209
13,15,0.084,0.309,0.082,0.163,0.062,0.082,0.092,0.126
14,0,0.060,0.085,0.103,0.183,0.032,0.378,0.027,0.132
14,1,0.081,0.148,0.103,0.177,0.101,0.110,0.151,0.129
14,2,0.035,0.073,0.077,0.158,0.022,0.417,0.064,0.154
14,3,0.105,0.289,0.126,0.071,0.067,0.070,0.071,0.201
14,4,0.064,0.209,0.024,0.273,0.086,0.168,0.084,0.092
14,5,0.055,0.160,0.032,0.183,0.107,0.149,0.068,0.246
14,6,0.038,0.328,0.075,0.219,0.064,0.066,0.016,0.194
14,7,0.104,0.168,0.073,0.138,0.097,0.138,0.079,0.203
14,8,0.118,0.068,0.096,0.088,0.121,0.156,0.095,0.258
14,9,0.152,0.107,0.069,0.088,0.117,0.346,0.073,0.048
14,10,0.031,0.288,0.068,0.437,0.035,0.047,0.020,0.074
14,11,0.121,0.106,0.080,0.126,0.068,0.097,0.140,0.262
14,12,0.100,0.248,0.030,0.044,0.111,0.129,0.058,0.280
14,13,0.091,0.133,0.159,0.213,0.096,0.090,0.127,0.091
14,15,0.079,0.138,0.102,0.138,0.085,0.169,0.144,0.145
15,0,0.087,0.274,0.077,0.181,0.120,0.106,0.056,0.099
15,1,0.113,0.128,0.082,0.272,0.090,0.062,0.114,0.139
15,2,0.063,0.129,0.073,0.284,0.037,0.121,0.102,0.191
15,3,0.067,0.075,0.093,0.107,0.146,0.277,0.083,0.152
15,4,0.052,0.233,0.068,0.176,0.062,0.076,0.031,0.302
15,5,0.079,0.189,0.134,0.198,0.066,0.090,0.113,0.131
15,6,0.099,0.107,0.122,0.170,0.106,0.142,0.105,0.149
15,7,0.107,0.106,0.069,0.126,0.059,0.173,0.068,0.292
15,8,0.091,0.182,0.050,0.078,0.084,0.299,0.128,0.088
15,9,0.044,0.436,0.062,0.179,0.054,0.068,0.022,0.135
15,10,0.081,0.139,0.095,0.115,0.040,0.213,0.066,0.251
15,11,0.082,0.065,0.064,0.074,0.074,0.429,0.091,0.121
15,12,0.075,0.154,0.105,0.125,0.092,0.294,0.062,0.093
15,13,0.094,0.081,0.084,0.196,0.116,0.070,0.116,0.243
15,14,0.095,0.239,0.067,0.238,0.080,0.081,0.050,0.150