ipvs
南风未起 2024-11-20 20:36:22 kubeproxy
# 1.ipvs
# 1.1.简介
ipvs是linux内核实现的四层负载均衡，利用iptables进行数据包过滤及SNAT，基于ipset存储DROP或SNAT的流量，确保iptables规则的数量恒定。相比于iptables，ipvs利用更高效的数据结构(Hash表)，支持更复杂的负载均衡算法及连接重试功能，允许几乎无限的规模扩张。--- 优势 1.专为大规模服务负载均衡设计,实现tcp/udp服务的流量转发给后端的真实服务 2.iptables正在使用情况动态更新规则(ipset),性能几乎无损 3.支持服务健康检查/连接重试/端口映射/会话保持 4.支持更复杂的负载均衡算法(最少负载/最少连接/局部性/加权/随机)1
2
3
4
5
注意
1.
iptables规则混乱，出现问题极难排查2.
iptables规则串行执行，算法复杂度为O(n)，规则膨胀后性能极差3.
iptables支持负载均衡功能有限，尤其不支持连接重试
# 1.2.DR模式
dr(direct routing)是性能最高的模式，工作在L2，VIP和RIP必须位于相同网段。工作时，请求报文经过ipvs director将MAC地址设置为RS的Mac再转发，服务响应后结果直接返回给客户端，不再经过负载均衡器。--- 特点 1.数据包转发时,源/目的IP端口不变,数据包的MAC改写为RS的MAC进行转发 2.RS必须在环回网卡(lo)绑定VIP,避免RS无法识别数据包(目的IP为VIP)进行丢弃 3.RS的业务进程必须监听环回网卡的VIP,端口必须与LB的虚拟服务端口一致,避免RS拒绝数据包 4.请求处理后,响应直接返回客户端,不会经过LB,因此RS和CIP之间网络可达1
2
3
4
5
# 1.3.NAT模式
nat模式支持端口映射，回程报文经过ipvs director，又称为masq(伪装)模式。工作时，LB会对请求包做目的地址转换(DNAT)，将请求包的目的IP改写为RSIP，响应包做源地址转换(SNAT)，将响应包的源IP改写为VIP。--- 特点 1.由于请求包和响应包经过ipvs director,会进行DNAT和SNAT 2.LB仅完成部分地址转换,RS收到的请求包可以看到CIP 3.RS的网关地址需配置为LB的VIP地址,确保响应包正确回到LB 4.LB和RS必须位于相同子网,客户端和LB/RS不能位于相同子网,避免RS根据客户端MAC直接返回,避免数据包目的IP是VIP造成客户端丢弃1
2
3
4
5
注意
由于回包经过
ipvs director进行一次SNAT，效率相比DR模式略低
# 1.4.IP隧道模式
tunning模式会将原始请求包封装在另一个IP包，封好的包转发给RS，RS拆包获取目的为VIP的原始报文，服务器发现VIP配置在本地的IP隧道设备就处理，响应报文会根据路由表直接返给客户端。--- 特点 1.LB和RS节点不用处在相同子网,解除NAT模式和DR模式限制 2.节点必须支持隧道技术,RS节点的TUN网卡需配置VIP地址,避免拆包后无法解析丢弃 3.RS节点与Client网络可达,确保响应包可以直接返给源1
2
3
4
注意
由于请求过程的
封包和拆包造成额外开销，性能最低
# 1.5.kube扩展链
ipvs会在5链的基础上进行链扩展，新增kube-services、kube-firewall、kube-postrouting、kube-mark-masq、kube-node-port、kube-mark-drop、kube-forward和kube-load-balancer共8链。
注意
扩展链会插入
5链进行包劫持，确保数据包可以流向ipvs处理
# 1.6.回包问题
kubeproxy针对clusterIP流量依赖conntrack表以记录连接，确保DNAT到目标的回包基于conntrack表匹配连接反向NAT，形成连接链路闭环。由于conntrack工作在三层，网桥是虚拟的二层转发设备，直接访问同一网桥内的地址不会经过conntrack，造成回包问题。
注意
这也是为什么开启
conntrack时，ipvs proxier验证br_netfilter/bridge-nf-call-iptables开启
# 1.7.流量倾斜
ipvs也存在长连接流量倾斜问题，请求流量首次到达会经过LB进行负载，一旦连接建立请求目标就会固定，无法再触发负载均衡。因此，真正的长连接负载均衡更可能的是请求方实现，根据不同数据目标提前规划长连接作为endpoints，每次发起请求选择一个长连接达到负载目的。
# 2.ipvsproxier
# 2.1.newProxier
proxier是ipvs核心实现模块，用于配合informer进行事件处理、缓存更新，以便基于监听的service/endpoints最新状态刷新规则，维护ipset及iptables路由信息。// NewProxier returns a new Proxier given an iptables and ipvs Interface instance. func NewProxier(...) (*Proxier, error) { // 1.br_netfilter和bridge-nf-call-iptables内核参数检查,确保bridge容器流量经过iptables sysctl.GetSysctl(sysctlBridgeCallIPTables) ... // 2.设置net.ipv4.vs.conntrack=1内核参数,开启ipvs连接跟踪,用于SNAT utilproxy.EnsureSysctl(sysctl, sysctlVSConnTrack, 1) ... // 3.内核版本处在4.1~5.9关闭ipvs连接复用(存在问题) if kernelVersion.In(4.1,5.9) { // Set the connection reuse mode utilproxy.EnsureSysctl(sysctl, sysctlConnReuse, 0) ... } // 4.设置内核参数expire_nodest_conn=1,目标IP不存在及时清理连接 utilproxy.EnsureSysctl(sysctl, sysctlExpireNoDestConn, 1) ... // 5.设置内核参数expire_quiescent_template=1,清理失效的模板连接 utilproxy.EnsureSysctl(sysctl, sysctlExpireQuiescentTemplate, 1) ... // 6.设置内核参数ip_forward=1,允许进程路由转发 utilproxy.EnsureSysctl(sysctl, sysctlForward, 1) ... // 7.设置arp内核参数,避免多个节点响应同一个VIP的ARP请求 if strictARP { // 设置arp_ignore=1(请求IP是配置在接收请求的网卡才响应) utilproxy.EnsureSysctl(sysctl, sysctlArpIgnore, 1) ... // 设置arp_announce=2(限制节点仅用真实拥有的IP宣布,避免使用VIP地址广播) utilproxy.EnsureSysctl(sysctl, sysctlArpAnnounce, 2) ... } // 8.设置内核参数,配置ipvs超时时间 if tcpTimeout > 0 || tcpFinTimeout > 0 || udpTimeout > 0 { ipvs.ConfigureTimeouts(tcpTimeout, tcpFinTimeout, udpTimeout) ... } // 9.SNAT标记 masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x", masqueradeValue) ... // 10.设置默认调度算法(rr) if len(scheduler) == 0 { scheduler = DefaultScheduler } ... // 11.初始化ipvsproxier proxier := &Proxier{ ... } // 12.初始化ipset规则 proxier.ipsetList = make(map[string]*IPSet) for _, is := range ipsetInfo { proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, (ipFamily == v1.IPv6Protocol), is.comment) } ... // 13.初始化syncRunner proxier.syncRunner = async.NewBoundedFrequencyRunner("sync-runner", proxier.syncProxyRules, minSyncPeriod, syncPeriod, burstSyncs) // 14.优雅删除ipvs虚拟服务 proxier.gracefuldeleteManager.Run() return proxier, nil } // Run start a goroutine to try to delete rs in the graceful delete rsList with an interval 1 minute func (m *GracefulTerminationManager) Run() { // 间隔1min周期执行 go wait.Until(m.tryDeleteRs, rsCheckDeleteInterval, wait.NeverStop) } // 尝试删除RS func (m *GracefulTerminationManager) tryDeleteRs() { if !m.rsList.flushList(m.deleteRsFunc) { klog.ErrorS(nil, "Try flush graceful termination list error") } }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# 2.2.eventWatch
ipvs proxier也会实现serviceHandler/endpointSliceHandler/nodeHandler相关接口，service/endpointSlice/node变更时触发回调驱动syncLoop()进行规则更新。// OnServiceAdd is called whenever creation of new service object is observed. func (proxier *Proxier) OnServiceAdd(service *v1.Service) { proxier.OnServiceUpdate(nil, service) } // OnServiceUpdate is called whenever modification of an existing service object is observed. func (proxier *Proxier) OnServiceUpdate(oldService, service *v1.Service) { if proxier.serviceChanges.Update(oldService, service) && proxier.isInitialized() { proxier.Sync() } } // OnServiceDelete is called whenever deletion of an existing service object is observed. func (proxier *Proxier) OnServiceDelete(service *v1.Service) { proxier.OnServiceUpdate(service, nil) } // OnEndpointSliceAdd is called whenever creation of a new endpoint slice object is observed. func (proxier *Proxier) OnEndpointSliceAdd(endpointSlice *discovery.EndpointSlice) { proxier.OnEndpointSliceUpdate(nil,endpointSlice) } // OnEndpointSliceUpdate is called whenever modification of an existing endpoint slice object is observed. func (proxier *Proxier) OnEndpointSliceUpdate(_, endpointSlice *discovery.EndpointSlice) { if proxier.endpointsChanges.EndpointSliceUpdate(endpointSlice, false) && proxier.isInitialized() { proxier.Sync() } } // OnEndpointSliceDelete is called whenever deletion of an existing endpoint slice object is observed. func (proxier *Proxier) OnEndpointSliceDelete(endpointSlice *discovery.EndpointSlice) { proxier.OnEndpointSliceUpdate(endpointSlice,nil) } // OnNodeAdd is called whenever creation of new node object is observed. func (proxier *Proxier) OnNodeAdd(node *v1.Node) { proxier.OnNodeUpdate(nil,node) } // OnNodeUpdate is called whenever modification of an existing node object is observed. func (proxier *Proxier) OnNodeUpdate(oldNode, node *v1.Node) { if node.Name != proxier.hostname { return } if reflect.DeepEqual(proxier.nodeLabels, node.Labels) { return } proxier.mu.Lock() proxier.nodeLabels = map[string]string{} for k, v := range node.Labels { proxier.nodeLabels[k] = v } proxier.syncProxyRules() } // OnNodeDelete is called whenever deletion of an existing node object is observed. func (proxier *Proxier) OnNodeDelete(node *v1.Node) { if node.Name != proxier.hostname { return } proxier.mu.Lock() proxier.nodeLabels = nil proxier.mu.Unlock() proxier.syncProxyRules() } // Sync is called to synchronize the proxier state to iptables and ipvs as soon as possible. func (proxier *Proxier) Sync() { ... proxier.syncRunner.Run() } // Run the function as soon as possible. If this is called while Loop is not // running, the call may be deferred indefinitely. func (bfr *BoundedFrequencyRunner) Run() { // signal select { case bfr.run <- struct{}{}: default: } }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
注意
1.
service/endpointSlice的增量事件基于syncRunner.sync()通知信号触发更新2.
node的增量事件对比nodeLabel差异，调用proxier.syncProxyRules()更新规则
# 2.3.syncProxyRules
ipvs支持DR/Tunning/NAT三种模式，仅NAT支持端口映射，因此kubeproxy实现上用的是NAT模式，由于内核原生的ipvs仅支持DNAT，部分场景下ipvs仍依赖iptables规则。// SyncLoop runs periodic work. This is expected to run as a goroutine or as the main loop of the app. It does not return. func (proxier *Proxier) SyncLoop() { ... proxier.syncRunner.Loop(wait.NeverStop) } // Loop handles the periodic timer and run requests. This is expected to be called as a goroutine. func (bfr *BoundedFrequencyRunner) Loop(stop <-chan struct{}) { ... for { select { // 退出 case <-stop: bfr.stop() return // 间隔30s周期执行 case <-bfr.timer.C(): bfr.tryRun() // 事件驱动 case <-bfr.run: bfr.tryRun() case <-bfr.retry: bfr.doRetry() } } } // assumes the lock is not held func (bfr *BoundedFrequencyRunner) tryRun() { bfr.mu.Lock() defer bfr.mu.Unlock() // 令牌申请 if bfr.limiter.TryAccept() { // 执行syncProxyRules bfr.fn() ... // 重置timer下一次周期(30s) bfr.timer.Stop() bfr.timer.Reset(bfr.maxInterval) return } ... // 重置timer--min((剩余定时触发时间,距上一次触发剩余时间)) bfr.timer.Stop() bfr.timer.Reset(nextScheduled) } // This is where all of the ipvs calls happen. func (proxier *Proxier) syncProxyRules() { proxier.mu.Lock() defer proxier.mu.Unlock() // don't sync rules till we've received services and endpoints if !proxier.isInitialized() { return } ... // 1.service同步 serviceUpdateResult := proxier.serviceMap.Update(proxier.serviceChanges) // 2.endpoints同步 endpointUpdateResult := proxier.endpointsMap.Update(proxier.endpointsChanges) // 3.整理udp state service(udp是无连接协议,旧链接不会自然结束,clusterIP改变必须手动清理conntrack) staleServices := serviceUpdateResult.UDPStaleClusterIP for _, svcPortName := range endpointUpdateResult.StaleServiceNames { if svcInfo, ok := proxier.serviceMap[svcPortName]; ok && svcInfo != nil && conntrack.IsClearConntrackNeeded(svcInfo.Protocol()) { staleServices.Insert(svcInfo.ClusterIP().String()) for _, extIP := range svcInfo.ExternalIPStrings() { staleServices.Insert(extIP) } for _, extIP := range svcInfo.LoadBalancerIPStrings() { staleServices.Insert(extIP) } } } ... // 4.创建及链接kube专用链 proxier.createAndLinkKubeChain() // 5.确保dummy网卡存在(kube-ipvs0),用于绑定clusterIP _, err := proxier.netlinkHandle.EnsureDummyDevice(DefaultDummyDevice) ... // 6.ipset检查 for _, set := range proxier.ipsetList { // ipset创建 ensureIPSet(set) ... // 清空entry,重新写入最新端点信息 set.resetEntries() } ... // 7.获取dummy网卡绑定地址 bindedAddresses, err := proxier.ipGetter.BindedIPs() ... // 8.nodeport可用IP整理 if hasNodePort { // 获取满足nodePortAddress网段的所有网卡地址 nodeAddrSet, err := utilproxy.GetNodeAddresses(proxier.nodePortAddresses, proxier.networkInterfacer) ... nodeAddresses = nodeAddrSet.List() for _, address := range nodeAddresses { a := netutils.ParseIPSloppy(address) // 环回地址跳过,nodeport不能暴露loopback if a.IsLoopback() { continue } // 零地址,可以绑定所有IP if utilproxy.IsZeroCIDR(address) { // 获取节点所有网卡IP(剔除kube-ipvs0和loopback网卡地址) nodeIPs, err = proxier.ipGetter.NodeIPs() ... break } nodeIPs = append(nodeIPs, a) } } ... // 9.service ipvs rule维护 for svcName, svc := range proxier.serviceMap { ... svcNameString := svcName.String() // pod回环流量SNAT处理 for _, e := range proxier.endpointsMap[svcName] { ... // 本地endpoint ipset entry的协议及IP proxier.ipsetList[kubeLoopBackIPSet].validateEntry(entry) ... // 记录loopback ipset entry proxier.ipsetList[kubeLoopBackIPSet].activeEntries.Insert(entry.String()) } // 校验及记录clusterIP ipset entry proxier.ipsetList[kubeClusterIPSet].validateEntry(entry) ... proxier.ipsetList[kubeClusterIPSet].activeEntries.Insert(entry.String()) ... // clusterIP vs绑定到dummy网卡 if err := proxier.syncService(svcNameString, serv, true, bindedAddresses); err == nil { ... // vip绑定endpoint rs,清理无效的endpoint rs proxier.syncEndpoint(svcName, internalNodeLocal, serv) ... } // externalIP流量处理 for _, externalIP := range svcInfo.ExternalIPStrings() { ... // ipvs local entry if svcInfo.ExternalPolicyLocal() { // 校验及记录external ipset entry proxier.ipsetList[kubeExternalIPLocalSet].validateEntry(entry) ... proxier.ipsetList[kubeExternalIPLocalSet].activeEntries.Insert(entry.String()) // ipvs cluster entry } else { // 校验及记录external ipset entry proxier.ipsetList[kubeExternalIPSet].validateEntry(entry) ... proxier.ipsetList[kubeExternalIPSet].activeEntries.Insert(entry.String()) } ... // externalIP绑定到dummy网卡 if err := proxier.syncService(svcNameString, serv, true, bindedAddresses); err == nil { ... // vip绑定endpoint rs,清理无效endpoint rs proxier.syncEndpoint(svcName, svcInfo.ExternalPolicyLocal(), serv) ... } } // LB流量处理 for _, ingress := range svcInfo.LoadBalancerIPStrings() { ... // 校验及记录lb ipvs entry proxier.ipsetList[kubeLoadBalancerSet].validateEntry(entry) ... proxier.ipsetList[kubeLoadBalancerSet].activeEntries.Insert(entry.String()) // service local policy if svcInfo.ExternalPolicyLocal() { // 校验及记录lb ipvs local entry proxier.ipsetList[kubeLoadBalancerLocalSet].validateEntry(entry) ... proxier.ipsetList[kubeLoadBalancerLocalSet].activeEntries.Insert(entry.String()) } // 限制访问lb的cidr if len(svcInfo.LoadBalancerSourceRanges()) != 0 { // 校验及记录lb firwall ipvs entry(命中kubefw规则才放行) proxier.ipsetList[kubeLoadbalancerFWSet].validateEntry(entry) ... proxier.ipsetList[kubeLoadbalancerFWSet].activeEntries.Insert(entry.String()) // 白名单cidr for _, src := range svcInfo.LoadBalancerSourceRanges() { ... // 校验及记录允许访问lb的source ipvs entry proxier.ipsetList[kubeLoadBalancerSourceCIDRSet].validateEntry(entry) ... proxier.ipsetList[kubeLoadBalancerSourceCIDRSet].activeEntries.Insert(entry.String()) ... } // 节点访问VIP(vip loopback) if allowFromNode { ... // 校验及记录vip loopback ipvs entry proxier.ipsetList[kubeLoadBalancerSourceIPSet].validateEntry(entry) ... proxier.ipsetList[kubeLoadBalancerSourceIPSet].activeEntries.Insert(entry.String()) } } ... // lbIP vs绑定到dummy网卡 if err := proxier.syncService(svcNameString, serv, true, bindedAddresses); err == nil { ... // vip绑定endpoint rs,清理无效的endpoint rs proxier.syncEndpoint(svcName, svcInfo.ExternalPolicyLocal(), serv) } } // nodeport流量处理 if svcInfo.NodePort() != 0 { ... // 构建本地端口对象 for _, address := range nodeAddresses { lp := netutils.LocalPort{ Description: "nodePort for " + svcNameString, IP: address, IPFamily: localPortIPFamily, Port: svcInfo.NodePort(), Protocol: netutils.Protocol(svcInfo.Protocol()), } if utilproxy.IsZeroCIDR(address) { // Empty IP address means all lp.IP = "" lps = append(lps, lp) break } lps = append(lps, lp) } // 清理udp连接追踪 for _, lp := range lps { if svcInfo.Protocol() != v1.ProtocolSCTP && lp.Protocol == netutils.UDP { conntrack.ClearEntriesForPort(proxier.exec, lp.Port, isIPv6, v1.ProtocolUDP) } } ... if nodePortSet != nil { // 校验及记录nodeport ipvs entry for _, entry := range entries { nodePortSet.validateEntry(entry) ... nodePortSet.activeEntries.Insert(entry.String()) } ... } // nodeport local建立独立ipset if svcInfo.ExternalPolicyLocal() { ... if nodePortLocalSet != nil { // 校验及记录nodeport ipvs entry for _, entry := range entries { nodePortLocalSet.validateEntry(entry) ... nodePortLocalSet.activeEntries.Insert(entry.String()) } } } // Build ipvs kernel routes for each node ip address for _, nodeIP := range nodeIPs { ... // 注册nodeport vs(不绑定dummy网卡) if err := proxier.syncService(svcNameString, serv, false, bindedAddresses); err == nil { ... // vip绑定endpoint rs,清理无效的endpoint rs proxier.syncEndpoint(svcName, svcInfo.ExternalPolicyLocal(), serv) } } } // 端口健康检查,切薄本地kubeproxy或LB可以访问健康检查端口 if svcInfo.HealthCheckNodePort() != 0 { nodePortSet := proxier.ipsetList[kubeHealthCheckNodePortSet] ... nodePortSet.validateEntry(entry) ... nodePortSet.activeEntries.Insert(entry.String()) } } // 10.ipset同步到内核 for _, set := range proxier.ipsetList { set.syncIPSetEntries() } // 11.根据ipset生成iptables规则 proxier.writeIptablesRules() ... // 12.iptables规则写到内核 proxier.iptables.RestoreAll(proxier.iptablesData.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters) ... // 13.获取dummy网卡绑定IP currentBindAddrs, err := proxier.netlinkHandle.ListBindAddress(DefaultDummyDevice) ... // 14.获取以前绑定不再使用IP legacyBindAddrs := proxier.getLegacyBindAddr(activeBindAddrs, currentBindAddrs) // 15.获取内核已存在的ipvs虚拟服务 appliedSvcs, err := proxier.ipvs.GetVirtualServers() ... for _, appliedSvc := range appliedSvcs { currentIPVSServices[appliedSvc.String()] = appliedSvc } // 16.清理不再使用的ipvs虚拟服务,解绑dummy网卡不再使用IP proxier.cleanLegacyService(activeIPVSServices, currentIPVSServices, legacyBindAddrs) ... // 17.清理连接追踪 for _, svcIP := range staleServices.UnsortedList() { conntrack.ClearEntriesForIP(proxier.exec, svcIP, v1.ProtocolUDP) ... } // 18.旧endpoint proxier.deleteEndpointConnections(endpointUpdateResult.StaleEndpoints) ... }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
注意
1.
clusterIP/loadbalancer/externalIP/nodeport模式均会将vip绑定到dummy网卡，作为LB负载2.
endpoint的变化最终体现在ipset entry，涉及新规则补充及旧规则删除3.
dummy网卡及virtual server同步会触发过期IP和VS清理
# 2.4.ipvs规则
注意
ipvs模式数据包劫持还是基于iptables chain，iptables chain规则命中后转发到ipset进行DNAT及更新conntrack条目