sockmap
南风未起 2026-08-25 21:39:22 cni
# 1.简介
# 1.1.痛点
TCP连接交换数据是常见的需求，本质上是socket间的通信，传统模块基于read/write系统调用和数据拷贝完成，会发生多次上下文切换及拷贝。
Linux后续提供的零拷贝计数诸如sendfile，无法支持socket-->socket转发，本质上需唤醒进程的上下文进行系统调用及利用缓冲区完成交换。
注意
Linux Kernel 4.14引入sockmap，基于eBPF技术直接将数据转发offload到内核
# 1.2.变革
sockmap是Linux内核提供的一种BPF Map类型，允许将socket指针存到内核sockmap及挂载sk_msg，实现socket数据直通。--- BPF Hook 1.sock_ops: 挂载到cgroup根，监听socket时间，连接建立时将符合条件的socket加入sockmap 2.sk_msg: 挂载到soctmap，发起sendmsg/send调用会截获消息，通过bpf_msg_redirect_hash()重定向到对端socket接收队列1
2
3
注意
1.内核会检查发送
sendmsg/send的socket是否注册到挂载sk_msg的sockmap，注册的话主动触发一下sk_msg2.利用这一机制，数据包无需经过完整的
TCP/IP协议栈、iptables和conntrack层层处理，显著降低延迟和CPU开销3.
calico实现仅处理访问envoy sidecar连接，也就是加速不会作用于所有socket，只会作用于app<-->envoy sidecar
# 1.3.限制
sockmap对内核版本、环境及CPU架构都有一定限制，同时主机环境必须具备bpftool工具，Go侧实现依赖该工具操作BPF对象。// bpf/bpf.go#L2133-L2144 func SupportsSockmap() error { // 内核版本限制 if err := isAtLeastKernel(v4Dot20Dot0); err != nil { return err } // BPF实现仅支持小端序 if nativeEndian != binary.LittleEndian { return fmt.Errorf("this bpf library only supports little endian architectures") } return nil }1
2
3
4
5
6
7
8
9
10
11
12
13依赖项 要求 说明 内核版本 ≥ 4.20.0 完整功能支持限制 Cgroup V2 必须挂载 sock_ops程序需挂载到cgroup v2BPF Filesystem 必须挂载 用于 Pin BPF Map和Progbpftool 必须在 PATH Go侧通过bpftool操作BPF对象CPU架构 Little Endian 代码显式检查小端序 注意
sockmap也是进程级的，一个连接会有两个socket，对应client/server两个进程
# 2.实现
# 2.1.map
calico sockmap会用到两个BPF Map协调工作，calico_sock_map存储socket引用，calico_sk_endpoints存储加速网段。// calico_sock_map, sk_msg重定向的索引表 struct bpf_map_def __attribute__((section("maps"))) calico_sock_map = { .type = BPF_MAP_TYPE_SOCKHASH, .key_size = sizeof(struct sock_key), // 12 {ip,port,0:app/1:envoy} .value_size = sizeof(__u32), // 4 bytes (socket pointer stored by kernel) .max_entries = 65535, }; func (b *BPFLib) NewSockmap() (string, error) { // Pin路径: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 mapPath := filepath.Join(b.sockmapDir, sockMapName) keySize := 12 // sizeof(struct sock_key) valueSize := 4 return newMap(sockMapName, mapPath, "sockhash", 65535, keySize, valueSize, 0) } // calico_sk_endpoints, sock_ops地址加速检查表 struct bpf_map_def __attribute__((section("maps"))) calico_sk_endpoints = { .type = BPF_MAP_TYPE_LPM_TRIE, .key_size = sizeof(union ip4_bpf_lpm_trie_key), // 8 bytes .value_size = sizeof(__u32), // 4 bytes .max_entries = 65535, .map_flags = BPF_F_NO_PREALLOC, }; func (b *BPFLib) NewSockmapEndpointsMap() (string, error) { // Pin路径: /sys/fs/bpf/calico/sockmap/calico_sk_endpoints_v1 mapPath := filepath.Join(b.sockmapDir, sockmapEndpointsMapName) keySize := 8 valueSize := 4 // flag=1,BPF_F_NO_PREALLOC return newMap(sockmapEndpointsMapName, mapPath, "lpm_trie", 65535, keySize, valueSize, 1) }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33注意
基于
LPM Trie高效检查加速地址，Go侧通过Felix的回调机制维护BPF Map
# 2.2.sockops
sock_ops会挂载到cgroup v2，监听连接建立事件获取socket，将符合条件(访问envoy sidecar)的socket注册到calico_sock_map。// 连接过滤 switch (skops->op) { // connect case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: // accept case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: break; default: return BPF_OK; // 只关心连接建立事件 } // endpoint匹配检查 ip4val_to_lpm(&sip, 32, skops->local_ip4); // calico_sock_map key包装 ip4val_to_lpm(&dip, 32, skops->remote_ip4); // 两端都不是Pod if (NULL == bpf_map_lookup_elem(&calico_sk_endpoints, &dip) && NULL == bpf_map_lookup_elem(&calico_sk_endpoints, &sip)) { return BPF_OK; // 不加速 } // envoy识别 #define ENVOY_IP 0x100007f // 127.0.0.1(小端序: 1.0.0.127 → 0x7f000001 → 注意这里是0x100007f) #define ENVOY_PORT 0x993a // 15001(小端序: 0x3a99) // Envoy--->App, key = {dip,dport,1} if (sip.ip.addr == ENVOY_IP && sport == ENVOY_PORT) { key.envoy_side = 1; key.ip4 = dip.ip.addr; key.port = dport; // App--->Envoy, key = {sip,sport,0} } else { key.ip4 = sip.ip.addr; key.port = sport; key.envoy_side = 0; } // socket加入calico_sock_map err = bpf_sock_hash_update(skops, &calico_sock_map, &key, BPF_ANY);1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
注意
bpf_sock_hash_update是BPF helper函数，会将当前socket作为value存入calico_sock_map
# 2.3.skmsg
sk_msg会挂载到calico_sock_map，应用通过sockmap注册的 socket 发送消息时触发，匹配对端socket执行重定向逻辑。// 基于方向取key,匹配对端socket if (sip == ENVOY_IP && sport == ENVOY_PORT) { // Envoy → App {dip,dport,0} key.ip4 = dip; key.port = dport; key.envoy_side = 0; } else { // App → Envoy {sip,sport,1} // 非App → Envoy流量无法匹配对端socket,相当于不加速 key.ip4 = sip; key.port = sport; key.envoy_side = 1; } // 重定向,消息的元数据和数据指针发到对端的socket接收队列 err = bpf_msg_redirect_hash(msg, &calico_sock_map, &key, BPF_REDIR_INGRESS); return SK_PASS;1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
注意
无论重定向是否成功始终返回
SK_PASS，sockmap匹配不到对端socket内核会自动将消息传递给正常的网络栈处理
# 3.注册
# 3.1.dataplane
dataplane初始化会根据配置尝试创建sockmap_state，负责维护calico_sk_endpoints_v1加速地址及挂载sock_ops/sk_msg。func NewIntDataplaneDriver(config Config) *InternalDataplane { ... // 启用envoy sidecar连接加速 if config.SidecarAccelerationEnabled { // sockmap条件检查(内核版本/小端序) bpf.SupportsSockmap() ... // 初始化sockmapState st, err := NewSockmapState() ... dp.sockmapState = st dp.sockmapState.PopulateCallbacks(callbacks) // 注册回调端点 dp.sockmapState.SetupSockmapAcceleration() // 挂载sock_ops和sk_msg ... } // 未加速或加速失败,尝试清理可能残留的旧sockmap,回退到内核网络栈连接 if dp.sockmapState == nil { st, err := NewSockmapState() ... // 卸载sock_ops和sk_msg st.WipeSockmap(bpf.FindInBPFFSOnly) } ... return dp } func (s *sockmapState) PopulateCallbacks(cbs *common.Callbacks) { cbIDs := []*common.CbID{ // 注册Hook,workload endpoint变更触发 cbs.UpdateWorkloadEndpointV4.Append(s.updateWorkload), cbs.RemoveWorkloadEndpointV4.Append(s.removeWorkload), } // 记录一下,卸载用 s.cbIDs = append(s.cbIDs, cbIDs...) }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
注意
加速失败会尝试创建一个临时
sockmapState执行WipeSockmap()，清理残留的sockmap挂载及BPF Map，避免影响流量
# 3.2.initialize
NewSockmapState()会实例化sockmapState对象，加载eBPF驱动用于后续基于workload endpint更新sockmap内容。type sockmapState struct { bpfLib bpf.BPFDataplane // BPF 操作接口 cbIDs []*common.CbID // 回调注册 ID（用于注销） workloadEndpoints map[string][]string // name -> []CIDR(当前所有端点) } func NewSockmapState() (*sockmapState, error) { // bpfLib初始化 lib, err := bpf.NewBPFLib("/usr/lib/calico/bpf/") ... return &sockmapState{ bpfLib: lib, cbIDs: nil, workloadEndpoints: make(map[string][]string), }, nil } func NewBPFLib(binDir string) (*BPFLib, error) { _, err := exec.LookPath("bpftool") // 检查bpftool存在 bpfDir, err := utils.MaybeMountBPFfs() // 挂载BPF FS(/sys/fs/bpf) cgroupV2Dir, err := utils.MaybeMountCgroupV2() // 挂载cgroup v2 calicoDir := filepath.Join(bpfDir, "calico") // 根目录/sys/fs/bpf/calico sockmapDir := filepath.Join(calicoDir, "sockmap") // sockmap目录/sys/fs/bpf/calico/sockmap xdpDir := filepath.Join(calicoDir, "xdp") // xdp目录/sys/fs/bpf/calico/xdp return &BPFLib{ binDir, bpffsDir, calicoDir, sockmapDir, cgroupV2Dir, xdpDir, }, nil } // Pin路径,NewBPFLib和xdp共用,所以也会生成xdp目录 /sys/fs/bpf/ └── calico/ ├── sockmap/ │ ├── calico_sock_map_v1 # sockhash map │ ├── calico_sk_endpoints_v1 # lpm_trie map │ ├── calico_sockops_v1 # sock_ops program │ └── calico_sk_msg_v1 # sk_msg program └── xdp/ └── ...1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40注意
BPFLib封装了所有对BPF内核对象的操作，底层通过exec.Command调用bpftool完成
# 4.加载
# 4.1.setup
s.SetupSockmapAcceleration()会创建相关BPF Map Pin文件，加载sock prog程序，将程序挂载到cgroup和sockmap。func (s *sockmapState) SetupSockmapAcceleration() error { // 先清理sockmap旧数据(下面动作的逆过程) s.WipeSockmap(bpf.FindByID) // 创建sockmap map pin文件,sock_ops会注册socket s.bpfLib.NewSockmap() ... // 创建endpoints map pin文件,sk_msg加速检查Pod端用 s.bpfLib.NewSockmapEndpointsMap() ... // 加载sockops prog s.bpfLib.LoadSockopsAuto() ... // 加载sk_msg prog s.bpfLib.LoadSkMsgAuto() ... // sk_msg挂载到sockmap s.bpfLib.AttachToSockmap() ... // sockops挂载到cgroup s.bpfLib.AttachToCgroup() ... return nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
注意
BPF Map创建、Sockmap Prog加载及挂载其实就是执行bpftool命令执行系统调用
# 4.2.map创建
calico_sock_map_v1和calico_sk_endpoints_v1创建底层都是调用newMap通用函数执行bpftool命令执行系统调用。func (b *BPFLib) NewSockmap() (string, error) { // mapPath: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 mapPath := filepath.Join(b.sockmapDir, sockMapName) ... return newMap(sockMapName, mapPath, "sockhash", 65535, keySize, valueSize, 0) } func (b *BPFLib) NewSockmapEndpointsMap() (string, error) { // mapPath: /sys/fs/bpf/calico/sockmap/calico_sk_endpoints_v1 mapPath := filepath.Join(b.sockmapDir, sockmapEndpointsMapName) ... return newMap(sockmapEndpointsMapName, mapPath, "lpm_trie", 65535, keySize, valueSize, 1) } func newMap(name, path, kind string, entries, keySize, valueSize, flags int) (string, error) { // Pin文件已存在直接返回 if _, err := os.Stat(path); err == nil { return path, nil } ... args := []string{ "map", "create", path, "type", kind, // sockhash或lpm_trie "key", fmt.Sprintf("%d", keySize), "value", fmt.Sprintf("%d", valueSize), "entries", fmt.Sprintf("%d", entries), "name", name, "flags", fmt.Sprintf("%d", flags), // BPF_F_NO_PREALLOC 等 } // bpftool map create将内容Pin到指定路径 output, err := exec.Command("bpftool", args...).CombinedOutput() ... }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37注意
bpftool map create直接将map Pin到指定路径，无需额外Pin步骤
# 4.3.prog加载
sock_ops和sk_msg加载都需要重用已Pin的maps，通过getSockmapArgs/getSkMsgArgs获取Pin参数配合bpftool生成命令。// sockops加载 func (b *BPFLib) LoadSockops(objPath string) error { // 编译前: /usr/lib/calico/bpf/sockops.o objPath = path.Join(b.binDir, objPath) // 编译后: /sys/fs/bpf/calico/sockmap/calico_sockops_v1 progPath := filepath.Join(b.sockmapDir, sockopsProgName) // Pin文件加载,确保map用的是一个 sockmapArgs, err := b.getSockmapArgs() ... // bpftool prog load <obj_file> <pin_path> type <prog_type> [map name <symbol> pinned <map_path> ...] return b.loadBPF(objPath, progPath, "sockops", sockmapArgs) } // sk_msg加载 func (b *BPFLib) LoadSkMsg(objPath string) error { // 编译前: /usr/lib/calico/bpf/redir.o objPath = path.Join(b.binDir, objPath) // 编译后: /sys/fs/bpf/calico/sockmap/calico_sk_msg_v1 progPath := filepath.Join(b.sockmapDir, skMsgProgName) // Pin文件加载,确保map用的是一个 mapArgs, err := b.getSkMsgArgs() ... // 关联已Pin Map加载prog程序,将程序Pin到指定路径,后续程序工作会用这个map fd return b.loadBPF(objPath, progPath, "sk_msg", mapArgs) } // 通用函数 // sock_ops依赖calico_sock_map_v1和calico_sk_endpoints_v1 func (b *BPFLib) getSockmapArgs() ([]string, error) { // sockmap Pin路径: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 sockmapPath := filepath.Join(b.sockmapDir, sockMapName) // endpoint Pin路径:/sys/fs/bpf/calico/sockmap/calico_sk_endpoints_v1 sockmapEndpointsPath := filepath.Join(b.sockmapDir, sockmapEndpointsMapName) maps := map[string]string{ "calico_sock_map": sockmapPath, "calico_sk_endpoints": sockmapEndpointsPath, } ... for n, p := range maps { mapArgs = append(mapArgs, []string{"map", "name", n, "pinned", p}...) } return mapArgs, nil } // sk_msg仅依赖calico_sock_map_v1 func (b *BPFLib) getSkMsgArgs() ([]string, error) { // sockmap Pin路径: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 sockmapPath := filepath.Join(b.sockmapDir, sockMapName) // key: symbol of the map definition in the XDP program // value: path where the map is pinned maps := map[string]string{ "calico_sock_map": sockmapPath, } ... for n, p := range maps { ... mapArgs = append(mapArgs, []string{"map", "name", n, "pinned", p}...) } return mapArgs, nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
注意
复用
map确保BPF程序加载使用的是之前创建及Pin的文件，避免ELF定义新的map造成数据丢失
# 4.4.attach
sock_ops会附着到cgroup根，将匹配socket加入关联的sockmap，sk_msg附着到sockmap实现加速流量劫持和直接转发。// sk_msg挂在sockmap func (b *BPFLib) AttachToSockmap() error { // sockmap路径: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 mapPath := filepath.Join(b.sockmapDir, sockMapName) // sk_msg Pin路径: /sys/fs/bpf/calico/sockmap/calico_sk_msg_v1 progPath := filepath.Join(b.sockmapDir, skMsgProgName) ... // bpftool prog attach pinned <progPath> msg_verdict pinned <mapPath> output, err := exec.Command(prog, args...).CombinedOutput() ... return nil } // sock_ops挂在cgroup func (b *BPFLib) AttachToCgroup() error { // sock_ops Pin路径: /sys/fs/bpf/calico/sockmap/calico_sockops_v1 progPath := filepath.Join(b.sockmapDir, sockopsProgName) ... // bpftool cgroup attach <cgroupPath> sock_ops pinned <progPath> output, err := exec.Command(prog, args...).CombinedOutput() ... return nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27注意
attach基于bpftool将Pin出来的prog程序附着到Pin出来的BPF Map或cgroup根目录
# 5.端点
# 5.1.callback
workload endpoint变化触发callback，基于内存期望的cidr和eBPF endpoint map对比，增量更新维护sockmap加速地址。func (s *sockmapState) updateWorkload(old, new *proto.WorkloadEndpoint) { // wep cidr集合 s.workloadEndpoints[new.Name] = new.Ipv4Nets desired := s.flattenWorkloadEndpoints() // 同步 s.processWorkloadUpdates(desired) ... } func (s *sockmapState) removeWorkload(old *proto.WorkloadEndpoint) { // wep cidr集合 delete(s.workloadEndpoints, old.Name) desired := s.flattenWorkloadEndpoints() // 同步 s.processWorkloadUpdates(desired) ... } // desired vs BPF Map func (s *sockmapState) processWorkloadUpdates(desired set.Set[string]) error { ... // bpftool --json --pretty map dump pinned <calico_sk_endpoints_v1> cidrs, err := s.bpfLib.DumpSockmapEndpointsMap(bpf.IPFamilyV4) ... // calico_sk_endpoints_v1保存的cidr for _, rawCIDR := range cidrs { ipnet := rawCIDR.ToIPNet() current.Add(ipnet.String()) } // 对比 toAdd := setDifference[string](desired, current) toDrop := setDifference[string](current, desired) // 清理多余的cidr toDrop.Iter(func(cidr string) error { ip, mask, err := bpf.MemberToIPMask(cidr) ... s.bpfLib.RemoveItemSockmapEndpointsMap(*ip, mask) ... return nil }) // 注册新增的cidr toAdd.Iter(func(cidr string) error { ip, mask, err := bpf.MemberToIPMask(cidr) ... s.bpfLib.UpdateSockmapEndpoints(*ip, mask) ... return nil }) return nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
注意
processWorkloadUpdates更新的是calico_sk_endpoints_v1索引表，sock_ops用于Pod端点检查，sockmap由底层维护
# 5.2.map更新
wep对象更新会调整calico_sk_endpoints_v1的cidr内容集合，这份数据由sock_ops读取用于检查socket连接至少一端是否属于Pod。func (b *BPFLib) UpdateSockmapEndpoints(ip net.IP, mask int) error { // endpoint Pin路径: /sys/fs/bpf/calico/sockmap/calico_sk_endpoints_v1 mapPath := filepath.Join(b.sockmapDir, sockmapEndpointsMapName) ... // cidr小端序序列化 cidr := fmt.Sprintf("%s/%d", ip.String(), mask) hexKey, err := CidrToHex(cidr) ... hexValue := []string{"01", "00", "00", "00"} prog := "bpftool" args := []string{ "map", "update", "pinned", mapPath, "key", "hex" } args = append(args, hexKey...) args = append(args, "value", "hex") args = append(args, hexValue...) // bpftool map update pinned <calico_sk_endpoints_v1> key hex <hexKey> value hex 01 00 00 00 output, err := exec.Command(prog, args...).CombinedOutput() ... return nil } func (b *BPFLib) RemoveItemSockmapEndpointsMap(ip net.IP, mask int) error { mapPath := filepath.Join(b.sockmapDir, sockmapEndpointsMapName) ... // cidr小端序序列化 cidr := fmt.Sprintf("%s/%d", ip.String(), mask) hexKey, err := CidrToHex(cidr) ... prog := "bpftool" args := []string{ "--json", "--pretty", "map", "delete", "pinned", mapPath, "key", "hex" } args = append(args, hexKey...) // bpftool --json --pretty map delete pinned <calico_sk_endpoints_v1> key hex <hexKey> output, err := exec.Command(prog, args...).CombinedOutput() ... return nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
小端序
10.0.0.1/32会拆解为20 00 00 00(32)和0a 00 00 01(10.0.0.1)，两者拼接作为hesKey
# 6.清理
# 6.1.widesock
s.WipeSockmap()清理顺序与初始化顺序相反，会先detach解除挂载 ，清理程序，最后执行bpftool命令清理Pin Map。func (s *sockmapState) WipeSockmap(mode bpf.FindObjectMode) { // sockmap分离sk_msg s.bpfLib.DetachFromSockmap(mode) // cgroup分离sock_ops s.bpfLib.DetachFromCgroup(mode) // 清理sock_ops Pin文件 s.bpfLib.RemoveSockops() // 清理sk_msg Pin文件 s.bpfLib.RemoveSkMsg() // 清理calico_sk_endpoints_v1 s.bpfLib.RemoveSockmapEndpointsMap() // 清理calico_sock_map_v1 s.bpfLib.RemoveSockmap(mode) }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
注意
prog/map清理本质就是将Pin文件删掉，核心还是detach，会基于正常模式和回退模式清理，避免仅基于Pin文件卸载失败
# 6.2.map清理
calico_sk_endpoints_v1仅记录cidr集合，直接清理文件不会造成资源泄漏，sockmap持有socket引用，需要先解引用再删。func (b *BPFLib) RemoveSockmap(mode FindObjectMode) error { // sockmap Pin路径: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 mapPath := filepath.Join(b.sockmapDir, sockMapName) defer os.Remove(mapPath) // sockmap内容清理 if err := clearSockmap([]string{"pinned", mapPath}); err != nil { ... // mapPath异常,bpftool map show遍历找type==sockhash m, err := b.getSockMap() ... if m != nil { clearSockmap([]string{"id", fmt.Sprintf("%d", m.Id)}) ... } } return nil } func clearSockmap(mapArgs []string) error { ... for { ... // 执行bpftool map --json getnext pinned <calico_sock_map_v1> output, _ := exec.Command(prog, args...).CombinedOutput() json.Unmarshal(output, &e) ... // 删除当前key keyArgs := jsonKeyToArgs(e.NextKey) ... // 正常模式: bpftool map --json delete pinned <calico_sock_map_v1> key hex <keyArgs> // FindByID模式: bpftool map --json delete id <sockmap_id> key hex <keyArgs> output, err = exec.Command(prog, args...).CombinedOutput() ... } }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
注意
1.
sockmap value不是普通数据，写的是struct sock*指针，sockops更新sockmap内核会进行socket引用计数2.直接清理
sockmap Pin文件，内核的socket引用技术不会变，视为仍有引用，此时会造成socket无法释放
# 6.3.detach
sock_ops/sk_msg会执行bpftool由sockmap和cgroup解除附着，提供基于Pin路径正常卸载和FindByID异常卸载两种方式。func (b *BPFLib) DetachFromSockmap(mode FindObjectMode) error { // sockmap Pin: /sys/fs/bpf/calico/sockmap/calico_sock_map_v1 mapPath := filepath.Join(b.sockmapDir, sockMapName) // prog Pin: /sys/fs/bpf/calico/sockmap/calico_sk_msg_v1 progPath := filepath.Join(b.sockmapDir, skMsgProgName) ... // bpftool prog detach pinned <progPath> msg_verdict pinned <mapPath> output, err := exec.Command(prog, args...).CombinedOutput() if err != nil { ... // FindByID模式,bpftool prog show遍历找type=sk_msg progID, err2 := b.getSkMsgID() ... if progID >= 0 { // bpftool map show遍历找type==sockhash mapID, err2 := b.getSockMapID(progID) ... // bpftool prog detach id <progID> msg_verdict id <mapID> output2, err2 := exec.Command(prog, args...).CombinedOutput() ... } } return nil } func (b *BPFLib) DetachFromCgroup(mode FindObjectMode) error { // prog Pin: /sys/fs/bpf/calico/sockmap/calico_sockops_v1 progPath := filepath.Join(b.sockmapDir, sockopsProgName) ... // bpftool cgroup detach <cgroupPath> sock_ops pinned <progPath> output, err := exec.Command(prog, args...).CombinedOutput() if err != nil { ... // FindByID模式 progID, err2 := b.getAttachedSockopsID() ... if progID >= 0 { ... // bpftool cgroup detach <cgroupPath> sock_ops id <progID> output2, err2 := exec.Command(prog, args...).CombinedOutput() ... } } return nil }1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
注意
1.
Pin文件可能因进程崩溃或手动删除丢失，内核中的BPF程序和Map仍然存在，会导致残留对象2.
DetachFromSockmap/DetachFromCgroup/RemoveSockmap三个方法都实现两级查找的容错机制