BPF之巅：洞悉Linux系统和应用性能

定　　价：199 元

• 作者：（美）Brendan Gregg（布兰登.格雷格）
• 出版时间：2020/11/1
• ISBN：9787121399725
• 出 版 社：电子工业出版社

• 中图法分类：TP316.85 
• 页码：856
• 版次：01
• 开本：16K

内容简介

基于BPF的性能优化工具提供了前所未有的系统级和应用程序级的观察能力，使用这些工具可以优化性能、调试代码、增强安全性、降低成本。本书是使用这些观察工具的全面指南。本书作为全面介绍 BPF 技术的图书，从 BPF 技术的起源到未来发展方向都有涵盖，不仅全面介绍了 BPF 的编程模型，还完整介绍了两个主要的 BPF 前端编程框架 — BCC 和 bpftrace，更给出了一系列实现范例，全面展示了 BPF技术的实际能力和未来发展前景，以及使用BPF工具优化性能、修复问题、探索线上系统的内部情况。本书的另一个关注方向是 Linux 系统性能和应用程序性能的调优。本书全面介绍了系统性能调优的策略、工具与实践案例，不仅介绍了对应的 BPF 工具，还着重介绍了这些工具如何与 Linux 传统性能工具进行互补，这样读者可以有选择地进行使用。本书介绍的工具小巧精致，且包含了简单易读的源代码，这就是 BPF 技术的魅力所在 ：安全、高效、快捷的系统扩展力。未来 BPF 技术在 Linux 中的应用场景会越来越多，越来越重要。希望本书能在大家学习这项技术并关注它的发展时提供一定的便利。本书是系统管理员、应用程序开发者、运维人员，以及其他IT从业者在企业内部或云上使用各种Linux发行版时都可参考的资料。

作者介绍

Netflix 高级性能工程师 Brendan Gregg 是 BPF（eBPF）的主要贡献者，他帮助开发和维护了两个主要的 BPF 前端框架，开创了 BPF 用于可观测性的先河，并创建了数十种基于 BPF 的性能分析工具。他编著的畅销书有《性能之巅：洞悉系统、企业与云计算》。
译者介绍孙宇聪：现任 Facebook 运维工程师经理，曾在谷歌工作多年，任谷歌高级SRE（Senior Site Reliblity Engineer），Coding.net 前技术负责人，译有《SRE：Google运维解密》《架构整洁之道》等经典畅销技术图书。吕宏利，资深SRE，现任职于谷歌基础架构部。之前曾负责谷歌搜索广告和内容广告系统运维工作，有多年分布式系统研发与运维经验。对运维工具平台建设、监控、应用性能跟踪及分析、数据化运维等方面有深入的研究。刘晓舟，毕业于北京大学计算机系，现供职于字节跳动公司系统部，任系统架构师。他在字节跳动主持构建了基于 eBPF 的大规模性能分析和网络监控诊断平台，闲暇时间也在相关开源社区提交代码。在加入字节跳动之前，他有 10 年国家部委电子政务和大数据研究经历。

目 录

第1章　引　言.................................................................................................................1
1.1　BPF和eBPF是什么 .............................................................................................. 1
1.2　跟踪、嗅探、采样、剖析和可观测性分别是什么 .......................................... 2
1.3　BCC、bpftrace和IO Visor ................................................................................... 3
1.4　初识BCC：快速上手 .......................................................................................... 4
1.5　BPF跟踪的能见度 ............................................................................................... 7
1.6　动态插桩：kprobes和uprobes ............................................................................ 8
1.7　静态插桩：tracepoint和USDT ........................................................................... 9
1.8　初识bpftrace：跟踪open() ................................................................................ 10
1.9　再回到BCC：跟踪open() ................................................................................. 13
1.10　小结 .................................................................................................................. 15
第2章　技术背景...........................................................................................................16
2.1　图释BPF ............................................................................................................. 16
2.2　BPF ..................................................................................................................... 17
2.3　扩展版BPF ......................................................................................................... 18
2.3.1　为什么性能工具需要 BPF 技术 ......................................................... 21
2.3.2　BPF 与内核模块的对比 ......................................................................23
2.3.3　编写 BPF 程序 .....................................................................................23
2.3.4　使用 BPF 查看指令集 ：bpftool ..........................................................24
2.3.5　使用 bpftrace 查看 BPF 指令集 .......................................................... 32
2.3.6　BPF API ................................................................................................ 33
2.3.7　BPF 并发控制 ......................................................................................37
2.3.8　BPF sysfs 接口 ..................................................................................... 38
2.3.9　BPF 类型格式 ......................................................................................38
2.3.10　BPF CO-RE ........................................................................................ 39
2.3.11　BPF 的局限性 ..................................................................................... 40
2.3.12　BPF 扩展阅读资料 ............................................................................40
2.4　调用栈回溯 ........................................................................................................ 41
2.4.1　基于帧指针的调用栈回溯 ................................................................... 41
2.4.2　调试信息 ............................................................................................... 42
2.4.3　最后分支记录 ....................................................................................... 43
2.4.4　ORC ...................................................................................................... 43
2.4.5　符号 ....................................................................................................... 43
2.4.6　扩展阅读 ............................................................................................... 43
2.5　火焰图 ................................................................................................................ 44
2.5.1　调用栈信息 ........................................................................................... 44
2.5.2　对调用栈信息的剖析 ........................................................................... 44
2.5.3　火焰图 ................................................................................................... 45
2.5.4　火焰图的特性 ....................................................................................... 47
2.5.5　火焰图的变体 ....................................................................................... 48
2.6　事件源 ................................................................................................................ 48
2.7　kprobes ............................................................................................................... 49
2.7.1　kprobes 是如何工作的 ......................................................................... 49
2.7.2　kprobes 接口 ......................................................................................... 51
2.7.3　BPF 和 kprobes .....................................................................................51
2.7.4　关于 kprobes 的更多内容 .................................................................... 53
2.8　uprobes ............................................................................................................... 53
2.8.1　uprobes 是如何工作的 ......................................................................... 53
2.8.2　uprobes 接口 ......................................................................................... 55
2.8.3　BPF 与 uprobes .....................................................................................55
2.8.4　uprobes 的开销和未来的工作 ............................................................. 56
2.8.5　扩展阅读 ............................................................................................... 57
2.9　跟踪点 ................................................................................................................ 57
2.9.1　如何添加跟踪点 ................................................................................... 58
2.9.2　跟踪点的工作原理 ............................................................................... 59
2.9.3　跟踪点的接口 ....................................................................................... 60
2.9.4　跟踪点和 BPF ...................................................................................... 61
2.9.5　BPF 原始跟踪点 ..................................................................................62
2.9.6　扩展阅读 ............................................................................................... 62
2.10　USDT ............................................................................................................... 62
2.10.1　添加 USDT 探针 ................................................................................ 63
2.10.2　USDT 是如何工作的 ......................................................................... 65
2.10.3　BPF 与 USDT ..................................................................................... 66
2.10.4　USDT 的更多信息 ............................................................................. 66
2.11　动态USDT ........................................................................................................ 66
2.12　性能监控计数器 .............................................................................................. 68
2.12.1　PMC 的模式 .......................................................................................68
2.12.2　PEBS ................................................................................................... 69
2.12.3　云计算 ................................................................................................. 69
2.13　perf_events ....................................................................................................... 69
2.14　小结 .................................................................................................................. 70
第3章　性能分析...........................................................................................................71
3.1　概览 .................................................................................................................... 71
3.1.1　目标 ....................................................................................................... 71
3.1.2　分析工作 ............................................................................................... 72
3.1.3　多重性能问题 ....................................................................................... 73
3.2　性能分析方法论 ................................................................................................ 73
3.2.1　业务负载画像 ....................................................................................... 74
3.2.2　下钻分析 ............................................................................................... 75
3.2.3　USE 方法论 .......................................................................................... 76
3.2.4　检查清单法 ........................................................................................... 77
3.3　Linux 60秒分析 ................................................................................................. 77
3.3.1　uptime ................................................................................................... 77
3.3.2　dmesg | tail ............................................................................................ 78
3.3.3　vmstat 1 ................................................................................................. 78
3.3.4　mpstat -P ALL 1 ....................................................................................79
3.3.5　pidstat 1 ................................................................................................. 80
3.3.6　iostat -xz 1 ............................................................................................. 80
3.3.7　free -m ................................................................................................... 82
3.3.8　sar -n DEV 1 .........................................................................................82
3.3.9　sar -n TCP,ETCP 1 ................................................................................ 83
3.3.10　top ....................................................................................................... 83
3.4　BCC工具检查清单 ............................................................................................ 84
3.4.1　execsnoop .............................................................................................. 84
3.4.2　opensnoop ............................................................................................. 85
3.4.3　ext4slower ............................................................................................. 85
3.4.4　biolatency .............................................................................................. 86
3.4.5　biosnoop ................................................................................................ 86
3.4.6　cachestat ................................................................................................ 87
3.4.7　tcpconnect .............................................................................................87
3.4.8　tcpaccept ............................................................................................... 87
3.4.9　tcpretrans ............................................................................................... 88
3.4.10　runqlat ................................................................................................. 88
3.4.11　profile .................................................................................................. 89
3.5　小结 .................................................................................................................... 90
第4章　BCC..................................................................................................................91
4.1　BCC的组件 ........................................................................................................ 92
4.2　BCC的特性 ........................................................................................................ 92
4.2.1　BCC 的内核态特性 .............................................................................. 92
4.2.2　BCC 的用户态特性 .............................................................................. 93
4.3　安装BCC ............................................................................................................ 94
4.3.1　内核要求 ............................................................................................... 94
4.3.2　Ubuntu ................................................................................................... 94
4.3.3　RHEL .................................................................................................... 95
4.3.4　其他发行版 ........................................................................................... 95
4.4　BCC的工具 ........................................................................................................ 96
4.4.1　重点工具 ............................................................................................... 96
4.4.2　工具的特点 ........................................................................................... 97
4.4.3　单一用途工具 ....................................................................................... 98
4.4.4　多用途工具 ........................................................................................... 99
4.5　funccount .......................................................................................................... 100
4.5.1　funccount 的示例 ................................................................................ 101
4.5.2　funccount 的语法 ................................................................................ 103
4.5.3　funccount 的单行程序 ........................................................................ 103
4.5.4　funccount 的帮助信息 ........................................................................ 104
4.6　stackcount ......................................................................................................... 105
4.6.1　stackcount 的示例 .............................................................................. 105
4.6.2　stackcount 的火焰图 .......................................................................... 107
4.6.3　stackcount 残缺的调用栈 .................................................................. 108
4.6.4　stackcount 的语法 .............................................................................. 108
4.6.5　stackcount 的单行程序 ...................................................................... 109
4.6.6　stackcount 的帮助信息 ...................................................................... 109
4.7　trace .................................................................................................................. 110
4.7.1　trace 的示例 .........................................................................................111
4.7.2　trace 的语法 .........................................................................................111
4.7.3　trace 的单行程序 ................................................................................ 113
4.7.4　trace 的结构体 .................................................................................... 113
4.7.5　trace 调试文件描述符泄露问题 ........................................................ 114
4.7.6　trace 的帮助信息 ................................................................................ 115
4.8　argdist ............................................................................................................... 117
4.8.1　argdist 的语法 ..................................................................................... 118
4.8.2　argdist 的单行程序 ............................................................................. 119
4.8.3　argdist 的帮助信息 ............................................................................. 119
4.9　工具文档 .......................................................................................................... 121
4.9.1　man 帮助文档 ：opensnoop ............................................................... 121
4.9.2　示例文件 ：opensnoop ....................................................................... 125
4.10　开发BCC工具 ................................................................................................ 126
4.11　BCC的内部实现 ............................................................................................ 127
4.12　BCC的调试 .................................................................................................... 128
4.12.1　printf() 调试 ...................................................................................... 129
4.12.2　BCC 调试输出 ..................................................................................131
4.12.3　BCC 的调试标志位 .......................................................................... 132
4.12.4　bpflist ................................................................................................. 133
4.12.5　bpftool ............................................................................................... 134
4.12.6　dmesg ................................................................................................ 134
4.12.7　重置事件 ........................................................................................... 134
4.13　小结 ................................................................................................................ 136
第5章　bpftrace...........................................................................................................137
5.1　bpftrace的组件 ................................................................................................. 138
5.2　bpftrace的特性 ................................................................................................. 139
5.2.1　bpftrace 的事件源 .............................................................................. 139
5.2.2　bpftrace 的动作 .................................................................................. 139
5.2.3　bpftrace 的一般特性 .......................................................................... 140
5.2.4　bpftrace 与其他观测工具的比较 ...................................................... 140
5.3　bpftrace的安装 ................................................................................................. 141
5.3.1　内核版本要求 ..................................................................................... 141
5.3.2　Ubuntu ................................................................................................. 142
5.3.3　Fedora ................................................................................................. 142
5.3.4　构建后的安装步骤 ............................................................................. 143
5.3.5　其他发行版 ......................................................................................... 143
5.4　bpftrace工具 ..................................................................................................... 143
5.4.1　重点工具 ............................................................................................. 144
5.4.2　工具特征 ............................................................................................. 144
5.4.3　工具的运行 ......................................................................................... 145
5.5　bpftrace单行程序 ............................................................................................. 145
5.6　bpftrace的文档 ................................................................................................. 146
5.7　bpftrace编程 ..................................................................................................... 146
5.7.1　用法 ..................................................................................................... 147
5.7.2　程序结构 ............................................................................................. 148
5.7.3　注释 ..................................................................................................... 148
5.7.4　探针格式 ............................................................................................. 149
5.7.5　探针通配符 ......................................................................................... 149
5.7.6　过滤器 ................................................................................................. 150
5.7.7　动作 ..................................................................................................... 150
5.7.8　Hello, World! .......................................................................................151
5.7.9　函数 ..................................................................................................... 151
5.7.10　变量 ................................................................................................... 152
5.7.11　映射表函数 ....................................................................................... 153
5.7.12　对 vfs_read() 计时 ............................................................................ 154
5.8　bpftrace的帮助信息 ......................................................................................... 155
5.9　bpftrace的探针类型 ......................................................................................... 157
5.9.1　tracepoint ............................................................................................. 157
5.9.2　usdt ...................................................................................................... 159
5.9.3　kprobe 和 kretprobe ............................................................................160
5.9.4　uprobe 和 uretprobe ............................................................................ 160
5.9.5　software 和 hardware ..........................................................................161
5.9.6　profile 和 interval ................................................................................ 162
5.10　bpftrace的控制流 ........................................................................................... 163
5.10.1　过滤器 ............................................................................................... 163
5.10.2　三元操作符 ....................................................................................... 163
5.10.3　if 语句 ............................................................................................... 163
5.10.4　循环展开 ........................................................................................... 164
5.11　bpftrace的运算符 ........................................................................................... 164
5.12　bpftrace的变量 ............................................................................................... 165
5.12.1　内置变量 ........................................................................................... 165
5.12.2　内置变量 ：pid、comm 和 uid ........................................................166
5.12.3　内置变量 ：kstack 和 ustack ............................................................166
5.12.4　内置变量 ：位置参数 ....................................................................... 168
5.12.5　临时变量 ........................................................................................... 169
5.12.6　映射表变量 ....................................................................................... 169
5.13　bpftrace的函数 ............................................................................................... 170
5.13.1　printf() ............................................................................................... 171
5.13.2　join() .................................................................................................. 172
5.13.3　str() .................................................................................................... 173
5.13.4　kstack() 和 ustack() ........................................................................... 173
5.13.5　ksym() 和 usym() .............................................................................. 174
5.13.6　kaddr() 和 uaddr() .............................................................................175
5.13.7　system() ............................................................................................. 176
5.13.8　exit() .................................................................................................. 176
5.14　bpftrace映射表的操作函数 ........................................................................... 177
5.14.1　count() ............................................................................................... 177
5.14.2　sum()、avg()、min() 和 max() ........................................................178
5.14.3　hist() .................................................................................................. 179
5.14.4　lhist() ................................................................................................. 180
5.14.5　delete() .............................................................................................. 181
5.14.6　clear() 和 zero() .................................................................................181
5.14.7　print() ................................................................................................ 182
5.15　bpftrace的下一步工作 ................................................................................... 183
5.15.1　显式区分地址模式 ........................................................................... 183
5.15.2　其他扩展 ........................................................................................... 184
5.15.3　ply ..................................................................................................... 184
5.16　bpftrace的内部运作 ....................................................................................... 185
5.17　bpftrace的调试 ............................................................................................... 186
5.17.1　printf() 调试 ...................................................................................... 186
5.17.2　调试模式 ........................................................................................... 187
5.17.3　详情模式 ........................................................................................... 188
5.18　小结 ................................................................................................................ 190
第6章　CPU................................................................................................................191
6.1　背景知识 .......................................................................................................... 192
6.1.1　CPU 基础知识 ....................................................................................192
6.1.2　BPF 的分析能力 ................................................................................194
6.1.3　分析策略 ............................................................................................. 196
6.2　传统工具 .......................................................................................................... 197
6.2.1　内核统计 ............................................................................................. 197
6.2.2　硬件统计 ............................................................................................. 200
6.2.3　硬件采样 ............................................................................................. 202
6.2.4　定时采样 ............................................................................................. 203
6.2.5　事件统计与事件跟踪 ......................................................................... 207
6.3　BPF工具 ........................................................................................................... 210
6.3.1　execsnoop ............................................................................................ 211
6.3.2　exitsnoop ............................................................................................. 214
6.3.3　runqlat ................................................................................................. 215
6.3.4　runqlen ................................................................................................ 219
6.3.5　runqslower ........................................................................................... 222
6.3.6　cpudist ................................................................................................. 223
6.3.7　cpufreq ................................................................................................ 224
6.3.8　profile .................................................................................................. 227
6.3.9　offcputime ........................................................................................... 232
6.3.10　syscount ............................................................................................. 236
6.3.11　argdist 和 trace ..................................................................................239
6.3.12　funccount ........................................................................................... 242
6.3.13　softirqs ............................................................................................... 244
6.3.14　hardirqs ............................................................................................. 245
6.3.15　smpcalls ............................................................................................. 246
6.3.16　llcstat ................................................................................................. 250
6.3.17　其他工具 ........................................................................................... 251
6.4　BPF单行程序 ................................................................................................... 251
6.4.1　BCC 工具 ............................................................................................251
6.4.2　bpftrace 版本 ...................................................................................... 252
6.5　可选练习 .......................................................................................................... 253
6.6　小结 .................................................................................................................. 254
第7章　内存................................................................................................................255
7.1　背景知识 .......................................................................................................... 256
7.1.1　内存基础知识 ..................................................................................... 256
7.1.2　BPF 的分析能力 ................................................................................260
7.1.3　分析策略 ............................................................................................. 262
7.2　传统工具 .......................................................................................................... 263
7.2.1　内核日志 ............................................................................................. 263
7.2.2　内核统计信息 ..................................................................................... 264
7.2.3　硬件统计和硬件采样 ......................................................................... 268
7.3　BPF工具 ........................................................................................................... 269
7.3.1　oomkill ................................................................................................ 270
7.3.2　memleak .............................................................................................. 271
7.3.3　mmapsnoop ......................................................................................... 274
7.3.4　brkstack ............................................................................................... 275
7.3.5　shmsnoop ............................................................................................ 277
7.3.6　faults .................................................................................................... 277
7.3.7　ffaults .................................................................................................. 280
7.3.8　vmscan ................................................................................................ 281
7.3.9　drsnoop ................................................................................................ 284
7.3.10　swapin ............................................................................................... 285
7.3.11　hfaults ................................................................................................ 287
7.3.12　其他工具 ........................................................................................... 287
7.4　BPF单行程序 ................................................................................................... 288
7.4.1　BCC ..................................................................................................... 288
7.4.2　bpftrace ............................................................................................... 288
7.5　可选练习 .......................................................................................................... 289
7.6　小结 .................................................................................................................. 290
第8章　文件系统.........................................................................................................291
8.1　背景知识 .......................................................................................................... 292
8.1.1　文件系统基础知识 ............................................................................. 292
8.1.2　BPF 的分析能力 ................................................................................294
8.1.3　分析策略 ............................................................................................. 295
8.2　传统工具 .......................................................................................................... 296
8.2.1　df ......................................................................................................... 297
8.2.2　mount .................................................................................................. 297
8.2.3　strace ................................................................................................... 298
8.2.4　perf ...................................................................................................... 298
8.2.5　fatrace .................................................................................................. 301
8.3　BPF工具 ........................................................................................................... 302
8.3.1　opensnoop ........................................................................................... 303
8.3.2　statsnoop ............................................................................................. 306
8.3.3　syncsnoop ............................................................................................ 308
8.3.4　mmapfiles ............................................................................................ 309
8.3.5　scread .................................................................................................. 311
8.3.6　fmapfault ............................................................................................. 312
8.3.7　filelife .................................................................................................. 313
8.3.8　vfsstat .................................................................................................. 315
8.3.9　vfscount ............................................................................................... 317
8.3.10　vfssize ............................................................................................... 318
8.3.11　fsrwstat .............................................................................................. 320
8.3.12　fileslower ........................................................................................... 322
8.3.13　filetop ................................................................................................ 325
8.3.14　writesync ........................................................................................... 327
8.3.15　filetype ............................................................................................... 328
8.3.16　cachestat ............................................................................................ 331
8.3.17　writeback ........................................................................................... 334
8.3.18　dcstat ................................................................................................. 336
8.3.19　dcsnoop ............................................................................................. 338
8.3.20　mountsnoop ....................................................................................... 340
8.3.21　xfsslower ........................................................................................... 341
8.3.22　xfsdist ................................................................................................ 342
8.2.23　ext4dist .............................................................................................. 345
8.3.24　icstat .................................................................................................. 348
8.3.25　bufgrow ............................................................................................. 350
8.3.26　readahead .......................................................................................... 351
8.3.27　其他工具 ........................................................................................... 353
8.4　BPF单行程序 ................................................................................................... 353
8.4.1　BCC ..................................................................................................... 353
8.4.2　bpftrace ............................................................................................... 354
8.4.3　BPF 单行程序示范 ............................................................................356
8.5　可选练习 .......................................................................................................... 359
8.6　小结 .................................................................................................................. 360
第9章　磁盘I/O............................................................................................................361
9.1　背景知识 .......................................................................................................... 362
9.1.1　磁盘系统基础知识 ............................................................................. 362
9.1.2　BPF 的分析能力 ................................................................................365
9.1.3　分析策略 ............................................................................................. 366
9.2　传统工具 .......................................................................................................... 367
9.2.1　iostat .................................................................................................... 367
9.2.2　perf ...................................................................................................... 369
9.2.3　blktrace ................................................................................................ 370
9.2.4　SCSI 日志 ...........................................................................................371
9.3　BPF工具 ........................................................................................................... 372
9.3.1　biolatency ............................................................................................373
9.3.2　biosnoop .............................................................................................. 379
9.3.3　biotop .................................................................................................. 383
9.3.4　bitesize ................................................................................................ 384
9.3.5　seeksize ............................................................................................... 386
9.3.6　biopattern ............................................................................................ 388
9.3.7　biostacks .............................................................................................. 390
9.3.8　bioerr ................................................................................................... 393
9.3.9　mdflush ................................................................................................ 395
9.3.10　iosched .............................................................................................. 397
9.3.11　scsilatency ......................................................................................... 399
9.3.12　scsiresult ............................................................................................ 401
9.3.13　nvmelatency ......................................................................................403
9.4　BPF单行程序 ................................................................................................... 406
9.4.1　BCC ..................................................................................................... 406
9.4.2　bpftrace ............................................................................................... 407
9.4.3　BPF 单行程序示范 ............................................................................408
9.5　可选练习 .......................................................................................................... 409
9.6　小结 .................................................................................................................. 410
第10章　网络..............................................................................................................411
10.1　背景知识 ........................................................................................................ 412
10.1.1　网络基础知识 ................................................................................... 412
10.1.2　BPF 的分析能力 ..............................................................................419
10.1.3　分析策略 ........................................................................................... 421
10.1.4　常见的跟踪错误 ............................................................................... 421
10.2　传统工具 ........................................................................................................ 422
10.2.1　ss ....................................................................................................... 423
10.2.2　ip ....................................................................................................... 424
10.2.3　nstat ................................................................................................... 425
10.2.4　netstat ................................................................................................ 425
10.2.5　sar ...................................................................................................... 428
10.2.6　nicstat ................................................................................................ 429
10.2.7　ethtool ............................................................................................... 429
10.2.8　tcpdump ............................................................................................. 431
10.2.9　/proc .................................................................................................. 432
10.3　BPF工具 ......................................................................................................... 433
10.3.1　sockstat .............................................................................................. 435
10.3.2　sofamily ............................................................................................. 437
10.3.3　soprotocol .......................................................................................... 440
10.3.4　soconnect ........................................................................................... 442
10.3.5　soaccept .............................................................................................445
10.3.6　socketio ............................................................................................. 447
10.3.7　socksize ............................................................................................. 450
10.3.8　sormem .............................................................................................. 452
10.3.9　soconnlat ........................................................................................... 455
10.3.10　so1stbyte ......................................................................................... 459
10.3.11　tcpconnect ....................................................................................... 461
10.3.12　tcpaccept .........................................................................................464
10.3.13　tcplife .............................................................................................. 467
10.3.14　tcptop ............................................................................................... 472
10.3.15　tcpsnoop .......................................................................................... 473
10.3.16　tcpretrans .........................................................................................474
10.3.17　tcpsynbl ........................................................................................... 477
10.3.18　tcpwin .............................................................................................. 479
10.3.19　tcpnagle ...........................................................................................481
10.3.20　udpconnect ...................................................................................... 483
10.3.21　gethostlatency ................................................................................. 485
10.3.22　ipecn ................................................................................................ 487
10.3.23　superping ......................................................................................... 488
10.3.24　qdisc-fq ........................................................................................... 491
10.3.25　qdisc-cbq、qdisc-cbs、qdisc-codel、qdisc-fq_codel、qdisc-red、qdisc-tbf ....................................................................... 493
10.3.26　netsize ............................................................................................. 495
10.3.27　nettxlat ............................................................................................. 498
10.3.28　skbdrop ............................................................................................ 500
10.3.29　skblife .............................................................................................. 503
10.3.30　ieee80211scan .................................................................................505
10.3.31　其他工具 ......................................................................................... 507
10.4　BPF单行程序 ................................................................................................. 507
10.4.1　BCC ................................................................................................... 507
10.4.2　bpftrace ............................................................................................. 508
10.4.3　BPF 单行程序示范 ..........................................................................510
10.5　可选练习 ........................................................................................................ 513
10.6　小结 ................................................................................................................ 515
第11章　安全..............................................................................................................516
11.1　背景知识 ........................................................................................................ 516
11.1.1　BPF 的分析能力 ............................................................................... 517
11.1.2　无特权 BPF 用户 .............................................................................. 521
11.1.3　配置 BPF 安全策略 .......................................................................... 521
11.1.4　分析策略 ........................................................................................... 523
11.2　BPF工具 ......................................................................................................... 523
11.2.1　execsnoop .......................................................................................... 524
11.2.2　elfsnoop ............................................................................................. 524
11.2.3　modsnoop .......................................................................................... 526
11.2.4　bashreadline....................................................................................... 527
11.2.5　shellsnoop .......................................................................................... 528
11.2.6　ttysnoop ............................................................................................. 530
11.2.7　opensnoop ......................................................................................... 532
11.2.8　eperm ................................................................................................. 532
11.2.9　tcpconnect 和 tcpaccept .................................................................... 534
11.2.10　tcpreset ............................................................................................ 534
11.2.11　capable ............................................................................................. 536
11.2.12　setuids .............................................................................................. 540
11.3　BPF单行程序 ................................................................................................. 542
11.3.1　BCC ................................................................................................... 542
11.3.2　bpftrace ..............................................................................................543
11.3.3　BPF 单行程序示范 ........................................................................... 543
11.4　小结 ................................................................................................................ 544
第12章　编程语言.......................................................................................................545
12.1　背景知识 ........................................................................................................ 545
12.1.1　编译型语言 ....................................................................................... 546
12.1.2　即时编译型语言 ............................................................................... 547
12.1.3　解释型语言 ....................................................................................... 548
12.1.4　BPF 的分析能力 ..............................................................................549
12.1.5　分析策略 ........................................................................................... 550
12.1.6　BPF 工具 ..........................................................................................550
12.2　C ..................................................................................................................... 551
12.2.1　C 函数符号 ....................................................................................... 552
12.2.2　C 调用栈 ........................................................................................... 555
12.2.3　C 函数跟踪 ....................................................................................... 557
12.2.4　C 函数偏移量跟踪 ........................................................................... 558
12.2.5　C USDT ............................................................................................. 558
12.2.6　C 单行程序 ....................................................................................... 559
12.3　Java ................................................................................................................. 560
12.3.1　跟踪 libjvm ....................................................................................... 561
12.3.2　jnistacks ............................................................................................. 563
12.3.3　Java 线程名字 .................................................................................. 565
12.3.4　Java 方法的符号 .............................................................................. 566
12.3.5　Java 调用栈 ...................................................................................... 569
12.3.6　Java USDT 探针 ............................................................................... 573
12.3.7　profile ................................................................................................ 579
12.3.8　offcputime ......................................................................................... 583
12.3.9　stackcount ......................................................................................... 589
12.3.10　javastat ............................................................................................ 593
12.3.11　javathreads....................................................................................... 594
12.3.12　javacalls ........................................................................................... 596
12.3.13　javaflow ........................................................................................... 597
12.3.14　javagc .............................................................................................. 599
12.3.15　javaobjnew ...................................................................................... 599
12.3.16　Java 单行程序 ................................................................................ 600
12.4　bash shell ........................................................................................................ 601
12.4.1　函数计数 ........................................................................................... 603
12.4.2　函数参数跟踪（bashfunc.bt） .......................................................... 604
12.4.3　函数执行时长（bashfunclat.bt） ...................................................... 607
12.4.4　/bin/bash ............................................................................................ 609
12.4.5　/bin/bash USDT ................................................................................. 613
12.4.6　bash 单行程序 .................................................................................. 613
12.5　其他语言 ........................................................................................................ 614
12.5.1　JavaScript（Node.js） ........................................................................ 614
12.5.2　C++ ................................................................................................... 616
12.5.3　Golang ............................................................................................... 616
12.6　小结 ................................................................................................................ 619
第13章　应用程序.......................................................................................................620
13.1　背景知识 ........................................................................................................ 621
13.1.1　应用程序基础信息 ........................................................................... 621
13.1.2　应用程序示例 ：MySQL 服务器 ....................................................622
13.1.3　BPF 的能力 ......................................................................................623
13.1.4　分析策略 ........................................................................................... 624
13.2　BPF工具 ......................................................................................................... 625
13.2.1　execsnoop .......................................................................................... 626
13.2.2　threadsnoop ....................................................................................... 626
13.2.3　profile ................................................................................................ 629
13.2.4　threaded ............................................................................................. 632
13.2.5　offcputime ......................................................................................... 634
13.2.6　offcpuhist .......................................................................................... 638
13.2.7　syscount ............................................................................................. 641
13.2.8　ioprofile ............................................................................................. 642
13.2.9　libc 帧指针 ........................................................................................ 644
13.2.10　mysqld_qslower .............................................................................. 645
13.2.11　mysqld_clat ..................................................................................... 648
13.2.12　signals ............................................................................................. 652
13.2.13　killsnoop .......................................................................................... 654
13.2.14　pmlock 和 pmheld ........................................................................... 655
13.2.15　naptime ............................................................................................ 660
13.2.16　其他工具 ......................................................................................... 662
13.3　BPF单行程序 ................................................................................................. 662
13.3.1　BCC ................................................................................................... 662
13.3.2　bpftrace ............................................................................................. 663
13.4　BPF单行程序示范 ......................................................................................... 664
13.5　小结 ................................................................................................................ 664
第14章　内核..............................................................................................................665
14.1　背景知识 ........................................................................................................ 666
14.1.1　内核基础知识 ................................................................................... 666
14.1.2　BPF 的分析能力 ..............................................................................668
14.2　分析策略 ........................................................................................................ 669
14.3　传统工具 ........................................................................................................ 670
14.3.1　Ftrace ................................................................................................. 670
14.3.2　perf sched .......................................................................................... 673
14.3.3　slabtop ............................................................................................... 674
14.3.4　其他工具 ........................................................................................... 675
14.4　BPF工具 ......................................................................................................... 675
14.4.1　loads .................................................................................................. 676
14.4.2　offcputime ......................................................................................... 677
14.4.3　wakeuptime .......................................................................................679
14.4.4　offwaketime ...................................................................................... 681
14.4.5　mlock 和 mheld ................................................................................. 683
14.4.6　自旋锁 ............................................................................................... 687
14.4.7　kmem ................................................................................................. 688
14.4.8　kpages ............................................................................................... 689
14.4.9　memleak ............................................................................................690
14.4.10　slabratetop ....................................................................................... 691
14.4.11　numamove ....................................................................................... 692
14.4.12　workq .............................................................................................. 694
14.4.13　小任务 ............................................................................................. 695
14.4.14　其他工具 ......................................................................................... 696
14.5　BPF单行程序 ................................................................................................. 697
14.5.1　BCC ................................................................................................... 697
14.5.2　bpftrace ............................................................................................. 698
14.6　BPF单行程序示范 ......................................................................................... 699
14.6.1　按系统调用函数对系统调用进行计数 ........................................... 699
14.6.2　对内核函数开始的 hrtimer 进行计数 ............................................. 699
14.7　挑战 ................................................................................................................ 700
14.8　小结 ................................................................................................................ 700
第15章　容器..............................................................................................................701
15.1　背景知识 ........................................................................................................ 701
15.1.1　BPF 的分析能力 ..............................................................................703
15.1.2　挑战 ................................................................................................... 703
15.1.3　分析策略 ........................................................................................... 706
15.2　传统工具 ........................................................................................................ 706
15.2.1　从主机上分析 ................................................................................... 706
15.2.2　在容器内分析 ................................................................................... 707
15.2.3　systemd-cgtop ................................................................................... 707
15.2.4　kubectl top ......................................................................................... 708
15.2.5　docker stats ........................................................................................ 708
15.2.6　/sys/fs/cgroups ................................................................................... 709
15.2.7　perf .................................................................................................... 709
15.3　BPF工具 ......................................................................................................... 710
15.3.1　runqlat ............................................................................................... 710
15.3.2　pidnss ................................................................................................ 711
15.3.3　blkthrot .............................................................................................. 714
15.3.4　overlayfs ............................................................................................ 715
15.4　BPF单行程序 ................................................................................................. 717
15.5　可选练习 ........................................................................................................ 717
15.6　小结 ................................................................................................................ 718
第16章　虚拟机管理器................................................................................................719
16.1　背景知识 ........................................................................................................ 719
16.1.1　BPF 的分析能力 ..............................................................................721
16.1.2　建议的分析策略 ............................................................................... 722
16.2　传统工具 ........................................................................................................ 722
16.3　访客系统的BPF工具 ..................................................................................... 723
16.3.1　Xen 超级调用 ................................................................................... 723
16.3.2　xenhyper ............................................................................................ 727
16.3.3　Xen 回调 ........................................................................................... 729
16.3.4　cpustolen ........................................................................................... 731
16.3.5　HVM 退出跟踪 ................................................................................ 732
16.4　宿主机BPF工具 ............................................................................................. 732
16.4.1　kvmexits ............................................................................................ 733
16.4.2　未来的工作 ....................................................................................... 737
16.5　小结 ................................................................................................................ 737
第17章　其他BPF性能工具.........................................................................................738
17.1　Vector和Performance Co-Pilot（PCP） ....................................................... 738
17.1.1　可视化 ............................................................................................... 739
17.1.2　可视化 ：热图 ................................................................................... 740
17.1.3　可视化 ：表格形式的数据 ............................................................... 742
17.1.4　BCC 提供的指标 ..............................................................................743
17.1.5　内部实现 ........................................................................................... 743
17.1.6　安装 PCP 和 Vector ..........................................................................744
17.1.7　连接并显示数据 ............................................................................... 744
17.1.8　配置 BCC PMDA ............................................................................. 746
17.1.9　改进工作 ........................................................................................... 747
17.1.10　进一步阅读 ..................................................................................... 747
17.2　Grafana和Performance Co-Pilot .................................................................... 747
17.2.1　安装和配置 ....................................................................................... 748
17.2.2　连接并查看数据 ............................................................................... 748
17.2.3　改进工作 ........................................................................................... 750
17.2.4　进一步阅读 ....................................................................................... 750
17.3　Cloudflare eBPF Prometheus Exporter（配合Grafana） ............................. 750
17.3.1　构建并运行 ebpf 导出器 ................................................................. 750
17.3.2　配置 Prometheus 监控 ebpf_exporter 实例 ..................................... 751
17.3.3　在 Grafana 中设置一个查询 ............................................................ 751
17.3.4　进一步阅读 ....................................................................................... 751
17.4　kubectl-trace ................................................................................................... 752
17.4.1　跟踪节点 ........................................................................................... 752
17.4.2　跟踪 pod 和容器 ............................................................................... 753
17.4.3　进一步阅读 ....................................................................................... 755
17.5　其他工具 ........................................................................................................ 755
17.6　小结 ................................................................................................................ 755
第18章　建议、技巧和常见问题. .................................................................................756
18.1　典型事件的频率和额外开销 ........................................................................ 756
18.1.1　频率 ................................................................................................... 757
18.1.2　执行的操作 ....................................................................................... 758
18.1.3　自行测试 ........................................................................................... 760
18.2　以49Hz或99Hz为采样频率 .......................................................................... 760
18.3　黄猪和灰鼠 .................................................................................................... 760
18.4　开发目标软件 ................................................................................................ 762
18.5　学习系统调用 ................................................................................................ 763
18.6　保持简单 ........................................................................................................ 764
18.7　事件缺失 ........................................................................................................ 764
18.8　调用栈缺失 .................................................................................................... 766
18.8.1　如何修复损坏的调用栈 ................................................................... 767
18.9　打印时符号缺失（函数名称） .................................................................... 767
18.9.1　如何修复符号缺失 ：JIT 运行时（Java、Node.js、...） ................ 768
18.9.2　如何修复符号缺失 ：ELF 二进制文件（C、C++、...） ................ 768
18.10　跟踪时函数缺失 .......................................................................................... 768
18.11　反馈回路 ...................................................................................................... 769
18.12　被丢掉的事件 .............................................................................................. 769
附录A　bpftrace单行程序............................................................................................770
附录B　bpftrace备忘单...............................................................................................775
附录C　BCC工具的开发.............................................................................................778
附录D　C.BPF. ............................................................................................................793
附录E　BPF指令.........................................................................................................812