The Complete Guide to Optimizing Systems Performance   Written by the winner of the 2013 LISA Award for Outstanding Achievement in System Administration Large-scale enterprise, cloud, and virtualized computing systems have introduced serious performance challenges. Now, internationally renowned performance expert Brendan Gregg has brought together proven methodologies, tools, and metrics for analyzing and tuning even the most complex environments. Systems Performance: Enterprise and the Cloud focuses on Linux® and Unix® performance, while illuminating performance issues that are relevant to all operating systems. You’ll gain deep insight into how systems work and perform, and learn methodologies for analyzing and improving system and application performance. Gregg presents examples from bare-metal systems and virtualized cloud tenants running Linux-based Ubuntu®, Fedora®, CentOS, and the illumos-based Joyent® SmartOS™ and OmniTI OmniOS®. He systematically covers modern systems performance, including the “traditional” analysis of CPUs, memory, disks, and networks, and new areas including cloud computing and dynamic tracing. This book also helps you identify and fix the “unknown unknowns” of complex performance: bottlenecks that emerge from elements and interactions you were not aware of. The text concludes with a detailed case study, showing how a real cloud customer issue was analyzed from start to finish.   Coverage includes • Modern performance analysis and tuning: terminology, concepts, models, methods, and techniques • Dynamic tracing techniques and tools, including examples of DTrace, SystemTap, and perf • Kernel internals: uncovering what the OS is doing • Using system observability tools, interfaces, and frameworks • Understanding and monitoring application performance • Optimizing CPUs: processors, cores, hardware threads, caches, interconnects, and kernel scheduling • Memory optimization: virtual memory, paging, swapping, memory architectures, busses, address spaces, and allocators • File system I/O, including caching • Storage devices/controllers, disk I/O workloads, RAID, and kernel I/O • Network-related performance issues: protocols, sockets, interfaces, and physical connections • Performance implications of OS and hardware-based virtualization, and new issues encountered with cloud computing • Benchmarking: getting accurate results and avoiding common mistakes   This guide is indispensable for anyone who operates enterprise or cloud environments: system, network, database, and web admins; developers; and other professionals. For students and others new to optimization, it also provides exercises reflecting Gregg’s extensive instructional experience.  

Table of Contents:
Acknowledgments xxxiii About the Author xxxv   Chapter 1: Introduction 1 1.1 Systems Performance 1 1.2 Roles 2 1.3 Activities 3 1.4 Perspectives 4 1.5 Performance Is Challenging 4 1.6 Latency 6 1.7 Dynamic Tracing 7 1.8 Cloud Computing 8 1.9 Case Studies 9   Chapter 2: Methodology 15 2.1 Terminology 16 2.2 Models 17 2.3 Concepts 18 2.4 Perspectives 32 2.5 Methodology 35 2.6 Modeling 57 2.7 Capacity Planning 65 2.8 Statistics 69 2.9 Monitoring 74 2.10 Visualizations 76 2.11 Exercises 82 2.12 References 82   Chapter 3: Operating Systems 85 3.1 Terminology 86 3.2 Background 87 3.3 Kernels 105 3.4 Exercises 113 3.5 References 113   Chapter 4: Observability Tools 115 4.1 Tool Types 116 4.2 Observability Sources 120 4.3 DTrace 133 4.4 SystemTap 144 4.5 perf 149 4.6 Observing Observability 150 4.7 Exercises 151 4.8 References 151   Chapter 5: Applications 153 5.1 Application Basics 153 5.2 Application Performance Techniques 158 5.3 Programming Languages 163 5.4 Methodology and Analysis 167 5.5 Exercises 186 5.6 References 187   Chapter 6: CPUs 189 6.1 Terminology 190 6.2 Models 191 6.3 Concepts 193 6.4 Architecture 199 6.5 Methodology 214 6.6 Analysis 224 6.7 Experimentation 254 6.8 Tuning 256 6.9 Exercises 260 6.10 References 262   Chapter 7: Memory 265 7.1 Terminology 266 7.2 Concepts 267 7.3 Architecture 272 7.4 Methodology 289 7.5 Analysis 295 7.6 Tuning 314 7.7 Exercises 319 7.8 References 320   Chapter 8: File Systems 323 8.1 Terminology 324 8.2 Models 325 8.3 Concepts 326 8.4 Architecture 337 8.5 Methodology 353 8.6 Analysis 362 8.7 Experimentation 383 8.8 Tuning 387 8.9 Exercises 391 8.10 References 392   Chapter 9: Disks 395 9.1 Terminology 396 9.2 Models 397 9.3 Concepts 399 9.4 Architecture 407 9.5 Methodology 421 9.6 Analysis 431 9.7 Experimentation 465 9.8 Tuning 467 9.9 Exercises 470 9.10 References 471   Chapter 10: Network 473 10.1 Terminology 474 10.2 Models 474 10.3 Concepts 476 10.4 Architecture 483 10.5 Methodology 493 10.6 Analysis 503 10.7 Experimentation 535 10.8 Tuning 536 10.9 Exercises 542 10.10 References 543   Chapter 11: Cloud Computing 545 11.1 Background 546 11.2 OS Virtualization 551 11.3 Hardware Virtualization 563 11.4 Comparisons 581 11.5 Exercises 583 11.6 References 584   Chapter 12: Benchmarking 587 12.1 Background 588 12.2 Benchmarking Types 597 12.3 Methodology 602 12.4 Benchmark Questions 613 12.5 Exercises 614 12.6 References 615   Chapter 13: Case Study 617 13.1 Case Study: The Red Whale 617 13.2 Comments 633 13.3 Additional Information 634 13.4 References 634   Appendix A: USE Method: Linux 637 Physical Resources 637 Software Resources 640 Reference 641   Appendix B: USE Method: Solaris 643 Physical Resources 643 Software Resources 646 References 647   Appendix C: sar Summary 649 Linux 649 Solaris 650   Appendix D: DTrace One-Liners 651 syscall Provider 651 proc Provider 655 profile Provider 655 sched Provider 657 fbt Provider 658 pid Provider 659 io Provider 660 sysinfo Provider 660 vminfo Provider 661 ip Provider 661 tcp provider 662 udp provider 663   Appendix E: DTrace to SystemTap 665 Functionality 665 Terminology 666 Probes 666 Built-in Variables 667 Functions 668 Example 1: Listing syscall Entry Probes 668 Example 2: Summarize read() Returned Size 668 Example 3: Count syscalls by Process Name 670 Example 4: Count syscalls by syscall Name, for Process ID 123 671 Example 5: Count syscalls by syscall Name, for "httpd" Processes 672 Example 6: Trace File open()s with Process Name and Path Name 672 Example 7: Summarize read() Latency for "mysqld" Processes 672 Example 8: Trace New Processes with Process Name and Arguments 673 Example 9: Sample Kernel Stacks at 100 Hz 674 References 674   Appendix F: Solutions to Selected Exercises 675 Chapter 2 Methodology 675 Chapter 3 Operating Systems 675 Chapter 6 CPUs 675 Chapter 7 Memory 676 Chapter 8 File Systems 676 Chapter 9 Disks 677 Chapter 11 Cloud Computing 677   Appendix G: Systems Performance Who's Who 679   Glossary 683 Bibliography 689 Index 697