Profile-guided optimization - The Go Programming Language
Profile-guided optimization Starting in Go 1.20, the Go compiler supports profile-guided optimization (PGO) to further optimize builds. Table of Contents: Overview Collecting profiles Building with PGO Notes Frequently Asked Questions Appendix: alternative profile sources Overview Profile-guided optimization (PGO), also known as feedback-directed optimization (FDO), is a compiler optimization technique that feeds information (a profile) from representative runs of the application back into to the compiler for the next build of the application, which uses that information to make more informed optimization decisions.
For example, the compiler may decide to more aggressively inline functions which the profile indicates are called frequently. In Go, the compiler uses CPU pprof profiles as the input profile, such as from runtime/pprof or net/http/pprof . As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%. We expect performance gains to generally increase over time as additional optimizations take advantage of PGO in future versions of Go.
Collecting profiles The Go compiler expects a CPU pprof profile as the input to PGO. Profiles generated by the Go runtime (such as from runtime/pprof and net/http/pprof ) can be used directly as the compiler input. It may also be possible to use/convert profiles from other profiling systems. See the appendix for additional information. For best results, it is important that profiles are representative of actual behavior in the application’s production environment.
Using an unrepresentative profile is likely to result in a binary with little to no improvement in production. Thus, collecting profiles directly from the production environment is recommended, and is the primary method that Go’s PGO is designed for. The typical workflow is as follows: Build and release an initial binary (without PGO). Collect profiles from production. When it’s time to release an updated binary, build from the latest source and provide the production profile.
GOTO 2 Go PGO is generally robust to skew between the profiled version of an application and the version building with the profile, as well as to building with profiles collected from already-optimized binaries. This is what makes this iterative lifecycle possible. See the AutoFDO section for additional details about this workflow. If it is difficult or impossible to collect from the production environment (e.g., a command-line tool distributed to end users), it is also possible to collect from a representative benchmark.
Note that constructing representative benchmarks is often quite difficult (as is keeping them representative as the application evolves). In particular, microbenchmarks are usually bad candidates for PGO profiling , as they only exercise a small part of the application, which yields small gains when applied to the whole program. Building with PGO The standard approach to building is to store a pprof CPU profile with filename default.pgo in the main package directory of the profiled binary.
By default, go build will detect default.pgo files automatically and enable PGO. Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source. For more complex scenarios, the go build -pgo flag controls PGO profile selection.
This flag defaults to -pgo=auto for the default.pgo behavior described above. Setting the flag to -pgo=off disables PGO optimizations entirely. If you cannot use default.pgo (e.g., different profiles for different scenarios of one binary, unable to store profile with source, etc), you may directly pass a path to the profile to use (e.g., go build -pgo=/tmp/foo.pprof ). Note: A path passed to -pgo applies to all main packages. e.g., go build -pgo=/tmp/foo.pprof ./cmd/foo ./cmd/bar applies foo.pprof to both binaries foo and bar , which is often not what you want.
Usually different binaries should have different profiles, passed via separate go build invocations. Note: Before Go 1.21, the default is -pgo=off . PGO must be explicitly enabled. Notes Collecting representative profiles from production Your production environment is the best source of representative profiles for your application, as described in Collecting profiles . The simplest way to start with this is to add net/http/pprof to your application and then fetch /debug/pprof/profile?seconds=30 from an arbitrary instance of your service.
This is a great way to get started, but there are ways that this may be unrepresentative: This instance may not be doing anything at the moment it gets profiled, even though it is usually busy. Traffic patterns may change throughout the day, making behavior change throughout the day. Instances may perform long-running operations (e.g., 5 minutes doing operation A, then 5 minutes doing operation B, etc). A 30s profile will likely only cover a single operation type.
Instances may not receive fair distributions of requests (some instances receive more of one type of request than others). A more robust strategy is collecting multiple profiles at different times from different instances to limit the impact of differences between individual instance profiles. Multiple profiles may then be merged into a single profile for use with PGO. Many organizations run “continuous profiling” services that perform this kind of fleet-wide sampling profiling automatically, which could then be used as a source of profiles for PGO.
Merging profiles The pprof tool can merge multiple profiles like this: $ go tool pprof -proto a.pprof b.pprof > merged.pprof This merge is effectively a straightforward sum of samples in the input, regardless of wall duration of the profile. As a result, when profiling a small time slice of an application (e.g., a server that runs indefinitely), you likely want to ensure that all profiles have the same wall duration (i.e., all profiles are collected for 30s).
Otherwise, profiles with longer wall duration will be overrepresented in the merged profile. AutoFDO Go PGO is designed to support an “ AutoFDO ” style workflow. Let’s take a closer look at the workflow described in Collecting profiles : Build and release an initial binary (without PGO). Collect profiles from production. When it’s time to release an updated binary, build from the latest source and provide the production profile.
GOTO 2 This sounds deceptively simple, but there are a few important properties to note here: Development is always ongoing, so the source code of the profiled version of the binary (step 2) is likely slightly different from the latest source code getting built (step 3). Go PGO is designed to be robust to this, which we refer to as source stability . This is a closed loop. That is, after the first iteration the profiled version of the binary is already PGO-optimized with a profile from a previous iteration.
Go PGO is also designed to be robust to this, which we refer to as iterative stability . Source stability is achieved using heuristics to match samples from the profile to the compiling source. As a result, many changes to source code, such as adding new functions, have no impact on matching existing code. When the compiler is not able to match changed code, some optimizations are lost, but note that this is a graceful degradation .
A single function failing to match may lose out on optimization opportunities, but overall PGO benefit is usually spread across many functions. See the source stability section for more details about matching and degradation. Iterative stability is the prevention of cycles of variable performance in successive PGO builds (e.g., build #1 is fast, build #2 is slow, build #3 is fast, etc). We use CPU profiles to identify hot fu
The rest of this page is on the live site.