Text of github.com › golang › go › issues › 57175 as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

runtime: performance and diagnostics meeting notes · Issue #57175 · golang/go · GitHub

Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert runtime: performance and diagnostics meeting notes #57175 New issue Copy link New issue Copy link Open Open runtime: performance and diagnostics meeting notes #57175 Copy link Assignees Labels NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. compiler/runtime Issues related to the Go compiler and/or runtime. Issues related to the Go compiler and/or runtime. meeting-minutes Meeting minutes/notes from periodic meetings. Meeting minutes/notes from periodic meetings. Milestone Backlog Description mknyszek opened on Dec 8, 2022 Issue body actions Since 2022, members of the Go team and the Go community have met every two weeks to discuss Go runtime topics.

Until 2025 the focus was mainly on diagnostics, but we've also delved into plenty of other topics related to the runtime as well. To reflect this, I'm renaming this group to the "performance and diagnostics biweekly meeting." We currently have a virtual sync every 2 weeks, Thursdays at 11 AM NYC time. Please ping me at mknyszek -- at -- golang.org for an invite if you're interested in attending. This issue will be updated regularly with meeting notes from those meetings.

The reasons for starting this meeting in 2022 are still relevant today. So for the sake of some nostalgia, and also as a reminder of the importance of diagnostics to making Go an effective tool for our users, I leave behind some of the original text from this GitHub issue below. As the Go user base grows, more and more Go developers are seeking to understand the performance of their programs and reduce resource costs.

However, they are locked into the relatively limited diagnostic tools we provide today. Some teams build their own tools, but right now that requires a large investment. This issue extends to the Go team as well, where we often put significant effort into ad-hoc performance tooling to analyze the performance of Go itself. This issue is a tracking issue for improving the state of Go runtime diagnostics and its tooling, focusing primarily on runtime/trace traces and heap analysis tooling.

Reacted by tsingson, rain.cai, Rajender Reddy Kompally, Terry Chan, vadyus, Julio Guerra, Nikolay Dubina, Sen Han, Señor Gus, Leonard Wang and 19 more Reacted by tsingson, Rajender Reddy Kompally, Julio Guerra, Kemal Akkoyun, Eliah Rusin, Paolo Facchinetti and Redowan Delowar Reacted by Felix Geisendörfer, Nick Ripley, R. Aidan Campbell, Kemal Akkoyun, Valery Piashchynski, Michal Matczuk, Sung Yoon Whang, Sayed Mustafa Hussaini, Oleg Kovalov, Sandeep Bandi and 9 more Reacted by Umut Tezduyar Lindskog and someview Reacted by Eliah Rusin and Thomas Løkkeborg Activity Next mknyszek added compiler/runtime Issues related to the Go compiler and/or runtime.

Issues related to the Go compiler and/or runtime. on Dec 8, 2022 mknyszek added this to the Go1.21 milestone on Dec 8, 2022 mknyszek added this to Go Compiler / Runtime / Command on Dec 8, 2022 mknyszek commented on Dec 8, 2022 mknyszek on Dec 8, 2022 Contributor Author More actions 2022-12-07 Sync Attendees: @mknyszek @aclements @prattmic @felixge @nsrip-dd @rhysh Notes: pprof labels in execution traces Michael K: I need to follow up on the CL and issue.

Michael P: Have you considered runtime/trace regions? Nick: Yes, but it doesn't quite hit our use-cases. Rhys: Could use logs instead. Is the point to follow goroutines? Felix: Yeah. Rhys: Started parsing the trace format for some internal tooling. Felix: Prefer labels because inheritance. Mostly like it for profile tools on top, but maybe doesn't matter for tracing tools. The parent goroutine gets recorded so things like request IDs can be tracked in post-processing.

Also useful for attributing CPU time. Michael P: Tasks are inherited, but inheritance is processed downstream. Would be nice to have pprof labels just so you don't have to think about which one to use. Michael K: Useful even just to bridge pprof and runtime/trace. Austin: Agreed. We can one day get to the point of deprecating the old APIs as well. Rhys: RE: Attributing CPU time, can see at better than 10 ms of granularity already (even if Ps aren't CPU time, it's still time the rest of the app couldn't run).

Michael P: There's an issue about measure CPU time on-line. (TODO: Find it.) Trace parsing API Michael K: How important is this? Priority? Felix: Important for the community, but we can't make use of it in our case. Will the trace format change in between releases? Michael K: I think we can always guarantee a trace format for a release. Michael P: How high level should this API be? cmd/trace has two levels: Low-level that understands format and events.

Higher-level that understands relationships between goroutines, etc. Michael K: Page trace splits this into "parser" and "simulator." The latter is more stateful. Felix: Intuitive feeling toward lower level API. Rhys: +1 to low level. Austin: Scalability of processing traces. Currently not in a good state in low or high level format (currently requires the whole trace). Can fix trace wire format for low-level parsing scalability issues, but it's much less clear how to do this for the high-level format.

Austin: Flight recorder idea. Interacts interestingly with simulation. Current trace snapshots everything. Solved this in debuglog; reads its own tail and keeps local state updated. Complicated trade-offs in this space. Felix: We use a lot of JFR, one thing that's nice is it's broken down into self-contained chunks. Michael K sent out a very half-baked trace format revamp. (Thanks for the comments! Far from ready to share more widely.) The next step is to measure the actual current overhead.

Maybe add a mode to Sweet? Rhys: Have been collecting CPU profiles and execution traces. 20% of CPU time during execution trace is for execution trace itself. 95% of overhead is collecting stack traces. Collect 1 second every 5000 seconds and no one complains. People do complain about goroutine profiles every 2 minutes. Michael K: Shooting for KUTrace overhead, so making stack traces optional/faster is just step 1. Felix: Trace effect on tail latency.

Rhys: Traces are my view of tail latency. Felix: Benchmark for pathological cases and worst case. Austin: Linked to trace overhead issue , Dmitry proposed switching to frame pointer unwinding. Felix: At some point implemented frame pointer unwinding in userland and it was 50x faster ( link ). Rhys: Not sure what kind of tool you could build without stack traces in an app that doesn't set pprof labels, tasks, regions, trace logs, etc.

Michael K: Integration with structured logging? Michael P: It does add yet another way to add things to runtime/trace. Rhys: The standard library (e.g. database/sql) doesn't currently use runtime/trace at all, maybe it should. Michael K: This connects to deciding what goes into a trace. I think this is a very good idea. Felix: +1. Java world and JFR does this. Deciding what goes into a trace Disabling stack tracing / reduce stack trace depth Filtering by pprof labels Specific event classes Standard library events Rhys: I've made this decision for my organization.

Expected that you do profiling for a long running service. No opportunity for app owners to express opinions. People who complained forked the package, turned it off, and now coming back. I kind of want everything. Felix: I would love to be in a place where we can do that, but we get pushback from users when the overhead is too high. Rhys: The question is how close we get within 1% overhead. My choice was to get everything, but less often.

Felix: Desire to get more of everything is in conflict w

The rest of this page is on the live site.