cmd/cover: extend coverage testing to include applications · Issue #51430 · golang/go · GitHub
Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert cmd/cover: extend coverage testing to include applications #51430 New issue Copy link New issue Copy link Closed Closed cmd/cover: extend coverage testing to include applications #51430 Copy link Assignees Labels FrozenDueToAge Proposal Proposal-Accepted compiler/runtime Issues related to the Go compiler and/or runtime.
Issues related to the Go compiler and/or runtime. Milestone Go1.20 Description thanm opened on Mar 2, 2022 Issue body actions Proposal: extend code coverage testing to include applications Author(s): Than McIntosh Last updated: 2022-03-02 Detailed design document: markdown , CL 388857 Abstract This document contains a proposal for improving/revamping the system used in Go for code coverage testing. Background Current support for coverage testing The Go toolchain currently includes support for collecting and reporting coverage data for Golang unit tests; this facility is made available via the "go test -cover" and "go tool cover" commands.
The current workflow for collecting coverage data is baked into "go test" command; the assumption is that the source code of interest is a Go package or set of packages with associated tests. To request coverage data for a package test run, a user can invoke the test(s) via: go test -coverprofile=<filename> [package target(s)] This command will build the specified packages with coverage instrumentation, execute the package tests, and write an output file to "filename" with the coverage results of the run.
The resulting output file can be viewed/examined using commands such as go tool cover -func=<covdatafile> go tool cover -html=<covdatafile> Under the hood, the implementation works by source rewriting: when "go test" is building the specified set of package tests, it runs each package source file of interest through a source-to-source translation tool that produces an instrumented/augmented equivalent, with instrumentation that records which portions of the code execute as the test runs.
A function such as func ABC ( x int ) { if x < 0 { bar () } } is rewritten to something like func ABC ( x int ) { GoCover_0_343662613637653164643337 . Count [ 9 ] = 1 ; if x < 0 { GoCover_0_343662613637653164643337 . Count [ 10 ] = 1 ; bar () } } where "GoCover_0_343662613637653164643337" is a tool-generated structure with execution counters and source position information. The "go test" command also emits boilerplate code into the generated "_testmain.go" to register each instrumented source file and unpack the coverage data structures into something that can be easily accessed at runtime.
Finally, the modified "_testmain.go" has code to call runtime routines that emit the coverage output file when the test completes. Strengths and weaknesses of what we currently provide The current implementation is simple and easy to use, and provides a good user experience for the use case of collecting coverage data for package unit tests. Since "go test" is performing both the build and the invocation/execution of the test, it can provide a nice seamless "single command" user experience.
A key weakness of the current implementation is that it does not scale well-- it is difficult or impossible to gather coverage data for applications as opposed to collections of packages, and for testing scenarios involving multiple runs/executions. For example, consider a medium-sized application such as the Go compiler ("gc"). While the various packages in the compiler source tree have unit tests, and one can use "go test" to obtain coverage data for those tests, the unit tests by themselves only exercise a small fraction of the code paths in the compiler that one would get from actually running the compiler binary itself on a large collection of Go source files.
For such applications, one would like to build a coverage-instrumented copy of the entire application ("gc"), then run that instrumented application over many inputs (say, all the Go source files compiled as part of a "make.bash" run for multiple GOARCH values), producing a collection of coverage data output files, and finally merge together the results to produce a report or provide a visualization. Many folks in the Golang community have run into this problem; there are large numbers of blog posts and other pages describing the issue, and recommending workarounds (or providing add-on tools that help); doing a web search for "golang integration code coverage" will turn up many pages of links.
An additional weakness in the current Go toolchain offering relates to the way in which coverage data is presented to the user from the "go tool cover") commands. The reports produced are "flat" and not hierarchical (e.g. a flat list of functions, or a flat list of source files within the instrumented packages). This way of structuring a report works well when the number of instrumented packages is small, but becomes less attractive if there are hundreds or thousands of source files being instrumented.
For larger applications, it would make sense to create reports with a more hierarchical structure: first a summary by module, then package within module, then source file within package, and so on. Finally, there are a number of long-standing problems that arise due to the use of source-to-source rewriting used by cmd/cover and the go command, including #23883 "cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./..." #23910 "cmd/go: -coverpkg packages imported by all tests, even ones that otherwise do not use it" #27336 "cmd/go: test coverpkg panics when defining the same flag in multiple packages" Most of these problems arise because of the introduction of additional imports in the _testmain.go shim created by the Go command when carrying out a coverage test run in combination with the "-coverpkg" option.
Proposed changes Building for coverage While the existing "go test" based coverage workflow will continue to be supported, the proposal is to add coverage as a new build mode for "go build". In the same way that users can build a race-detector instrumented executable using "go build -race", it will be possible to build a coverage-instrumented executable using "go build -cover". To support this goal, the plan will be to migrate the support for coverage instrumentation into the compiler, moving away from the source-to-source translation approach.
Running instrumented applications Applications are deployed and run in many different ways, ranging from very simple (direct invocation of a single executable) to very complex (e.g. gangs of cooperating processes involving multiple distinct executables). To allow for more complex execution/invocation scenarios, it doesn't make sense to try to serialize updates to a single coverage output data file during the run, since this would require introducing synchronization or some other mechanism to ensure mutually exclusive access.
For non-test applications built for coverage, users will instead select an output directory as opposed to a single file; each run of the instrumented executable will emit data files within that directory. Example: $ go build -o myapp.exe -cover ... $ mkdir /tmp/mycovdata $ export GOCOVERDIR=/tmp/mycovdata $ <run test suite, resulting in multiple invocations of myapp.exe> $ go tool cover -html=/tmp/mycovdata $ For coverage runs in the context of "go test", the default will continue to be emitting a single named output file when the test is run.
File names within the output directory will be chosen at runtime so as to minimize the possibility of collisions, e.g. possibly something to the effect of covdata.<metafilehash>.<processid>.<nanotimevalue>.out When invoked for reporting, the coverage tool itself will test its input argument to see whether i
The rest of this page is on the live site.