runtime: memory corruption on Linux 5.2+ · Issue #35777 · golang/go · GitHub
Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert runtime: memory corruption on Linux 5.2+ #35777 New issue Copy link New issue Copy link Closed Closed runtime: memory corruption on Linux 5.2+ #35777 Copy link Labels FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
The path to resolution is known, but the work has not been done. release-blocker umbrella Milestone Go1.14 Description aclements opened on Nov 22, 2019 Issue body actions We've had several reports of memory corruption on Linux 5.3.x (or later) kernels from people running tip since asynchronous preemption was committed. This is a super-bug to track these issues. I suspect they all have one root cause. Typically these are "runtime error: invalid memory address or nil pointer dereference" or "runtime: unexpected return pc" or "segmentation violation" panics.
They can also appear as self-detected data corruption. If you encounter a crash that could be random memory corruption, are running Linux 5.3.x or later, and are running a recent tip Go (after commit 62e53b7 ), please file a new issue and add a comment here. If you can reproduce it, please try setting "GODEBUG=asyncpreemptoff=1" in your environment and seeing if you can still reproduce it. Duplicate issues (I'll edit this comment to keep this up-to-date): runtime: corrupt binary export data seen after signal preemption CL ( #35326 ): Corruption in file version header observed by vet.
Medium reproducible. Strong leads. cmd/compile: panic during early copyelim crash ( #35658 ): Invalid memory address in cmd/compile/internal/ssa.copyelim. Not reproducible. Nothing obvious in stack trace. Haven't dug into assembly. runtime: SIGSEGV in mapassign_fast64 during cmd/vet ( #35689 ): Invalid memory address in runtime.mapassign_fast64 in vet. Stack trace includes random pointers. Some assembly decoding work. runtime: unexpected return pc for runtime.(*mheap).alloc ( #35328 ): Unexpected return pc.
Stack trace includes random pointers. Not reproducible. cmd/dist: I/O error: read src/xxx.go: is a directory ( #35776 ): Random misbehavior. Not reproducible. runtime: "fatal error: mSpanList.insertBack" in mallocgc ( #35771 ): Bad mspan next pointer (random and unaligned). Not reproducible. cmd/compile: invalid memory address or nil pointer dereference in gc.convlit1 ( #35621 ): Invalid memory address in cmd/compile/internal/gc.convlit1.
Evidence of memory corruption, though no obvious random pointers. Not reproducible. cmd/go: unexpected signal during runtime execution ( #35783 ): Corruption in file version header observed by vet. Not reproducible. runtime: unexpected return pc for runtime.systemstack_switch ( #35592 ): Unexpected return pc. Stack trace includes random pointers. Not reproducible. cmd/compile: random compile error running tests ( #35760 ): Compiler data corruption.
Not reproducible. Reacted by Dmitri Shuralyov, Ben Schumacher, Zhennan LI, Jamie Hall, Scott Stensland, Camal Cakar and Anton Kurochkin Reacted by Octávio Richter Reacted by Bryan C. Mills, Daniel Martí, Alberto Donizetti, Zik Aeroh, Josh Bleecher Snyder, Ingo Oeser, Constantine, lni, Changkun Ou, Valery Piashchynski and 27 more Activity Next aclements added NeedsFix The path to resolution is known, but the work has not been done.
The path to resolution is known, but the work has not been done. release-blocker on Nov 22, 2019 aclements added this to the Go1.14 milestone on Nov 22, 2019 aclements self-assigned this on Nov 22, 2019 aclements pinned this issue on Nov 22, 2019 mvdan commented on Nov 22, 2019 mvdan on Nov 22, 2019 Member More actions @aclements for your records, #35328 and #35776 might be related as well. Those two were on the same Linux 5.3.x machine of mine.
Reacted by Bryan C. Mills, Vladyslav Dalechyn and Isogram aclements commented on Nov 22, 2019 aclements on Nov 22, 2019 Member Author More actions Thanks @mvdan . I've folded those into the list above. zikaeroh commented on Nov 22, 2019 zikaeroh on Nov 22, 2019 Contributor More actions #35621 from me. One time, no repro. myitcv commented on Nov 22, 2019 myitcv on Nov 22, 2019 Member More actions @aclements just saw #35783 for the record.
If you think we have enough "evidence" please say and I'll stop creating issues for now 😄 bradfitz commented on Nov 22, 2019 bradfitz on Nov 22, 2019 Contributor More actions Have we roughly bisected which Linux versions are affected? Looking at the kernel changes in that region might yield a clue about where and whose the bug is. 5.3 = bad. 5.2 = ? zikaeroh commented on Nov 22, 2019 zikaeroh on Nov 22, 2019 Contributor More actions In #35326 (comment) , I used Arch's 4.19 LTS and could not reproduce the bexport corruption.
However, the kernel configuration differs between 4.19 and 5.3, so that may be unscientific. ( I'm letting my machine rebuild 5.3 without PREEMPT set to see if that's the problem, but I have doubts. EDIT: It was not PREEMPT, so setting up a builder with a newer kernel would likely be good regardless.) What set of kernels do the current Linux builders use? That might provide a lower bound, as I've never seen the issue there. (I'd bring up #9505 to advocate for an Arch builder, but that issue is more about everything but the kernel version.
I feel like there should be some builder which is at the latest Linux kernel, whatever that may be.) bradfitz commented on Nov 22, 2019 bradfitz on Nov 22, 2019 Contributor More actions The existing Go Linux builders use Container Optimized OS with a Linux kernel 4.19.72+ . aclements commented on Nov 22, 2019 aclements on Nov 22, 2019 Member Author More actions Thanks @myitcv , I think we have enough reports. If you do happen to find another one that's reproducible, that would be very helpful, though.
Reacted by Paul Jolly dr2chase commented on Nov 25, 2019 dr2chase on Nov 25, 2019 Contributor More actions To recap experiments last Friday (and I rechecked the test for the more mystifying of these Sunday afternoon), Cherry and I tried the following: Double the size of the sigaltstack, just in case. Also sanity check the bounds within gdb, they were okay. Modified the definition of fpstate to conform to what is defined in the linux header files.
Modified sigcontext to use the new Xstate: fpstate *Xstate // *fpstate1 Wrote a method to allow us to store the ymm registers that were supplied (as registers) to the signal handler, tried an experiment in the assembly language handler to trash the YMM registers (not the data structures) before return. We never saw any sign of the trash but this seemed to raise the rate of the failures (running "go vet all"). The trashing string stored was "This_is_a_test.
" tried printing the saved and current ymm registers in sigtrampgo. The saved ones looked like memmove artifacts (source code while running vet all), and the current ones were always zero . The memmove artifacts stayed unchanged, a lot, between signals. I rechecked the code that did this earlier today, just in case we got it wrong. made a copy of the saved xmm and ymm registers on sigtrampgo entry, then checked the copy against the saved registers, to see if our code ever somehow modified them.
That never fired. I spent some time Saturday looking for "interesting" comments in the Linux git log, I have some to review. What I am wondering is if there was some attempt to optimize saving of the ymm registers and that got fouled up. One thing I wonder a little about was what they are doing for power management with AVX use, I saw some mention of that. (I.e., what triggers AVX use, can they "save power" if they don't touch the registers, if they believe AVX is not being used?
Suppose they rely on some hardware bit that isn't set under exactly the expected conditions?) type Xstate struct {
The rest of this page is on the live site.