Issue 40237: Test code coverage (C) job of Travis CI fails on test_distutils which creates _configtest.gcno file - Python tracker
Python Home About News Documentation Downloads Community Foundation Developer's Guide Issue Tracker Issues Search Random Issue Summaries Issues with patch Easy issues Stats User Administration User List Committer List Help Tracker Documentation Tracker Development Report Tracker Problem Issue40237 ➜ This issue tracker has been migrated to GitHub , and is currently read-only . For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on 2020-04-09 12:52 by vstinner , last changed 2022-04-11 14:59 by admin . This issue is now closed . Pull Requests URL Status Linked Edit PR 19460 closed shihai1991 , 2020-04-10 16:49 Messages (16) msg366047 - (view) Author: STINNER Victor (vstinner) * Date: 2020-04-09 12:52 Example: https://travis-ci.org/github/python/cpython/jobs/672698615 Output: =========================================== 0:09:01 load avg: 3.96 [104/421/1] test_distutils failed -- running: test_multiprocessing_forkserver (1 min 13 sec), test_unparse (6 min 21 sec) Warning -- files was modified by test_distutils Before: [] After: ['_configtest.gcno'] test test_distutils failed -- Traceback (most recent call last): File "/home/travis/build/python/cpython/Lib/ distutils/tests/test_build_ext.py ", line 112, in test_build_ext assert_python_ok('-c', code) File "/home/travis/build/python/cpython/Lib/ test/support/script_helper.py ", line 156, in assert_python_ok return _assert_python(True, *args, **env_vars) File "/home/travis/build/python/cpython/Lib/ test/support/script_helper.py ", line 142, in _assert_python res.fail(cmd_line) File "/home/travis/build/python/cpython/Lib/ test/support/script_helper.py ", line 70, in fail raise AssertionError("Process return code is %d\n" AssertionError: Process return code is 1 command line: ['/home/travis/build/python/cpython/python', '-X', 'faulthandler', '-I', '-c', "\ntmp_dir = '/tmp/tmp3y4hpawx'\n\nimport sys\nimport unittest\nfrom test import support\n\nsys.path.insert(0, tmp_dir)\nimport xx\n\nclass Tests(unittest.TestCase):\n def test_xx(self):\n for attr in ('error', 'foo', 'new', 'roj'):\n self.assertTrue(hasattr(xx, attr))\n\n self.assertEqual(xx.foo(2, 5), 7)\n self.assertEqual(xx.foo(13,15), 28)\n self.assertEqual(xx.new().demo(), None)\n if support.HAVE_DOCSTRINGS:\n doc = 'This is a template module just for instruction.'\n self.assertEqual(xx.__doc__, doc)\n self.assertIsInstance(xx.Null(), xx.Null)\n self.assertIsInstance(xx.Str(), xx.Str)\n\n\nunittest.main()\n"] stdout: --- --- stderr: --- Traceback (most recent call last): File "<string>", line 9, in <module> ImportError: /tmp/tmp3y4hpawx/xx.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add --- =========================================== Python is built GCC with -ftest-coverage option.
But it seems like this option is "leaked" to C flags used to build third party extensions in distutils. Maybe CFLAGS_NODIST should be used instead of CFLAGS. The Travis CI job runs these commands: --- ./configure xvfb-run make -j4 coverage-report make pythoninfo bash <(curl -s https://codecov.io/bash ) -y .github/codecov.yml --- msg366140 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-10 16:55 > Maybe CFLAGS_NODIST should be used instead of CFLAGS.
Looks like it worked. > Python is built GCC with -ftest-coverage option. More exact description is: Python is built GCC with --coverage option, --coverage option including -ftest-coverage option. msg366141 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-10 16:58 > Looks like it worked. Oh, sorry, I checked the wrong gate. msg366198 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-11 10:41 After `CFLAGS` replcaced by `CFLAGS_NODIST`, some extension module built failed, some info like: *** WARNING: renaming "_struct" since importing it failed: build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add the possible reason: building '_struct' extension gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -O0 -pg --coverage -I./Include/internal -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_struct.c -o build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o gcc -pthread -shared (xxxxxxxxxxlacking --coverage in here xxxxxxxxx) build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o -L/usr/local/lib -o build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so msg366199 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-11 10:51 FWIW, gcc Instrumentation Options in https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html msg366679 - (view) Author: STINNER Victor (vstinner) * Date: 2020-04-17 21:12 I understand that with PR 19460 , code coverage is not longer run on C extension modules of the stdlib.
I'm not using C code coverage, so I don't know how it's supposed to be used or analyzed. Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave. The other option is to only fix test_distutils. msg366693 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-18 05:24 > Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave. Copy that.
I found PR7773 updated the travis' coverage ci and a discussion issue in https://github.com/python/core-workflow/issues/2 , so I add brett and ammar to the noisy list, MAYBE they know the historical background. msg366867 - (view) Author: Brett Cannon (brett.cannon) * Date: 2020-04-20 19:06 The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it. msg366921 - (view) Author: Hai Shi (shihai1991) * Date: 2020-04-21 15:23 > The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.
Hm, code coverage of C code including two parts: 1) coverage of inner c code; 2) extension modules built by distutils; If I understand correct, we force on 1) is good enough, no? msg366924 - (view) Author: STINNER Victor (vstinner) * Date: 2020-04-21 15:41 Brett: > The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it. If there is no user of code coverage *on pull requests*, maybe the simplest fix is to remove this job.
There is also Codecov service running somehow on Python branches through GitHub integration. msg368869 - (view) Author: STINNER Victor (vstinner) * Date: 2020-05-14 21:48 It seems like results at published at: https://codecov.io/gh/python/cpython/ It seems like there is coverage on both C and Python code. I never used this service. msg368926 - (view) Author: Hai Shi (shihai1991) * Date: 2020-05-15 05:01 > It seems like there is coverage on both C and Python code.
Great, MAYBE we can use this coverage result to improve our testcases. msg371665 - (view) Author: STINNER Victor (vstinner) * Date: 2020-06-16 15:06 See also bpo-40993 : "Don't run Python and C coverage jobs of Travis CI on pull requests". msg371716 - (view) Author: STINNER Victor (vstinner) * Date: 2020-06-17 08:00 See also Coverage report in Github CI for standard library discussion: https://discuss.python.org/t/coverage-report-in-github-ci-for-standard-library/2836 msg371717 - (view) Author: STINNER Victor (vstinner) * Date: 2020-06-17 08:03 Another issue: the "Test code coverage (C)" job on Travis CI times out after 50 minutes: https://travis-ci.org/github/python/cpython/jobs/699170857 0:44:44 load avg: 7.76 running: test_multiprocessing_spawn (11 min 38 sec), test_concurrent_futures (9 min 33 sec), test_unparse (25 min 7 sec), test_capi (1 min) 0:45:14 load avg: 7.91 running: test_multiprocessing_spawn (12 min 8 sec), test_concurrent_futures (10 min 3 sec), test_unparse (25 min 37 sec), test_capi (
The rest of this page is on the live site.