Repository navigation
Test discovery should be fault tolerant and display all successfully imported files #13301
Description
Activity
Were you able to get
pytestorunittestto show tests via command line, even with import errors?Yes,
pytest --collect-onlycorrectly detect the valid tests:============================= test session starts ============================== platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /home/epot/__A__/Code/datasets, configfile: pytest.ini plugins: xdist-1.34.0, forked-1.3.0 collected 1685 items / 1 error / 1684 selected <Package tensorflow_datasets> <Module import_test.py> <UnitTestCase ImportTest> <TestCaseFunction test_import> <TestCaseFunction test_session> <Module import_testing_test.py> <UnitTestCase ImportTestingTest> ... # The 1684 detected tests are correctly displayed =========================== short test summary info ============================ ERROR tensorflow_datasets/scripts/documentation/build_api_docs_test.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! ======================== 5 warnings, 1 error in 35.20s =========================Additionally,
pytest --continue-on-collection-errorscorrectly execute the valid tests.Shouldn't the
info neededlabel be removed ?--continue-on-collection-errorsmakes test discovery bit better when there are issues with loading test files. However, adding support that will require a bit of work in the current test adapter. We have plans to update the adapter to be more robust. We also have to see if similar support can be provided on other supported test frameworks.3 remaining items
I am facing the same error with the plugin. For manual test discovery in a terminal I can fix the error by executing:
pip install -e LOCAL_PACKAGE_PATHsource ~/ROS2_WORKSPACE
Can I prepend these command somehow to the plugin, so that I could use the test discovery?
Hello! This should be fixed in the rewrite which is currently underway! The meta-issue is here and that will track progress but I will keep this issue open as well until resolved. Thanks!
- added a commit that references this issue
on Jul 13, 2023 Hello! A fix for this is out on insiders! If someone could tests and let me know it worked that would be great!
Verification steps:
- take following code and make two files with each block
- run pytest or unittest discovery
- check that there is a node with the tests that can be discovered and one saying there was a failure
def test_example(): assert 1 == 1 def test_two(): assert Trueimport unittest import foo #doesn't exist so will error class OtherTest(unittest.TestCase): def testtwo(self): self.assertEquals(add(4, 5), 9) if __name__ == "__main__": unittest.main()- locked as resolved and limited conversation to collaborators
on Aug 25, 2023
Currently, pytest discovery fail for the entire project if a single import is missing on a single file:
Only 1/600 file failed, but VS Code does not show the 599 tests which were correctly discovered.
Many other project have "optional" deps which add extra features. Those extra deps shouldn't be required when the full project do not need to be tested.
My project has many independent modules, each requiring a specific set of dependencies (
pip install .[module0]). Contributors should not be forced to install the full dependencies of all modules / features when they only want to test a subpart.Related issues:
Those issues are not duplicate but seems related: