Repository navigation
unittest runner thinks failing test is not run #17561
Description
Environment data
- VS Code version: 1.60.2
- Extension version (available under the Extensions sidebar): v2021.9.1246542782
- OS and version:
Ubuntu 20.04.3 LTS
Linux x86_64 5.4.0-88-generic - Python version (& distribution if applicable, e.g. Anaconda): 3.8.10
- Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
- Relevant/affected Python packages and their versions: none?
- Relevant/affected Python-related VS Code extensions and their versions: none?
- Value of the
python.languageServersetting: pylance
Expected behaviour
Reports failed test
Actual behaviour
Reports passing test/thinks test is still not run
Steps to reproduce:
Run below test using vscode:
import unittest
class Tests(unittest.TestCase):
def test(self):
with self.subTest():
assert FalseLogs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
> ~/.virtualenvs/libre-riscv-venv/bin/python ~/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=*test*.py --uvInt=2 --result-port=35131 -ttest_reproducer.Tests.test --testFile=/home/jacob/projects/libreriscv/simd_signal/test_reproducer.py
cwd: ~/projects/libreriscv/simd_signal
Output for Test Output in the Terminal panel
Running tests (unittest): /home/jacob/projects/libreriscv/simd_signal/test_reproducer.py
Running tests: [object Object]
Total number of tests expected to run: 1
Total number of tests run: 1
Total number of tests passed: 0
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Finished running tests!
> Test run finished at 9/28/2021, 5:56:12 PM <
Output for Python Test Log in the Output panel
test (test_reproducer.Tests) ...
======================================================================
FAIL: test (test_reproducer.Tests) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jacob/projects/libreriscv/simd_signal/test_reproducer.py", line 7, in test
assert False
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Activity
coppinri commentedon Sep 29, 2021
This seems to be the behaviour particular to subtests.
Worse though, if the test passed previously, but now the test assertion fails, the UI still shows it as passed.
karthiknadig commentedon Sep 30, 2021
Our test adapter for
unittest, when it was written was not designed to handleunittestsubtests. This is primarily due tounittestdiscovery cannot detect them before running the test. when subtests are used, the results are generated slightly differently from when they are not used. So we currently don't have a way to match the before subtest state.We are in the process of re-writing the test adapter. With the redesign we will take unittest sub tests into consideration.
karthiknadig commentedon Sep 30, 2021
Related #17242
karthiknadig commentedon Sep 30, 2021
Jacob Lifshay (@programmerjake) I made a PR (#17578) that addresses this issue in some form:
But it won't make it into up coming release. I will share the build from this PR when it is done here so you can try it out. For the code that yo shared the result should look like this:
coppinri commentedon Oct 1, 2021
Wow, I was only expecting it to update the UI on a failed run, but if it's able to delve into- and report on- each subtest that would be great.
karthiknadig commentedon Oct 1, 2021
Richard Coppin (@coppinri) There are limitations to this. We can only detect sub-test at run time,
unittestdiscovery does not know about sub-tests. The subtest nodes you see above are only reporting nodes, you can specifically select them and run.if you want to give this a try, you will need insiders version of VS Code (v 1.61.0-insider) and this build of python extensions https://github.com/microsoft/vscode-python/suites/3925208299/artifacts/98213954