Skip to content

Investigate run in terminal tool hanging issues #255396

Description

@Tyriar

This is a meta issue for the plan to investigate all issues around hanging in the terminal. For the June release one major case was fixed related to conpty being off #252489, but we still get many reports for all OS's.

Some of these will likely be helped with the move of the tool from core as we don't need to rely on the extension API and can touch terminal internals #252650

Activity

  1. Tyriar commented on Jul 21, 2025

    @Tyriar
    ContributorAuthor

    Previous details from manual testing in https://github.com/microsoft/vscode-copilot/issues/14065#issuecomment-2741375544 back in March

    Test matrix below.

    • Success means that the output ("run ls in the terminal") was correct exactly in the tool call for rich calls, and correct as expected for basic/none (may include surrounding lines).
    • SI sequences are what shows in the terminal hover after running the test
    OS Shell Prompt SI injection SI quality SI sequences Success?
    macOS zsh default rich P, D, A, B, E, C ✅
    macOS zsh default disabled none N/A ✅
    macOS zsh oh-my-zsh rich P, 7;file://..., D, A, B, E, C ✅
    macOS zsh powerlevel10k basic A, B, C, D ✅1
    macOS zsh starship rich P, D, A, B, H, I, E, C ✅
    macOS bash default rich P, A, B, E, C, D ✅
    macOS bash default disabled none N/A ✅
    macOS bash WT SI disabled basic D, A, B ✅
    macOS bash starship rich P, A, B, E, C, D ✅
    macOS fish default disabled none N/A ✅
    macOS fish default rich P, E, C, D, A, B ✅2
    macOS pwsh default rich P, A, B, E, C, D ✅
    macOS pwsh default disabled none N/A ✅
    macOS pwsh oh-my-posh rich P, A, B, D, E, C ❌3
    macOS pwsh WT SI basic A, 9;9, B, D ✅
    Windows pwsh default disabled rich N/A ✅
    Windows pwsh default rich P, A, B, E, C, D ✅
    Windows pwsh posh-git rich P, A, B, E, C, D ✅
    Windows pwsh oh-my-posh rich P, A, B, D, E, C ❌3
    Windows pwsh starship rich P, A, B, E, C, D ✅
    Windows pwsh starship disabled none N/A ✅
    Windows pwsh WT SI disabled basic A, 9;9, B, D ✅
    Windows Git bash default rich P, A, B, E, C, D ✅
    Windows Git bash default disabled none N/A ✅

    1: Prompt input model does really not work well, was using transient/instant prompt and it has a lot of different configurations in the wizard
    2: E, C, D happens before first prompt #244236
    3: No output was tracked, D happens before E, C #244169

  2. Tyriar commented on Jul 21, 2025

    @Tyriar
    ContributorAuthor

    Current telemetry is showing approximately 98-99% of tool calls "succeed", this means specifically that it did not throw in the block that executes the execution strategy:

    } catch (e) {
    this._logService.debug(`RunInTerminalTool: Threw exception`);
    toolTerminal.instance.dispose();
    error = 'threw';
    throw e;
    } finally {
    const timingExecuteMs = Date.now() - timingStart;
    this._sendTelemetry(toolTerminal.instance, {
    didUserEditCommand,
    didToolEditCommand,
    isBackground: false,
    shellIntegrationQuality: toolTerminal.shellIntegrationQuality,
    error,
    isNewSession,
    outputLineCount,
    exitCode,
    timingExecuteMs,
    timingConnectMs,
    });
    }

    This 98-99% is pretty consistent across shell types and platforms.

  3. Tyriar commented on Jul 21, 2025

    @Tyriar
    ContributorAuthor

    We could use execution time as a proxy for hanging, this should drop if we reduce hanging cases as manual intervention will always take longer. It is certainly expected that some commands take this long though.

    Image
  4. Tyriar commented on Jul 22, 2025

    @Tyriar
    ContributorAuthor

    Going forward we can differentiate canceled requests vs an unexpected exception, plus we break down bucket sizes. So we have a much better mechanism for tracking improvements we make in this area now.

  5. Tyriar commented on Jul 23, 2025

    @Tyriar
    ContributorAuthor

    Collecting all issues, this will be updated over time:

    ➡️: Deferred
    ✅: Verified and fixed
    ⛔: Not related to terminal tool

    Searched all open issues for terminal stuck and terminal hang:

    ? Link VS Code Ext OS Shell Notes
    microsoft/vscode-copilot-release#9331 1.100.2 0.27.1 Linux bash?
    microsoft/vscode-copilot-release#10992 1.100.0 0.27.0 Linux bash?
    microsoft/vscode-copilot-release#12495 1.102.0-ins ? Windows pwsh 5?
    ➡️ #258291 1.100.2 ? macOS zsh? Related to requiring input
    ✅ microsoft/vscode-copilot-release#12821 1.101.0 ? macOS bash/starship API continuation prompt problem, fixed with move to core
    #252492 (1) 1.100.2 ? macOS zsh/oh-my-zsh/powerlevel10k
    #252492 (2) ? ? macOS zsh/oh-my-zsh
    #252493 ? ? ? bash/bash-git-prompt
    #253782 1.101.2 0.28.5 Windows pwsh 5?
    ✅ #254447 1.99.0-ins 0.26 Windows pwsh 7 OP likely fixed conpty problem, big meta issue. Closed/locked and added comment on how to report issues
    ✅ #254724 1.102.0-ins ? Linux bash? API sub-execution problem, fixed with move to core
    ✅ #256855 1.102.1 0.29.1 Linux/snap bash? API sub-execution problem, fixed with move to core
    ✅ #256419 1.102.1 ? macOS zsh? API sub-execution problem, fixed with move to core
    ➡️ microsoft/vscode-copilot-release#13456 1.101.1 0.28.2 macOS zsh Idle timeout
    ➡️ #254471 1.101.2 0.28.5 Windows pwsh 5 Idle timeout
    ➡️ #255932 1.101.2 0.28.5 Windows->Container sh? Idle timeout
    ➡️ #254426 1.102.0-ins 0.29 macOS zsh/powerlevel10k? Agent gave bad quoting, dquote> hint on line
    ⛔ microsoft/vscode-copilot-release#12405 1.100.2 0.27.3 macOS zsh? Hang after terminal has finished

    WSL issues:

    ? Link VS Code Ext OS Shell Notes
    ✅ #252084 1.101.1 ? Windows->Linux bash?
    ✅ #253078 1.101.1 0.28.4 Windows->Linux bash?
    ✅ #257065 1.102.1 ? Windows->Linux bash?
    ✅ #255651 1.102.0 0.29.0 Windows->Linux bash? Large python command
  6. Tyriar commented on Jul 23, 2025

    @Tyriar
    ContributorAuthor

    Considering bringing back these recording style tests to help cover all these shell cases: #240299

  7. Tyriar commented on Jul 24, 2025

    @Tyriar
    ContributorAuthor

    Adding some recording tests in #257517, this will let us so some important assertions for shell edge cases we find as part of this reliability work.

  8. shanselman commented on Jul 25, 2025

    @shanselman
    Member

    Mark Russinovich and his TA Yanan Cai see this all day. Both are using either a remote SSH session or WSL. I also see it often in WSL on my arm machine. Both could provide reliable repros.

    And I wouldn't necessarily call this a hang, it's the copilot can't SEE what's happening in the term

  9. Tyriar commented on Jul 25, 2025

    @Tyriar
    ContributorAuthor

    Mark Russinovich and his TA Yanan Cai see this all day. Both are using either a remote SSH session or WSL. I also see it often in WSL on my arm machine. Both could provide reliable repros.

    And I wouldn't necessarily call this a hang, it's the copilot can't SEE what's happening in the term

    Investigated and it looks like it's fixed in Insiders as I can only reproduce in Stable 🎉

  10. Tyriar commented on Jul 30, 2025

    @Tyriar
    ContributorAuthor

    Going to call this completed, here's a summary of what happened this iteration:

    • Moved the terminal tool to core, this improved performance+stability as it moved the tool to the richer/lower level internal APIs as opposed to the more ergonomic/higher level/more complex APIs exposed to extensions. Move run in terminal tool to core #255804
    • Added a bunch of properties to the telemetry event. We can now more accurately monitor when things are going wrong in production. Highlights:
      • Error rate (canceled or exception) has been reduced from 2.4% to ~1%. We also can now also differentiate unexpected exceptions and the tool call being canceled, with the vast majority now being cancelation. Cancelations are expected in normal use, but they would be higher when the tool hangs. Actual unexpected exceptions are negligible (< 0.1%) Differentiate unexpected exceptions from canceled errors #258225
    • Tested many of the issues, some of them were marked deferred above and are known edge cases that are out of the scope of this.
    • Added recorded regression tests for command detection Add recorded tests for command detection #257517
  11. EmilStenstrom commented on Aug 1, 2025

    @EmilStenstrom

    Will this fix be part of the 1:st Aug stable release?

  12. Tyriar commented on Aug 2, 2025

    @Tyriar
    ContributorAuthor

    Emil Stenström (@EmilStenstrom) it'll come in the upcoming stable release, likely to be released around the 7th of August

  13. timaro commented on Aug 20, 2025

    @timaro

    Not sure if this is relevant, but since update to Version: 1.103.1 (Universal) on OSX, I'm getting hangs in tests under Ruby MiniTest -- parallel or sequential -- whenever code forks. In my case, specifically, I'm forking using Open3.capture3 to run Tika:

    stdout, stderr, status = Open3.capture3("#{tika_cmd} -x #{file.path.shellescape}")

    Shell is Bash. Hanging only in VSCode.

  14. Tyriar commented on Aug 20, 2025

    @Tyriar
    ContributorAuthor

    Tim Robertson (@timaro) are you talking about Copilot specifically? If not then it's unrelated and not sure what you mean by hanging. See microsoft/vscode-copilot-release#12701 (comment) for the type of info I need to investigate if you are talking about Copilot.

  15. timaro commented on Aug 20, 2025

    @timaro
  16. timaro commented on Aug 21, 2025

    @timaro

    Here's two screenshots - one of the requested terminal info, and another of the outcome of interrupting the deadlocked process with Ctrl-C. This is completely reproducible, and it doesn't matter if I'm running a single-threaded test, or multi-threaded test (single threaded shown in screenshots). I haven't tried with another command, but I suspect it doesn't matter -- it's the fork that's causing the issue.

    This test runs reliably outside of VSCode.

    Image Image
  17. Tyriar commented on Aug 21, 2025

    @Tyriar
    ContributorAuthor

    Tim Robertson (@timaro) if the terminal never returns to the shell's prompt it means it's unrelated to this issue. You're better off creating a new issue for this.

    This test runs reliably outside of VSCode.

    The main difference with us and other terminals wrt these types of problems is the shell's env, diffing that might reveal a problem

  18. locked as off topic and limited conversation to collaborators on Aug 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

chat-terminalThe run in terminal tool in chatplan-itemVS Code - planned item for upcoming

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions