WSL2 -very slow bash autocompletion since version 1.26.0

[returntrip]

What happened:

Since updating to kubectl 1.26.0 on both my WSL2 distros (Ubuntu and Fedora Remix). Bash automcompletion is super slow.

Ubuntu was 1.25.2
Fedora was 1.25.4

What you expected to happen:

Autocompletion should be as fast as < 1.26.0

How to reproduce it (as minimally and precisely as possible):

Update to 1.26.0. Type kubectl followed by TAB

Anything else we need to know?:

This seems to be a known issue in WSL. But, I am not experiencing it with other tools like: flux or tanzu (for which I installed completions manually). So it seems a combination of a new version of kubectl and the WSL Windows Path "issue".

Environment:

• Kubernetes client and server versions (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

• Cloud provider or hardware configuration: NA
• OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

NAME="Fedora Remix for WSL"
VERSION="37"
ID=fedoraremixforwsl
ID_LIKE=fedora
VERSION_ID=37
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Remix for WSL"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
HOME_URL="https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL"
SUPPORT_URL="https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL"
BUG_REPORT_URL="https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL/issues"
PRIVACY_POLICY_URL="https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL/blob/master/PRIVACY.md"
FEDORA_REMIX_VERSION=37.0.2

[k8s-ci-robot]

@returntrip: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[marckhouzam]

Thanks @returntrip for the report.
What version of kubectl were you running before which was not having this problem?

How do you setup shell completion?

[returntrip]

Ubuntu was 1.25.2
Fedora was 1.25.4

I just bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

But I have also trying sourcing directly on the shell source <(kubectl completion bash) with same results

[marckhouzam]

So now you are running kubectl 1.26. What version of kubectl were you running before?

I'm asking because the only change I'm aware of between 1.25 and 1.26 is that we've enabled completion descriptions. However we've also tested for performance for this change. On the other hand, if you were previously running an even older kubectl, then the problem may be something else that changed before 1.26

[returntrip]

So now you are running kubectl 1.26. What version of kubectl were you running before?

I'm asking because the only change I'm aware of between 1.25 and 1.26 is that we've enabled completion descriptions. However we've also tested for performance for this change. On the other hand, if you were previously running an even older kubectl, then the problem may be something else that changed before 1.26

Sorry maybe I was not clear when I provided the versions in the original message and also above:

The kubectl versions were:

For Ubuntu: 1.25.2
For Fedora: 1.25.4

Or, are you looking for some other version numbers or the output of kubectl version?

Please let me know!

[marckhouzam]

Sorry, my mistake. I saw "Ubuntu" and didn't even look at the version assuming it was your os version.

[marckhouzam]

There's a couple of things you could do to help us know what is the root cause:

1- turn off completion descriptions, which were added in 1.26.
To do that, generate the completion script into a file and replace the __complete string with __completeNoDesc, then source the file. When you do shell completion you should not see any descriptions, and you can let us know if it is noticeably faster.
Here is what I would do:

kubectl completion bash | sed s,__complete,__completeNoDesc, > k
source k
kubectl [tab][tab]

2- Use the shell completion file of kubectl 1.25 with kubectl 1.26. This will show us if the slowdown is in the shell completion script or kubectl itself. Something like:

# install kubectl 1.25.  Downloading it is probably the simplest way
# Then source its completion script
source <(kubectl-1.25/kubectl completion bash)
# Now run kubectl 1.26
kubectl [tab]tab]

[returntrip]

Thanks for providing troubleshooting steps. I have tried both ways and neither seem to speed up kubectl 1.26.0 completions. The interesting thing is that 1.25.5 works fine with the 1.26.0 completion descriptions.

[marckhouzam]

@returntrip Are you still having this problem?

The interesting thing is that 1.25.5 works fine with the 1.26.0 completion descriptions.

That's interesting.
What if you run a completion request by hand, is it slow with 1.26 and fast with 1.25? To test that:

kubectl __complete desc<ENTER>

This will request completions from the kubectl binary just like the actual completion script does.

[Karocyt]

I had the exact same problem, but as a newcomer to WSL (among Windows other blessings cough), I only experienced 1.26.0.
Your completion request was behaving the same as autocompletion (hanging my WSL for about 20 seconds with 20% CPU usage on a couple cores).

Trying 1.25.5 fixed it, tested with bash and zsh.

1.25.5 works with both generated completions, 1.26.0 hangs with both, so either the binary is doing something messed up enough to unsettle WSL, or the other way around.

The last commits seems to list stuff when completing plugin stuff, can't it be related to WSL slow IO ?

Adding a single dummy plugin with 2 options doesn't change much, but hangs while completing mid second word too, while this seems to behave correctly with 1.25.5.

Happy to just downgrade to 1.25.5 for now anyway, and learned quite a lot about completion intricacies.

[marckhouzam]

Thanks for the info @Karocyt. This points to the fact that kubectl 1.26 now completes plugin names also, which goes through the entire list of paths present in $PATH.

To confirm you can check if listing plugins is slow by doing : kubectl plugin list

[Karocyt]

kubectl plugin list seems to be the culprit indeed, but if it's about listing plugins, why is it hanging for plugins args too ? At this stage there would be no point listing plugins as a name was already given and is set in stone...

EDIT: Removing all the Windows paths from $PATH fixed it !

Windows appends the content of the whole Windows PATH variable to Ubuntu's $PATH. I can't find a specific culprit but it seems that the more Windows files on the Path, the slower it gets. I got down to 5 seconds (from 20-30) removing the bigger half of it, and it's instant if I clean everything.
I guess there is some kind of translation layer for Windows files, and listing all these folders/files eats up a lot.

So it is technically a WSL "bug", but this one was closed long ago (it seems that accessing Windows executables from WSL was one of their more appraised features, don't ask me why), so this will need a hack/fix on Kubernetes side...

Maybe stripping /mnt/one-letter-drive/* from PATH when on WSL ?

if [[ $(grep -i Microsoft /proc/version) ]]; then
    echo "Bash is running on WSL"
fi

[marckhouzam]

Ok so we know problem now. Thanks again @Karocyt ! Using the very long PATH should only affect completion for a limited set of scenarios:

• completing the first argument (right after "kubectl")
• completion for any plugin and its args

why is it hanging for plugins args too?

Once kubectl detects it is doing completion for a plugin's arguments, it looks through the PATH variable looking for a script called kubectl_complete-<pluginName> which will provide the completion choices. Most plugins don't yet provide such a script so kubectl goes through the entire PATH looking for the script.

BTW, this PATH problem will also make using any kubectl plugin very slow.

This comment says you can actually configure WSL to not append the windows paths: microsoft/WSL#4234 (comment)

But if you do need the windows paths to be included in PATH, there isn't much you can do... What I can think of is wrapping kubectl in a function that changes the PATH variable just for the kubectl invocation.

[Karocyt]

The only Windows binary that I'm using (consciously) from within WSL is VSCode.exe aliased as code (for the code . command, to open my working directory on Windows VSCode).
So I could delete all the /mnt/c/* from my PATH and keep VSCode as an alias...

A documentation page on usage with WSL should recommend to disable Windows Paths and set specific aliases for the binaries you need.
This should be easy enough to stumble upon while confronted to that kind of stuff.

[eddiezane]

Adding a section on kubectl + WSL to the docs sounds like a good idea. Please open that over at https://github.com/kubernetes/website. We can collect what should be in it there.

/close

[k8s-ci-robot]

@eddiezane: Closing this issue.

In response to this:

Adding a section on kubectl + WSL to the docs sounds like a good idea. Please open that over at https://github.com/kubernetes/website. We can collect what should be in it there.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rlipscombe]

Most plugins don't yet provide such a script so kubectl goes through the entire PATH looking for the script.

The plugin I'm using -- https://github.com/rlipscombe/kubectl-ssh -- does provide a completion script, and it's in the first entry in PATH, so why is it still slow?

[marckhouzam]

Most plugins don't yet provide such a script so kubectl goes through the entire PATH looking for the script.

The plugin I'm using -- https://github.com/rlipscombe/kubectl-ssh -- does provide a completion script, and it's in the first entry in PATH, so why is it still slow?

Is it always slow? Or only for the first argument after kubectl?

Is it still slow when you do kubectl ssh <TAB>?

[marckhouzam]

Most plugins don't yet provide such a script so kubectl goes through the entire PATH looking for the script.

The plugin I'm using -- https://github.com/rlipscombe/kubectl-ssh -- does provide a completion script, and it's in the first entry in PATH, so why is it still slow?

I looked at the code again. I see that when it’s doing completion for a specific plugin, the code still looks for all plugins. That would explain why things remain slow when the WSL path contains all the windows paths.

I'm talking about this function:

kubectl/pkg/cmd/plugin/plugin_completion.go

Line 82 in bc9a57e

func addPluginCommands(cmd *cobra.Command) {

this could be optimized.

[rlipscombe]

Is it still slow when you do kubectl ssh ?

To be clear: it's still (tens of seconds) slow for kubectl ssh <TAB> and kubectl ssh -n foo<TAB>.

[rlipscombe]

this could be optimized.

Should this issue be reopened, or another one created?

[marckhouzam]

this could be optimized.

Should this issue be reopened, or another one created?

The only way to avoid the slow down in every scenario is to remove the Windows paths as described in comments further above.

Optimizing the code as we've discussed would probably improve the situation for completing commands of a plugin but only if, as you have taken the time to insure, the completion script of the plugin is in the first path on the list.

It would still not improve completion for the first argument, e.g.,kubectl <TAB>.

So I would suggest opening a different issue explaining things, if you feel it's worth it, and after that it will depend if anyone feels it's worth working on.

[allanwilson]

I know this is old and I can open a new issue if needed. I experience this same slowness on MacOS (Venturs 13.5) as well.

kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.3
Kustomize Version: v4.5.7
Server Version: v1.25.12-eks-2d98532

kubectl plugin list

Unable to read directory "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin" from your PATH: open /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin: no such file or directory. Skipping...
Unable to read directory "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin" from your PATH: open /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin: no such file or directory. Skipping...
Unable to read directory "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin" from your PATH: open /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin: no such file or directory. Skipping...
The following compatible plugins are available:

/Users/xxxx/bin/kubectl-convert