bash command completion has a problem with sudo command

[mug896]

Describe the bug

When the command completion function starts running
there are three variables automatically setting $1, $2, $3.
and $2 is not always the same as ${COMP_WORDS[COMP_CWORD]}.

1. if $2 value is a member of $COMP_WORDBREAKS variable
   then it's value is empty by default

   ex) command --foo=[tab]
   $2 : (empty) ${COMP_WORDS[COMP_CWORD]} : =

2. if $2 value start with " or ' (single or double quote)
   then it's quote removed ( "foo --> foo )

   ex) command --foo "bar[tab]
   $2 : bar ${COMP_WORDS[COMP_CWORD]} : "bar

this is an intended work for making completion function simpler
but if i use sudo in front of a command then
$2 value is always the same as ${COMP_WORDS[COMP_CWORD]}
This is a different behavior to the bash default and problematic.
and ends up can not use $2 variable and force to use
${COMP_WORDS[COMP_CWORD]} to support sudo

Versions

• Operating system name/distribution and version:
  Operating System: Ubuntu 22.04.1 LTS
  Kernel: Linux 5.15.0-43-generic
  Architecture: x86-64
• bash version: 5.1.16(1)-release
• bash-completion version: 2.11

[akinomyoga]

The call of the completion functions for the commands after another command (such as sudo) is emulated by _command_offset. In particular, the arguments passed to the completion function is determined here.

• if $2 value is a member of $COMP_WORDBREAKS variable
  then it's value is empty by default
  ex) command --foo=[tab]
  $2 : (empty) ${COMP_WORDS[COMP_CWORD]} : =
• if $2 value start with " or ' (single or double quote)
  then it's quote removed ( "foo --> foo )
  ex) command --foo "bar[tab]
  $2 : bar ${COMP_WORDS[COMP_CWORD]} : "bar

Is this the exact rule? If the exact rule is known, I think we can implement it in _command_offset, but Bash Reference Manual doesn't seem to clarify the details.

Bash Reference Manual - §8.6 Programmable Completion

[...] When the function or command is invoked, the first argument ($1) is the name of the command whose arguments are being completed, the second argument ($2) is the word being completed, and the third argument ($3) is the word preceding the word being completed on the current command line. [...]

I have tried it a little with the plain Bash (5.1.16/Fedora 36), but the rule seems to be a bit more complicated. For special characters in COMP_WORDBREAKS (particularly = and :),

$ _test1() { printf '\e7\n\e[J'; declare -p COMP_WORDS COMP_CWORD; (($#)) && printf '<%s>' "$@"; printf '\e8'; } && complete -F _test1 test1
$ cat -A <<< "$COMP_WORDBREAKS"
 ^I$
"'@><=;|&(:$
$ test1 --foo==[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo" [2]="==")
declare -- COMP_CWORD="2"
<test1><><--foo>
$ test1 --foo=:[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo" [2]="=:")
declare -- COMP_CWORD="2"
<test1><><--foo>
$ test1 --foo'=[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo'=")
declare -- COMP_CWORD="1"
<test1><=><test1>
$ test1 --foo =[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo" [2]="=")
declare -- COMP_CWORD="2"
<test1><><--foo>
$ test1 --foo '=[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo" [2]="'=")
declare -- COMP_CWORD="2"
<test1><=><--foo>
$ test1 --foo '='[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="--foo" [2]="'='")
declare -- COMP_CWORD="2"
<test1><'='><--foo>

For double quotation,

$ test1 a"b[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="a\"b")
declare -- COMP_CWORD="1"
<test1><b><test1>
$ test1 a"b"[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="a\"b\"")
declare -- COMP_CWORD="1"
<test1><a"b"><test1>
$ test1 a"b"c[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="a\"b\"c")
declare -- COMP_CWORD="1"
<test1><a"b"c><test1>
$ test1 a"b"c"[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="a\"b\"c\"")
declare -- COMP_CWORD="1"
<test1><><test1>

Also, $2 changes depending on the position of the cursor (where the cursor position is indicated by [TAB]):

$ test1 a[TAB]bcd
declare -a COMP_WORDS=([0]="test1" [1]="abcd")
declare -- COMP_CWORD="1"
<test1><a><test1>
$ test1 ab[TAB]cd
declare -a COMP_WORDS=([0]="test1" [1]="abcd")
declare -- COMP_CWORD="1"
<test1><ab><test1>
$ test1 abc[TAB]d
declare -a COMP_WORDS=([0]="test1" [1]="abcd")
declare -- COMP_CWORD="1"
<test1><abc><test1>
]$ test1 abcd[TAB]
declare -a COMP_WORDS=([0]="test1" [1]="abcd")
declare -- COMP_CWORD="1"
<test1><abcd><test1>

@mug896 Could you investigate and summarize the exact rule?

[mug896]

I asked this problem to the current bash maintainer.
and he said without bash-completion installed get the same null second argument
with or without specifying 'sudo'

I think no needs to parse quotes directly
In _sudo function just backup the $2, $3 variables
and pass the values to the _hello function something like this

bash$ sudo hello aa bb cc

complete -F _sudo sudo
complete -F _hello hello

_sudo() {
	local two_bak=$2 three_bak=$3     # backup $2, $3
	. . .
	_sudo_sub
}
_sudo_sub() {
	. . .
	_hello "hello" "$two_bak" "$three_bak"
}

[akinomyoga]

I think that would only work if we would just care about sudo, but we also need to care about completions for other commands that calls _command_offset. Also, to save the original arguments passed to the completion functions, we need to change the basic use pattern of bash-completion utility functions, which starts from the declaration of a pre-defined set of local variables and _init_completion. Furthermore, _command_offset and other utilities of bash-completion are called by external completions, i.e., custom completion functions written by users or third-party completion functions written by authors of applications, which makes it difficult to introduce breaking changes to the existing utility functions.

For these reasons, to follow your suggestion, we need to pay large efforts. We need to 1) design a new basic interface of bash-completion utilities for completion functions with new names or options, 2) rewrite all the existing completion functions to follow the new interface (there are currently 567 such functions in the repository), and 3) deprecate related existing functions or usages like _init_completion || return. We also need to 4) announce the interface changes to the authors of external completions and 5) submit pull requests if necessary. The question is whether it is worth making such large efforts to solve this issue. In particular, basically, we cannot change the basic interface used by external completions unless there is a sufficient reason. Even if we decided to break the existing interface, this would be a several-year project (including design discussion, implementations, tests, and reviews), and some continuous contributors need to voluntarily work on them, but at least I'm not willing to work on them.

[mug896]

check this script out.
i made this for fixing the bug of sudo completion function.
and it works for me
though i'm not using bash-completion package in my completion function.
i think you should fix this bug anyway.
because of this bug every completion function has to start with
local cmd=$1 cur=${COMP_WORDS[COMP_CWORD]} prev=${COMP_WORDS[COMP_CWORD-1]};
[[ ${COMP_LINE:COMP_POINT-1:1} = " " ]] && cur=""
and adds [[ $cur == "=" ]] && cur="" or
[[ $cur == @(\"*|\'*) ]] && cur=${cur:1} lines
instead of just writing local cmd=$1 cur=$2 prev=$3
to support bash-completion package

_sudo() 
{
    local arr func cmd=${COMP_WORDS[1]} cur=$2 i
    if (( COMP_CWORD == 1 )); then
        COMPREPLY=($( compgen -A command -- "$cur" ))
        return
    fi
    if ! complete -p "$cmd" &> /dev/null; then
        _completion_loader "$cmd" &> /dev/null
    fi
    if arr=( $(complete -p "$cmd" 2> /dev/null) ); then
        for (( i = 1; i < ${#arr[@]}; i++ )); do 
            [[ ${arr[i]} == -F ]] && { func=${arr[i + 1]}; break ;}
        done
        if [[ -n $func ]]; then 
            COMP_LINE=${COMP_LINE##$1+( )}
            let COMP_POINT-="COMP_POINT - ${#COMP_LINE}"
            unset -v 'COMP_WORDS[0]'
            COMP_WORDS=( "${COMP_WORDS[@]}" )
            let COMP_CWORD--
            "$func" "$cmd" "$2" "$3"
        fi
    fi
}

complete -o default -o bashdefault -F _sudo sudo

[akinomyoga]

Thanks, but I know how to fix the issue if it is fine to fix only sudo as you are suggesting.

As I have explained in #790 (comment), the point is that we wouldn't like to fix the issue for each command like sudo, ionice, ccache, find, gdb, screen, strace, sudo, timeout, valgrind, watch, xvfb-run ... one by one separately. We should rather fix the utility function _command_offset because the implementation of _command_offset causes the present problem. The problem is that we cannot easily change the interface of _command_offset, so

• One option is to implement the special rule to determine $2 from ${COMP_WORDS{COMP_CWORD]} in _command_offset. This is what I initially suggested. This solves the issue without requiring any changes to the completion functions in bash-completion and in external completions.
• Another option is to save the original $2 passed by the readline to a fixed position and let _command_offset pick up from that position. However, this requires a change of the standard pattern local cur prev words cword split; _init_completion -s || return.
• Another option is to add a new implementation of _command_offset2 or add a new option to _command_offset to specify a preferred $2 as an argument. However, this also involves the interface change so we need to rewrite all the calls of _command_offset. Of course, we can rewrite all such instances in the bash-completion project, but it is not the case for external completions.

[mug896]

The problem is that the bash-completion package installed by default all linux distros.
so even users not using the package have to use the sudo completion function in the package.
The only command to concern is sudo command, not any other commands.
and the sudo completion function in the package change $2 default value
to ${COMP_WORDS[COMP_CWORD]}.
I had some tests with the sudo function in the package and my sudo function above
with ccache, find, gcc, strace, timeout, valgrind, watch ... commands.
and they all work the same without problems
because that commands use _init_completion || return. and not use $2 variable directly

[akinomyoga]

The problem is that the bash-completion package installed by default all linux distros. so even users not using the package have to use the sudo completion function in the package.

I am not trying to say that the current bash-completion doesn't need to be fixed. You don't have to repeat in every your reply that the current behavior is a problem and it needs to be fixed. We are talking about the solution. As I have mentioned in my previous replies, the solution that you are suggesting has another maintenance problem though I know that it solves the original problem of sudo. You are repeating that your suggestion solves the original problem, but you don't seem to provide a solution to the maintenance problem. Unless you provide a satisfactory answer to the maintenance problem or consider other solutions, I think the discussion doesn't proceed.

The only command to concern is sudo command, not any other commands.

Why?

and the sudo completion function in the package change $2 default value to ${COMP_WORDS[COMP_CWORD]}.

These other commands are also affected by the current behavior of bash-completion because of the same problem in _command_offset. I won't create the test cases of every command for you, but that can be easily reproduced by, for example,

$ _test1() { printf '\e7\n\e[J'; printf '<%s>' "$@"; printf '\e8'; } && complete -F _test1 test1
$ test1 --foo=
<test1><><--foo>
$ sudo test1 --foo=
<test1><=><--foo>
$ ccache test1 --foo=
<test1><=><--foo>
$ find . -exec test1 --foo=
<test1><=><--foo>
$ gdb --args test1 --foo=
<test1><=><--foo>
$ strace test1 --foo=
<test1><=><--foo>

I had some tests with the sudo function in the package and my sudo function above with ccache, find, gcc, strace, timeout, valgrind, watch ... commands. and they all work the same without problems because that commands use _init_completion || return. and not use $2 variable directly

Could you describe in more detail how you implemented or patched the completion functions for those commands and how did you test them?

[mug896]

I never thought of completions in the command line options that way.
I'm sorry about my ignorance.
my preferred way is to adding the second argument of $2 value
to the _command_offset function using the command like sed.
but there are external completions as you said.
It seems your first option is way to go
but that needs to parse the ${COMP_WORDS{COMP_CWORD]} value twice
in the script (including quotes, escape sequence). I think it's terrible.

[mug896]

how about using shopt -e extdebug option ?
if you enable extdebug option then you can obtain ancestor function's
argument count and values like this

#!/bin/bash

_command_offset() {
    echo "${FUNCNAME[0]} function argc: ${BASH_ARGC[0]}"
    echo "${FUNCNAME[0]} function arg1: ${BASH_ARGV[1]}"
    echo "${FUNCNAME[0]} function arg2: ${BASH_ARGV[0]}"
    echo "${FUNCNAME[1]} function argc: ${BASH_ARGC[1]}"
    echo "${FUNCNAME[1]} function arg1: ${BASH_ARGV[4]}"
    echo "${FUNCNAME[1]} function arg2: ${BASH_ARGV[3]}"
    echo "${FUNCNAME[1]} function arg3: ${BASH_ARGV[2]}"
}

_foo() {
    _command_offset 44 55
}

main() {
    _foo 11 22 33
}

shopt -s extdebug
main

#####################################

bash$ ./test.sh
_command_offset function argc: 2
_command_offset function arg1: 44
_command_offset function arg2: 55
_foo function argc: 3
_foo function arg1: 11
_foo function arg2: 22
_foo function arg3: 33

[akinomyoga]

my preferred way is to adding the second argument of $2 value > to the _command_offset function using the command like sed. but there are external completions as you said.

I'm actually currently thinking of the possibility of the hybrid solution of analyzing ${COMP_WORDS{COMP_CWORD]} and passing the original $2: We can add a new option to _command_offset to pass the original $2 (like _command_offset --cur="$2" $offset). When the original $2 is passed, _command_offset can just use the passed value. When the option is not specified, _command_offset analyzes ${COMP_WORDS{COMP_CWORD]} to determine a value. For the completion functions inside the bash-completion project, we can modify the codes to pass the original $2. It is better to modify also external completions so that the original $2 is passed to _command_offset, but they would still work mostly even if they are not modified because $2 is reconstructed from ${COMP_WORDS{COMP_CWORD]}.

It seems your first option is way to go but that needs to parse the ${COMP_WORDS{COMP_CWORD]} value twice in the script (including quotes, escape sequence). I think it's terrible.

I'm currently not sure how hard that is. Does it really require us an authentic shell parser? That is the reason we need to investigate the exact rule used by Bash in order to estimate whether this solution can be useful. If that can be processed by regular expressions like this in most cases, that is fine. It is possibly impossible to perfectly reproduce the original rule, but even in that case, I guess the most realistic cases can be covered by some regular expressions.

how about using shopt -e extdebug option ?

Actually, I also thought about that direction independently, but this direction also seems to have other problems that are hard to solve.

• First, extdebug doesn't just enable BASH_ARGV and BASH_ARGC, but also changes the semantics of DEBUG / ERR / RETURN traps. Specifically, we can set these traps for each function separately when extdebug is not turned off, but it becomes impossible when extdebug is turned on. This means that if we set extdebug globally, it can break existing configurations that use these traps. In particular, DEBUG trap is used by various Bash configurations like bash-preexec, iTerm2 shell integration, wezterm shell integration, starship, liqiuidprompt, etc.
• Another concern is the performance. With extdebug, the above traps start to be inherited by all the function calls, so the shell can be significantly slower than without extdebug. Even when no traps are set, I have doubts on the performance of BASH_ARGV and extdebug. Once I have measured the performance of a utility function that uses the hack of BASH_ARGV with extdebug introduced in Pure Bash Bible, but the result was by far inferior to the naive implementation. (By the way, the function of Pure Bash Bible is also broken by globally set extdebug, though it can be easily fixed.)

Then, another possibility that comes to our mind is to locally set extdebug instead of globally setting extdebug. We might think of temporarily turning extdebug on in _command_offset, getting the arguments, and finally unsetting extdebug (or restoring the original state of extdebug). However, this naive solution doesn't work because BASH_ARGV only contains the arguments after we set extdebug. When _command_offset is called, it is too late to set extdebug in order to retrieve the arguments of the caller by BASH_ARGV.

Next, we might consider locally setting extdebug inside the top-level completion function prior to the call of _command_offset. This can be technically possible if the top-level completion function calls _init_completion at the beginning. We can set extdebug when _init_completion is called. At the same time, _init_completion shall set the RETURN trap for the top-level completion function. We can reset extdebug in the RETURN trap. This kind of strategy was experimentally implemented in . PR #739 but is not yet fully tested. This strategy also has some other problems:

• We cannot ensure that the RETURN trap is called, for example, when the user cancels the completion by C-c (SIGINT). In such a case, we fail to restore the global setting of extdebug. Issue Aborting a completion attempt leaves set -o noglob in effect #786 is essentially the same problem. We may think of trapping SIGINT, but we are not sure if that covers all the similar cases.
• This doesn't always work for external completions. For this to work, _init_completion needs to be called from the top level of the completion function, but _init_completion has never been put such a restriction, so external completions may call it from inside child functions. Also, there seem to be even some external completion functions that do not call _init_completion but only call _command_offset. We can find such completion functions, e.g., here and there.

[mug896]

As you already know, quotes are members of the $COMP_WORDBREAKS variable
and have different behavior than other WORDBREAKERS.
The primary usage of quotes in tab completions is
to input multiple words sentence something like this

1) install                              
2) Linked Base for node-1 and node-2     # completion list
3) Linked Base for node-1 and node-3
4) hostname hosts setup

sh$ hello --foo "Linked Ba[tab]
sh$ hello --foo="Linked Ba[tab]
sh$ hello -A "Linked Ba[tab]

and if a user wants to include quotes in the completion list
then he first has to escape the quotes like (1) when assign COMPREPLY variable
in case (2) will not appear the quotes in the completion list when user enter tab
how to tab complete the quotes in the prompt ? you also need to escape the quotes to complete.

install
Linked Base for \"node-1\" and \"node-2\"     ---- (1)
Linked Base for "node-1" and "node-3"         ---- (2)
hostname hosts setup

nobody uses quotes in tab completion like this.

test1 a"b[TAB]
test1 a"b"[TAB]
test1 a"b"c[TAB]
test1 a"b"c"[TAB]

# but this is possible
test1 "ab[TAB]

also nobody uses tab completion on already completed word like this.
so these two cases are no need to concern to set the bash's original $2 variable.

test1 a[TAB]bcd
test1 ab[TAB]cd
test1 abc[TAB]d

# but this case is need to concern because following usage pattern
test1 [TAB]abcd    
sh$ hello -a foo -b bar      # user wants to change option argument of -a
sh$ hello -a [TAB]-b bar     # $2: empty    ${COMP_WORDS[COMP_CWORD]}: -b

# because of the current bash-completion bug every completion function 
# that does not use base-completion package has to add this line on top.
# if a user can use the bash's original $2 value then don't need this line
# [[ ${COMP_LINE:COMP_POINT-1:1} = " " ]] && cur=""

So my recommandation is that there is no need to parse
${COMP_WORDS[COMP_CWORD]} variable to set $2 variable.
and this is enough to set the original $2 variable.

if [[ ${COMP_WORDS[COMP_CWORD]} == @(\"*|\'*) ]]; then
    set -- "$1" "${COMP_WORDS[COMP_CWORD]:1}" "${COMP_WORDS[COMP_CWORD-1]}"
elif [[ $COMP_WORDBREAKS == *${COMP_WORDS[COMP_CWORD]}* ]] ||
    [[ ${COMP_LINE:COMP_POINT-1:1} = " "  ]]; then
    set -- "$1" "" "${COMP_WORDS[COMP_CWORD-1]}"
else    
    set -- "$1" "${COMP_WORDS[COMP_CWORD]}" "${COMP_WORDS[COMP_CWORD-1]}"
fi

I currently noticed that bash-completion package also set bash's default $3 value to empty then you have to use ${COMP_WORDS[COMP_CWORD-1]} for setting $3 variable

All these are the factors that make people stop trying to learn completion functions and only a few people make it. think about how confusing it ! if the bug I suggested is fixed, then a lot of confusion will be removed..