Sources accepts variables which represent arrays.

[trulede]

If a source (or generate) glob is a templated variable, and that variable is/was an array, then add that array of globs. The following example demonstrates.

• Fixes Allow : separator to define multiple paths on a single sources entry #1018
• Fixes Dynamic variables in sources cause glob expansion to hang due to unpopulated placeholders and symlink loops #2400
• Fixes Specify task sources from dynamic list of files #2483
• Fixes Ability to dynamically define sources using sh #88
• Fixes sources to support reading an ignore, and looking at the opposite of what's ignored #709

version: '3'

method: none

vars:
  GLOB_SINGLE: 'foo.*'
  GLOB_ARRAY: ['*.txt', '*.yml'] 
  GLOB_STRING: '*.txt:*.yml' 
  GLOB_SPLIT: '{{splitList ":" .GLOB_STRING}}'

tasks:
  default:
    cmds:
      - cmd: touch foo.txt
      - cmd: touch bar.txt
      - cmd: touch foo.yml
      - cmd: touch bar.yml
      - task: direct
      - task: single
      - task: array
      - task: string
      - task: split

  direct:
    sources:
      - '*.txt'
    cmds:
      - echo "== {{.TASK}} =="
      - for: sources 
        cmd: echo {{.ITEM}}
  single:
    sources:
      - '{{.GLOB_SINGLE}}'
    cmds:
      - echo "== {{.TASK}} =="
      - for: sources 
        cmd: echo {{.ITEM}}
  array:
    sources:
      - '{{.GLOB_ARRAY}}'
      - exclude: Taskfile.yml
    cmds:
      - echo "== {{.TASK}} =="
      - for: sources 
        cmd: echo {{.ITEM}}
  string:
    sources:
      - '{{splitList ":" .GLOB_STRING}}'
      - exclude: Taskfile.yml
    cmds:
      - echo "== {{.TASK}} =="
      - for: sources 
        cmd: echo {{.ITEM}}
  split:
    sources:
      - '{{.GLOB_SPLIT}}'
      - exclude: Taskfile.yml
    cmds:
      - echo "== {{.TASK}} =="
      - for: sources 
        cmd: echo {{.ITEM}}

[timrulebosch]

It might be worthwhile combining changes from #2624 including/extending test coverage.

[guettli]

Until this PR is merged, I use a work-around which I explain here: skip this task unless files in this path changed since the last successful run (feedback is welcome).

[vmaerten]

Thanks for the PR! I have some concerns about the string-parsing approach in ReplaceGlobs.

The detection logic strings.HasPrefix(v, "[") && strings.HasSuffix(v, "]") conflicts with valid glob syntax. [a-z] is a standard glob character class, but this code would interpret it as an array:

  tasks:
    example:
      sources:
        - '[a-z]'  # valid glob matching single-char files a, b, c... z
      cmds:
        - echo {{.ITEM}}

Currently this works (in main) and matches files like a, b, c. With this PR, [a-z] would be parsed as an array containing a-z.

I know glob are not widely used and it can be edge cases.

I think relying on string representation of arrays is fragile.

IMHO, Go's string representation of slices ([a b c]) is a debug format, no escaping, no quoting, no delimiter.
strings.Fields may also break on elements containing spaces.

I think we can work on a type-aware resolution

We already have template.Resolve() which returns typed values (slices, strings, etc.) instead of stringifying them. This is the same mechanism used by ResolveRef for matrix variables.

If we try Resolve() before falling back to Replace(), we can detect actual arrays at the type level.

I can try to make some suggestion in the code if you want

[vmaerten]

I think this should be globs because if it's call for generates it'll return always sources

[trulede]

Regarding the glob array type. I think this is for the case (#88):

GLOB_DYNAMIC:
    sh: 'echo "[*.txt *.text]"'

where the glob pattern really is a string encoded array (i.e. JSON), not a Go type. By the time this code is hit the globs have already been decoded from arrays to strings, so its only a matter of decoding the string itself ... if it seems like an array.

Of course, this might be better as a real JSON string, perhaps, or dropped from the PR altogether.

[timrulebosch]

I think reading in globs from a command is interesting, but a JSON encoding might be better and also not collide with the glob expressions?

So, code like this:

func appendGlobsFromJSON(v string, g *ast.Glob, new []*ast.Glob) []*ast.Glob {
	var decoded any
	if err := json.Unmarshal([]byte(v), &decoded); err != nil {
		return new
	}

	switch x := decoded.(type) {
	case []any:
		for _, item := range x {
			if glob, ok := item.(string); ok {
				new = append(new, &ast.Glob{Glob: glob, Negate: g.Negate})
			}
		}
	case string:
		return appendGlobsFromJSON(x, g, new)
	}

	return new
}