@W-24630680: Wire and package embedded Excel/CSV datasources for data apps

src/plugin/skills/tableau-data-app-authoring/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: tableau-data-app-authoring
-description: Build and publish a Tableau data app, a custom viz-extension web app that queries a published datasource live. Scaffolds the app with the scaffold-data-app MCP tool, writes its query and visualization from the user's criteria or vibe, then packages it as a .twbx and publishes it. Requires scaffold-data-app, which may be unavailable depending on the user's site role, site settings, or Tableau Server version; don't invoke this skill unless it is available. Use to create, build, vibe-code, or publish a data app. Not for native workbooks or dashboards built from marks, Show Me, or the chart catalog (tableau-workbook-authoring), or for opening existing content, including published data apps (tableau-content-viewer).
+description: Build and publish a Tableau data app, a custom viz-extension web app that queries a published datasource live, or a data file embedded in the workbook. Scaffolds the app with the scaffold-data-app MCP tool, writes its query and visualization from the user's criteria or vibe, then packages it as a .twbx and publishes it. Requires scaffold-data-app, which may be unavailable depending on the user's site role, site settings, or Tableau Server version; don't invoke this skill unless it is available. Use to create, build, vibe-code, or publish a data app. Not for native workbooks or dashboards built from marks, Show Me, or the chart catalog (tableau-workbook-authoring), or for opening existing content, including published data apps (tableau-content-viewer).
 ---
 
 # Tableau Data App Authoring
@@ -160,6 +160,8 @@ At the end of this stage you have a finalized workspace directory:
     content/src/app.js      ← the authoring surface
     content/src/styles.css
     content/src/tableau.extensions.1.latest.min.js
+  Data/.gitkeep             ← placeholder; embedded datasource files go here
+  Resources/.agents/README.md   ← which agent docs travel with the workbook
 ```
 
 **Sheet name:** the worksheet is named after the app's display name. It's a
@@ -181,10 +183,15 @@ user only wants to publish the starter to prove packaging. Only add, remove, or
 swap datasources the user asked for; if you think a different or extra
 datasource would help, ask first.
 
-The app queries a datasource that already exists on the server (a `sqlproxy`
-connection, resolved live by VDS). Its identity can come straight from the
-user, or be reused from a datasource already wired into some other existing
-workbook — see Reuse an existing workbook's datasource below.
+The app queries either kind of datasource, in any mix:
+
+- **Published:** a datasource that already exists on the server (a `sqlproxy`
+  connection, resolved live by VDS). Its identity can come straight from the
+  user, or be reused from a datasource already wired into some other existing
+  workbook (see Reuse an existing workbook's datasource below).
+- **Embedded file:** a data file the user supplies, copied into
+  the workspace's `Data/` folder and packed into the `.twbx` (see Embedded
+  file datasource below).
 
 Do **not** hand-edit the XML — the anchors must be filled atomically with a
 matching join key across multiple coordinated locations, or the app silently
@@ -263,6 +270,37 @@ Trust that failure over patching the XML by hand. `datatype` maps to the column
 `type` (`real`/`integer` → quantitative, `date`/`datetime` → ordinal, else
 nominal); `role: "measure"` gets a `Sum` aggregation, `dimension` a `Count`.
 
+**Embedded file datasource:** use this when the user hands you a data file
+instead of naming a published datasource. The same script wires it (and rejects
+file types it can't); the descriptor just has `"type": "file"`.
+
+1. **Copy the file into the workspace** under `Data/`, conventionally
+   `Data/Datasources/<file>`. The script fails if the file isn't there, or if
+   its path is outside `Data/`.
+2. **Write a descriptor.** `file` is the path relative to the `.twb`. `table`
+   picks the table in a file that holds several (an Excel sheet); the script
+   says when it's needed. Field names must match the column names Tableau reads
+   from the file, usually its header row. No server, site, or `repositoryId`:
+
+   ```bash
+   cat > "$WORK/descriptor.json" <<'DS_JSON'
+   {
+     "type": "file",
+     "caption": "Sample - Superstore",
+     "file": "Data/Datasources/Sample - Superstore.xls",
+     "table": "Orders",
+     "fields": [
+       { "name": "Sales",  "datatype": "real",   "role": "measure"   },
+       { "name": "Region", "datatype": "string", "role": "dimension" }
+     ]
+   }
+   DS_JSON
+   ```
+
+   To mix it with published datasources, put every entry in one
+   `datasources` array (primary first) as above.
+3. **Run the wiring script** the same way as for a published datasource.
+
 **Reuse an existing workbook's datasource:** point the Published path above
 at a datasource already wired into some *other* already-published workbook,
 instead of a fresh published DS. Only applies when that datasource actually
@@ -274,8 +312,10 @@ have produced).
    if ambiguous), then call `get-workbook({ workbookId })` and read
    `upstreamDatasources[]` — ask which entry if it lists more than one.
 2. Check the matched entry's `datasourceType`:
-   - **`"embedded"`** — it can't be reused via this path; ask the user for a
-     published datasource instead.
+   - **`"embedded"`**: its data lives inside that workbook, so it can't be
+     reused by reference. Ask the user for either a published datasource or
+     the source data file, and wire the file as an Embedded file
+     datasource above.
    - **`"published"`** — still run the rest of Published datasource step 1
      (`list-datasources` filtered by this LUID, for `contentUrl`/site/server
      — `get-workbook` doesn't carry those — plus
@@ -328,6 +368,10 @@ hand:
 python3 "$SKILL_DIR/scripts/package_twbx.py" "<App Name>"   # the finalized workspace dir
 ```
 
+It also packs `Data/` and `Resources/.agents/` at the archive root. Under
+`Data/`, only the files the `.twb` references are packed. Hidden files are
+never packed.
+
 It writes `<App Name>.twbx` next to the workspace dir (pass a second arg to
 override), overwriting any existing file, and prints the path for Publish.
 
@@ -378,7 +422,8 @@ Uses the MCP publish tools (not available to Slack clients).
    source (the normal case after `wire_datasource.py`, or when reusing one),
    with its published name from `list-datasources` as `<name>` (not the
    descriptor's caption, which can differ). Repeat it once per wired data
-   source. Omit it when no data source was wired. Do no extra permission lookups. Relay its stdout to the user
+   source. Omit it for embedded file datasources and when no data source was
+   wired. Do no extra permission lookups. Relay its stdout to the user
    verbatim.
 
    If `publish-workbook` returns `status: "invalid"` (or an error), surface the
@@ -394,7 +439,8 @@ Uses the MCP publish tools (not available to Slack clients).
 ## Non-negotiable limits
 
 - Don't zip the `.twbx` freehand. Use `scripts/package_twbx.py` — it keeps
-  `.twb` + `Packages/` at the archive root, never nested under `<App Name>/`.
+  `.twb`, `Packages/`, and any `Data/`/`Resources/.agents/` at the archive root, never
+  nested under `<App Name>/`.
 - Don't apply a postUnzip plan freehand. Use `scripts/apply_plan.py` — edits
   before renames, renames deepest-first, verified; see its header comment for
   the full contract.

src/plugin/skills/tableau-data-app-authoring/references/build-data-app.md

@@ -8,9 +8,10 @@
 
 ## Live-query model (read this first)
 
-**The data app queries its datasource live via the Tableau Extensions API — there is NO local data
-snapshot.** The shipped app calls `readMetadataAsync()` / `queryAsync()` at view time against the
-published datasource, so it always reflects current data. Two consequences:
+**The data app queries its datasource live via the Tableau Extensions API: no data is baked into
+the app code.** The shipped app calls `readMetadataAsync()` / `queryAsync()` at view time against the
+wired datasource. A published datasource always reflects current data; an embedded file is
+packed into the `.twbx` and is current only as of packaging. Two consequences:
 
 1. **The app reaches datasources through the workbook, and queries them directly.** It is a viz
    extension hosted on a worksheet, so it uses
@@ -40,7 +41,9 @@ published datasource, so it always reflects current data. Two consequences:
 2. **Identify the datasource(s).** Find the target published datasource and its LUID with
    `list-datasources` (ask the user which one if ambiguous). You can wire any number if the app
    genuinely needs them — all in one 'Wire a datasource in' run, primary first. Each LUID drives
-   both the `.twb` wiring and your introspection queries. In `app.js`, find each datasource by
+   both the `.twb` wiring and your introspection queries. If the user supplies a data file
+   instead, copy it into `Data/` and wire it as an embedded file; the SKILL's
+   'Embedded file datasource' covers it. In `app.js`, find each datasource by
    `name` (its caption) in `getAllDataSourcesAsync()`, not by position. Query each separately and
    combine results in `app.js` — Tableau data blending doesn't apply, so don't add blend XML.
 3. **Introspect the datasource before writing `app.js`.**

src/plugin/skills/tableau-data-app-authoring/scripts/file_formats.py

@@ -0,0 +1,45 @@
+"""
+File formats `wire_datasource.py` can embed. To add one, write a class like
+these and list it in `FILE_FORMATS`.
+"""
+
+
+def bracket(name):
+    return '[' + name.replace(']', ']]') + ']'
+
+
+class ExcelFile:
+    extensions = ('.xls', '.xlsx', '.xlsm')
+    connection_class = 'excel-direct'
+
+    def __init__(self, path, table):
+        if not isinstance(table, str) or not table:
+            raise ValueError('Excel datasources need "table": the name of the sheet to query.')
+        self.path = path
+        self.relation_name = table
+        self.relation_table = bracket(f'{table}$')
+
+    def connection_attrs(self):
+        return {'cleaning': 'no', 'compat': 'no', 'dataRefreshTime': '', 'filename': self.path,
+                'interpretationMode': '0', 'password': '', 'server': '', 'validate': 'no'}
+
+
+class TextFile:
+    extensions = ('.csv',)
+    connection_class = 'textscan'
+
+    def __init__(self, path, table):
+        if table is not None:
+            raise ValueError('"table" applies only to Excel files; a CSV file is its own table.')
+        self.directory, self.filename = path.rsplit('/', 1)
+        stem, extension = self.filename.rsplit('.', 1)
+        self.relation_name = self.filename
+        self.relation_table = bracket(f'{stem}#{extension}')
+
+    def connection_attrs(self):
+        return {'directory': self.directory, 'filename': self.filename, 'password': '', 'server': ''}
+
+
+FILE_FORMATS = (ExcelFile, TextFile)
+
+FORMAT_BY_EXTENSION = {extension: file_format for file_format in FILE_FORMATS for extension in file_format.extensions}

src/plugin/skills/tableau-data-app-authoring/scripts/package_twbx.py

@@ -4,14 +4,17 @@
   <App Name>/
     <App Name>.twb
     Packages/<package id>/extensions/*.trex, content/...
+    Data/...                (optional: files the .twb references)
+    Resources/.agents/...   (optional: agent docs)
 
-The archive holds the `.twb` (written first) and `Packages/` at its ROOT, never
-nested under `<App Name>/` — nesting fails to open on publish with
-`PackageValidationException: Package directory contains no extension .trex
-files under extensions/`. OS cruft (`.DS_Store`, `__MACOSX`) is skipped.
+The archive holds the `.twb` (written first), `Packages/`, `Data/`, and
+`Resources/.agents/` at its ROOT, never nested under `<App Name>/`: nesting fails
+to open on publish with `PackageValidationException: Package directory contains
+no extension .trex files under extensions/`.
 
 Hard-fails unless the workspace root has exactly one `.twb`, a `Packages/` dir,
-and at least one `.trex` under `Packages/*/extensions/`.
+at least one `.trex` under `Packages/*/extensions/`, and every `Data/` file the
+`.twb` references.
 
 Output defaults to `<App Name>.twbx` next to the workspace dir; an existing
 file is overwritten. Prints the output path on stdout.
@@ -22,16 +25,68 @@
 import glob
 import os
 import sys
+import xml.etree.ElementTree as ET
 import zipfile
 
+PACKED_DIRS = ('Packages', 'Data', 'Resources/.agents')
+
+FILE_PATH_ATTRS = ('filename', 'dbname')
+
 
 def die(message):
     print(f'✗ {message}', file=sys.stderr)
     sys.exit(1)
 
 
-def is_cruft(rel_path):
-    return any(part.endswith('.DS_Store') or part.startswith('__MACOSX') for part in rel_path.split('/'))
+def collect_entries(workspace, top, allowed=None):
+    entries = []  # (path, archive name, is_dir)
+    for root, dirs, files in os.walk(os.path.join(workspace, top)):
+        dirs[:] = sorted(name for name in dirs if not name.startswith(('.', '__MACOSX')))
+        rel_root = os.path.relpath(root, workspace).replace(os.sep, '/')
+        entries.append((root, rel_root, True))
+        for name in sorted(files):
+            rel = f'{rel_root}/{name}'
+            if name.startswith('.') or (allowed is not None and rel not in allowed):
+                continue
+            entries.append((os.path.join(root, name), rel, False))
+
+    ancestor_dirs = {rel.rsplit('/', i)[0] for _, rel, is_dir in entries if not is_dir
+                     for i in range(1, rel.count('/') + 1)}
+    return [(path, rel) for path, rel, is_dir in entries if not is_dir or rel in ancestor_dirs]
+
+
+def referenced_data_files(workspace, twb):
+    try:
+        root = ET.parse(os.path.join(workspace, twb)).getroot()
+    except ET.ParseError as error:
+        die(f'Could not parse {twb}: {error}')
+    referenced = set()
+    for element in root.iter('connection'):
+        directory = element.get('directory', '').replace('\\', '/').rstrip('/')
+        for attr in FILE_PATH_ATTRS:
+            path = element.get(attr, '').replace('\\', '/')
+            if attr == 'filename' and directory and path:
+                path = f'{directory}/{path}'
+            if path.startswith('Data/'):
+                referenced.add(path)
+    return referenced
+
+
+def collect_packed_dirs(workspace, twb):
+    referenced = referenced_data_files(workspace, twb)
+    entries_by_top = {}
+    for top in PACKED_DIRS:
+        if os.path.isdir(os.path.join(workspace, top)):
+            entries = collect_entries(workspace, top, referenced if top == 'Data' else None)
+            if entries:
+                entries_by_top[top] = entries
+
+    packed = {rel for entries in entries_by_top.values() for _, rel in entries}
+    missing = sorted(referenced - packed)
+    if missing:
+        die(f'{twb} references files missing from the workspace (no data at runtime): '
+            + ', '.join(missing))
+    return entries_by_top
 
 
 def main():
@@ -60,22 +115,16 @@ def main():
     output = os.path.abspath(argv[2]) if len(argv) == 3 else os.path.join(
         os.path.dirname(workspace), os.path.basename(workspace) + '.twbx')
 
+    entries_by_top = collect_packed_dirs(workspace, twb)
+
     # Paths are workspace-relative so nothing lands under an <App Name>/ prefix.
     with zipfile.ZipFile(output, 'w', zipfile.ZIP_DEFLATED) as archive:
         archive.write(os.path.join(workspace, twb), twb)
-        for root, dirs, files in os.walk(packages):
-            dirs.sort()
-            rel_root = os.path.relpath(root, workspace).replace(os.sep, '/')
-            if is_cruft(rel_root):
-                dirs[:] = []
-                continue
-            archive.write(root, rel_root)
-            for name in sorted(files):
-                rel = f'{rel_root}/{name}'
-                if not is_cruft(rel):
-                    archive.write(os.path.join(root, name), rel)
+        for entries in entries_by_top.values():
+            for path, rel in entries:
+                archive.write(path, rel)
 
-    print(f'✓ Packaged {twb} + Packages/ into {output}', file=sys.stderr)
+    print(f"✓ Packaged {twb} + {', '.join(top + '/' for top in entries_by_top)} into {output}", file=sys.stderr)
     print(output)