diff --git a/src/plugin/skills/tableau-data-app-authoring/SKILL.md b/src/plugin/skills/tableau-data-app-authoring/SKILL.md index 97429f7..70159b6 100644 --- a/src/plugin/skills/tableau-data-app-authoring/SKILL.md +++ b/src/plugin/skills/tableau-data-app-authoring/SKILL.md @@ -1,6 +1,6 @@ --- name: tableau-data-app-authoring -description: Build and publish a Tableau data app, a custom viz-extension web app that queries a published datasource live. Scaffolds the app with the scaffold-data-app MCP tool, writes its query and visualization from the user's criteria or vibe, then packages it as a .twbx and publishes it. Requires scaffold-data-app, which may be unavailable depending on the user's site role, site settings, or Tableau Server version; don't invoke this skill unless it is available. Use to create, build, vibe-code, or publish a data app. Not for native workbooks or dashboards built from marks, Show Me, or the chart catalog (tableau-workbook-authoring), or for opening existing content, including published data apps (tableau-content-viewer). +description: Build and publish a Tableau data app, a custom viz-extension web app that queries a published datasource live, or a data file embedded in the workbook. Scaffolds the app with the scaffold-data-app MCP tool, writes its query and visualization from the user's criteria or vibe, then packages it as a .twbx and publishes it. Requires scaffold-data-app, which may be unavailable depending on the user's site role, site settings, or Tableau Server version; don't invoke this skill unless it is available. Use to create, build, vibe-code, or publish a data app. Not for native workbooks or dashboards built from marks, Show Me, or the chart catalog (tableau-workbook-authoring), or for opening existing content, including published data apps (tableau-content-viewer). --- # Tableau Data App Authoring @@ -160,6 +160,8 @@ At the end of this stage you have a finalized workspace directory: content/src/app.js ← the authoring surface content/src/styles.css content/src/tableau.extensions.1.latest.min.js + Data/.gitkeep ← placeholder; embedded datasource files go here + Resources/.agents/README.md ← which agent docs travel with the workbook ``` **Sheet name:** the worksheet is named after the app's display name. It's a @@ -181,10 +183,15 @@ user only wants to publish the starter to prove packaging. Only add, remove, or swap datasources the user asked for; if you think a different or extra datasource would help, ask first. -The app queries a datasource that already exists on the server (a `sqlproxy` -connection, resolved live by VDS). Its identity can come straight from the -user, or be reused from a datasource already wired into some other existing -workbook — see Reuse an existing workbook's datasource below. +The app queries either kind of datasource, in any mix: + +- **Published:** a datasource that already exists on the server (a `sqlproxy` + connection, resolved live by VDS). Its identity can come straight from the + user, or be reused from a datasource already wired into some other existing + workbook (see Reuse an existing workbook's datasource below). +- **Embedded file:** a data file the user supplies, copied into + the workspace's `Data/` folder and packed into the `.twbx` (see Embedded + file datasource below). Do **not** hand-edit the XML — the anchors must be filled atomically with a matching join key across multiple coordinated locations, or the app silently @@ -263,6 +270,37 @@ Trust that failure over patching the XML by hand. `datatype` maps to the column `type` (`real`/`integer` → quantitative, `date`/`datetime` → ordinal, else nominal); `role: "measure"` gets a `Sum` aggregation, `dimension` a `Count`. +**Embedded file datasource:** use this when the user hands you a data file +instead of naming a published datasource. The same script wires it (and rejects +file types it can't); the descriptor just has `"type": "file"`. + +1. **Copy the file into the workspace** under `Data/`, conventionally + `Data/Datasources/`. The script fails if the file isn't there, or if + its path is outside `Data/`. +2. **Write a descriptor.** `file` is the path relative to the `.twb`. `table` + picks the table in a file that holds several (an Excel sheet); the script + says when it's needed. Field names must match the column names Tableau reads + from the file, usually its header row. No server, site, or `repositoryId`: + + ```bash + cat > "$WORK/descriptor.json" <<'DS_JSON' + { + "type": "file", + "caption": "Sample - Superstore", + "file": "Data/Datasources/Sample - Superstore.xls", + "table": "Orders", + "fields": [ + { "name": "Sales", "datatype": "real", "role": "measure" }, + { "name": "Region", "datatype": "string", "role": "dimension" } + ] + } + DS_JSON + ``` + + To mix it with published datasources, put every entry in one + `datasources` array (primary first) as above. +3. **Run the wiring script** the same way as for a published datasource. + **Reuse an existing workbook's datasource:** point the Published path above at a datasource already wired into some *other* already-published workbook, instead of a fresh published DS. Only applies when that datasource actually @@ -274,8 +312,10 @@ have produced). if ambiguous), then call `get-workbook({ workbookId })` and read `upstreamDatasources[]` — ask which entry if it lists more than one. 2. Check the matched entry's `datasourceType`: - - **`"embedded"`** — it can't be reused via this path; ask the user for a - published datasource instead. + - **`"embedded"`**: its data lives inside that workbook, so it can't be + reused by reference. Ask the user for either a published datasource or + the source data file, and wire the file as an Embedded file + datasource above. - **`"published"`** — still run the rest of Published datasource step 1 (`list-datasources` filtered by this LUID, for `contentUrl`/site/server — `get-workbook` doesn't carry those — plus @@ -328,6 +368,10 @@ hand: python3 "$SKILL_DIR/scripts/package_twbx.py" "" # the finalized workspace dir ``` +It also packs `Data/` and `Resources/.agents/` at the archive root. Under +`Data/`, only the files the `.twb` references are packed. Hidden files are +never packed. + It writes `.twbx` next to the workspace dir (pass a second arg to override), overwriting any existing file, and prints the path for Publish. @@ -378,7 +422,8 @@ Uses the MCP publish tools (not available to Slack clients). source (the normal case after `wire_datasource.py`, or when reusing one), with its published name from `list-datasources` as `` (not the descriptor's caption, which can differ). Repeat it once per wired data - source. Omit it when no data source was wired. Do no extra permission lookups. Relay its stdout to the user + source. Omit it for embedded file datasources and when no data source was + wired. Do no extra permission lookups. Relay its stdout to the user verbatim. If `publish-workbook` returns `status: "invalid"` (or an error), surface the @@ -394,7 +439,8 @@ Uses the MCP publish tools (not available to Slack clients). ## Non-negotiable limits - Don't zip the `.twbx` freehand. Use `scripts/package_twbx.py` — it keeps - `.twb` + `Packages/` at the archive root, never nested under `/`. + `.twb`, `Packages/`, and any `Data/`/`Resources/.agents/` at the archive root, never + nested under `/`. - Don't apply a postUnzip plan freehand. Use `scripts/apply_plan.py` — edits before renames, renames deepest-first, verified; see its header comment for the full contract. diff --git a/src/plugin/skills/tableau-data-app-authoring/references/build-data-app.md b/src/plugin/skills/tableau-data-app-authoring/references/build-data-app.md index 595c6a3..764ad33 100644 --- a/src/plugin/skills/tableau-data-app-authoring/references/build-data-app.md +++ b/src/plugin/skills/tableau-data-app-authoring/references/build-data-app.md @@ -8,9 +8,10 @@ ## Live-query model (read this first) -**The data app queries its datasource live via the Tableau Extensions API — there is NO local data -snapshot.** The shipped app calls `readMetadataAsync()` / `queryAsync()` at view time against the -published datasource, so it always reflects current data. Two consequences: +**The data app queries its datasource live via the Tableau Extensions API: no data is baked into +the app code.** The shipped app calls `readMetadataAsync()` / `queryAsync()` at view time against the +wired datasource. A published datasource always reflects current data; an embedded file is +packed into the `.twbx` and is current only as of packaging. Two consequences: 1. **The app reaches datasources through the workbook, and queries them directly.** It is a viz extension hosted on a worksheet, so it uses @@ -40,7 +41,9 @@ published datasource, so it always reflects current data. Two consequences: 2. **Identify the datasource(s).** Find the target published datasource and its LUID with `list-datasources` (ask the user which one if ambiguous). You can wire any number if the app genuinely needs them — all in one 'Wire a datasource in' run, primary first. Each LUID drives - both the `.twb` wiring and your introspection queries. In `app.js`, find each datasource by + both the `.twb` wiring and your introspection queries. If the user supplies a data file + instead, copy it into `Data/` and wire it as an embedded file; the SKILL's + 'Embedded file datasource' covers it. In `app.js`, find each datasource by `name` (its caption) in `getAllDataSourcesAsync()`, not by position. Query each separately and combine results in `app.js` — Tableau data blending doesn't apply, so don't add blend XML. 3. **Introspect the datasource before writing `app.js`.** diff --git a/src/plugin/skills/tableau-data-app-authoring/scripts/file_formats.py b/src/plugin/skills/tableau-data-app-authoring/scripts/file_formats.py new file mode 100644 index 0000000..1ef9ab0 --- /dev/null +++ b/src/plugin/skills/tableau-data-app-authoring/scripts/file_formats.py @@ -0,0 +1,45 @@ +""" +File formats `wire_datasource.py` can embed. To add one, write a class like +these and list it in `FILE_FORMATS`. +""" + + +def bracket(name): + return '[' + name.replace(']', ']]') + ']' + + +class ExcelFile: + extensions = ('.xls', '.xlsx', '.xlsm') + connection_class = 'excel-direct' + + def __init__(self, path, table): + if not isinstance(table, str) or not table: + raise ValueError('Excel datasources need "table": the name of the sheet to query.') + self.path = path + self.relation_name = table + self.relation_table = bracket(f'{table}$') + + def connection_attrs(self): + return {'cleaning': 'no', 'compat': 'no', 'dataRefreshTime': '', 'filename': self.path, + 'interpretationMode': '0', 'password': '', 'server': '', 'validate': 'no'} + + +class TextFile: + extensions = ('.csv',) + connection_class = 'textscan' + + def __init__(self, path, table): + if table is not None: + raise ValueError('"table" applies only to Excel files; a CSV file is its own table.') + self.directory, self.filename = path.rsplit('/', 1) + stem, extension = self.filename.rsplit('.', 1) + self.relation_name = self.filename + self.relation_table = bracket(f'{stem}#{extension}') + + def connection_attrs(self): + return {'directory': self.directory, 'filename': self.filename, 'password': '', 'server': ''} + + +FILE_FORMATS = (ExcelFile, TextFile) + +FORMAT_BY_EXTENSION = {extension: file_format for file_format in FILE_FORMATS for extension in file_format.extensions} diff --git a/src/plugin/skills/tableau-data-app-authoring/scripts/package_twbx.py b/src/plugin/skills/tableau-data-app-authoring/scripts/package_twbx.py index 6b850d7..6aa35e0 100644 --- a/src/plugin/skills/tableau-data-app-authoring/scripts/package_twbx.py +++ b/src/plugin/skills/tableau-data-app-authoring/scripts/package_twbx.py @@ -4,14 +4,17 @@ / .twb Packages//extensions/*.trex, content/... + Data/... (optional: files the .twb references) + Resources/.agents/... (optional: agent docs) -The archive holds the `.twb` (written first) and `Packages/` at its ROOT, never -nested under `/` — nesting fails to open on publish with -`PackageValidationException: Package directory contains no extension .trex -files under extensions/`. OS cruft (`.DS_Store`, `__MACOSX`) is skipped. +The archive holds the `.twb` (written first), `Packages/`, `Data/`, and +`Resources/.agents/` at its ROOT, never nested under `/`: nesting fails +to open on publish with `PackageValidationException: Package directory contains +no extension .trex files under extensions/`. Hard-fails unless the workspace root has exactly one `.twb`, a `Packages/` dir, -and at least one `.trex` under `Packages/*/extensions/`. +at least one `.trex` under `Packages/*/extensions/`, and every `Data/` file the +`.twb` references. Output defaults to `.twbx` next to the workspace dir; an existing file is overwritten. Prints the output path on stdout. @@ -22,16 +25,68 @@ import glob import os import sys +import xml.etree.ElementTree as ET import zipfile +PACKED_DIRS = ('Packages', 'Data', 'Resources/.agents') + +FILE_PATH_ATTRS = ('filename', 'dbname') + def die(message): print(f'✗ {message}', file=sys.stderr) sys.exit(1) -def is_cruft(rel_path): - return any(part.endswith('.DS_Store') or part.startswith('__MACOSX') for part in rel_path.split('/')) +def collect_entries(workspace, top, allowed=None): + entries = [] # (path, archive name, is_dir) + for root, dirs, files in os.walk(os.path.join(workspace, top)): + dirs[:] = sorted(name for name in dirs if not name.startswith(('.', '__MACOSX'))) + rel_root = os.path.relpath(root, workspace).replace(os.sep, '/') + entries.append((root, rel_root, True)) + for name in sorted(files): + rel = f'{rel_root}/{name}' + if name.startswith('.') or (allowed is not None and rel not in allowed): + continue + entries.append((os.path.join(root, name), rel, False)) + + ancestor_dirs = {rel.rsplit('/', i)[0] for _, rel, is_dir in entries if not is_dir + for i in range(1, rel.count('/') + 1)} + return [(path, rel) for path, rel, is_dir in entries if not is_dir or rel in ancestor_dirs] + + +def referenced_data_files(workspace, twb): + try: + root = ET.parse(os.path.join(workspace, twb)).getroot() + except ET.ParseError as error: + die(f'Could not parse {twb}: {error}') + referenced = set() + for element in root.iter('connection'): + directory = element.get('directory', '').replace('\\', '/').rstrip('/') + for attr in FILE_PATH_ATTRS: + path = element.get(attr, '').replace('\\', '/') + if attr == 'filename' and directory and path: + path = f'{directory}/{path}' + if path.startswith('Data/'): + referenced.add(path) + return referenced + + +def collect_packed_dirs(workspace, twb): + referenced = referenced_data_files(workspace, twb) + entries_by_top = {} + for top in PACKED_DIRS: + if os.path.isdir(os.path.join(workspace, top)): + entries = collect_entries(workspace, top, referenced if top == 'Data' else None) + if entries: + entries_by_top[top] = entries + + packed = {rel for entries in entries_by_top.values() for _, rel in entries} + missing = sorted(referenced - packed) + if missing: + die(f'{twb} references files missing from the workspace (no data at runtime): ' + + ', '.join(missing)) + return entries_by_top def main(): @@ -60,22 +115,16 @@ def main(): output = os.path.abspath(argv[2]) if len(argv) == 3 else os.path.join( os.path.dirname(workspace), os.path.basename(workspace) + '.twbx') + entries_by_top = collect_packed_dirs(workspace, twb) + # Paths are workspace-relative so nothing lands under an / prefix. with zipfile.ZipFile(output, 'w', zipfile.ZIP_DEFLATED) as archive: archive.write(os.path.join(workspace, twb), twb) - for root, dirs, files in os.walk(packages): - dirs.sort() - rel_root = os.path.relpath(root, workspace).replace(os.sep, '/') - if is_cruft(rel_root): - dirs[:] = [] - continue - archive.write(root, rel_root) - for name in sorted(files): - rel = f'{rel_root}/{name}' - if not is_cruft(rel): - archive.write(os.path.join(root, name), rel) + for entries in entries_by_top.values(): + for path, rel in entries: + archive.write(path, rel) - print(f'✓ Packaged {twb} + Packages/ into {output}', file=sys.stderr) + print(f"✓ Packaged {twb} + {', '.join(top + '/' for top in entries_by_top)} into {output}", file=sys.stderr) print(output) diff --git a/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_package_twbx.py b/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_package_twbx.py index 5ab9e6e..bec08be 100644 --- a/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_package_twbx.py +++ b/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_package_twbx.py @@ -10,6 +10,15 @@ SCRIPT = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'package_twbx.py') PACKAGE = 'Packages/com.tableau.mcp.sales-demo' +TWB_WITH_XLS = ("" + "" + "") + + +def twb_referencing(*paths): + connections = ''.join(f"" for p in paths) + return f'{connections}' def run_package(*args): @@ -30,6 +39,13 @@ def make_workspace(tmp_path, with_trex=True): return workspace +def add_template_dirs(workspace): + (workspace / 'Data').mkdir() + (workspace / 'Data' / '.gitkeep').write_text('') + (workspace / 'Resources' / '.agents').mkdir(parents=True) + (workspace / 'Resources' / '.agents' / 'README.md').write_text('# Resources/.agents/') + + def names(twbx_path): with zipfile.ZipFile(twbx_path) as archive: return archive.namelist() @@ -69,6 +85,150 @@ def test_excludes_ds_store_and_macosx(tmp_path): assert f'{PACKAGE}/content/src/app.js' in entries +def test_packs_data_at_root(tmp_path): + workspace = make_workspace(tmp_path) + datasources = workspace / 'Data' / 'Datasources' + datasources.mkdir(parents=True) + (datasources / 'Sample.xls').write_bytes(b'\xd0\xcf\x11\xe0') + (datasources / '.DS_Store').write_text('x') + (workspace / 'Sales Demo.twb').write_text(TWB_WITH_XLS) + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert entries[0] == 'Sales Demo.twb' + assert 'Data/Datasources/Sample.xls' in entries + assert f'{PACKAGE}/extensions/data-app.trex' in entries + assert not any(entry.startswith('Sales Demo/') for entry in entries) + assert not any('.DS_Store' in entry for entry in entries) + with zipfile.ZipFile(result.stdout.strip()) as archive: + assert archive.read('Data/Datasources/Sample.xls') == b'\xd0\xcf\x11\xe0' + assert '✓ Packaged Sales Demo.twb + Packages/, Data/ into' in result.stderr + + +def test_template_data_dir_with_only_placeholder_is_not_packed(tmp_path): + workspace = make_workspace(tmp_path) + add_template_dirs(workspace) + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert not any(entry.startswith('Data') for entry in entries) + assert 'Resources/.agents/README.md' in entries + assert '✓ Packaged Sales Demo.twb + Packages/, Resources/.agents/ into' in result.stderr + + +def test_fails_when_twb_references_missing_data_file(tmp_path): + workspace = make_workspace(tmp_path) + (workspace / 'Sales Demo.twb').write_text(TWB_WITH_XLS) + + result = run_package(str(workspace)) + + assert result.returncode == 1 + assert '✗ Sales Demo.twb references files missing from the workspace' in result.stderr + assert 'Data/Datasources/Sample.xls' in result.stderr + assert not (tmp_path / 'Sales Demo.twbx').exists() + + +def test_packs_only_data_files_the_twb_references(tmp_path): + workspace = make_workspace(tmp_path) + datasources = workspace / 'Data' / 'Datasources' + datasources.mkdir(parents=True) + (datasources / 'Sample.xls').write_bytes(b'\xd0\xcf\x11\xe0') + (datasources / 'credentials.json').write_text('{"token": "x"}') + (workspace / 'Data' / 'Scratch').mkdir() + (workspace / 'Data' / 'Scratch' / 'notes.txt').write_text('x') + (workspace / 'Sales Demo.twb').write_text(TWB_WITH_XLS) + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert 'Data/Datasources/Sample.xls' in entries + assert not any(e.startswith('Data/Scratch') or 'credentials' in e for e in entries) + + +def test_checks_directory_plus_filename_connections(tmp_path): + workspace = make_workspace(tmp_path) + (workspace / 'Sales Demo.twb').write_text( + "" + "" + "") + + result = run_package(str(workspace)) + + assert result.returncode == 1 + assert 'Data/orders.csv' in result.stderr + + (workspace / 'Data').mkdir() + (workspace / 'Data' / 'orders.csv').write_text('a,b\n1,2\n') + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + assert 'Data/orders.csv' in names(result.stdout.strip()) + + +def test_packs_only_agents_under_resources(tmp_path): + workspace = make_workspace(tmp_path) + agents = workspace / 'Resources' / '.agents' + (agents / 'history').mkdir(parents=True) + (agents / 'README.md').write_text('# agents') + (agents / 'history' / 'spec.md').write_text('# old spec') + (workspace / 'Resources' / 'scratch.md').write_text('scratch') + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert 'Resources/.agents/README.md' in entries + assert 'Resources/.agents/history/spec.md' in entries + assert 'Resources/scratch.md' not in entries + assert '✓ Packaged Sales Demo.twb + Packages/, Resources/.agents/ into' in result.stderr + + +def test_skips_hidden_entries_below_packed_dirs(tmp_path): + workspace = make_workspace(tmp_path) + content = workspace / PACKAGE / 'content' + (content / '.git').mkdir() + (content / '.git' / 'config').write_text('x') + (content / '.env').write_text('SECRET=x') + (content / '.DS_Store').write_text('x') + (content / '.config' / 'gh').mkdir(parents=True) + (content / '.config' / 'gh' / 'hosts.yml').write_text('token: x') + data = workspace / 'Data' + data.mkdir() + (data / '.env').write_text('SECRET=x') + agents = workspace / 'Resources' / '.agents' + agents.mkdir(parents=True) + (agents / 'README.md').write_text('# agents') + (agents / '.env').write_text('SECRET=x') + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert 'Resources/.agents/README.md' in entries + assert not any(part.startswith('.') for entry in entries + for part in entry.removeprefix('Resources/.agents/').split('/')) + assert not any(entry.startswith('Data') for entry in entries) + assert '✓ Packaged Sales Demo.twb + Packages/, Resources/.agents/ into' in result.stderr + + +def test_skips_other_workspace_dirs(tmp_path): + workspace = make_workspace(tmp_path) + (workspace / 'node_modules' / 'pkg').mkdir(parents=True) + (workspace / 'node_modules' / 'pkg' / 'index.js').write_text('//') + + result = run_package(str(workspace)) + + assert result.returncode == 0, result.stderr + entries = names(result.stdout.strip()) + assert not any(entry.startswith('node_modules') for entry in entries) + assert not any(entry.startswith(('Data', 'Resources')) for entry in entries) + + def test_explicit_output_overwrites_existing_file(tmp_path): workspace = make_workspace(tmp_path) output = tmp_path / 'out' / 'custom.twbx' diff --git a/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_wire_datasource.py b/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_wire_datasource.py index eedab16..d0c3ba5 100644 --- a/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_wire_datasource.py +++ b/src/plugin/skills/tableau-data-app-authoring/scripts/tests/test_wire_datasource.py @@ -382,3 +382,198 @@ def test_default_site_omits_site_path_and_attribute(tmp_path): wired = open(twb_path).read() assert "" in wired assert 'site=' not in wired + + +XLS_PATH = 'Data/Datasources/Sample - Superstore.xls' + + +def add_data_file(tmp_path, rel=XLS_PATH, content=b'\xd0\xcf\x11\xe0'): + path = tmp_path / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + + +def excel_descriptor(**overrides): + ds = { + "type": "file", + "caption": "Sample - Superstore", + "file": XLS_PATH, + "table": "Orders", + "fields": [ + {"name": "Sales", "datatype": "real", "role": "measure"}, + {"name": "Region", "datatype": "string", "role": "dimension"}, + {"name": "Order Date", "datatype": "date", "role": "dimension"}, + ], + } + ds.update(overrides) + return ds + + +def csv_descriptor(**overrides): + ds = { + "type": "file", + "caption": "Orders CSV", + "file": "Data/orders.csv", + "fields": [ + {"name": "Profit", "datatype": "real", "role": "measure"}, + {"name": "Ship Date", "datatype": "date", "role": "dimension"}, + ], + } + ds.update(overrides) + return ds + + +def test_excel_file_wires_federated_datasource_with_matching_join_keys(tmp_path): + add_data_file(tmp_path) + twb_path = write_twb(tmp_path) + result = run_wire(twb_path, write_descriptor(tmp_path, excel_descriptor())) + + assert result.returncode == 0, result.stderr + assert result.stdout.strip() == twb_path + wired = open(twb_path).read() + root, view = view_of(wired) + root_ds = root.find('./datasources/datasource') + name = root_ds.get('name') + assert re.fullmatch(r'federated\.[a-z0-9]{22}', name) + assert f"✓ Wired datasource 'Sample - Superstore' ({name}) with 3 field(s)" in result.stderr + key = 'excel-direct.' + name.split('.', 1)[1] + assert root_ds.get('caption') == 'Sample - Superstore' + assert root_ds.find('./connection').get('class') == 'federated' + named = root_ds.find('./connection/named-connections/named-connection') + assert named.get('name') == key + file_connection = named.find('./connection') + assert file_connection.get('class') == 'excel-direct' + assert file_connection.get('filename') == XLS_PATH + relation = root_ds.find('./connection/relation') + assert relation.attrib == {'connection': key, 'name': 'Orders', 'table': '[Orders$]', 'type': 'table'} + records = {r.findtext('remote-name'): r for r in root_ds.iter('metadata-record')} + assert records['Sales'].findtext('remote-type') == '5' + assert records['Region'].findtext('remote-type') == '129' + assert records['Order Date'].findtext('remote-type') == '129' + assert {r.findtext('parent-name') for r in records.values()} == {'[Orders]'} + assert 'sqlproxy' not in wired + + assert [d.get('name') for d in view.findall('./datasources/datasource')] == [name] + assert [d.get('datasource') for d in view.findall('./datasource-dependencies')] == [name] + assert "" in wired + assert "" in wired + + +def test_csv_file_splits_directory_and_filename(tmp_path): + add_data_file(tmp_path, 'Data/orders.csv', b'Profit,Ship Date\n1.5,2026-01-31\n') + twb_path = write_twb(tmp_path) + result = run_wire(twb_path, write_descriptor(tmp_path, csv_descriptor())) + + assert result.returncode == 0, result.stderr + root, _ = view_of(open(twb_path).read()) + root_ds = root.find('./datasources/datasource') + named = root_ds.find('./connection/named-connections/named-connection') + assert named.get('name') == 'textscan.' + root_ds.get('name').split('.', 1)[1] + assert named.find('./connection').attrib == { + 'class': 'textscan', 'directory': 'Data', 'filename': 'orders.csv', 'password': '', 'server': ''} + relation = root_ds.find('./connection/relation') + assert relation.get('name') == 'orders.csv' + assert relation.get('table') == '[orders#csv]' + records = {r.findtext('remote-name'): r for r in root_ds.iter('metadata-record')} + assert records['Profit'].findtext('remote-type') == '5' + assert records['Ship Date'].findtext('remote-type') == '129' + assert {r.findtext('parent-name') for r in records.values()} == {'[orders.csv]'} + + +def test_published_and_file_datasources_wire_in_one_run(tmp_path): + add_data_file(tmp_path) + twb_path = write_twb(tmp_path) + descriptor = {"datasources": [descriptor_happy(), excel_descriptor()]} + result = run_wire(twb_path, write_descriptor(tmp_path, descriptor)) + + assert result.returncode == 0, result.stderr + wired = open(twb_path).read() + root, view = view_of(wired) + names = [d.get('name') for d in root.findall('./datasources/datasource')] + assert names[0] == 'sqlproxy.abc123' and names[1].startswith('federated.') + assert [d.get('name') for d in view.findall('./datasources/datasource')] == names + assert [d.get('datasource') for d in view.findall('./datasource-dependencies')] == ['sqlproxy.abc123'] + + +def test_file_missing_from_the_workspace_is_rejected(tmp_path): + twb_path = write_twb(tmp_path) + result = run_wire(twb_path, write_descriptor(tmp_path, excel_descriptor())) + + assert result.returncode == 1 + assert result.stderr.strip() == ( + f"✗ Datasource 'Sample - Superstore' file \"{XLS_PATH}\" is not in the workspace ({tmp_path}). " + 'Copy it there first, at that path relative to the .twb.' + ) + assert open(twb_path).read() == TWB_TEMPLATE + + +def test_file_outside_data_folder_is_rejected(tmp_path): + twb_path = write_twb(tmp_path) + for bad in ('orders.csv', 'Data/../orders.csv', '/abs/Data/orders.csv', 'Resources/orders.csv'): + result = run_wire(twb_path, write_descriptor(tmp_path, csv_descriptor(file=bad))) + assert result.returncode == 1 + assert '"file" must be a path inside the workspace Data/ folder' in result.stderr + assert open(twb_path).read() == TWB_TEMPLATE + + +def test_unsupported_file_type_is_rejected(tmp_path): + twb_path = write_twb(tmp_path) + result = run_wire(twb_path, write_descriptor(tmp_path, csv_descriptor(file='Data/orders.json'))) + + assert result.returncode == 1 + assert result.stderr.strip() == '✗ "file" must end in .xls, .xlsx, .xlsm, .csv (got "Data/orders.json").' + + +def test_excel_requires_a_table_and_csv_rejects_one(tmp_path): + twb_path = write_twb(tmp_path) + descriptor = excel_descriptor() + del descriptor['table'] + result = run_wire(twb_path, write_descriptor(tmp_path, descriptor)) + assert result.returncode == 1 + assert result.stderr.strip() == '✗ Excel datasources need "table": the name of the sheet to query.' + + result = run_wire(twb_path, write_descriptor(tmp_path, csv_descriptor(table='orders'))) + assert result.returncode == 1 + assert result.stderr.strip() == '✗ "table" applies only to Excel files; a CSV file is its own table.' + + +def test_unknown_type_is_rejected(tmp_path): + twb_path = write_twb(tmp_path) + result = run_wire(twb_path, write_descriptor(tmp_path, excel_descriptor(type='extract'))) + + assert result.returncode == 1 + assert result.stderr.strip() == '✗ Descriptor "type" must be "published" or "file" (got "extract").' + + +def test_rewiring_a_file_wired_twb_is_rejected(tmp_path): + add_data_file(tmp_path) + twb_path = write_twb(tmp_path) + desc_path = write_descriptor(tmp_path, excel_descriptor()) + assert run_wire(twb_path, desc_path).returncode == 0 + + result = run_wire(twb_path, desc_path) + + assert result.returncode == 1 + assert '✗ This .twb is already wired.' in result.stderr + + +def test_wired_file_twb_packages_with_its_data(tmp_path): + import zipfile + workspace = tmp_path / 'Sales Demo' + ext_dir = workspace / 'Packages' / 'com.tableau.mcp.sales-demo' / 'extensions' + ext_dir.mkdir(parents=True) + (ext_dir / 'data-app.trex').write_text('') + add_data_file(workspace) + add_data_file(workspace, 'Data/orders.csv', b'Profit,Ship Date\n') + twb_path = write_twb(workspace, filename='Sales Demo.twb') + descriptor = {"datasources": [excel_descriptor(), csv_descriptor()]} + assert run_wire(twb_path, write_descriptor(tmp_path, descriptor)).returncode == 0 + + package = os.path.join(os.path.dirname(SCRIPT), 'package_twbx.py') + result = subprocess.run([sys.executable, package, str(workspace)], capture_output=True, text=True) + + assert result.returncode == 0, result.stderr + with zipfile.ZipFile(result.stdout.strip()) as archive: + entries = archive.namelist() + assert XLS_PATH in entries + assert 'Data/orders.csv' in entries diff --git a/src/plugin/skills/tableau-data-app-authoring/scripts/wire_datasource.py b/src/plugin/skills/tableau-data-app-authoring/scripts/wire_datasource.py index 8ff49bb..86e83ce 100644 --- a/src/plugin/skills/tableau-data-app-authoring/scripts/wire_datasource.py +++ b/src/plugin/skills/tableau-data-app-authoring/scripts/wire_datasource.py @@ -1,12 +1,13 @@ #!/usr/bin/env python3 """ -Wire one or more published datasources into a scaffolded data-app `.twb`. +Wire one or more datasources (published, or files in the workspace's `Data/`) +into a scaffolded data-app `.twb`. `scaffold-data-app` emits a workbook with TWO empty `` anchors (workbook root + worksheet ``). Until both are filled the extension's `getAllDataSourcesAsync()` finds nothing and renders "no data source found in -the workbook." This fills both with N published `sqlproxy` (Data Server) -datasources, keeping each `sqlproxy.` join key byte-identical everywhere. +the workbook." This fills both with N datasources, keeping each +join key (`sqlproxy.` or `federated.`) byte-identical everywhere. A script, not freehand XML, because the wiring spans coordinated locations that must agree exactly (root datasource `name`, root `relation connection`, view @@ -36,7 +37,15 @@ { "name": "Region", "datatype": "string", "role": "dimension" } ] } -or any number of them, primary first, wired in a single run: +or an embedded file (the file must already be in the workspace): + { + "type": "file", + "caption": "Sample - Superstore", + "file": "Data/Datasources/Sample - Superstore.xls", // Excel or CSV + "table": "Orders", // Excel only: the sheet + "fields": [ ...as above... ] + } +or any number of either, primary first, wired in a single run: { "datasources": [ { ...as above... }, { ...as above... } ] } """ @@ -47,10 +56,11 @@ import sys import xml.etree.ElementTree as ET +from file_formats import FORMAT_BY_EXTENSION, bracket + # The exact empty anchor emitted by the scaffold template. Matched literally. EMPTY_ANCHOR = '' - def die(message): print(f'✗ {message}', file=sys.stderr) sys.exit(1) @@ -106,15 +116,27 @@ def derive_field(field, ordinal, error_prefix=''): } -def generate_connection_name(): +def generate_connection_name(prefix): token = lambda: ''.join(random.choices(string.ascii_lowercase + string.digits, k=11)) - return f'sqlproxy.{token()}{token()}' + return f'{prefix}.{token()}{token()}' + + +def parse_fields(descriptor, error_prefix): + fields_in = descriptor.get('fields') if isinstance(descriptor.get('fields'), list) else [] + if len(fields_in) == 0: + die(f'{error_prefix}Descriptor "fields" must list at least one field the app will query.') + return [derive_field(f, i, error_prefix) for i, f in enumerate(fields_in)] # Validate one datasource descriptor. `error_prefix` names the datasource in errors when wiring several. def parse_datasource(descriptor, error_prefix=''): if not isinstance(descriptor, dict): die(f'{error_prefix}Descriptor must be a JSON object.') + kind = descriptor.get('type', 'published') + if kind == 'file': + return parse_file_datasource(descriptor, error_prefix) + if kind != 'published': + die(f'{error_prefix}Descriptor "type" must be "published" or "file" (got {json.dumps(kind)}).') for key in ('caption', 'repositoryId', 'site', 'server'): value = descriptor.get(key) # site may be "" (the Default site); everything else must be non-empty. @@ -123,26 +145,55 @@ def parse_datasource(descriptor, error_prefix=''): channel = descriptor.get('channel') or 'https' port = descriptor.get('port') if descriptor.get('port') is not None else (443 if channel == 'https' else 80) - fields_in = descriptor.get('fields') if isinstance(descriptor.get('fields'), list) else [] - if len(fields_in) == 0: - die(f'{error_prefix}Descriptor "fields" must list at least one field the app will query.') + fields = parse_fields(descriptor, error_prefix) connection_name = descriptor.get('connectionName') if connection_name and not connection_name.startswith('sqlproxy.'): die(f'{error_prefix}connectionName must start with "sqlproxy." (got "{connection_name}").') return { + 'type': 'published', 'caption': descriptor['caption'], 'repositoryId': descriptor['repositoryId'], 'site': descriptor['site'], 'server': descriptor['server'], 'channel': channel, 'port': port, - 'fields': [derive_field(f, i, error_prefix) for i, f in enumerate(fields_in)], + 'fields': fields, 'connectionName': connection_name, } +def parse_file_datasource(descriptor, error_prefix): + for key in ('caption', 'file'): + value = descriptor.get(key) + if not isinstance(value, str) or not value: + die(f'{error_prefix}Descriptor is missing required string "{key}".') + file_path = descriptor['file'].replace('\\', '/') + parts = file_path.split('/') + if len(parts) < 2 or parts[0] != 'Data' or any(part in ('', '.', '..') for part in parts): + die(f'{error_prefix}"file" must be a path inside the workspace Data/ folder, ' + f'e.g. "Data/Datasources/Sales.xlsx" (got "{descriptor["file"]}").') + file_format = FORMAT_BY_EXTENSION.get(os.path.splitext(file_path)[1].lower()) + if not file_format: + die(f'{error_prefix}"file" must end in {", ".join(FORMAT_BY_EXTENSION)} (got "{file_path}").') + try: + source = file_format(file_path, descriptor.get('table')) + except ValueError as error: + die(f'{error_prefix}{error}') + + fields = parse_fields(descriptor, error_prefix) + + return { + 'type': 'file', + 'caption': descriptor['caption'], + 'file': file_path, + 'source': source, + 'fields': fields, + 'connectionName': None, # always generated + } + + def parse_descriptor(descriptor): if isinstance(descriptor, dict) and 'datasources' in descriptor: extra_keys = [key for key in descriptor if key != 'datasources'] @@ -158,6 +209,8 @@ def parse_descriptor(descriptor): seen_repository_ids = set() for i, ds in enumerate(datasources): + if ds['type'] != 'published': + continue if ds['repositoryId'] in seen_repository_ids: die(f'datasources[{i}]: repositoryId "{ds["repositoryId"]}" is listed more than once.') seen_repository_ids.add(ds['repositoryId']) @@ -177,11 +230,16 @@ def parse_descriptor(descriptor): used = set(explicit_names) for ds in datasources: if not ds['connectionName']: - name = generate_connection_name() + prefix = 'sqlproxy' if ds['type'] == 'published' else 'federated' + name = generate_connection_name(prefix) while name in used: - name = generate_connection_name() + name = generate_connection_name(prefix) ds['connectionName'] = name used.add(name) + if ds['type'] == 'published': + ds['relationConnection'] = ds['connectionName'] + else: + ds['relationConnection'] = f"{ds['source'].connection_class}.{ds['connectionName'].split('.', 1)[1]}" return datasources @@ -193,6 +251,45 @@ def repository_location_xml(repository_id, site): def root_datasource_xml(ds): + return published_datasource_xml(ds) if ds['type'] == 'published' else file_datasource_xml(ds) + + +def file_connection_xml(source): + attrs = ' '.join(f"{name}='{esc(value)}'" for name, value in source.connection_attrs().items()) + return f"" + + +def file_datasource_xml(ds): + metadata_records = '\n'.join( + f""" + {esc(f['name'])} + {5 if f['type'] == 'quantitative' else 129} + {esc(f['localName'])} + {esc(bracket(ds['source'].relation_name))} + {esc(f['name'])} + {f['ordinal']} + {esc(f['datatype'])} + {f['aggregation']} + true + """ + for f in ds['fields'] + ) + return f""" + + + + {file_connection_xml(ds['source'])} + + + + +{metadata_records} + + + """ + + +def published_datasource_xml(ds): metadata_records = '\n'.join( f""" {esc(f['name'])} @@ -254,8 +351,13 @@ def verify_wiring(wired, datasources): root_ds = root_datasources.get(name) if root_ds is None: die(f'Root is missing {name} — wiring incomplete.') - if root_ds.find(f".//relation[@connection='{name}']") is None: - die(f'Root datasource {name} has no relation connection to itself — wiring incomplete.') + relation_connection = ds['relationConnection'] + if not any(r.get('connection') == relation_connection for r in root_ds.iter('relation')): + target = 'itself' if relation_connection == name else relation_connection + die(f'Root datasource {name} has no relation connection to {target}, so wiring is incomplete.') + if ds['type'] == 'file' and not any( + c.get('name') == relation_connection for c in root_ds.iter('named-connection')): + die(f'Root datasource {name} has no named connection {relation_connection}, so wiring is incomplete.') # The server only connects datasources listed on the host sheet's view. host_view = next( @@ -290,6 +392,12 @@ def main(): datasources = parse_descriptor(descriptor) primary = datasources[0] + workspace = os.path.dirname(twb_path) + for ds in datasources: + if ds['type'] == 'file' and not os.path.isfile(os.path.join(workspace, ds['file'])): + die(f"Datasource '{ds['caption']}' file \"{ds['file']}\" is not in the workspace ({workspace}). " + 'Copy it there first, at that path relative to the .twb.') + # --- build the XML blocks ---------------------------------------------- root_datasources = '\n' + '\n'.join(root_datasource_xml(ds) for ds in datasources) + '\n ' @@ -318,7 +426,7 @@ def main(): head = content[:split_idx] tail = content[split_idx:] - if "class='sqlproxy'" in head: + if "class='sqlproxy'" in head or "class='federated'" in head: die('This .twb is already wired. To change its datasources, re-scaffold, copy content/src/app.js ' 'into the new workspace, and wire every datasource in one run.')