Modernize the big-tests WDL (Ubuntu 24.04, dead links, toolchain)

test/big-tests/aws/.gitignore

@@ -0,0 +1,5 @@
+.terraform/
+*.tfstate
+*.tfstate.*
+*.tfvars
+crash.log

test/big-tests/aws/AGENTS.md

@@ -0,0 +1 @@
+README.md

test/big-tests/aws/README.md

@@ -0,0 +1,154 @@
+# Big tests on one EC2 instance
+
+After each use, update this runbook with what will make future runs smoother and
+repeatable. Keep it concise and broadly applicable; avoid unnecessary detail
+and instructions that depend on incidental choices from a particular run.
+
+This Terraform configuration runs the [big-test WDL](../genomicsqlite_big_tests.wdl)
+with miniwdl on an on-demand `c7gd.4xlarge` in the default VPC. The instance
+has 16 Arm64 vCPUs, 32 GiB RAM, a 950 GB NVMe instance store, and a separate
+50 GiB encrypted gp3 root volume. The default region is `us-west-2`.
+
+Allow several hours and at least 550 GiB of scratch space for the default
+inputs, which occupy about 33 GiB; actual scratch usage can be higher,
+especially during the SAM QNAME sort. The WDL requests 16 CPUs per task, so
+miniwdl runs one at a time, although a task may use fewer CPUs.
+
+## Deploy and connect
+
+The configuration uses Canonical's public Ubuntu 26.04 Arm64 AMI, a default
+VPC and subnet, internet access and a public IPv4 address. It creates an
+outbound-only security group and an instance role for Session Manager and
+CloudWatch metrics; it does not require SSH keys or account-specific IDs. The
+caller needs Terraform's EC2 and IAM permissions (including `iam:PassRole`),
+Session Manager permission, and the AWS CLI Session Manager plugin. Set
+`region` or `availability_zone` with Terraform variables if needed.
+
+From the repository root:
+
+```sh
+cd test/big-tests/aws
+terraform init
+terraform plan
+terraform apply
+# Run the command printed by:
+terraform output -raw session_command
+```
+
+On the instance, wait for bootstrap and enter a fresh Ubuntu login so Docker
+group membership takes effect:
+
+```sh
+cloud-init status --wait
+sudo -iu ubuntu
+findmnt /scratch
+df -h /scratch
+docker info --format '{{.DockerRootDir}}'
+grep '^root' /etc/containerd/config.toml
+miniwdl --version
+```
+
+The instance-store device should be mounted at `/scratch`, with Docker at
+`/scratch/docker` and containerd at `/scratch/containerd`. The scratch setup
+formats a blank instance store, preserves its filesystem on reboot, and refuses
+unexpected disk layouts. Docker and containerd depend on that setup. If boot
+fails, inspect `journalctl -u genomicsqlite-scratch`,
+`journalctl -u genomicsqlite-downloader`, and `/var/log/cloud-init-output.log`.
+Docker group membership grants root-equivalent access.
+
+## Monitor in CloudWatch
+
+Bootstrap installs the CloudWatch agent and publishes host-wide CPU, memory,
+`/scratch` capacity, and block-device I/O metrics every 10 seconds to the
+`GenomicSQLite/BigTests` namespace. In CloudWatch Metrics, select that
+namespace, filter by the instance ID printed by Terraform, and graph CPU
+`usage_active`/`usage_iowait`, memory `used_percent`/`available_percent`,
+`disk_used_percent`/`disk_free` for `/scratch`, and `diskio_*` for the NVMe
+instance-store device shown by `findmnt /scratch` and `lsblk`. For disk I/O
+byte and time counters, use CloudWatch's **Sum** statistic for each period;
+for CPU, memory and capacity use **Average** or **Maximum** as appropriate.
+These are host metrics rather than per-task measurements.
+
+Check `systemctl status amazon-cloudwatch-agent` and the agent log under
+`/opt/aws/amazon-cloudwatch-agent/logs/` if metrics do not appear. The
+instance role can publish only to this namespace and can describe EC2 tags;
+it cannot write CloudWatch logs. The 10-second samples are billed as custom
+metrics and may create multiple series for disk devices. CloudWatch keeps
+sub-minute data for only three hours, then rolls it up to one-minute data;
+export during the run if the full 10-second history matters. The ordinary
+`AWS/EC2` CPU metric remains available separately at five-minute resolution.
+
+For a durable local 10-second copy spanning a multi-hour run, optionally use
+`sar -A -o /scratch/miniwdl/sysstat.sa 10` in a detached tmux session (install
+`sysstat` if needed). Stop it with Ctrl-C after the run. Its raw file includes
+the hostname and must be sanitized before publication.
+
+## Run
+
+Choose the WDL revision and source revision deliberately. They are separate:
+the WDL's `git_revision` input selects the GenomicSQLite source to build and
+defaults to the moving `main` branch. The build task clones
+`https://github.com/mlin/GenomicSQLite.git`, so the chosen source commit must
+be available there. The following example resolves `main` to a commit; fetch
+the WDL revision from its source if it is not in the clone.
+
+```sh
+git clone https://github.com/mlin/GenomicSQLite.git
+cd GenomicSQLite
+# Fetch if necessary, then check out the desired WDL revision.
+git rev-parse HEAD                    # record the WDL revision
+miniwdl check test/big-tests/genomicsqlite_big_tests.wdl
+(cd /scratch/miniwdl && miniwdl run_self_test)
+```
+
+Run in tmux so an interrupted Session Manager connection does not stop the
+workflow. Within the tmux session, record the console output and exit status:
+
+```sh
+tmux new-session -s big-tests
+# Inside tmux, from the GenomicSQLite checkout:
+set -o pipefail
+source_rev=$(git ls-remote origin refs/heads/main | cut -f1)
+test -n "$source_rev" || { echo 'Could not resolve source revision' >&2; exit 1; }
+printf 'source revision: %s\n' "$source_rev"
+miniwdl run --no-color --dir /scratch/miniwdl/runs \
+  test/big-tests/genomicsqlite_big_tests.wdl "git_revision=$source_rev" \
+  2>&1 | tee /scratch/miniwdl/big-tests-console.log
+printf '%s\n' "${PIPESTATUS[0]}" > /scratch/miniwdl/big-tests.exit
+```
+
+Detach with `Ctrl-b`, then `d`; reconnect with `tmux attach -t big-tests`.
+Monitor `df -h /scratch`, the console log, and the task logs in the run
+directory. The workflow prints query results but does not assert their expected
+values, so a successful exit is a stability check, not a full correctness
+proof. The `.dbi` 404 in the no-index web test is expected fallback behavior.
+
+miniwdl 1.16.0 uses the Docker backend and the experimental `inlineDockerfile`
+extension. Bootstrap builds an Arm64 aria2/AWS downloader image because the
+default image is amd64-only; Google Storage uses the multi-architecture Google
+Cloud CLI image. Inputs and successful calls are cached under
+`/scratch/miniwdl`; call-cache entries reference earlier run directories, so
+keep those directories for retries. Cache lookup does not validate mutable
+remote URIs or image tags; use immutable revisions or disable cache for a
+fresh comparison. The public GCS input and S3 CRAM require internet access;
+the CRAM is in `us-east-1`.
+
+## Preserve results and clean up
+
+Before teardown, save the workflow and task logs, inputs/outputs, WDL and
+source revisions, timings, system configuration, exported CloudWatch metrics,
+and a run-directory size inventory. Omit the large databases and inputs if they
+are throwaway. A sanitized example is in [runs](../runs/). Check logs
+and sysstat data for account IDs, hostnames, network details, and other
+identifiers before publishing. The instance role has no S3 upload permissions;
+use a separately authorized transfer or add narrowly scoped permissions.
+
+Instance-store contents—including Docker images, caches, logs, and run
+files—are lost on stop, hibernation, termination, or host replacement. Preserve
+anything needed **before** stopping the instance. Terraform state is local and
+may contain infrastructure details. When finished, from the repository root:
+
+```sh
+cd test/big-tests/aws
+terraform destroy
+```

test/big-tests/aws/cloudwatch-agent.json

@@ -0,0 +1,28 @@
+{
+  "agent": {
+    "metrics_collection_interval": 10
+  },
+  "metrics": {
+    "namespace": "GenomicSQLite/BigTests",
+    "append_dimensions": {
+      "InstanceId": "${aws:InstanceId}"
+    },
+    "metrics_collected": {
+      "cpu": {
+        "totalcpu": true,
+        "measurement": ["usage_active", "usage_iowait"]
+      },
+      "mem": {
+        "measurement": ["used_percent", "available_percent"]
+      },
+      "disk": {
+        "resources": ["/scratch"],
+        "measurement": ["used_percent", "free"]
+      },
+      "diskio": {
+        "resources": ["*"],
+        "measurement": ["read_bytes", "write_bytes", "io_time"]
+      }
+    }
+  }
+}

test/big-tests/aws/main.tf

@@ -0,0 +1,174 @@
+terraform {
+  required_version = ">= 1.5.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.0"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.region
+}
+
+data "aws_vpc" "default" {
+  default = true
+}
+
+data "aws_subnets" "default" {
+  filter {
+    name   = "vpc-id"
+    values = [data.aws_vpc.default.id]
+  }
+  filter {
+    name   = "default-for-az"
+    values = ["true"]
+  }
+  dynamic "filter" {
+    for_each = var.availability_zone == null ? [] : [var.availability_zone]
+    content {
+      name   = "availability-zone"
+      values = [filter.value]
+    }
+  }
+}
+
+data "aws_ssm_parameter" "ubuntu_arm64" {
+  name = "/aws/service/canonical/ubuntu/server/resolute/stable/current/arm64/hvm/ebs-gp3/ami-id"
+}
+
+data "aws_iam_policy_document" "assume_ec2" {
+  statement {
+    actions = ["sts:AssumeRole"]
+    principals {
+      type        = "Service"
+      identifiers = ["ec2.amazonaws.com"]
+    }
+  }
+}
+
+resource "aws_iam_role" "instance" {
+  name_prefix        = "genomicsqlite-big-tests-"
+  assume_role_policy = data.aws_iam_policy_document.assume_ec2.json
+}
+
+# The minimum AWS-documented permissions for Session Manager. These APIs do
+# not support resource-level restrictions; no S3 or Parameter Store access is
+# granted to the instance. The person starting a session needs their own IAM
+# permission to do so.
+data "aws_iam_policy_document" "session_manager" {
+  statement {
+    actions = [
+      "ssm:UpdateInstanceInformation",
+      "ssmmessages:CreateControlChannel",
+      "ssmmessages:CreateDataChannel",
+      "ssmmessages:OpenControlChannel",
+      "ssmmessages:OpenDataChannel",
+    ]
+    resources = ["*"]
+  }
+}
+
+resource "aws_iam_role_policy" "session_manager" {
+  name   = "session-manager"
+  role   = aws_iam_role.instance.id
+  policy = data.aws_iam_policy_document.session_manager.json
+}
+
+resource "aws_iam_role_policy" "metrics" {
+  name = "publish-metrics"
+  role = aws_iam_role.instance.id
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect   = "Allow"
+        Action   = "cloudwatch:PutMetricData"
+        Resource = "*"
+        Condition = {
+          StringEquals = { "cloudwatch:namespace" = "GenomicSQLite/BigTests" }
+        }
+      },
+      {
+        Effect   = "Allow"
+        Action   = "ec2:DescribeTags"
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "aws_iam_instance_profile" "instance" {
+  name_prefix = "genomicsqlite-big-tests-"
+  role        = aws_iam_role.instance.name
+}
+
+resource "aws_security_group" "instance" {
+  name_prefix = "genomicsqlite-big-tests-"
+  description = "Outbound access for GenomicSQLite big tests; access via Session Manager"
+  vpc_id      = data.aws_vpc.default.id
+
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+}
+
+resource "aws_instance" "runner" {
+  ami                         = data.aws_ssm_parameter.ubuntu_arm64.value
+  instance_type               = "c7gd.4xlarge"
+  subnet_id                   = sort(data.aws_subnets.default.ids)[0]
+  associate_public_ip_address = true
+  vpc_security_group_ids      = [aws_security_group.instance.id]
+  iam_instance_profile        = aws_iam_instance_profile.instance.name
+
+  root_block_device {
+    volume_type           = "gp3"
+    volume_size           = 50
+    encrypted             = true
+    delete_on_termination = true
+  }
+
+  metadata_options {
+    http_endpoint = "enabled"
+    http_tokens   = "required"
+  }
+
+  user_data = templatefile("${path.module}/user-data.sh.tftpl", {
+    scratch_setup    = filebase64("${path.module}/scratch-setup.sh")
+    downloader_image = filebase64("${path.module}/miniwdl-tools.Dockerfile")
+    metrics_config   = filebase64("${path.module}/cloudwatch-agent.json")
+  })
+  user_data_replace_on_change = true
+
+  tags = {
+    Name = "genomicsqlite-big-tests"
+  }
+
+  depends_on = [aws_iam_role_policy.session_manager, aws_iam_role_policy.metrics]
+}
+
+variable "region" {
+  description = "AWS region containing the default VPC and public Ubuntu AMI parameter."
+  type        = string
+  default     = "us-west-2"
+}
+
+variable "availability_zone" {
+  description = "Optional availability zone for its default subnet; otherwise select a default subnet automatically."
+  type        = string
+  default     = null
+}
+
+output "instance_id" {
+  value = aws_instance.runner.id
+}
+
+output "session_command" {
+  description = "Connect using AWS CLI and the Session Manager plugin."
+  value       = "aws ssm start-session --region ${var.region} --target ${aws_instance.runner.id}"
+}