diff --git a/other/materials_designer/Introduction.ipynb b/other/materials_designer/Introduction.ipynb index ba6dce33a..95c6da565 100644 --- a/other/materials_designer/Introduction.ipynb +++ b/other/materials_designer/Introduction.ipynb @@ -77,6 +77,7 @@ "\n", "### 5.1. 3D\n", "#### [5.1.1. Maxwell-Boltzmann thermal disorder `X-3D-PER`](create_maxwell_disorder.ipynb)\n", + "#### [5.1.2. Create initial/final materials](create_initial_final_materials.ipynb). Transform a material into initial/intermediate/final images for a Nudged Elastic Band path.\n", "\n", "### 5.2. 2D\n", "#### [5.2.1. Perturbation using sine wave `X-2D-PER`](create_perturbation.ipynb)\n", @@ -97,6 +98,10 @@ "\n", "This notebook demonstrates a workflow for converting materials data from the [JARVIS](https://jarvis.nist.gov/) database into ESSE format for use with the Mat3ra.com platform.\n", "\n", + "### 6.2. Materials sets.\n", + "\n", + "#### [6.2.1. Create material set (ordered or unordered)](utils_create_material_set.ipynb). Save materials to a platform set for NEB, convex hull, EOS, etc.\n", + "\n", "## 6.3. Development.\n", "\n", "#### [6.3.1. Custom Transformation](custom_transformation.ipynb). Notebook setup for development of custom transformations on materials.\n", @@ -105,7 +110,7 @@ "\n", "### 7.1. Under the hood.\n", "\n", - "#### [7.1.1. More info about the conventions used](under_the_hood.ipynb)." + "#### [7.1.1. More info about the conventions used](under_the_hood.ipynb).\n" ] }, { diff --git a/other/materials_designer/create_initial_final_materials.ipynb b/other/materials_designer/create_initial_final_materials.ipynb new file mode 100644 index 000000000..99b799258 --- /dev/null +++ b/other/materials_designer/create_initial_final_materials.ipynb @@ -0,0 +1,273 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Create Initial/Final Materials\n", + "\n", + "Build an ordered initial → (optional intermediates) → final set of materials from one starting structure, and write them to a subfolder under `uploads/` in path order for [`utils_create_material_set.ipynb`](utils_create_material_set.ipynb).\n", + "\n", + "Any calculation that takes an ordered start/end pair can use the output; a Nudged Elastic Band path ([`neb.ipynb`](workflows/neb.ipynb)) is one consumer.\n", + "\n", + "Order is preserved by **numbering material names** (`00_...`, `01_...`, …): `utils_create_material_set.ipynb` (and `load_materials_from_folder`) sort by filename, and filenames come from material names.\n", + "\n", + "## Usage\n", + "\n", + "1. Set the material and the names in cell 1.2.\n", + "1. Run 2.1, copy the coordinates of the atom you want to move, and paste them into 2.2.\n", + "1. Run the rest to build and write the path materials.\n", + "1. Open [`utils_create_material_set.ipynb`](utils_create_material_set.ipynb), set the same `SUBFOLDER_NAME` and `IS_ORDERED = True`, and run it to save the materials and create the platform set.\n", + "1. Use the printed set name as `MATERIAL_SET` in [`neb.ipynb`](workflows/neb.ipynb).\n", + "\n", + "## Summary\n", + "\n", + "1. Install packages and set parameters.\n", + "1. Load the starting material.\n", + "1. Clone it as the initial image; move one atom to make the final image.\n", + "1. Name members in path order and write them to `uploads//`." + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## 1. Set up the environment and parameters\n", + "### 1.1. Install packages (JupyterLite)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.packages import install_packages\n", + "\n", + "await install_packages(\"made|api_examples\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "### 1.2. Set parameters\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "# Starting material (uploads folder or Standata name match)\n", + "FOLDER = \"uploads\"\n", + "MATERIAL_NAME = \"Silicon (100) surface\"\n", + "\n", + "# Short base name for the written images: 00_.json, 01_.json.\n", + "PATH_NAME = \"initial-final-materials\"\n", + "\n", + "# Subfolder under uploads/ to write path materials into — use the same value as\n", + "SUBFOLDER_NAME = \"initial_final_materials\"" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## 2. Build path materials\n", + "### 2.1. Load the starting material\n", + "\n", + "To read an atom's coordinates from the viewer below: open **Measurements** (the ruler icon), turn on\n", + "**Copy Coordinates [C]** — or press `C` — then click the atom. Its coordinates go to the clipboard,\n", + "ready to paste into 2.2." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.made.material import Material\n", + "from mat3ra.standata.materials import Materials\n", + "from mat3ra.notebooks_utils.ipython.entity.material.visualize import visualize_materials as visualize\n", + "from mat3ra.notebooks_utils.material import load_material_from_folder\n", + "\n", + "source_material = load_material_from_folder(FOLDER, MATERIAL_NAME) or Material.create(\n", + " Materials.get_by_name_first_match(MATERIAL_NAME)\n", + ")\n", + "visualize(source_material, viewer=\"wave\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "### 2.2. Choose the atom to move\n", + "\n", + "`ATOM_COORDINATE` is the atom you copied above, in crystal coordinates; the nearest atom to it is the\n", + "one that moves.\n", + "\n", + "`TRANSLATION` is vector to move it, in Ångström." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "# Coordinates of the atom to move, copied from the viewer above.\n", + "ATOM_COORDINATE = [0.0, 0.5, 0.5633]\n", + "\n", + "# Displacement in Angstrom.\n", + "TRANSLATION = [0.0, 0.0, -2.0]" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "### 2.3. Clone as the initial image\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "initial_material = source_material.clone()\n", + "visualize(initial_material, rotation=\"-90x\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "11", + "metadata": {}, + "source": [ + "### 2.4. Transform into the final image (example)\n", + "\n", + "Default: move the atom at `ATOM_COORDINATE` by `TRANSLATION`. Replace with any other transformation (defects, swaps, custom coordinates, …).\n", + "\n", + "2.5 below shows the same move done as a displacement *field* instead, which is what you want when the neighbours should relax along with the atom.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.made.tools.analyze.other import get_closest_site_id_from_coordinate\n", + "from mat3ra.made.tools.operations.core.unary import translate_atoms\n", + "\n", + "atom_id = get_closest_site_id_from_coordinate(initial_material, ATOM_COORDINATE)\n", + "final_material = translate_atoms(initial_material, [atom_id], TRANSLATION)\n", + "\n", + "visualize([initial_material, final_material], viewer=\"wave\")" + ] + }, + { + "cell_type": "markdown", + "id": "13", + "metadata": {}, + "source": [ + "### 2.5. Alternative: move the atom with a perturbation function (example)\n", + "\n", + "`translate_atoms` above applies a fixed vector. A perturbation function instead returns `∆z` for\n", + "*every* atom from `f(x, y, z)`, so a Gaussian centred on one atom keeps the displacement local to\n", + "it — and lets the path be shaped (neighbours relaxing along with it, a wave, a decaying tail)\n", + "rather than a rigid shift.\n", + "\n", + "Left commented out so \"Run All\" uses the simple translation. Uncomment it to overwrite\n", + "`final_material`, run, and compare the two structures in the viewer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14", + "metadata": {}, + "outputs": [], + "source": [ + "# import numpy as np\n", + "# import sympy as sp\n", + "# from mat3ra.made.tools.build_components.operations.core.modifications.perturb import FunctionHolder\n", + "# from mat3ra.made.tools.operations.core.unary import perturb\n", + "#\n", + "# SIGMA = 0.5 # Angstrom — how tightly the displacement is localised around the atom\n", + "#\n", + "# center_x, center_y, center_z = np.array(ATOM_COORDINATE) @ np.array(\n", + "# initial_material.lattice.vector_arrays\n", + "# )\n", + "#\n", + "# x, y, z = sp.symbols(\"x y z\")\n", + "# displacement_function = TRANSLATION[2] * sp.exp(\n", + "# -(((x - center_x) ** 2 + (y - center_y) ** 2 + (z - center_z) ** 2) / (2 * SIGMA**2))\n", + "# )\n", + "#\n", + "# final_material = perturb(\n", + "# initial_material,\n", + "# FunctionHolder(function=displacement_function),\n", + "# use_cartesian_coordinates=True,\n", + "# )\n", + "# visualize([initial_material, final_material], viewer=\"wave\")" + ] + }, + { + "cell_type": "markdown", + "id": "15", + "metadata": {}, + "source": [ + "### 2.6. Name members in path order and write to the subfolder\n", + "\n", + "Numeric prefixes control load order in `utils_create_material_set.ipynb` (filenames are sorted; filenames come from material names).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "16", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.material import set_materials\n", + "from mat3ra.notebooks_utils.settings import UPLOADS_FOLDER\n", + "\n", + "path_materials = [initial_material, final_material]\n", + "for index, material in enumerate(path_materials):\n", + " material.name = f\"{index:02d}_{PATH_NAME}\"\n", + "\n", + "subfolder_path = f\"{UPLOADS_FOLDER}/{SUBFOLDER_NAME}\"\n", + "set_materials(path_materials, folder_path=subfolder_path)\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/other/materials_designer/utils_create_material_set.ipynb b/other/materials_designer/utils_create_material_set.ipynb new file mode 100644 index 000000000..10e0c6533 --- /dev/null +++ b/other/materials_designer/utils_create_material_set.ipynb @@ -0,0 +1,248 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Create Material Set\n", + "\n", + "Save materials to a platform materials set — **ordered** (path order, e.g. NEB) or **unordered** (e.g. convex hull, EOS series).\n", + "\n", + "This notebook only packages materials — it does not build or transform them. Build materials elsewhere (Materials Designer, a `create_*` notebook, or a dedicated builder notebook such as [`create_initial_final_materials.ipynb`](create_initial_final_materials.ipynb)), then either:\n", + "\n", + "- Select them as **Input Materials** (outer runtime), or\n", + "- Write them to a subfolder under `uploads/` with `set_materials(materials, folder_path=f\"uploads/{SUBFOLDER_NAME}\")` — filename sort order becomes set order for ordered sets, so number material names (`00_...`, `01_...`, …).\n", + "\n", + "## Usage\n", + "\n", + "1. Set `MATERIAL_SET_NAME`, `IS_ORDERED`, and `SUBFOLDER_NAME` in cell 1.2.\n", + "1. Click \"Run\" > \"Run All\".\n", + "\n", + "## Summary\n", + "\n", + "1. Install packages and set parameters.\n", + "1. Authenticate and select account.\n", + "1. Load materials — from `SUBFOLDER_NAME` if set, otherwise Input Materials / the `uploads/` root.\n", + "1. Save materials on the platform and get or create the materials set (reuses `MATERIAL_SET_NAME` if it already exists, otherwise creates it).\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## 1. Set up the environment and parameters\n", + "### 1.1. Install packages (JupyterLite)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.packages import install_packages\n", + "\n", + "await install_packages(\"made|api_examples\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "### 1.2. Set parameters\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "# Auth / organization\n", + "ORGANIZATION_NAME = None\n", + "\n", + "# Materials set on the platform (use this name in workflow notebooks)\n", + "MATERIAL_SET_NAME = \"My Materials Set\"\n", + "# True = path order via inSet.index (NEB); False = bag of members (hull, EOS, …)\n", + "IS_ORDERED = True\n", + "\n", + "# Subfolder under uploads/ to load materials from (written there by a builder notebook via\n", + "# set_materials(materials, folder_path=f\"uploads/{SUBFOLDER_NAME}\")).\n", + "# None = use Input Materials (outer runtime), falling back to the uploads/ root.\n", + "SUBFOLDER_NAME = None # e.g. \"my_materials\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## 2. Authenticate and initialize API client\n", + "### 2.1. Authenticate\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.auth import authenticate\n", + "\n", + "await authenticate()\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "### 2.2. Initialize API client\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.api_client import APIClient\n", + "\n", + "client = APIClient.authenticate()\n", + "client\n" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "### 2.3. Select account\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "client.list_accounts()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11", + "metadata": {}, + "outputs": [], + "source": [ + "selected_account = client.my_account\n", + "\n", + "if ORGANIZATION_NAME:\n", + " selected_account = client.get_account(name=ORGANIZATION_NAME)\n", + "\n", + "ACCOUNT_ID = selected_account.id\n", + "print(f\"✅ Selected account ID: {ACCOUNT_ID}, name: {selected_account.name}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "12", + "metadata": {}, + "source": [ + "## 3. Load materials for the set\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.ipython.entity.material.visualize import visualize_materials as visualize\n", + "from mat3ra.notebooks_utils.material import get_materials, load_materials_from_folder\n", + "from mat3ra.notebooks_utils.settings import UPLOADS_FOLDER\n", + "\n", + "if SUBFOLDER_NAME:\n", + " materials = load_materials_from_folder(f\"{UPLOADS_FOLDER}/{SUBFOLDER_NAME}\")\n", + "else:\n", + " materials = get_materials()\n", + "\n", + "print(f\"Loaded {len(materials)} material(s): {[material.name for material in materials]}\")\n", + "visualize(materials)\n" + ] + }, + { + "cell_type": "markdown", + "id": "14", + "metadata": {}, + "source": [ + "## 4. Save to a materials set\n", + "### 4.1. Create or reuse materials on the platform\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.core.entity.material.api import get_or_create_material\n", + "\n", + "saved_materials = [get_or_create_material(client, material, ACCOUNT_ID) for material in materials]\n", + "for material, saved in zip(materials, saved_materials):\n", + " print(f\"{material.name}: {saved['_id']}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "16", + "metadata": {}, + "source": [ + "### 4.2. Get or create the materials set and move members\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.core.entity.material.api import get_or_create_materials_set\n", + "\n", + "materials_set = get_or_create_materials_set(\n", + " client,\n", + " ACCOUNT_ID,\n", + " MATERIAL_SET_NAME,\n", + " saved_materials,\n", + " is_ordered=IS_ORDERED,\n", + ")\n", + "print(f\"MATERIAL_SET = {materials_set['name']!r} (is_ordered={IS_ORDERED})\")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/other/materials_designer/workflows/Introduction.ipynb b/other/materials_designer/workflows/Introduction.ipynb index 03d695968..17d13c292 100644 --- a/other/materials_designer/workflows/Introduction.ipynb +++ b/other/materials_designer/workflows/Introduction.ipynb @@ -81,7 +81,7 @@ "## 7. Chemistry\n", "\n", "### 7.1. Nudged Elastic Band (NEB)\n", - "#### 7.1.1. NEB reaction pathway calculation. *(to be added)*\n", + "#### [7.1.1. NEB reaction pathway calculation.](neb.ipynb)\n", "\n", "### 7.2. HOMO-LUMO (NWChem)\n", "#### 7.2.1. HOMO-LUMO gap calculation. *(to be added)*\n", diff --git a/other/materials_designer/workflows/analyze_convex_hull.ipynb b/other/materials_designer/workflows/analyze_convex_hull.ipynb index bebe464e2..d50ebf299 100644 --- a/other/materials_designer/workflows/analyze_convex_hull.ipynb +++ b/other/materials_designer/workflows/analyze_convex_hull.ipynb @@ -16,7 +16,7 @@ "3. The notebook finds materials, retrieves total energies, and builds the convex hull.\n", "\n", "**Prerequisites:**\n", - "1. Save materials to a material set (for simpler search).\n", + "1. Save materials to a material set (for simpler search) — see [`utils_create_material_set.ipynb`](../utils_create_material_set.ipynb) (`IS_ORDERED = False`).\n", "2. Relax if needed and store relaxed structures there.\n", "3. Calculate total energies for the materials with the same formalism (e.g. DFT functional) and needed precision.\n", "\n", @@ -26,7 +26,7 @@ "2. **Preview** the materials (formula, structure type, space group)\n", "3. **Retrieve total energies** from completed jobs\n", "4. **Build convex hull** using pymatgen PhaseDiagram\n", - "5. **Analyze stability** — formation energies, energy above hull, decomposition products" + "5. **Analyze stability** — formation energies, energy above hull, decomposition products\n" ] }, { @@ -140,18 +140,14 @@ "metadata": {}, "outputs": [], "source": [ - "# Resolve material set (if specified)\n", - "set_id = None\n", + "from mat3ra.notebooks_utils.core.entity.material.api import list_materials_by_set\n", + "\n", + "set_materials = None\n", "if MATERIAL_SET:\n", - " set_query = {\"owner._id\": ACCOUNT_ID, \"isEntitySet\": True,\n", - " \"name\": {\"$regex\": MATERIAL_SET, \"$options\": \"i\"}}\n", - " sets = client.materials.list(set_query)\n", - " if not sets:\n", - " raise ValueError(f\"No material set matching '{MATERIAL_SET}'\")\n", - " set_id = sets[0][\"_id\"]\n", - " print(f\"✅ Using set: {sets[0]['name']} ({set_id})\")\n", + " set_materials = list_materials_by_set(client, ACCOUNT_ID, MATERIAL_SET)\n", + " print(f\"✅ Using set '{MATERIAL_SET}': {len(set_materials)} material(s)\")\n", "else:\n", - " print(\"ℹ️ No material set specified — searching all materials in account.\")" + " print(\"ℹ️ No material set specified — searching all materials in account.\")\n" ] }, { @@ -169,23 +165,22 @@ "metadata": {}, "outputs": [], "source": [ - "# Search materials by formula\n", + "# Search materials by formula (optionally restricted to MATERIAL_SET)\n", "all_materials = []\n", "\n", "for formula in FORMULAS:\n", - " query = {\"formula\": formula, \"owner._id\": ACCOUNT_ID}\n", - " if set_id:\n", - " query[\"inSet._id\"] = set_id\n", - " matches = client.materials.list(query)\n", - " # Filter out entity sets\n", - " matches = [m for m in matches if not m.get(\"isEntitySet\")]\n", + " if set_materials is not None:\n", + " matches = [material for material in set_materials if material.get(\"formula\") == formula]\n", + " else:\n", + " matches = client.materials.list({\"formula\": formula, \"owner._id\": ACCOUNT_ID})\n", + " matches = [material for material in matches if not material.get(\"isEntitySet\")]\n", " for material in matches:\n", " material[\"_search_formula\"] = formula # track which formula query found this\n", " all_materials.extend(matches)\n", "\n", " print(f\"{formula}: {len(matches)} material(s) found\")\n", "\n", - "print(f\"\\nTotal: {len(all_materials)} materials\")" + "print(f\"\\nTotal: {len(all_materials)} materials\")\n" ] }, { diff --git a/other/materials_designer/workflows/neb.ipynb b/other/materials_designer/workflows/neb.ipynb new file mode 100644 index 000000000..5917955e2 --- /dev/null +++ b/other/materials_designer/workflows/neb.ipynb @@ -0,0 +1,612 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Nudged Elastic Band (NEB)\n", + "\n", + "Calculate the reaction energy barrier and energy profile along a reaction path using Quantum ESPRESSO `neb.x` on the Mat3ra platform.\n", + "\n", + "## Prerequisites — where do the path materials come from?\n", + "\n", + "This notebook needs an **ordered** path: **first** = initial image, **last** = final image, optional **middle** intermediates. Order follows the materials-set indices (same as the job designer), not material names.\n", + "\n", + "If you do not already have those materials on the platform, prepare them first in Materials Designer:\n", + "\n", + "1. Open [`create_initial_final_materials.ipynb`](../create_initial_final_materials.ipynb) (under Materials / Perturbations). Build the initial → final (and optional intermediate) images and run it so they are written to `uploads//` with numbered names (`00_…`, `01_…`, …).\n", + "\n", + "Then choose **one** of these ways to feed this NEB notebook:\n", + "\n", + "1. **From the subfolder (this notebook packages the set):** set `SUBFOLDER_NAME` to that folder name (e.g. `\"neb_silicon\"`). This notebook loads the files, saves the materials, and gets or creates the ordered set named `MATERIAL_SET`.\n", + "1. **From a materials set you build separately:** leave `SUBFOLDER_NAME = None`, run [`utils_create_material_set.ipynb`](../utils_create_material_set.ipynb) with the same `SUBFOLDER_NAME` and `IS_ORDERED = True`, then set `MATERIAL_SET` here to that set’s name.\n", + "1. **From a materials set you already have:** leave `SUBFOLDER_NAME = None` and set `MATERIAL_SET` to an existing ordered set on your account that already contains the path (first → … → last).\n", + "\n", + "If the set has only first+last (no middle images), set `N_IMAGES` (e.g. `20`) so Quantum ESPRESSO interpolates intermediates.\n", + "\n", + "

Usage

\n", + "\n", + "1. Set `MATERIAL_SET` and optional `SUBFOLDER_NAME` in cell 1.2; NEB settings (`N_IMAGES`, k-grid) in cell 1.3.\n", + "1. Click \"Run\" > \"Run All\" to run all cells.\n", + "1. Wait for the job to complete.\n", + "1. Scroll down to view the reaction energy profile.\n", + "\n", + "## Summary\n", + "\n", + "1. Set up the environment and parameters: install packages (JupyterLite only) and configure materials lookup, workflow, compute, and job.\n", + "1. Authenticate and initialize API client: authenticate via browser, initialize the client, then select account and project.\n", + "1. Resolve NEB path materials — from `SUBFOLDER_NAME` (get or create set) or from an existing ordered materials set.\n", + "1. Create workflow and set its parameters: load the NEB workflow from Standata, set k-grid / `N_IMAGES`, and save the workflow.\n", + "1. Configure compute: get list of clusters and create compute configuration.\n", + "1. Create the multi-material NEB job.\n", + "1. Submit the job and monitor the status.\n", + "1. Retrieve results: visualize the reaction energy profile along the path.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## 1. Set up the environment and parameters\n", + "### 1.1. Install packages (JupyterLite)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.packages import install_packages\n", + "\n", + "await install_packages(\"made|api_examples\")" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "### 1.2. Set parameters and configurations for the workflow and job" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "from mat3ra.ide.compute import QueueName\n", + "\n", + "# 2. Auth and organization parameters\n", + "# Set organization name to use it as the owner, otherwise your personal account is used\n", + "ORGANIZATION_NAME = None\n", + "\n", + "# 3. Material parameters\n", + "# Order in the set: first = initial, middle = intermediates (optional), last = final\n", + "MATERIAL_SET = \"NEB-ordered-material-set\"\n", + "# If set, load path materials from ../uploads/{SUBFOLDER_NAME} and get or create MATERIAL_SET.\n", + "# If None, reuse the existing ordered set named MATERIAL_SET on the platform.\n", + "SUBFOLDER_NAME = None # e.g. \"neb_silicon\"\n", + "\n", + "# 4. Workflow parameters\n", + "WORKFLOW_SEARCH_TERM = \"neb.json\"\n", + "APPLICATION_NAME = \"espresso\"\n", + "MY_WORKFLOW_NAME = \"Nudged Elastic Band (NEB)\"\n", + "\n", + "# 5. Compute parameters\n", + "CLUSTER_NAME = None # specify full or partial name i.e. \"cluster-001\" to select\n", + "QUEUE_NAME = QueueName.D\n", + "PPN = 1\n", + "\n", + "# 6. Job parameters\n", + "timestamp = datetime.now().strftime(\"%Y-%m-%d %H:%M\")\n", + "POLL_INTERVAL = 30 # seconds" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "### 1.3. Set specific NEB parameters" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "# Total image count passed to QE; applied whenever set (QE interpolates up to this count)\n", + "N_IMAGES = None # e.g. 20; defaults to 1 when the set holds exactly two materials\n", + "\n", + "# K-grid for the NEB unit\n", + "NEB_KGRID = [1, 1, 1]" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## 2. Authenticate and initialize API client\n", + "### 2.1. Authenticate\n", + "Authenticate in the browser and have credentials stored in environment variable \"OIDC_ACCESS_TOKEN\".\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.auth import authenticate\n", + "\n", + "\n", + "await authenticate()" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "### 2.2. Initialize API Client\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.api_client import APIClient\n", + "\n", + "client = APIClient.authenticate()\n", + "client" + ] + }, + { + "cell_type": "markdown", + "id": "11", + "metadata": {}, + "source": [ + "### 2.3. Select account to work under" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "client.list_accounts()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13", + "metadata": {}, + "outputs": [], + "source": [ + "selected_account = client.my_account\n", + "\n", + "if ORGANIZATION_NAME:\n", + " selected_account = client.get_account(name=ORGANIZATION_NAME)\n", + "\n", + "ACCOUNT_ID = selected_account.id\n", + "print(f\"✅ Selected account ID: {ACCOUNT_ID}, name: {selected_account.name}\")" + ] + }, + { + "cell_type": "markdown", + "id": "14", + "metadata": {}, + "source": [ + "### 2.4. Select project" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15", + "metadata": {}, + "outputs": [], + "source": [ + "projects = client.projects.list({\"isDefault\": True, \"owner._id\": ACCOUNT_ID})\n", + "project_id = projects[0][\"_id\"]\n", + "print(f\"✅ Using project: {projects[0]['name']} ({project_id})\")" + ] + }, + { + "cell_type": "markdown", + "id": "16", + "metadata": {}, + "source": [ + "## 3. Resolve NEB path materials\n", + "### 3.1. Load from subfolder or reuse an ordered set\n", + "If `SUBFOLDER_NAME` is set, load numbered files from `../uploads/{SUBFOLDER_NAME}`, save them, and get or create `MATERIAL_SET`. Otherwise load members from the existing ordered set in index order: first → intermediates (if any) → last." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.made.material import Material\n", + "from mat3ra.notebooks_utils.core.entity.material.api import (\n", + " find_material_set,\n", + " get_or_create_material,\n", + " get_or_create_materials_set,\n", + " list_materials_in_set,\n", + ")\n", + "from mat3ra.notebooks_utils.ipython.entity.material.visualize import visualize_materials as visualize\n", + "from mat3ra.notebooks_utils.material import load_materials_from_folder\n", + "from mat3ra.notebooks_utils.settings import UPLOADS_FOLDER\n", + "\n", + "if SUBFOLDER_NAME:\n", + " folder_materials = load_materials_from_folder(f\"../{UPLOADS_FOLDER}/{SUBFOLDER_NAME}\")\n", + " saved_responses = [\n", + " get_or_create_material(client, material, ACCOUNT_ID) for material in folder_materials\n", + " ]\n", + " materials_set = get_or_create_materials_set(\n", + " client,\n", + " ACCOUNT_ID,\n", + " MATERIAL_SET,\n", + " saved_responses,\n", + " is_ordered=True,\n", + " )\n", + " MATERIAL_SET = materials_set[\"name\"]\n", + "else:\n", + " materials_set = find_material_set(client, ACCOUNT_ID, MATERIAL_SET, require_ordered=True)\n", + "\n", + "material_dicts = list_materials_in_set(client, ACCOUNT_ID, materials_set)\n", + "\n", + "if len(material_dicts) < 2:\n", + " raise ValueError(\n", + " f\"Ordered set '{MATERIAL_SET}' must contain at least first and last images \"\n", + " f\"(found {len(material_dicts)}).\"\n", + " )\n", + "print(f\"✅ Loaded {len(material_dicts)} material(s) from ordered set '{MATERIAL_SET}'\")\n", + "materials = [Material.create(material_dict) for material_dict in material_dicts]\n", + "print(f\" first (initial): {materials[0].name} ({materials[0].id})\")\n", + "if len(materials) > 2:\n", + " for material in materials[1:-1]:\n", + " print(f\" intermediate: {material.name} ({material.id})\")\n", + "print(f\" last (final): {materials[-1].name} ({materials[-1].id})\")\n", + "visualize(materials)\n" + ] + }, + { + "cell_type": "markdown", + "id": "18", + "metadata": {}, + "source": [ + "### 3.2. Use resolved materials for the job\n", + "Path materials are on the platform in the ordered set (reused or just created from `SUBFOLDER_NAME`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "19", + "metadata": {}, + "outputs": [], + "source": [ + "saved_materials = materials\n", + "for saved_material in saved_materials:\n", + " print(f\"✅ Material: {saved_material.name} ({saved_material.id})\")" + ] + }, + { + "cell_type": "markdown", + "id": "20", + "metadata": {}, + "source": [ + "## 4. Create workflow and set its parameters\n", + "### 4.1. Get list of applications and select one" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.standata.applications import ApplicationStandata\n", + "from mat3ra.ade.application import Application\n", + "\n", + "app_config = ApplicationStandata.get_by_name_first_match(APPLICATION_NAME)\n", + "app = Application(**app_config)\n", + "print(f\"Using application: {app.name}\")" + ] + }, + { + "cell_type": "markdown", + "id": "22", + "metadata": {}, + "source": [ + "### 4.2. Create workflow from standard workflows and preview it" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "23", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.standata.workflows import WorkflowStandata\n", + "from mat3ra.wode.workflows import Workflow\n", + "from mat3ra.notebooks_utils.ipython.entity.workflow.visualize import visualize_workflow\n", + "\n", + "workflow_config = WorkflowStandata.filter_by_application(app.name).get_by_name_first_match(WORKFLOW_SEARCH_TERM)\n", + "workflow = Workflow.create(workflow_config)\n", + "workflow.name = MY_WORKFLOW_NAME\n", + "\n", + "visualize_workflow(workflow)" + ] + }, + { + "cell_type": "markdown", + "id": "24", + "metadata": {}, + "source": [ + "### 4.3. Modify important settings\n", + "Set k-grid for the NEB unit. When the set has only first+last, set `N_IMAGES` (e.g. 20) so QE interpolates intermediates." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.wode.context.providers import PointsGridDataProvider\n", + "\n", + "neb_subworkflow = workflow.subworkflows[0]\n", + "unit_to_modify = neb_subworkflow.get_unit_by_name(name=\"neb\")\n", + "\n", + "if NEB_KGRID is not None:\n", + " new_context_kgrid = PointsGridDataProvider(dimensions=NEB_KGRID, isEdited=True).get_context_item_data()\n", + " unit_to_modify.add_context(new_context_kgrid)\n", + "\n", + "effective_n_images = N_IMAGES\n", + "if len(saved_materials) == 2 and effective_n_images is None:\n", + " effective_n_images = 1\n", + "if effective_n_images is not None:\n", + " unit_to_modify.add_context(\n", + " {\"name\": \"neb\", \"isEdited\": True, \"data\": {\"nImages\": effective_n_images}, \"extraData\": {}}\n", + " )\n", + " print(f\"Using N_IMAGES={effective_n_images}\")\n", + "\n", + "neb_subworkflow.set_unit(unit_to_modify)\n", + "visualize_workflow(workflow)" + ] + }, + { + "cell_type": "markdown", + "id": "26", + "metadata": {}, + "source": [ + "### 4.4. Save workflow to collection" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "27", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.utils.namespace import dict_to_namespace_recursive\n", + "from mat3ra.notebooks_utils.core.entity.workflow.api import get_or_create_workflow\n", + "\n", + "saved_workflow_response = get_or_create_workflow(client, workflow, ACCOUNT_ID)\n", + "saved_workflow = Workflow.create(saved_workflow_response)\n", + "print(f\"Workflow ID: {saved_workflow.id}\")" + ] + }, + { + "cell_type": "markdown", + "id": "28", + "metadata": {}, + "source": [ + "## 5. Create the compute configuration\n", + "### 5.1. Get list of clusters" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29", + "metadata": {}, + "outputs": [], + "source": [ + "clusters = client.clusters.list()\n", + "print(f\"Available clusters: {[cluster['hostname'] for cluster in clusters]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "30", + "metadata": {}, + "source": [ + "### 5.2. Create compute configuration for the job\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "31", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.ide.compute import Compute\n", + "\n", + "if CLUSTER_NAME:\n", + " cluster = next((c for c in clusters if CLUSTER_NAME in c[\"hostname\"]), None)\n", + "else:\n", + " cluster = clusters[0]\n", + "\n", + "compute = Compute(\n", + " cluster=cluster,\n", + " queue=QUEUE_NAME,\n", + " ppn=PPN\n", + ")\n", + "print(f\"Using cluster: {compute.cluster.hostname}, queue: {QUEUE_NAME}, ppn: {PPN}\")" + ] + }, + { + "cell_type": "markdown", + "id": "32", + "metadata": {}, + "source": [ + "## 6. Create the job with materials and workflow configuration\n", + "### 6.1. Create job" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "33", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.job import create_job\n", + "from mat3ra.notebooks_utils.ui import display_JSON\n", + "\n", + "print(f\"Materials: {[material.id for material in saved_materials]}\")\n", + "print(f\"Workflow: {saved_workflow.id}\")\n", + "print(f\"Project: {project_id}\")\n", + "\n", + "job_name = f\"{MY_WORKFLOW_NAME} {MATERIAL_SET} {timestamp}\"\n", + "job_response = create_job(\n", + " api_client=client,\n", + " materials=saved_materials,\n", + " workflow=workflow,\n", + " project_id=project_id,\n", + " owner_id=ACCOUNT_ID,\n", + " prefix=job_name,\n", + " compute=compute.to_dict(),\n", + " materials_set=materials_set,\n", + ")\n", + "\n", + "job = dict_to_namespace_recursive(job_response)\n", + "job_id = job._id\n", + "print(\"✅ Job created successfully!\")\n", + "print(f\"Job ID: {job_id}\")\n", + "display_JSON(job_response)" + ] + }, + { + "cell_type": "markdown", + "id": "34", + "metadata": {}, + "source": [ + "## 7. Submit the job and monitor the status" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "35", + "metadata": {}, + "outputs": [], + "source": [ + "client.jobs.submit(job_id)\n", + "print(f\"✅ Job {job_id} submitted successfully!\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "36", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.api.job import wait_for_jobs_to_finish_async\n", + "\n", + "await wait_for_jobs_to_finish_async(client.jobs, [job_id], poll_interval=POLL_INTERVAL)\n", + "\n", + "job_status = client.jobs.get(job_id)[\"status\"]\n", + "if job_status != \"finished\":\n", + " job_errors = (client.jobs.get(job_id).get(\"compute\") or {}).get(\"errors\") or []\n", + " raise RuntimeError(f\"Job {job_id} ended with status={job_status!r}, errors={job_errors!r}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "37", + "metadata": {}, + "source": [ + "## 8. Retrieve results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38", + "metadata": {}, + "outputs": [], + "source": [ + "from mat3ra.notebooks_utils.ipython.entity.property.visualize import visualize_properties\n", + "\n", + "profile_data = client.properties.get_for_job(job_id, property_name=\"reaction_energy_profile\")\n", + "if not profile_data:\n", + " raise ValueError(f\"Job {job_id} finished but returned no reaction_energy_profile.\")\n", + "visualize_properties(profile_data, title=\"Reaction Energy Profile\")\n", + "\n", + "energies = profile_data[0][\"yDataSeries\"][0]\n", + "print(f\"Peak reaction energy = {max(energies):.3f} eV\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "39", + "metadata": {}, + "outputs": [], + "source": [ + "barrier_data = client.properties.get_for_job(job_id, property_name=\"reaction_energy_barrier\")\n", + "visualize_properties(barrier_data, title=\"Reaction Energy Barrier\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/py/mat3ra/notebooks_utils/core/entity/job/api.py b/src/py/mat3ra/notebooks_utils/core/entity/job/api.py index 5bec1189a..9e2650b01 100644 --- a/src/py/mat3ra/notebooks_utils/core/entity/job/api.py +++ b/src/py/mat3ra/notebooks_utils/core/entity/job/api.py @@ -1,8 +1,10 @@ import urllib.request -from typing import List, Optional, Union +from typing import Any, Dict, List, Optional, Union from mat3ra.api_client import APIClient, JobEndpoints +MATERIALS_SET_ENTITY_CLASS = "Material" + def save_files(job_id: str, job_endpoint: JobEndpoints, filename_on_cloud: str, filename_on_disk: str) -> None: """ @@ -37,6 +39,33 @@ def get_jobs_statuses_by_ids(endpoint: JobEndpoints, job_ids: List[str]) -> List return [job["status"] for job in jobs] +def _materials_set_reference(materials_set: Dict[str, Any]) -> Dict[str, str]: + """ + Builds the `_materialsSet` reference a job config expects. + + Mirrors what the job designer sends: the set's ID, the entity class it holds, + and a slug. The platform resolves members from the ID, so `slug` is only a + label — falling back to `name` keeps it readable when the response omits it. + + Args: + materials_set (dict): Materials set document. + + Returns: + dict: The `_materialsSet` reference. + + Raises: + KeyError: If the set document carries neither `slug` nor `name`. + """ + slug = materials_set.get("slug") or materials_set.get("name") + if not slug: + raise KeyError(f"Materials set {materials_set['_id']} has neither 'slug' nor 'name'.") + return { + "_id": materials_set["_id"], + "cls": MATERIALS_SET_ENTITY_CLASS, + "slug": slug, + } + + def create_job( api_client: APIClient, materials: List[dict], @@ -45,6 +74,7 @@ def create_job( owner_id: str, prefix: str, compute: Optional[dict] = None, + materials_set: Optional[Dict[str, Any]] = None, ) -> Union[dict, List[dict]]: """ Creates jobs using pre-serialised material and workflow dicts. @@ -57,6 +87,8 @@ def create_job( owner_id (str): Account ID. prefix (str): Job name prefix. compute (dict, optional): Compute configuration dict. + materials_set (dict, optional): Ordered/unordered materials set document + (same contract as the job designer `_materialsSet`). Returns: dict | list[dict]: Created job(s). @@ -75,6 +107,9 @@ def create_job( if is_multimaterial: config["_materials"] = [{"_id": m["_id"]} for m in materials] + if materials_set is not None: + config["_materialsSet"] = _materials_set_reference(materials_set) + if compute: config["compute"] = compute diff --git a/src/py/mat3ra/notebooks_utils/core/entity/material/api.py b/src/py/mat3ra/notebooks_utils/core/entity/material/api.py index 3051b42ad..97be0959b 100644 --- a/src/py/mat3ra/notebooks_utils/core/entity/material/api.py +++ b/src/py/mat3ra/notebooks_utils/core/entity/material/api.py @@ -1,8 +1,14 @@ +import re +from typing import Any, Dict, List, Optional + from mat3ra.api_client import APIClient from mat3ra.made.material import Material from .analysis import get_slab_bulk_crystal, resolve_bulk_query_from_crystal +ORDERED_ENTITY_SET_TYPE = "ordered" +UNORDERED_ENTITY_SET_TYPE = "unordered" + def get_or_create_material(api_client: APIClient, material, owner_id: str) -> dict: """ @@ -72,3 +78,140 @@ def _require_material_for_owner(api_client: APIClient, query: dict, owner_id: st "Run the Total Energy notebook for that bulk material first, then rerun this notebook." ) return Material.create(material_response) + + +def _index_in_set(material: Dict[str, Any], material_set_id: str) -> float: + """ + Path-order sort key. A member with no recorded index sorts last, so a + partially indexed set degrades to "known order first" instead of reshuffling. + """ + for entry in material.get("inSet") or []: + if entry.get("_id") == material_set_id: + index = entry.get("index") + return float(index) if index is not None else float("inf") + return float("inf") + + +def find_material_set( + api_client: APIClient, + owner_id: str, + material_set_name: str, + require_ordered: bool = False, +) -> Dict[str, Any]: + """ + Find a materials entity set by name (case-insensitive substring match). + + Args: + api_client (APIClient): API client instance carrying the authorization context. + owner_id (str): Account ID that owns the set. + material_set_name (str): Substring matched against set names under the owner. + require_ordered (bool): Reject the match unless it carries path order. + + Returns: + dict: The first matching materials set document. + + Raises: + ValueError: If no set matches, or if `require_ordered` and the match is unordered. + """ + material_sets = api_client.materials.list( + { + "owner._id": owner_id, + "isEntitySet": True, + "name": {"$regex": re.escape(material_set_name), "$options": "i"}, + } + ) + if not material_sets: + raise ValueError(f"No material set matching '{material_set_name}'") + material_set = material_sets[0] + + # Only an ordered set gets inSet.index values, so sorting an unordered one leaves + # every member tied — an arbitrary path, submitted without an error. + entity_set_type = material_set.get("entitySetType") + if require_ordered and entity_set_type != ORDERED_ENTITY_SET_TYPE: + raise ValueError( + f"Materials set '{material_set.get('name')}' is '{entity_set_type}', not " + f"'{ORDERED_ENTITY_SET_TYPE}'. Its members carry no path order." + ) + return material_set + + +def list_materials_in_set(api_client: APIClient, owner_id: str, material_set: Dict[str, Any]) -> List[Dict[str, Any]]: + """ + Members of an already-resolved set, ascending by `inSet.index` + (first -> optional intermediates -> last). Takes a resolved set so callers + that already have one do not re-query for it. + """ + material_set_id = material_set["_id"] + matches = api_client.materials.list({"owner._id": owner_id, "inSet._id": material_set_id}) + members = [material for material in matches if not material.get("isEntitySet")] + return sorted(members, key=lambda material: _index_in_set(material, material_set_id)) + + +def list_materials_by_set( + api_client: APIClient, + owner_id: str, + material_set_name: str, + require_ordered: bool = False, +) -> List[Dict[str, Any]]: + """Resolve a materials set by name and list its members in path order.""" + material_set = find_material_set(api_client, owner_id, material_set_name, require_ordered=require_ordered) + return list_materials_in_set(api_client, owner_id, material_set) + + +def get_or_create_materials_set( + api_client: APIClient, + owner_id: str, + material_set_name: str, + materials: List[Any], + is_ordered: bool = False, +) -> Dict[str, Any]: + """ + Reuse an existing materials set by name, or create one, then move members into it. + + Members are moved one at a time in list order: the platform assigns `inSet.index` + in the order it receives them, which is what makes an ordered set's path order + match the caller's list. + + Args: + api_client (APIClient): API client instance carrying the authorization context. + owner_id (str): Account ID under which to find or create the set. + material_set_name (str): Name of the set to reuse or create. + materials (list): Members to include (dict responses or Made objects with `.id`). + is_ordered (bool): Whether path order (`inSet.index`) matters for this set. + + Returns: + dict: The existing or newly created materials set document. + + Raises: + ValueError: If materials are empty, if an ordered set has fewer than two members, + or if an existing set of that name has the opposite `entitySetType`. + """ + if not materials: + raise ValueError("Materials set needs at least one material.") + if is_ordered and len(materials) < 2: + raise ValueError("Ordered materials set needs at least two materials.") + + entity_set_type = ORDERED_ENTITY_SET_TYPE if is_ordered else UNORDERED_ENTITY_SET_TYPE + try: + materials_set: Optional[Dict[str, Any]] = find_material_set(api_client, owner_id, material_set_name) + except ValueError: + materials_set = None + + if materials_set is None: + materials_set = api_client.materials.create_set( + {"name": material_set_name, "owner": {"_id": owner_id}, "entitySetType": entity_set_type} + ) + print(f"✅ Materials set '{materials_set['name']}' ({entity_set_type}, {materials_set['_id']})") + else: + existing_type = materials_set.get("entitySetType") + if existing_type != entity_set_type: + raise ValueError( + f"Materials set '{materials_set['name']}' already exists as '{existing_type}', but " + f"'{entity_set_type}' was requested. Reusing it would silently drop path order." + ) + print(f"♻️ Reusing materials set '{materials_set['name']}' ({existing_type}, {materials_set['_id']})") + + for material in materials: + identifier = material["_id"] if isinstance(material, dict) else material.id + api_client.materials.move_to_set(identifier, "", materials_set["_id"]) + return materials_set diff --git a/tests/py/unit/core/entity/test_job_api.py b/tests/py/unit/core/entity/test_job_api.py new file mode 100644 index 000000000..762cd5e54 --- /dev/null +++ b/tests/py/unit/core/entity/test_job_api.py @@ -0,0 +1,72 @@ +from typing import Any, Dict, List +from unittest.mock import MagicMock + +import pytest +from mat3ra.notebooks_utils.core.entity.job.api import create_job + +OWNER_ID = "account-1" +PROJECT_ID = "project-1" +JOB_PREFIX = "NEB H2+H" +MATERIAL_SET_ID = "set-1" +MATERIAL_SET_NAME = "H2+H" + +MATERIAL_INITIAL: Dict[str, Any] = {"_id": "m-initial", "name": "initial"} +MATERIAL_FINAL: Dict[str, Any] = {"_id": "m-final", "name": "final"} +MATERIALS: List[Dict[str, Any]] = [MATERIAL_INITIAL, MATERIAL_FINAL] + +MULTI_MATERIAL_WORKFLOW: Dict[str, Any] = { + "_id": "workflow-1", + "name": "NEB", + "isMultiMaterial": True, +} +SINGLE_MATERIAL_WORKFLOW: Dict[str, Any] = { + "_id": "workflow-2", + "name": "Total Energy", + "isMultiMaterial": False, +} +MATERIALS_SET: Dict[str, Any] = { + "_id": MATERIAL_SET_ID, + "name": MATERIAL_SET_NAME, + "slug": MATERIAL_SET_NAME, + "isEntitySet": True, +} +CREATED_JOB: Dict[str, Any] = {"_id": "job-1", "name": JOB_PREFIX} + + +@pytest.mark.parametrize( + ("workflow", "materials_set", "expected_materials_set"), + [ + (MULTI_MATERIAL_WORKFLOW, MATERIALS_SET, True), + (MULTI_MATERIAL_WORKFLOW, None, False), + (SINGLE_MATERIAL_WORKFLOW, MATERIALS_SET, True), + ], +) +def test_create_job_sets_materials_set_when_provided(workflow, materials_set, expected_materials_set): + client = MagicMock() + client.jobs.create.return_value = CREATED_JOB + workflow_payload = dict(workflow) + + job = create_job( + api_client=client, + materials=MATERIALS, + workflow=workflow_payload, + project_id=PROJECT_ID, + owner_id=OWNER_ID, + prefix=JOB_PREFIX, + materials_set=materials_set, + ) + + assert job["_id"] == "job-1" + config = client.jobs.create.call_args.args[0] + assert "_id" not in workflow_payload + assert ("_materialsSet" in config) is expected_materials_set + if expected_materials_set: + assert config["_materialsSet"] == { + "_id": MATERIAL_SET_ID, + "cls": "Material", + "slug": MATERIAL_SET_NAME, + } + if workflow.get("isMultiMaterial"): + assert config["_materials"] == [{"_id": "m-initial"}, {"_id": "m-final"}] + else: + assert "_materials" not in config diff --git a/tests/py/unit/core/entity/test_material_api.py b/tests/py/unit/core/entity/test_material_api.py new file mode 100644 index 000000000..4325d74e4 --- /dev/null +++ b/tests/py/unit/core/entity/test_material_api.py @@ -0,0 +1,223 @@ +import re +from typing import Any, Dict, List +from unittest.mock import MagicMock + +import pytest +from mat3ra.notebooks_utils.core.entity.material.api import ( + find_material_set, + get_or_create_materials_set, + list_materials_by_set, + list_materials_in_set, +) + +OWNER_ID = "account-1" +MATERIAL_SET_NAME = "H2+H" +MATERIAL_SET_ID = "set-1" + +MATERIAL_INITIAL: Dict[str, Any] = { + "_id": "m-initial", + "name": "path-start", + "isEntitySet": False, + "inSet": [{"_id": MATERIAL_SET_ID, "index": 0}], +} +MATERIAL_IMAGE: Dict[str, Any] = { + "_id": "m-image", + "name": "path-mid", + "isEntitySet": False, + "inSet": [{"_id": MATERIAL_SET_ID, "index": 1}], +} +MATERIAL_FINAL: Dict[str, Any] = { + "_id": "m-final", + "name": "path-end", + "isEntitySet": False, + "inSet": [{"_id": MATERIAL_SET_ID, "index": 2}], +} +ENTITY_SET: Dict[str, Any] = { + "_id": MATERIAL_SET_ID, + "name": "H2+H", + "isEntitySet": True, + "entitySetType": "ordered", +} + +SET_MEMBER_MATERIALS_OUT_OF_ORDER: List[Dict[str, Any]] = [ + MATERIAL_FINAL, + MATERIAL_INITIAL, + MATERIAL_IMAGE, + ENTITY_SET, +] +EXPECTED_ORDERED_IDS = ["m-initial", "m-image", "m-final"] +EXPECTED_SINGLE_MEMBER_IDS = ["m-initial"] + + +def _client_with_list_responses(responses: List[List[Dict[str, Any]]]) -> MagicMock: + client = MagicMock() + client.materials.list.side_effect = responses + return client + + +def test_find_material_set_returns_first_match(): + client = _client_with_list_responses([[ENTITY_SET]]) + + material_set = find_material_set(client, OWNER_ID, MATERIAL_SET_NAME) + + assert material_set["_id"] == MATERIAL_SET_ID + client.materials.list.assert_called_once_with( + { + "owner._id": OWNER_ID, + "isEntitySet": True, + "name": {"$regex": re.escape(MATERIAL_SET_NAME), "$options": "i"}, + } + ) + assert "+" in MATERIAL_SET_NAME + assert re.escape(MATERIAL_SET_NAME) != MATERIAL_SET_NAME + + +def test_find_material_set_raises_when_missing(): + client = _client_with_list_responses([[]]) + + with pytest.raises(ValueError, match="No material set matching"): + find_material_set(client, OWNER_ID, MATERIAL_SET_NAME) + + +def test_find_material_set_rejects_unordered_when_order_required(): + client = _client_with_list_responses([[{**ENTITY_SET, "entitySetType": "unordered"}]]) + + with pytest.raises(ValueError, match="is 'unordered', not 'ordered'"): + find_material_set(client, OWNER_ID, MATERIAL_SET_NAME, require_ordered=True) + + +def test_list_materials_by_set_rejects_unordered_when_order_required(): + client = _client_with_list_responses([[{**ENTITY_SET, "entitySetType": "unordered"}]]) + + with pytest.raises(ValueError, match="is 'unordered', not 'ordered'"): + list_materials_by_set(client, OWNER_ID, MATERIAL_SET_NAME, require_ordered=True) + assert client.materials.list.call_count == 1 + + +def test_list_materials_in_set_does_not_re_resolve_the_set(): + client = _client_with_list_responses([SET_MEMBER_MATERIALS_OUT_OF_ORDER]) + + materials = list_materials_in_set(client, OWNER_ID, ENTITY_SET) + + assert [material["_id"] for material in materials] == EXPECTED_ORDERED_IDS + client.materials.list.assert_called_once_with({"owner._id": OWNER_ID, "inSet._id": MATERIAL_SET_ID}) + + +@pytest.mark.parametrize( + ("members", "expected_ids"), + [ + (SET_MEMBER_MATERIALS_OUT_OF_ORDER, EXPECTED_ORDERED_IDS), + ([MATERIAL_INITIAL], EXPECTED_SINGLE_MEMBER_IDS), + ], +) +def test_list_materials_by_set_orders_by_inset_index(members, expected_ids): + client = _client_with_list_responses([[ENTITY_SET], members]) + + materials = list_materials_by_set(client, OWNER_ID, MATERIAL_SET_NAME) + + assert [material["_id"] for material in materials] == expected_ids + assert client.materials.list.call_args_list[1].args[0] == { + "owner._id": OWNER_ID, + "inSet._id": MATERIAL_SET_ID, + } + + +@pytest.mark.parametrize( + ("is_ordered", "expected_entity_set_type", "materials"), + [ + (True, "ordered", [MATERIAL_INITIAL, MATERIAL_FINAL]), + (False, "unordered", [MATERIAL_INITIAL]), + ], +) +def test_get_or_create_materials_set_creates_when_missing(is_ordered, expected_entity_set_type, materials): + client = _client_with_list_responses([[]]) + client.materials.create_set.return_value = { + **ENTITY_SET, + "entitySetType": expected_entity_set_type, + } + + materials_set = get_or_create_materials_set( + client, + OWNER_ID, + MATERIAL_SET_NAME, + materials, + is_ordered=is_ordered, + ) + + assert materials_set["_id"] == MATERIAL_SET_ID + client.materials.create_set.assert_called_once_with( + { + "name": MATERIAL_SET_NAME, + "owner": {"_id": OWNER_ID}, + "entitySetType": expected_entity_set_type, + } + ) + assert client.materials.move_to_set.call_args_list[0].args == ( + materials[0]["_id"], + "", + MATERIAL_SET_ID, + ) + + +def test_get_or_create_materials_set_reuses_when_found(): + client = _client_with_list_responses([[ENTITY_SET]]) + materials = [MATERIAL_INITIAL, MATERIAL_FINAL] + + materials_set = get_or_create_materials_set( + client, + OWNER_ID, + MATERIAL_SET_NAME, + materials, + is_ordered=True, + ) + + assert materials_set["_id"] == MATERIAL_SET_ID + client.materials.create_set.assert_not_called() + assert client.materials.move_to_set.call_args_list[0].args == ( + materials[0]["_id"], + "", + MATERIAL_SET_ID, + ) + + +def test_get_or_create_materials_set_rejects_reuse_with_mismatched_type(): + unordered_set = {**ENTITY_SET, "entitySetType": "unordered"} + client = _client_with_list_responses([[unordered_set]]) + + with pytest.raises(ValueError, match="already exists as 'unordered'"): + get_or_create_materials_set( + client, + OWNER_ID, + MATERIAL_SET_NAME, + [MATERIAL_INITIAL, MATERIAL_FINAL], + is_ordered=True, + ) + client.materials.move_to_set.assert_not_called() + + +def test_get_or_create_materials_set_ordered_requires_two_materials(): + client = MagicMock() + + with pytest.raises(ValueError, match="at least two materials"): + get_or_create_materials_set( + client, + OWNER_ID, + MATERIAL_SET_NAME, + [MATERIAL_INITIAL], + is_ordered=True, + ) + client.materials.list.assert_not_called() + + +def test_get_or_create_materials_set_requires_one_material(): + client = MagicMock() + + with pytest.raises(ValueError, match="at least one material"): + get_or_create_materials_set( + client, + OWNER_ID, + MATERIAL_SET_NAME, + [], + is_ordered=False, + ) + client.materials.list.assert_not_called()