chore: migrate xmllint to Python xml.etree.ElementTree in CI scripts and workflows - #14181
chore: migrate xmllint to Python xml.etree.ElementTree in CI scripts and workflows#14181blakeli0 wants to merge 2 commits into
Conversation
…and workflows Migrate xmllint (libxml2-utils) usages across Kokoro scripts and generation tools to Python 3's built-in xml.etree.ElementTree. This eliminates the need for installing libxml2-utils in GitHub Actions workflows, resolving CI timeouts caused by transient mirror stalls (fixes googleapis#14132).
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces the dependency on xmllint with inline Python scripts using xml.etree.ElementTree to parse and modify Maven pom.xml files across various Kokoro and generation shell scripts. The review feedback highlights two main issues with the new Python implementations: first, ElementTree.parse() discards XML comments (such as license headers) when writing back to files, which can be resolved by using a custom parser with insert_comments=True; second, embedding shell variables like $1 directly inside Python script strings can cause syntax errors, so passing them as command-line arguments via sys.argv is recommended.
| python3 -c " | ||
| import xml.etree.ElementTree as ET, sys | ||
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | ||
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | ||
| tree = ET.parse(file) | ||
| root = tree.getroot() | ||
| updated = False | ||
| for elem in root.iter(): | ||
| art = elem.find('{*}artifactId') | ||
| ver = elem.find('{*}version') | ||
| if art is not None and art.text == artifact_id and ver is not None: | ||
| ver.text = new_ver | ||
| updated = True | ||
| if updated: | ||
| tree.write(file, encoding='utf-8', xml_declaration=True) | ||
| " "$2" "$3" |
There was a problem hiding this comment.
By default, xml.etree.ElementTree.parse() discards all XML comments (including crucial Apache license headers at the top of pom.xml files). When the modified tree is written back to the file, these comments are permanently lost.
To preserve comments in Python 3.8+, you can pass a custom parser configured with insert_comments=True to ET.parse().
| python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | |
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | |
| tree = ET.parse(file) | |
| root = tree.getroot() | |
| updated = False | |
| for elem in root.iter(): | |
| art = elem.find('{*}artifactId') | |
| ver = elem.find('{*}version') | |
| if art is not None and art.text == artifact_id and ver is not None: | |
| ver.text = new_ver | |
| updated = True | |
| if updated: | |
| tree.write(file, encoding='utf-8', xml_declaration=True) | |
| " "$2" "$3" | |
| python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | |
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | |
| parser = ET.XMLParser(target=ET.TreeBuilder(insert_comments=True)) | |
| tree = ET.parse(file, parser=parser) | |
| root = tree.getroot() | |
| updated = False | |
| for elem in root.iter(): | |
| art = elem.find('{*}artifactId') | |
| ver = elem.find('{*}version') | |
| if art is not None and art.text == artifact_id and ver is not None: | |
| ver.text = new_ver | |
| updated = True | |
| if updated: | |
| tree.write(file, encoding='utf-8', xml_declaration=True) | |
| " "$2" "$3" |
| python3 -c " | ||
| import xml.etree.ElementTree as ET, sys | ||
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | ||
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | ||
| tree = ET.parse(file) | ||
| root = tree.getroot() | ||
| updated = False | ||
| for elem in root.iter(): | ||
| art = elem.find('{*}artifactId') | ||
| ver = elem.find('{*}version') | ||
| if art is not None and art.text == artifact_id and ver is not None: | ||
| ver.text = new_ver | ||
| updated = True | ||
| if updated: | ||
| tree.write(file, encoding='utf-8', xml_declaration=True) | ||
| " "$2" "$3" |
There was a problem hiding this comment.
By default, xml.etree.ElementTree.parse() discards all XML comments (including crucial Apache license headers at the top of pom.xml files). When the modified tree is written back to the file, these comments are permanently lost.
To preserve comments in Python 3.8+, you can pass a custom parser configured with insert_comments=True to ET.parse().
| python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | |
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | |
| tree = ET.parse(file) | |
| root = tree.getroot() | |
| updated = False | |
| for elem in root.iter(): | |
| art = elem.find('{*}artifactId') | |
| ver = elem.find('{*}version') | |
| if art is not None and art.text == artifact_id and ver is not None: | |
| ver.text = new_ver | |
| updated = True | |
| if updated: | |
| tree.write(file, encoding='utf-8', xml_declaration=True) | |
| " "$2" "$3" | |
| python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| file, artifact_id, new_ver = 'pom.xml', sys.argv[1], sys.argv[2] | |
| ET.register_namespace('', 'http://maven.apache.org/POM/4.0.0') | |
| parser = ET.XMLParser(target=ET.TreeBuilder(insert_comments=True)) | |
| tree = ET.parse(file, parser=parser) | |
| root = tree.getroot() | |
| updated = False | |
| for elem in root.iter(): | |
| art = elem.find('{*}artifactId') | |
| ver = elem.find('{*}version') | |
| if art is not None and art.text == artifact_id and ver is not None: | |
| ver.text = new_ver | |
| updated = True | |
| if updated: | |
| tree.write(file, encoding='utf-8', xml_declaration=True) | |
| " "$2" "$3" |
| result=$(python3 -c " | ||
| import xml.etree.ElementTree as ET, sys | ||
| try: | ||
| root = ET.parse('$1/pom.xml').getroot() | ||
| v = root.find('{*}version') | ||
| if v is None: | ||
| v = root.find('{*}parent/{*}version') | ||
| if v is not None and v.text: | ||
| print(v.text.strip()) | ||
| else: | ||
| sys.exit(1) | ||
| except Exception: | ||
| sys.exit(1) | ||
| ") |
There was a problem hiding this comment.
Using shell expansion ($1) directly inside the Python script string can lead to syntax errors or unexpected behavior if the directory path contains spaces, single quotes, or other special characters. It is much safer and more robust to pass the file path as a command-line argument to the Python script and access it via sys.argv.
| result=$(python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| try: | |
| root = ET.parse('$1/pom.xml').getroot() | |
| v = root.find('{*}version') | |
| if v is None: | |
| v = root.find('{*}parent/{*}version') | |
| if v is not None and v.text: | |
| print(v.text.strip()) | |
| else: | |
| sys.exit(1) | |
| except Exception: | |
| sys.exit(1) | |
| ") | |
| result=$(python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| try: | |
| root = ET.parse(sys.argv[1]).getroot() | |
| v = root.find('{*}version') | |
| if v is None: | |
| v = root.find('{*}parent/{*}version') | |
| if v is not None and v.text: | |
| print(v.text.strip()) | |
| else: | |
| sys.exit(1) | |
| except Exception: | |
| sys.exit(1) | |
| " "$1/pom.xml") |
| result=$(python3 -c " | ||
| import xml.etree.ElementTree as ET, sys | ||
| try: | ||
| root = ET.parse('$1/pom.xml').getroot() | ||
| v = root.find('{*}version') | ||
| if v is None: | ||
| v = root.find('{*}parent/{*}version') | ||
| if v is not None and v.text: | ||
| print(v.text.strip()) | ||
| else: | ||
| sys.exit(1) | ||
| except Exception: | ||
| sys.exit(1) | ||
| ") |
There was a problem hiding this comment.
Using shell expansion ($1) directly inside the Python script string can lead to syntax errors or unexpected behavior if the directory path contains spaces, single quotes, or other special characters. It is much safer and more robust to pass the file path as a command-line argument to the Python script and access it via sys.argv.
| result=$(python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| try: | |
| root = ET.parse('$1/pom.xml').getroot() | |
| v = root.find('{*}version') | |
| if v is None: | |
| v = root.find('{*}parent/{*}version') | |
| if v is not None and v.text: | |
| print(v.text.strip()) | |
| else: | |
| sys.exit(1) | |
| except Exception: | |
| sys.exit(1) | |
| ") | |
| result=$(python3 -c " | |
| import xml.etree.ElementTree as ET, sys | |
| try: | |
| root = ET.parse(sys.argv[1]).getroot() | |
| v = root.find('{*}version') | |
| if v is None: | |
| v = root.find('{*}parent/{*}version') | |
| if v is not None and v.text: | |
| print(v.text.strip()) | |
| else: | |
| sys.exit(1) | |
| except Exception: | |
| sys.exit(1) | |
| " "$1/pom.xml") |
… script Extract XML parsing and manipulation logic into standalone pom_utils.py scripts with dedicated CLI commands (get-version, has-versioned-dep, get-dep-version, update-dep-version, update-property, get-coordinates) for cleaner readability and reusability.
1 participant
Description
Migrate all
xmllint(libxml2-utils) usages across Kokoro scripts and generation tools to Python 3's built-inxml.etree.ElementTree.This eliminates the need for
sudo apt-get install libxml2-utilssteps in GitHub Actions workflows, resolving intermittent CI timeouts caused by stalled Ubuntu/Azure package mirrors.Fixes #14132
Changes
.kokoro/common.sh&sdk-platform-java/.kokoro/presubmit/common.sh:parse_pom_version: Extracts project version using Pythonxml.etree.ElementTree.find_all_poms_with_versioned_dependency: Checks for versioned artifact dependencies using Python.update_pom_dependency: Updates target dependency version nodes inpom.xmlvia Python..kokoro/common_test.sh:xmllint --shell.generation/check_existing_release_versions.sh:xmllint --xpathcalls with a single Python invocation to extractgroupId,artifactId, andversion..kokoro/client-library-check.sh&.kokoro/client-library-check-doclet.sh:replace_java_shared_config_version,replace_java_shared_dependencies_version, andreplace_sdk_platform_java_config_versionwith Python XML modification..kokoro/presubmit/downstream-build.sh&sdk-platform-java/scripts/create_native_image_test_env.sh:modify_shared_config,modify_shared_dependencies, and XPath lookups with Python.libxml2-utils/xmllintinstallation steps insdk-platform-java-downstream.yaml,versions.yaml,java-shared-config-downstream-maven-plugins.yaml, andjava-shared-config-downstream-dependencies.yaml.