fix(profiler): prevent filtering target package name in module discovery

[hebaalazzeh]

Fix module resolution for packages like google-cloud-testutils whose top-level Python package (test_utils) was being filtered out by ignored_prefixes in find_module_from_package.

[gemini-code-assist]

Code Review

This pull request introduces PEP 0810 explicit lazy imports (lazy_modules) across several packages in google-cloud-core and updates the import profiler script to better handle namespace packages. The review feedback suggests placing the lazy_modules definitions before import statements in accordance with PEP 8, which would also eliminate the need for .flake8 E402 bypasses and typing imports. Additionally, the feedback points out a logic issue in the import profiler's package filtering that causes false positives for packages like google-cloud-build, recommending a more specific exception for google-cloud-testutils.

I am having trouble creating individual review comments. Click here to see my feedback.

scripts/import_profiler/profiler.py (413-424)

The current logic pkg_clean.endswith(top.replace("_", "")) introduces false positives for other packages whose names end with an ignored prefix.

For example:

• For google-cloud-build, pkg_clean is "googlecloudbuild". If a standard "build" directory exists, top is "build". Since "googlecloudbuild" ends with "build", the "build" directory will NOT be ignored, which is incorrect as it is just a build artifact directory.
• For google-cloud-notebooks, the "notebooks" directory (which contains Jupyter notebooks and is in ignored_prefixes) will NOT be ignored because "googlecloudnotebooks" ends with "notebooks".

Since google-cloud-testutils is the only package in the repository whose top-level Python package is actually in ignored_prefixes (test_utils), we should make this exception specific to google-cloud-testutils and test_utils. This avoids false positives for all other packages.

            ignored_prefixes = ("tests", "samples", "examples", "benchmark", "benchmarks", "third_party", "testing", "test_utils", "docs", "build", "dist", "bin", "ci", "scripts", "cloudbuild", "notebooks", "assets", "scratch", "specs")
            ignored_starts = ("test_", "sample_", "bench_", "example_", "doc_", "notebook_")
            
            filtered = []
            for p in pkgs:
                top = p.split(".")[0]
                is_ignored_top = top in ignored_prefixes or top.startswith(ignored_starts)
                if is_ignored_top and pkg == "google-cloud-testutils" and top == "test_utils":
                    is_ignored_top = False
                if is_ignored_top or p in ("google", "google.cloud"):
                    continue
                filtered.append(p)

[chalmerlowe]

I am glad you jumped on this in an effort to get some of our other PRs unblocked.
Thanks.

I have some strong preferences for approaches that should make the code simpler, more readable, and more maintainable.

I also provide some tips and facts that may help with justifying why these changes are a good idea.

[chalmerlowe]

#PREFERENCE

In line with the idea of making the approach a positive one (i.e. identify which packages to process) we can simplify this code, as well.

1. We ditch the continue keyword which complicates understanding of for loops (in large measure it is basically a goto statement which makes for spaghetti code)
2. By skipping the continue, we end up doing the thing we intend to do immediately after getting the answer to our question with zero intervening logic AND clear logic of why the append() step is happening. That clean step-like logic can be lost when we use a negative formulation and we inject a continue:

• "Should we process this?"
• "Yes?"
• "Cool, append to filtered"

Suggested change

	filtered = []
	filtered = []
	for p in pkgs:
	top = p.split(".")[0]
	if _should_process_namespace_package(top, pkg):
	filtered.append(p)

[chalmerlowe]

#PREFERENCE

If we choose to go with the positive approach, then this test will need to be renamed and potentially inverted to ensure we are getting back the packages to process not packages to ignore.

[parthea]

Can we determine the list programatically ? There could be information in setup.py that we can use to determine which namespaces should be profiled:

google-cloud-python/packages/bigframes/setup.py

Lines 113 to 121 in 9f1477c

	packages = [
	package
	for package in setuptools.find_namespace_packages()
	if package.startswith("bigframes")
	] + [
	package
	for package in setuptools.find_namespace_packages("third_party")
	if package.startswith("bigframes_vendored")
	]