Skip to content

Add automatic saliency cropping - #46

Closed
TorstenDittmann wants to merge 8 commits into
mainfrom
feat/semantic-focus-crop
Closed

Add automatic saliency cropping#46
TorstenDittmann wants to merge 8 commits into
mainfrom
feat/semantic-focus-crop

Conversation

@TorstenDittmann

@TorstenDittmann TorstenDittmann commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add Image::GRAVITY_AUTO for saliency-aware cropping
  • use the bundled full U2NET model through ankane/onnxruntime
  • add Image::detect() so model inference can run in a dedicated worker
  • allow crop() to consume a persisted detection result without loading ONNX
  • validate persisted saliency masks before using them
  • fall back to centered cropping for empty or uniform saliency maps
  • cache the ONNX model once per PHP worker
  • use Debian Bullseye/glibc test images for the supported ONNX Runtime binaries

Usage

// Detection worker
$image = new Image(\file_get_contents('image.jpg'));
$detectionJson = json_encode($image->detect(), JSON_THROW_ON_ERROR);
// Store $detectionJson in the database.

// Image worker
$image = new Image(\file_get_contents('image.jpg'));
$detection = json_decode($detectionJson, true, flags: JSON_THROW_ON_ERROR);
$image->crop(400, 300, Image::GRAVITY_AUTO, $detection);

The detection result contains width, height, and a normalized two-dimensional mask.

Model and runtime

  • full U2NET ONNX model stored with Git LFS
  • SHA-256: 8d10d2f3bb75ae3b6d527c77944fc5e7dcd94b29809d47a739a7a728a912b491
  • model source and Apache-2.0 attribution are in resources/models/NOTICE.md
  • applications must add OnnxRuntime\Vendor::check to root Composer post-install and post-update scripts
  • prebuilt Linux ONNX Runtime artifacts require glibc; Alpine/musl needs a compatible custom runtime

Performance

Measured on an Apple M3 Pro with a 1280x837 JPEG cropped to 180x320:

  • first automatic crop: about 491 ms
  • warm automatic crop: about 405-438 ms
  • regular centered crop: about 14 ms
  • first-crop process RSS: about 550 MiB
  • long-running process RSS: about 766-768 MiB

Native ONNX Runtime and Imagick allocations are not fully represented by PHP's memory counter. Automatic detection is best suited to an asynchronous, controlled-concurrency worker.

Testing

  • vendor/bin/pint --test
  • PHPStan level max
  • Docker PHP 8.3: 60 tests, 328 assertions reached; the two new detection-worker tests pass
  • the end-to-end U2NET test cannot load the model in this checkout because only its Git LFS pointer is present

@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds Image::GRAVITY_AUTO, a saliency-aware crop mode powered by the bundled U2NET ONNX model via ankane/onnxruntime. A new detect() method allows saliency inference to run in a dedicated worker and its result to be passed into crop() later, avoiding repeated model loading.

  • detectSaliency() preprocesses a 320×320 resized frame, runs the U2NET model through a process-level static cache, and normalizes the output mask; findSalientCrop() locates the highest-salience crop window using a 2D prefix-sum (verified correct) with center-proximity tie-breaking.
  • normalizeDetection() validates and clamps any externally persisted mask before use, providing a clear injection point for the worker-split workflow.
  • The integral-image score computation, coordinate mapping back to image space, and the upper-bound clamp (min($resizeWidth - $width, ...)) are all mathematically correct; the flat-mask center fallback and the epsilon-based tie-break are covered by dedicated tests.

Confidence Score: 4/5

  • The Image.php logic is correct and well-tested, but the dependency wiring in composer.json forces the ONNX runtime and FFI extension onto every consumer of this library — including those that never use GRAVITY_AUTO.
  • The core saliency algorithm, coordinate math, and validation are solid. The unresolved issue is in composer.json: ankane/onnxruntime sits in require rather than suggest, and ext-ffi is declared as a hard requirement, so any downstream package that installs utopia-php/image will unconditionally pull platform-specific ONNX binaries and fail entirely on FFI-disabled environments. This was called out in a previous review comment but remains unaddressed in this diff.
  • composer.json — the ONNX runtime dependency and FFI extension requirement affect all consumers of the library, not just those using the new automatic crop feature.

Important Files Changed

Filename Overview
src/Image/Image.php Adds GRAVITY_AUTO with U2NET saliency detection, detect(), normalizeDetection(), detectSaliency(), and findSalientCrop(). Algorithm is correct (integral image verified), validation in normalizeDetection() is thorough, static model caching is intentional per-worker design. No new bugs introduced in this file.
composer.json ankane/onnxruntime added to require (not suggest), ext-ffi required unconditionally, and post-install/post-update hooks download platform binaries for all consumers — even those that never use GRAVITY_AUTO. This is a pre-existing P1 already flagged in a previous comment on this PR.
tests/Image/ImageTest.php Comprehensive new tests for GRAVITY_AUTO: horizontal/vertical saliency steering, flat-mask center fallback, precomputed detection passthrough, center tie-breaking, coordinate clamping, and the end-to-end U2NET path. detectSaliency() is protected for easy subclassing in tests. All existing tests updated to snake_case naming convention.
resources/models/NOTICE.md Proper Apache 2.0 attribution for the U2NET ONNX model including source URL, MD5, and SHA-256 hash matching the LFS pointer.
.gitattributes Correctly registers *.onnx files under Git LFS with diff=lfs merge=lfs -text attributes.
README.md Documents GRAVITY_AUTO usage, detect()/crop() two-step workflow, Composer hook requirement, and Alpine/musl limitation clearly.

Reviews (6): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile

Comment thread src/Image/Image.php Outdated
Comment thread src/Image/Image.php Outdated
@TorstenDittmann TorstenDittmann changed the title Add semantic focus cropping Add automatic saliency cropping Jul 30, 2026
# Conflicts:
#	.gitignore
#	Dockerfile-php-8.1
#	Dockerfile-php-8.2
#	Dockerfile-php-8.3
#	composer.json
#	composer.lock
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant