Optimised database registration logic in CLEM workflow

[tieneupin]

Fixes a couple of issues with the database registration logic in the CLEM workflow:

Handle the possibility of there being multiple atlas images associated with a grid
The CLEM workflow currently assumes that all of the datasets stored under the top level folder belong to the same grid. It currently also uses the field of view of the incoming dataset to decide if it is to be treated as an atlas image or a grid. We thus occasionally run into a situation where more than one datasets in a grid get registered as atlases, at which point the workflow breaks because the SQL query expects a single matching database entry or no result.

This can be resolved by adjusting the logic slightly so that the database query returns all matching atlas entries, sorted by insertion order. It then uses the latest one to perform the ISPyB updates. Because the grid squares and data collection groups are re-updated for every new image set received by the Murfey backend, all entries for a given sample will be re-updated correctly with reference to the last atlas-class image registered.

Accidental insert when updating ImagingSite entries after registering GridSquare entries
In the _register_dcg_and_atlas function, we previously would create an ImagingSite entry if none was found. In production, this logic led to the accidental creation of partially populated duplicate rows due to race conditions when registering ImagingSite entries in the database (one worker thinks there is no entry, and tries to create a duplicate row before the transaction done by another worker is set). _register_dcg_and_atlas should not be creating any new ImagingSite rows, but should error if such a race condition crops up.

Aggressive commits
In register_grid_square, db.commit() was previously run multiple times within a single for loop. By moving it out of the loop, this should improve performance.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.88%. Comparing base (3d23916) to head (9e1d4af).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #787      +/-   ##
==========================================
+ Coverage   50.72%   50.88%   +0.16%     
==========================================
  Files          96       96              
  Lines       10059    10089      +30     
  Branches     1322     1337      +15     
==========================================
+ Hits         5102     5134      +32     
+ Misses       4690     4686       -4     
- Partials      267      269       +2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.