Handle keystore and VACUUM failures on the database-open path instead of crash-looping (#2213) - #2219
Merged
Merged
Conversation
…aled (#2213) KeyStoreHelper.unseal rethrows every crypto failure as AssertionError, and the database secret was only ever dereferenced from the openHelper lazy, outside any handler. The process died on every launch instead of reaching the error screen that already exists. Resolve the secret inside the migration flow and catch Throwable so the failure becomes MigrationState.Error, and end openHelper's wait on Error as well as Completed — waiting for Completed alone would have hung every database caller once that state was reachable. "Clear Device and Restore" is now hidden when there is no login state, because it restores from the in-memory state and would otherwise silently behave as "clear and restart" after the user accepted a warning promising an account recovery. Keystore failures now log the KeyStoreException error code, which is the only thing distinguishing a transient fault, where the data is intact and a retry may succeed, from a key that can no longer decrypt what it sealed.
…cret (#2213) Routing the keystore failure into MigrationState.Error was not enough on its own: the database is reached during startup by components that construct themselves eagerly and by flows that start on login state, and the first of them to be handed an exception took the process down before the migration screen could appear. Split the startup components so everything that reaches the database is resolved through a Provider that is only asked for once the migration reports Completed, which defers construction as well as the callback — the pollers start their own work from their constructors, so deferring onPostAppStarted alone would have started them anyway. openHelper goes back to waiting rather than throwing. Parking a caller that cannot proceed is better than handing it a failure it does not expect, and the callers that reach the database outside the startup sequence are not enumerable from one place. A retry that succeeds reaches Completed and releases everyone waiting. Verified on an emulator by corrupting the sealed secret in place: the app now survives and shows the database error screen with Retry, Export Logs and Clear Device and Restart, logs "Keystore failure: code=10, transient=false, systemError=false", drops Clear Device and Restore when the login state is unsealable too, and starts normally with no keystore failures once the secret is sound.
catch (Throwable) also caught OutOfMemoryError, StackOverflowError and LinkageError, turning a process that is already lost into a database error screen offering Retry and Clear Data. Catch Exception and AssertionError instead: AssertionError is the one Error on this path thrown deliberately, as KeyStoreHelper's way of reporting a crypto failure, which is what makes singling it out defensible rather than arbitrary.
The VACUUM in postKey runs on the database-open path, unwrapped, so any failure reached the caller as a database that would not open rather than as failed maintenance. It also recorded only success, so a VACUUM that threw was retried on every open from then on — one full disk became a launch that never worked again. Record the attempt before making it, skip it when there is less than twice the database file's size free since that is roughly what rebuilding it needs, and catch what is left. Note the timestamp is persisted with apply(), so on a genuinely full disk the write can be dropped and the attempt repeats next launch; the catch, not the ordering, is what stops that being fatal. Verified only in part: the block is entered when due and the timestamp now advances on entry, and startup is unaffected across repeated runs. The skip and catch branches were not observed executing — a full emulator disk reclaims cache at exactly the boundary that would trigger them — so those two paths are reasoned, not tested.
mpretty-cyro
marked this pull request as ready for review
September 22, 2026 03:22
mpretty-cyro
marked this pull request as draft
September 28, 2026 03:44
…g again The login state is unsealed once, when its repository is built, and a failure is indistinguishable from having no account: both leave it null. Surviving a keystore fault instead of crashing turns that into data loss. The process now stays alive with the account pinned to null, routing sends the user to the welcome screen on top of their own data, and creating an account there re-seals a new seed over the old one. Crashing was self-healing by accident — every relaunch read both secrets again. Record that the state was unreadable rather than absent, and re-attempt the read when the migration reaches Completed, which is the point where there is fresh evidence the keystore is working. A recovered read repopulates the state before anything routes on it; the flag keeps it a no-op in the ordinary case. Verified on an emulator: the re-attempt fires with the flag set and genuinely re-reads (a corrupt blob logs the unseal failure twice, once per attempt), stays a no-op when the first read succeeded, and leaves healthy startup unchanged. The recovery branch itself is not covered — a transient keystore fault cannot be induced, and a corrupt blob fails identically on every attempt.
…into fix/keystore-unseal-crash-loop
CurrentActivityObserver, AppDisguiseManager and NotificationChannelManager were held back with the rest, and the comment claimed everything in the group reached the database, which was not true of any of the three. CurrentActivityObserver registers the activity lifecycle callbacks in its own constructor, so deferring it dropped the events arriving before the migration finished. They now start immediately. The remainder still waits as a group rather than being classified one by one: transitive reach through injected dependencies is not cheap to establish per component, and starting one too early costs more than starting one too late. Renamed to match what the group now claims about itself.
The comment argued that blocking beat throwing, which was the reasoning at the time and is not a property of the code. What a reader needs is the hazard: Error is terminal and is not a terminating condition for the wait, so a caller arriving after a failed migration blocks for the life of the process and its thread is never returned.
mpretty-cyro
marked this pull request as ready for review
September 28, 2026 04:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent startup failures, both of which end with the app dying on every launch and neither of
which could reach the database error screen the app already ships. Fixes #2213.
No new strings.
1. A keystore failure could not reach the error screen
KeyStoreHelper.unsealrethrows every crypto failure asAssertionError. The database secret wasonly dereferenced from the
openHelperlazy, outside any handler, so once the sealed secret could nolonger be decrypted the process died on every launch with no message and no way out.
Two things had to change for the existing screen to be reachable:
AssertionErroralongsideExceptionso the failure becomes
MigrationState.Error.migrateCipherSettingsreturns early for anyonealready past the KDF migration and so never touched the secret at all — guarding only
openHelperwould have caught nothing for the users actually affected.
AssertionErroris caught becauseKeyStoreHelperthrows it on purpose;OutOfMemoryError,StackOverflowErrorandLinkageErrorare deliberately left to kill the process.
Completed. Getting the stateright was not enough on its own: the first component handed a failure took the process down before
the screen could appear. The pollers start their own work from their constructors, so the
components are now resolved through a
Providerasked for only once the migration completes, whichdefers construction as well as the callback. And not all database work goes through the startup
sequence —
BlindMappingRepositoryreaches it from a flow started on login state — so the callersare not enumerable from one place. For that reason
openHelperwaits rather than throwing:parking a caller that cannot proceed beats handing it a failure it does not expect, and a retry
that succeeds releases everyone waiting.
2. A login state that was only unreadable was treated as absent
The login state is unsealed once, when its repository is built, and a failure is indistinguishable
from having no account — both leave it null. Surviving a keystore fault instead of crashing turns
that into data loss: the process stays alive with the account pinned to null, routing sends the user
to the welcome screen on top of their own data, and creating an account there re-seals a new seed
over the old one. Crashing was self-healing by accident, because every relaunch read both secrets
again.
LoginStateRepositorynow records that the state was unreadable rather than absent, and re-attemptsthe read when the migration reaches
Completed— the point at which there is fresh evidence thekeystore is working. A recovered read repopulates the state before anything routes on it, and the flag
keeps it a no-op in the ordinary case.
Found by a release review, and it is why this PR went back to draft.
3. "Clear Device and Restore" could not restore
It preserves the account by re-applying the in-memory login state, so when that state could not
be unsealed either it silently did what "Clear Device and Restart" does — after the user accepted a
warning saying their account would be restored. Now hidden in that case, which is the same
precondition the implementation itself checks.
4. Startup components that take no database dependency are no longer deferred
CurrentActivityObserver,AppDisguiseManagerandNotificationChannelManagerwere held back withthe rest, and the comment claimed everything in that group reached the database — not true of any of
the three.
CurrentActivityObserverregisters the activity lifecycle callbacks in its ownconstructor, so deferring it dropped the events arriving before the migration finished.
The remainder still waits as a group rather than being classified one by one: transitive reach
through injected dependencies is not cheap to establish per component, and starting one too early
costs more than starting one too late.
5. The keystore error code is now logged
android.security.KeyStoreException(API 33+) carriesgetNumericErrorCode()andisTransientFailure(). That is the only thing distinguishing a transient keystore fault — where thedata is intact and a retry may well succeed — from a key that can no longer decrypt what it sealed.
Nothing else in the crash carries the distinction, and it was being discarded.
6. Separate defect: the weekly VACUUM could prevent startup
SQLCipherOpenHelper'spostKeyhook runs on the database-open path, and the VACUUM there wasunwrapped — so any failure reached the caller as a database that would not open rather than as failed
maintenance. It also recorded only success, so a VACUUM that threw was retried on every open from
then on: one full disk became a launch that never worked again.
It now records the attempt before making it, skips when there is less than twice the database file's
size free (roughly what rebuilding it needs), and catches what is left. Note this path is not
covered by the rest of this PR —
postKeyfires on the first lazy open, inside whichever callertouches the database first, not inside the guarded migration block.
Verification
On an emulator, by corrupting the sealed secret's base64 in place — IV and keystore key untouched, so
a genuine GCM tag failure.
CurrentActivityObservernow receives events immediatelyCurrentActivityObserverdoes; 0 uncaught exceptions, 0 ANRs — so none of the three un-deferred components reaches the databaseKeystore failure: code=10, transient=false, systemError=falselogged in each failing case —ERROR_KEYMINT_FAILURE, correctly classified non-transient for a tag failure.Pre-existing routing fault, unmasked by this change — needs fixing before this ships
This is the case that matters most, because if the keystore key itself is gone then everything
sealed under it fails — the database secret and the login state together. Measured sequence:
ScreenLockActionBarActivity.getApplicationState()readsmigrationState.valuewhile it is stillIdle, falls through, finds no login state, and routes to the welcome screen. The migration reachesError19ms later, too late to affect the decision.So the user is shown a fresh-install welcome screen with their database still intact on disk
(verified:
session.dbpresent, 1,130,496 bytes). That invites starting over, which destroys datathat a retry might have recovered — worse in that one respect than the crash loop it replaces, which
at least did not invite anything.
The fall-through is not introduced here.
ScreenLockActionBarActivity'sshouldShowUIcoversonly
MigratingandError(:246-247), and the initial state isIdle, so the gap is already ondev— this branch does not touch that file. It does not bite today only because theAssertionErrorkills the process before routing completes; catching the error unmasks it.The second arm of the same race is also dev's.
openHelperwaitsfirst { it == MigrationState.Completed }with no timeout, andErroris terminal — so once thestate settles there, that wait can never be satisfied and
runBlockingholds the calling thread forthe life of the process. Both the terminal
Errorstate and the unterminating wait are already ondev; what changes here is only which faults reach them. Ondevthe keystore fault leavesmigration
Completed, because the secret is never touched inside the try, and the process dies atthe
SQLCipherOpenHelper(...)line instead. An ordinaryExceptionout ofmigrateCipherSettingsreaches
Errorondevtoday and lands in the same wait.Calling that "parking" was too comfortable a description on my part. The comment justifies blocking
until the migration resolves;
Erroris resolved, and the code cannot express it. So it is apermanent thread leak rather than a wait, and
openHelperis the entry point to the whole databaselayer — bounded only by the IO dispatcher's thread cap, after which unrelated coroutine work starves
too. The foreground case is the visible one; the background case is the worse one to diagnose. A leak with
a ceiling is not a smaller problem than an unbounded one — its symptom arrives all at once and
somewhere else, as unrelated coroutine work stalling, with nothing in it pointing back at the
database layer. An ANR at least names its own thread.
So the routing change is necessary but not sufficient:
WELCOMENORMALAny background caller reaching
openHelperafter anErrorstill blocks forever, routing or norouting. Fully closing it needs
Erroras a terminating condition on the wait —first { it is Completed || it is Error }— plus a decision about what callers then get. Worth noting for whoevertakes that on: throwing there was tried in this branch's history and it killed the process, but that
was measured before the startup gating existed. With the gating and the routing change in place far
fewer callers arrive, so throwing may now be survivable — that is a thing to test, not an assertion.
Neither change is in this PR. Both are wider than it and neither is mine to decide.
Gaps a reviewer should know about
re-read, but not to succeed: a transient keystore fault cannot be induced on an emulator, and a
corrupt blob fails identically on every attempt. The success path is reasoned from the code.
reaches
Completed, including with no user action — but the review's scenario proper (error screen→ Retry → recovered) still requires reaching the error screen, and which screen the user gets is the
race described above. Observed going both ways across runs.
advances on entry, and startup is unaffected across repeated runs — but the skip and catch branches
were never observed executing. A full emulator disk reclaims cache at exactly the boundary that
would trigger them. Those two branches are reasoned, not tested.
openHelperparks for the life of the process, andrunBlockingholds a thread. The gating keeps that to a handful. This is what the original codeintended, but the
Errorstate was previously unreachable, so it never actually happened.keystore fault — where the retry would succeed and release the parked callers — was not reproduced.
an unexplained crash loop into a handled failure that can produce a log.