Skip to content

Handle keystore and VACUUM failures on the database-open path instead of crash-looping (#2213) - #2219

Merged
mpretty-cyro merged 10 commits into
devfrom
fix/keystore-unseal-crash-loop
Sep 28, 2026
Merged

mpretty-cyro merged 10 commits into
devfrom
fix/keystore-unseal-crash-loop

Conversation

@mpretty-cyro

@mpretty-cyro mpretty-cyro commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Two independent startup failures, both of which end with the app dying on every launch and neither of
which could reach the database error screen the app already ships. Fixes #2213.

No new strings.

1. A keystore failure could not reach the error screen

KeyStoreHelper.unseal rethrows every crypto failure as AssertionError. The database secret was
only dereferenced from the openHelper lazy, outside any handler, so once the sealed secret could no
longer be decrypted the process died on every launch with no message and no way out.

Two things had to change for the existing screen to be reachable:

  • Resolve the secret inside the migration flow, and catch AssertionError alongside Exception
    so the failure becomes MigrationState.Error. migrateCipherSettings returns early for anyone
    already past the KDF migration and so never touched the secret at all — guarding only openHelper
    would have caught nothing for the users actually affected. AssertionError is caught because
    KeyStoreHelper throws it on purpose; OutOfMemoryError, StackOverflowError and LinkageError
    are deliberately left to kill the process.
  • Hold database-backed startup work until the migration reports Completed. Getting the state
    right was not enough on its own: the first component handed a failure took the process down before
    the screen could appear. The pollers start their own work from their constructors, so the
    components are now resolved through a Provider asked for only once the migration completes, which
    defers construction as well as the callback. And not all database work goes through the startup
    sequence — BlindMappingRepository reaches it from a flow started on login state — so the callers
    are not enumerable from one place. For that reason openHelper waits rather than throwing:
    parking a caller that cannot proceed beats handing it a failure it does not expect, and a retry
    that succeeds releases everyone waiting.

2. A login state that was only unreadable was treated as absent

The login state is unsealed once, when its repository is built, and a failure is indistinguishable
from having no account — both leave it null. Surviving a keystore fault instead of crashing turns
that into data loss: the process stays alive with the account pinned to null, routing sends the user
to the welcome screen on top of their own data, and creating an account there re-seals a new seed
over the old one. Crashing was self-healing by accident, because every relaunch read both secrets
again.

LoginStateRepository now records that the state was unreadable rather than absent, and re-attempts
the read when the migration reaches Completed — the point at which there is fresh evidence the
keystore is working. A recovered read repopulates the state before anything routes on it, and the flag
keeps it a no-op in the ordinary case.

Found by a release review, and it is why this PR went back to draft.

3. "Clear Device and Restore" could not restore

It preserves the account by re-applying the in-memory login state, so when that state could not
be unsealed either it silently did what "Clear Device and Restart" does — after the user accepted a
warning saying their account would be restored. Now hidden in that case, which is the same
precondition the implementation itself checks.

4. Startup components that take no database dependency are no longer deferred

CurrentActivityObserver, AppDisguiseManager and NotificationChannelManager were held back with
the rest, and the comment claimed everything in that group reached the database — not true of any of
the three. CurrentActivityObserver registers the activity lifecycle callbacks in its own
constructor, so deferring it dropped the events arriving before the migration finished.

The remainder still waits as a group rather than being classified one by one: transitive reach
through injected dependencies is not cheap to establish per component, and starting one too early
costs more than starting one too late.

5. The keystore error code is now logged

android.security.KeyStoreException (API 33+) carries getNumericErrorCode() and
isTransientFailure(). That is the only thing distinguishing a transient keystore fault — where the
data is intact and a retry may well succeed — from a key that can no longer decrypt what it sealed.
Nothing else in the crash carries the distinction, and it was being discarded.

6. Separate defect: the weekly VACUUM could prevent startup

SQLCipherOpenHelper's postKey hook runs on the database-open path, and the VACUUM there was
unwrapped — so any failure reached the caller as a database that would not open rather than as failed
maintenance. It also recorded only success, so a VACUUM that threw was retried on every open from
then on: one full disk became a launch that never worked again.

It now records the attempt before making it, skips when there is less than twice the database file's
size free (roughly what rebuilding it needs), and catches what is left. Note this path is not
covered by the rest of this PR — postKey fires on the first lazy open, inside whichever caller
touches the database first, not inside the guarded migration block.

Verification

On an emulator, by corrupting the sealed secret's base64 in place — IV and keystore key untouched, so
a genuine GCM tag failure.

Scenario Result
Healthy secret normal startup, stable
Database secret corrupted process survives, error screen stays up, 0 uncaught exceptions, 0 ANRs
Database secret and login state corrupted does not reach this screen at all — see below
Retry tapped on a permanent failure re-attempts the unseal, stays up, no crash
Both restored, relaunched normal startup, database keyed, deferred components start as expected
Login state unreadable, database secret fine the re-attempt fires with the flag set and genuinely re-reads — a corrupt blob logs the unseal failure twice, once per attempt — then routes to onboarding, correctly, because the account really is unreadable
Healthy start, and database-secret-only corruption the re-attempt stays a no-op; startup and the error screen are unchanged
Healthy start, after the component reordering unchanged; the deferred group still starts (pollers run) and CurrentActivityObserver now receives events immediately
Database secret corrupted, after the reordering error screen with all four buttons; pollers do not start (still deferred) while CurrentActivityObserver does; 0 uncaught exceptions, 0 ANRs — so none of the three un-deferred components reaches the database

Keystore failure: code=10, transient=false, systemError=false logged in each failing case —
ERROR_KEYMINT_FAILURE, correctly classified non-transient for a tag failure.

Pre-existing routing fault, unmasked by this change — needs fixing before this ships

This is the case that matters most, because if the keystore key itself is gone then everything
sealed under it fails — the database secret and the login state together. Measured sequence:

12:20:00.562  LoginStateRepository: Unable to unseal login state    -> login state becomes null
12:20:00.634  routeApplicationState() - STATE_WELCOME_SCREEN        -> routes to onboarding
12:20:00.653  DatabaseMigrationManager: Keystore failure            -> migration reaches Error, 19ms late

ScreenLockActionBarActivity.getApplicationState() reads migrationState.value while it is still
Idle, falls through, finds no login state, and routes to the welcome screen. The migration reaches
Error 19ms later, too late to affect the decision.

So the user is shown a fresh-install welcome screen with their database still intact on disk
(verified: session.db present, 1,130,496 bytes). That invites starting over, which destroys data
that a retry might have recovered — worse in that one respect than the crash loop it replaces, which
at least did not invite anything.

The fall-through is not introduced here. ScreenLockActionBarActivity's shouldShowUI covers
only Migrating and Error (:246-247), and the initial state is Idle, so the gap is already on
dev — this branch does not touch that file. It does not bite today only because the
AssertionError kills the process before routing completes; catching the error unmasks it.

The second arm of the same race is also dev's. openHelper waits
first { it == MigrationState.Completed } with no timeout, and Error is terminal — so once the
state settles there, that wait can never be satisfied and runBlocking holds the calling thread for
the life of the process. Both the terminal Error state and the unterminating wait are already on
dev; what changes here is only which faults reach them. On dev the keystore fault leaves
migration Completed, because the secret is never touched inside the try, and the process dies at
the SQLCipherOpenHelper(...) line instead. An ordinary Exception out of migrateCipherSettings
reaches Error on dev today and lands in the same wait.

Calling that "parking" was too comfortable a description on my part. The comment justifies blocking
until the migration resolves; Error is resolved, and the code cannot express it. So it is a
permanent thread leak rather than a wait, and openHelper is the entry point to the whole database
layer — bounded only by the IO dispatcher's thread cap, after which unrelated coroutine work starves
too. The foreground case is the visible one; the background case is the worse one to diagnose. A leak with
a ceiling is not a smaller problem than an unbounded one — its symptom arrives all at once and
somewhere else, as unrelated coroutine work stalling, with nothing in it pointing back at the
database layer. An ANR at least names its own thread.

So the routing change is necessary but not sufficient:

Race outcome Condition Closed by the routing change?
routes to WELCOME login state also unsealable yes
routes to NORMAL login state still readable only the foreground path

Any background caller reaching openHelper after an Error still blocks forever, routing or no
routing. Fully closing it needs Error as a terminating condition on the wait — first { it is Completed || it is Error } — plus a decision about what callers then get. Worth noting for whoever
takes that on: throwing there was tried in this branch's history and it killed the process, but that
was measured before the startup gating existed. With the gating and the routing change in place far
fewer callers arrive, so throwing may now be survivable — that is a thing to test, not an assertion.

Neither change is in this PR. Both are wider than it and neither is mine to decide.

Gaps a reviewer should know about

  • The login-state recovery branch is not covered. The re-attempt is verified to fire and to
    re-read, but not to succeed: a transient keystore fault cannot be induced on an emulator, and a
    corrupt blob fails identically on every attempt. The success path is reasoned from the code.
  • The routing race still gates part of the benefit. The re-attempt fires whenever the migration
    reaches Completed, including with no user action — but the review's scenario proper (error screen
    → Retry → recovered) still requires reaching the error screen, and which screen the user gets is the
    race described above. Observed going both ways across runs.
  • The VACUUM change is only partly verified. The block is entered when due and the timestamp now
    advances on entry, and startup is unaffected across repeated runs — but the skip and catch branches
    were never observed executing. A full emulator disk reclaims cache at exactly the boundary that
    would trigger them. Those two branches are reasoned, not tested.
  • On a permanent failure, whatever reaches openHelper parks for the life of the process, and
    runBlocking holds a thread. The gating keeps that to a handful. This is what the original code
    intended, but the Error state was previously unreachable, so it never actually happened.
  • Only exercised on one API level, and only the permanent (tag-failure) case. A genuinely transient
    keystore fault — where the retry would succeed and release the parked callers — was not reproduced.
  • The cause of the underlying key failures is still unidentified and is not addressed here. This turns
    an unexplained crash loop into a handled failure that can produce a log.

…aled (#2213)

KeyStoreHelper.unseal rethrows every crypto failure as AssertionError, and the
database secret was only ever dereferenced from the openHelper lazy, outside any
handler. The process died on every launch instead of reaching the error screen that
already exists. Resolve the secret inside the migration flow and catch Throwable so
the failure becomes MigrationState.Error, and end openHelper's wait on Error as well
as Completed — waiting for Completed alone would have hung every database caller once
that state was reachable.

"Clear Device and Restore" is now hidden when there is no login state, because it
restores from the in-memory state and would otherwise silently behave as "clear and
restart" after the user accepted a warning promising an account recovery.

Keystore failures now log the KeyStoreException error code, which is the only thing
distinguishing a transient fault, where the data is intact and a retry may succeed,
from a key that can no longer decrypt what it sealed.
…cret (#2213)

Routing the keystore failure into MigrationState.Error was not enough on its own: the
database is reached during startup by components that construct themselves eagerly and
by flows that start on login state, and the first of them to be handed an exception
took the process down before the migration screen could appear.

Split the startup components so everything that reaches the database is resolved
through a Provider that is only asked for once the migration reports Completed, which
defers construction as well as the callback — the pollers start their own work from
their constructors, so deferring onPostAppStarted alone would have started them
anyway.

openHelper goes back to waiting rather than throwing. Parking a caller that cannot
proceed is better than handing it a failure it does not expect, and the callers that
reach the database outside the startup sequence are not enumerable from one place. A
retry that succeeds reaches Completed and releases everyone waiting.

Verified on an emulator by corrupting the sealed secret in place: the app now survives
and shows the database error screen with Retry, Export Logs and Clear Device and
Restart, logs "Keystore failure: code=10, transient=false, systemError=false", drops
Clear Device and Restore when the login state is unsealable too, and starts normally
with no keystore failures once the secret is sound.
catch (Throwable) also caught OutOfMemoryError, StackOverflowError and LinkageError,
turning a process that is already lost into a database error screen offering Retry and
Clear Data. Catch Exception and AssertionError instead: AssertionError is the one Error
on this path thrown deliberately, as KeyStoreHelper's way of reporting a crypto
failure, which is what makes singling it out defensible rather than arbitrary.
The VACUUM in postKey runs on the database-open path, unwrapped, so any failure
reached the caller as a database that would not open rather than as failed
maintenance. It also recorded only success, so a VACUUM that threw was retried on
every open from then on — one full disk became a launch that never worked again.

Record the attempt before making it, skip it when there is less than twice the
database file's size free since that is roughly what rebuilding it needs, and catch
what is left. Note the timestamp is persisted with apply(), so on a genuinely full
disk the write can be dropped and the attempt repeats next launch; the catch, not the
ordering, is what stops that being fatal.

Verified only in part: the block is entered when due and the timestamp now advances on
entry, and startup is unaffected across repeated runs. The skip and catch branches were
not observed executing — a full emulator disk reclaims cache at exactly the boundary
that would trigger them — so those two paths are reasoned, not tested.
@mpretty-cyro mpretty-cyro changed the title Handle a keystore failure on startup instead of crash-looping (#2213) Handle keystore and VACUUM failures on the database-open path instead of crash-looping (#2213) Sep 22, 2026
@mpretty-cyro
mpretty-cyro marked this pull request as ready for review September 22, 2026 03:22
@mpretty-cyro
mpretty-cyro marked this pull request as draft September 28, 2026 03:44
…g again

The login state is unsealed once, when its repository is built, and a failure is
indistinguishable from having no account: both leave it null. Surviving a keystore
fault instead of crashing turns that into data loss. The process now stays alive with
the account pinned to null, routing sends the user to the welcome screen on top of
their own data, and creating an account there re-seals a new seed over the old one.
Crashing was self-healing by accident — every relaunch read both secrets again.

Record that the state was unreadable rather than absent, and re-attempt the read when
the migration reaches Completed, which is the point where there is fresh evidence the
keystore is working. A recovered read repopulates the state before anything routes on
it; the flag keeps it a no-op in the ordinary case.

Verified on an emulator: the re-attempt fires with the flag set and genuinely re-reads
(a corrupt blob logs the unseal failure twice, once per attempt), stays a no-op when
the first read succeeded, and leaves healthy startup unchanged. The recovery branch
itself is not covered — a transient keystore fault cannot be induced, and a corrupt
blob fails identically on every attempt.
CurrentActivityObserver, AppDisguiseManager and NotificationChannelManager were held
back with the rest, and the comment claimed everything in the group reached the
database, which was not true of any of the three. CurrentActivityObserver registers the
activity lifecycle callbacks in its own constructor, so deferring it dropped the events
arriving before the migration finished.

They now start immediately. The remainder still waits as a group rather than being
classified one by one: transitive reach through injected dependencies is not cheap to
establish per component, and starting one too early costs more than starting one too
late. Renamed to match what the group now claims about itself.
The comment argued that blocking beat throwing, which was the reasoning at the time and
is not a property of the code. What a reader needs is the hazard: Error is terminal and
is not a terminating condition for the wait, so a caller arriving after a failed
migration blocks for the life of the process and its thread is never returned.
@mpretty-cyro
mpretty-cyro marked this pull request as ready for review September 28, 2026 04:53
@mpretty-cyro
mpretty-cyro merged commit 34fd2ba into dev Sep 28, 2026
5 checks passed
@mpretty-cyro
mpretty-cyro deleted the fix/keystore-unseal-crash-loop branch September 28, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session app every time that i open the app it crashes

1 participant