Mutation Testing Results :: Dead Code + Missing Tests #1396

Activity

1. 81reap commented on Aug 22, 2026

   

   81reap

   on Aug 22, 2026

   Last edited by 81reap

   CollaboratorAuthor

   More actions

   LLM Breakdown

   since this is a large amount of issues, I also asked claude to break it up by type. I am not aware of a programmatic way to make this breakdown without the use of an LLM.

   note :: while an LLM was used for this part it was only used to process the files attached at the top of this issue, the data in SQLPage/mutants.out was created from a deterministic program.

   Harness gap (74)(maybe, up to maintainers if these are added benifits or not)

   File	Survivors	Why nothing catches it	PR
   telemetry.rs	52	no test asserts on log or span output; 19 replace a writer or layer hook with (), 16 of those in logfmt	#1434 WIP10.2
   webserver/lambda_http.rs	15	only compiled/run under the lambda-web feature on AWS	todo
   cli/commands.rs	6	100% survival — no test invokes a subcommand	todo
   main.rs	1	process entry point	todo

   Dialect-gated (33)(no-op, further deep dive required)

   The run used the default sqlite::memory:. These arms are unreachable without the matching driver.

   File	Survivors	Example	PR
   webserver/database/sql_to_json.rs	21	delete match arm "UUID" | "UNIQUEIDENTIFIER", "JSONB", "INT8RANGE", matches!(db_type, Mssql(_)) guards	todo
   webserver/database/mod.rs	4	delete match arm "postgres" | "postgresql" in SupportedDatabase::from_dbms_name	todo
   filesystem.rs	3	delete match arm SupportedDatabase::Mssql in DbFsQueries::get_create_table_sql	todo
   webserver/database/csv_import.rs	2	delete match arm AnyConnectionKind::Postgres in run_csv_import	todo
   webserver/database/execute_queries.rs	2	delete match arm sqlx::any::AnyKind::Postgres in set_trace_context	todo
   webserver/database/sql/rewrite.rs	1	match guard self.database.database_type == SupportedDatabase::Mssql	#1422

   Re-running with DATABASE_URL pointed at each backend would reclassify these; CI already has the matrix, but I do not have my dev machine properly setup to be able to re-run tests against these other DBs.

   Dead code (7)(yes, we should clean up/refactor these)

   Every item in #1389 shows up here independently.

   Mutant	Item	PR
   template_helpers.rs:525:9 replace <impl CanHelp for H0>::call with Ok(Default::default())	E2	#1412 #1419
   template_helpers.rs:526:13 delete match arm [] in <impl CanHelp for H0>::call	E2	#1412 #1419
   database/mod.rs:76:9 replace SupportedDatabase::otel_name_from_kind with ""	E3	todo
   database/mod.rs:76:9 replace SupportedDatabase::otel_name_from_kind with "xyzzy"	E3	todo
   error_with_status.rs:22:9 replace <impl ResponseError for ErrorWithStatus>::status_code with Default::default()	E4	todo
   error_with_status.rs:27:24 replace == with != in error_response	E4	todo
   url_parameters.rs:132:9 replace <impl From<URLParameters> for String>::from with Default::default()	E5	todo

   Logic gaps worth a test (subset of the 255)(yes, this is just adding more tests which is always a good thing)

   Each row was checked against the source; the last column is what a test would have to assert.

   Location	Survivors	Behaviour left unpinned	PR
   render.rs HeaderContext::add_cookie	10	secure and http_only default to on, remove/max_age handling. The only cookie test (tests/sql_test_files/data/cookie.sql) reads a cookie; nothing sets one through the component.	#1429 #1431 #1432 #1433
   template_helpers.rs plus/minus/sum	12	+→-, +→*, +=→-= all survive; no test exercises the arithmetic helpers	#1412
   template_helpers.rs MarkdownConfig for AppConfig	4	allow_dangerous_html and allow_dangerous_protocol can each be hardcoded true or false unnoticed	todo
   oidc.rs validate_redirect_url	3	delete ! in !url.starts_with(redirect_uri) and returning "" both survive; the existing test covers is_safe_relative_redirect, not this wrapper	todo
   oidc.rs set_auth_cookie / verify_logout_params	6	expiry and redirect-count comparisons (>→>=, >→==)	#1423 #1424 #1426
   sql/rewrite.rs recognize_sqlpage_function	5	every `
   sql/rewrite.rs has_single_row_shape / emulated_function	8	same, plus delete match arm "concat"	#1422
   app_config.rs default_*	12	defaults for max file size, ACME URL, markdown_allow_dangerous_protocol, system_root_ca_certificates, OIDC client id	todo
   app_config.rs validation & durations	6	AppConfig::validate, validate_smtp_host, cache_stale_duration_ms, deserialize_duration_seconds, resolve_timeout	#1436
   error_highlighting.rs quote_source_with_highlight	4	+→- and >→>= on the caret column arithmetic	todo
   content_security_policy.rs	4	is_enabled can return a constant either way	#1414
   http_client.rs default_system_root_ca_certificates_from_env	3	TLS trust-store selection	#1435
   csv_import.rs CopyCsvOption::{quote,header,null,escape}	7	COPY option parsing	todo
   server_timing.rs ServerTiming::header_value	4	/→%, %→+ in the duration formatting	todo
   filesystem.rs io_error_status + is_path_missing_error guards	10	mapping ErrorKind to a status code, and the missing-path guards in read_file, modified_since, file_exists	#1411
   file_cache.rs Cached::needs_check	1	<→<= on the staleness comparison	todo
   http_metrics.rs middleware call	1	>=→< on the status-class bucket	todo
   blob_to_data_url.rs detect_mime_type	2	magic-byte comparisons	#1415

   Found while writing the tests (not in the tables above)

   Change	PR
   migrations.rs unwraps the version of the failing migration. sqlx::migrate::MigrateError is #[non_exhaustive], so an added variant turns a "migration N failed" message into a server panic.	#1413
   OidcState::maybe_refresh was only pinned in the stale direction. A mutant that refreshes a snapshot that is still fresh survives, so every request could hammer the identity provider unnoticed.	#1425
   The documented facet example selects distinct over a column the rewriter computes, which SQLPage rejects.	#1420
   configuration.md lists a DuckDB DSN as a possible database_url default, but create_default_database only ever builds a sqlite:// URL.	#1421
   configuration.md lists on_connect.sql and on_rest.sql but no test coverage	#1437
2. lovasoa commented on Aug 22, 2026

   

   lovasoa

   on Aug 22, 2026

   Collaborator

   More actions

   Hi @81reap ! Did you review the results? Did you find anything that would be likely to cause problems? Before changing CI, I'd be keen to see actual interesting PRs that fix real issues this uncovered. 100% code coverage is not a goal, what I'd like to see is a realistic assessment of the risk for anything that is not covered. Most of the machine output above does not seem to have enough context to effectively prune the results.
3. lovasoa commented on Aug 22, 2026

   

   lovasoa

   on Aug 22, 2026

   Collaborator

   More actions

   Would you be ok with starting with small targeted PRs that add actually useful tests based on what you uncovered here?
4. lovasoa commented on Aug 22, 2026

   

   lovasoa

   on Aug 22, 2026

   Last edited by lovasoa

   Collaborator

   More actions

   Plus, looking at the output, I think your mutation testing exercised only the rust crate tests. I think a lot of what it reports is actually covered in end to end playwright tests (the best kind of tests), and in hurl tests from the./examples/ folder.
5. 81reap commented on Aug 22, 2026

   

   81reap

   on Aug 22, 2026

   Last edited by 81reap

   CollaboratorAuthor

   More actions

   hey @lovasoa! btw, are you the only maintainer? we seem to interact a lot lol

   Did you review the results? Did you find anything that would be likely to cause problems?

   yes. so I was able to use the results to deterministically identify the same issues I had discovered with claude in #1389

   100% code coverage is not a goal, what I'd like to see is a realistic assessment of the risk for anything that is not covered.

   interesting, then I may be following the same principals I do at work too much then. we have a goal to reach +95% test coverage in unit and integ tests as well as adding more layers of automated checks in order to be more confident that the changes do not have unintended side effects. I had assumed this would also be a benefit for an open source project.

   Personally, I also feel more confident in my changes when it goes through a swiss cheese layer of robust tests first. I'm not trying to imply there isn't this high bar set in SQLPage's codebase already, just that tools like mutation testing help push that high bar even further.

   Most of the machine output above does not seem to have enough context to effectively prune the results.

   yes, so the way I read these are code smells. there may be something there, or there may just be a missing test case it's trying to flag. There's a lot of strings to pull on here and the end of the rope isn't always "just add a test". for example #1389 were manual code smells I found, but the actual fix was #1394 . To me sharing the current set of reports helps string the PRs together into a bigger picture, rather than a flood of random OPS & OE changes to the codebase

   Would you be ok with starting with small targeted PRs that add actually useful tests based on what you uncovered here?

   yes, ofc!! this is why I created the report and issue to share :) I was hoping to also do the work to fix the gaps. I know there are a lot of changes here and not all of them may be applicable but from my understanding, the boon of mutation testing is to be able to find code smells that we can improve.

   Plus, looking at the output, I think your mutation testing exercised only the rust crate tests. I think a lot of what it reports is actually covered in end to end playwright tests (the best kind of tests), and in hurl tests from the./examples/ folder.

   ah, this is my first time setting up mutation testing from scratch myself and the results were already quite long so I missed that. I can run mutation testing again on the codebase once we knock out this batch of fixes, if that works with you?

   Prayag, why are you doing this?

   so my main goal is to still use SQLPage to power my machine dashboards. this is the main thing I care about as customer trying to use SQLPage and so far my concerns have been

   • stacked area graphs / stacked line graphs #1362 was fixed with fix(chart) :: align stacked series on their X values #1370 and fix(chart) :: line series up on a category axis for every chart type #1371
   • chart: reference lines for limits and for events #1374 will be fixed by feat(chart) :: draw horizontal reference lines #1375 , feat(chart) :: draw vertical reference lines #1376 , and feat(chart) :: turn a reference line into a band #1377

   But I also like to follow the principal of leaving a codebase better than I found it so I also raised these smaller fixes :: #1366 , #1367 , #1368 , #1369 , #1372 , and #1373 . IMO this also helps AI tooling perform better as they are susceptible to broken windows but for code quality.

   While making those changes I also discovered more issues like B3 from #1386 , C1 & C2 from #1387 , and sql issues in #1385 . That's why I raised these issues and am also breaking them up and raising PRs to address them .

   if there's a more github native way to do this (Stacked PRs, Project Planning, etc.) I'm happy to learn and use that instead. we don't use github at work and my personal usage thus far has been limited to personal projects or addressing small issues in open source ones
6. 81reap

   self-assigned this

   on Sep 24, 2026