mutator

Overview

mutator is an automated mutation testing tool for the R language. It applies mutation testing principles to help developers improve test suite quality by introducing small, systematic changes (mutations) to source code and verifying if tests can detect these changes.

Features

• Comprehensive Mutation Testing: Applies various mutation operators to R source code
• AST-Based Mutation: Uses Abstract Syntax Tree analysis for intelligent code mutations
• Parallel Test Execution: Runs tests in parallel for improved performance
• Coverage-guided Test Selection: Runs only the tests that cover mutated lines, also for improved performance
• Equivalent Mutant Detection: Uses AI to identify mutants that are functionally equivalent to the original code
• Detailed Reporting: Provides mutation scores and analysis of test suite effectiveness

Installation

# Install from GitHub; package dependencies are installed automatically
# install.packages("remotes")
remotes::install_github("PRL-PRG/mutator")

# Or install from local source, in an R console
# install.packages("devtools")
setwd("path/to/mutator")
devtools::install()

Quick Start Guide

library(mutator)

# Mutate a single file
mutants <- mutate_file("path/to/your/file.R")

# Optional: cap returned mutants by random selection
mutants <- mutate_file("path/to/your/file.R", max_mutants = 20)

# Mutate an entire package and run tests
result <- mutate_package("path/to/your/package")

# Optional: cap tested mutants across the whole package
result <- mutate_package("path/to/your/package", max_mutants = 100)

# Optional: set a fixed timeout (seconds) per mutant test run
result <- mutate_package("path/to/your/package", timeout_seconds = 60)

# Optional: control where mutant files are written
result <- mutate_package("path/to/your/package", mutation_dir = tempdir())

See the pkgdown reference for the full argument and return-value documentation.

Mutation testing modes

mutator selects a package test strategy automatically:

• If tests/testthat/ exists, mutator loads the mutant in-process with pkgload::load_all() and mirrors the package's own tests/testthat.R harness by forwarding extractable arguments (notably any filter) from testthat::test_check() to testthat::test_dir(), without paying for an install per mutant.
• Otherwise, if tests/ exists, mutator falls back to tools::testInstalledPackage(..., types = "tests") after installing each mutant with --install-tests.

The fallback path supports non-testthat layouts (for example tinytest-driven packages that run through tests/ scripts). To avoid recompiling C/C++ on every mutant, mutator installs the unmutated package, compiling its shared objects, once into a template library, then installs each mutant with --no-libs (R code only) and restores the template's prebuilt shared objects before running its tests. This is safe because mutator does not mutate compiled code, so the shared object is identical for every mutant; it also means concurrent mutant installs no longer write into a shared src/ (see Parallel execution and isolation in the Configuration article).

Each mutant test run uses a timeout. By default, mutator calibrates it from the baseline suite runtime under the current parallelism, with a small floor. You can override this by setting timeout_seconds explicitly.

Mutant outcomes are reported as:

• SURVIVED: tests passed for the mutant
• KILLED: tests failed (or execution error)
• HANG: mutant exceeded timeout

Configuration

mutate_package() exposes a number of options to control how mutants are run, which tests are selected, and how results are refined. Each is covered in depth in the Configuration article:

• Timeouts and the contended baseline — how the per-mutant HANG timeout is self-calibrated from a parallelism-aware baseline, and timeout_seconds to override it.
• CRAN mode (cran) — run the tests CRAN would (skipping guarded slow/network tests) or the full suite.
• Fail-fast (fail_fast) — stop each mutant's run at the first failing test.
• Parallel execution and isolation (isolate, cores) — symlink-vs-copy of the package tree and how to handle non-hermetic tests, plus the optional pbmcapply progress bar.
• Excluding code from mutation — exclude_files, in-source # mutator:ignore-* directives, covr # nocov annotations, and .covrignore.
• Coverage-guided test selection (coverage_guided, coverage_backend) — run only the tests that cover each mutated line.
• Precise mutant locations — the optional imputesrcref package for narrower operator-mutant source ranges.
• Equivalent mutant detection (detectEqMutants) — configuring the OpenAI-compatible API used to flag equivalent mutants.

Mutation Operators

mutator currently generates these mutation families:

Family	Mutations
Arithmetic operators	+ ↔ -, * ↔ /
Comparison operators	== ↔ !=, < ↔ >, <= ↔ >=
Logical operators	& ↔ |, && ↔ ||, removes !, and negates if / while conditions
Assignment and call values	Replaces assignment right-hand sides and ordinary function calls with 42
Scalar constants	Replaces numeric zero with 42, numeric non-zero values with 0, constants with a typed NA, and constants with NULL
Returns	Replaces non-constant direct return() values with NULL, for example return(x) → return(NULL)
Deletions	Deletes statements inside { ... } blocks and, as a fallback, valid source lines

Direct literal return values are not rewritten by the return-to-NULL mutation; for example, return(1) is left alone by that mutation.

Dependencies

mutator depends on:

• R Packages:
  • pkgload: For loading mutated package copies
  • testthat: For test execution
  • xml2: For running C++ tests through testthat::run_cpp_tests()
  • covr: For coverage-guided test selection (coverage_guided = TRUE)
  • future and furrr: For parallel execution
  • callr: For subprocess test execution and hard timeouts
  • R6: For the per_file coverage backend reporter
  • httr and jsonlite: For OpenAI API integration
• LinkingTo: testthat (for Catch2 C++ test headers)
• C++17: For native mutation engine implementation

mutator's test suite

mutator itself uses testthat for its own R tests and testthat + Catch2 for C++ tests.

• C++ tests are located in src/test-*.cpp
• The C++ test runner is src/test-runner.cpp
• C++ tests are executed from tests/testthat/test-cpp.R via run_cpp_tests("mutator")

Run the full test suite with:

devtools::test()