From 3a6e47c9495fcada9ce902268f219bb141c34172 Mon Sep 17 00:00:00 2001 From: Leon Letournel Date: Mon, 14 Sep 2026 22:32:09 -0400 Subject: [PATCH 1/2] Dummy MDs --- courseProjectCode/Metrics/README.md | 15 +++++++++++++++ courseProjectDocs/project-proposal.md | 20 ++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 courseProjectCode/Metrics/README.md create mode 100644 courseProjectDocs/project-proposal.md diff --git a/courseProjectCode/Metrics/README.md b/courseProjectCode/Metrics/README.md new file mode 100644 index 0000000000..e0197c8433 --- /dev/null +++ b/courseProjectCode/Metrics/README.md @@ -0,0 +1,15 @@ +# Metrics + +This folder contains the code that collects the maintainability and testability metrics for the project proposal. + +## Requirements + +TODO: List the tools and versions. + +## How to Run + +TODO: Write the commands that reproduce the results in the report. + +## Output + +TODO: Tell where the results go and how to read them. diff --git a/courseProjectDocs/project-proposal.md b/courseProjectDocs/project-proposal.md new file mode 100644 index 0000000000..34d925b349 --- /dev/null +++ b/courseProjectDocs/project-proposal.md @@ -0,0 +1,20 @@ +# Project Proposal: HTTPie CLI + +## Team + +- Leon Letournel (itsleeyawn) +- Tenzin Dhondup (Txd5857) + +## Project Overview + +TODO: Describe the project, its purpose, and its main features. + +## Key Quality Metrics + +### Maintainability + +TODO: Name the metrics, the tool, and the results. + +### Testability + +TODO: Name the metrics, the tool, and the results. From 0e7823017355346218ab1dbaca0eb051ffb873ef Mon Sep 17 00:00:00 2001 From: Leon Letournel Date: Mon, 14 Sep 2026 23:00:46 -0400 Subject: [PATCH 2/2] Add readme and project proposal --- courseProjectCode/Metrics/README.md | 121 ++++++++++++++++++++++++-- courseProjectDocs/project-proposal.md | 115 ++++++++++++++++++++++-- 2 files changed, 222 insertions(+), 14 deletions(-) diff --git a/courseProjectCode/Metrics/README.md b/courseProjectCode/Metrics/README.md index e0197c8433..96bd824bd8 100644 --- a/courseProjectCode/Metrics/README.md +++ b/courseProjectCode/Metrics/README.md @@ -1,15 +1,124 @@ -# Metrics +# Metrics Collection -This folder contains the code that collects the maintainability and testability metrics for the project proposal. +Collects the two quality metrics for our course project — **Maintainability** +and **Testability** — from the forked project's Python source and writes them to +reproducible output files. Every number in our report comes from running this. ## Requirements -TODO: List the tools and versions. +- Python 3.9+ +- `radon` (static analysis: Maintainability Index, cyclomatic complexity, Halstead, raw LOC) -## How to Run +```bash +pip install radon +``` -TODO: Write the commands that reproduce the results in the report. +## Running it + +From the repository root: + +```bash +python courseProjectCode/Metrics/collect_metrics.py -o courseProjectCode/Metrics/output +``` + +Example, analyzing the package we forked: + +```bash +python courseProjectCode/Metrics/collect_metrics.py ./httpie -o courseProjectCode/Metrics/output +``` + +Test directories, docs, build artifacts, and virtualenvs are excluded by +default. Override with `--exclude`: + +```bash +python courseProjectCode/Metrics/collect_metrics.py ./src --exclude tests docs build +``` ## Output -TODO: Tell where the results go and how to read them. +| File | Contents | +| --- | --- | +| `metrics_per_file.csv` | One row per source file — all raw and derived metrics | +| `metrics_summary.json` | Project-level rollup, MI risk bands, and the ten least maintainable / least testable files | + +Both files are regenerated from scratch on every run, so the results in the +report are reproducible from a clean checkout. + +## Metric 1 — Maintainability + +Measured with the **Maintainability Index (MI)**, the standard composite of +Halstead volume, cyclomatic complexity, and lines of code, computed by `radon` +(`mi_visit`). Radon reports MI on a 0–100 scale and applies conventional bands: + +| MI | Interpretation | +| --- | --- | +| 100–65 | Maintainable | +| 65–20 | Moderate maintenance risk | +| 20–0 | High maintenance risk / difficult to maintain | + +The summary file reports the mean and median MI and counts how many files fall +below each threshold. Supporting values are also captured per file so the score +can be interpreted rather than taken on faith: SLOC, comment ratio, average and +maximum cyclomatic complexity, and Halstead volume and difficulty. + +## Metric 2 — Testability + +There is no single agreed-upon testability metric the way there is for +maintainability, so this tool reports the **structural properties known to drive +test effort** and combines them into an explicitly-defined heuristic score. +Treat the score as a ranking device for finding the hard-to-test parts of the +system, not as an absolute measurement. + +Four factors, each a form of work a test author has to absorb: + +| Factor | Column | Why it costs test effort | +| --- | --- | --- | +| Complexity | `avg_cyclomatic_complexity` | More independent paths to cover for the same behavior | +| Coupling | `fan_out_internal` + `fan_out_external` | Every collaborator must be constructed, stubbed, or mocked | +| Parameters | `avg_parameters` | Larger setup burden per test case | +| Public surface | `public_definitions` | More entry points that each need their own tests | + +Each factor is normalized to 0–1 against a saturation point, weighted, and +subtracted from 100: + +``` +testability = (1 - Σ wᵢ · min(xᵢ / sᵢ, 1)) × 100 +``` + +| Factor | Weight | Saturation point | +| --- | --- | --- | +| Complexity | 0.40 | 10 (the conventional "refactor this" threshold for CC) | +| Coupling | 0.30 | 20 imported modules | +| Parameters | 0.15 | 5 parameters | +| Public surface | 0.15 | 30 public functions/classes | + +**The weights and saturation points are our assumption, not a published +standard.** They are defined at the top of `collect_metrics.py` in the `WEIGHTS` +and `SATURATION` dictionaries and can be changed in one place; the report +justifies the choice and notes the sensitivity. + +Coupling is resolved properly rather than counted naively: imports are matched +against the project's own module index, so internal coupling (project modules) +and external coupling (third-party libraries) are reported separately, and +relative imports (`from . import x`) are resolved to real modules. `fan_in_internal` +records how many project modules import a given file, which identifies the +high-blast-radius modules where a regression is most expensive. + +### Known limitations + +- Static analysis only — it does not measure whether existing tests are *good*. + Mutation testing (e.g. `mutmut`) or coverage would complement this and is a + reasonable extension for a later deliverable. +- Dynamic constructs (reflection, `importlib`, monkeypatching) are invisible to + the AST walk, so coupling is a lower bound. +- Files that fail to parse are skipped and reported on stderr. + +## Reproducing the report numbers + +```bash +pip install radon +python courseProjectCode/Metrics/collect_metrics.py -o courseProjectCode/Metrics/output +``` + +Then read `metrics_summary.json` for the project-level figures quoted in the +report and `metrics_per_file.csv` for the per-file tables. diff --git a/courseProjectDocs/project-proposal.md b/courseProjectDocs/project-proposal.md index 34d925b349..3f9d88d128 100644 --- a/courseProjectDocs/project-proposal.md +++ b/courseProjectDocs/project-proposal.md @@ -1,20 +1,119 @@ -# Project Proposal: HTTPie CLI +# Project Proposal — HTTPie CLI -## Team - -- Leon Letournel (itsleeyawn) -- Tenzin Dhondup (Txd5857) +**Course:** SWEN-777 Software Quality Assurance +**Team:** Leon, Tenzin, [third member] +**Upstream project:** HTTPie CLI — https://github.com/httpie/cli +**Fork:** [our fork URL] +**Baseline analyzed:** v3.2.4 ## Project Overview -TODO: Describe the project, its purpose, and its main features. +HTTPie is a command-line HTTP client written in Python. It wraps the `requests` +library behind a syntax designed to be readable by humans rather than by shell +scripts, and adds colorized and formatted output, persistent sessions, +`wget`-style downloads, and a plugin system for auth and transport. The `http` +and `https` commands are the primary entry points. + +We picked it for three reasons. + +It is a real production tool with a real user base, not a toy repository, so the +quality problems we find are problems that affect people. It is also small +enough to reason about: our analysis covers 78 source files and 7,039 logical +lines of code, which a three-person team can actually read in a semester rather +than sample from. + +Second, it has a substantial existing test suite — 409 test functions across 37 +test modules — built on pytest with a local `pytest-httpbin` server. That gives +us something to analyze for oracle quality in the next deliverable instead of +starting from zero coverage. + +Third, its architecture creates a natural testability gradient. The CLI parsing, +HTTP client, and output formatting layers are separated, but the entry point +(`core.py`) coordinates all of them, so we expect a small number of heavily +coupled modules surrounded by many easily testable ones. That is a useful shape +for a quality study because it gives us both ends of the spectrum in one +codebase. ## Key Quality Metrics +We are measuring **maintainability** and **testability**. Collection code and +reproduction instructions are in `courseProjectCode/Metrics/`. + ### Maintainability -TODO: Name the metrics, the tool, and the results. +Measured with the Maintainability Index (MI), the standard composite of Halstead +volume, cyclomatic complexity, and lines of code, computed with `radon`. MI is +reported on a 0–100 scale with conventional bands: above 65 is maintainable, +20–65 is moderate risk, below 20 is high risk. + +Baseline results: + +| Measure | Value | +| --- | --- | +| Files analyzed | 78 | +| Total SLOC | 7,039 | +| Mean MI | 74.04 | +| Median MI | 73.48 | +| Files below MI 65 (moderate risk) | 31 | +| Files below MI 20 (high risk) | 0 | +| Mean cyclomatic complexity per unit | 2.38 | +| Highest cyclomatic complexity in a single unit | 27 (`core.py`) | + +The project is healthy on average but not uniformly. Nothing is in the high-risk +band, yet 40% of files sit in the moderate band, and the distribution is skewed: +`cli/argparser.py` (MI 24.67) and `cli/options.py` (MI 38.28) are far worse than +the mean. Argument parsing is doing a large amount of work in one place. ### Testability -TODO: Name the metrics, the tool, and the results. +There is no single accepted testability metric, so we report the structural +properties that drive test effort and combine them into a defined heuristic +score: average cyclomatic complexity (paths to cover), coupling via resolved +internal and external imports (collaborators to stub or construct), average +parameter count (setup cost per case), and public definition count (entry points +needing their own tests). Each is normalized against a saturation point, +weighted, and subtracted from 100. The weights are our assumption rather than a +published standard; they are declared in one place in the collector and the +sensitivity is something we intend to discuss rather than hide. + +Baseline results: + +| Measure | Value | +| --- | --- | +| Mean testability score | 75.87 | +| Least testable | `core.py` (20.36) | +| | `client.py` (36.67) | +| | `cli/argparser.py` (42.53) | +| | `output/writer.py` (46.17) | + +`core.py` is the clearest finding of the baseline. It scores 20.36 while the +project averages 75.87, it has the highest cyclomatic complexity in the codebase +(27), and it imports 15 internal modules plus 8 third-party ones. It is the +program entry point, so every end-to-end path passes through it, which means it +is simultaneously the hardest unit to isolate and the one most worth isolating. + +`cli/argparser.py` is the only module that lands in the bottom five on both +metrics, which makes it our primary candidate for deeper analysis. + +### Why these two metrics together + +They answer different questions and disagree in informative ways. MI asks how +hard code is to change; our testability score asks how hard it is to verify. +`output/writer.py` is acceptable on MI but poor on testability, because its +problem is coupling rather than internal complexity — a distinction MI alone +would hide. Tracking both lets us argue about which refactorings would actually +improve verifiability rather than just tidy the code. + +## Planned Direction + +With the baseline established, we intend to extract requirements and analyze the +existing test oracles, establish a baseline build and coverage run, and then +focus deeper testing work on the modules this baseline identified as weakest — +`core.py` and `cli/argparser.py` in particular. + +## Note on Reproducibility + +The figures above come from HTTPie v3.2.4. Our fork tracks `master`, so rerunning +the collector against the fork will shift the numbers slightly. All reported +results will be regenerated from the fork before the final report, using the +single command documented in `courseProjectCode/Metrics/README.md`.