Skip to content

targeted documentation pages and trimmed readme #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anacarolinaleote merged 1 commit into from
Oct 23, 2026
Merged

targeted documentation pages and trimmed readme #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Builds the docs on every pull request, so a broken page fails there, and
# publishes them to GitHub Pages on every merge to main.
name: docs

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: '3.12'
- run: pip install -r docs/requirements.txt
- run: sphinx-build -W --keep-going docs _site
- if: github.event_name == 'push'
uses: actions/upload-pages-artifact@v5
with:
path: _site

deploy:
if: github.event_name == 'push'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
concurrency:
group: pages
cancel-in-progress: false
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v5
114 changes: 11 additions & 103 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,38 +11,28 @@
- **Automated archetype annotation** — contrastive image analysis and standardized reports generated with multimodal language models and user review.
- **Morphological program definition** — integration of co-regulated archetypes into interpretable programs of coordinated cellular change.

## Requirements
## Documentation

Most of CHART is Python and arrives with the package. The archetype
analysis is the exception: every step of it but the first and last is an
R script that `chart archetypes` runs through `Rscript`, so that stage
needs R and eight R packages. `environment.yml` describes them:
The [documentation](https://genentech.github.io/CHART/) describes the input data, each stage's usage, outputs and method, the configuration, and running on a cluster.

## Installation

```
mamba env create -f environment.yml
conda activate chart-r
Rscript -e 'install.packages("archetypes", repos="https://cloud.r-project.org")'
pip install .
```

The archetype annotation is optional. It needs two more packages and a key
for the language model, read from the environment:
The archetype analysis also needs R and the packages in `environment.yml`:

```
pip install ".[annotation]"
export OPENAI_API_KEY="sk-..."
mamba env create -f environment.yml
conda activate chart-r
Rscript -e 'install.packages("archetypes", repos="https://cloud.r-project.org")'
```

`src/chart/annotation/llm.py` lists the gateway and Azure alternatives.
The optional annotation stage needs `pip install ".[annotation]"` and credentials for a language model in the environment. [Installation](https://genentech.github.io/CHART/installation.html) covers these, scallops for reading images, and remote storage.

## Quickstart

Install the package, plus the environment above if you want the archetype
analysis:

```
pip install .
```

Write a config saying where the input is and where output should go:

```yaml
Expand All @@ -67,86 +57,4 @@ chart effects --config my_screen.yaml --all-channels
chart programs --config my_screen.yaml
```

`chart effects` tests each gene against the controls on every archetype
of the weights `chart archetypes` left, and writes one table per channel
to `effects/` under `data_output_dir`. Once every channel in
`archetypes.channels` has its table, it corrects all their p-values
together and writes them, with q-values, to `effects/pooled.parquet`.

`chart programs` finds programs, sets of archetypes that genes move
together, by sparse PCA on the genes and archetypes in
`effects/pooled.parquet` with any q-value below 0.01. It writes to
`programs/` under `data_output_dir`: each program's loadings and each
gene's scores; the genes' modules, from how often they cluster together
across 150 refits on part of them, which takes about 25 minutes; and every
cell's scores, with 20 example cells per program. `--steps` runs only some
of `matrix`, `fit`, `modules` and `cells`. `examples/programs.ipynb` shows
what each table holds and how to read them; Jupyter is not installed with
CHART.

The last step of `chart archetypes`, `crops`, draws each archetype's most
heavily weighted cells from the images, five to a PNG, under
`archetypes/<channel>/crops/` in `local_output_dir`. It reads the images
through scallops, from the locations under `preprocessing_bywell`. A
channel that `archetypes.image_channels` does not list, or a machine
without scallops, gets a warning and no crops, or an error if the step
was asked for with `--steps crops`; `example_config.yaml` lists its
settings.

`chart annotation` describes each archetype from those crops with a
language model, leaving out the archetypes `archetypes_to_remove` lists,
and writes the reports and a dashboard to `annotation/<channel>/` under
`local_output_dir`. Nothing downstream reads them.

Wells are read off the input filenames. Name wells
individually with `--well` and channels with `--channel`, and use
`--steps` on every stage but `chart effects` to run only part of a pipeline.
`chart archetypes --dry-run` prints every command the stage would run
without running any of them.

`chart archetypes` runs one channel at a time.
`archetypes.parallel_channels`, or `--parallel-channels`, runs several
at once, each in its own process and logging to
`archetypes/<channel>/` under `local_output_dir`; the screen says when
each starts and ends. Memory sets how many fit, not cores: the metacells
step needed about 62 GB for one channel of 875,000 cells, and uses one
core. The other stages use every core the run may use, which under SLURM
is the job's share. `n_jobs` at the top of the config, or `--n-jobs`,
sets a number instead.

## Configuration

Every step reads one YAML file, given with `--config`:

```
chart preprocessing --config my_screen.yaml --all-wells
```

Use `example_config.yaml` as a starting point: copy it and edit the paths. Every key in it except the two output directories is optional, and the values shown are the defaults.

The `schema` block maps CHART variables to the input screen, accounting for differences in input column names and feature naming conventions:

```yaml
schema:
cell_column: object_number
guide_column: sgrna_id
gene_column: target_gene
feature_patterns: ['^Nuclei_', '^Cells_', '^Cytoplasm_']
control_gene: safe_harbour
control_prefix: null
centroid_columns: [Nuclei_Location_Center_Y, Nuclei_Location_Center_X]
```

Every column in `feature_patterns` must be numeric.

Input filenames are configurable in the same file, under
`preprocessing_bywell`. Only `.parquet` input files are accepted.

```yaml
preprocessing_bywell:
objects_pattern: 'plate1_{well}.objects.parquet'
features_pattern: 'plate1_{well}.features.parquet'
```

When one file per well holds objects and features together, set
`premerged: true` and name it with `merged_pattern` instead.
Before `chart effects`, inspect each archetype's crops under `archetypes/<channel>/crops/` in `local_output_dir`, and list those reflecting segmentation errors or staining artefacts under `archetypes.archetypes_to_remove`, as described in [Curating archetypes](https://genentech.github.io/CHART/archetypes.html#curating-archetypes). `example_config.yaml` lists every setting, and [Configuration](https://genentech.github.io/CHART/configuration.html) explains which matter most.
85 changes: 85 additions & 0 deletions docs/annotation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Annotation

`chart annotation` describes each archetype from its crops with a multimodal language model. For every archetype it writes a morphological summary, three proposed names and a functional hypothesis, and collects them with the images in one HTML dashboard per channel. The stage is optional: nothing downstream reads its output.

It reads the grids drawn by the `crops` step of `chart archetypes`, leaving out the archetypes listed under `archetypes.archetypes_to_remove`. The packages and credentials it needs are described under [Installation](installation.md).

## Usage

Each channel needs one or more normal archetypes, chosen from the crops, against which every archetype is compared:

```yaml
annotation:
normal_archetypes:
TOM20: [10]
FAK: [5, 8, 11]
```

```
chart annotation --config my_screen.yaml --all-channels
chart annotation --config my_screen.yaml --channel TOM20 --steps annotate baseline
```

| Step | Output |
|---|---|
| `annotate` | An observational description of each of the archetype's first two grids |
| `baseline` | The archetype's first grid compared with the normal archetypes' |
| `synthesize` | A summary from the first grid's description and the baseline note; three names and a functional hypothesis |
| `refine` | Summaries that read alike, contrasted pairwise and rewritten |
| `dashboard` | The final reports beside their grids, in one HTML file |

The steps run in order, each reading the previous step's output. `--resume` keeps the descriptions `annotate` has already written, so an interrupted run continues where it stopped. `--continue-on-error` carries on with the remaining channels after a failure. `dashboard` on its own needs no model or credentials.

> [!NOTE]
> Calls are made one at a time. For a channel with $K$ archetypes, `annotate` makes up to $2K$ image calls, `baseline` $K$, `synthesize` $2K$ text calls, and `refine` two image calls per similar pair and one rewrite per archetype in a pair. `refine` compares summaries with the sentence-transformers model `all-MiniLM-L6-v2`, downloaded from Hugging Face on first use; on a machine without internet access, download it beforehand.

## Outputs

All files are written to `annotation/<channel>/` under `local_output_dir`. Archetype numbers are those of the crops, `<channel>_Archetype<N>`.

| File | Contents |
|---|---|
| `archetype_annot/archetype<N>_grid_<g>_annotation.md` | Description of one grid |
| `contrastive_baselines/contrastive_baseline_notes.json` | Each archetype against the normal archetypes; `contrastive_baseline_report.md` holds the same notes |
| `archetype_reviews/archetype<N>_synthesis.md` | Summary, names and hypothesis |
| `archetype_reviews/archetype<N>_contrast_vs<M>.md` | Traits distinguishing archetype N from M |
| `archetype_reviews/archetype<N>_final.md` | Final summary, names, hypothesis, and the archetypes it was contrasted with |
| `archetype_reviews/combined_archetype_synthesis.md`, `combined_archetype_final.md` | Every archetype's report in one file |
| `archetype_dashboard.html` | The final reports with their grids embedded, readable anywhere |

## Configuration

```yaml
annotation:
reports_dir: annotation/ # under local_output_dir
normal_archetypes: {} # per channel, by archetype number
model: gpt-5.5
similarity_threshold: 0.08 # cosine distance below which summaries are contrasted
max_pairs_per_archetype: 0 # closest partners kept per archetype; 0 keeps all
```

The values shown are the defaults. `normal_archetypes` has to be set for every channel before `baseline` runs. The stage also reads `archetypes.channels`, `archetypes.archetypes_to_remove`, and `archetypes.reports_dir`, under which the crops are.

## Method

**Grids.** The crops step draws each archetype's most heavily weighted cells, `archetypes.top_cells` (10 by default), five to a grid: the stain in inverted grey, the cell outlined in red and its nucleus in blue. The first two grids of each archetype are described; only the first enters the later steps.

**Observation.** The descriptions, the baseline comparisons and the pairwise contrasts are restricted to visible features: intensity, localisation relative to the two outlines, texture, cell and nuclear size and shape, and artefacts. The model is instructed not to infer biology or to assume the stain's usual pattern. The only interpretive call is the one in `synthesize` that proposes names and a hypothesis, from the first grid's description.

**Baseline.** Each archetype's first grid is sent together with the normal archetypes' first grids in one request, at reduced resolution (longest edge at most 1024 pixels).

**Refinement.** The summaries are embedded with `all-MiniLM-L6-v2`, and each pair closer than `similarity_threshold` in cosine distance is contrasted in both directions, each call showing both first grids. Every archetype in a pair then has its summary rewritten to include what distinguishes it. With `max_pairs_per_archetype` above 0, only each archetype's closest partners are kept. Archetypes in no pair keep their synthesis summary.

**Validation.** Every reply must contain the sections the next step reads; a reply without them is requested once more, then the step fails. Empty and truncated replies are errors, and rate-limited calls are retried with increasing delays. A step therefore either writes every archetype's report or stops.

## Troubleshooting

| Message | Cause and remedy |
|---|---|
| `No top_cells.parquet in ...; run the crops step of chart archetypes for ...` | The channel has no crops; run `chart archetypes --steps crops` for it |
| `top_cells.parquet lists archetype ..., but ... is not in ...; rerun the crops step` | The grid images are incomplete; rerun the crops step |
| `'annotation.normal_archetypes' has to name archetypes of ... that the crops step drew and 'archetypes.archetypes_to_remove' keeps` | No normal archetype is set for the channel, or one named was removed or not drawn |
| `Missing langchain_openai, sentence_transformers. ...` | The annotation extra is not installed |
| `No LLM provider is configured ...` | No credentials in the environment; see [Installation](installation.md) |
| `... model returned no text ...`, `... reply was cut off at the token ceiling ...` | The model spent its token budget; rerun the step, with `--resume` for `annotate` |
| `... reply is missing required section header(s) ...` | The model twice ignored the output format; rerun the step |
Loading
Loading