Feature/notebook 03 carbon analysis#64
Open
Goldokpa wants to merge 53 commits into
Open
Conversation
- Expanded config.yaml with per-analysis-type configuration for deforestation, ice melting, and flooding including band configs, alert thresholds, and model paths - Added config/train.yaml for production training configuration - Expanded db.py with full SQLite schema: organisations, subscriptions, alerts tables; API key generation; all CRUD operations - Added requirements-install.txt for streamlined dependency installation Co-authored-by: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-authored-by: John Edoh Onuh <onuhj47@gmail.com> Co-authored-by: Francis Umo <Francisumoh@360yahoo.com> Co-authored-by: Olufemi Taiwo <olufemitaiwo23@gmail.com> Co-authored-by: Godswill Chukwu Okoroafor <godswillchukwu21@gmail.com>
- Added inference/pipeline.py: full GEE-integrated inference engine with NDVI computation, model loading, file and bbox inference paths, synthetic NDVI fallback with bbox-seeded reproducibility - Updated inference/__init__.py to export run_inference, run_inference_from_file, run_inference_from_gee - Added analysis/ module: base class, registry, and dedicated analysers for deforestation, flooding and ice melting detection - Added training/ module: production trainer with EMA, checkpointing, early stopping, and combined loss functions (BCE + Dice + Focal) - Updated models/unet.py with minor architecture improvements - Updated __init__.py package exports Co-authored-by: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-authored-by: Francis Umo <Francisumoh@360yahoo.com> Co-authored-by: Godswill Chukwu Okoroafor <godswillchukwu21@gmail.com> Co-authored-by: Victor Mbachu <victor.c.mbachu@gmail.com>
- Expanded api/main.py with full production API: organisation and NGO management, subscription system, alert and notification endpoints, all three analysis types wired to inference pipeline, run history, file upload endpoint, health check and API key authentication - Added run_api.sh: server startup script with venv activation, environment setup and uvicorn hot-reload configuration - Added docs/API_REFERENCE.md: full endpoint reference with request and response schemas for all routes Co-authored-by: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-authored-by: John Edoh Onuh <onuhj47@gmail.com> Co-authored-by: Olufemi Taiwo <olufemitaiwo23@gmail.com> Co-authored-by: Victor Mbachu <victor.c.mbachu@gmail.com> Co-authored-by: Godswill Chukwu Okoroafor <godswillchukwu21@gmail.com>
- Fixed repository clone URL to Climate-Vision/ClimateVision - Updated Quick Start to use run_api.sh instead of raw uvicorn command - Corrected tech stack: SQLite (not PostgreSQL), Google Maps API (not Leaflet) - Fixed API Reference doc link to docs/API_REFERENCE.md - Updated Phase 3 roadmap to reflect Google Maps and Recharts as completed - Fixed Star History tracking link Co-authored-by: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-authored-by: John Edoh Onuh <onuhj47@gmail.com> Co-authored-by: Francis Umo <Francisumoh@360yahoo.com> Co-authored-by: Olufemi Taiwo <olufemitaiwo23@gmail.com> Co-authored-by: Godswill Chukwu Okoroafor <godswillchukwu21@gmail.com> Co-authored-by: Victor Mbachu <victor.c.mbachu@gmail.com> Co-authored-by: Paul <46930375+cutewizzy11@users.noreply.github.com>
…ipts - prepare_data.py: GEE + synthetic Sentinel-2 patch downloader with Dynamic World forest labels, train/val/test split, normalizer fitting - train.py: production Attention U-Net training entry-point with YAML config, focal+dice loss, EMA weights, cosine LR schedule, early stopping - run_training.py: end-to-end training + inference pipeline - evaluate.py: per-class IoU/F1/precision/recall on held-out test set - export_model.py: ONNX and TorchScript model export - infer.py: CLI inference runner for single images or GEE bbox Co-Authored-By: Emmanuel Edoh <edoh-Onuh@users.noreply.github.com> Co-Authored-By: Godswill Okoroafor <godswillchukwu21@gmail.com> Co-Authored-By: Gold Okpa <okpagold@gmail.com> Co-Authored-By: Victor Mbachu <victor.c.mbachu@gmail.com>
- pipeline.py: authenticate GEE via service account key when GEE_SERVICE_ACCOUNT and GEE_SERVICE_ACCOUNT_KEY env vars are set; falls back to synthetic NDVI when GEE is unavailable instead of zeros - .gitignore: protect secrets/ directory and *.json key files Co-Authored-By: Gold Okpa <okpagold@gmail.com>
Notebook handles: GEE service account auth, multi-region patch download (Amazon/Congo/Borneo), Attention U-Net training on T4 GPU, evaluation, and checkpoint download back to local machine. Co-Authored-By: Gold Okpa <okpagold@gmail.com>
- prepare_data.py: reads GEE_SERVICE_ACCOUNT / GEE_SERVICE_ACCOUNT_KEY env vars to authenticate via service account instead of requiring earthengine authenticate - notebook: sets env vars with absolute key path in Cell 3 so all subprocess calls in Cells 5 and 6 inherit them automatically Co-Authored-By: Gold Okpa <okpagold@gmail.com>
Split each region into 0.5° tiles at 30m resolution instead of downloading the whole bbox at 10m (which hit GEE's pixel grid cap). Each tile is ~1850x1850px — well under the 32768 limit. Patches are accumulated across tiles until max_patches is reached. Co-Authored-By: Gold Okpa <okpagold@gmail.com>
…E limit Previous 30m/0.5° tiles were ~130MB each, exceeding GEE's 48MB cap. At 100m resolution each 0.25° tile is ~1.5MB — well within limits. Also fixes NameError on profile when all tiles failed, and adds a clear error exit when no patches are extracted. Co-Authored-By: Gold Okpa <okpagold@gmail.com>
… config - App.tsx: main application shell with routing, global state and sidebar navigation between Dashboard, Analysis, NGO and Settings - api.ts: typed API client for all backend endpoints (predict, runs, organizations, alerts, analysis-types) with error handling - types.ts: shared TypeScript interfaces for Run, Organization, Alert, NDVIStats, InferenceResult and API responses - styles.css: design-system CSS variables (cv-* tokens), component base styles, skeleton loader, scrollbar and animation utilities - tailwind.config.js: extended theme with cv-* color palette, shadow tokens, and custom font stack matching the dark forest UI - main.tsx: React 18 createRoot entry-point with StrictMode - index.html: updated meta tags, font preload and app title - package.json: added lucide-react, recharts, react-router-dom deps - .env.example: documents VITE_GOOGLE_MAPS_API_KEY and VITE_API_BASE_URL Co-Authored-By: Emmanuel Edoh <edoh-Onuh@users.noreply.github.com> Co-Authored-By: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-Authored-By: Gold Okpa <okpagold@gmail.com> Co-Authored-By: Victor Mbachu <victor.c.mbachu@gmail.com>
- Validate bbox has exactly 4 values [west, south, east, north] - Enforce longitude bounds (-180 to 180) and latitude bounds (-90 to 90) - Ensure west < east and south < north - Validate date strings follow YYYY-MM-DD format - Ensure start_date is earlier than end_date
- Add offset query parameter for cursor-based pagination
- Return total record count alongside results for frontend page controls
- Restructure response to {total, limit, offset, runs} envelope
- Refactor WHERE clause building to avoid SQL injection via safe parameterisation
- Returns total run count, completed runs in last 7 days - Breakdown by status (pending, running, completed, failed) - Breakdown by analysis type (deforestation, ice_melting, flooding) - Alert summary: total alerts and unacknowledged count - Feeds directly into the frontend Dashboard KPI summary cards
- Log every request: method, path, status code, duration_ms, client IP - Attach X-Response-Time-Ms header to all responses for frontend monitoring - Uses Starlette BaseHTTPMiddleware for non-blocking request interception - Helps trace slow endpoints and detect unusual access patterns in production
- Reduce from 874 lines to ~100 lines (~5000 words to 596 words) - Move installation to top (line 18) - visible without scrolling - Replace imaginary API examples with real working curl + uvicorn commands - Replace fabricated benchmarks with honest in-progress markers - Remove community growth strategy, team descriptions, and execution plan - Add satellite band details to analysis types table - Keep citation, contributing, and docs links
…ments feat(api): Olufemi - API validation, pagination, stats & audit logging
- Add React components: Map, Charts, Layout, UI elements - Add contexts: AppContext, ToastContext - Add hooks: useGeocoding, useRunPolling - Add pages: Analytics, NewAnalysis, RunHistory, Settings, Upload - Update SETUP_COMPLETE.md Co-authored-by: Adeolu Mary Oshadare <nifemi996@gmail.com> Co-authored-by: John Edoh Onuh <onuhj47@gmail.com> Co-authored-by: Francis Umo <Francisumoh@360yahoo.com> Co-authored-by: Olufemi Taiwo <olufemitaiwo23@gmail.com> Co-authored-by: Godswill Chukwu Okoroafor <godswillchukwu21@gmail.com> Co-authored-by: Victor Mbachu <victor.c.mbachu@gmail.com> Co-authored-by: Paul <46930375+cutewizzy11@users.noreply.github.com> Co-authored-by: Gold Okpa <okpagold@gmail.com>
Prevent accidental commits of large .pth model files that exceed GitHub's 100MB limit.
Add centralized constants for API config, map settings, analysis types, polling intervals, and UI configurations.
Add helper utilities for date formatting, number formatting, area calculations, debouncing, and value clamping.
Add comprehensive validation utilities for the data pipeline: - Image shape validation (channels, dimensions) - Mask shape validation - Value range checking - NaN detection - Dataset split verification
Add quality metrics computation for satellite imagery: - Cloud coverage estimation - Missing data detection - Class distribution analysis - Quality score calculation - Dataset quality reports
Update data package exports to include: - DataValidationError and validation functions - QualityReport and quality assessment utilities Enables: from climatevision.data import validate_sample, assess_dataset_quality
feat(data): add sampling and transform utilities - Add stratified split, balanced sampling, k-fold CV - Add custom satellite imagery transforms - Add cloud masking and normalization utilities
Add comprehensive type validation schemas including: - BoundingBox and PredictionRequest for input validation - OrganizationCreate and SubscriptionCreate for entity creation - Response schemas for health, predictions, alerts, and errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add postprocess.py with confidence thresholding, small region removal, and anomaly detection for inference outputs. Includes quality scoring to flag predictions that may need review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add CarbonEstimator class with: - Allometric equations for biomass estimation by forest type - IPCC-based biomass-to-carbon conversion - Monte Carlo uncertainty quantification with confidence intervals - Regional adjustment factors for Amazon, Congo, Southeast Asia Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GroundTruthValidator with: - IoU, F1, precision, recall, accuracy, and Kappa metrics - Per-class IoU computation for multi-class segmentation - Automated recommendations based on metric thresholds - Model version comparison utility for A/B testing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…schemas and inference postprocessing
…rbon analytics and validation
Implements structured logging middleware with request tracing and audit trails for compliance monitoring. Includes: - RequestLoggingMiddleware: logs all requests with timing - AuditLogMiddleware: tracks sensitive operations - X-Request-ID headers for distributed tracing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements secure API key authentication for protected endpoints: - APIKeyAuth class for key generation and validation - Secure key hashing using SHA-256 - Organization context extraction for access control - FastAPI dependencies for required/optional auth - Comprehensive audit logging for auth events Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements comprehensive statistical analysis capabilities: - Two-sample t-test (Welch's) for model comparison - Mann-Whitney U test for non-parametric analysis - Linear trend analysis for time series - A/B testing framework for model version comparison - Bootstrap confidence intervals for uncertainty estimation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements automated report generation for stakeholders: - RegionalMetrics dataclass for environmental KPIs - ImpactReport with carbon, validation, and recommendations - ReportGenerator for JSON and HTML report output - Trend analysis integration for year-over-year comparisons - Actionable recommendations based on threshold analysis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updates module __init__.py to expose complete public API: - Carbon estimation: CarbonEstimator, estimate_carbon - Validation: GroundTruthValidator, validate_predictions - Statistics: t_test, mann_whitney, trend_analysis, ab_test - Reporting: ReportGenerator, generate_report Enables clean imports like: from climatevision.analytics import estimate_carbon, generate_report Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merging Olufemi's API middleware and auth modules
Merging Francis's analytics statistics and reporting modules
* feat(data): add GEE tile downloader with analysis-aware band selection - Downloads real Sentinel-2 composites via Google Earth Engine - Reads required bands from config.yaml per analysis_type - Includes SCL band for downstream cloud masking - Synthetic fallback with explicit is_synthetic flag when GEE unavailable - Fix .gitignore so src/climatevision/data/ is no longer ignored * feat(data): add analysis-specific Sentinel-2 band mapping utilities - get_bands_for_analysis() reads correct bands from config.yaml - get_band_indices() maps band names to canonical 13-band stack positions - is_analysis_enabled() and list_enabled_analysis_types() for config validation - Includes SCL band helpers for downstream cloud masking * feat(data): integrate SCL cloud masking and export new pipeline modules - apply_scl_cloud_mask() masks cloudy pixels using Sentinel-2 SCL band - Default clear labels: vegetation, bare soils, water, snow - Update __init__.py to expose gee_downloader and band_mapping utilities * refactor(data): address PR review feedback - Remove duplicated config logic in gee_downloader.py; import from band_mapping - Cache config.yaml load in band_mapping.py via lru_cache - Read synthetic tile size from config.yaml instead of hardcoding 256 - Remove unused json import in gee_downloader.py - Add shape validation in apply_scl_cloud_mask --------- Co-authored-by: Adeolu Mary Oshadare <adeolu@placeholder.com>
…ing (#8) * feat(inference): make pipeline analysis-aware with dynamic model loading - _load_model() now accepts analysis_type and reads in_channels/num_classes from config.yaml - Per-analysis-type model cache prevents cross-contamination between deforestation/ice/flood models - _find_best_checkpoint() prefers config.yaml weight path per analysis type - run_inference() accepts analysis_type, pads/crops to correct n_channels, and returns dynamic class counts - run_inference_from_file() and run_inference_from_gee() propagate analysis_type parameter * feat(api): wire analysis_type into prediction endpoints - Pass body.analysis_type to run_inference_from_gee() in /api/predict - Pass analysis_type to run_inference_from_file() in /api/predict/upload - Enables the API to load the correct model and return correct class counts per analysis type --------- Co-authored-by: Olufemi Taiwo <Olufemitaiwo23@gmail.com>
#29) - Add governance module with SHAPExplainer class - Implement band-level and spatial attribution using DeepExplainer - Add /api/explain endpoint for SHAP-based explanations - Create 06_explainability.ipynb with visualization examples - Add shap>=0.42.0 to requirements.txt Closes #22 Co-authored-by: Linda Oraegbunam <obielinda@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add SECURITY.md for security policy and reporting Added a security policy document outlining supported versions, vulnerability reporting, scope, and disclosure policy. * chore: add PR template for contributor guidance Add a pull request template to guide contributors. * chore: add CODEOWNERS assigning @Goldokpa as default reviewer Added CODEOWNERS file to define code ownership and review assignments. * chore: add Dependabot config for pip, npm, and GitHub Actions Configured Dependabot for Python, GitHub Actions, and npm dependencies with specified schedules and reviewers. * chore: add CHANGELOG.md following Keep a Changelog format Document notable changes, additions, modifications, and removals for ClimateVision. * chore: add CITATION.cff to enable GitHub Cite this repository button Added citation file for ClimateVision software. * fix: replace #email placeholder in CODE_OF_CONDUCT with Security Advisory link This change updates the Code of Conduct document by removing the original content and replacing it with a new version that includes various sections on community standards, enforcement responsibilities, and guidelines. * chore: remove SETUP_COMPLETE.md (internal artifact not suited for public repo) * chore: remove internal team_docs (Francis_Umo_Role.pdf) from public repo * chore: remove internal team_docs (Olufemi_Taiwo_Role.pdf) from public repo
End-to-end notebook that loads (or simulates) a biomass-labelled spectral dataset, trains a Random Forest BiomassRegressor, evaluates RMSE/MAE/R^2/MAPE, converts biomass predictions to carbon and CO2e using IPCC defaults, plots feature importances, and persists the model + metrics for the analytics API and the model-card pipeline. Falls back to a synthetic dataset when the labelled parquet file is not present, so the notebook is runnable in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.