Spectra

Spectra is an iOS accessibility assistant for people who are blind or have low vision. The user points their iPhone at a scene, speaks a request, and the app speaks back an answer: what objects are around, where a specific object is relative to their hand, what a sign or label says, or a short description of the scene.

It is a two-part system: a SwiftUI iPhone client and a self-hosted Python computer-vision backend that the phone talks to over the local network.

Architecture

  iPhone (SwiftUI client)                 Mac / host on same Wi-Fi
  ┌──────────────────────────┐           ┌────────────────────────────────┐
  │ Camera capture            │           │ FastAPI server (main.py)       │
  │ Voice commands (Speech)   │  query +  │  - YOLOv8 object detection     │
  │ Azure neural TTS playback │   image   │  - MediaPipe hand tracking     │
  │ Onboarding / voice select │ ────────► │  - Depth-Anything-V2 depth     │
  │                           │           │  - EasyOCR text reading        │
  │                           │ ◄──────── │  - SentenceTransformer intent  │
  │  speaks the result        │  text     │  - OpenAI GPT-4o scene desc.   │
  └──────────────────────────┘           └────────────────────────────────┘

The client POSTs a multipart request (a text query plus a JPEG image) to POST /process on the backend, and speaks the returned text back to the user.

Features

These are the features actually implemented in the code:

• Object detection - YOLOv8 (yolov8l.pt) detects objects in the camera frame and reports what is present.
• Hand-guided object finding - MediaPipe hand tracking plus depth estimation guide the user toward a requested object relative to their hand.
• Text reading (OCR) - EasyOCR reads text from signs, labels, and documents in view.
• Scene description - OpenAI GPT-4o produces a natural-language description of the scene.
• Intent matching - a SentenceTransformer model maps a spoken request to the right backend capability.
• Voice commands - on-device speech recognition (SFSpeechRecognizer) captures the user's request.
• Neural text-to-speech - Azure Speech synthesizes the spoken response; the app includes onboarding and a voice-selection flow.

Tech stack

iOS client (Swift)

• SwiftUI
• AVFoundation (camera capture, audio playback)
• Speech (SFSpeechRecognizer) for voice commands
• Azure Cognitive Services Speech (neural TTS) via REST

Backend (Python)

• FastAPI + Uvicorn
• Ultralytics YOLOv8 (object detection)
• MediaPipe (hand tracking)
• Hugging Face Transformers - Depth-Anything-V2 (depth estimation)
• EasyOCR (text reading)
• Sentence-Transformers (intent matching)
• OpenAI (GPT-4o scene description)
• gTTS, OpenCV, Pillow, NumPy

Setup

Backend

1. From the Backend Python Files/ directory, install the dependencies. The project shipped without a lockfile; an unpinned requirements.txt is provided as a starting point:

   cd "Backend Python Files"
   pip install -r requirements.txt

2. Set your OpenAI API key. Copy the example env file and fill it in:

   cp .env.example .env
   # then edit .env and set OPENAI_API_KEY=...

   The backend loads OPENAI_API_KEY via python-dotenv (see main.py).
3. The YOLOv8 weight file (yolov8l.pt) is committed in this directory and is loaded directly by the server.
4. Run the server:

   uvicorn main:app --host 0.0.0.0 --port 8000

   A GPU/mps device is expected for the depth model (the backend was developed on Apple Silicon).

Backend tests

The pure guidance/geometry logic (hand-to-object direction, depth/distance bucketing, OCR text assembly, query parsing) lives in Backend Python Files/spectra_guidance.py with no heavy dependencies, so it can be linted and tested without the CV/ML stack, a GPU, or any network access:

cd "Backend Python Files"
pip install ruff pytest
ruff check .
pytest -q

This is exactly what the CI workflow runs on every push and pull request.

iOS client

1. Open the project in Xcode:

   open Spectra.xcodeproj

   This is a plain Xcode project. There are no CocoaPods or Swift Package dependencies to install.
2. Point the app at your backend. The server IP is currently hardcoded as 172.17.96.86:8000 in APIService.swift (and allowlisted for insecure HTTP in Spectra/Info.plist). Change it to your host's LAN IP. See Status and limitations.
3. (Optional) Enable text-to-speech. The app reads AzureSpeechKey and AzureSpeechRegion from the bundle Info.plist at runtime. Copy Secrets.example.plist to Secrets.plist, fill in your Azure Speech credentials, and wire those values into the build. Without a key, the app still runs but speech output is disabled.
4. Build and run on a device. The camera, microphone, and speech features need real hardware; the simulator is not sufficient.
5. Grant camera, microphone, and speech-recognition permissions when prompted.

Status and limitations

This is a student/research prototype, not a shipped product. Known limitations:

• Hardcoded backend IP. The phone talks to a fixed LAN address (172.17.96.86:8000) baked into APIService.swift and Info.plist. There is no in-app server configuration; you must edit the source and rebuild for your network.
• No Python dependency lockfile. The included requirements.txt is unpinned and was reconstructed from the imports. Pin versions before relying on it.
• Self-supplied keys required. Scene description needs an OpenAI key; neural TTS needs an Azure Speech key. Neither is included.
• Large model file in git. yolov8l.pt (~85 MB) is committed directly. Git LFS would be a better home for it in the future.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contact

For any inquiries, contact Sammy Tourani.