MiniSearch — Full-Stack Search Engine Prototype

Google-inspired full-stack search engine prototype featuring asynchronous web crawling, inverted indexing, BM25 ranking, PageRank scoring, autocomplete, spell correction, and a custom search analytics dashboard.

Built with Python, FastAPI, SQLite, and Vanilla JavaScript, focused on information retrieval systems, ranking algorithms, backend engineering, and search architecture design.

---

Features

• Asynchronous Web Crawler — BFS-based multi-level crawler with robots.txt compliance
• Inverted Index Engine — Positional indexing with optimized posting lists
• Hybrid Ranking System — BM25 relevance scoring + PageRank authority fusion
• Autocomplete Engine — Trie-based query suggestion system
• Spell Correction — Edit-distance typo handling and correction
• Search Analytics Dashboard — Indexed pages, query stats, and ranking insights
• Snippet Generation — Context-aware keyword highlighting
• Full-Stack Search UI — Responsive custom search interface with admin controls
• REST API Backend — FastAPI-powered search endpoints
• Lightweight Architecture — SQLite-based retrieval engine with modular services

---

Screenshots

Architecture Overview

Web Crawler Admin Panel

Search Results

System Analytics Dashboard

---

Search Pipeline

Query Input
      ↓
Spell Correction
      ↓
Tokenization + Query Parsing
      ↓
Posting List Retrieval
      ↓
BM25 Ranking
      ↓
PageRank Fusion
      ↓
Snippet Generation
      ↓
Pagination
      ↓
Ranked Results

---

Core Systems

Web Crawling Engine

Features:

• BFS traversal strategy
• robots.txt compliance
• Duplicate URL detection
• Configurable crawl depth
• Async crawling with rate limiting

---

Indexing Engine

Supports:

• Inverted indexing
• Positional indexing
• Token normalization
• Posting-list generation
• Term-frequency statistics

---

Ranking Algorithms

Implements:

• Okapi BM25 ranking
• PageRank authority scoring
• Hybrid rank fusion
• Phrase-aware retrieval
• Query parsing optimization

---

Query Intelligence

Includes:

• Trie-based autocomplete
• Prefix suggestions
• Typo correction
• Edit-distance matching
• Query refinement

---

Analytics Dashboard

Tracks:

• Indexed pages
• Unique indexed terms
• Query statistics
• Discovered links
• Top-ranked pages
• Crawl metrics

---

Tech Stack

Layer	Technology
Backend	Python 3.11
API Framework	FastAPI
Database	SQLite
Crawling	aiohttp + BeautifulSoup4
Frontend	HTML5, CSS3, Vanilla JavaScript
Templating	Jinja2
Ranking	BM25 + PageRank
Data Structures	Trie, Inverted Index
Architecture	Modular search engine pipeline

---

Project Structure

MiniSearch-Engine/
├── backend/
│   ├── api/
│   │   ├── database.py
│   │   ├── main.py
│   │   ├── routes.py
│   │   └── search.py
│   │
│   ├── crawler/
│   │   └── spider.py
│   │
│   ├── indexer/
│   │   ├── autocomplete.py
│   │   ├── indexer.py
│   │   └── spell_correct.py
│   │
│   ├── ranker/
│   │   └── pagerank.py
│   │
│   └── data/
│       └── search.db
│
├── frontend/
│   ├── static/
│   └── templates/
│
├── tests/
│   ├── test_autocomplete.py
│   ├── test_crawler.py
│   ├── test_indexer.py
│   └── test_pagerank.py
│
├── screenshots/
│   ├── architecture.png
│   ├── local-search-engine.png
│   ├── system-stats.png
│   └── web-crawler.png
│
├── requirements.txt
├── pytest.ini
└── README.md

---

Getting Started

Prerequisites

• Python 3.11+
• pip

Installation

# 1. Clone the repository
git clone https://github.com/manyahh07/MiniSearch-Engine.git

# 2. Enter project directory
cd MiniSearch-Engine

# 3. Create virtual environment
python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

# 4. Install dependencies
pip install -r requirements.txt

---

Run Application

uvicorn backend.api.main:app --reload --port 8000

Open:

http://127.0.0.1:8000

---

Example Workflow

1. Open admin panel
2. Enter seed URLs
3. Crawl websites asynchronously
4. Build inverted indexes
5. Generate PageRank scores
6. Query indexed documents
7. Analyze ranked search results

---

Technical Highlights

Hybrid Ranking System

Combines:

• BM25 content relevance
• PageRank authority scoring
• Rank fusion normalization

for balanced search quality.

---

Inverted Index Architecture

Implements positional indexing for:

• Fast term lookup
• Phrase matching
• Efficient retrieval
• Scalable posting-list traversal

---

Async Crawling Pipeline

Crawler uses asynchronous networking for efficient multi-page crawling with configurable limits and duplicate protection.

---

Query Intelligence Engine

Supports:

• Autocomplete
• Spell correction
• Prefix matching
• Query parsing
• Contextual snippets

---

Concepts Demonstrated

• Information Retrieval Systems
• Search Engine Architecture
• Ranking Algorithms
• Backend API Engineering
• Asynchronous Programming
• Database Systems
• Full-Stack Development
• Data Structures & Algorithms
• Query Optimization
• Search Analytics

---

Future Improvements

• Semantic vector search
• Embedding-based ranking
• Redis caching
• PostgreSQL migration
• Distributed crawling workers
• Query caching
• Authentication system
• Docker deployment
• Cloud hosting

---

License

MIT © Manya Singh