pywikiscrape

pywikiscrape is a user-friendly but powerful open-source Wikipedia scraper written in Python that lets users export articles to an SQLite3 database using Beautiful Soup.

---

Features

• Automated Scraping: Automatically scrapes Wikipedia articles with requests and BeautifulSoup.
• Zero-Config: Simply run the script, and it will handle everything for you.
• Two SQLite3 Tables: Stores two tables, one for the content of the article, the other for the links contained in it, in one file.
• Minimal Dependencies: Relying only on Python standard library modules, requests, and BeautifulSoup.
• Error Handling: Handles HTTP error codes without user input.
• Logging: Uses the Python Standard Library module logging over print statements.

Usage

Currently, the only way to use pywikiscrape is to clone the GitHub repository and manually run the script. To do so, follow this code block. You must ensure that you have all required dependencies available to your Python interpreter.

# Clone the repository
git clone https://github.com/tyleruploads/pywikiscrape.git
cd pywikiscrape

# Set up a virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Runs the script
python3 src/main.py

The script saves its output to an SQLite3 database with the name of pywikiscrape.db. You do not need to pass any arguments to the script.

To choose a seed, the script uses a few link-rich preset Wikipedia articles and one link to a completely random Wikipedia article.

Database Schemas

Table 1: Articles

This table contains the title of the Wikipedia article and the text inside of it.

Column	Type	Description
id	INTEGER	Primary Key
title	TEXT	Wikipedia article Title
text	TEXT	Content of Wikipedia article

Table 2: Links

This table contains the id correlating an article to its entry in table 1 and the links contained inside of it.

Column	Type	Description
id	INTEGER	Primary Key
links_json	TEXT	The links contained in the Wikipedia article

Roadmap

• Package and publish to PyPI
• Have the CLI ask the user for a seed, a custom path for the output database, and other settings, all with defaults the user can easily select
• Support scraping non-English Wikipedia domains
• Store more metadata about article like categories, images, and when it was added to the database
• Allow more export options for data like JSON, CSV, or Parquet alongside SQLite
• Add Unit testing
• Add a Dockerfile and docker-compose.yml for zero-setup execution

Contributing

Contributions are what make open-source projects one of a kind. All contributions are highly appreciated.

• Found a bug or issue: Open an Issue and show the output of the script, the steps to reproduce it, and as much information as possible
• Have an idea: Open an Issue and explain your idea as much as possible, why you think it would be a good addition to the project, and any other important information.

License

This project is licensed under the MIT License - see the LICENSE file for more information.