Doc Scraper

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files.

The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is:

This installs the doc-scraper binary to your GOPATH/bin directory (usually ~/go/bin or %USERPROFILE%\go\bin ). Make sure this directory is in your PATH .

This creates an executable named doc-scraper in the project root.

A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag.

HTTP Client Settings: (Global; cannot be overridden per site. Pool, dialer, and TLS timings are baked into pkg/fetch with sane defaults and are not exposed as config knobs.)