# Doc Scraper

> Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).

- **Website:** https://github.com/Sriram-PR/doc-scraper
- **Pricing:** unknown
- **Categories:** Writing & Content, Developer Tools
- **Tags:** writing-content, developer-tools, design
- **Platforms:** Web, CLI
- **Last verified:** 2026-09-10
- **Canonical page:** https://linkrena.com/tools/doc-scraper

## About

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files.

The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is:

This installs the doc-scraper binary to your GOPATH/bin directory (usually ~/go/bin or %USERPROFILE%\go\bin ). Make sure this directory is in your PATH .

This creates an executable named doc-scraper in the project root.

A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag.

HTTP Client Settings: (Global; cannot be overridden per site. Pool, dialer, and TLS timings are baked into pkg/fetch with sane defaults and are not exposed as config knobs.)

## Related tools

- [TexLite](https://linkrena.com/tools/texlite): TexLite: Lightweight self-hosted collaborative LaTeX editor and Overleaf alternative - SWUFE-DB-Group/TexLite
- [IndexFlow core](https://linkrena.com/tools/indexflow-core): Open-source search engine indexing infrastructure with inline SEO quality gates, fair multi-site scheduling, and rolling quota circuit breaker.
- [Unsung](https://linkrena.com/tools/unsung): A blog about software craft and quality
- [Web Iterate](https://linkrena.com/tools/web-iterate): Google today reached the final milestone in a browser-extension transition that has been years in the making, all remaining Manifest V2 extensions were removed…
- [Openclip](https://linkrena.com/tools/openclip): Fast, open-source macOS floating text actions.
- [WebToolsEdge](https://linkrena.com/tools/webtoolsedge): Free browser-based tools for text, SEO, images, documents, calculators, development, and everyday productivity.