Harvest is a multithreaded C# library for web-crawling. With a lightweight and flexible architecture, it makes common crawling tasks easy.
The API is minimal, and you can control pretty much everything.
[https://github.com/alexandernyquist/Harvest/wiki/Crawling-a-site-for-all-external-links](Crawling a site for all external links)
Contributions are very welcome. If you think Harvest is interesting, please drop me an email, issue or pull request. Thank you!
Yep, for now. Note that Harvest is pretty much a work in progress, but it's already used in production to do some cool things.