Webscrapers is basically the short name for us. A bunch of people that are interested and knowledgable in web scraping, HTTP scripting, bot writing and internet spidering. For fun and profit.

We have a mailing list setup for discussions and questions around issues within this area.

We started this on July 2nd 2010. We are already over 70 subscribers, and among these we have two of the authors of the books mentioned below and the primary author of curl and libcurl.

To fully master the art of web scraping, we recommend and use the following:

• Command line tools: curl, Wget
• Network analyzers: Wireshark
• Firefox extensions: LiveHTTPHeaders, iMacros, Tamper Data, Selenium
• Internet Explorer tool: ieHTTPHeaders
• ...
• [please tell us more good tools on the list]

Webbots, Spiders, and Screen Scrapers by Michael Schrenk:

php|architect's Guide to Web Scraping with PHP by Matthew Turland:

Spidering Hacks By Tara Calishain and Kevin Hemenway:

If you have other favourites, your own or others, please tell us on the list!