The Spokesman-Review’s Ryan Pitts on Web Scraping
http://www.journalismaccelerator.com/resources/the-spokesman-review%E2%80%99s-ryan-pitts-on-web-scraping/
Courtesy matters, when Web scraping is necessary, design scripts that honor the source.
|
The Register-Guard’s John Heasly on Web Scraping
http://www.journalismaccelerator.com/resources/the-register-guard%E2%80%99s-john-heasly-on-web-scraping/
An interview with The Register-Guard’s John Heasly by JA’s Tram Whitehurst John Heasly is the Web content editor at The Register-Guard in Eugene, OR
|
Scrapy
http://scrapy.org/
A screen scraping and web crawling framework used to crawl websites and extract structured data from their pages.
|
ScraperWiki
http://scraperwiki.com/
A collection of tools for Web scraping, data mining and visualization.
|
OutWit
http://www.outwit.com/
Helps to find, grab and organize all kinds of data and media from online sources.
|
Yahoo! Pipes
http://pipes.yahoo.com/pipes/
A composition tool to aggregate, manipulate, and mashup content from around the web.
|
Coding for Journalists 101: A 4-Part Series
http://danwin.com/2010/04/coding-for-journalists-101-a-four-part-series/
A series of tutorials to guide the non-coding-but-computer-savvy journalist through enough programming fundamentals so that he/she can write a Web scraper to collect data from public websites.
|
Python Tutorial
http://www.palewire.com/posts/2008/04/20/python-recipe-grab-a-page-scrape-a-table-download-a-file/
Instructions on using Python for scraping.
|
Perl Tutorial
http://nicar-phoenix.s3.amazonaws.com/scraping-presentation.html
Instructions on using Perl for scraping.
|
Scraping for Journalism: A Guide for Collecting Data
http://www.propublica.org/nerds/item/doc-dollars-guides-collecting-the-data
Lots of links to online tools and guides.
|
An Introduction to Compassionate Screen Scraping
http://lethain.com/entry/2008/aug/10/an-introduction-to-compassionate-screenscraping/
Step-by-step guide to scraping, with a discussion of the ethics of Web scraping.
|
State Data Scraper
http://openstates.sunlightlabs.com/docs/scrapers.html
Tools and information for scraping data from state websites.