<img class="size-full wp-image-1329 alignright" title="ScraperWiki" src="http://www.journalismaccelerator.com/wp-content/uploads/2011/04/logo_large1.png" alt="ScraperWiki" width="225" height="141" />

An interview with The Spokesman-Review’s <a title="Ryan Pitts" href="http://www.spokesmanreview.com/local/staff.asp?bylinename=Ryan%20Pitts">Ryan Pitts</a> by JA’s <a title="Tram Whitehurst" href="http://www.journalismaccelerator.com/about/ja-launch-and-beta-crew/">Tram Whitehurst</a>.<em></em>

<em>Ryan</em><em> Pitts is the online director at <a title="The Spokesman Review" href="http://www.spokesman.com/">The Spokesman-Review</a> in Spokane, W</em><em>ash.</em>

“We primarily use data made available through various APIs, in either JSON or XML format, or we take data exports in CSV and then import them into our own system. As far as writing scripts to go out and scrape pages that don't have corresponding data made available in one of those other ways, well, we don't really do much of it. Not because we can't, but because there are enough projects to keep us plenty busy with data that's easier to get at.

That said, if a project came up where we could ONLY get what we needed via scrape, the tool du jour these days definitely seems to be <a title="ScraperWIKI" href="http://www.journalismaccelerator.com/resources/technology-resources/scraperwiki/"><span style="text-decoration: underline;">ScraperWIKI</span></a>. There's a <a title="Scraper Tutorial" href="http://scraperwiki.com/help/tutorials/python/">tutorial page</a> that can help you get going, even with nominal understanding of programming.

There's also a great tutorial <a title="Ben Welsh" href="http://www.palewire.com/posts/2008/04/20/python-recipe-grab-a-page-scrape-a-table-download-a-file/"><span style="text-decoration: underline;">here</span></a> by Ben Welsh of the Los Angeles Times.

As far as scraping advice, <strong>I think one of the key things to keep in mind is the same as we do for data use via API — be considerate of your data source. </strong> Don't hammer their web server with your script. In whatever way you're taking in the data, build in enough delays that their web server doesn't suffer and keep other people from being able to access it.”