An interview with The Register-Guard’s <img class="size-medium wp-image-1391 alignright" title="scraper" src="http://www.journalismaccelerator.com/wp-content/uploads/2011/04/scraper-300x237.jpg" alt="scraper" width="225" height="177" /> <a title="John Heasly" href="http://blogs.registerguard.com/rga/index.php/contacts/john_heasly/" target="_blank">John Heasly</a> by JA’s <a title="Tram Whitehurst" href="http://www.journalismaccelerator.com/about/ja-launch-and-beta-crew/">Tram Whitehurst</a>

<em>John Heasly is the Web content editor at The Register-Guard in Eugene, OR.</em>

<em><strong>What are some of the ways you're using web scraping at your newspaper? How has it enhanced your work?</strong></em>

“Here's <a href="http://special.registerguard.com/police/eugene/" target="_blank">one</a> of my favorites. It updates every fifteen minutes and uses <a href="http://www.eugene-or.gov/portal/server.pt/gateway/PTARGS_0_0_11080_219_0_43/http%3B/ceppportlet.eugene1.net/epdpubliccad2/cadday.aspx?date=03/13/2011" target="_blank">this page</a> as its source. Another big scrape is elections… Here’s the full <a href="http://special.registerguard.com/ballot/results/full/" target="_blank">results page</a>.

We also have a smaller customizable widget of selected races that we put on the homepage. It uses both <a href="http://egov.sos.state.or.us/division/elections/results/2010G/index.html" target="_blank">county</a> and <a href="http://egov.sos.state.or.us/division/elections/results/2010G/index.html" target="_blank">state</a> pages for data. The data also gets formatted into InDesign templates and is reverse-published into the paper.”

<strong><em>Where do you see the technology headed? Are there new ways in which it can be used?</em></strong>

“I don't see any wildly revolutionary changes in the field of screen-scraping, as it's kind of a hack/workaround to begin with. I think as data publishers-news sources-governments get their acts together, there will be more APIs, so people can get at the data directly. I think the ease of geolocating events is going to continually increase.”

<strong><em>Who else is doing web scraping well?</em></strong>

“Well, any of the EveryBlock sites, of course. The L.A. Times' <a href="http://crime.latimes.com/" target="_blank">crime map</a> recently came to my attention. It seems pretty amazing.”

<strong><em>What is your advice to journalists looking to start using web scraping in their own work?</em></strong>

“Find a problem, attack it! Make sure it's something you're passionate about, otherwise, when you hit a bump — and you will hit bumps — you'll get de-railed. Ask questions in newsgroups, Google groups, <a title="Hackskackers" href="http://help.hackshackers.com/" target="_blank">help.hackshackers.com</a>. The open-source software and the advice/help are free. All you need is a computer and an Internet connection and an appropriate pig-headedness and you're set!”

<strong><em>What are your favorite Web scraping tools and guides?</em></strong>

“I like Python as a language. I like the Python module BeautifulSoup for taming what I've scraped and Django as a Web framework for serving the scrapings.”

&nbsp;