Web archives as big data

This one-day conference is organised as part of the AHRC-funded ‘Big UK Domain Data for the Arts and Humanities’ project (see http://buddah.projects.history.ac.uk/ for more information).
The web is an integral part of our daily lives, whether we are shopping online, booking cinema tickets, registering to vote or checking whether or not it is going to rain today. It is also of enormous importance to arts and humanities researchers, as the site of digitised historical material, as a primary source in its own right, and as a means of promoting and communicating research to the widest possible audience. It is hard to imagine how you would write the history of the late 20th and early 21st centuries without access to all of this data.
But web archives offer unique challenges for researchers. The data is messy, collected irregularly to varying degrees of completeness, and as it grows will contain multiple duplicates of some web pages. The depth of collection is uneven: some important sites are crawled in their entirety, others to only a shallower level. Different media within a page may not be acquired successfully, so that only partial content is preserved (the scale of the data collection precludes manual checking). Web pages contain many different types of information beyond what might be viewed as their main textual content – links to other sites, advertisements, contact information, and so on – making analysis problematic. As things stand, we have neither the expertise nor the tools to exploit this invaluable resource reflectively.
The ‘Big UK Domain Data for the Arts and Humanities’ project is seeking to address these challenges, working with humanities researchers from a range of disciplines. This conference will showcase the groundbreaking work being undertaken using the archive of UK web space from 1996 to 2010, as well as exploring how the web is archived by institutions such as the British Library, and the ethical implications of working with this kind of data.