Open Content Index

About the Index

(Psst. Print this page. It is designed to read on a piece of paper.)

Open access content abounds, and all the library profession needs to do to leverage it is: 1) collect it, 2) organize it, 3) index it, and 4) provide access to it. This Web page illustrates one way these ideas can be implemented.

There exists a wide range of open access content appropo to the users of libraries. Some of this content includes but is not limited to:

• Open Directory Project - Human-cataloged web resources. (4,780,821 records)
• Project Gutenberg - High-quality clean-text ebooks, some audio-books. (20,800 records)
• OAIster - A Union catalog of digital resources, chiefly open archives of journals, etc. (9,558,628 records)
• Open Content Alliance (OCA) - All of the ebooks made available by the Internet Archive as part of the Open Content Alliance (OCA). Includes high-quality, searchable PDFs, online book-readers, audio books, and much more. Excludes the Gutenberg sub-collection, which is available as a separate database. (134,250 records)
• OCA American Libraries collection - A sub-collection of the OCA. (48,299 records)
• The Internet Archive Children's Library - Books for children from around the world. (669 records)
• Open Source Books - A collection of community-contributed books at the Internet Archive. (2,489 records)
• OCA Canadian Libraries - A sub-collection of the Open Content Alliance. (36,730 records)
• The Universal Library - A digitzation project founded at Carnegie-Mellon University. Content hosted at the Internet Archive. (30,888 records)
• Wikipedia - Titles and abstracts from Wikipedia, the open encyclopedia. (1,657,443 records)

Besides existing in their native Web interfaces, these sets of content are also available in machine-readable formats. (Think MARC.) After systematically harvesting this content and indexing it, it is possible to provide a myriad of interfaces to it. A long time ago and in a galaxy far far away, Z39.50 used to be the interface. Today there are additional interfaces including SRW/U and OpenSearch.

This page -- Open Access Index -- is an example of the idea outlined above. A company named Index Data harvested metadata from the sources above, indexed it, and provided a number of interfaces to their index. Open Access Index takes advantage of the SRW/U interface. Users select a specific index to search, enter a query, and click Go. The query is sent to a client application that converts the query into SRW/U and sends it to a server at Index Data. The server searches the selected index and returns a stream of XML. The client application transforms the XML into XHTML and displays the result on the user's screen. The whole client application is written in Perl, and the heart of the application is almost trivial in nature:

param( 'query' ));
  
  # build an SRU url
  my $url = SRUTEMPLATE;
  $url =~ s/##QUERY##/$query/e;
  $url =~ s/##INDEX##/$cgi->param( 'index' )/e;
  
  # create a user agent, create a request, send the url, and get a response
  my $ua       = LWP::UserAgent->new;
  my $request  = HTTP::Request->new( GET => $url );
  my $response = $ua->request( $request );
  
  # transform the response
  my $parser     = XML::LibXML->new;
  my $xslt       = XML::LibXSLT->new;
  my $source     = $parser->parse_string( $response->content ) or croak $!;
  my $style      = $parser->parse_string( &get_sru_to_html )   or croak $!;
  my $stylesheet = $xslt->parse_stylesheet( $style )           or croak $!;
  my $results    = $stylesheet->transform( $source )           or croak $!;
  
  # create the results page
  my $html = &get_template;
  $html =~ s/##FORM##/&get_form/e;
  $html =~ s/##CONTENT##/$stylesheet->output_string( $results )/e;
  $html =~ s/##QUERY##/$cgi->param( 'query' )/ge;
  
  # done
  &gracefulExit ( $html );
 
]]>

For more information about what Index Data did and its machine-to-machine interfaces see www.indexdata.com/opencontent. If you want to the client source code to Open Access Index, then see: mylibrary.library.nd.edu/oci/oci.txt. As you will see, most of the program is about user interface, not searching. (For a good time, it might be fun to add a Did You Mean? function and/or a synonym function to the interface. Hmmm...)

Summary

Again, open access content abounds. Much of it is relevant to needs of library users. By collecting this content, indexing it, and providing sets of services againt the index meeting the needs of users, libraries can exploit this thing called the Internet to a greater degree. By enhancing its skills and evolving with the environment librarianship can retain its traditional missions and at the same time provide useful collections and services to it clientele. The opportunities are almost limitless.