Enterprise Search Sourcebook 2009: An Info Pro's Federated Search Pros and Cons

Information Today Inc. each year publishes the Enterprise Search Sourcebook. This year's edition was published in print and digitally, with the digital edition actually having more content. On pages 18 & 19(in the digital edition) is an article I authored entitled"An Info Pro's Federated Search Pros and Cons". If you are interested in search (federated or otherwise), then you may want to thumb through the Sourcebook.

This has been out for a while and I am very remiss in not mentioning it sooner.

---

Technorati tag: Federated Search

Vendor news

I don't include a lot of vendor news in this blog, except if it really peeks my interest. Here, though, are a few things you might want to know:

Blogs --

• Atiz, who manufactures and sells book scanners, has a blog, http://blog.atiz.com/
• Sol Lederman continues to blog about federated search at http://federatedsearchblog.com/ Lederman has really found his stride and is blogging constantly about federated search. This blog is not geared towards one specific vendor's product.
  
• MuseGlobal CEO Kate Noerr's is blogging about federal search at http://blog.museglobal.com/ (calling MUSEings)
• Vivisimo is blogging about search at http://searchdoneright.com/
• One blog that has floundered is Peter McCracken's blog for Serials Solution at http://blog.serialssolutions.com/ McCracken was co-founder of Serials Solutions and is now Director of Research at Serials Solution, which is owned by ProQuest. Too bad.
  

Deals --

• I saw in email this morning that OCLC is offering a summer deal on microfilm digitization.

If you are going to outsource any digitization work, remember talk to your vendors about what they can do for you. Are they offering any deals? Can the work be bundled in a specific way in other to make the cost a bit better? I know, I know...we're all horrible negotiators, but you may be surprised what a few questions will get you.

---

Technorati tags: Digitization, Federated Search

More on federated search

On Dec. 29, I wrote a blog post that listed several resources on federated search. On Dec. 30, Sol Lederman expanded on what I had pulled together, including a link to 21 presentations on the topic. So if federated search is of interest to you, be sure to check out Sol's post and look at the comments, too.

---

Technorati tag:Federated Search

Digitization 101: 2008 Year in Review

As has become my tradition at year-end in recent years, I'm stopping to take a look back and a look forward. What has stood out to me in 2008?

• Google: (posts on Google)
  
• continued to grow its books digitization program by adding more collections/libraries to it.
  
• branched into digitizing magazines.
  
• joined an effort to digitize newspapers.
  
• added photos from Life magazine to its image collection.
• settled issues related to copyright and its book digitization efforts.
  
• Microsoft ended its book digitization efforts (post).
  
• A side-by-side demonstration of four automated book scanners was done in Germany (post, post). Since most organizations could never do that, it was good to see a group take on the task. Also good to see that a report to be written on it.
• JPEG2000 has gained wider acceptance. (related posts)
• "Digital preservation" discussed as not being the correct term. Should we instead use words that describe the outcome not the process? (post)

Personally, I need to mention a few things, too:

• My work on the New York State Regents Advisory Council on Libraries. This has taught me things about libraries, library support agencies, and state budgets that I would have never learned otherwise. (posts)
• Writing Federated Search Report and Tool Kit for Free Pint. This report took much effort and thankfully has been well-received. (post)
• The changes coming to my work life in 2009. Actually, when you teach at the college level, there is a lot of work that must be done before classes begin, so I've already been working at my new job, but officially begin in a few days. I am definitely looking forward to the balance and interplay of teaching and consulting. (post)
  

And what do I see for 2009?

• Google will continue to grow. I do wonder if/when Google will find itself with the same fate as DEC, IBM and others (fallen and no longer relevant).
  
• Organizations and government entities will start to ask questions (more loudly) about the wisdom of relying on major corporations for creating, managing and preserving digital assets.
  
• More organizations will cooperate to build sustainable digitization programs. Not only will libraries, museums and archives cooperate, but for-profit organizations will also find it useful to collaborate (on digitization, equipment and software development, and digital archives).
• Shifting budgets will cause organizations to be more creative. An unusual benefit of a down economy is that organizations turn to more creative solutions; solutions that they would ignore in good economic times.
  
• Unfortunately, many small organizations will still be unable to digitize materials that deserve broader access. I keep waiting for something that will help all small organizations jump on the digitization bandwagon...and waiting...and waiting.
  

Thank you for continuing to read this blog, for the comments and emails that you send, and for recommending this blog to other people. As you can see from the ClusterMap on the left side of this blog, this blog is read by people around the world. Your comments and questions, as well as the number of people who read this blog (and their locations) help me stay dedicated to writing nearly daily on digitization and its related topics.

If you are reading this blog and would like to receive updates in email or through your RSS reader, it is easy to subscribe to this blog to that can happen. Having updates delivered to you can make reading any blog easier for you.

Finally, may you have a Happy New Year! I know that 2008 has been a tough year for many people and organization because of the economy. At the moment, 2009 looks like it will throw us more challenges. However, may you and your organization find ways of getting through 2009 with your sanity, budget, etc. intact.

Related blog posts:

• Thank you (and some statistics) (2007)
• Digitization 101 Year in Review: Words to Take into 2007 (2006)
• Digitization 101 Year in Review: News (2006)
• Digitization 101 Year in Review: Influences (2006)

---

Technorati tags: JPEG2000,Copyright,Digitization,Digital Preservation,Regents,RAC,Book,Federated Search

Federated search blogs, presentations, and a report

My interest in federated search, over the last few years, had led me to follow a few federated search blogs, attended related presentations and -- this year -- write a report on federated search. So let's take a quick look at what's out there, in case this topic also interests you.

Blogs: Here are five that are in my blog reader. There may, of course, be others.

• The Blog Libific by Peter McCracken, librarian and co-founder of Serials Solutions
• Federated Search Blog written primarily by Sol Lederman and sponsored by Deep Web Technologies
• MUSEings written by MuseGlobal CEO Kate Noerr
• Search Done Right, a team blog by Vivisimo, which is a more broadly focused blog
• Beyond Search by Steve Arnold at Arnold IT, which focuses on all corners of the search industry, including federated search

Articles: Well, there are ton of articles on federated search and many of them are good. If you are interested in the topic, I would suggest skimming many articles so you can get a feel for the points of view being expressed. That will help you to narrow down on the types of articles that will be useful to you. For example, are you interested in articles about the industry or in articles on specific implementations? Do you only want to read articles on specific software?

Besides using an Internet search engine to find articles, be sure to use the databases at your library.

Presentations: I suspect that every library-related conference contains several presentations on federated search. Here are a few from 2008:

• Federated Searching: Giving Users What They Want, October 2008, Bonnie Swoger and three others
• Bearcat Search: Implementing Federated Searching at the Newman Library, March 2008, Stephen Francoeur and four others
• What’s new in Federated Search, April 2008, Frank Cervone and Jeff Wisniewski
• Trends and Patterns of MetaLib Usage at a Large Academic Library, August 2008, by Frank Cervone

A Report: The Federated Search Report and Tool Kit, which I wrote for Free Pint, has been well-received. The report includes chapters and activities that guide you through the process of selecting a federated solution. And at £39.00 (or ~US $62.40), it is worthwhile investment. (Of course, I'm biased!)

---

Technorati tag: Federated Search

FUMSI: Federated Search Report and Tool Kit

This report for Free Pint was a long time coming! Thanks to Robin Neidorf at Free Pint for asking me to write this. From the web site...

So many datasets, so little time. Federated search can streamline the process of searching across internal, subscription and web-based sources of information. But when is an organisation a strong candidate for investing in federated search solutions? And how can the information staff research, plan and implement a federated search solution that meets the organisation’s needs?

Jill Hurst-Wahl, a noted industry expert in digital collections, federated search, the nexus between information and technology and recent full-time appointee to the faculty of the information school at the University of Syracuse, provides a compact yet comprehensive resource with her Federated Search Report and Tool Kit. She begins with an explanation of federated search – what it does well and what it does not do well – and then lays out a process for making informed decisions about if and how to move forward.

The report’s 5 chapters provide guidance on conducting a needs assessment, building a project team and gaining buy-in – all essential yet often overlooked steps. Hurst-Wahl also covers researching the industry, considering open source as well as proprietary solutions, researching specific projects, and establishing success criteria so that the project can be measured.

Everyone faces the challenge of choosing the best resource to search for the right information… and we all want it to function as easily as a certain search engine whose name rhymes with ‘frugal’. Whilst that may not be a realistic goal, the Federated Search Report and Tool Kit can help you develop solutions that save your team time, improve usage of key datasets and help users more efficiently get to the information they need to make better decisions.

More...including the table of contents and a sample from the report.

11/4/2008 addendum: Sol Lederman reviewed the report.

---

Technorati tag: Federated Search

Blog post: Federated search is part of collection development

Today I'm the guest blogger on Federated Search Blog. My post comes on the heels of the Library Camp @ Syracuse that occurred last week, which focused on collection development during its second day. Federated search is part of collection development, as Ed Shephard reminded us during his presentation. Unfortunately, we tend to forget that.

---

Technorati tag: Federated Search

Federated Search Blog

Deep Web Technologies is sponsoring a blog on federated search. Begun in December 2007, this is not a blog to promote DWT's product, but instead is aimed "to keep you abreast of happenings in the federated search industry and to serve as a forum for discussion of issues, questions, and concerns among vendors, customers, and potential customers." Sol Lederman is writing nearly daily on federated search and I would think has enough fodder to keep him writing daily for a long time. For example, one of the recent posts is about a web presentation that Frank Cervone and Jeff Wisniewski did on federated search. Frank and Jeff speak frequently on the topic, and give a lively update on how the industry is changing. Sol was able to write and publish extensive notes from the session. (My notes from their IL2007 are in this post.)

---

Technorati tag: Federated Search

IL2007: Notes from Monday

Tom Hogan started off the day by welcoming people to the 11th Internet Librarian and the 8th in Monterey. He said that 48 U.S. plus Washington D.C. are represented at this conference. There are also people from 11 foreign countries. In total there are 1,554 people attending, including 102 exhibitors and 67 people who are attending the exhibits only.

The phrase "Internet Librarian" was first used in June 1993. What is the retronym that describes a non-Internet librarian? Excellent question! He's asked people to put suggestions on the evaluation forms.

Lee Rainie, from the Pew Internet & American Life Project, did the keynote, "2.0 & the Internet World." Lee always has lots of data...too much to takes notes on. Thankfully, his presentation will be online on the IL2007 site. I would also encourage people to check out the Project's web site and read materials there. Their stuff is always very readable. (Once I get more time, I may place some info from his talk online. 11/26/2007 -- Well, I guess I not going to get any notes online from his talk. His presentation, though, is here.)

Sarah Houghton-Jan and Aaron Schmidt did a session on "Online Outreach: 2.0 Marketing Strategies for Libraries." Their presentation -- and most others -- will also be on the Information Today web site. They talked through many tools/strategies for getting the word out about a library: (categories)

• Search engine findability
• Library directory listings
• Blog search engines
• Wikipedia
• Wikimapia
• Community web site info
• Blogs and forums
• Presence where it's warranted
• Social review web sites
• Q&A sites
• Blog geo-search engines
• Online phone directories
• ..and more..

They did mention many specific tools which are listed on their slides.

"Putting Evidence-Based Practice to Work" presented information from Frank Cervone and Amanda Hollister. The key is -- to me -- is that (1) we know how to gather the evidence (and its not hard) and (2) it allows us to make better decisions about the content of library web sites (which was the focus of both presenters). Evidence-based practice has come to us out of the medical community. It is a more accurate -- less emotional way -- of making decisions, since you are making those decisions based on evidence (information). There are many workshops on this topic, so information professionals who need to acquire this skills should be able to learn it.

Meredith Farkas and Joshua Petrusa did a presentation on "User Generated Content." Their presentation and other information is here. Why do user generated content?

• We don't know everything
• Insufficient metadata exists (so let others help build it)
• Findability and refindability
• Stories people tel about items are of value
• Interaction leads to personal connections
• People are doing it!

Why shouldn't you do user generated content?

• No control
• Multiple terms are used to describe the same thing (no authority control)
• People tag selfishly (content is self-centered)
• People tag incorrectly
• Cheap cataloguing but harder to do accurate retrieval

Frank Cervone and Jeff Wisniewski did a lively presentation about "What's New with Federated Search." Research at their institutions has shown that users are using federated search software and that usage is increasing. In addition, people are looking at full-text that they receive from their searches, although it is not known if the users like the articles.

They note that there continues to be consolidation in the industry and vendors swapping the tools that they are using. The slide they used to demonstrate this became a jumbled mess! Companies/products mentioned during the presentation included: (there may have been more)

• SirsiDynix
• Innovative
• VTLS
• Muse
  
• TLC
• WebFeat
• Vivisimo
• Aquabrowser
• Grokker
• Autonomy
• Siderian –
• dbWiz (Simon Frasier Univ. Lib.)
  
• Keystone DLS (Index Data...as well as their newer product)
  
• LibraryFind (Oregon St. Univ. Lib.)

Trends:

• Federated search for articles --> federated search for OPAC and articles --> federated search for all content
  
• Number of vendors is shrinking
• Open source is increasing
• Progress on standards
• Resources migrating to XML format feeds (XML gateways)
• More holistic approach to content
• More affordable turnkey solutions
• Data pre-processing options
  
• Visualization and clustering
• Greater possibility for off-site hosting

As part of the session "Federated Searching Feedback, "I spoke on "Federated Searching Feedback:Walking the Talk?" This presentation was born out of the conversations I've had over the last year with libraries and federated search companies. I've found that companies don't want to provide "too much" information about their products, so it can be difficult to quickly and easily learn the details that you want to know.

Besides talking about the benefits and shortcomings of federated search, I also gave advice to libraries and federated search companies. Advice for federated search companies included:

•

• Talk to people (everyone) about your products
• Use language we understand -- don't use "your own terminology" that doesn't relate to anyone else's terminology
  
• Provide screenshots & explanations -- especially of key/unique features
  
• Be more transparent
  
• Talk openly about how you price, even if you won't talk openly about your prices (e.g., full-time employees, full-time students, number of connectors, etc.)
  
• Help us compare “apples to apples”
  
• Realize our growing sophistication -- our requirements are changing
  

The trends I mentioned included:

• Clustering
  
• Visualization
  
• Incl. OPAC & Internet sources as a norm
  
• Open source
  
• dbWiz (Simon Frasier Univ. Lib.)
  
• Keystone DLS (Index Data)
  
• LibraryFind (Oregon St. Univ. Lib.)
  
• OpenSite Search
  
• More installations
  
• Built-in to database products, not separate software
• Ability to search across databases from one product automatically
  

Finally, Sarah Williams and Angela Bonnell talked about the feedback they gathered on their installation of their federated search product ("Federated Searching Feedback From Usability Testing & Focus Groups"). They presented interesting data and I'll post a link to their presentation once I have it. (11/2/2007: Their presentation is here.)

09/29/2008: Corrected URL for federated search presentation to: http://www.hurstassociates.com/ppt/jhw_a105.ppt

---

Technorati tags: IL2007, Federated Search

Thinking about federated search again (still)

I'm giving a presentation at Internet Librarian on federated search and am beginning to write the summary that will go into the conference proceedings. During my part of the session I'm going to talk about the benefits of federated search, as well as the shortcomings. Shortcomings? mmm...are they shortcomings or are they instances where the library didn't do enough upfront to understand what they were getting (and what they needed in order to make it work)? I'll explore that in the session. And I'll give some tips for vendors who selling federated search products to libraries. It should be a lively session!

BTW the title of my part of the session is "Federated Searching Feedback: Walking the Talk?" (Session A105)

I do have two requests for you. First, let me know what interests you about federated search. This will help me as a prepare for my presentation. Second, if you know of any blogs that focus on federated search, please tell me (I haven't found any yet). You can leave a comment on this blog post with the information. Thanks!

---

Technorati tags: Federated Search, IL2007

CONTENTdm Multi-Site Server

The CONTENTdm Multi-Site Server is not a true federated search product. It does, however, allow CONTENTdm to search across several CONTENTdm installations and present the user with one list of search results (one hit list). As the web site says:

The Multi-Site Server enables users to query multiple CONTENTdm Servers from a single search interface. Geographically remote organizations can create and maintain their own local collections and at the same time provide users with seamless access to cross collection searching on multiple CONTENTdm servers.

This is not just for remote organizations. With many institutions installing CONTENTdm, the Multi-Site Server is a way for them to connect their collections together and create a larger digital presence.

One place that is using the Multi-Site Server is the Mountain West Digital Library. It seems to work seemlessly, although the search can be slow. Results appear to be returned un-ranked, but appear to be grouped by collection.

In addition, the Mountain West Digital Library is using ZContent (An Open Source Z39.50 Connection to CONTENTdm Digital Collection Management Software), which MWDL developed. You can read more about ZContent at http://www.lib.utah.edu/digital/about/zcontent.php.

It is always useful to see software in action and to be able to "bang" on it. If you're using CONTENTdm, you may want to check out the MWDL site and think about the advantages of the Multi-Site Server. For some, the product may be more appropriated than installing federated search software, although the product is limited to searching only CONTENTdm sites. (Some regions have so many CONTENTdm sites, that just connecting them together under one search umbrella could be quite useful.)

---

Technorati tag: federated search

CIL2007: Federated Search

[04/22/2007 -- corrected formatting.]

Frank Cervone and Jeff Wisnewski – Federated Search: State of the Art

Standard features:

• Support for multiple protocols
• Z39.50, SRU/SRW. OAI. Other XML
• Simple and advanced search
• Post processing of results
• Statistics

Advanced search features, e.g., integration with other software

Library Journal Automated System marketplace 2006 lists the competitors in this product space.

• Ex Libris
• WebFeat
• Products based on MuseGlobal
• And others…

However, we need to remove the confusion from having to make choices.

Gone from the marketplace: Endeavor. SirsiDynix’s solution is going away.

Database vendors are enhancing their market share by offering federated search. (e.g., ProQuest CSA Illumina/MultiSearch). But do our users understand the categories, for example, that are in MultiSearch?

Will ProQuest CSA kill MultiSearch in favor of Serial Solutions?

Ovid SearchSolver:

• Based on Muse technology
• Hosted
• Results post-processing
• Customizable, brandable
• Integrates faceted browsing

Ex Libris – Endeavor: all of the Endeavor federated search is being integrated into the Ex Libris product line.

• MetaLib
• SFX
• Primo

Metalib 4.0:

• Now with faceted browsing
• Interface with eventually be supplanted by Primo

Aquabrowser:

• Visual navigation
• Different way of information discovery

Grokker in EBSCO:

• Visual navigation
• Exposes relationships between pieces of data that may not be apparent in a text environment
• A pleasant experience for the user

Encore (by Innovative):

• Includes tag clouds – associates works with LCSH (Library of Congress Subject Headings) and then associates those LCSH with phrases and creates a phrase cloud

Fretwell-Downing (OCLC PICA):

• FDI Portal
• OL2
• Sounds like there were in the forefront at one point but no more. Frank showed a cartoon of a man yawning.

WebFeat Express:

• Hosted version of WebFeat Prism
• $7950 for an annual subscription
• Several interface choices – brandable
• OpenURL compliant
• ProCite/EndNote compatible
• EBSCO A-Z Integration
• Trying to make it plug-n-play as much as possible

Who else is in the game?

• Agent
• Carlweb
• Cameleon
• DOM2
• Millennium Access Plus
• Encore
• Explorit
• Keystone
• MetaSearch Solution
• Muse
• OpenSiteSearch
• Polaris
• Serials Solution
• Searcher Analyzer
• WebClarity

Looking outside of libraryland:

• Autonomy -- with Universal Repository Interfacing
• Endeca
• Siderian -- Take Guided Navigation a step further…

Trends:

• Number of vendors is shrinking
• Migration to XML information feeds
• Integration of content with external systems
• Organizational portals
• Bibliographic management software
• Course management/ learning management systems
• Increased use of visualization and clustering
• Greater possibilities for off-site hosting
• Data pre-processing options
• Google Scholar / Microsoft Academic Live

They skimmed through a lot of information. I wish they had gone more in-depth and perhaps covered less (deeper information on fewer topics, rather than a small bit of information on a lot of topics).

---

Technorati tag: CIL2007

Texas Heritage Online: A federated search system

We try to learn from others, but often other projects and programs don't want to share information about how they did it. This could be due to a need for confidentiality, a belief that they don't have anything to share, a desire to hide their mistakes, or just not enough time to put the information online. Therefore, I was very pleased to see the information available for the Texas Heritage Online, "a federated search portal for Texas libraries, archives, and museums with digital collections of cultural heritage materials." Here is a one-page document that gives a technical overview of their federated search software. The information is easy to read and only uses necessary jargon.

Thanks to whomever spearheaded the effort to put this information online. I'm sure that others besides myself appreciate it.

---

Technorati tag: Federated Search

Thinking about federated search again (still!)

I've been submerged in federated searching "articles" for a few hours today and compiling information (and thoughts). There is much that has been written on federated search, but very little of it seems to talk about using federated search software to search across digitization programs. We might think "oh, of course it can do that" but I know that one vendor seemed a bit iffy on whether theirs would work with CONTENTdm and some of the other digital asset management systems.

Below is the beginnings of a bibliography I'm compiling on federated search. These are pieces that I think are relevant to what I'm working on and the list will likely get longer as my research continues.

I want to point out that the Library of Congress Portal Applications Issues Group has created a list of federated search portal products and vendors. Unfortunately, the list is two years old, so it is a snapshot in time. Still a good starting place since there are likely few lists like it.

When you talk to people about federated search, a few names always rise to the surface, yet -- looking at the LOC list -- there are many products that have a federated search component to them including products built into integrated library systems (ILS). So when you think about implementing a federated search option for searching across digitization programs, be sure to talk to the companies that are supplying those systems to you and your collaborators. They may have have the functionality that you need or be willing to work with you to develop those capabilities.

By the way, I find it interesting to see what companies say about their federated search capabilities. Some like Polaris Library System describe functionality that I would describe as being federated search, but they don't use those words. We might shy away from the term "federated search" because it seems too technical or unfamiliar, yet those of us who are researching federated search products look for those words (and then the other details to back it up). So please don't shy away from the "federated search" phrase!

Short Bibliography:

• A Checklist of Considerations for Selecting Metasearch Software, 2004, http://www.cdlib.org/inside/projects/metasearch/metasearch_checklist.pdf
  
• Federated Search Portal Products & Vendors, Library of Congress, 2005, http://www.loc.gov/catdir/lcpaig/portalproducts.html
  
• Federated Search: Practical Applications and Future Directions (presentation slides), 2005, http://www.nfais.org/events/event_details.cfm?id=30
  
• Resources on Metasearching, California Digital Library, 2005, http://www.cdlib.org/inside/projects/metasearch/info.html
  
• Federated Search Symposium wrap-up, Paul R. Pival (Distance Education Librarian at the University of Calgary), 2006, http://distlib.blogs.com/distlib/2006/02/federated_searc.html

An off-topic thought: When we think of companies that provide integrated library systems, we often think immediate of behemoth companies like SirsiDynix, yet there are non-behemoths like Polaris we should be paying attention to. Cast a broad net and look to see who will provide the functionality and support that you need. Don't make assumptions!

---

Technorati tag: federated search