IRF at PIUG Annual Conference 2010

2010 | 04 | 30

At this year's PIUG meeting in Baltimore John Tait presents: "Evaluating the Effectiveness of Patent Search Systems".

John Tait speaks on Monday, May 3rd at 5pm.

More information about the conference can be obtained here.

Abstract

During 2009 the Information Retrieval Facility, a Vienna, Austria, based not-for-profit organization promoted two major evaluations of patent search systems.

The first, TREC CHEM, was run under the auspices of the US federal National Institute of Standards and Technology, and focused on chemistry searching using a collection of chemical patents (supplied by Matrixware) and academic papers supplied by the Royal Society of Chemistry.

The second, CLEF IP, was run under the auspices of the somewhat informal European Cross Language Evaluation Forum, and focused on searching patent data in English, French and German across all subject domains.

The presentation will review the results of evaluations and point out some preliminary conclusions and future directions. Very briefly, the primary conclusions is that knowledge of patents helps with effective retrieval, but it is difficult to integrate domain knowledge (e.g. of chemistry) into essentially text retrieval systems in a way which genuinely improves effectiveness. Both evaluation campaigns will run again in 2010 in improved form, learning from the lessons of the 2009 campaigns and trying to make the evaluations more relevant to the requirements of real patent searchers.

Disappointingly all the entrants were academic or research institutes and we at the IRF are hoping to either get engagement from commercial suppliers or to find other types of evaluation campaign with which they are willing to engage.

On a more positive note the campaigns have radically raised the profile of patent search as a problem and potential market for the technology providers, and hopefully this will lead to significant improvements in the tools available to patent searchers in the not-too-distant future.