Uncategorized

The Accuracy of SHERPA/FACT

Today sees the release of figures from an accuracy survey that was carried out on SHERPA/FACT under the control of the FACT Advisory Group, which includes stakeholders from libraries, publishers, funders and open access works from the sector.

A sample of queries for the compliance of a journal with various funder policies were assessed independently by professionals from UKCoRR working from available information and then by SHERPA/FACT. We are very pleased to say that SHERPA/FACT shows an accuracy of 95% in assessing the combined Gold and Green compliance of a journal. The figure was even higher in assessing either the Gold or the Green.

This level of accuracy comfortably exceeds the level of accuracy asked of the system by a previous Jisc survey of users, who felt that 90% would be sufficient to support the use of SHERPA/FACT.

It is also significant that the system exceeded the level of accuracy of the professionals checking independently. We see this as a strong argument for the use of SHERPA/FACT by professionals to save time and expertise in dealing with the increasing complexity of open access policy requirements.

The results have prompted the funders of SHERPA/FACT to publicly approve the service as the authoritative source to check if a journal is compliant. From today’s Press Release:

Mark Thorley, chair of RCUK Research Outputs Network, said: “I am pleased that an independent test has validated that the information in FACT is highly accurate. The Research Councils support FACT as the authoritative source of information to check if a journal is compliant with the RCUK policy on open access. We hope that all those that we fund will use FACT as the most efficient and accurate way to check journal compliance with our OA policy.”

These findings match internal quality assurance exercises that we have previously carried out. We are pleased that SHERPA/FACT has now shown independently that it can deal with the complexity of the policy environment and produce consistently accurate results. We hope that our users will continue to find SHERPA/FACT as a useful and accurate assistant in dealing with policy compliance.

The Methodology paper for the survey is available through FigShare, along with the data from the assessments.

http://figshare.com/articles/SHERPA_FACT_Accuracy_Testing_Methodology/1409594

http://figshare.com/articles/FACT_UK_CORR_Test_Results_v1/1408538

The study was commissioned by the SHERPA/FACT Advisory Board – which includes representatives from UKCoRR, Jisc, Wellcome Trust, Research Councils UK (RCUK), CRC Nottingham, Association of Learned and Professional Society Publishers (ALPSP), Higher Education Funding Council for England (HEFCE), Publishers Association and SCONUL

 Bill