An Analysis of Applicability using Quality Metrics for Ontologies on Ontology Design Patterns

• Date: 01 January 2015
• Author: Birger Lantow,Kurt Sandkuhl
• Published date: 01 January 2015
• DOI: http://doi.org/10.1002/isaf.1360

AN ANALYSIS OF APPLICABILITY USING QUALITY METRICS

FOR ONTOLOGIES ON ONTOLOGY DESIGN PATTERNS

BIRGER LANTOW*AND KURT SANDKUHL

University of Rostock, Chair of Business Information Systems, Rostock, Germany

SUMMARY

Ontologydesign patterns (ODPs) provide best-practice solutionsfor common or recurring ontology design problems.

This work focuses on content ODPs, which form small ontologies themselves and thus can be subject to ontology

quality metrics in general. We investigate the use of such metrics for content ODP evaluation in terms of metrics

applicability and validity. The quality metrics used for this investigation are taken from existing work in the area of

ontology quality evaluation. We discuss the general applicability to content ODP of each metric considering its

definition, ODP characteristics, and the defined goals of ODPs. The research process presented in this paper has

two phases. In thefirst phase, we conducted a literatureresearch in the area of metrics for assessing ontology quality.

The secondphase consisted of a two-stepevaluation of theontology metrics identifiedin the literature analysis.During

the first step, we investigated whether the me trics are appropriat e to differentiate between cont ent ODPs of different

quality. Metrics that proved to be applicable werecalculated for a random set of 14 content ODPs. In the secondstep,

a controlled experiment, the quality indicated by the metric value was contrasted with the perception of ontology

engineers; that is,do ‘measured quality’and ‘perceived quality’match?. Copyright © 2015 John Wiley& Sons, Ltd.

Keywords: Ontology Design Patterns; Quality Metrics; Semantic Web; Ontology Engineering

1. INTRODUCTION

In many engineering disciplines, quality is considered an essential factor for acceptance of technologies

and solutions, for efficiency of the processes and for robustness and usabilityof products. Quality is often

considered from different perspectives, including the product-based and the user-based perspectives

(Garvin, 1984).The product-based perspectiveusually views quality as a precise and measurable variable;

that is, differences in quality are reflected in differences of some attribute possessed by a product(Hallak

and Schott, 2011).The user-based perspectiveassumes that quality showswhen using a product or service

and to some extent ‘lies in the eyes of the user’(Bevan, 1995). International standardization attempts of

quality includethese two perspectivesand often add other aspectslike, for example, efficiencyand reliabil-

ity (Jung et al., 2004).Independently of perspectiveand definition, how to determine andmeasure quality

continues to be an issue that is subject of research and industrial development; for example, see Garrido

et al. (2014) or Jozsef and Blaga (2014). With an increasing use of ontologies in industrial applications,

standards, procedures and metrics for quality assessment of ontology construction processes and the arte-

facts produced during these processes also gain in importance. Although considerable efforts have been

spent on developing ontology assessment and evaluationapproaches, including metrics and ways to mea-

sure quality (see Section 2.3), generally accepted practices for industrial use are still missing.

The objective of this paper is to contribute to quality ontologies by focusing on ontology design pat-

terns (ODPs) and ways to determine their quality. ODPs have been proposed as encodings of best

* Correspondence to: Birger Lantow,University of Rostock, Chair of Business Information Systems, Rostock, Germany. E-mail:

birger.lantow@uni-rostock.de

Copyright © 2015 John Wiley & Sons, Ltd.

INTELLIGENT SYSTEMS IN ACCOUNTING, FINANCE AND MANAGEMENT

Intell. Sys. Acc. Fin. Mgmt. 22,81–99 (2015)

Published online 19 January 2015 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/isaf.1360

practices (see Section 2.2) supporting ontology construction by facilitating reuse of proven solution

principles. Different kinds of ODPs have been proposed, like logical, transformation or content ODPs,

which represent different aspects of best practices. This paper focuses specifically on content ODPs and

on investigating the transferability of ontology quality metrics to content ODP. The long-term objective

is to create an instrument for quality assurance in practice; that is, the main intention is not to develop

new fundamental knowledge about ODP characteristics and measurement options, but rather to evalu-

ate how to transfer metrics from the ontology area and what metrics to transfer.

Researchresults presentedin this paper are basedon a research processwith two phases.In the first phase

we conducted a literature research in the area of metrics for assessing ontology quality. The results of this

step are summarized in Section 2.3 and Section 3 respectively. The second phase consisted of a two-step

evaluation of the ontology metrics identified in the literature analysis. Duringthe first step, we investigated

whether the metrics are appropriate to differentiate between content ODPs of proposedly different quality.

For large ontologiesa metric value may well characterize an ontology,but for small ODPs the same metric

mayalways showvery similar or identicalvalues, whichare unlikely to helpdifferentiatingquality.We used

the measurement procedures defined for a metric and determined the actual value for a given set of patterns.

The set of patterns used consisted of 14 randomly selected patterns from one of the major sources for ODPs,

the ODP portal at http://www.ontologydesignpatterns.org/. The results are discussed in Section 4.1. In the

second step, we only considered those metrics that passed the differentiation test during the first step. In a

controlled experiment, the quality indicated by the metric value was contrasted with the perception of

ontology engineers; that is, do ‘measured quality’and ‘perceived quality’match (see Section 4.2)?

The contributions of this paper are (1) the evaluation of a selected set of ontology metrics regarding

their applicability for content ODPs, (2) an assessment of applicable metrics regarding their ability to

differentiate between ODPs of different quality and (3) an investigation on how the proposed metrics

correlate with perceived quality.

The remainder of this paper is structured as follows. Section 2 gives an overview to related work,

which includes the area of content ODPs and ontology evaluation. We discuss possible quality metrics

and their calculation and applicability in Section 3. The metrics that qualify for content ODPs are val-

idated by a set of experiments that we describe in Section 4. Section 5 aggregates our findings and gives

an outlook on future research needs.

2. BACKGROUND AND RELATED WORK

Relevant background for this paper includes knowledge patterns (Section 2.1), ODPs (Section 2.2) and

approaches for quality assurance of ontologies and ODPs (Section 2.3).

2.1. Knowledge Patterns

For more than 20 years patterns have been popular in computer science, and they were introduced for

numerous areas, like software design, information modelling or business processes. Although there is

no generally accepted definition of the term pattern, most publications in the field get some inspiration

from Christopher Alexander’sdefinition (Alexander et al., 1977):

Each pattern describes a problem which occurs over and over again in our environment, and then describes the

core of the solution to that problem, in such a way that you can use this solution a million times over, without

ever doing it the same way twice.

82 B. LANTOW AND K. SANDKUHL

Copyright © 2015 John Wiley & Sons, Ltd. Intell. Sys. Acc. Fin. Mgmt., 22,81–99 (2015)

DOI: 10.1002/isaf