Systems engineering-based tool for identifying critical research systems.

Author:Abbott, Rodman P.
Position:Report - Abstract

Abstract: This study investigates the relationship between the designated research project system independent variables of Labor, Travel, Equipment, and Contract total annual costs and the dependent variables of both the associated matching research project total annual academic publication output and thesis/dissertation number output. The Mahalanobis Taguchi System (MTS) pattern recognition methodology was utilized in the three-year, 3000+ research project case study to identify which research project system variables are responsible in both magnitude and degree for the associated publication & thesis and dissertation research project outputs. The selection of the MTS "abnormal" and "normal" data set boundary was specifically chosen in an attempt to define a "successful" from "unsuccessful" research project metric at the research unit, tenure track versus non-tenure track, and individual research project principal investigator organizational levels. The findings of the study are directly compared against the associated level budget percentages changes over the same time period. Through such concrete research system identification, research administrative personnel have the possibility of directly identifying both the research project system "imp actors" as well as the association between likely effects of reduced or affected research system impacts on the research outputs themselves.

Keywords: Faculty Integrated Outputs, Mahalanobis Taguchi System, System of Systems


Modern university research projects, even though they are still primarily leadership driven by individual Principal Investigator (PI)/Program Director (PD)-equivalent faculty members, must today utilize and rely on a series of institutional infrastructure systems for their facilities, instrumentation, travel, contracting, labor, and administrative needs to function effectively. Each of these systems has various costs associated with them (Haley, 2009; Haley 2011; Grieb, Horon, Wong, Durkin, & Kunkel, 2014).

This paper addresses the use of systems engineering-based management concepts to respond to and develop an advanced research systems administrative and management information approach prototype in order to allow research universities to improve the effectiveness of their research infrastructure administrative tools and policies.

The International Organization of Standards (ISO) Subcommittee on Software Engineering (SC/7) developed the current measurement standard for software measurement processes. "This International Standard identifies the activities and tasks that are necessary to successfully identify, define, select, apply and improve measurement within an overall project or organizational measurement structure" (ISO, 2007). However, while this standards title (referred to as ISO/IEC 15939) uses the terminology of software engineering, it is explicitly meant to also refer to systems engineering (Frenz, Roedler, Gantzer, & Baxter, 2010). ISO/IEC 15939 is also defined in terms of fields of application. In the context of a university research administrative organization, one of the fields of application, "by a supplier to implement a measurement process to address specific project or organizational information requirements" can be seen as representing the necessary measurement needs of a research university administrative component organization (ISO/IEC 15939, 2007).

Of note, ISO/IEC 15939 is not a library of measurements nor does it provide any recommendation on which measures apply to an individual project or organization. It merely defines a process supporting the construction of defined and tailored measures for an organization's individual requirements.

From an organizational management perspective, ISO/IEC 15939 also details the steps necessary for an organization to ensure that their measurement processes are optimized to form a set of "requirements" to ensure the maximum utilization of these same measurement processes. Figure 1 depicts this process, including the four fundamental measurement task activities.

Institutional Background

Between 2009 and 2012, the Naval Post Graduate School (NPS) experienced an unprecedented growth rate exceeding thirty percent annually in its research funding. From ~$75M in 2009 to over ~$150M in 2012; the bredth and scope of research work done at NPS increased dramatically (NPS, 2014). Such growth was not without its problems, however. Several major new research program investments failed to achieve their intended results and goals. A root cause analysis was conducted at the project level by the Chairs and Dean of the associated schools. Additionally, several internal and external reviews were conducted of both institutional research project acceptance and research project review processes. As a consequence, the top level research management and administrative level organization at NPS, the Research and Sponsored Programs Office (RSPO), began exploring possible additional technical and management processes in order to assist in preventing such similar occurrences from occurring again.

As a result of these technical and management process actions, the RSPO has adopted a multi-pronged institutional approach to managing the acquisition of key information required for support, improvement, and strategic planning for critical research activities. These include: (1) the development, formation, and maintenance of an institutional work acceptance policy and joint academic, research, and leadership committee to monitor and serve as a senior decision body for research program acceptance according to the institutional strategic plan; and (2) the development, formation, and maintenance of an evolving research system, staff, and project research output measurement information construct. This latter element will form the major portion of this paper.

Research Administration Background

Education, research, and services constitute the three major responsibilities of universities (Boyer, 1996). Because of the inherent complexity of optimizing and simultaneously balancing these three different mission areas, the resultant university structure that results from it is one that requires a critical, evolving, and well thought out management process. This would include the specific research management structure employed (Bosch & Taylor, 2011; Pettigrew, Lee, Meek, & Barros, 2013). Furthermore, in selecting the elements necessary in formulating the elements of this research management structure, Mintzberg (1979) lists four generic parameters, including an information based decision making system as being necessary (Haines, 2012).

With respect to this decision making system Kirkland (2008) notes "a system to identify any emerging problems at an early stage" as being critical. Taylor (2006) also supports this concept by stating that research management administration should be seen as "encouraging, supporting and monitoring" project entities.

The general advantages of developing a systematic information based measurements tool basis as a tool for research administration are not new. Haines (2012) has pointed out that their use includes establishing and oversight of research business processes, defining responsibilities, controlling expectations, driving team motivations, assessing research staff performance, as well as upgrading tools for both research decision making and prioritization.

While many research administrative organizations use various information measurement constructs as a key portion of their overall responsibility, as Nguyen, Huong, and Meek (2015) point out, "the need for an effective, evidence-based metric standard that captures the complexity of the (Research Management) field remains unmet."

As Nguyen et al. (2015) have also pointed out, universities typically use an amalgam of publication information, peer review, or a combination of the two former to quantify research personnel outputs at the individual principal investigator, department, or school levels. This combination, or bibliometrics, includes such elements as impact factors and/or citation rates. While there are both pro arguments (Taylor, 2011) and contra arguments (Adams, 2009) to the use of bibliometrics, the inclusion of specific system derived information directly relevant to the integrated System of System (SoS) research project outputs has, to the best of the authors' knowledge, not been attempted for inclusion.

Literature Review

Cost functions as almost a universal sanctioned means of exchange and value throughout communities (Newlyn, 1978). Investments in a particular quantity, whether it be equipment, labor, project, etc., can all be associated with the appropriate process if there is an accurate accounting system to assign "cost" in dollars to these processes and interactions (Langford, 2012). The system engineering usage of cost constituting the basis for modelling is also well known (Boehm et al., 2000; Blanchard, 1998).

Any performance management information system which has the capability to actively predict performance, must also meet two additional criteria: (1) it must have a program management framework; and (2) it must also have a procedural framework (Folan & Browne, 2005). The numeric basis of cost makes it attractive as many other gauges of systems measurement (Beamon, 1998). Beamon (1998) also concluded, however, that it is improbable that a unitary performance measure like cost will be sufficient, hence a conglomerate of performance measures is demanded for precise assessment.

The need to further classify research university project performance measures as they relate to performance information management is apparent from the fact that effective performance information management involves more than merely quantifying usefulness or benefit as the produced outcome of any organizational undertaking (Macbryde & Mendibil, 2003). Monitoring the processes responsible for those outcomes themselves is...

To continue reading