The Journal of Systems and Software 70 (2004) 165–176
www.elsevier.com/locate/jss
Research in computer science: an empirical study
V. Ramesh
a
a,*
, Robert L. Glass
b,1
, Iris Vessey
a,2
Department of Information Systems, Kelley School of Business, Indiana University, Bloomington, IN 47405, USA
b
Computing Trends, 1416 Sare Rd., Bloomington, IN 47401, USA
Received 20 May 2002; received in revised form 7 August 2002; accepted 3 November 2002
Abstract
In this paper, we examine the state of computer science (CS) research from the point of view of the following research questions:
1.
2.
3.
4.
5.
What topics do CS researchers address?
What research approaches do CS researchers use?
What research methods do CS researchers use?
On what reference disciplines does CS research depend?
At what levels of analysis do CS researchers conduct research?
To answer these questions, we examined 628 papers published between 1995 and 1999 in 13 leading research journals in the CS field.
Our results suggest that while CS research examines a variety of technical topics it is relatively focused in terms of the level at which
research is conducted as well as the research techniques used. Further, CS research seldom relies on work outside the discipline for
its theoretical foundations. We present our findings as an evaluation of the state of current research and as groundwork for future
CS research efforts.
2003 Elsevier Inc. All rights reserved.
Keywords: Topic ¼ Computing research; Research Approach ¼ Evaluative-Other; Research Method ¼ Literature analysis; Reference Discipline ¼ Not applicable; Level of Analysis ¼ Profession
1. Introduction
Computer science is a well-established discipline that
is represented in almost all institutions of higher education. 3 As part of their faculty responsibilities, computer scientists conduct research in several different
areas, such as artificial intelligence, databases, distributed systems, etc. Research is published in journals
dedicated to fostering research in those specific areas.
*
Corresponding author. Tel.: +1-812-855-2641; fax: +1-812-8554985.
E-mail addresses: venkat@indiana.edu (V. Ramesh), rlglass@acm.
org (R.L. Glass), ivessey@indiana.edu (I. Vessey).
1
Tel.: +1-812-337-8047.
2
Tel.: +1-812-855-3485.
3
Throughout this paper, CS is an abbreviation for Computer
Science unless the context explicitly states otherwise (e.g., where CS is
used to represent Computer System (in Unit/Level of Analysis) or Case
Study (under Research Methods)).
0164-1212/$ - see front matter 2003 Elsevier Inc. All rights reserved.
doi:10.1016/S0164-1212(03)00015-3
Thus, it is not surprising that most papers that examine
the nature of research within computer science tend to
focus on specific areas of computer science (see, for
example, Gruman, 1990; Rice, 1995; Wegner and Doyle,
1996; Gallopoulous and Sameh, 1997) or even subareas, for example, heterogeneous databases (Sheth and
Larson, 1990) or data modeling (Hull and King, 1987),
rather than on the discipline as a whole. From a broader
perspective, we also find articles that address the nature
of computer science research at a country level, e.g.,
Ramamritham (1997) on India and Estivili-Castro
(1995) on Mexico. With the exception of studies by
Glass (1995) and Tichy et al. (1995), however, very few
studies have examined the nature of research in the field
as a whole. And even these studies have a relatively
narrow focus in that they examine only commonly researched topics and/or the research methods used.
Our objective in this study is to provide a detailed
characterization of computer science research, along the
166
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
dimensions identified above, by examining articles
published in major computer science journals from
1995–1999. Our interest in this study goes beyond topic
and research methods and includes other ways of characterizing research such as research approach, which
identifies the way in which a research study is conducted, the level of analysis, which identifies the object
that is studied, and reference discipline, which identifies
the theoretical foundation of the research.
2. The current study
This section describes the classification scheme used
to characterize CS research in this study. It also presents
details regarding the journals examined and the classification process.
2.1. Classification scheme
Given that our objective was to characterize research
in computer science, our first task was to identify a
classification scheme that would enable us to capture the
richness of CS research. We found that traditional
classification schemes such as the ACM computing
classification scheme (ACM CCS, 1998) characterize
research only along one dimension, i.e., topic. Classification schemes in related disciplines such as information
systems, e.g., ISRL categories (Barki et al., 1988), also
tend to focus on topic with research method as a secondary consideration. However, researchers often wish
to know how the findings of studies of interest were
obtained (i.e., the research approach and research
method used). In addition, the level at which a study is
conducted is also of interest to researchers; a study
might, for example, focus on an abstract concept (AC)
such as a data model, or a computing element (CE) such
as an algorithm, or it might focus on a system, a project,
or an organization. Finally, the origin of the studyÕs
theoretical base, the reference discipline, is also interesting to researchers because it may suggest richer conceptualizations of the phenomena of interest.
Because none of the existing classification schemes
was sufficiently rich in the desired dimensions, we developed a multi-faceted classification system that characterizes research along the five dimensions outlined
above. The classification system was comprehensive in a
further way; it was developed to describe research in
three computing-related disciplines: computer science,
software engineering, and information systems (see
Vessey et al., 2001). Thus, some of the categories in our
scheme may be less relevant to mainstream CS research.
For brevity, the classification system is presented with
the results of our study in Tables 3–7. Below, we present
a brief description of how the classification system was
developed.
2.1.1. Classifying topic
To ensure that our list of topics was sufficiently broad
to include all areas of computing research, we used
several sources of topics from the general discipline of
computing, viz., the ACM computing reviews classification scheme (ACM CCS, 1998), the categories in
Barki et al. (1988), and the topic areas identified by
Glass (1992). In particular, we used the classification
scheme proposed by Glass (1992) as the starting point
for arriving at the high-level categories shown in Table
3 because its stated objective of presenting a comprehensive set of topics in the fields of computer science,
software engineering, and information systems best fit
our completeness criterion.
The overall classification scheme, which is shown in
Table 3, divides the topics of the computing research
field into several major categories:
•
•
•
•
•
•
•
•
•
Problem-solving concepts
Computer concepts
Systems/software concepts
Data/information concepts
Problem-domain-specific concepts
Systems/software management concepts
Organizational concepts
Societal concepts
Disciplinary issues
Each of these categories, is further divided into several subordinate categories.
2.1.2. Classifying research approach
We also categorized the research techniques used. We
divided those techniques into research approach, the
overall approach undertaken in performing the research,
and research method, the more detailed techniques used.
In this section, we discuss research approach.
Surprisingly, there is very little information in the
field to aid in the classification of research techniques.
We used Morrison and GeorgeÕs (1995) categorization
of research approaches as a starting point for determining the research approaches to be examined in this
study. Based on an analysis of articles in both software
engineering and information systems between 1986 and
1991, they characterized the four major research approaches as descriptive, developmental, formulative,
and evaluative. These correspond roughly to the scientific method categories of: observe, formulate, and
evaluate (Glass, 1995). We included developmental in
the descriptive category because such research primarily
involved describing systems.
We further subdivided these categories to reflect a
rich set of research approaches. Table 4 shows the
categories used to classify research approach in this
study. The descriptive approach has three subcategories.
Subcategory descriptive-system (DS) is based on Mor-
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
rison and GeorgeÕs descriptive category and is used to
capture papers whose primary focus is describing a
system. Descriptive-other (DO) was added to capture
those papers that used a descriptive approach for describing something other than a system, for example, an
opinion piece. We added descriptive-review (DR) as a
subcategory into which we categorized papers whose
primary content was a review of the literature.
The formulative research approach was subcategorized into a rich set of possible entities being formulated,
including processes/procedures/methods/algorithms (all
categorized under FP), and frameworks and guidelines/
standards (FF and FG, respectively). In all, there are six
subcategories of the formulative research approach.
Our evaluative categories are based on the three
alternative ‘‘evaluative’’ epistemologies identified by
Orlikowski and Baroudi (1991): positivist (evaluativedeductive in our system), interpretive (evaluative-interpretive), and critical (evaluative-critical). We added an
‘‘Other’’ category here to characterize those papers that
have an evaluative component but that did not use any
of the three approaches identified above. For example,
we classified papers that used opinion surveys to gather
data (as opposed to questionnaires that used established
research instruments) under evaluative-other.
2.1.3. Classifying research method
Research method describes the specific technique
used in a given study. While the choice of research approach narrows the set of possible applicable research
methods, there is typically a one-to-many relationship
between a given research approach and method. Hence,
in addition to research approach, we also coded the
detailed technique used by a study.
Unlike research approach, where there were few
candidate categories from which to choose, in the case
of research method, there were numerous classifications
from which to choose. Recall that, while the objective of
this paper is to characterize the nature of research in
computer science, the categories and taxonomies used in
this paper were intended to cover the whole of the
computing field, including computer science.
Arguably, the computing discipline most concerned
with research method is Information Systems where
many prior publications have identified a number of
commonly used methods (see, for example, Alavi and
Carlson, 1992; Farhoomand and Drury, 2000). These
articles identify, for example, laboratory experiments
(using human subjects), field studies, case studies, and
field experiment. Several other research methods have
also been identified; for example, conceptual analysis (or
conceptual study), literature review (Lai and Mahapatra, 1997), instrument development (Alavi and Carlson, 1992), and exploratory survey (Cheon et al., 1993).
Some studies have examined research methods specific to a software engineering context. Both Zelkowitz
167
and Wallace (1997) and Harrison and Wells (2000)
proposed a number of research methods similar to those
identified in the information systems studies cited above.
In addition, we are aware of two papers that address
research methods in both computer science and software
engineering. Glass (1995), for example, suggested a
fairly simplistic approach, derived from prior literature,
which categorized methods as scientific, engineering,
empirical, and analytical. Tichy et al. (1995) conducted a
more general survey of articles in CS journal and conferences and found that CS research was lacking in its
use of experimental methods.
To assist in the categorization of the CS component
of computing research, we added the following categories to the above list: conceptual analysis/mathematical
(CA/M) and mathematical proof to facilitate the classification of papers that utilize mathematical techniques;
Simulation, to allow categorization of papers that utilized simulation as their primary research method; and
concept implementation for papers whose prime research method was to demonstrate proof of a concept by
building a prototype system. We also added the category
laboratory experiment (software) to characterize those
papers that, for example, compare the performance of a
newly-proposed system with other (existing) systems. It
is important to note that not all of the research methods
included in Table 5 are appropriate for computer science
research.
2.1.4. Classifying unit/level of analysis
Level of analysis refers to the notion that research
work may be conducted at one or more of several levels;
for example, at a high level, the research may be technical or behavioral in nature. Example of technical research would be focused on the computing system (CS),
computing element (CE, representing a program, component, algorithm, or object) or abstract concept level
(AC, e.g., graph-based representations). An example of
behavioral research is the Watts Humphrey work on
Team Software Process (http://www.sei.cmu.edu/tsp/
tsp.html), which would be categorized as GP for Group/
Team, and his Personal Software Process work, which
would be categorized as IN for individual (http://
www.sei.cmu.edu/tsp/psp.html). Some research work is
done at the level of the profession (PRO), of which this
paper is an example, as are those papers referenced in
the introduction that address CS research in a particular
country, while others may be conducted within an enterprise at the organizational (OC) level. Table 6 presents the levels of analysis used in this study.
2.1.5. Classifying reference discipline
By reference discipline, we mean any discipline outside the CS field that CS researchers have relied upon for
theory and/or concepts. Generally, a reference discipline
is one that provides an important basis, such as theory,
168
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
for the research work being conducted. Various researchers have characterized the reference disciplines
used in research (see, for example, Swanson and Ramiller, 1993; Westin et al., 1994). Swanson and Ramiller
(1993) identified computer science, management science,
and cognitive science, organizational science, and economics as four key reference disciplines for information
systems. Barki et al. (1988) also include behavioral science, organizational theory, management theory, language theories, artificial intelligence, ergonomics,
political science, and psychology, while Westin et al.
(1994) further identified mathematics/statistics and engineering. The reference discipline categories presented
in Table 7 represent a comprehensive aggregation of the
categories addressed in prior research, i.e., some of our
categories subsumed one or more of the categories
outlined above. The management category, for example, subsumes organizational theory and management
theory. Similarly, artificial intelligence is subsumed
within computer science.
2.2. Journal and article selection
For our study to truly reflect the field of computer
science, we needed to ensure that we evaluated a representative sample of research articles. We began with
the ACM and IEEE journals that Geist et al. used in
their 1996 study of faculty productivity and eliminated
two software engineering journals (ACM Transactions
on Software Engineering and Methodology and IEEE
Transactions on Software Engineering) as well as IEEE
Transactions on VLSI, which does not appear in the list
of IEEE Computer Society publications. 4 Table 1 presents the 13 journals examined.
We used a sampling approach that enabled us to select approximately 500 articles for evaluation. We
wanted to ensure that, as a group, the two primary
publication outlets, IEEE and ACM Transactions were
reflected equally in the sample set. 5 Based on the
number of articles published during the years 1995–
1999, inclusive, we selected 1 in 10 articles from the
IEEE journals and 1 in 3 articles from ACM journals.
This approach resulted in approximately 309 articles in
IEEE journals and 286 articles in ACM journals, as well
as 33 articles from a joint IEEE/ACM publication
Table 1
The journals examined
Journal title
Abbreviation
IEEE Transactions on Computers
Journal of the ACM
IEEE Transactions on Knowledge and
Data Engineering
IEEE Transactions on Pattern Analysis and
Machine Intelligence
IEEE Transactions on Parallel and Distributed
Systems
ACM Transactions on Human–Computer
Interaction
ACM Transactions on Database Systems
ACM Transactions on Graphics
ACM Transactions on Information Systems
ACM Transactions on Modeling and Computer
Simulation
IEEE/ACM Transactions on Networking
ACM Transactions on Programming Languages
and Systems
IEEE Transactions on Visualization and Computer
Graphics
COMP
JACM
KDE
PAMI
PDS
TOCHI
TODS
TOG
TOIS
TOMCS
TON
TOPLAS
VCG
(Transactions on Networking). Table 2 presents raw data
for the number of articles examined in each of the
journals during the five-year period.
2.3. The classification process
Two of the three authors of this paper independently
classified each article using just one category in each of
the five characteristics. Hence the coding reflected the
primary focus of the research. Following the individual
codings, the first author resolved differences by reexamining the article and choosing a final coding that
was typically one of the two original codings.
Agreement varied among categories. For example,
high levels of agreement were achieved for research
method and reference discipline coding (close to 90%),
while coding of level of analysis and topic was somewhat
more problematic (70% and 75% agreement, respectively). Disagreement occurred most often when a paper
could legitimately have been coded in more than one
way. Original agreements across all categories averaged
around 80%.
3. Findings
4
ACM Transactions on Software Engineering and Methodology
(TOSEM) and IEEE Transactions on Software Engineering (TSE) were
not included in this study because they are primarily software
engineering journals and were therefore examined in our analysis of
the software engineering literature. This analysis is reported in Glass
et al. (2002).
5
An alternative distribution scheme based on the amount of
research published could also have been used here. However, we did
not want our sample to be overwhelmed by publications in IEEE
Transactions, which publish more issues per year and more articles per
issue than ACM Transactions.
In the following section the study findings are presented by research question; that is, we address the
topics, research approaches, research methods, levels of
analysis, and reference disciplines that CS researchers
use, in turn. Tables 3–7 summarize the findings. Although this study was designed to characterize research
in the CS discipline, it is also interesting to examine
differences in the journals themselves. Hence, in each of
the sections, we also highlight the findings by journal.
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
169
Table 2
Numbers of publications examined by journal and year
Overall
COMP
JACM
KDE
PAMI
PDS
TOCHI
1995
1996
1997
1998
1999
141
128
135
122
102
21
20
18
17
15
20
16
16
15
11
10
10
10
8
8
17
17
19
19
17
16
14
14
12
10
4
7
6
5
3
Totals
628
91
78
46
89
66
25
Table 8 presents the data by journal for each of the
categories examined. While some of the results for the
discipline as a whole and the journals are somewhat
predictable, some are fairly surprising.
3.1. Findings for topic
Table 3 shows that research in computer science is
spread evenly among the five categories: computer
concepts (28.67%), problem-domain-specific concepts
(21.50%), systems/software concepts (19.11%), data/
information concepts (15.45%), and problem-solving
concepts (14.65%). Two other categories, systems/software management concepts, and organizational concepts, are represented minimally, while two categories,
societal concepts and disciplinary issues are not represented at all.
The leading sub-category was computer graphics/
pattern analysis within the problem-domain-specific
concepts category. Twenty percent of articles were devoted to this category, while 17.68% were devoted to
inter-computer communication (part of computer concepts), which includes such topics as networking and
distributed systems. Other notable topics were computer/hardware principles/architecture at 10.19% (again
part of computer concepts) and database/warehouse/
mart organization at 8.44% (part of data/information
concepts), while papers focusing on mathematics/computational science (part of problem-solving concepts)
were next at 6.69%.
Table 8 (Panel A) presents the topics by journal. The
results show that most journals tended to have a single
dominant topic as suggested by their title. These topics,
then, broadly define the sub-fields that make up the
discipline of computer science. We found that 2 or 3 of
the 13 journals typically focused on the same topic area.
For example, the principal topic category in Journal of
the ACM and ACM Transactions on Modeling and
Computer Simulation was problem-solving concepts; in
IEEE Transactions on Computers, IEEE Transactions on
Parallel and Distributed Systems, and IEEE/ACM
Transactions on Networking it was computer concepts;
in ACM Transactions on Computer–Human Interaction
and ACM Transactions on Programming Languages and
Systems, it was systems/software concepts, in IEEE
Transactions on Knowledge and Data Engineering, ACM
TODS
TOG
TOIS
TOMCS
TON
TOPLAS
VCG
10
3
7
5
3
7
7
7
6
4
9
7
6
7
6
6
6
9
8
8
4
8
9
6
6
13
9
11
11
8
4
4
3
3
3
28
31
35
37
33
52
17
Transactions on Database Systems, and ACM Transactions on Information Systems, it was data/information
concepts, and in IEEE Transactions on Pattern Analysis
and Machine Intelligence, ACM Transactions on Graphics, and IEEE Transactions on Visualization and
Computer Graphics, it was problem-domain-specific
concepts.
3.2. Findings for research approach
Table 4 shows the primary research approaches used
by CS researchers. Formulative was by far the dominant
research approach representing 79.15% of the papers
assessed, followed by evaluative and descriptive approaches, which were virtually equivalent at 10.98% and
9.88%, respectively.
Examination of the sub-categories of research approach shows that FP, a multifaceted subcategory that
includes formulating processes, procedures, methods, or
algorithms is the most important of the formulative subcategories. Approximately half of computer science research (50.55%) fell into this category. The next largest
category was FC (e.g., formulating a concept such as a
data model), at 17.04%. Papers whose primary focus
was evaluation using techniques other than deductive,
interpretive, or critical approaches (evaluative-other)
were third at 9.87%.
Table 8 (Panel B) shows the primary research approaches by journal. The data shows that FP (formulate-process, method, or algorithm) was the most
important research approach in 12 of the 13 journals
examined while formulate-concept (FC) was the second
most important approach (in 8 out of those 12 journals).
ACM Transactions on Computer–Human Interaction was
the only journal in which the formulative research
category did not dominate. Instead, 40% of the papers in
that journal were devoted to evaluative studies (evaluative-deductive and evaluative-other at 20% each), with
a further 20% devoted to system descriptions (DS).
Other journals with significant numbers of evaluative
studies were ACM Transactions on Programming Languages and Systems (21.15%) and Journal of the ACM
(20.51%).
Our results suggest, therefore, that the focus in most
areas of computer science research is primarily on formulating things.
170
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Table 3
Findings for computing topics
1.0
1.1
1.2
1.3
1.4
Problem-solving concepts
Algorithms
Mathematics/computational science
Methodologies (object, function/process, information/data, event, business rules,. . .)
Artificial intelligence
14.65%
5.57%
6.69%
–
2.39%
2.0
2.1
2.2
2.3
2.4
Computer concepts
Computer/hardware principles/architecture
Inter-computer communication (networks, distributed systems)
Operating systems (as an augmentation of hardware)
Machine/assembler-level data/instructions
28.67%
10.19%
17.68%
0.80%
–
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Systems/software concepts
System architecture/engineering
Software life-cycle/engineering (including requirements, design, coding, testing, maintenance)
Programming languages
Methods/techniques (including reuse, patterns, parallel processing, process models, data models. . .)
Tools (including compilers, debuggers)
Product quality (including performance, fault tolerance)
Human–computer interaction
System security
19.11%
0.48%
–
3.82%
3.82%
5.25%
1.75%
3.18%
0.80%
4.0
4.1
4.2
4.3
4.4
4.5
Data/information concepts
Data/file structures
Data base/warehouse/mart organization
Information retrieval
Data analysis
Data security
15.45%
1.91%
8.44%
3.98%
0.64%
0.48%
5.0
5.1
5.2
5.3
5.4
5.5
Problem-domain-specific concepts (use as a secondary subject, if applicable, or as a primary subject if there is no other choice)
Scientific/engineering (including bio-informatics)
Information systems (including decision support, group support systems, expert systems)
Systems programming
Real-time (including robotics)
Computer graphics/pattern analysis
21.50%
0.48%
0.64%
–
0.16%
20.22%
6.0
6.1
6.2
6.3
6.4
Systems/software management concepts
Project/product management (including risk management)
Process management
Measurement/metrics (development and use)
Personnel issues
0.32%
0.32%
–
–
–
7.0
7.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
7.9
7.11
7.11
7.12
Organizational concepts
Organizational structure
Strategy
Alignment (including business process reengineering)
Organizational learning /knowledge management
Technology transfer (including innovation, acceptance, adoption, diffusion)
Change management
Information technology implementation
Information technology usage/operation
Management of ‘‘computing’’ function
IT impact
Computing/information as a business
Legal/ethical/cultural/political (organizational) implications
0.32%
–
–
–
–
0.16%
–
–
–
0.16%
–
–
–
8.0
8.1
8.2
8.3
8.4
Societal concepts
Cultural implications
Legal implications
Ethical implications
Political implications
–
–
–
–
–
9.0
9.1
9.2
Disciplinary issues
‘‘Computing’’ research
‘‘Computing’’ curriculum/teaching
–
–
–
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Table 4
Findings for research approach
Descriptive:
DS
Descriptive-system
DO
Descriptive-other
DR
Review of literature
Table 7
Findings for reference discipline
9.88%
4.14%
5.10%
0.64%
Evaluative:
ED
EI
EC
EO
Evaluative-deductive
Evaluative-interpretive
Evaluative-critical
Evaluative-other
10.98%
1.11%
–
–
9.87%
Formulative:
FF
FG
FM
FP
FT
FC
Formulative-framework
Formulative-guidelines/standards
Formulative-model
Formulative-process, method, algorithm
Formulative-classification/taxonomy
Formulative-concept
79.15%
2.39%
0.64%
5.73%
52.55%
0.80%
17.04%
Table 5
Findings for research method
AR
CA
CAM
CI
CS
DA
DI
ET
FE
FS
GT
HE
ID
LH
LR
LS
MA
MP
PA
PH
SI
SU
Action research
Conceptual analysis
Conceptual analysis/mathematical
Concept implementation (proof of concept)
Case study
Data analysis
Discourse analysis
Ethnography
Field experiment
Field study
Grounded theory
Hermeneutics
Instrument development
Laboratory experiment (human subjects)
Literature review/analysis
Laboratory experiment (software)
Meta-analysis
Mathematical proof
Protocol analysis
Phenomenology
Simulation
Descriptive/exploratory survey
171
–
15.13%
73.41%
2.87%
0.16%
0.16%
–
–
–
0.16%
–
–
–
1.75%
0.32%
1.91%
–
2.39%
–
–
1.75%
–
CP
SB
CS
SC
EN
EC
LS
MG
MS
PA
PS
MA
OT
Cognitive psychology
Social and behavioral science
Computer science
Science
Engineering
Economics
Library science
Management
Management science
Public administration
Political science
Mathematics
Other
0.80%
–
89.33%
0.96%
–
–
–
–
–
–
–
8.60%
0.32%
3.3. Findings for research method
Table 5 presents the primary research methods used
by CS researchers. Conceptual Analysis/Mathematical
(CA/M) (73.41%) was the primary research method with
conceptual analysis (not using mathematical techniques)
next at 15.13%. Categories such as laboratory experiment (using human subjects), laboratory experiment
(software), simulation, and concept implementation are
also represented, although none reached double-digits.
Table 8 (Panel C) shows the findings for research
method by journal. CA/M was the most important research method in all journals except ACM Transactions
on Computer–Human Interaction (TOCHI). The figures
ranged from a low of 37.14% in ACM Transactions on
Information Systems (TOIS) to a high of 90.32% in
ACM Transactions on Graphics (TOG). In TOCHI,
which published no studies using CA/M, the leading
research methods were conceptual analysis (40%) and
laboratory experiment (36%). Concept implementation
as a research method was primarily used in TOCHI
(16%) and TOIS (11.43%). TOIS was also the only
journal in which comparative studies of systems (laboratory experiment (software)) was used as the primary
research method (14.29%).
3.4. Findings for level of analysis
Table 6
Findings for level of analysis
SOC
PRO
EXT
OC
PR
GP
IN
CS
CE
AC
Society
Profession
External business context
Organizational context
Project
Group/team
Individual
Computing system
Computing element––program, component,
algorithm
Abstract concept
–
0.32%
–
–
–
–
1.91%
5.57%
53.34%
38.85%
Table 6 presents the levels of analysis used by CS
researchers. It shows that, similar to research approach
and research method, CS research was also relatively
focused in terms of levels of analysis. The most dominant level of analysis was the Computing Element (CE)
category (53.34%), which relates to algorithms, methods, and techniques, e.g., a scheduling algorithm for a
crossbar switch. The Abstract Concept (AC) category,
which relates to concepts such as the definition of global
predicates in the context of distributed computations,
was the next largest at 38.85%. Finally, 5.57% of the
papers focused on the computing system (CS) level. Two
other categories (individual (IN) and profession (PRO))
COMP
JACM
KDE
PAMI
PDS
TOCHI
14.65%
28.66%
19.11%
15.45%
21.50%
0.32%
0.32%
–
–
12.09%
69.23%
16.48%
1.10%
1.10%
–
–
–
–
44.87%
26.92%
11.54%
16.67%
–
–
–
–
–
28.26%
2.17%
6.52%
60.87%
2.17%
–
–
–
–
5.62%
–
–
4.49%
89.89%
–
–
–
–
6.06%
68.18%
22.73%
1.51%
1.52%
–
–
–
–
–
–
80.00%
8.00%
8.00%
–
4.00%
–
–
Panel B:
DO
DR
DS
ED
EI
EO
EC
FC
FF
FG
FM
FP
FT
5.10%
0.64%
4.14%
1.11%
–
9.87%
–
17.04%
2.39%
0.64%
5.73%
52.55%
0.80%
5.49%
–
3.30%
–
–
7.69%
–
23.08%
–
3.30%
6.59%
50.55%
–
–
–
–
–
–
20.51%
–
29.49%
1.28%
–
1.28%
46.15%
1.28%
2.17%
4.35%
6.52%
–
–
6.52%
–
17.39%
2.17%
–
13.04%
47.83%
–
5.62%
1.12%
2.25%
–
–
6.74%
–
13.48%
8.99%
–
5.62%
56.18%
–
–
–
1.52%
–
–
3.03%
–
7.58%
–
–
7.58%
80.30%
–
Panel C:
AR
CA
CAM
CI
CS
DA
ET
FE
FS
GT
HE
ID
LH
LR
LS
MP
PA
SI
SU
–
15.13%
73.41%
2.87%
0.16%
0.16%
–
–
0.16%
–
–
–
1.75%
0.32%
1.91%
2.39%
–
1.75%
–
–
19.78%
69.23%
2.20%
–
–
–
–
–
–
–
–
–
1.10%
1.10%
4.40%
–
2.20%
–
–
–
89.74%
–
–
–
–
–
–
–
–
–
–
–
–
10.26%
–
–
–
–
36.96%
52.17%
4.35%
–
–
–
–
–
–
–
–
–
–
–
–
–
6.52%
–
–
4.49%
89.89%
–
–
–
–
–
–
–
–
–
–
1.12%
1.12%
3.37%
–
–
–
–
12.12%
81.82%
–
–
–
–
–
–
–
–
–
–
–
1.52%
–
–
4.55%
–
TODS
TOG
TOIS
TOMCS
TON
TOPLAS
VCG
–
–
3.57%
96.43%
–
–
–
–
–
3.23%
–
6.45%
–
90.32%
–
–
–
–
2.86%
–
17.14%
54.29%
17.14%
5.71%
2.86%
–
–
51.35%
37.84%
5.41%
2.70%
2.70%
–
–
–
–
–
96.97%
3.03%
–
–
–
–
–
–
1.92%
7.69%
88.46%
1.92%
–
–
–
–
–
11.76%
–
–
–
88.24
–
–
–
–
–
4.00%
20.00%
20.00%
–
20.00%
–
8.00%
–
4.00%
4.00%
16.00%
4.00%
7.14%
–
3.57%
–
–
7.14%
–
17.86%
3.57%
–
–
50.00%
10.71%
3.23%
–
–
6.45%
–
–
–
16.13%
–
–
3.23%
70.97%
–
8.57%
–
11.43%
–
–
8.57%
–
8.57%
8.57%
–
17.14%
37.14%
–
24.32%
–
10.81%
–
–
10.81%
–
18.92%
–
–
2.70%
32.43%
–
3.03%
–
–
–
–
9.09%
–
3.03%
–
–
12.12%
72.73%
–
7.69%
–
1.92%
–
–
21.15%
–
28.85%
1.92%
–
–
38.46%
–
5.88%
–
11.76%
–
–
–
–
–
–
–
–
82.35%
–
–
40.00%
–
16.00%
–
–
–
–
4.00%
–
–
–
36.00%
–
4.00%
–
–
–
–
–
10.71%
78.57%
3.57%
–
–
–
–
–
–
–
–
–
–
7.14%
–
–
–
–
–
3.23%
90.32%
–
–
–
–
–
–
–
–
–
6.45%
–
–
–
–
–
–
–
34.29%
37.14%
11.43%
–
2.86%
–
–
–
–
–
–
–
–
14.29%
–
–
–
–
–
16.22%
72.97%
5.41%
2.70%
–
–
–
–
–
–
–
–
–
–
–
–
2.70%
–
–
24.24%
69.70%
–
–
–
–
–
–
–
–
–
–
–
–
–
–
6.06%
–
–
7.69%
86.54%
3.85%
–
–
–
–
–
–
–
–
–
–
1.92%
–
–
–
–
–
23.53%
70.59%
5.88%
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Overall
Panel A:
Prob-solving
Computer
Systs/SW
Data/Info
Prob-domain
Sys/SW Mgt
OrgÕal
Societal
Disc issues
172
Table 8
Representation by topic (Panel A), representation by reasearch approach (Panel B), representation by research method (Panel C); representation by level of analysis (Panel D), representation by
reference discipline (Panel E)
173
–
3.03%
–
96.97%
–
–
–
–
–
–
–
5.77%
–
94.23%
–
–
–
–
–
–
–
–
–
82.35%
–
–
–
–
–
17.65%
were below 2%, while the five categories of societal,
organizational context, external business context, project, and group/team were not represented.
Table 8 (Panel D) presents level of analysis by journal. The data shows that CE was the primary level of
analysis in 8 of the 13 journals. The figures ranged from
a low of 51.69% in IEEE Transactions on Pattern
Analysis and Machine Intelligence to a high of 88.24%
in IEEE Transactions on Visualization and Computer
Graphics (VCG). Further, AC was the primary level of
analysis in four journals ranging from 42.86% to
56.04%, while Individual was the primary level of analysis in ACM Transactions on Computer–Human Interaction (TOCHI). In addition, TOCHI and ACM
Transactions on Graphics (TOG) were the only journals
to publish articles that used a non-technical level of
analysis (i.e., levels of analysis other than AC, CS 6 or
CE) with 40% of the articles in TOCHI and 6.45% of the
articles in TOG focusing on the individual level.
Table 1 presents the journal titles; The shaded areas represent the highest percentage and 2nd highest percentage, respectively.
–
18.92%
–
81.08%
–
–
–
–
–
–
–
2.86%
–
97.14%
–
–
–
–
–
–
–
6.45%
–
90.32%
–
–
–
–
–
3.23%
–
41.03%
–
57.69%
–
–
–
–
–
1.28%
0.80%
8.60%
–
89.33%
–
–
–
–
0.32%
0.96%
Panel E:
CP
MA
SB
CS
EC
IS
MG
MS
OT
SC
–
1.10%
–
98.90%
–
–
–
–
–
–
–
–
–
97.83%
–
–
–
–
–
2.17%
–
7.87%
–
92.13%
–
–
–
–
–
–
–
–
–
100.00%
–
–
–
–
–
–
20.00%
–
–
72.00%
–
–
–
–
8.00%
–
–
–
–
100.00%
–
–
–
–
–
–
–
–
–
–
–
–
–
42.86%
17.14%
40.00%
–
–
–
–
–
–
–
42.31%
–
57.69%
–
0.32%
–
–
–
–
1.91%
38.85%
5.57%
53.34%
Panel D:
SOC
PRO
EXT
OC
PR
GP
IN
AC
CS
CE
–
–
–
–
–
–
–
56.04%
4.40%
39.56%
–
2.17%
–
–
–
–
–
34.78%
10.87%
52.17%
–
1.12%
–
–
–
–
–
44.94%
2.25%
51.69%
–
–
–
–
–
–
–
12.12%
–
87.88%
–
–
–
–
–
–
40.00%
24.00%
20.00%
16.00%
–
–
–
–
–
–
–
35.71%
3.57%
60.71%
–
–
–
–
–
–
6.45%
22.58%
–
70.97%
–
–
–
–
–
–
–
45.95%
16.22%
37.84%
–
–
–
–
–
–
–
36.36%
–
63.84%
–
–
–
–
–
–
–
55.77%
7.69%
36.54%
–
–
–
–
–
–
–
–
11.76%
88.24%
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
3.5. Findings for reference discipline
Table 7 presents the reference disciplines used by CS
researchers. The results suggest that, for the most part,
CS research does not rely on other fields for its fundamental theories and/or concepts. Of the papers examined, Computer Science itself was the reference
discipline in 89.33% of the cases. The only other discipline that emerged was mathematics (8.60%). There
were trivial instances of papers that relied on cognitive
psychology (0.80%) and science (0.96%).
Table 8 (Panel E) presents the breakdown of reference
discipline by journal. Not surprisingly, computer science
was the primary reference discipline in all journals,
ranging from a low of 57.69% in Journal of the ACM
(JACM) to a high of 100% in IEEE Transactions on
Parallel and Distributed Systems. Mathematics was a
major reference discipline in JACM with 41% of the articles using concepts directly from that discipline. Only
two journals did not have mathematics as their second
most important reference discipline (TOCHI and VCG).
Cognitive psychology emerged as a major reference discipline in TOCHI (20%) and Science in VCG (17.65%).
4. Discussion and implications
In this study, we sought to analyze the characteristics
of computer science research according to five research
characteristics all of which are recorded in the literature
as being important aspects of any research study. We first
provide a brief summary of the key findings, followed by
a discussion of the some of the limitations of our study.
6
Computer System.
174
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
CS research is fairly evenly distributed across five
major topic areas: problem-solving concepts, computer
concepts, systems/software concepts, data/information
concepts and problem-domain-specific concepts. The
leading category is computer concepts, with problemdomain-specific concepts (principally computer graphics
and pattern analysis) second. As would be expected,
there is very little work in the area of systems/software
management concepts (two papers), one paper on organizational concepts, and no papers that examined
societal concepts or disciplinary issues.
In terms of both research approach and research
method, CS research tends to be quite focused. The
‘‘formulate’’ research approach category accounts for
almost 80% of the research with a majority of papers
being devoted to formulating a process, method, or algorithm. The preferred research method is conceptual
analysis based on mathematical techniques.
With regard to levels of analysis, CS research falls
primarily into the CE or AC categories confirming that
the mission of CS is to conduct research that is focused
on technical levels of analysis. As would be expected,
very little research focused on the society or profession
categories.
With respect to reference disciplines, our data shows
that CS research seldom relies on research in other
disciplines and in the rare instances that it does, it relies
primarily on mathematics.
Table 9 presents a summary of the most important
research characteristics in each of the 13 journals. The
data indicate that while CS research addresses a diverse
range of topics, there is a high degree of consistency in
terms of the research approaches, research methods, and
levels of analysis used to study these topics. Further,
across the 13 journals studied, ACM Transactions on
Computer–Human Interaction is a clear outlier. It is, for
example, the only journal not to have FP (formulate
process, method or algorithm) as the predominant research approach and CA/M as the predominant research method. From the viewpoint of level of analysis,
CE dominates AC by eight journals to four. It is,
however, interesting to note that each of the four journals in which AC is dominant focuses on one of the
major topic categories; the only topic category that is
not the focus of one or more of the journals we studied is
problem-domain-specific concepts.
Note that we used our classification system to record
the keywords describing this paper (following the abstract). The paper is classified as follows: (a) the topic is
computing research (9.1); (b) the research approach is
EO (evaluative-other) because our paper is about evaluating CS research; (c) the research method is LR (literature review/analysis); (d) the level of analysis is the
profession (PRO); and (e) the reference discipline is
none because we did not use concepts from other disciplines in performing the study. We encourage authors
in the future to use our classification system not only to
select keywords but also to write abstracts. Such a
practice would aid researchers to assess the relevance of
published research to their own endeavors.
A study of this nature is not without limitations. The
first limitation stems from the choice of journals. The
results of our study reflect the nature of computer science research to the extent that these journals are representative of the field. While there are many other
magazines, and high-quality research conferences that
publish CS research articles, we chose to analyze only
articles published in journals because of the traditional
and enduring role that journals play in the development
of academic disciplines. A second potential limitation
arises from the fact that we coded only a sample of the
articles published in the selected journals. Given, however, that we used a systematic sampling procedure, we
have no reason to believe that the results are biased. A
final limitation arises from the subjective nature of the
coding process. We attempted to reduce the subjectivity
by using two independent coders who revisited the
articles to resolve any disagreements. The relatively
high-level of raw agreements suggests that articles were
indeed coded in a consistent manner.
Table 9
Summary of characteristics of journals
Journal
Principal topic
Research approach
Research method
Level of analysis
Reference discipline
TOMCS
JACM
COMP
PDS
TON
TOIS
TODS
KDE
PAMI
TOG
VCG
TOPLAS
TOCHI
Problem-solving
Problem-solving
Computer
Computer
Computer
Data/information
Data/information
Data/information
Problem-domain-specific
Problem-domain-specific
Problem-domain-specific
Systems/software
Systems/software
FP
FP
FP
FP
FP
FP
FP
FP
FP
FP
FP
FP
DS, ED, EO
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA/M
CA
AC
CE
AC
CE
CE
AC
CE
CE
CE
CE
CE
AC
IN
CS
CS
CS
CS
CS
CS
CS
CS
CS
CS
CS
CS
CS
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
5. Conclusion
We examined 628 papers (over a 5-year period) in 13
leading research journals in the CS field from 1995 to
1999 to answer questions regarding the nature of CS
research. We characterized CS research in terms of the
topics, research approaches, research methods, levels of
analysis, and reference disciplines used. Our results
suggest that CS research focuses on a variety of technical topics, using formulative approaches to study new
entities that are either computing elements or abstract
concepts, principally using mathematically-based research methods.
The results from our study should be of value to both
researchers and doctoral students engaged in computer
science research. For example, our study provides a
characterization of the types of articles that computer
science journals publish. Researchers can use this
knowledge to make choices when deciding on a target
journal for their research. Our results can also be used to
provide insights into areas of CS research that are receiving little research attention. For example, in terms of
research approaches, our results clearly suggest that
insufficient emphasis is being placed on the use of
evaluative methodologies. However, while our results
clearly support Tichy et al.Õs (1995) claim regarding the
lack of focus on evaluation in CS research, Fletcher
(1995) cautions that the use of experimental methods
may not always be appropriate in computer science, a
caveat that should be kept in mind.
Further, funding organizations such as NSF could
use the findings of our research to provide focused calls
for proposals aimed at fostering research in particular
areas or using particular approaches/methods. It is important to note, however, that any interpretation of gaps
represented in or findings must take into account the
fact the classification scheme was developed to cover a
broader scope than computer science, alone, by also
including the disciplines of software engineering, and
information systems. Hence, for example, while our results clearly show that there is a lack of emphasis on
organizational aspects of computing, that is the focus of
IS researchers (see Vessey et al., 2002) and does not
necessarily represent opportunities for CS researchers.
We hope that our evaluation of the state of current
CS research fosters future CS research efforts.
References
ACM CCS: ACM Computing Classification System [1998 version],
http://www.acm.org/class/1998, 1998, accessed 11/06/01.
Alavi, M., Carlson, P., 1992. A review of MIS research and
disciplinary development: implications for deans/administrators.
Journal of Management Information Systems 8, 45–62.
175
Barki, H., Rivard, S., Talbot, J., 1988. An information systems
keyword classification scheme. MIS Quarterly 12 (2), 299–
322.
Cheon, M.J., Grover, V., Sabherwal, R., 1993. The evolution of
empirical research in IS: a study in IS maturity. Information and
Management 24 (5), 107–119.
Estivili-Castro, V., 1995. Computer Science research in Mexico. IEEE
Computer 28 (8), 56–62.
Farhoomand, A.F., Drury, D.H., 2000. A Historiographical Examination of Information Systems. Communication of the AIS Vol. 1,
Article 19, 2000.
Fletcher, P., 1995. The role of experiments in Computer Science.
Journal of Systems and Software 30 (1–2), 61–163.
Gallopoulous, E., Sameh, A., 1997. CSE: content and product. IEEE
Computational Science and Engineering 4 (2), 39–43.
Geist, R., Chetuparambil, M., Hedetniemi, S., Turner, A.J., 1996.
Computing research programs in the US. Communications of the
ACM 39 (12), 96–99.
Glass, R.L., 1992. A comparative analysis of the topic areas of
Computer Science, Software Engineering, and Information Systems. Journal of Systems and Software (November), 277–289.
Glass, R.L., 1995. A structure-based critique of contemporary
computing research. Journal of Systems and Software (January).
Glass, R.L., Vessey, I., Ramesh, V., 2002. Research in software
engineering: an analysis of the literature. Information and Software
Technology 44, 491–506.
Gruman, G., 1990. Conference tries to define computing-research
agenda. IEEE Software 7 (1), 127–132.
Harrison, R., Wells, M., 2000. A Meta-Analysis of Multidisciplinary
Research. In: Papers from The Conference on Empirical Assessment in Software Engineering (EASE), pp. 1–15.
Hull, R., King, R., 1987. Semantic database modeling: survey,
applications and research issues. ACM Computing Surveys 19
(3), 201–260.
Lai, V.S., Mahapatra, R.K., 1997. Exploring the research in information technology implementation. Information and Management 32,
187–201.
Morrison, J., George, J.F., 1995. Exploring the Software Engineering
component in MIS research. Communications of the ACM 38 (7),
80–91.
Orlikowski, W., Baroudi, J.J., 1991. Studying information technology
in organizations: research approaches and assumptions. Information Systems Research 2 (1), 1–28.
Ramamritham, K., 1997. Computer Science research in India. IEEE
Computer, 40–47.
Rice, J.R., 1995. Computational science and the future of computing
research. IEEE Computational Science and Engineering 2 (4), 34–
41.
Sheth, A., Larson, J., 1990. Federated database systems for managing
distributed, heterogeneous, and autonomous databases. ACM
Computing Surveys 22 (3), 183–236.
Swanson, E.B., Ramiller, N., 1993. Information systems research
thematics: submissions to a new journal 1987–1992. Information
Systems Research 4 (4), 299–330.
Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E., 1995. Experimental
evaluation in computer science: a quantitative study. Journal of
Systems and Software (January).
Vessey, I, Ramesh, V., Glass, R.L., 2001. Research in Computing
Disciplines: A Comprehensive Classification Scheme, Indiana
University: Working Paper (TR 107-1), http://www.kelley.iu.edu/
ardennis/wp/ (accessed 02/08/02), 2001.
Wegner, P., Doyle, J., 1996. Strategic directions in computing research.
ACM Computing Surveys 28 (4), 565–574.
Westin, S.M., Roy, M., Kim, C.K., 1994. Cross-fertilizations of
knowledge: the case of MIS and its reference disciplines. Information Resources Management Journal 7 (2), 24–34.
176
V. Ramesh et al. / The Journal of Systems and Software 70 (2004) 165–176
Zelkowitz, M.V., Wallace, D., 1997. Experimental validation in
software engineering. Information and Software Technology 39,
735–743.
Ramesh Venkataraman, Ph.D., is an Assistant Professor of Information Systems and Ford Motor Company Teaching Fellow in the Department of Accounting and Information Systems at Kelley School of
Business, Indiana University. He has published over 25 papers in
leading journals, books, and conferences. His areas of expertise are in
database modeling and design, systems design and development, heterogeneous databases, and groupware systems.
Robert L. Glass is president of Computing Trends, publishers of The
Software Practitioner. He has been active in the field of computing and
software for over 45 years, largely in industry (1954–1982 and 1988–
present), but also as an academic (1982–1988). He is the author of over
20 books and 70 papers on computing subjects, Editor Emeritus of
ElsevierÕs Journal of Systems and Software, and a columnist for several
periodicals including Communications of the ACM (the ‘‘Practical
Programmer’’ column) and IEEE Software (‘‘The Loyal Opposition’’).
He was for 15 years a Lecturer for the ACM, and was named a Fellow
of the ACM in 1998. He received an honorary Ph.D. from Linkoping
University in Sweden in 1995. He describes himself by saying ‘‘my head
is in the academic area of computing, but my heart is in its practice.’’
Iris Vessey is Professor of Information Systems at Indiana UniversityÕs
Kelley School of Business, Bloomington. She received her M.Sc.,
MBA, and Ph.D. in MIS from the University of Queensland, Australia. Her research into evaluating emerging information technologies
from both cognitive and analytical perspectives has been published in
journals such as Communications of the ACM, Information Systems
Research, Journal of Management Information Systems, and MIS
Quarterly. In recent years, her interests have focused on managerial
issues associated with the management and implementation of enterprise systems. She currently serves as Secretary of the Association for
Information Systems and of the International Conference on Information Systems and is an AIS Fellow.
