Empirical Studies of Software Engineering Expertise

« Think-aloud books available online | Main | Expertise in Professional Software Design: A Process Study by Sabine Sonnentag »

May 1, 2005

The Effects of Semantic Complexity on Expert and Novice Computer Program Recall and Comprehension by Bernard Guerin and Anthony Matthews

In this paper, the authors wanted to explore the effects of semantic complexity on both recall and comprehension while comparing novices and experts programmers. To do so, they conducted three experiments.

Experiment 1
Purpose: If experts use higher level semantic knowledge to help them understand programs, then removing some of the semantic structure should to lead to performance closer to that of novices
Method:
- 104 subjects (52 experts with 4.7-5 years of programming experience and 52 novices with 0.5 years of experience).
- Subjects received a booklet with a two-page computer program (116 line COBOL program) with 4 versions: the original version, a version with lines randomized within a module but keeping the module order intact, a version with only the modules randomized, and a version with both lines with a module randomized and module order randomized.
- The subjects were tested in groups of two to five.
- Subjects had to take 10 minutes to learn and memorize the program and then write down as many lines or program code that they could remember. Then they had to write down a brief summary of the purpose of the program and how it worked to achieve that purpose.
Results: The experiment provided strong evidence that experts use superior semantic knowledge when memorizing a computer program. However, when line order was randomized, they performed at the same level as a novice.

Experiment 2
Purpose: To see how two versions of a program (semantically complex and semantically simple) affect recall and comprehension.
Method:
- 52 subjects (26 experts and 26 novices)
- Each subject received 2 procedures written in Pascal for pattern-matching (one simple, one complex). In the first program, it is implemented using simple data structures. In the second program, it is implemented using more complex structures (linked lists).
- Subjects were tested in small groups.
- They were given 5 minutes to memorize the program, 5 minutes for recall, and 3 minutes to describe the function and operation of the program.
Results: The complex program was recalled less well than the simple version. Novices performed poorly on both versions. When the complex version was used, experts performed at the level of novices. Expertise was only significant for comprehension.

Experiment 3
Purpose: One reason that experts perform well in the previous experiments is because they already know the keywords so they can memorize and comprehend the program more quickly.
Method:
- To test this idea subjects were given a program to memorize. One half received the original COBOL program and the other half received a version with the keywords replaced by common English words
- 24 novices and 24 experts.
- 10 minutes to memorize, 8 minutes to recall, and 5 minutes to write a brief description of what the program was doing
Results: Removal of the keywords did not affect the experts more than it did the novices for either recall or comprehension. This means that the experts still have an advantage with their semantic knowledge about computer languages.

Overall, the experiments revealed that experts actively search for program functions. The experiments show that experts pay more attention to actual functions than novices do.

This paper relates to our study, because it provides more qualities of experts that we can look for. Moreover, the paper provides examples of how we can examine how our subjects think. It is important not only to monitor what type of code they produce, but also what their general understanding of the problem is. Also, the actual procedures of their experiments can be looked at as an example.

Posted by Justin at May 1, 2005 10:26 PM

Comments

Post a comment