« August 2004 | Main | May 2005 »

April 28, 2005

"The Elements of Expertise" by Steven K. S. Tan

The article explores the characteristics of an expert and attempts to explain how a person earns the label expert. An expert possessing qualities and attributes that account for their performances seem to be a combination of innate characteristics and those that grow from practice and experience. Experts are consistently superior in performance on a specific task. It is also of note that experts, specifically chess masters, don't think ahead further than novices. They might even pick fewer, leaving only superior moves to choose from.

Experts have a significantly larger knowledge base that is both hierarchical and complex. This complexity allows them to retrieve information faster than non-experts. Through this larger knowledge base experts also able to perform many tasks subconsciously and automatically. Although experts are slower in the early stages of problem solving, they are still able to solve problems faster than non-experts.

Other interesting facts from this article include:

This article seems to fall under the cognitive science aspect of our research. There aren't any specific experiments run, but there is some talk about forward reasoning in experts versus backward reasoning being used in non-experts that we should look into further perhaps for our experiment analysis.

Posted by Matthew at 11:15 AM | Comments (0)

April 26, 2005

Week 3 Progress

  1. Read the 4 papers
  2. Took the IRB tutorial
  3. Started doing questions for panel of experts

Posted by Matthew at 11:09 AM | Comments (0)

April 25, 2005

Progress Report Week 3

1. I finished writing the last 2 summaries.
2. Read 2 of the 4 papers I received last week. Might be able to squeeze in a one more before the meeting.
3. Read about 5 or 6 of the 14 papers I found through ACM.

Posted by Teerawat at 1:57 PM | Comments (0)

week 3 progress report

1. read 3 of the 4 papers, since one was mine.
2. researched "popular books", found a few. need to see if the library has them.
3. took the irb tutorial.

Posted by John at 3:01 AM | Comments (0)

Expertise in context: Chapter 6 - Cognitive Conceptions of Expertise

This chapter presents answers to the question “what is it that makes an expert?” Nine views of expertise are presented.
- General-process: This view is based on superiority in general information processing. Experts use different processes from novices to solve a specific problem, or use the same processes but are faster.
- The quantity-of-knowledge: In a study that requires expert and novice chess players to remember chessboard configurations, the results show that differences between expert and novices are not in their ability to general information, but rather in their ability to encode and remember information. Since experts have more knowledge about chessboard configuration than novices do, they can remember and encode them easily than novices.
- The knowledge-organization: In a study of experts in physics, the results show that differences between experts and novices were not only in the amount of knowledge, but also in the way they organized knowledge that allows them to exploit the knowledge easier.
- Superior analytical ability in solving problem: Experts can use knowledge more effective and can infer things from information that novices cannot.
- Superior creative ability: Experts have creative insight that takes information that other people see in one way, but they see in another, and reach insightful solution that others cannot reach. These process includes: 1) filtering relevant and identify irrelevant information; 2) combining information in a way that is not obvious to other people, and 3) applying information acquired from another context to the problem at hand.
- Superior automatization: Experts have more information processed automatically. Automatization frees an expert’s cognitive resources so that he has more resources available for solving novel aspects of the problems.
- Superior practical ability: Experts know how the field operates, and how to navigate within it.
- Being labeled: A person is an expert because he is labeled, and treated as such.
- Synthetic view: Expertise is a prototype that comprises all aspects in other views, and others aspects not presented. Some aspects of an expert in one domain might be irreverent to an expert in other domains. People are experts in varying degrees depending on the extent to which they fulfill the criteria of expertise in their domains.

Posted by Sukanya at 2:37 AM | Comments (0)

Expertise in context:Chapter 4: Experience and expertise: the role of memory in planning for opportunities

This chapter emphasizes the role of specific past experiences in memory that may form the basis of knowledge organization that differentiates experts from novices. Expertise in planning tasks is studied based on hypothesis that "expert planners are able to anticipate the circumstances related to satisfying goals, and therefore maximize their ability to respond to opportunities when they arise”. Three planning models are presented: AI model, Case-based planning model, Opportunism planning model.

AI model requires complete preplanning. Then the effect of the plan is projected on the world model to determine whether the outcome is successful before execution. The model requires the complete knowledge of the world and that the world model is stable. These requirements lead to costly information, complex planning, and lengthy execution time. In addition, many real world tasks require experts to work with uncertain and incomplete information. This model also cannot deal with multiple goals and postponed execution.

Case-based planning model - let past experiences tell the planner when and where plans work and do not work. Reuse past successful plan and avoid failed plan. "By experiencing failures and successes within a domain, expert planner is prepared to deal with future failure and opportunities when they occur. The framework of case-based planning modules that act to anticipate and avoid problems, repair faulty plan and store them. In this model, experts learn from failure by anticipating and avoiding them. However, it does not address the fact that expert also can predict and exploit opportunities as they arise.

Opportunism planning model: planner should be able to shift to a different strategy when notice opportunity even an unanticipated one.
• Demon model: Goals are characterized as agents having their own inferential power. A suspend goal continue to independently examine the ongoing input, realized satisfy condition and insert plan into current agenda. The reason for this model is there is no way to determine at the time of suspending goal in which condition the will be reactivated, and it is difficult to provide description of features that will be used to activate suspended goal.

The author argues that successful recognition of opportunities depend on feature use to index the suspended goal in memory. The model addresses the problem of how people recognize opportunities during plan execution. Power of opportunism rests in the abilities to reason at the time of goal suspension.
• Predictive encoding Model: Planners anticipate features in the world that indicate potential opportunities (condition that success execution should be possible), use features to index goal in memory. Activation of the features while pursue other goals will reactivate the suspended goal. The experience acquired in a domain allows experts to anticipate the circumstances that are related to opportunities to satisfy goal, and easily describe them in easily observable feature. The more one learn about the circumstances, the better one will be able to predict the feature and encode them.
The predictive encoding model is verified using “Trucker” computer simulation and an empirical study whose results support the model. Trucker involves receiving delivery requests, making decisions about which truck to assign a request, and which order parcels should be picked up. The empirical involves commonsense planning in a dorm room using college students as subjects. The results confirm that: 1) recognition is likely to occur based on features that could be anticipated from planning; 2) features associated with opportunities at the suspension time is more likely to lead to pending goal; 3) opportunities not anticipated may be missed; and features requiring inferential links to connect to goals is less likely to lead to recognition.

Posted by Sukanya at 2:35 AM | Comments (0)

April 24, 2005

Week 3 Progress

1. Read all the papers.
2. Took IRB Tutorial Test.
3. I found a couple of psych papers on expertise, but some of the findings weren't entirely relevant to our topic. Wrote up summaries for 1 psych paper concering expertise and 1 paper from the ESP.
4. I have a general idea for the "Expections" for the background of our study. (i.e. Experts will use "schemas" to solve problems, while novices will focus on more concrete information)

Posted by Justin at 11:32 PM | Comments (0)

Comprehension Differences in Debugging by Skilled and Novice Programmers by Leo Gugerty and Gary M. Olson (from the First Workshop of Empirical Studies of Programmers)

In this study, experts and novices are compared in terms of program debugging. Experts were those who were advanced graduate students in computer science and novices were just finishing their first or second course in Pascal. The study focused on two programming languages: LOGO and Pascal. In this first experiment, all of the subjects, both experts and novices, had no knowledge of LOGO. Subjects were taught the basics of the programming language. Following training, each subject had to debug three LOGO programs. In the second experiment, all of the subjects had to debug a Pascal program.

The results from the study suggest that experts perform better when programming because they could determine what the program did faster. Novices, on the other hand, did a poorer job because they lacked sufficient programming language and introduced new bugs. Due to the more advanced domain knowledge which they possessed, experts could solve a problem more quickly.

This paper is relevant to our study because it provides a basic setup for conducting an experiment to compare expert and novice programmers. Each subject was not only monitored but also asked questions concerning their understanding of the programs. Also, an observer recorded where the subject was looking (i.e. the screen, keyboard, notepad, etc) at particular times. This study emphasizes the importance of tracking as much information as possible with corresponding timestamps.

Posted by Justin at 11:30 PM | Comments (0)

Week 3 Progress

1. Read all 4 papers that were given to read.
2. Took tutorial "test" for IRB
3. Found 2 more papers regarding programming aptitude tests.
4. Thought about tasks on what to do for experiments and measurement of performance.

Posted by Derrick at 10:36 PM | Comments (0)

Maps as Representations: Expert Novice Comparison of Projection Understanding by Anderson and Leinhardt

This paper focuses on a study comparing the understanding of maps by experts and novices. The study divided subjects into four groups: experts, advanced novices, novices, and teachers. Experts had at least 10 years of experience in geography. Advanced Novices were geography undergraduates who had taken at least two cartography classes. Novices were geography undergraduates who were enrolled in their first geography class. Teachers were social studies teachers with no geography experience.

The subjects were given a map and asked to draw a line on the map to indicate the shortest distance between pairs of cities. The subjects were then given a second map and asked to draw lines of shortest distance between three cities in a particular order. The subjects were then asked if additional information may have helped accomplish the tasks.

Although the subject matter in this paper has little to do with software engineering, this paper is relevant to our study because, in the end, the findings were similar to those found in the examination of experts and novices computer science. In this study, experts approached problems by using rules obtained through experience. Experts could solve problems more quickly because they retrieved pre-established solutions which were used in similar problems. Likewise, advanced novices, with similar knowledge, were able to use rules to generate solutions. Novices and teachers, however, could not solve problems correctly because they lacked the basic rules for problem solving. Many times novices and teachers used incorrect rules to infer more incorrect rules. Overall, this particular examination of experts and novices show that similar observations and conclusions concerning expertise appear can appear in very different subject areas.

EDIT: I found similar observations in different subject areas in the following papers as well:

The Role of Visual Representations in Advanced Mathematical Problem Solving: An Examination of Expert–Novice Similarities and Differences by Stylianou and Silver

Expert–Novice Differences in the Understanding and Explanation of Complex Political Conflicts by Jones and Read

Posted by Justin at 9:43 PM | Comments (0)

Frequently Abused Words

In the various write-ups that I've read in the last couple of weeks, I've found a couple terms that are frequently misuse, not just by the students on this project, but in general.

Methodology is the study of methods, e.g., research methods, research designs, and software development methods. Unfortunately, this word is often used as a drop-in replacement for "method," which is incorrect.

Case study is a term multiple meanings. In the context of research methods, it's an analytic method, suitable for building explanations and theories. In medicine and law, cases or patient/client histories are studied for lessons learned. This definition often leaks over into software engineering. Worse yet, sometimes these two terms are conflated into a shoddy empirical study.

I'm sure there are others, but these are the ones that have cropped up so far.

Susan

Posted by ses at 6:51 PM | Comments (0)

What is a "popular" book?

A popular book is not an academic book. It's usually written for a general audience, i.e. the average software developer. The author is typically somebody with a lot of industry experience, rather than a professor or researcher. You can usually find popular books on Amazon. Some publishers include Addison-Wesley, Microsoft Press, Dorset House. Conference proceedings and other works by IEEE and ACM are not considered popular books.

Susan

Posted by ses at 6:46 PM | Comments (0)

April 21, 2005

"The Stories of Expert and Novice Student Teachers' Supervisors: Perspectives on Professional Development" by Lya Kremer-Hayon

Like the title says this paper was pretty much a compilation of interviews of supervisors. The participants were chosen through recommendations and experience. The participants consisted of 3 experts and 3 novices. One participant from each group was associated with a Kibbutz college and the rest were from State colleges. The inclusion of two different institutions was intentional because they wanted to observe possible effects caused by environmental factors.

The interviews were aimed at discovering the development changes that occur as expertise is acquired. The results of the interview were split into seven categories that were exhibited by each interviewee. The categories are: Knowledge and Change (KC), Reflection and Criticism (RC), Human Relations (HR), Theory and Research (TR), Ideologies and Values (IV), and Difficulties (D). KC, RC, and TR were classified as cognitive aspects, HR and IV were classified as affective aspects, and D covered both aspects. They discovered that experts displayed more characteristics in the cognitive aspects. Novices also displayed characteristics in coginitive aspects but at a smaller percentage than experts. These results showed that cognitive skills developed with expertise for supervisors.

Even though this paper has nothing to do with software engineering, the concept that could be relevant to our study is environmental factors. Environmental factors are something to consider when we select subjects for the test. Different policies and procedures at different companies could have an effect on how a software engineer develops.

Posted by Teerawat at 2:16 AM | Comments (0)

"Knowledge restructuring and the acquisition of programming expertise" by Simon P. Davies

This paper tries to introduce a model which would explain the development in expertise that separates an expert from a novice. The currently existing model associates schemata acquisition with skill development. However, in this paper they introduce a new model that emphasizes knowledge restructuring processes over schemata acquisition. The idea of knowledge restructuring processes is that the expert programmer would develop the focal points of the program first and then build the rest of the program around those points.

In order to support their model, they devised an experiment that consisted of three groups of equal size and different skill level. One group consisted of novices, another of intermediates, and the last being experts. Each group was presented with four Pascal programs and they had 10 minutes to study each program. After they have had a chance to study all four programs, they were presented with a series of 40 probe items that were either focal lines or non-focal lines. For each probe item they were asked to state whether the probe item was present in any of the four programs. The results revealed that experts had the highest percentage of correct results as well as the fastest response time, novices had the lowest percentage correct and the slowest response time, and intermediates were in the middle for both. The results also revealed that the differences in response time and percentage of correct answers were significant for focal lines, but not significant for non-focal lines. This basically supported their model by showing that experts are more aware of focal lines in a program since they build programs starting with focal points due to knowledge restructuring processes.

This paper is relevant to our research in regards to the results that show experts are more observant of focal lines in a program. This just shows us another way that differentiates an expert from a novice. By giving us a characteristic that differentiates the two groups, we can take note of it in our own experiment as well.

Posted by Teerawat at 1:22 AM | Comments (0)

April 18, 2005

Analysing the Novice Analyst: Cognitve Models in Software Engineering by A. G. Sutcliffe and N. A. M. Maiden

This empirical study focused on novice system analysts instead of expert ones because CASE ( Computer-Aided Software Engineering ) tools benefit novices more. The results will be used in the future design of CASE tools. Thirteen novice system analysts were used in this study. They were given a task of developing the specifications for a delivery scheduling system. During the task, which was recorded by audio tape, the subjects were told to think aloud. The recorded results were divided into two main categories: mental and non-mental behaviours. Within these two main ones, they were divided further down. Recognise goal, assertions, reasoning, and planning fell under mental behaviours while information acquisiton and conceptual modelling fell under non-mental.

While our project will focus on both novices and experts, we can incorporate their methodology of thinking aloud and recording their thoughts. I'm not sure if any of the task we're giving in our experiment deals with systems analysis. If it does, then we can model their procedures and methods of analyzing the data.

Posted by John at 1:09 PM | Comments (0)

"The Programmer's Burden: Developing Experise in Programming" by Robert L. Campbell, Normal R. Brown, and Lia A. DiBello

This article describes and illustrates a developmental approach to programming knowledge. A constructivist developmental approach using Piagetian structural interview and tape diaries are used to gather information. The study interviews programming experts and follows individuals learning Smalltalk for 2 months.

Results of the experiment:

Interviews conducted required programmers to introspect and find in retrospect moments in which they viewed themselves becoming more aquinted with a language. The moments describe a metaphor of climbing a mountain, where some moments they think they have climbed over, only to see more of the mountain ahead. Expert programmers could tell which programs have been coded by experts and those done by novices. There's a consensus in a progression from "cookbook" to "intuitive" understanding involved in programming among the interviewees.

The second method, involving tape diaries of programmers learning smalltalk, followed the development of learning the language. From this researches could deduce developmental levels for learning smalltlak. Professional programmers were much quicker to learn Smalltalk pedagogy than novices or non-programmers.

What's important to us:

The tape diaries of much use to us in this experiment. Since the team has less than 10 weeks, performing such tape diaries, let alone find people willing to participate, is impossible. We're better off conducting interviews and performing experiments in the lab.

Binary distinction between novice and expert is inadequate because of enormous variations in skill, knowledge, and productivity among programmers. One person's expert research could be another's novice programmer, so our study should make this distinction.

Novices differ from experts in four general areas

  1. Syntactic knowledge
  2. Semantic knowledge
  3. Schematic knowledge
  4. Strategies

By taking up the programmer's burden, professional programmers are working against the grain of Smalltalk, and they end up frustrated with its principles of code reuse. This material may be dated, but it's interesting to see that experts will try what they already know in programming problems.

Posted by Matthew at 10:43 AM | Comments (0)

Empirical Studies of Programmers: Sixth Workshop by Wayne D. Gray and Deborah A. Boehm-Davis

This brief article is a synopsis on a workshop discussing the empirical studies done on programmers. Large scale experiments, such as software process appraisals, are impossible to conduct, while smaller scaled experiments are possible. Results from successful experiments are being used in decision making and management. The workshop consists of five panelists. They will discuss recent experiments, methodologies used, implementation issues, and how to bridge the gap between collaborations with academe and industry.

Since our experiment is a collaboration between academe and industry, I’m very interested in what was said during the panel. Too bad this was just a synopsis of the workshop. Perhaps I can try to find the rest of the article.

Posted by John at 6:29 AM | Comments (0)

Characteristics of the Mental Representations of Novice and Expert Programmers: An Empirical Study by Susan Wiedenbeck and Vikki Fix

This study hypothesized that an expert programmer’s mental representation of a program has five abstract characteristics: hierarchical and multilayered, contains explicit mappings between the different layers, founded on the recognition of basic patterns, well connected internally, and well grounded in the program text. To test their hypothesis, they will conduct an experiment with both novices and experts. They hope to show that the novices’ understandings of programs are usually underdeveloped or absent in each of the five areas.

The experiment was conducted with twenty novices and twenty experts. They were given 15 minutes to memorize a 135 line Pascal program. After that, they were given a booklet with 11 questions. The questions tested the subject in all five areas. They were not allowed to look at the code to answer the questions, but they were given an infinite amount of time to respond.

The results did prove that novices did lack in the five areas mentioned, compared to the experts. However, in easy tasks such as naming the simplest recurring patterns, novices were able to perform just as well, if not better, than the experts. They attribute the novice’s lack of understanding in the five areas due to their program comprehension strategy. Experts use a “systematic” approach while novices use an “as-needed” approach.

This study is very relevant to our study. It is extremely similar to the experiment that we will be conducting near the end of this quarter. However, our hypothesis is not very specific as of right now. But we can use their methodology, by giving the subjects a chunk of code to memorize and then asking questions afterwards. When we have formed a more developed hypothesis, we can alter their methodology accordingly

Posted by John at 6:14 AM | Comments (0)

Week 2 Progress Report

1. I've read 3 of the 4 papers that everyone got and 2 of the 3 papers that only I have. I've posted only one the blogs so far but I'll get the second one tomorrow.

2. I haven't had much luck searching through PsychLit. I'm going to try searching more for cognitive abilities since Justin is searching for expertise. This way we'll probably be able to cover more material that could relate to the study.

3. IEEE hasn't really been giving me what I was looking for, but I stumbled upon some promising papers on ACM that revolve around other experiments between novices and experts. Unfortunately I couldn't view any of them because I need to purchase a membership, so I'll just access them on the school network tomorrow.

Posted by Teerawat at 12:41 AM | Comments (0)

April 17, 2005

Progress Report

1. Read all the papers
2. I had a hard time finding psychology literature concerning expertise in software engineering. Going to look for more concerning expertise in general.
3. I checked out 2 books on "Empirical Studies of Programmers" from the library. There was only one good article concering "expert/novice" comparisions.

Posted by Justin at 11:54 PM | Comments (0)

"The Advanced Programmer's Reliance on Program Semantics: Evidence from some Cognitive Tasks" by Siu-Yee Wong, Him Cheung, and Hsuan-Chih Chen

This paper examines the effect of programming experience on program semantics. According to the paper, expert programmers are characterized by their ability to organize or "chunk" code. It was also found that expert programmers were more likely than novices to fill in blank lines of code that were semantically intact. But when the program is scrambled then experts were not able to fill the blanks. This led them to the theory that expert programmers are more sensitive and reliant on the semantics of a program.

To test this theory they ran an experiment involving a BASIC program. The experiment consisted of 24 non-programmers, 24 novices with one course in BASIC, and 24 experts with 2+ years BASIC experience. Each participant was given 6 programs each consisting of 10 to 12 lines. They were given a brief amount of time to review and understand each program. Then they were given the same 6 programs with either semantic, syntactic, or no changes made to the program and were asked to decide if they had seen the program before. The results showed that the subjects with more BASIC experience were more able to detect the semantic changes than those with less experience.

This paper is relevant to our project because it discusses the difference in the "chunking" ability between experts and novices which leads to a difference in the reliance on program semantics. Since we are also studying the difference between experts and novices, we can see if an expert's "chunking" ability helps him problem solve better.

Posted by Teerawat at 11:49 PM | Comments (0)

Status update

1. Read all but 2 papers that were given. Will finish by meeting.
2. Updated blog with info on unique papers that were given.
3. Found 2 sample aptitude test online.
4. Found a paper that researched the outcome of programming aptitude tests. Was not able to read it yet.

Posted by Derrick at 11:25 PM | Comments (0)

Week 2 Progress

  1. I read the four papers.
  2. Somehow I only got 2 articles, but was able to read them. Posted about one of them so far. I won't be able to finish the second one until Monday.
  3. I took a look at some panel of experts documentation on the web to see what was covered and what types of questions are asked.

Posted by Matthew at 11:05 PM | Comments (0)

Progress Report: Week 2

1. I read the four papers that each member received.
2. Started but not finished reading the three papers that only I received. I'll finish before our meeting tomorrow.
3. Found three books for the "popular books" background research assigned to me and my teammate.

Posted by John at 10:12 PM | Comments (0)

"The Case for Case Studies of Programming Problems" by Marcia C. Linn & Michael J. Clancy

This article describes how programmers organize programming knowledge, analyze how program design skills are commonly taught, and describes the effectiveness of case studies. Case studies are used in an experiment to highlight effective and ineffective strategies for solving programming problems. It emphasizes teaching students design skills by applying general skills to specific domains in order to solve programming-specific problems. An experiment tested the effectiveness of case studies in 10 pre-college Pascal classes. Three conditions were met by the instructors in a random order. The conditions were: Providing a student solution and looking at expert code, providing a student solution plus expert commentary and expert code, and providing expert commentary and expert code.

Results of the experiment:

Students who received the expert commentary learned more about program design than did students who received the expert code without the commentary. Students who implemented their own solution to the problem and then studied the expert commentary learned no more about designing the solution to a computer problem than those who had the expert commentary alone. The students who didn't study the expert commentary were the least successful in the study. Given the fact that most pre-college students normally try to avoid reading, the results show strong evidence for expert commentary. Finally, students need more than just expert code to assist them in learning design skills.

Other highlights from the reading include:

Students organize programming knowledge in terms of language syntax, often causing fragmented knowledge and leading to an engagement of trial and error. Syntax and top-down learning are often used because that's how it is presented to them. There is no emphasis of recognizing & reusing patterns.

Experts organize programming knowledge in larger conceptual structures. Programs are often thought out in plans with goals in mind. Larger programs require combinations of plans. Templates are applied to reocurring problems and competent programmers use top-down, bottom-up, middle-out, and mixes of these styles to design programs.

What's relevant:

Expert commentary helps students at varied ability levels to design more effective programs in case studies. The study found that students complained when they had to read long commentary from experts, but I'm not sure how well highschool student compare with college students. The usage of templates and plans are helpful in teaching novices become better at programming. Top-down teaching is unlikely to develop skills that individuals need to be effective programmers and only a small number of students are capable of inferring program design skills via unguided discovery. A suggested approach is reciprocal teaching. Perhaps we can look into how templates and plans are formulated when we decide what questions to ask novices & experts in our project.

Posted by Matthew at 10:04 PM | Comments (0)

"When Novices Elicit Knowledge: Question Asking in Designing, Evaluating, and Learning to Use Software"

The main topic of the article focused on the interaction between expert and novice users and their learning capabilities. The method in question was the Q&A technique that many feel is a superior form of learning compared to think-out-loud methods. Multiple experiments were run with novice users(office secretaries), where the expert would “coach” the user by answering their questions about the tasks they must perform in a given time limit. Analysis of the experiments included the following:

- What types of question-and-answer dialogues occurred?
- What were questions about?
- Did users ask the right questions?
- Were answers to questions effective?
- What did we learn from questions?
- How are questions interpreted?

The main purpose of studying these questions were to find solutions within human-computer interaction. “In the HCI domain, we want to learn useful things about the users’ experience with computer systems, in order to improve the systems, guide design, and achieve some more general understanding of the mechanisms that underlie computer use as a cognitive skill.” In conclusion, the writer feels that Q&A method is more natural for qualitative analysis than the more familiar think-out-loud method.

Posted by Jonathan at 7:15 PM | Comments (0)

Toward a Unified Theory of Problem Solving: Views from the Content Domains by Richard Mayer

This paper sheds light on different concepts of problem solving. In problems solving there are two kinds of problems (routines and nonroutine); two kindas of problem solving strategies (weak and strong methods); two kinds of problems solvers (novice and experts); and two phases of problem solving (qualitive and quantitative reasoning). The paper defines each of these terms and how they are important in the research of how problem solvers *work*.

This paper is relevant because it gives insight on what to pay attention to in the subjects as they solve problems. It is important that we look into all aspects of our research approach so that we can gather relevant and important data.

Although this paper does not have any direct information on the empirical studies of software engineering, it does give us information on what to look for in our studies. When preparing experiments, we should look into what kind of problems we are creating. We should also research the problem solving strategies that the engineer uses and what problem solving phase they are at.

Posted by Derrick at 6:12 PM | Comments (0)

Expert and Novice Performance in Solving Physics Problems by Larkin, McDermott, Simon, Simon

This paper studies the differences in performance of novice and expert physic solvers. Researchers are looking into how "physical intuition" works. Physical intuition is the solving of "difficult problems rapidly and without much conscious deliberation about a plan of attack." These researchers first looked into how people solved word problems in algebra and how chess players visualize chess boards. This is because physics involve these two skills to solve physics problems. Physic problems are often given as word problems (like algebra) and require problem solving skills (chess playing).

Although this paper does not deal with empirical studies of computer software engineering, it does shed light on the approach of how we can gather information. This paper gives ideas of what to look for in the subjects as they solve problems (do they write diagrams? how do they approach the problem? forward thinking? backward thinking?).

A relevant theory that the paper looks into is memory "chunking". Thinking back into software engineering, a large number of tasks are linked and are inter-related with each other. Perhaps expert engineers are able to visualize larger chunks of this information.

Posted by Derrick at 5:48 PM | Comments (0)

"Expert-Novice Differences and Knowledge Elicitation" by Foley and Hart

In this study, the researchers examine expert-novice differences when developing systems. Foley and Hart argue that it is more important to model ways in which experts and novices interact than to solely model an expert’s knowledge. In this particular study, the researchers wanted to examine the differences between how novices and experts used their knowledge to solve problems.

The subjects in the study included first-year students, second-year students and staff from the computer science department (novice, intermediate, and expert respectively). The subjects were asked to solve three debugging problems. The problems became progressively harder. To track the debugging process of the subjects, portions of the code were written on different cards. This method allowed researchers to track the search path that subjects were using to solve the problem. Also, the subjects were taped and asked to think aloud to capture what they were thinking at the moment.

Overall, the study revealed that experts were more likely solve the problem. The researchers noticed that the experts examined more hypotheses for solving the problem. Moreover, the strategies between novices and experts were different. Ultimately, the researchers argue that it is important to note the differences between novices and experts during knowledge elicitation.

This study is relevant because it provides another model for conducting our own study. It highlighted the importance of recording the data through tape and thinking aloud. More importantly, it compared the differences between experts and novices just like our own study.

Posted by Justin at 4:53 PM | Comments (0)

"What Do Expert Programmers Communicate by Means of Descriptive Commenting?" by Riecken, Koenemann-Belliveau, Robertson

In this study, researchers examined what ways expert programmers use descriptive commenting to communicate program knowledge to novice programmers. To conduct the study, the researchers asked several expert programmers to comment a simple parsing program consisting of a main(), parse(), hash(), and errorExit() methods. The research focused on two methods of the methods in the study: parse() and hash(). These methods served as a focus because the parse method runs the majority of the programming, while the hash method contains complex code. The researchers argue that although novices would be more concerned with the syntax of the hash method, experts would be more concerned with the semantics of the parse method. Thus, the experts would be more likely to comment the parse method.

The subjects of the study were 16 expert programmers who had at least 10 years of broad programming language experience, 6 years C programming experience, and was a project leader. The subjects were asked to make the code of the parser easier to read for a novice programmer. Also, subjects were asked to "think aloud" to capture their thinking patterns.

Overall, the study revealed that the experts were more likely to comment the parse method. The researchers argue that expert programmers are more likely to "communicate semantic domain knowledge concerning the function and behavior of complex syntactical program knowledge." Novices, on the other hand, feel they also need help on the syntax of complex code.

This study is relevant to our project because it provides an example of a method for examining how expert programmers think. The study provides insight on how to get experts to “think aloud” and share their ideas. A similar method could be used in our own study.

Posted by Justin at 4:30 PM | Comments (0)

April 16, 2005

"The Initial Stage of Program Comprehension"

The study presented by this article reveals the importance of “beacons” in program comprehension. Program comprehension refers to the ability to understand what is the goal of a program, based on viewing the functions and documentation within the code. Wiedenbeck describes the term beacon as “segments of code which serve as typical indicators of the presence of a particular programming structure or operation.” In the experiments discussed, the example beacon used was the swap function which is correlated with sorting algorithms. Four different experiments were held to depict the importance of beacons:

Disguise Experiment – Two samples of code for a Shellsort algorithm were given to novice and advance users. The first sample contained the swap method, which is regularly found in this algorithm. The second sample, however, consisted of an alternate method of swap which allowed the function to still work. Results showed that a significant majority of users recognized the first sample as a sorting algorithm. Only one-fourth were able to tell that the second sample was another implementation of Shellsort. This suggests that the few lines of code which implemented the swap method played a significant role in determining the goal of the algorithm.

Prototypical/Non-Prototypical Sort Experiment – In this experiment, two samples of code were given to both parties in the same fashion. However, the algorithms given were two functions which both novice and expert users have never seen before. The first sample contained the swap beacon while the second performed the same sort but with different methods. Once again, results showed that a majority were able to deduce that the first function was a sorting algorithm. Virtually no one was able to tell what the purpose for the second algorithm. This experiment suggested that regardless of the familiarity of code surrounding it, the swapping beacon aided users in determining that it was a sorting algorithm.

Intrusion Experiment – This experiment was implemented to show the negative effects of beacons in program comprehension. The samples provided were both implementations of binary search, however in the second sample, a swapping method was added in the body of the code. The experiment recorded instances of whether the subjects classified the code as a binary search or a sort. Almost all expert users recognized the first sample as a binary search, and only half recognized the second sample with the intrusion as a binary search. Less than half of novice users recognized it as a binary search, and no novice users were able to determine the function of the second sample. Very few subjects classified the first sample as a sort but over 80% of both parties classified the second sample as a sorting algorithm. The result of this experiment showed that the presence of a false beacon in a program has a strong tendency to mislead programmers about the program’s function.

Defective Sort Experiment – The final experiment presented four different samples of code. The first was a correct implementation of a sort procedure which contained nested loops and the swap method. The second version contained all the same code except one of the nested loops which was deleted. The third sample took all loops out of the code altogether. And finally the last version contained all loops intact but removed the swap beacon. The purpose of this experiment was to determine if secondary beacons(nested loops) would effect the decisions of programmers as much as primary beacons(swap) which they are accustomed to. The results went against the hypothesis of the experiment where secondary beacons also have an effect on program comprehension yet reinforced the hypothesis that the primary beacon plays a significant role on comprehension.

In conclusion, the study was able to determine that beacons play both a positive and negative role in program comprehension. Since there are many repetitive and custom methods of programming, software engineers rely on these beacons to understand and recognize the functions presented to them.

Posted by Jonathan at 8:27 PM | Comments (0)

"A Note on the Quantification of Computer Programming Skill"

This article explains that recent studies have shown only one side of expertise when it comes to computer programming skill. Studies of expertise have usually compared expert programmers with novice programmers and examined “time-based” skill, in terms of time spent programming. The article emphasizes that the tasks that are performed and the languages used to perform them should also be measured to evaluate what it calls “multiskilling” in programming. There is difficulty in quantifying and measuring multiskilling, however, it was quantified using entropy measures derived from informational theory. Multiskilling was then divided into two separate parts:

1) ”Language entropy” which measures the frequency of use of multiple languages over a period of time
2) “Task entropy” which measures time spent and a variety of tasks in different categories (e.g. scientific or commercial computing).

Since there was a lack of multiskilling information, a survey was mailed to hundreds of professional computer programmers asking questions about the languages they know. Some examples included the year the language was learned, the frequency with which each language was used, and which language they wished to learn next. The results yielded that “actual time spent programming and interval since first learning to program loaded highly on the first factor(time-based skill), while language and task entropy and the number of languages known loaded highly on the second factor(multiskilling).” The article was able to derive evidence supporting the importance of multiskilling by examining the impact of ageing on work skills and obsolescence. “In order to remain current, programmers must learn new languages, update their skills, and demonstrate the same flexibility required of staff in other areas of work.” Basically, programmers must become multiskilled. Evidence also showed that older programmers had no desire to learn new programming languages while two thirds of other programmers surveyed had future intentions to learn C and C++ which were the newly introduced programming languages at the time. In conclusion, the article stressed that although expertise in one language have been valued in the past, employers should now consider those who have multiskilling abilities overall.

Posted by Jonathan at 2:12 PM | Comments (0)

April 13, 2005

Welcome to 199 Students

We had our first meeting this week at 9:30am on Tuesday. We talked about the overall plan for the study and gave everyone homework.

Homework

There were three papers that everyone would read.

In addition, there was a stack of papers and books that was divided among the group. Some of these papers are my only copies, so please be gentle with them. For these papers, please post something on the blog about them under the category of "Books and Papers". Give the citation, a summary, your evaluation of its relevance to our project, and identify what you think might be relevant (a theory, a result, an experimental manipulation).

Also, the 199's were given some other background to research.

Justin and Teerawat: psychology and ESP/PPIG/SE
John and Derrick: programming aptitude tests and popular books
Jonathan and Matt: field study and panel of experts

Please post a progress report the day before our meeting under the
"Progress Report" category.

I've uploaded an EndNote library containing references to the various papers that we have.
Download EndNote Library
If you have a copy of EndNote, you can add to this as we go along.

Next meeting will be Monday at 4pm.

Susan

Posted by ses at 4:15 PM | Comments (0)