May 23, 2005
Pyschology of Programming by J.M. Hoc
- Methodological Issues in the Study of Programming ( pg. 83 )
o Types of data collection – results of each need to be interpreted differently, and the results of each are often necessary for the design of further research using other approaches
hypothesis testing – contrasting theoretical positions make different predictions about the effect of some manipulation on behavior
Comparisons – discover which of two ( or more ) alternatives is easier to use, or whether some change to the programming process has an effect on performance.
- Evaluations – fine line between evaluations and comparisons, there is considerable overlap. “Can people use flowcharts?”, rather than “Are flowcharts better than structure diagrams?”, also used with the intention of improving a system.
- Exploration – the main research theme is novel and we lack enough information to engage in the other types of data collection
- The experimental task – “You can’t study people using one system and assume your results generalize; you can’t study a single task and assume you found out all the important facts”, for truly generalizable conclusions, a variety of tasks, a variety of type of programmers and a variety of languages should be studied.
- Sample tasks – asking subjects to write a program and “think aloud” while they do it, giving them a program and giving them a comprehension quiz about it, scrutinize code either to classify types of errors or to compare the frequencies of different solution types, timed question answering with miniature programming languages, recalling programs that have been viewed briefly,
- Majority of studies of program writing have used observational techniques, either video or post hoc analysis of code.
- While single tasks, single subject populations and single languages may be used to disprove a theoretical hypothesis, they are of little use when they confirm hypotheses.
- Important concepts in experimental design
- Statistical significance – provides a measure of the likelihood of the observed difference in scores being a result of chance variations in the performance of four subject groups, assuming that they were drawn from the same population and that the experimental manipulations had no influence on their performance.
- Effect size – this reflects the influence on a score of the experimental manipulations, though no just by magnitude, but also by variability.
- Sample size – the main problem with small samples arises when the results are not significant ( i.e. p > 5% ) since the use of a larger sample, all else being equal, might reduce p to below 5%.
- Power – measure of the quality of an experimental design, constructing an experiment of sufficient power to detect an effect, without overkill.
Observational techniques
- Interviews with programmers
- Video plus verbalization
- Constructive interaction
- Longitudinal studies of learning
- Social and organizational processes
- Applying research – what was the question?, interpreting causes
Experimental validity
- Internal – describes our ability to be sure that our explanation of the observed differences is the only likely explanation, and it is closely related to the statistical issues discussed earlier.
- External – describes our ability to make correct generalizations about the implications of the results.
- In most cases external validity is only an issue when we have high internal validity.
- Hypothesis-testing must place most emphasis on internal validity, while comparisons and evaluations need to emphasize external validity.
- Experts represent programs in terms of semantic structure, whereas novices encode them syntactically.
- Experts tend to spend more time than novices planning and evaluating.
- The pattern overall is that experts are able to handle information at different levels. They differ from novices in two important respects: their ability to develop overviews or abstract models of solutions, and their ability to understand the consequences of implementation detail. ( pg. 108 )
- For experts, particular languages or language types did not correspond to particular solution strategies, although there was some correspondence between languages and oversights. It seems that experts conceive an abstract model of the solution separate from its expression in a particular programming language. They envision the solution in an pseudo-language, which is a collage of models borrowed from many sources. ( pg. 108 )
- Experts were tenaciously resistant to the change of algorithm. ( pg. 108 )
Posted by John at 3:13 PM | Comments (0)
May 9, 2005
Code Complete by Steven McConnell
- Architecture – the quality of the architecture determines the conceptual integrity of the system, which in turn determines the ultimate quality of the system. ( pg. 44 )
- Attention to quality at the beginning has a greater influence on product quality that attention at the end. ( pg. 59 )
- Every programming language has weaknesses and strengths. Be aware of the specific strengths and weaknesses of the language that you’re using. Establish programming conventions before you begin programming( pg. 70 )
- Major design heuristics ( pg. 108 )
o Find real-world objects
o Form consistent abstractions
o Encapsulate implementation details
o Inherit when possible
o Hide secrets ( information hidings )
o Identify areas likely to change
o Keep coupling loose
o Look for common design pattersn
-Guidelines for using heuristics ( pg. 109 )
1. understanding the problem
2. devising a plan
3. carrying out the plan
4. looking back
- Good design is iterative; the more design possibilities you try, the better your final design will be. (pg. 123 )
- A class interface should hide something – a system interface, a design decision, or an implementation detail. Inheritance is a useful tool, but it adds complexity. Keep in mind that the primary tool for managing complexity is classes. ( pg. 160 )
- Routine guidelines: ( pg. 161 )
o Cohesion – the ultimate goal is to have each routine do one thing well and not do anything else
o Give routines a name that describes exactly what it does, no matter how long it is.
o Put parameters in input-modify-output order.
- Practice defensive programming. Use exception handling and assertions.
- PPP ( Pseudocode Programming Process ) is a useful tool for detailed design and makes coding easy. Pseudocode translates directly into comments, ensuring that the comments are accurate and useful. The steps in constructing routines by using the PPP are design the routine, code the routine, check the code, clean up loose ends, repeat as needed. ( pg. 215 )
- Variable guidelines: ( pg. 237 )
o Pay attention to data initialization, for it is one of the biggest source of error in computer programming.
o Minimize the scope of each variable. Keep references to a variable close together. Keep it local to a routine or class. Avoid global data.
o Use each variable for one and only one purpose.
o Names should be as specific as possible.
- Remember the individual rules for each data type. ( pg. 318 )
- Structures can help make programs less complicated, easier to understand, and easier to maintain, but always consider whether a class would work better.
- Pointers are error-prone. Protect yourself by using access routines or classes.
- Global variables should be avoided, but if you can’t, work with them through access routines. ( pg. 344 )
- The strongest principle for organizing straight-line code is ordering dependencies. Dependencies should be made obvious through the use of good routine names, parameter lists, and comments. ( pg. 353 )
- To trap errors, use the default clause in a case statement or the last else in a chain of if-then-else statements. ( pg. 366 )
- Recursion can provide elegant solutions but keep this tips in mind: make sure the recursion stops, use safety counters to prevent infinite recursion, and limit recursion to one routine. ( pg. 396 )
- Tables ( direct access, indexed access, stair-step access ) provide an alternative to complicated logic and inheritance structures. ( pg. 430 )
- Minimizing complexity is a key to writing high-quality code. ( pg. 460 )
- How software measure up against the external and internal software qualities?
o External – correctness, usability, efficiency, reliability, integrity, adaptability, accuracy, robustness
o Internal – maintainability, flexibility, portability, reusability, readability, testability, understandability
- Techniques for improving software quality ( pg. 466 )
o Software-quality objectives
o Explicit quality-assurance activity
o Testing strategy
o Software-engineering guidelines
o Informal technical reviews
o Formal technical reviews
o External audits
- Code-tuning techniques ( pg. 609 )
o Logic
o Loops
o Data transformations
o Expressions
o Routines
o Recoding in a low-level language
o The more things change, the more they stay the same
- The first priority of visual layout is to illuminate the logical organization of the code. Criteria used to assess whether that priority is achieved include accuracy, consistency, readability, and maintainability. ( pg. 775 )
- Good code is its own best documentation. If the code is bad enough to require extensive comments, try first to improve the code so that it doesn’t need extensive comments. ( pg. 817 )
- The characteristics that matter the most are humility, curiosity, intellectual honesty, creativity and discipline, and enlightened laziness. ( pg. 835 )
Posted by John at 11:08 AM | Comments (0)
May 8, 2005
“Communication Problems in Requirements Engineering: A Field Study”
This field study focuses on the communication characteristics of the requirements phase of a software project. The purpose of the study was to show that these projects have multiple problems in the communication between stakeholders and software engineers. In this case, the stakeholders can be seen as novice users while the requirements analysts can be seen as experts in their field. The three major communication barriers discussed deal with the “ineffectiveness of the current communication channels, restrictions on expressiveness imposed by notations, and the social and organizational barriers.” The study expresses that the one way communication channel, or “over-the-wall” method currently used in offices is hurting the flow of communication. Specification documentation is not adequate enough to communicate between phases of the software project. In addition, when asked if these documents were understandable by end users, a significant amount of experts expressed that their clients would need additional explanations in order to understand the notations that were given to them. Finally, organizational barriers arise when processes are divided into different phases and are performed by different groups. Managers chose these groups on a number of factors, rather than choosing them based on domain knowledge which is more ideal. Requirements committees are often composed of people with authority whom have limited knowledge of low-level tasks which require to be computerized. As a result of the study, it was determined that organizational and social issues posed the most important influence on communication problems in software engineering. It stresses that software professionals need to accommodate and identify these three different categories in their project management to determine the overall success or failure of the software project.
Posted by Jonathan at 7:49 PM | Comments (0)
"Is Software Engineering Training Enough for Software Engineers?"
The author of this article feels that the current curricula for software engineering is insufficient and inadequate when it comes to training students for “real world” software engineering projects. He argues that software engineering education should include the understanding and using classical engineering which is comprised of other engineering disciples such as mechanical engineering, electrical engineering, and different aspects of civil engineering. He proposes to update the current curriculum with adding elements such as analysis and modeling of non-software system related problems. As a case study, a software engineering course was analyzed to determine the importance of “system thinking”. The project purpose was to investigate the student’s ability to solve unfamiliar problems which are found in the profession but somewhat outside of the software engineering domain. They were given two requirements that hold emphasis in “real world” software engineering: modifiability and visualizing the mathematical model. The students understood that the software should be easy to modify but it was hard for them to incorporate this within their requirements and design. They were also able to derive a model for the project(some which did not work out at all) but had trouble with its documentation and analysis. Results determined that the students were not well prepared for engineering projects outside of the software spectrum. Even the teachers and their assistance were questioned on their knowledge of preparing students for what they will face in the future.
Posted by Jonathan at 7:13 PM | Comments (0)
May 2, 2005
Peopleware by Tom Demarco & Timothy Lister
- The major problems of our work are not so much technological as sociological in nature. ( pg. 4 )
- The main reason we tend to focus on the technical rather than the human side of work is not because it’s more crucial, but because it’s easier to do. ( pg. 5 )
- Bad estimates, hopelessly tight estimates, sap the builders’ energy. ( pg. 28 )
- Coding War Games ( pg. 44 )
- A pair of programmers from the same organization didn’t work together, each person given the same work, designing, coding, and testing a medium-sized program to our fixed specifications.
- Benefit to individual – learning how he or she compares with the rest of the competitors.
- Benefit to company – learning how well it does against other companies in the sample
- Benefit to us – learning a lot about what factors affect productivity
- Results:
- Best people outperforming the worst by about 10:1
- Best performer being about 2.5 times better than the median
- Half that are better-than-median performers outdoing the other half by more than 2:1.
- Years of experience was a productivity non-factor. People who had ten years of experience did not outperform those with two years of experience. Maybe it doesn’t matter so much for our experiment to consider experts people who’ve had 5 years of experience?
- It mattered a lot who your pairmate was. Even though the pairs didn’t work together, the two members of the pair came from the same organization. They worked in the same physical environment and shared the same corporate culture. Best performers are clustering in some organizations while the worst performers are clustering in others. We should keep this in mind when looking for expert candidates.
- Flow – a condition of deep, nearly meditative involvement. Unfortunately, you can’t turn on flow like a switch. It takes about 15 minutes or more of concentration before the state is locked in. During this immersion period, you are particularly sensitive to noise and interruption. Not really doing work during immersion. We might want to rethink having the subjects think aloud. ( pg. 63 )
- Jelled teams - a group of people so strongly knit that the whole is greater than the sum of the parts. once a team begins to jell, the probability of success goes up dramatically.( pg. 123 )
Posted by John at 7:15 AM | Comments (0)
May 1, 2005
Expertise in Professional Software Design: A Process Study by Sabine Sonnentag
In this study, the author wanted to examine expertise in software designers. He argues expert programmers should exhibit the following characteristics:
- Problem comprehension - create an adequate representation of the program
- Planning - decisions about future steps and their sequence are made
- Feedback processing - actively searching for feedback
- Task focus - spend less time on task-irrelevant cognitions
- Use of visualizations - use of sketches and diagrams
- Knowledge of strategies
To prove whether these characteristics actually appear in experts, he carried out the following experiment:
Method: 40 professional software designers participated in the study. The first task was to differentiate the high and moderate performers from the subjects. This was done through peer-nomination. The subjects had to design a software system that controls the movement of a lift system. The subjects were asked to think aloud.
Results: High performers spent more time on feedback processing and less time on task-irrelevant cognitions. They produced more visualizations and knew more about strategies. Across all phases, high performers spent less time analyzing requirements, suggesting they were able to build program representation more quickly. High performers did not show more planning ahead than moderate performers. Length of experience did not explain performance differences with the subjects in this particular study.
This study is relevant because it examines the attributes of an expert software designer. It provided a hypothesis for each of these attributes and tried to prove whether they were actually true in practice. Also, I thought it was interesting how they described expertise not merely but length of experience, but more importantly high performance.
Posted by Justin at 10:30 PM | Comments (0)
The Effects of Semantic Complexity on Expert and Novice Computer Program Recall and Comprehension by Bernard Guerin and Anthony Matthews
In this paper, the authors wanted to explore the effects of semantic complexity on both recall and comprehension while comparing novices and experts programmers. To do so, they conducted three experiments.
Experiment 1
Purpose: If experts use higher level semantic knowledge to help them understand programs, then removing some of the semantic structure should to lead to performance closer to that of novices
Method:
- 104 subjects (52 experts with 4.7-5 years of programming experience and 52 novices with 0.5 years of experience).
- Subjects received a booklet with a two-page computer program (116 line COBOL program) with 4 versions: the original version, a version with lines randomized within a module but keeping the module order intact, a version with only the modules randomized, and a version with both lines with a module randomized and module order randomized.
- The subjects were tested in groups of two to five.
- Subjects had to take 10 minutes to learn and memorize the program and then write down as many lines or program code that they could remember. Then they had to write down a brief summary of the purpose of the program and how it worked to achieve that purpose.
Results: The experiment provided strong evidence that experts use superior semantic knowledge when memorizing a computer program. However, when line order was randomized, they performed at the same level as a novice.
Experiment 2
Purpose: To see how two versions of a program (semantically complex and semantically simple) affect recall and comprehension.
Method:
- 52 subjects (26 experts and 26 novices)
- Each subject received 2 procedures written in Pascal for pattern-matching (one simple, one complex). In the first program, it is implemented using simple data structures. In the second program, it is implemented using more complex structures (linked lists).
- Subjects were tested in small groups.
- They were given 5 minutes to memorize the program, 5 minutes for recall, and 3 minutes to describe the function and operation of the program.
Results: The complex program was recalled less well than the simple version. Novices performed poorly on both versions. When the complex version was used, experts performed at the level of novices. Expertise was only significant for comprehension.
Experiment 3
Purpose: One reason that experts perform well in the previous experiments is because they already know the keywords so they can memorize and comprehend the program more quickly.
Method:
- To test this idea subjects were given a program to memorize. One half received the original COBOL program and the other half received a version with the keywords replaced by common English words
- 24 novices and 24 experts.
- 10 minutes to memorize, 8 minutes to recall, and 5 minutes to write a brief description of what the program was doing
Results: Removal of the keywords did not affect the experts more than it did the novices for either recall or comprehension. This means that the experts still have an advantage with their semantic knowledge about computer languages.
Overall, the experiments revealed that experts actively search for program functions. The experiments show that experts pay more attention to actual functions than novices do.
This paper relates to our study, because it provides more qualities of experts that we can look for. Moreover, the paper provides examples of how we can examine how our subjects think. It is important not only to monitor what type of code they produce, but also what their general understanding of the problem is. Also, the actual procedures of their experiments can be looked at as an example.
Posted by Justin at 10:26 PM | Comments (0)
Think-aloud books available online
There are two books available online that offer basic overviews about the think-aloud procedure. The first, Ericsson & Simon (1993), lays out the theoretical framework and procedures. It is the one most often cited in research articles (the online version appears to be read-only, as far as I can tell, so no saving or printing).
Posted by Erin at 10:13 PM | Comments (0)
Thinking aloud: Reconciling theory and practice by Boren & Ramey (2000)
Boren, M. T., & Ramey, J. (2000, September). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261-278.
Online version:
http://ieeexplore.ieee.org/iel5/47/18778/00867942.pdf
This paper begins with a discussion of think-aloud protocols. They first describe the original theoretical framework put forth by Ericsson & Simon (1984/1993), and then bring it into the context of research in usability testing. Field observations of usability testing showed that researchers do not appear to adhere to standards with regard to think-aloud protocols (the description of the variability is pretty interesting). Methods vary widely, and some are carried out without any explanation or basis in theory (i.e., no background given or sources cited in the literature). This paper is particularly useful because it summarizes some of the key points from Ericsson & Simon’s theory, and then offers an alternative theoretical framework (speech communication). One thing for us to consider is whether the concepts that this alternative framework is based upon are more pertinent to the realm of usability testing than they would be for us (i.e., should we consider departing from the original think-aloud framework).
According to Ericsson & Simon's (1984/1993) model, there are three types of verbalizations:
1. Level 1 verbalizations are those that need not be transformed before being verbalized during task performance, e.g., verbalizing a sequence of numbers while solving a math problem.
2. Level 2 verbalizations are those that must be transformed before being verbalized during task performance, e.g., images or abstract concepts, as long as this transformation into words is the only mediating cognitive process between short-term memory and verbalization.
3. Level 3 verbalizations require additional cognitive processing beyond what is required for task performance or verbalization; e.g., filtering (verbalizing only info that is related to a specific topic), making inferences about one’s own cognition, and retrieving information from long-term memory at the researcher’s request.
--Level 3 verbalizations are not considered reliable data under Ericsson & Simon's theory.
--Any outside influence during a task (e.g., a comment or prompt from a researcher) can turn verbalizations into Level 3 data, by altering the normal flow of information into short-term memory during the task.
--Verbalization has been critiqued by Nisbett & Wilson (1977), who claim that people cannot accurately report on their own mental processes. Ericsson & Simon, however, argued that this critique is directed at specific types of verbalization (Level 3), and that Level 1 and 2 verbalizations would constitute reliable data.
--If Level 1 or 2 verbalizations are collected properly, they can reveal what information the subject heeded and in what order.
--In their model, verbalizations are not to be valued for their subjective content (this aspect is addressed in another article, "Reconceptualizing think aloud methodology..." [Yang, 2003]).
--Additional verbalizations like feelings and value judgments would not be considered data.
Additional key points from Ericsson & Simon (as described in the current paper):
1. Collect and analyze only "hard" verbal data
--No inference, introspection, or opinion; only what would be present in short-term memory during a task, i.e., what the subject attends to and in what order
2. Give detailed initial instructions for thinking aloud
--Make the distinction between thinking aloud and explaining
--Encourage the participant to speak constantly as if he were alone, and warn him in advance that he will be given reminders if he falls silent
--Practice thinking aloud before the actual task!
3. Remind participants to think aloud
--After a predetermined period of time (e.g., 15-60 seconds)
--Short and nondirective (e.g., "keep talking")—do not make the subject more aware of the researcher’s presence or create a sense of personal contact
4. Otherwise, do not intervene
An alternative theoretical framework: Speech communication
1. Assumes that all verbal processes are meant for the purpose of communication, and the idea that the participant would suspend his awareness of the listener (so he may think aloud as if he were alone in the room) is flawed.
2. Speakers expect that listeners will react to what they say.
3. If this is true, and a communication framework is inevitable, then the goal becomes to create a highly asymmetrical speaker/listener relationship (where the speaker does almost all of the talking), rather than trying to have the researcher become essentially invisible.
--Roles: The participant should be cast as an important contributor (i.e., expert, and the researcher as the interested learner and listener). This should be defined at the start and maintained throughout.
--Make observational technology as unobtrusive as possible (decrease awareness of being observed).
--Use "acknowledgment tokens" continuously (e.g., "mm hmm" or "uh huh," which are both conversation "continuers," but remain uncommitted and carry no connotation). These provide the response of an "engaged listener" but still allow the researcher to keep a low profile. Tokens should carry little or no content and should not redirect the subject’s attention (and should require little or no processing on the subject’s part).
Posted by Erin at 8:55 PM | Comments (0)
"A Glimpse of Expert Programmers’ Mental Imagery" by Marian Petre and Alan F. Blackwell
This paper attempts to find correlations between different experts' mental imagery of a design. The experiment consisted of ten subjects all of which have 10+ years of experience. They were each asked to design a solution to one of four problems present, but they did not have to implement the solution. The experiment also had an interviewer present who was questioning the subject as to what they see, hear, or what they are imagining. The questions borke down the images into 3 categories: verbal imagery, visual imagery, and auditory imagery.
They found that there were several similarities between the different subjects' mental imagery. The similarities are as follows: each entity in the imagery had a label, each subject had multiple images, each image was dynamic yet could still be controlled such as pausing the image or returning it to its previous state, and the part of the image the subject had his/her attention on would be in focus while everything else was out of focus. Given these similarities the experimenters concluded that there is commonality among mental imagery between experts "suggesting that there are accessible
lessons to be elicited about how programmers construct solutions."
This is important to our study because it gives us another characteristic of expert programmers that we can probe during the experiment. It's also interesting to see that mental imagery is actually used to design a part of the program other than the GUI. The concept is interesting but the paper itself is pretty dry.
Posted by Teerawat at 6:03 PM | Comments (0)
"Expert Problem Solving Strategies For Program Comprehension" by Jurgen Koenemann and Scott P. Robertson
This paper studies the problem stategies that expert programmers use to solve a Pascal problem. The study consisted of 12 subjects, all of whom had 4 to 15 years of experience, had 4 years of Pascal experience, and have worked on a large project. Each subject was given a program and a list of 4 modification task descriptions for the program and they would have to describe how they would accomplish each task. Then they were randomly given one of the 4 tasks to complete and were asked to think aloud as well as verbalize and explain any requests for information.
Through this experiment they discovered that none of the subjects ever studied more than half of the code for the modification phase. The experimenters referred to this phenomenon as relevance strategy wherein subjects only study code or documents that they feel are relevant for the task. Furthermore some procedures were ignored by all subjects such as min, max, and isdigit. The experimenters concluded that the subjects spent most of their time searching for code segments that were relevant to their task and ignored parts of the program that either they were familiar with or had no relevance.
This is important for our studying because when we assign our experts a task we should observe not just what the expert does, but what he doesn't do to understand the problem presented to him. Until reading this paper it didn't occur to me to record things he doesn't do like reading all of the program. Just another trait of experts to keep in mind.
Posted by Teerawat at 5:30 PM | Comments (0)
Programming Aptitude Tests: Reivews
These are a few good papers I found that are discuss the problems of programming aptitude tests. They bring up good points about how many factors are left out that these tests do not consider.
Posted by Derrick at 4:10 PM | Comments (0)
"PROTOCOL ANALYSIS OF A NOVICE PROGRAMMER" by Catherine Bishop-Clark
This paper examines a novice's problem solving ability while writing a basic program. For the purpose of their experiment, their novice subject was a MBA student who was taking an introductory course in programming. They asked the student to write a program that would calculate the a student's class average and they also asked the student to speak aloud while writing the program. The results of the experiment were separated into 4 subgroups: Problem Representation, Planning and Design, Coding, and Debugging. They found that like most novices, their subject spent very little time understanding the problem and coming up with a plan. The subject spent most of her time coding and debugging. During the coding and debugging the subject kept reminding herself of syntax rules such as placing a semicolon at the end of a statement.
What's interesting about this paper is that their experiment shows that novices do not place much emphasis on the design and understanding of the problem. We can look for this in our experiment by timing the amount of time it takes for our novices to understand the problems we present to them. Additionally, the experiment reemphasizes the fact that novices focus a lot of attention on syntax which has been stated by most of the other papers we read.
Posted by Teerawat at 4:00 PM | Comments (0)
April 28, 2005
"The Elements of Expertise" by Steven K. S. Tan
The article explores the characteristics of an expert and attempts to explain how a person earns the label expert. An expert possessing qualities and attributes that account for their performances seem to be a combination of innate characteristics and those that grow from practice and experience. Experts are consistently superior in performance on a specific task. It is also of note that experts, specifically chess masters, don't think ahead further than novices. They might even pick fewer, leaving only superior moves to choose from.
Experts have a significantly larger knowledge base that is both hierarchical and complex. This complexity allows them to retrieve information faster than non-experts. Through this larger knowledge base experts also able to perform many tasks subconsciously and automatically. Although experts are slower in the early stages of problem solving, they are still able to solve problems faster than non-experts.
Other interesting facts from this article include:
- Experts rely on underlying principles and metaphors
- Experts engage in "forward" reasoning
- Knowledge is stored in stategically significant ways
- Experts are better at analyzing their mistakes
This article seems to fall under the cognitive science aspect of our research. There aren't any specific experiments run, but there is some talk about forward reasoning in experts versus backward reasoning being used in non-experts that we should look into further perhaps for our experiment analysis.
Posted by Matthew at 11:15 AM | Comments (0)
April 25, 2005
Expertise in context: Chapter 6 - Cognitive Conceptions of Expertise
This chapter presents answers to the question “what is it that makes an expert?” Nine views of expertise are presented.
- General-process: This view is based on superiority in general information processing. Experts use different processes from novices to solve a specific problem, or use the same processes but are faster.
- The quantity-of-knowledge: In a study that requires expert and novice chess players to remember chessboard configurations, the results show that differences between expert and novices are not in their ability to general information, but rather in their ability to encode and remember information. Since experts have more knowledge about chessboard configuration than novices do, they can remember and encode them easily than novices.
- The knowledge-organization: In a study of experts in physics, the results show that differences between experts and novices were not only in the amount of knowledge, but also in the way they organized knowledge that allows them to exploit the knowledge easier.
- Superior analytical ability in solving problem: Experts can use knowledge more effective and can infer things from information that novices cannot.
- Superior creative ability: Experts have creative insight that takes information that other people see in one way, but they see in another, and reach insightful solution that others cannot reach. These process includes: 1) filtering relevant and identify irrelevant information; 2) combining information in a way that is not obvious to other people, and 3) applying information acquired from another context to the problem at hand.
- Superior automatization: Experts have more information processed automatically. Automatization frees an expert’s cognitive resources so that he has more resources available for solving novel aspects of the problems.
- Superior practical ability: Experts know how the field operates, and how to navigate within it.
- Being labeled: A person is an expert because he is labeled, and treated as such.
- Synthetic view: Expertise is a prototype that comprises all aspects in other views, and others aspects not presented. Some aspects of an expert in one domain might be irreverent to an expert in other domains. People are experts in varying degrees depending on the extent to which they fulfill the criteria of expertise in their domains.
Posted by Sukanya at 2:37 AM | Comments (0)
Expertise in context:Chapter 4: Experience and expertise: the role of memory in planning for opportunities
This chapter emphasizes the role of specific past experiences in memory that may form the basis of knowledge organization that differentiates experts from novices. Expertise in planning tasks is studied based on hypothesis that "expert planners are able to anticipate the circumstances related to satisfying goals, and therefore maximize their ability to respond to opportunities when they arise”. Three planning models are presented: AI model, Case-based planning model, Opportunism planning model.
AI model requires complete preplanning. Then the effect of the plan is projected on the world model to determine whether the outcome is successful before execution. The model requires the complete knowledge of the world and that the world model is stable. These requirements lead to costly information, complex planning, and lengthy execution time. In addition, many real world tasks require experts to work with uncertain and incomplete information. This model also cannot deal with multiple goals and postponed execution.
Case-based planning model - let past experiences tell the planner when and where plans work and do not work. Reuse past successful plan and avoid failed plan. "By experiencing failures and successes within a domain, expert planner is prepared to deal with future failure and opportunities when they occur. The framework of case-based planning modules that act to anticipate and avoid problems, repair faulty plan and store them. In this model, experts learn from failure by anticipating and avoiding them. However, it does not address the fact that expert also can predict and exploit opportunities as they arise.
Opportunism planning model: planner should be able to shift to a different strategy when notice opportunity even an unanticipated one.
• Demon model: Goals are characterized as agents having their own inferential power. A suspend goal continue to independently examine the ongoing input, realized satisfy condition and insert plan into current agenda. The reason for this model is there is no way to determine at the time of suspending goal in which condition the will be reactivated, and it is difficult to provide description of features that will be used to activate suspended goal.
The author argues that successful recognition of opportunities depend on feature use to index the suspended goal in memory. The model addresses the problem of how people recognize opportunities during plan execution. Power of opportunism rests in the abilities to reason at the time of goal suspension.
• Predictive encoding Model: Planners anticipate features in the world that indicate potential opportunities (condition that success execution should be possible), use features to index goal in memory. Activation of the features while pursue other goals will reactivate the suspended goal. The experience acquired in a domain allows experts to anticipate the circumstances that are related to opportunities to satisfy goal, and easily describe them in easily observable feature. The more one learn about the circumstances, the better one will be able to predict the feature and encode them.
The predictive encoding model is verified using “Trucker” computer simulation and an empirical study whose results support the model. Trucker involves receiving delivery requests, making decisions about which truck to assign a request, and which order parcels should be picked up. The empirical involves commonsense planning in a dorm room using college students as subjects. The results confirm that: 1) recognition is likely to occur based on features that could be anticipated from planning; 2) features associated with opportunities at the suspension time is more likely to lead to pending goal; 3) opportunities not anticipated may be missed; and features requiring inferential links to connect to goals is less likely to lead to recognition.
Posted by Sukanya at 2:35 AM | Comments (0)
April 24, 2005
Comprehension Differences in Debugging by Skilled and Novice Programmers by Leo Gugerty and Gary M. Olson (from the First Workshop of Empirical Studies of Programmers)
In this study, experts and novices are compared in terms of program debugging. Experts were those who were advanced graduate students in computer science and novices were just finishing their first or second course in Pascal. The study focused on two programming languages: LOGO and Pascal. In this first experiment, all of the subjects, both experts and novices, had no knowledge of LOGO. Subjects were taught the basics of the programming language. Following training, each subject had to debug three LOGO programs. In the second experiment, all of the subjects had to debug a Pascal program.
The results from the study suggest that experts perform better when programming because they could determine what the program did faster. Novices, on the other hand, did a poorer job because they lacked sufficient programming language and introduced new bugs. Due to the more advanced domain knowledge which they possessed, experts could solve a problem more quickly.
This paper is relevant to our study because it provides a basic setup for conducting an experiment to compare expert and novice programmers. Each subject was not only monitored but also asked questions concerning their understanding of the programs. Also, an observer recorded where the subject was looking (i.e. the screen, keyboard, notepad, etc) at particular times. This study emphasizes the importance of tracking as much information as possible with corresponding timestamps.
Posted by Justin at 11:30 PM | Comments (0)
Maps as Representations: Expert Novice Comparison of Projection Understanding by Anderson and Leinhardt
This paper focuses on a study comparing the understanding of maps by experts and novices. The study divided subjects into four groups: experts, advanced novices, novices, and teachers. Experts had at least 10 years of experience in geography. Advanced Novices were geography undergraduates who had taken at least two cartography classes. Novices were geography undergraduates who were enrolled in their first geography class. Teachers were social studies teachers with no geography experience.
The subjects were given a map and asked to draw a line on the map to indicate the shortest distance between pairs of cities. The subjects were then given a second map and asked to draw lines of shortest distance between three cities in a particular order. The subjects were then asked if additional information may have helped accomplish the tasks.
Although the subject matter in this paper has little to do with software engineering, this paper is relevant to our study because, in the end, the findings were similar to those found in the examination of experts and novices computer science. In this study, experts approached problems by using rules obtained through experience. Experts could solve problems more quickly because they retrieved pre-established solutions which were used in similar problems. Likewise, advanced novices, with similar knowledge, were able to use rules to generate solutions. Novices and teachers, however, could not solve problems correctly because they lacked the basic rules for problem solving. Many times novices and teachers used incorrect rules to infer more incorrect rules. Overall, this particular examination of experts and novices show that similar observations and conclusions concerning expertise appear can appear in very different subject areas.
EDIT: I found similar observations in different subject areas in the following papers as well:
The Role of Visual Representations in Advanced Mathematical Problem Solving: An Examination of Expert–Novice Similarities and Differences by Stylianou and Silver
Expert–Novice Differences in the Understanding and Explanation of Complex Political Conflicts by Jones and Read
Posted by Justin at 9:43 PM | Comments (0)
April 21, 2005
"The Stories of Expert and Novice Student Teachers' Supervisors: Perspectives on Professional Development" by Lya Kremer-Hayon
Like the title says this paper was pretty much a compilation of interviews of supervisors. The participants were chosen through recommendations and experience. The participants consisted of 3 experts and 3 novices. One participant from each group was associated with a Kibbutz college and the rest were from State colleges. The inclusion of two different institutions was intentional because they wanted to observe possible effects caused by environmental factors.
The interviews were aimed at discovering the development changes that occur as expertise is acquired. The results of the interview were split into seven categories that were exhibited by each interviewee. The categories are: Knowledge and Change (KC), Reflection and Criticism (RC), Human Relations (HR), Theory and Research (TR), Ideologies and Values (IV), and Difficulties (D). KC, RC, and TR were classified as cognitive aspects, HR and IV were classified as affective aspects, and D covered both aspects. They discovered that experts displayed more characteristics in the cognitive aspects. Novices also displayed characteristics in coginitive aspects but at a smaller percentage than experts. These results showed that cognitive skills developed with expertise for supervisors.
Even though this paper has nothing to do with software engineering, the concept that could be relevant to our study is environmental factors. Environmental factors are something to consider when we select subjects for the test. Different policies and procedures at different companies could have an effect on how a software engineer develops.
Posted by Teerawat at 2:16 AM | Comments (0)
"Knowledge restructuring and the acquisition of programming expertise" by Simon P. Davies
This paper tries to introduce a model which would explain the development in expertise that separates an expert from a novice. The currently existing model associates schemata acquisition with skill development. However, in this paper they introduce a new model that emphasizes knowledge restructuring processes over schemata acquisition. The idea of knowledge restructuring processes is that the expert programmer would develop the focal points of the program first and then build the rest of the program around those points.
In order to support their model, they devised an experiment that consisted of three groups of equal size and different skill level. One group consisted of novices, another of intermediates, and the last being experts. Each group was presented with four Pascal programs and they had 10 minutes to study each program. After they have had a chance to study all four programs, they were presented with a series of 40 probe items that were either focal lines or non-focal lines. For each probe item they were asked to state whether the probe item was present in any of the four programs. The results revealed that experts had the highest percentage of correct results as well as the fastest response time, novices had the lowest percentage correct and the slowest response time, and intermediates were in the middle for both. The results also revealed that the differences in response time and percentage of correct answers were significant for focal lines, but not significant for non-focal lines. This basically supported their model by showing that experts are more aware of focal lines in a program since they build programs starting with focal points due to knowledge restructuring processes.
This paper is relevant to our research in regards to the results that show experts are more observant of focal lines in a program. This just shows us another way that differentiates an expert from a novice. By giving us a characteristic that differentiates the two groups, we can take note of it in our own experiment as well.
Posted by Teerawat at 1:22 AM | Comments (0)
April 18, 2005
Analysing the Novice Analyst: Cognitve Models in Software Engineering by A. G. Sutcliffe and N. A. M. Maiden
This empirical study focused on novice system analysts instead of expert ones because CASE ( Computer-Aided Software Engineering ) tools benefit novices more. The results will be used in the future design of CASE tools. Thirteen novice system analysts were used in this study. They were given a task of developing the specifications for a delivery scheduling system. During the task, which was recorded by audio tape, the subjects were told to think aloud. The recorded results were divided into two main categories: mental and non-mental behaviours. Within these two main ones, they were divided further down. Recognise goal, assertions, reasoning, and planning fell under mental behaviours while information acquisiton and conceptual modelling fell under non-mental.
While our project will focus on both novices and experts, we can incorporate their methodology of thinking aloud and recording their thoughts. I'm not sure if any of the task we're giving in our experiment deals with systems analysis. If it does, then we can model their procedures and methods of analyzing the data.
Posted by John at 1:09 PM | Comments (0)
"The Programmer's Burden: Developing Experise in Programming" by Robert L. Campbell, Normal R. Brown, and Lia A. DiBello
This article describes and illustrates a developmental approach to programming knowledge. A constructivist developmental approach using Piagetian structural interview and tape diaries are used to gather information. The study interviews programming experts and follows individuals learning Smalltalk for 2 months.
Results of the experiment:
Interviews conducted required programmers to introspect and find in retrospect moments in which they viewed themselves becoming more aquinted with a language. The moments describe a metaphor of climbing a mountain, where some moments they think they have climbed over, only to see more of the mountain ahead. Expert programmers could tell which programs have been coded by experts and those done by novices. There's a consensus in a progression from "cookbook" to "intuitive" understanding involved in programming among the interviewees.
The second method, involving tape diaries of programmers learning smalltalk, followed the development of learning the language. From this researches could deduce developmental levels for learning smalltlak. Professional programmers were much quicker to learn Smalltalk pedagogy than novices or non-programmers.
What's important to us:
The tape diaries of much use to us in this experiment. Since the team has less than 10 weeks, performing such tape diaries, let alone find people willing to participate, is impossible. We're better off conducting interviews and performing experiments in the lab.
Binary distinction between novice and expert is inadequate because of enormous variations in skill, knowledge, and productivity among programmers. One person's expert research could be another's novice programmer, so our study should make this distinction.
Novices differ from experts in four general areas
- Syntactic knowledge
- Semantic knowledge
- Schematic knowledge
- Strategies
By taking up the programmer's burden, professional programmers are working against the grain of Smalltalk, and they end up frustrated with its principles of code reuse. This material may be dated, but it's interesting to see that experts will try what they already know in programming problems.
Posted by Matthew at 10:43 AM | Comments (0)
Empirical Studies of Programmers: Sixth Workshop by Wayne D. Gray and Deborah A. Boehm-Davis
This brief article is a synopsis on a workshop discussing the empirical studies done on programmers. Large scale experiments, such as software process appraisals, are impossible to conduct, while smaller scaled experiments are possible. Results from successful experiments are being used in decision making and management. The workshop consists of five panelists. They will discuss recent experiments, methodologies used, implementation issues, and how to bridge the gap between collaborations with academe and industry.
Since our experiment is a collaboration between academe and industry, I’m very interested in what was said during the panel. Too bad this was just a synopsis of the workshop. Perhaps I can try to find the rest of the article.
Posted by John at 6:29 AM | Comments (0)
Characteristics of the Mental Representations of Novice and Expert Programmers: An Empirical Study by Susan Wiedenbeck and Vikki Fix
This study hypothesized that an expert programmer’s mental representation of a program has five abstract characteristics: hierarchical and multilayered, contains explicit mappings between the different layers, founded on the recognition of basic patterns, well connected internally, and well grounded in the program text. To test their hypothesis, they will conduct an experiment with both novices and experts. They hope to show that the novices’ understandings of programs are usually underdeveloped or absent in each of the five areas.
The experiment was conducted with twenty novices and twenty experts. They were given 15 minutes to memorize a 135 line Pascal program. After that, they were given a booklet with 11 questions. The questions tested the subject in all five areas. They were not allowed to look at the code to answer the questions, but they were given an infinite amount of time to respond.
The results did prove that novices did lack in the five areas mentioned, compared to the experts. However, in easy tasks such as naming the simplest recurring patterns, novices were able to perform just as well, if not better, than the experts. They attribute the novice’s lack of understanding in the five areas due to their program comprehension strategy. Experts use a “systematic” approach while novices use an “as-needed” approach.
This study is very relevant to our study. It is extremely similar to the experiment that we will be conducting near the end of this quarter. However, our hypothesis is not very specific as of right now. But we can use their methodology, by giving the subjects a chunk of code to memorize and then asking questions afterwards. When we have formed a more developed hypothesis, we can alter their methodology accordingly
Posted by John at 6:14 AM | Comments (0)
April 17, 2005
"The Advanced Programmer's Reliance on Program Semantics: Evidence from some Cognitive Tasks" by Siu-Yee Wong, Him Cheung, and Hsuan-Chih Chen
This paper examines the effect of programming experience on program semantics. According to the paper, expert programmers are characterized by their ability to organize or "chunk" code. It was also found that expert programmers were more likely than novices to fill in blank lines of code that were semantically intact. But when the program is scrambled then experts were not able to fill the blanks. This led them to the theory that expert programmers are more sensitive and reliant on the semantics of a program.
To test this theory they ran an experiment involving a BASIC program. The experiment consisted of 24 non-programmers, 24 novices with one course in BASIC, and 24 experts with 2+ years BASIC experience. Each participant was given 6 programs each consisting of 10 to 12 lines. They were given a brief amount of time to review and understand each program. Then they were given the same 6 programs with either semantic, syntactic, or no changes made to the program and were asked to decide if they had seen the program before. The results showed that the subjects with more BASIC experience were more able to detect the semantic changes than those with less experience.
This paper is relevant to our project because it discusses the difference in the "chunking" ability between experts and novices which leads to a difference in the reliance on program semantics. Since we are also studying the difference between experts and novices, we can see if an expert's "chunking" ability helps him problem solve better.
Posted by Teerawat at 11:49 PM | Comments (0)
"The Case for Case Studies of Programming Problems" by Marcia C. Linn & Michael J. Clancy
This article describes how programmers organize programming knowledge, analyze how program design skills are commonly taught, and describes the effectiveness of case studies. Case studies are used in an experiment to highlight effective and ineffective strategies for solving programming problems. It emphasizes teaching students design skills by applying general skills to specific domains in order to solve programming-specific problems. An experiment tested the effectiveness of case studies in 10 pre-college Pascal classes. Three conditions were met by the instructors in a random order. The conditions were: Providing a student solution and looking at expert code, providing a student solution plus expert commentary and expert code, and providing expert commentary and expert code.
Results of the experiment:
Students who received the expert commentary learned more about program design than did students who received the expert code without the commentary. Students who implemented their own solution to the problem and then studied the expert commentary learned no more about designing the solution to a computer problem than those who had the expert commentary alone. The students who didn't study the expert commentary were the least successful in the study. Given the fact that most pre-college students normally try to avoid reading, the results show strong evidence for expert commentary. Finally, students need more than just expert code to assist them in learning design skills.
Other highlights from the reading include:
Students organize programming knowledge in terms of language syntax, often causing fragmented knowledge and leading to an engagement of trial and error. Syntax and top-down learning are often used because that's how it is presented to them. There is no emphasis of recognizing & reusing patterns.
Experts organize programming knowledge in larger conceptual structures. Programs are often thought out in plans with goals in mind. Larger programs require combinations of plans. Templates are applied to reocurring problems and competent programmers use top-down, bottom-up, middle-out, and mixes of these styles to design programs.
What's relevant:
Expert commentary helps students at varied ability levels to design more effective programs in case studies. The study found that students complained when they had to read long commentary from experts, but I'm not sure how well highschool student compare with college students. The usage of templates and plans are helpful in teaching novices become better at programming. Top-down teaching is unlikely to develop skills that individuals need to be effective programmers and only a small number of students are capable of inferring program design skills via unguided discovery. A suggested approach is reciprocal teaching. Perhaps we can look into how templates and plans are formulated when we decide what questions to ask novices & experts in our project.
Posted by Matthew at 10:04 PM | Comments (0)
"When Novices Elicit Knowledge: Question Asking in Designing, Evaluating, and Learning to Use Software"
The main topic of the article focused on the interaction between expert and novice users and their learning capabilities. The method in question was the Q&A technique that many feel is a superior form of learning compared to think-out-loud methods. Multiple experiments were run with novice users(office secretaries), where the expert would “coach” the user by answering their questions about the tasks they must perform in a given time limit. Analysis of the experiments included the following:
- What types of question-and-answer dialogues occurred?
- What were questions about?
- Did users ask the right questions?
- Were answers to questions effective?
- What did we learn from questions?
- How are questions interpreted?
The main purpose of studying these questions were to find solutions within human-computer interaction. “In the HCI domain, we want to learn useful things about the users’ experience with computer systems, in order to improve the systems, guide design, and achieve some more general understanding of the mechanisms that underlie computer use as a cognitive skill.” In conclusion, the writer feels that Q&A method is more natural for qualitative analysis than the more familiar think-out-loud method.
Posted by Jonathan at 7:15 PM | Comments (0)
Toward a Unified Theory of Problem Solving: Views from the Content Domains by Richard Mayer
This paper sheds light on different concepts of problem solving. In problems solving there are two kinds of problems (routines and nonroutine); two kindas of problem solving strategies (weak and strong methods); two kinds of problems solvers (novice and experts); and two phases of problem solving (qualitive and quantitative reasoning). The paper defines each of these terms and how they are important in the research of how problem solvers *work*.
This paper is relevant because it gives insight on what to pay attention to in the subjects as they solve problems. It is important that we look into all aspects of our research approach so that we can gather relevant and important data.
Although this paper does not have any direct information on the empirical studies of software engineering, it does give us information on what to look for in our studies. When preparing experiments, we should look into what kind of problems we are creating. We should also research the problem solving strategies that the engineer uses and what problem solving phase they are at.
Posted by Derrick at 6:12 PM | Comments (0)
Expert and Novice Performance in Solving Physics Problems by Larkin, McDermott, Simon, Simon
This paper studies the differences in performance of novice and expert physic solvers. Researchers are looking into how "physical intuition" works. Physical intuition is the solving of "difficult problems rapidly and without much conscious deliberation about a plan of attack." These researchers first looked into how people solved word problems in algebra and how chess players visualize chess boards. This is because physics involve these two skills to solve physics problems. Physic problems are often given as word problems (like algebra) and require problem solving skills (chess playing).
Although this paper does not deal with empirical studies of computer software engineering, it does shed light on the approach of how we can gather information. This paper gives ideas of what to look for in the subjects as they solve problems (do they write diagrams? how do they approach the problem? forward thinking? backward thinking?).
A relevant theory that the paper looks into is memory "chunking". Thinking back into software engineering, a large number of tasks are linked and are inter-related with each other. Perhaps expert engineers are able to visualize larger chunks of this information.
Posted by Derrick at 5:48 PM | Comments (0)
"Expert-Novice Differences and Knowledge Elicitation" by Foley and Hart
In this study, the researchers examine expert-novice differences when developing systems. Foley and Hart argue that it is more important to model ways in which experts and novices interact than to solely model an expert’s knowledge. In this particular study, the researchers wanted to examine the differences between how novices and experts used their knowledge to solve problems.
The subjects in the study included first-year students, second-year students and staff from the computer science department (novice, intermediate, and expert respectively). The subjects were asked to solve three debugging problems. The problems became progressively harder. To track the debugging process of the subjects, portions of the code were written on different cards. This method allowed researchers to track the search path that subjects were using to solve the problem. Also, the subjects were taped and asked to think aloud to capture what they were thinking at the moment.
Overall, the study revealed that experts were more likely solve the problem. The researchers noticed that the experts examined more hypotheses for solving the problem. Moreover, the strategies between novices and experts were different. Ultimately, the researchers argue that it is important to note the differences between novices and experts during knowledge elicitation.
This study is relevant because it provides another model for conducting our own study. It highlighted the importance of recording the data through tape and thinking aloud. More importantly, it compared the differences between experts and novices just like our own study.
Posted by Justin at 4:53 PM | Comments (0)
"What Do Expert Programmers Communicate by Means of Descriptive Commenting?" by Riecken, Koenemann-Belliveau, Robertson
In this study, researchers examined what ways expert programmers use descriptive commenting to communicate program knowledge to novice programmers. To conduct the study, the researchers asked several expert programmers to comment a simple parsing program consisting of a main(), parse(), hash(), and errorExit() methods. The research focused on two methods of the methods in the study: parse() and hash(). These methods served as a focus because the parse method runs the majority of the programming, while the hash method contains complex code. The researchers argue that although novices would be more concerned with the syntax of the hash method, experts would be more concerned with the semantics of the parse method. Thus, the experts would be more likely to comment the parse method.
The subjects of the study were 16 expert programmers who had at least 10 years of broad programming language experience, 6 years C programming experience, and was a project leader. The subjects were asked to make the code of the parser easier to read for a novice programmer. Also, subjects were asked to "think aloud" to capture their thinking patterns.
Overall, the study revealed that the experts were more likely to comment the parse method. The researchers argue that expert programmers are more likely to "communicate semantic domain knowledge concerning the function and behavior of complex syntactical program knowledge." Novices, on the other hand, feel they also need help on the syntax of complex code.
This study is relevant to our project because it provides an example of a method for examining how expert programmers think. The study provides insight on how to get experts to “think aloud” and share their ideas. A similar method could be used in our own study.
Posted by Justin at 4:30 PM | Comments (0)
April 16, 2005
"The Initial Stage of Program Comprehension"
The study presented by this article reveals the importance of “beacons” in program comprehension. Program comprehension refers to the ability to understand what is the goal of a program, based on viewing the functions and documentation within the code. Wiedenbeck describes the term beacon as “segments of code which serve as typical indicators of the presence of a particular programming structure or operation.” In the experiments discussed, the example beacon used was the swap function which is correlated with sorting algorithms. Four different experiments were held to depict the importance of beacons:
Disguise Experiment – Two samples of code for a Shellsort algorithm were given to novice and advance users. The first sample contained the swap method, which is regularly found in this algorithm. The second sample, however, consisted of an alternate method of swap which allowed the function to still work. Results showed that a significant majority of users recognized the first sample as a sorting algorithm. Only one-fourth were able to tell that the second sample was another implementation of Shellsort. This suggests that the few lines of code which implemented the swap method played a significant role in determining the goal of the algorithm.
Prototypical/Non-Prototypical Sort Experiment – In this experiment, two samples of code were given to both parties in the same fashion. However, the algorithms given were two functions which both novice and expert users have never seen before. The first sample contained the swap beacon while the second performed the same sort but with different methods. Once again, results showed that a majority were able to deduce that the first function was a sorting algorithm. Virtually no one was able to tell what the purpose for the second algorithm. This experiment suggested that regardless of the familiarity of code surrounding it, the swapping beacon aided users in determining that it was a sorting algorithm.
Intrusion Experiment – This experiment was implemented to show the negative effects of beacons in program comprehension. The samples provided were both implementations of binary search, however in the second sample, a swapping method was added in the body of the code. The experiment recorded instances of whether the subjects classified the code as a binary search or a sort. Almost all expert users recognized the first sample as a binary search, and only half recognized the second sample with the intrusion as a binary search. Less than half of novice users recognized it as a binary search, and no novice users were able to determine the function of the second sample. Very few subjects classified the first sample as a sort but over 80% of both parties classified the second sample as a sorting algorithm. The result of this experiment showed that the presence of a false beacon in a program has a strong tendency to mislead programmers about the program’s function.
Defective Sort Experiment – The final experiment presented four different samples of code. The first was a correct implementation of a sort procedure which contained nested loops and the swap method. The second version contained all the same code except one of the nested loops which was deleted. The third sample took all loops out of the code altogether. And finally the last version contained all loops intact but removed the swap beacon. The purpose of this experiment was to determine if secondary beacons(nested loops) would effect the decisions of programmers as much as primary beacons(swap) which they are accustomed to. The results went against the hypothesis of the experiment where secondary beacons also have an effect on program comprehension yet reinforced the hypothesis that the primary beacon plays a significant role on comprehension.
In conclusion, the study was able to determine that beacons play both a positive and negative role in program comprehension. Since there are many repetitive and custom methods of programming, software engineers rely on these beacons to understand and recognize the functions presented to them.
Posted by Jonathan at 8:27 PM | Comments (0)
"A Note on the Quantification of Computer Programming Skill"
This article explains that recent studies have shown only one side of expertise when it comes to computer programming skill. Studies of expertise have usually compared expert programmers with novice programmers and examined “time-based” skill, in terms of time spent programming. The article emphasizes that the tasks that are performed and the languages used to perform them should also be measured to evaluate what it calls “multiskilling” in programming. There is difficulty in quantifying and measuring multiskilling, however, it was quantified using entropy measures derived from informational theory. Multiskilling was then divided into two separate parts:
1) ”Language entropy” which measures the frequency of use of multiple languages over a period of time
2) “Task entropy” which measures time spent and a variety of tasks in different categories (e.g. scientific or commercial computing).
Since there was a lack of multiskilling information, a survey was mailed to hundreds of professional computer programmers asking questions about the languages they know. Some examples included the year the language was learned, the frequency with which each language was used, and which language they wished to learn next. The results yielded that “actual time spent programming and interval since first learning to program loaded highly on the first factor(time-based skill), while language and task entropy and the number of languages known loaded highly on the second factor(multiskilling).” The article was able to derive evidence supporting the importance of multiskilling by examining the impact of ageing on work skills and obsolescence. “In order to remain current, programmers must learn new languages, update their skills, and demonstrate the same flexibility required of staff in other areas of work.” Basically, programmers must become multiskilled. Evidence also showed that older programmers had no desire to learn new programming languages while two thirds of other programmers surveyed had future intentions to learn C and C++ which were the newly introduced programming languages at the time. In conclusion, the article stressed that although expertise in one language have been valued in the past, employers should now consider those who have multiskilling abilities overall.
Posted by Jonathan at 2:12 PM | Comments (0)
August 19, 2004
On Abilities and Domains
The Psychology of Abilities, Competencies and Expertise
On Abilities and Domains
Author(s) : Michael W. Connell, Kimberly Sheridan, and Howard Gardner
Chapter 5, Page 126
Studies different individuals in different job areas who are successful.
Posted by kim at 8:55 AM | Comments (0)
The Psychology of Abilities, Competencies and Expertise
The Psychology of Abilities, Competencies and Expertise
Edited by Robert J. Sternberg & Elena L Grigorenko
Cambridge University Press
Cambridge, England
2003
[1]
Article Name: Trait Complexes, Cognitive Investment, and Domain Knowledge
Author(s): Philip Ackerman and Margaret E. Beier
Chapter 1, Page 1
Premise:
Seek to understand the development of expertise as an interaction between individual characteristics (abilities, personality, interests, self-concept, and so forth) and the environment, as jointly influencing which person develop expertise and which persons do not.
Methods:
Different types of tests tested on adults on general aptitude vs their domain knowledge.
Ideas:
Testing to prove either ability or achievement has its drawbacks because ability tests don't show what people have learned and achievement tests only focuses on the specialized knowledge domains. [Difficult to do both at same time] p2
Intelligence tests (ie SAT + GRE) "yield results that substantially underestimate the intellect of adults" p26 -- more comprehensive tests must be created to be able to encompass domain knowledge and expertise.
--------------------
[2]
Article Name: On Abilities and Domains
Author(s) : Michael W. Connell, Kimberly Sheridan, and Howard Gardner
Chapter 5, Page 126
Premise:
An "alternative approach for linking abilities to potential for expertise by identifying two quantitatively different kinds of human abilities (modular and integrative) that we believe correlate with corresponding categories of problems (modular tasks and integrative situations, respectively). " Also, "individuals with highly developed specific capacities are more likely to be attracted to and excel in targeted task areas, whereas individuals with strong integrative capacities are more likely to be attracted to and excel in professions that require situational competencies". p127
Methods:
Studies different individuals in different job areas who are successful.
Ideas:
Some people "assume that every human has the same set of abilities as every other human (that is, expertise is independent of ability, and mostly or only a function of practice)" p135
While others "assume that there is a tight coupling between general abilities (for example, general intelligence) and attainable expertise in any domain." p135
The third perspective "hold that individuals have profiles of specific abilities (for example, mathematics or music) that enable them to excel in particular domains (for example, in the field of mathematics or in musical composition, resepectively)." [This is the view of the article]. p135
-----------which one do i think?
List on page 153 of general ides of article:
1. it should be possible to focus not on raw abilities per se but rather on the fit between an individual's profile and the available career options.
2. Both within and across domains, it would be useful to develop measures of task-competence, on the one hand, and situational competence on the other.
3. For both cognitive and personality considerations, individuals may differen in their proclivities towards task- or situational-competence. Using our notion of an ability space, points close to an axis reflect a highly peaked ability profile and suggest an individual "at promise" for task-like pursuits. In contrast, individuals with points further from an axis and a flatter profile of intelligences, may suggest more potential for situateional mastery and integration of information across domains.
4. Greater efforts ought to be expended in modeling and training situational competence, synthesizing and integrating capacities and in documenting success in doing so. Computer-aided simulations are a promising means for accomplishing this end.
5. It may be useful to create a map charting the demands of different domains and sub-domains, in terms of both content demends and task-situation demands. On this map one could overlay a map of individual ability profiles known to be successful in those domains. In this way one might be able to develop assessments that have an early predictie ability using the formal mesaures of fit that have been defined.
--------------------
[3]
Article Name: Expertise, Competence, and Creative Ability The Perplexing Complexities
Author(s) Dean Keith Simonton
Chapter 8, Page 213
Premise:
"The phoenomenon of creativity highlights some critical issues about the nature of abilities, expertise, and competencies" p214.
-basically how much is all this affected by creativity? was einstein just very smart or did his creativity push his abilities further?
Methods
Empirical evidence from other studies. Studies aren't that interesting.
Ideas
How can creativity be tested?
-domain specific aptitude tests?
-personality tests?
Does creativity require expertise?
Though certain findings seem to say that "creativity is merely an expertise", this guy differs by saying that creativity is not always come from expertise. 222
Posted by kim at 8:44 AM | Comments (0)