Empirical Studies of Software Engineering Expertise

Design Expertise

Andre van der Hoek sent me the following reference to novice-expert differences in (architecture) design.

Variants in Design Cognition
Omer Akin
School of Architecture
Carnegie Mellon University

Week 10 Progress Report

Forgot to do this yesterday.

1. Ran my experiment with my subjects and Justin helping me.

2. Helped Derrick run his experiment.

3. Spent Sunday, Monday, and today going the videos and analyzing them.

4. Came up with my hypotheses on Sunday and revised them in the meeting yesterday.

5. Started writing the bits and pieces that I could for my paper.

6. Graded the implementations according to the answer key Justin created.

Week 9

- Attended experiment by Teerawat & Justin
- Found two expert software developers
- Scheduled Expert/Expert run with John for Friday 4pm
- Matt's assistant for tomorrow

Woohoo, I caught up.

Progress Report Week 9

1. Found a novice and an expert subject for the experiment.

2. Set up an experiment time for Tuesday June 7th, from 3:30 pm to 7:30 pm.

3. Ran a test run of the full experiment with Justin, Derrick, Matt, John, Sukanya, and Erin on Saturday.

Pilot Testing

Was test subject 1 in pilot study Saturday. Morae kept giving the strange "Unknown error has occurred" many times and we kept restarting until it finally worked. None of us knows what this descriptive error message means. Perhaps it's because we're using a larger monitor resolution than Morae recommends. Lowering the resolution only makes the screen blurry.

The only backup option I can think of is to capture with VNC and record with the DV camera we have.

Progress Report Week 8

-Updated the handbook (changed the text to a scenario-based script, modified the task to match the new task)
-Updated the Task Description to include the new task.

Week 8 Progress Report

- Pilot test for new task with Justin
- Worked more on online survey.
- Updated protocol handbook
- Still need to create grading script, need to talk to Erin on how to score IQ and personality tests

Pre-experiment questionnaire

IQ test

Personality test

Figures for IQ test (given as a separate sheet to the subject)

CS Questionnaire figure (given as a separate sheet to the subject)

Week 8 Progress Report

- pilot tested novice -> expert interaction on Friday
- updated handbook (task script, added new appendix)

Hardware Update

Jonathan and I were able to get part of the computer working the subject will be using. The webcam & Morae are working on the machine. We installed the Radeon 9700, only to remove it because the card only accepts DVI. We need a DVI->VGA adapter. They should be had for less than $20. Once we have it, the card should work. The firewire card is also installed.

We also tried running Eclipse through Remote Desktop. After testing we found it is much faster to run everything on one machine.

I'm leaving town Thursday and won't be available until Monday. John and Jonathan should be able to finish the rest of the equipment setup by the end of the week.

Pyschology of Programming by J.M. Hoc

- Methodological Issues in the Study of Programming ( pg. 83 )
o Types of data collection – results of each need to be interpreted differently, and the results of each are often necessary for the design of further research using other approaches
hypothesis testing – contrasting theoretical positions make different predictions about the effect of some manipulation on behavior
Comparisons – discover which of two ( or more ) alternatives is easier to use, or whether some change to the programming process has an effect on performance.
- Evaluations – fine line between evaluations and comparisons, there is considerable overlap. “Can people use flowcharts?”, rather than “Are flowcharts better than structure diagrams?”, also used with the intention of improving a system.
- Exploration – the main research theme is novel and we lack enough information to engage in the other types of data collection
- The experimental task – “You can’t study people using one system and assume your results generalize; you can’t study a single task and assume you found out all the important facts”, for truly generalizable conclusions, a variety of tasks, a variety of type of programmers and a variety of languages should be studied.
- Sample tasks – asking subjects to write a program and “think aloud” while they do it, giving them a program and giving them a comprehension quiz about it, scrutinize code either to classify types of errors or to compare the frequencies of different solution types, timed question answering with miniature programming languages, recalling programs that have been viewed briefly,
- Majority of studies of program writing have used observational techniques, either video or post hoc analysis of code.
- While single tasks, single subject populations and single languages may be used to disprove a theoretical hypothesis, they are of little use when they confirm hypotheses.
- Important concepts in experimental design
- Statistical significance – provides a measure of the likelihood of the observed difference in scores being a result of chance variations in the performance of four subject groups, assuming that they were drawn from the same population and that the experimental manipulations had no influence on their performance.
- Effect size – this reflects the influence on a score of the experimental manipulations, though no just by magnitude, but also by variability.
- Sample size – the main problem with small samples arises when the results are not significant ( i.e. p > 5% ) since the use of a larger sample, all else being equal, might reduce p to below 5%.
- Power – measure of the quality of an experimental design, constructing an experiment of sufficient power to detect an effect, without overkill.
Observational techniques
- Interviews with programmers
- Video plus verbalization
- Constructive interaction
- Longitudinal studies of learning
- Social and organizational processes
- Applying research – what was the question?, interpreting causes
Experimental validity
- Internal – describes our ability to be sure that our explanation of the observed differences is the only likely explanation, and it is closely related to the statistical issues discussed earlier.
- External – describes our ability to make correct generalizations about the implications of the results.
- In most cases external validity is only an issue when we have high internal validity.
- Hypothesis-testing must place most emphasis on internal validity, while comparisons and evaluations need to emphasize external validity.
- Experts represent programs in terms of semantic structure, whereas novices encode them syntactically.
- Experts tend to spend more time than novices planning and evaluating.
- The pattern overall is that experts are able to handle information at different levels. They differ from novices in two important respects: their ability to develop overviews or abstract models of solutions, and their ability to understand the consequences of implementation detail. ( pg. 108 )
- For experts, particular languages or language types did not correspond to particular solution strategies, although there was some correspondence between languages and oversights. It seems that experts conceive an abstract model of the solution separate from its expression in a particular programming language. They envision the solution in an pseudo-language, which is a collage of models borrowed from many sources. ( pg. 108 )
- Experts were tenaciously resistant to the change of algorithm. ( pg. 108 )

week 7 progress report

1. Wrote the Recruitment and Scheduling sections for the handbook.
2. Tested Morae with remote desktop with Jonathan and Matt.
3. Posted on Pyschology of Programming.

Arrival

Protocol Handbook

Arrival

Stimuli needed:
1. Consent form
2. Pen

Preparation:
1. If subject is a novice, get consent form A… Else if subject is an expert, get consent form B… Else get consent form C.
2. Get pen for subject.

(Take subject to library in ICS 2. (ICS2 257))

(Have subject take a seat at a table. Have consent form and pen at hand.)
“Thank you for taking the time to participate in our human research project. We would like to remind you that participating in this research project is completely voluntary and you are welcome to leave at any time. However, to receive your compensation, you must complete to the best of your ability all requested questionnaires and task experiments. We estimate the experiment to last for about two and a half to three hours. Is this okay with you?”

(If subject agrees, continue…)

(Give consent form to subject)

“This is a consent form that contains information about our studies and how our research will be carried out. It also contains important information on confidentiality and the risks involved. Please read over this form carefully. After reviewing this form, please print your name at the top of page one clearly and sign and date on page three. If you have any questions, feel free to ask them at any time.”

“Print name clearly here.”
(Highlight and point to top of page where subject prints name)
“Sign and date here.”
(Highlight and point to page three where subject signs name)
(Give pen to subject)
“Please hand in your form and pen to me when you are finished.”

(When subject hands in consent form, double check printed name and signature. Review with subject over the spelling of subjects name to ensure legibility. Then print and sign your name on page three).

(File subject’s consent form in correct filing folder).

(If subject = A, Go to Task prompt… Else go to Questionnaire)

Week 7 Progress Report

1. Wrote "Arrival" section for protocol handbook.
2. Talked with Erin and debated pros and cons of electronic questionnaire or paper questionnaire.
3. Decided to use VTSurvey to create surveys, IQ, and personality tests. Will use script to calculate scores.

Week 7 Progress Report

1. Wrote part of the handbook (Task A, Interaction, Task B)
2. Pilot tested with Yuzo (an expert) to determine the feasibility of the actual task. After looking at the code and the task he estimated that the task might take a "day" to finish.
3. Discussed alternatives to our existing task with Teerawat and Sukanya. We need to clarify more in person tomorrow.

Progress Report Week 7

1. Wrote part of the handbook and combined my section with Justin's. I posted the combined version in my previous post.