« April 2005 | Main | June 2005 »

May 31, 2005

Week 8 Progress Report

- Pilot test for new task with Justin
- Worked more on online survey.
- Updated protocol handbook
- Still need to create grading script, need to talk to Erin on how to score IQ and personality tests

Pre-experiment questionnaire

IQ test

Personality test

Figures for IQ test (given as a separate sheet to the subject)

CS Questionnaire figure (given as a separate sheet to the subject)

Posted by Derrick at 3:46 PM | Comments (0)

Week 8 Progress Report

- pilot tested novice -> expert interaction on Friday
- updated handbook (task script, added new appendix)

Posted by Justin at 8:58 AM | Comments (0)

May 25, 2005

Hardware Update

Jonathan and I were able to get part of the computer working the subject will be using. The webcam & Morae are working on the machine. We installed the Radeon 9700, only to remove it because the card only accepts DVI. We need a DVI->VGA adapter. They should be had for less than $20. Once we have it, the card should work. The firewire card is also installed.

We also tried running Eclipse through Remote Desktop. After testing we found it is much faster to run everything on one machine.

I'm leaving town Thursday and won't be available until Monday. John and Jonathan should be able to finish the rest of the equipment setup by the end of the week.

Posted by Matthew at 5:37 PM | Comments (0)

May 23, 2005

Pyschology of Programming by J.M. Hoc

- Methodological Issues in the Study of Programming ( pg. 83 )
o Types of data collection – results of each need to be interpreted differently, and the results of each are often necessary for the design of further research using other approaches
hypothesis testing – contrasting theoretical positions make different predictions about the effect of some manipulation on behavior
Comparisons – discover which of two ( or more ) alternatives is easier to use, or whether some change to the programming process has an effect on performance.
- Evaluations – fine line between evaluations and comparisons, there is considerable overlap. “Can people use flowcharts?”, rather than “Are flowcharts better than structure diagrams?”, also used with the intention of improving a system.
- Exploration – the main research theme is novel and we lack enough information to engage in the other types of data collection
- The experimental task – “You can’t study people using one system and assume your results generalize; you can’t study a single task and assume you found out all the important facts”, for truly generalizable conclusions, a variety of tasks, a variety of type of programmers and a variety of languages should be studied.
- Sample tasks – asking subjects to write a program and “think aloud” while they do it, giving them a program and giving them a comprehension quiz about it, scrutinize code either to classify types of errors or to compare the frequencies of different solution types, timed question answering with miniature programming languages, recalling programs that have been viewed briefly,
- Majority of studies of program writing have used observational techniques, either video or post hoc analysis of code.
- While single tasks, single subject populations and single languages may be used to disprove a theoretical hypothesis, they are of little use when they confirm hypotheses.
- Important concepts in experimental design
- Statistical significance – provides a measure of the likelihood of the observed difference in scores being a result of chance variations in the performance of four subject groups, assuming that they were drawn from the same population and that the experimental manipulations had no influence on their performance.
- Effect size – this reflects the influence on a score of the experimental manipulations, though no just by magnitude, but also by variability.
- Sample size – the main problem with small samples arises when the results are not significant ( i.e. p > 5% ) since the use of a larger sample, all else being equal, might reduce p to below 5%.
- Power – measure of the quality of an experimental design, constructing an experiment of sufficient power to detect an effect, without overkill.
Observational techniques
- Interviews with programmers
- Video plus verbalization
- Constructive interaction
- Longitudinal studies of learning
- Social and organizational processes
- Applying research – what was the question?, interpreting causes
Experimental validity
- Internal – describes our ability to be sure that our explanation of the observed differences is the only likely explanation, and it is closely related to the statistical issues discussed earlier.
- External – describes our ability to make correct generalizations about the implications of the results.
- In most cases external validity is only an issue when we have high internal validity.
- Hypothesis-testing must place most emphasis on internal validity, while comparisons and evaluations need to emphasize external validity.
- Experts represent programs in terms of semantic structure, whereas novices encode them syntactically.
- Experts tend to spend more time than novices planning and evaluating.
- The pattern overall is that experts are able to handle information at different levels. They differ from novices in two important respects: their ability to develop overviews or abstract models of solutions, and their ability to understand the consequences of implementation detail. ( pg. 108 )
- For experts, particular languages or language types did not correspond to particular solution strategies, although there was some correspondence between languages and oversights. It seems that experts conceive an abstract model of the solution separate from its expression in a particular programming language. They envision the solution in an pseudo-language, which is a collage of models borrowed from many sources. ( pg. 108 )
- Experts were tenaciously resistant to the change of algorithm. ( pg. 108 )

Posted by John at 3:13 PM | Comments (0)

week 7 progress report

1. Wrote the Recruitment and Scheduling sections for the handbook.
2. Tested Morae with remote desktop with Jonathan and Matt.
3. Posted on Pyschology of Programming.

Posted by John at 3:10 PM | Comments (0)

Arrival

Protocol Handbook

Arrival

Stimuli needed:
1. Consent form
2. Pen

Preparation:
1. If subject is a novice, get consent form A… Else if subject is an expert, get consent form B… Else get consent form C.
2. Get pen for subject.

(Take subject to library in ICS 2. (ICS2 257))

(Have subject take a seat at a table. Have consent form and pen at hand.)
“Thank you for taking the time to participate in our human research project. We would like to remind you that participating in this research project is completely voluntary and you are welcome to leave at any time. However, to receive your compensation, you must complete to the best of your ability all requested questionnaires and task experiments. We estimate the experiment to last for about two and a half to three hours. Is this okay with you?”

(If subject agrees, continue…)

(Give consent form to subject)

“This is a consent form that contains information about our studies and how our research will be carried out. It also contains important information on confidentiality and the risks involved. Please read over this form carefully. After reviewing this form, please print your name at the top of page one clearly and sign and date on page three. If you have any questions, feel free to ask them at any time.”

“Print name clearly here.”
(Highlight and point to top of page where subject prints name)
“Sign and date here.”
(Highlight and point to page three where subject signs name)
(Give pen to subject)
“Please hand in your form and pen to me when you are finished.”

(When subject hands in consent form, double check printed name and signature. Review with subject over the spelling of subjects name to ensure legibility. Then print and sign your name on page three).

(File subject’s consent form in correct filing folder).

(If subject = A, Go to Task prompt… Else go to Questionnaire)

Posted by Derrick at 9:42 AM | Comments (0)

Week 7 Progress Report

1. Wrote "Arrival" section for protocol handbook.
2. Talked with Erin and debated pros and cons of electronic questionnaire or paper questionnaire.
3. Decided to use VTSurvey to create surveys, IQ, and personality tests. Will use script to calculate scores.

Posted by Derrick at 9:38 AM | Comments (0)

May 22, 2005

Week 7 Progress Report

1. Wrote part of the handbook (Task A, Interaction, Task B)
2. Pilot tested with Yuzo (an expert) to determine the feasibility of the actual task. After looking at the code and the task he estimated that the task might take a "day" to finish.
3. Discussed alternatives to our existing task with Teerawat and Sukanya. We need to clarify more in person tomorrow.

Posted by Justin at 11:56 PM | Comments (0)

Progress Report Week 7

1. Wrote part of the handbook and combined my section with Justin's. I posted the combined version in my previous post.

Posted by Teerawat at 11:30 PM | Comments (0)

Handbook Section

Here's what Justin and I came up with so far for the handbook. Task B is still up in the air because we discovered that the task is harder than we initially thought and need a new task.

Handbook
Task A
- Provide subject with Program Description, Task Description, and CCB Form
- Inform the subject that he or she will have access to the following resources:
o Program Description
o Overall Task Description
o Source Code of VTSurvey
 Located under the “webapps” folder in the Apache Tomcat folder (still need to get exact URL)
o VTSurvey website (http://vtsurvey.sourceforge.net)
o Running version of the code
o Any resource on the Internet.
o Paper
o Pen/Pencil
o Eclipse IDE
o TextPad
- Inform them Java source files located in C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\survey\WEB-INF\src\edu\vt\ward\survey
- Inform them JSP source files located in C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\survey\WEB-INF
- Inform them that they can access a running version of the program by entering this URL into a web browser: http://localhost:8080/survey/index.jsp
- Inform them they have ___ hours to complete the task
- Explain to subject that their task is to go through and understand the program so they can fill in the CCB proposal form detailing the changes that need to be done.
- Ask the user to think aloud. Provide positive cues to keep the subject talking
- Once they have an understanding of the task and the program ask them to fill in the CCB form. Ask them to provide as much detail as possible.
Interaction (Optional)
- While Subject B is outside of the room have them, read the initial Program Description and Task Description.
- Inform Subject B that Subject A is going to provide him or her with a document which will further explain the task. Subject A will also give an explanation of the changes needed for the task. Finally, inform Subject B that he or she will be able to ask Subject A questions about the task once Subject A has finished his or her explanation
- With Subject A still in the room, bring in Subject B.
- Inform them that they can use paper/whiteboard to support explanations and questions
- Inform them they will have ___ hours to explain/ask questions
- Subject A explains task and changes needed
- Subject B asks questions
- Subject A leaves the room.
Task B
- Provide subject with Program Description, Task Description, and CCB Form
- Inform the subject that he or she will have access to the following resources:
o Program Description
o Overall Task Description
o Source Code of VTSurvey
 Located under the “webapps” folder in the Apache Tomcat folder (still need to get exact URL)
o VTSurvey website (http://vtsurvey.sourceforge.net)
o Running version of the code
o Any resource on the Internet.
o Paper
o Pen/Pencil
o Eclipse IDE
o TextPad
- Inform them they have ___ hours to complete the task
- Inform them Java source files located in C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\survey\WEB-INF\src\edu\vt\ward\survey
- Inform them JSP source files located in C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\survey\WEB-INF
- Inform them that they can access a running version of the program by entering this URL into a web browser: http://localhost:8080/survey/index.jsp
- Ask the subject to review the CCB proposal received from subject A
- Ask the subject to read the through the code and verify that the changes describe by subject A is correct and feasible.
- Ask the subject to make corrections to the CCB in a new CCB if corrections are needed.
- Ask the subject to explain whether the CCB written by subject A is feasible or not
Appendix:

Posted by Teerawat at 11:27 PM | Comments (0)

Week 7 Progress

Did 2nd run through (5/18) of video pilot and tried Remote Desktop Solution. One machine (Computer A) was using Morae. Another machine (Computer B) was used to access Computer A via Remote Desktop. Computer B would be the one the subjects would use. Result: unusably slow. John has video of it.

E-mailed script to Jonathan to check out errors and test out with using Morae.

Posted by Matthew at 7:25 PM | Comments (0)

May 20, 2005

Went shopping! Wheee

I ordered the following equipment today and it should arrive early next week. Total came to $771 with taxes and shipping, so if we end up purchasing Morae, we'll be on target with the budget. I forgot to order a microphone, but I should be able to find one that we can use. Also, I'll probably need to order a DVD burner, but that's less urgent.

Susan

Sony DCR-HC21 MiniDV Handycam Camcorder w/20x Optical Zoom
$370.49

Logitech QuickCam Pro 4000 (961239-0403)
$72.29

FlexStand - Flexible Web Camera Stand
$32.95

Sony DVM-60PR 60 Minutes Premium Mini DV Video Tape Cassette - 10 Pack
$32.99

Adaptec 1890600 Fireconnect 4300 Kit AFW-4300
$39.88

ATi All-in-Wonder RADEON 9700 PRO 128MB DDR AGP8X w/o Remote (OEM)
$151.00

Posted by ses at 2:57 PM | Comments (0)

May 18, 2005

Week 6 Progress

Video Pilot testing

1st run through - 5/18/05 - Concept

Johnathan, John, and I performed some pilot tests to test the usability of capturing data while a subject is performing assigned tasks. We attempted to perform tasks a subject might do such as opening applications, working on a project in Eclipse, instant messaging people, browsing the internet, and editing text files. We also tested out a firewire webcam to see if two users would be captured in a single frame. Tests were also conducted on the use of Morae itself and the playback of recorded video.

Installed Morae on 2 machines

Test case 1

Results

We noticed a slowdown when recording versus using only Eclipse. Since the slowdown occurred even without a webcam attached, we decided adding more programs will only take up more resources and slowdown the machine even more. The use of Windows Remote Desktop would only inhibit the machine further, also bringing about problems of video syncronization between what Morae records and the speed Remote Desktop would be able to capture from another machine. A more powerful video card should be used. On-board one not good enough.

Test case 2

Results

Eclipse's compiling speed was alright. Too a little bit longer to open program than normal. Scrolling was noticably slower. AIM had nothing problematic to report. Using internet Explorer for looking up JDK documentation was slow scrolling and laggy. TextPad had some slowdown in typing

Filesizes

Enoding WMV (takes 5 min) @ 640x480 Standard. File size: 18.8 MB

Lossless AVI (takes 3.5 min) @ 640x480 Standard. File size: 25.5 MB

Morae Project files are larger than AVI files. Test 1 was 6 MB & Test 2 was 33 MB.

Overall performance from test cases

Concerns the group has

Posted by Matthew at 3:20 PM | Comments (0)

week 6 progress report

1. created an gmail account for scheduling purposes: sims.ics199@gmail.com

2. read through "Psychology of Programming".

3. tested out Morae with Jonathan & Matt.

Posted by John at 2:03 PM | Comments (0)

Progress Report Week 6

1. Tried to implement the drop down menu, but it was more complicated than I thought. In the end I couldn't get it working.

2. A friend of ours helped out in the pilot test by performing the first part of the experiment and emailed us a CCB form. For the most part it met our expectations for a novice.

Posted by Teerawat at 12:06 AM | Comments (0)

May 17, 2005

Week 6 Progress Report

-Made updated changes to CS questionairre
-Performed pilot test (understanding VTSurvey and creating CPB Form)

Posted by Derrick at 10:06 PM | Comments (0)

Week 6 Progress Report

1. Tried to implement the actual solution.
2. Sent out the task to a couple of people (they returned the CCB form)
3. For the most part, I think the understanding of the task can be done, but the implemenation maybe a little more diffcult than originally expected.

Posted by Justin at 10:01 PM | Comments (0)

May 9, 2005

Week 5 Progress

Created shopping list with Jonathan & John (see previous post)

Posted by Matthew at 2:10 PM | Comments (0)

Code Complete by Steven McConnell

- Architecture – the quality of the architecture determines the conceptual integrity of the system, which in turn determines the ultimate quality of the system. ( pg. 44 )
- Attention to quality at the beginning has a greater influence on product quality that attention at the end. ( pg. 59 )
- Every programming language has weaknesses and strengths. Be aware of the specific strengths and weaknesses of the language that you’re using. Establish programming conventions before you begin programming( pg. 70 )
- Major design heuristics ( pg. 108 )
o Find real-world objects
o Form consistent abstractions
o Encapsulate implementation details
o Inherit when possible
o Hide secrets ( information hidings )
o Identify areas likely to change
o Keep coupling loose
o Look for common design pattersn
-Guidelines for using heuristics ( pg. 109 )
1. understanding the problem
2. devising a plan
3. carrying out the plan
4. looking back
- Good design is iterative; the more design possibilities you try, the better your final design will be. (pg. 123 )
- A class interface should hide something – a system interface, a design decision, or an implementation detail. Inheritance is a useful tool, but it adds complexity. Keep in mind that the primary tool for managing complexity is classes. ( pg. 160 )
- Routine guidelines: ( pg. 161 )
o Cohesion – the ultimate goal is to have each routine do one thing well and not do anything else
o Give routines a name that describes exactly what it does, no matter how long it is.
o Put parameters in input-modify-output order.
- Practice defensive programming. Use exception handling and assertions.
- PPP ( Pseudocode Programming Process ) is a useful tool for detailed design and makes coding easy. Pseudocode translates directly into comments, ensuring that the comments are accurate and useful. The steps in constructing routines by using the PPP are design the routine, code the routine, check the code, clean up loose ends, repeat as needed. ( pg. 215 )
- Variable guidelines: ( pg. 237 )
o Pay attention to data initialization, for it is one of the biggest source of error in computer programming.
o Minimize the scope of each variable. Keep references to a variable close together. Keep it local to a routine or class. Avoid global data.
o Use each variable for one and only one purpose.
o Names should be as specific as possible.
- Remember the individual rules for each data type. ( pg. 318 )
- Structures can help make programs less complicated, easier to understand, and easier to maintain, but always consider whether a class would work better.
- Pointers are error-prone. Protect yourself by using access routines or classes.
- Global variables should be avoided, but if you can’t, work with them through access routines. ( pg. 344 )
- The strongest principle for organizing straight-line code is ordering dependencies. Dependencies should be made obvious through the use of good routine names, parameter lists, and comments. ( pg. 353 )
- To trap errors, use the default clause in a case statement or the last else in a chain of if-then-else statements. ( pg. 366 )
- Recursion can provide elegant solutions but keep this tips in mind: make sure the recursion stops, use safety counters to prevent infinite recursion, and limit recursion to one routine. ( pg. 396 )
- Tables ( direct access, indexed access, stair-step access ) provide an alternative to complicated logic and inheritance structures. ( pg. 430 )
- Minimizing complexity is a key to writing high-quality code. ( pg. 460 )
- How software measure up against the external and internal software qualities?
o External – correctness, usability, efficiency, reliability, integrity, adaptability, accuracy, robustness
o Internal – maintainability, flexibility, portability, reusability, readability, testability, understandability
- Techniques for improving software quality ( pg. 466 )
o Software-quality objectives
o Explicit quality-assurance activity
o Testing strategy
o Software-engineering guidelines
o Informal technical reviews
o Formal technical reviews
o External audits
- Code-tuning techniques ( pg. 609 )
o Logic
o Loops
o Data transformations
o Expressions
o Routines
o Recoding in a low-level language
o The more things change, the more they stay the same
- The first priority of visual layout is to illuminate the logical organization of the code. Criteria used to assess whether that priority is achieved include accuracy, consistency, readability, and maintainability. ( pg. 775 )
- Good code is its own best documentation. If the code is bad enough to require extensive comments, try first to improve the code so that it doesn’t need extensive comments. ( pg. 817 )
- The characteristics that matter the most are humility, curiosity, intellectual honesty, creativity and discipline, and enlightened laziness. ( pg. 835 )

Posted by John at 11:08 AM | Comments (0)

Week 5 Progress Report

1. Met with Teerawat and Sukanya and decided on using Virginia Tech's open-source survey creation application, VTSurvey, for our experiement.
2. Analyzed the code to verify how feasible it is as our experiment program. It seems like it is.
3. Re-typed the "Experimental Task" and sent it to Erin
4. Typed up the "Change/Add Feature Description" and sent it to Erin.

Posted by Justin at 9:09 AM | Comments (0)

week 5 progress report

1. went over shopping list w/ matt and jonathon.
2. went over sample code for pilot
3. finished going through "code complete", starting on "psychology of programming."

Posted by John at 1:21 AM | Comments (0)

Week 5 Progress Report

1. I met with Sukanya and Justin on Thursday. We searched around for an open source program we could use for the experiment and decided on a program called VTSurvey. Sukanya has worked on this program before and is familiar with it. Also she verified that the program is well strutured and complex enough to challenge the subjects. We also decided on a feature addition which would be adding a drop down menu for answering questions.

2. Justin and I both went over the code to see the feasibility of adding the feature to the program. We've found that at least for a novice, this could prove to be a difficult task given the size of the program, but once the subject has a grasp of which files need to be modified then adding the new feature shouldn't be too difficult.

Posted by Teerawat at 12:02 AM | Comments (0)

May 8, 2005

Week 5 Progress Report

- Looked through aptitude test questions for possible problems
- Created 4 computer science questions. Emailed these to Erin. Posted also onto test and experiment page

Posted by Derrick at 9:23 PM | Comments (0)

CS Questionairre

clip_image002.jpg

Given that the starting vertex is node 56 (root), the breadth first traversal of the above sorted-binary tree is:
56 18 65 16 45 63 78 7 17 23 53 57 64 73 86

Question 1.

Give the pseudocode for the breadth-first traversal.

Question 2.

What is the running time (big-O) of the breadth-first traversal?

Question 3.

Given the following sentence:

“The fast dog jumped over the fence”

Give the pseudocode that will give the following result:

“fence the over jumped dog fast The”

The original sentence is given to you in a character array. Meaning “T” is in array position 0, “h” is in array position 1, “e” is in array position 2, “ “(space) is in array position 2, etc etc.
Your result should be returned in a character array.

Your algorithm should be as efficient as possible.

Question 4.

What are the stages of the Waterfall model of software development?

Question 5.

Match the following programming languages to their respective type.

C++ Functional

Pascal Object Oriented

Perl Logic

Lisp Scripting

VHDL Procedural

Prolog Hardware Descriptive

General questions

How much web development experience do you have?

Was this experience for personal or professional reasons?

If any, what web development software have you used?


Posted by Derrick at 9:18 PM | Comments (0)

Week 5 Progress

- Posted summaries for previous two articles

- Compiled shopping list with Matt & John

- Went over code for pilot

Posted by Jonathan at 8:30 PM | Comments (0)

“Communication Problems in Requirements Engineering: A Field Study”

This field study focuses on the communication characteristics of the requirements phase of a software project. The purpose of the study was to show that these projects have multiple problems in the communication between stakeholders and software engineers. In this case, the stakeholders can be seen as novice users while the requirements analysts can be seen as experts in their field. The three major communication barriers discussed deal with the “ineffectiveness of the current communication channels, restrictions on expressiveness imposed by notations, and the social and organizational barriers.” The study expresses that the one way communication channel, or “over-the-wall” method currently used in offices is hurting the flow of communication. Specification documentation is not adequate enough to communicate between phases of the software project. In addition, when asked if these documents were understandable by end users, a significant amount of experts expressed that their clients would need additional explanations in order to understand the notations that were given to them. Finally, organizational barriers arise when processes are divided into different phases and are performed by different groups. Managers chose these groups on a number of factors, rather than choosing them based on domain knowledge which is more ideal. Requirements committees are often composed of people with authority whom have limited knowledge of low-level tasks which require to be computerized. As a result of the study, it was determined that organizational and social issues posed the most important influence on communication problems in software engineering. It stresses that software professionals need to accommodate and identify these three different categories in their project management to determine the overall success or failure of the software project.

Posted by Jonathan at 7:49 PM | Comments (0)

"Is Software Engineering Training Enough for Software Engineers?"

The author of this article feels that the current curricula for software engineering is insufficient and inadequate when it comes to training students for “real world” software engineering projects. He argues that software engineering education should include the understanding and using classical engineering which is comprised of other engineering disciples such as mechanical engineering, electrical engineering, and different aspects of civil engineering. He proposes to update the current curriculum with adding elements such as analysis and modeling of non-software system related problems. As a case study, a software engineering course was analyzed to determine the importance of “system thinking”. The project purpose was to investigate the student’s ability to solve unfamiliar problems which are found in the profession but somewhat outside of the software engineering domain. They were given two requirements that hold emphasis in “real world” software engineering: modifiability and visualizing the mathematical model. The students understood that the software should be easy to modify but it was hard for them to incorporate this within their requirements and design. They were also able to derive a model for the project(some which did not work out at all) but had trouble with its documentation and analysis. Results determined that the students were not well prepared for engineering projects outside of the software spectrum. Even the teachers and their assistance were questioned on their knowledge of preparing students for what they will face in the future.

Posted by Jonathan at 7:13 PM | Comments (0)

May 6, 2005

Shopping Cart

Here's a list of items Johnathan, John and I were able to come up with for the lab experiment. Comments welcome.

Projected Costs: ~ $1,400

Posted by Matthew at 3:05 PM | Comments (0)

May 2, 2005

Progress Report - Week 4

- Researched field study papers:

Is Software Engineering Training Enough For Software Engineers?

Communication Problems in Requirements Engineering: A Field Study

- Derived questions for panel of experts

- Found software/hardware possibilities for experiment

Posted by Jonathan at 1:29 PM | Comments (0)

Peopleware by Tom Demarco & Timothy Lister

- The major problems of our work are not so much technological as sociological in nature. ( pg. 4 )
- The main reason we tend to focus on the technical rather than the human side of work is not because it’s more crucial, but because it’s easier to do. ( pg. 5 )
- Bad estimates, hopelessly tight estimates, sap the builders’ energy. ( pg. 28 )
- Coding War Games ( pg. 44 )
- A pair of programmers from the same organization didn’t work together, each person given the same work, designing, coding, and testing a medium-sized program to our fixed specifications.
- Benefit to individual – learning how he or she compares with the rest of the competitors.
- Benefit to company – learning how well it does against other companies in the sample
- Benefit to us – learning a lot about what factors affect productivity
- Results:
- Best people outperforming the worst by about 10:1
- Best performer being about 2.5 times better than the median
- Half that are better-than-median performers outdoing the other half by more than 2:1.
- Years of experience was a productivity non-factor. People who had ten years of experience did not outperform those with two years of experience. Maybe it doesn’t matter so much for our experiment to consider experts people who’ve had 5 years of experience?
- It mattered a lot who your pairmate was. Even though the pairs didn’t work together, the two members of the pair came from the same organization. They worked in the same physical environment and shared the same corporate culture. Best performers are clustering in some organizations while the worst performers are clustering in others. We should keep this in mind when looking for expert candidates.
- Flow – a condition of deep, nearly meditative involvement. Unfortunately, you can’t turn on flow like a switch. It takes about 15 minutes or more of concentration before the state is locked in. During this immersion period, you are particularly sensitive to noise and interruption. Not really doing work during immersion. We might want to rethink having the subjects think aloud. ( pg. 63 )
- Jelled teams - a group of people so strongly knit that the whole is greater than the sum of the parts. once a team begins to jell, the probability of success goes up dramatically.( pg. 123 )

Posted by John at 7:15 AM | Comments (0)

Week 4 Progress Report

1. discussed with jonathan and matt about the hardware and software needed for the setup and researched what software to use to get everything in-sync.
2. Read through "Peopleware" and "Code Complete." Will attempt to read through "Rapid Development" if time permitts.

Posted by John at 6:31 AM | Comments (0)

May 1, 2005

Experimental Task

Here's an initial design that Sukanya, Teerawat, and I thought of:

Experimental Task

Posted by Justin at 11:23 PM | Comments (0)

Week 4 Progress Report

1. Posted 2 summaries of psych papers on expertise.
2. Posted our initial design of the experiment

Posted by Justin at 11:14 PM | Comments (0)

Expertise in Professional Software Design: A Process Study by Sabine Sonnentag

In this study, the author wanted to examine expertise in software designers. He argues expert programmers should exhibit the following characteristics:

- Problem comprehension - create an adequate representation of the program
- Planning - decisions about future steps and their sequence are made
- Feedback processing - actively searching for feedback
- Task focus - spend less time on task-irrelevant cognitions
- Use of visualizations - use of sketches and diagrams
- Knowledge of strategies

To prove whether these characteristics actually appear in experts, he carried out the following experiment:

Method: 40 professional software designers participated in the study. The first task was to differentiate the high and moderate performers from the subjects. This was done through peer-nomination. The subjects had to design a software system that controls the movement of a lift system. The subjects were asked to think aloud.
Results: High performers spent more time on feedback processing and less time on task-irrelevant cognitions. They produced more visualizations and knew more about strategies. Across all phases, high performers spent less time analyzing requirements, suggesting they were able to build program representation more quickly. High performers did not show more planning ahead than moderate performers. Length of experience did not explain performance differences with the subjects in this particular study.

This study is relevant because it examines the attributes of an expert software designer. It provided a hypothesis for each of these attributes and tried to prove whether they were actually true in practice. Also, I thought it was interesting how they described expertise not merely but length of experience, but more importantly high performance.

Posted by Justin at 10:30 PM | Comments (0)

The Effects of Semantic Complexity on Expert and Novice Computer Program Recall and Comprehension by Bernard Guerin and Anthony Matthews

In this paper, the authors wanted to explore the effects of semantic complexity on both recall and comprehension while comparing novices and experts programmers. To do so, they conducted three experiments.

Experiment 1
Purpose: If experts use higher level semantic knowledge to help them understand programs, then removing some of the semantic structure should to lead to performance closer to that of novices
Method:
- 104 subjects (52 experts with 4.7-5 years of programming experience and 52 novices with 0.5 years of experience).
- Subjects received a booklet with a two-page computer program (116 line COBOL program) with 4 versions: the original version, a version with lines randomized within a module but keeping the module order intact, a version with only the modules randomized, and a version with both lines with a module randomized and module order randomized.
- The subjects were tested in groups of two to five.
- Subjects had to take 10 minutes to learn and memorize the program and then write down as many lines or program code that they could remember. Then they had to write down a brief summary of the purpose of the program and how it worked to achieve that purpose.
Results: The experiment provided strong evidence that experts use superior semantic knowledge when memorizing a computer program. However, when line order was randomized, they performed at the same level as a novice.

Experiment 2
Purpose: To see how two versions of a program (semantically complex and semantically simple) affect recall and comprehension.
Method:
- 52 subjects (26 experts and 26 novices)
- Each subject received 2 procedures written in Pascal for pattern-matching (one simple, one complex). In the first program, it is implemented using simple data structures. In the second program, it is implemented using more complex structures (linked lists).
- Subjects were tested in small groups.
- They were given 5 minutes to memorize the program, 5 minutes for recall, and 3 minutes to describe the function and operation of the program.
Results: The complex program was recalled less well than the simple version. Novices performed poorly on both versions. When the complex version was used, experts performed at the level of novices. Expertise was only significant for comprehension.

Experiment 3
Purpose: One reason that experts perform well in the previous experiments is because they already know the keywords so they can memorize and comprehend the program more quickly.
Method:
- To test this idea subjects were given a program to memorize. One half received the original COBOL program and the other half received a version with the keywords replaced by common English words
- 24 novices and 24 experts.
- 10 minutes to memorize, 8 minutes to recall, and 5 minutes to write a brief description of what the program was doing
Results: Removal of the keywords did not affect the experts more than it did the novices for either recall or comprehension. This means that the experts still have an advantage with their semantic knowledge about computer languages.

Overall, the experiments revealed that experts actively search for program functions. The experiments show that experts pay more attention to actual functions than novices do.

This paper relates to our study, because it provides more qualities of experts that we can look for. Moreover, the paper provides examples of how we can examine how our subjects think. It is important not only to monitor what type of code they produce, but also what their general understanding of the problem is. Also, the actual procedures of their experiments can be looked at as an example.

Posted by Justin at 10:26 PM | Comments (0)

Think-aloud books available online

There are two books available online that offer basic overviews about the think-aloud procedure. The first, Ericsson & Simon (1993), lays out the theoretical framework and procedures. It is the one most often cited in research articles (the online version appears to be read-only, as far as I can tell, so no saving or printing).


Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (revised edition). Cambridge, MA: Bradford Books/MIT Press.


Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modeling cognitive processes. London: Academic Press.

Posted by Erin at 10:13 PM | Comments (0)

Reconceptualizing think-aloud methodology: Refining the encoding and categorizing techniques via contextualized perspectives (Yang, 2003)

Yang, S. C. (2003). Reconceptualizing think-aloud methodology: Refining the encoding and categorizing techniques via contextualized perspectives. Computers in Human Behavior, 19, 95-115.


This paper summarizes several aspects of Ericsson & Simon's protocol analysis (a procedure to identify the mental processes that take place during problem-solving tasks). In particular, the author describes features of data analysis, and suggests straying from some of Ericsson and Simon's prescriptions.

Points from Ericsson & Simon (1984/1993):

1. There are three levels of verbalization in verbal protocols, and it is important that verbalization be kept at levels 1 and 2. Otherwise, the protocol analysis is subject to error.

2. Data analysis consists of transcribing, segmenting, and encoding verbal protocols.

3. Transcribing may be extended to include other data such as video recordings. Difficulties may arise depending on the clarity and fluency of subjects, technical quality of recordings, and ambient noise/distractions.

4. Once transcribed, data is broken down into discrete segments, which are large enough to contain all the information needed for making an encoding decision.

5. Encoding should be context-free, according to Ericsson & Simon (this is the point the author focuses on). In order to prevent bias, the researcher should select segments of data randomly, and categorize them one at a time, without regard for what came before or after. Thus, the researcher is less likely to hypothesize about what the subject was thinking, and just encodes (categorizes) what the subject actually said.


The author then describes a series of studies that examined learners' cognitive processing while using a Greek culture database. The researcher deliberately did not follow all of Ericsson and Simon's procedures for protocol analysis. In particular, examples are given of mental processes that might not have been revealed had the segments been encoded in random order, without paying attention to context.

The author feels that analysts should engage in "context-appreciative" (vs. "context-free") encoding, especially since it is becoming more and more difficult to adhere to Ericsson and Simon's prescriptions for developing discrete and mutually exclusive categories (the cognitive processes used by learners may be interrelated and complex). However, there is still the possibility of bias and problems of interpretation, and researchers should provide full accounts of their methods for collecting and encoding data, as well as their theoretical framework.

In evaluating this paper's recommendations, we should consider that the tasks used in the studies presented are complex (using a database to complete thematic assignments related to Classical Greek Studies, e.g., attitudes toward competition and its manifestations in Classical Greece), and undoubtedly produced especially rich verbal protocols. If our data will not be as complex, we may not need to worry about deviating from Ericsson and Simon's recommendations; however, the issue of context is well worth considering.

Posted by Erin at 10:03 PM | Comments (0)

Week 4 Progress

Posted by Matthew at 9:38 PM | Comments (0)

Thinking aloud: Reconciling theory and practice by Boren & Ramey (2000)

Boren, M. T., & Ramey, J. (2000, September). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261-278.

Online version:
http://ieeexplore.ieee.org/iel5/47/18778/00867942.pdf

This paper begins with a discussion of think-aloud protocols. They first describe the original theoretical framework put forth by Ericsson & Simon (1984/1993), and then bring it into the context of research in usability testing. Field observations of usability testing showed that researchers do not appear to adhere to standards with regard to think-aloud protocols (the description of the variability is pretty interesting). Methods vary widely, and some are carried out without any explanation or basis in theory (i.e., no background given or sources cited in the literature). This paper is particularly useful because it summarizes some of the key points from Ericsson & Simon’s theory, and then offers an alternative theoretical framework (speech communication). One thing for us to consider is whether the concepts that this alternative framework is based upon are more pertinent to the realm of usability testing than they would be for us (i.e., should we consider departing from the original think-aloud framework).


According to Ericsson & Simon's (1984/1993) model, there are three types of verbalizations:

1. Level 1 verbalizations are those that need not be transformed before being verbalized during task performance, e.g., verbalizing a sequence of numbers while solving a math problem.

2. Level 2 verbalizations are those that must be transformed before being verbalized during task performance, e.g., images or abstract concepts, as long as this transformation into words is the only mediating cognitive process between short-term memory and verbalization.

3. Level 3 verbalizations require additional cognitive processing beyond what is required for task performance or verbalization; e.g., filtering (verbalizing only info that is related to a specific topic), making inferences about one’s own cognition, and retrieving information from long-term memory at the researcher’s request.

--Level 3 verbalizations are not considered reliable data under Ericsson & Simon's theory.
--Any outside influence during a task (e.g., a comment or prompt from a researcher) can turn verbalizations into Level 3 data, by altering the normal flow of information into short-term memory during the task.
--Verbalization has been critiqued by Nisbett & Wilson (1977), who claim that people cannot accurately report on their own mental processes. Ericsson & Simon, however, argued that this critique is directed at specific types of verbalization (Level 3), and that Level 1 and 2 verbalizations would constitute reliable data.
--If Level 1 or 2 verbalizations are collected properly, they can reveal what information the subject heeded and in what order.
--In their model, verbalizations are not to be valued for their subjective content (this aspect is addressed in another article, "Reconceptualizing think aloud methodology..." [Yang, 2003]).
--Additional verbalizations like feelings and value judgments would not be considered data.


Additional key points from Ericsson & Simon (as described in the current paper):

1. Collect and analyze only "hard" verbal data
--No inference, introspection, or opinion; only what would be present in short-term memory during a task, i.e., what the subject attends to and in what order

2. Give detailed initial instructions for thinking aloud
--Make the distinction between thinking aloud and explaining
--Encourage the participant to speak constantly as if he were alone, and warn him in advance that he will be given reminders if he falls silent
--Practice thinking aloud before the actual task!

3. Remind participants to think aloud
--After a predetermined period of time (e.g., 15-60 seconds)
--Short and nondirective (e.g., "keep talking")—do not make the subject more aware of the researcher’s presence or create a sense of personal contact

4. Otherwise, do not intervene


An alternative theoretical framework: Speech communication

1. Assumes that all verbal processes are meant for the purpose of communication, and the idea that the participant would suspend his awareness of the listener (so he may think aloud as if he were alone in the room) is flawed.

2. Speakers expect that listeners will react to what they say.

3. If this is true, and a communication framework is inevitable, then the goal becomes to create a highly asymmetrical speaker/listener relationship (where the speaker does almost all of the talking), rather than trying to have the researcher become essentially invisible.

--Roles: The participant should be cast as an important contributor (i.e., expert, and the researcher as the interested learner and listener). This should be defined at the start and maintained throughout.
--Make observational technology as unobtrusive as possible (decrease awareness of being observed).
--Use "acknowledgment tokens" continuously (e.g., "mm hmm" or "uh huh," which are both conversation "continuers," but remain uncommitted and carry no connotation). These provide the response of an "engaged listener" but still allow the researcher to keep a low profile. Tokens should carry little or no content and should not redirect the subject’s attention (and should require little or no processing on the subject’s part).

Posted by Erin at 8:55 PM | Comments (0)

Progress Report for Week 4

1. Discussed the possible tasks for the subjects with Justin and Sukanya. We've come up with a list of possible tasks, expections, and ways to measure our results.

2. Wrote a few summaries on the articles I have from ACM. I'll write some more summaries on the remaining articles later in the week after I finish my paper.

Posted by Teerawat at 6:44 PM | Comments (0)

"A Glimpse of Expert Programmers’ Mental Imagery" by Marian Petre and Alan F. Blackwell

This paper attempts to find correlations between different experts' mental imagery of a design. The experiment consisted of ten subjects all of which have 10+ years of experience. They were each asked to design a solution to one of four problems present, but they did not have to implement the solution. The experiment also had an interviewer present who was questioning the subject as to what they see, hear, or what they are imagining. The questions borke down the images into 3 categories: verbal imagery, visual imagery, and auditory imagery.

They found that there were several similarities between the different subjects' mental imagery. The similarities are as follows: each entity in the imagery had a label, each subject had multiple images, each image was dynamic yet could still be controlled such as pausing the image or returning it to its previous state, and the part of the image the subject had his/her attention on would be in focus while everything else was out of focus. Given these similarities the experimenters concluded that there is commonality among mental imagery between experts "suggesting that there are accessible
lessons to be elicited about how programmers construct solutions."

This is important to our study because it gives us another characteristic of expert programmers that we can probe during the experiment. It's also interesting to see that mental imagery is actually used to design a part of the program other than the GUI. The concept is interesting but the paper itself is pretty dry.

Posted by Teerawat at 6:03 PM | Comments (0)

"Expert Problem Solving Strategies For Program Comprehension" by Jurgen Koenemann and Scott P. Robertson

This paper studies the problem stategies that expert programmers use to solve a Pascal problem. The study consisted of 12 subjects, all of whom had 4 to 15 years of experience, had 4 years of Pascal experience, and have worked on a large project. Each subject was given a program and a list of 4 modification task descriptions for the program and they would have to describe how they would accomplish each task. Then they were randomly given one of the 4 tasks to complete and were asked to think aloud as well as verbalize and explain any requests for information.

Through this experiment they discovered that none of the subjects ever studied more than half of the code for the modification phase. The experimenters referred to this phenomenon as relevance strategy wherein subjects only study code or documents that they feel are relevant for the task. Furthermore some procedures were ignored by all subjects such as min, max, and isdigit. The experimenters concluded that the subjects spent most of their time searching for code segments that were relevant to their task and ignored parts of the program that either they were familiar with or had no relevance.

This is important for our studying because when we assign our experts a task we should observe not just what the expert does, but what he doesn't do to understand the problem presented to him. Until reading this paper it didn't occur to me to record things he doesn't do like reading all of the program. Just another trait of experts to keep in mind.

Posted by Teerawat at 5:30 PM | Comments (0)

Week 4

1. Posted book/paper entry with links to papers read. (Recommended reading)
2. Thought of more questions for questionairre.
3. Thought of possible tasks the subjects could perform.

Posted by Derrick at 4:17 PM | Comments (0)

Programming Aptitude Tests: Reivews

These are a few good papers I found that are discuss the problems of programming aptitude tests. They bring up good points about how many factors are left out that these tests do not consider.

Paper 1 file

Paper 2 file

Posted by Derrick at 4:10 PM | Comments (0)

"PROTOCOL ANALYSIS OF A NOVICE PROGRAMMER" by Catherine Bishop-Clark

This paper examines a novice's problem solving ability while writing a basic program. For the purpose of their experiment, their novice subject was a MBA student who was taking an introductory course in programming. They asked the student to write a program that would calculate the a student's class average and they also asked the student to speak aloud while writing the program. The results of the experiment were separated into 4 subgroups: Problem Representation, Planning and Design, Coding, and Debugging. They found that like most novices, their subject spent very little time understanding the problem and coming up with a plan. The subject spent most of her time coding and debugging. During the coding and debugging the subject kept reminding herself of syntax rules such as placing a semicolon at the end of a statement.

What's interesting about this paper is that their experiment shows that novices do not place much emphasis on the design and understanding of the problem. We can look for this in our experiment by timing the amount of time it takes for our novices to understand the problems we present to them. Additionally, the experiment reemphasizes the fact that novices focus a lot of attention on syntax which has been stated by most of the other papers we read.

Posted by Teerawat at 4:00 PM | Comments (0)