June 14, 2005

Week 10 Progress Report

Forgot to do this yesterday.

1. Ran my experiment with my subjects and Justin helping me.

2. Helped Derrick run his experiment.

3. Spent Sunday, Monday, and today going the videos and analyzing them.

4. Came up with my hypotheses on Sunday and revised them in the meeting yesterday.

5. Started writing the bits and pieces that I could for my paper.

6. Graded the implementations according to the answer key Justin created.

Posted by Teerawat at 11:02 PM | Comments (0)

June 7, 2005

Week 9

- Attended experiment by Teerawat & Justin
- Found two expert software developers
- Scheduled Expert/Expert run with John for Friday 4pm
- Matt's assistant for tomorrow

Woohoo, I caught up.

Posted by Jonathan at 10:06 PM | Comments (0)

June 6, 2005

Progress Report Week 9

1. Found a novice and an expert subject for the experiment.

2. Set up an experiment time for Tuesday June 7th, from 3:30 pm to 7:30 pm.

3. Ran a test run of the full experiment with Justin, Derrick, Matt, John, Sukanya, and Erin on Saturday.

Posted by Teerawat at 12:14 PM | Comments (0)

June 5, 2005

Pilot Testing

Was test subject 1 in pilot study Saturday. Morae kept giving the strange "Unknown error has occurred" many times and we kept restarting until it finally worked. None of us knows what this descriptive error message means. Perhaps it's because we're using a larger monitor resolution than Morae recommends. Lowering the resolution only makes the screen blurry.

The only backup option I can think of is to capture with VNC and record with the DV camera we have.

Posted by Matthew at 1:58 PM | Comments (0)

June 1, 2005

Progress Report Week 8

-Updated the handbook (changed the text to a scenario-based script, modified the task to match the new task)
-Updated the Task Description to include the new task.

Posted by Teerawat at 10:53 AM | Comments (0)

May 31, 2005

Week 8 Progress Report

- Pilot test for new task with Justin
- Worked more on online survey.
- Updated protocol handbook
- Still need to create grading script, need to talk to Erin on how to score IQ and personality tests

Pre-experiment questionnaire

IQ test

Personality test

Figures for IQ test (given as a separate sheet to the subject)

CS Questionnaire figure (given as a separate sheet to the subject)

Posted by Derrick at 3:46 PM | Comments (0)

Week 8 Progress Report

- pilot tested novice -> expert interaction on Friday
- updated handbook (task script, added new appendix)

Posted by Justin at 8:58 AM | Comments (0)

May 25, 2005

Hardware Update

Jonathan and I were able to get part of the computer working the subject will be using. The webcam & Morae are working on the machine. We installed the Radeon 9700, only to remove it because the card only accepts DVI. We need a DVI->VGA adapter. They should be had for less than $20. Once we have it, the card should work. The firewire card is also installed.

We also tried running Eclipse through Remote Desktop. After testing we found it is much faster to run everything on one machine.

I'm leaving town Thursday and won't be available until Monday. John and Jonathan should be able to finish the rest of the equipment setup by the end of the week.

Posted by Matthew at 5:37 PM | Comments (0)

May 23, 2005

week 7 progress report

1. Wrote the Recruitment and Scheduling sections for the handbook.
2. Tested Morae with remote desktop with Jonathan and Matt.
3. Posted on Pyschology of Programming.

Posted by John at 3:10 PM | Comments (0)

Week 7 Progress Report

1. Wrote "Arrival" section for protocol handbook.
2. Talked with Erin and debated pros and cons of electronic questionnaire or paper questionnaire.
3. Decided to use VTSurvey to create surveys, IQ, and personality tests. Will use script to calculate scores.

Posted by Derrick at 9:38 AM | Comments (0)

May 22, 2005

Week 7 Progress Report

1. Wrote part of the handbook (Task A, Interaction, Task B)
2. Pilot tested with Yuzo (an expert) to determine the feasibility of the actual task. After looking at the code and the task he estimated that the task might take a "day" to finish.
3. Discussed alternatives to our existing task with Teerawat and Sukanya. We need to clarify more in person tomorrow.

Posted by Justin at 11:56 PM | Comments (0)

Progress Report Week 7

1. Wrote part of the handbook and combined my section with Justin's. I posted the combined version in my previous post.

Posted by Teerawat at 11:30 PM | Comments (0)

Week 7 Progress

Did 2nd run through (5/18) of video pilot and tried Remote Desktop Solution. One machine (Computer A) was using Morae. Another machine (Computer B) was used to access Computer A via Remote Desktop. Computer B would be the one the subjects would use. Result: unusably slow. John has video of it.

E-mailed script to Jonathan to check out errors and test out with using Morae.

Posted by Matthew at 7:25 PM | Comments (0)

May 20, 2005

Went shopping! Wheee

I ordered the following equipment today and it should arrive early next week. Total came to $771 with taxes and shipping, so if we end up purchasing Morae, we'll be on target with the budget. I forgot to order a microphone, but I should be able to find one that we can use. Also, I'll probably need to order a DVD burner, but that's less urgent.

Susan

Sony DCR-HC21 MiniDV Handycam Camcorder w/20x Optical Zoom
$370.49

Logitech QuickCam Pro 4000 (961239-0403)
$72.29

FlexStand - Flexible Web Camera Stand
$32.95

Sony DVM-60PR 60 Minutes Premium Mini DV Video Tape Cassette - 10 Pack
$32.99

Adaptec 1890600 Fireconnect 4300 Kit AFW-4300
$39.88

ATi All-in-Wonder RADEON 9700 PRO 128MB DDR AGP8X w/o Remote (OEM)
$151.00

Posted by ses at 2:57 PM | Comments (0)

May 18, 2005

Week 6 Progress

Video Pilot testing

1st run through - 5/18/05 - Concept

Johnathan, John, and I performed some pilot tests to test the usability of capturing data while a subject is performing assigned tasks. We attempted to perform tasks a subject might do such as opening applications, working on a project in Eclipse, instant messaging people, browsing the internet, and editing text files. We also tested out a firewire webcam to see if two users would be captured in a single frame. Tests were also conducted on the use of Morae itself and the playback of recorded video.

Installed Morae on 2 machines

Test case 1

Results

We noticed a slowdown when recording versus using only Eclipse. Since the slowdown occurred even without a webcam attached, we decided adding more programs will only take up more resources and slowdown the machine even more. The use of Windows Remote Desktop would only inhibit the machine further, also bringing about problems of video syncronization between what Morae records and the speed Remote Desktop would be able to capture from another machine. A more powerful video card should be used. On-board one not good enough.

Test case 2

Results

Eclipse's compiling speed was alright. Too a little bit longer to open program than normal. Scrolling was noticably slower. AIM had nothing problematic to report. Using internet Explorer for looking up JDK documentation was slow scrolling and laggy. TextPad had some slowdown in typing

Filesizes

Enoding WMV (takes 5 min) @ 640x480 Standard. File size: 18.8 MB

Lossless AVI (takes 3.5 min) @ 640x480 Standard. File size: 25.5 MB

Morae Project files are larger than AVI files. Test 1 was 6 MB & Test 2 was 33 MB.

Overall performance from test cases

Concerns the group has

Posted by Matthew at 3:20 PM | Comments (0)

week 6 progress report

1. created an gmail account for scheduling purposes: sims.ics199@gmail.com

2. read through "Psychology of Programming".

3. tested out Morae with Jonathan & Matt.

Posted by John at 2:03 PM | Comments (0)

Progress Report Week 6

1. Tried to implement the drop down menu, but it was more complicated than I thought. In the end I couldn't get it working.

2. A friend of ours helped out in the pilot test by performing the first part of the experiment and emailed us a CCB form. For the most part it met our expectations for a novice.

Posted by Teerawat at 12:06 AM | Comments (0)

May 17, 2005

Week 6 Progress Report

-Made updated changes to CS questionairre
-Performed pilot test (understanding VTSurvey and creating CPB Form)

Posted by Derrick at 10:06 PM | Comments (0)

Week 6 Progress Report

1. Tried to implement the actual solution.
2. Sent out the task to a couple of people (they returned the CCB form)
3. For the most part, I think the understanding of the task can be done, but the implemenation maybe a little more diffcult than originally expected.

Posted by Justin at 10:01 PM | Comments (0)

May 9, 2005

Week 5 Progress

Created shopping list with Jonathan & John (see previous post)

Posted by Matthew at 2:10 PM | Comments (0)

Week 5 Progress Report

1. Met with Teerawat and Sukanya and decided on using Virginia Tech's open-source survey creation application, VTSurvey, for our experiement.
2. Analyzed the code to verify how feasible it is as our experiment program. It seems like it is.
3. Re-typed the "Experimental Task" and sent it to Erin
4. Typed up the "Change/Add Feature Description" and sent it to Erin.

Posted by Justin at 9:09 AM | Comments (0)

week 5 progress report

1. went over shopping list w/ matt and jonathon.
2. went over sample code for pilot
3. finished going through "code complete", starting on "psychology of programming."

Posted by John at 1:21 AM | Comments (0)

Week 5 Progress Report

1. I met with Sukanya and Justin on Thursday. We searched around for an open source program we could use for the experiment and decided on a program called VTSurvey. Sukanya has worked on this program before and is familiar with it. Also she verified that the program is well strutured and complex enough to challenge the subjects. We also decided on a feature addition which would be adding a drop down menu for answering questions.

2. Justin and I both went over the code to see the feasibility of adding the feature to the program. We've found that at least for a novice, this could prove to be a difficult task given the size of the program, but once the subject has a grasp of which files need to be modified then adding the new feature shouldn't be too difficult.

Posted by Teerawat at 12:02 AM | Comments (0)

May 8, 2005

Week 5 Progress Report

- Looked through aptitude test questions for possible problems
- Created 4 computer science questions. Emailed these to Erin. Posted also onto test and experiment page

Posted by Derrick at 9:23 PM | Comments (0)

Week 5 Progress

- Posted summaries for previous two articles

- Compiled shopping list with Matt & John

- Went over code for pilot

Posted by Jonathan at 8:30 PM | Comments (0)

May 6, 2005

Shopping Cart

Here's a list of items Johnathan, John and I were able to come up with for the lab experiment. Comments welcome.

Projected Costs: ~ $1,400

Posted by Matthew at 3:05 PM | Comments (0)

May 2, 2005

Progress Report - Week 4

- Researched field study papers:

Is Software Engineering Training Enough For Software Engineers?

Communication Problems in Requirements Engineering: A Field Study

- Derived questions for panel of experts

- Found software/hardware possibilities for experiment

Posted by Jonathan at 1:29 PM | Comments (0)

Week 4 Progress Report

1. discussed with jonathan and matt about the hardware and software needed for the setup and researched what software to use to get everything in-sync.
2. Read through "Peopleware" and "Code Complete." Will attempt to read through "Rapid Development" if time permitts.

Posted by John at 6:31 AM | Comments (0)

May 1, 2005

Experimental Task

Here's an initial design that Sukanya, Teerawat, and I thought of:

Experimental Task

Posted by Justin at 11:23 PM | Comments (0)

Week 4 Progress Report

1. Posted 2 summaries of psych papers on expertise.
2. Posted our initial design of the experiment

Posted by Justin at 11:14 PM | Comments (0)

Week 4 Progress

Posted by Matthew at 9:38 PM | Comments (0)

Progress Report for Week 4

1. Discussed the possible tasks for the subjects with Justin and Sukanya. We've come up with a list of possible tasks, expections, and ways to measure our results.

2. Wrote a few summaries on the articles I have from ACM. I'll write some more summaries on the remaining articles later in the week after I finish my paper.

Posted by Teerawat at 6:44 PM | Comments (0)

Week 4

1. Posted book/paper entry with links to papers read. (Recommended reading)
2. Thought of more questions for questionairre.
3. Thought of possible tasks the subjects could perform.

Posted by Derrick at 4:17 PM | Comments (0)

April 26, 2005

Week 3 Progress

  1. Read the 4 papers
  2. Took the IRB tutorial
  3. Started doing questions for panel of experts

Posted by Matthew at 11:09 AM | Comments (0)

April 25, 2005

Progress Report Week 3

1. I finished writing the last 2 summaries.
2. Read 2 of the 4 papers I received last week. Might be able to squeeze in a one more before the meeting.
3. Read about 5 or 6 of the 14 papers I found through ACM.

Posted by Teerawat at 1:57 PM | Comments (0)

week 3 progress report

1. read 3 of the 4 papers, since one was mine.
2. researched "popular books", found a few. need to see if the library has them.
3. took the irb tutorial.

Posted by John at 3:01 AM | Comments (0)

April 24, 2005

Week 3 Progress

1. Read all the papers.
2. Took IRB Tutorial Test.
3. I found a couple of psych papers on expertise, but some of the findings weren't entirely relevant to our topic. Wrote up summaries for 1 psych paper concering expertise and 1 paper from the ESP.
4. I have a general idea for the "Expections" for the background of our study. (i.e. Experts will use "schemas" to solve problems, while novices will focus on more concrete information)

Posted by Justin at 11:32 PM | Comments (0)

Week 3 Progress

1. Read all 4 papers that were given to read.
2. Took tutorial "test" for IRB
3. Found 2 more papers regarding programming aptitude tests.
4. Thought about tasks on what to do for experiments and measurement of performance.

Posted by Derrick at 10:36 PM | Comments (0)

April 18, 2005

Week 2 Progress Report

1. I've read 3 of the 4 papers that everyone got and 2 of the 3 papers that only I have. I've posted only one the blogs so far but I'll get the second one tomorrow.

2. I haven't had much luck searching through PsychLit. I'm going to try searching more for cognitive abilities since Justin is searching for expertise. This way we'll probably be able to cover more material that could relate to the study.

3. IEEE hasn't really been giving me what I was looking for, but I stumbled upon some promising papers on ACM that revolve around other experiments between novices and experts. Unfortunately I couldn't view any of them because I need to purchase a membership, so I'll just access them on the school network tomorrow.

Posted by Teerawat at 12:41 AM | Comments (0)

April 17, 2005

Progress Report

1. Read all the papers
2. I had a hard time finding psychology literature concerning expertise in software engineering. Going to look for more concerning expertise in general.
3. I checked out 2 books on "Empirical Studies of Programmers" from the library. There was only one good article concering "expert/novice" comparisions.

Posted by Justin at 11:54 PM | Comments (0)

Status update

1. Read all but 2 papers that were given. Will finish by meeting.
2. Updated blog with info on unique papers that were given.
3. Found 2 sample aptitude test online.
4. Found a paper that researched the outcome of programming aptitude tests. Was not able to read it yet.

Posted by Derrick at 11:25 PM | Comments (0)

Week 2 Progress

  1. I read the four papers.
  2. Somehow I only got 2 articles, but was able to read them. Posted about one of them so far. I won't be able to finish the second one until Monday.
  3. I took a look at some panel of experts documentation on the web to see what was covered and what types of questions are asked.

Posted by Matthew at 11:05 PM | Comments (0)

Progress Report: Week 2

1. I read the four papers that each member received.
2. Started but not finished reading the three papers that only I received. I'll finish before our meeting tomorrow.
3. Found three books for the "popular books" background research assigned to me and my teammate.

Posted by John at 10:12 PM | Comments (0)

April 13, 2005

Welcome to 199 Students

We had our first meeting this week at 9:30am on Tuesday. We talked about the overall plan for the study and gave everyone homework.

Homework

There were three papers that everyone would read.

In addition, there was a stack of papers and books that was divided among the group. Some of these papers are my only copies, so please be gentle with them. For these papers, please post something on the blog about them under the category of "Books and Papers". Give the citation, a summary, your evaluation of its relevance to our project, and identify what you think might be relevant (a theory, a result, an experimental manipulation).

Also, the 199's were given some other background to research.

Justin and Teerawat: psychology and ESP/PPIG/SE
John and Derrick: programming aptitude tests and popular books
Jonathan and Matt: field study and panel of experts

Please post a progress report the day before our meeting under the
"Progress Report" category.

I've uploaded an EndNote library containing references to the various papers that we have.
Download EndNote Library
If you have a copy of EndNote, you can add to this as we go along.

Next meeting will be Monday at 4pm.

Susan

Posted by ses at 4:15 PM | Comments (0)