Software Engineering Expertise

Study Recruitment

Current Projects

  • Web Application Development Expertise

    • Expertise is the consistently superior performance on a set of tasks in some area of human activity. Software engineering expertise is difficult to define, and characterize empirically. In this project, we are evaluating three candidate criteria for assessing software engineering expertise. These three criteria are: experience; characteristics common to experts across fields; and software-specific proficiencies. We conducted an initial empirical study to evaluate these three definitions. We are observing expert and novice subjects in a laboratory experiment in which they are asked to perform software engineering tasks on a web application. Our initial findings indciate that all three criteria should be used to provide a full categorization of an individual’s level of expertise. Number of years of experience is a useful first filter, but cannot be used as the only criterion. Domain-independent (i.e. cognitive) characteristics are most useful for assessing the quality of the criteria for labeling an individual as novice or expert. Software-specific proficiencies appear to be the most promising, but assessments for various skills, technologies, and problem domains will need to be developed separately.

    We are conducting further experiments to refine a proficiency-based definition of expertise. The tasks in these experiments will be focused on web applications and will involve a larger number of subjects. We will be constructing a ladder of skills and knowledge areas, where the rungs on the ladder represent different levels of expertise. For more information about work-in-progress, see our project blog and wiki.

  • Requirements Engineering Expertise
    • The project aims to characterize requirements engineering expertise, by conducting interviews with experts and elites into the important skills for requirements engineering and project management.

Background

There are important differences in the software engineering prowess of undergraduate students and professional software developers. Industry is keenly aware of these differences, as job advertisements typically require a certain number of years of experience. What are these differences? What characteristics of experts make them incommensurate with novices? What can their insights tell us about the nature of software development?

There have been many studies of expert vs. novice performance in psychology. The effect of expertise on performance has been examined in a number of domains, most notably in physics, chess, and mathematics. Techniques, measures, and frameworks have been developed for characterizing differences between novices and experts. One finding that is pervasive across the domains is that experts perceive problems differently than novices. In other words, they attend to the "deep structure" of the problem rather than surface features. Figure 1, below, provides a visual illustration of this difference comes from chess playing. Here, eye movements were used as indicators of the heuristics used to plan the next move. The novice’s eye movements focused on the queen, while the master’s eye movements covered the board more broadly, suggesting greater attention to overall board position.

Eye movements of chess master over example board Image of example chess board Eye movements of chess novice over example board

While the same visual indicators may not be present in software development, similar differences likely exist between novices and experts in this domain as well. For instance, experts likely attend to different features of a problem, identify its deep structure, and use different strategies. These differences likely exist in both design and implementation of software. The studies from psychology will serve as a starting point for our investigations into differences between novice and expert software developers.

We can learn a great deal by studying expert software engineers. Through them, we can access many years of experience, thereby improving our understanding of software engineering itself. Being able to characterize, and even quantify, expertise is beneficial for human resource management, software engineering education, and even empirical evaluation of software tools and methods. This knowledge can be used in human resources to more effectively recruit and select team members. By identifying differences between novices and experts, we can teach students strategies and approaches so they can solve problems in the same manner as experts. Finally, it can be used empirical evaluations of tools and methods as a base for developing metrics to measure performance of software engineers. In other words, a software technology can be considered helpful if it assists a user to work more proficiently, that is, to behave more like an expert software engineer.