The science behind C'reer: A talk with Dr. Jane Swanson

We completed development of the career assessment that’s in C’reer a few months ago. In fact, it was probably the biggest undertaking in the first year of the company – to develop and prove a vocational assessment could be mobile friendly and scientifically accurate. With the app now in the market, we finally got some time to sit down with one of our lead researchers on the project – Dr. Jane Swanson – to reflect on the process and methodology that went into developing our assessment.

C’reer: What were your initial thoughts when we proposed the idea of developing a vocational assessment specifically for mobile?

JS: The underlying idea was intriguing, for a couple of reasons. First, there were the technological considerations. Other than the use of high-speed computers to expedite data processing, little has changed in measuring interests over the past 100 years. We had to consider how that longstanding approach would work in a mobile environment. Second, access to traditional career assessments is limited due to their control by mostly educational institutions, expense, and dependence on trained professionals to interpret the results. An assessment on a mobile platform changes that dynamic and expands the range of individuals that can take advantage of career assistance.  

C’reer: You’ve worked on the Strong Interest Inventory in the past — how was the process in developing the assessment in C’reer different or similar?

JS: The work on the Strong Interest Inventory focused on gathering data from individuals in occupational criterion groups – the samples that are used to build scales measuring interest and similarity to people already in occupations. For C’reer, we started at the beginning, in terms of choosing, developing and evaluating items to use in the assessment. We also were working on a much shorter time interval for C’reer, from initial concept to viable product and targeting pre-career individuals as target and test groups.

C’reer: Were there any surprises or a-ha moments in this for you?

JS: We knew going into the design process that we could not utilize the large set of items that are typically found in normal career inventories. The C’reer assessment needed to be engaging, relatively quick to complete, and provide a smooth user experience. That meant showing one question at a time and balancing the right size rigor with the requirements of a mobile application. User feedback thus far indicates we’ve successfully achieved that.

C’reer: What do you say to the person who gets an assessment recommendation and they say, ‘that’s not me’?

JS: I’d want to know more about the person’s experience and in what context they took the assessment. What type of exposure have they had to career information or knowledge about the variety of careers available? In other words, is the assessment based on some real-world knowledge about the world of work? If the person does not have much hands-on experience, how might they go about obtaining this experience to augment or pursue additional information?  Also, distraction can be factor when evaluating yourself for career potential – individuals should take care to respond quickly, going with their gut response, but not carelessly. 

C’reer: What do you think the biggest surprise will be for the person taking our assessment inside C’reer?

JS: If they are used to traditional 100+ text based assessments, they will be surprised by how quickly they can engage with the questions and obtain results that can help them better understand their own career interests. They may have the same quick and relatively fun experience of taking a Buzzfeed quiz, but the outcome will be something far more useful and based on science!

C’reer: Was the process in testing the various stages of the C’reer assessment different because it was intended solely for mobile?

JS: Despite the mobile application, the development was still rooted in tried-and-true scientific methods. The same basic processes were followed in each stage of developing the assessment, including basing it on well-established theoretical constructs, writing high quality items using content experts, multiple pilot testing, choosing appropriate samples, refining items, and estimating reliability.

profile_iphone7gold_portrait.png

C’reer: Did you have to rethink how you asked specific questions due to space constraints?

JS: Yes, but that was the nature of the challenge. The items themselves needed to be short and to the point, and the number of items needed to be reduced, all to conform to the mobile platform. We also had to identify outdated questions in some of the older inventories and eliminate them.

C’reer: What do you think the biggest challenges our assessment will get from educators and your fellow researchers?

JS: The brevity of the assessment feature itself. Traditional psychometric methods build scales using considerable more items, which is a standard tenet of reliability – all things being equal, longer scales are more reliable. And reliability helps establish validity. In other words, larger assessments can be viewed as typically being more reliable. We put C’reer through multiple pilot tests and are confident that the length of the assessment in the C’reer app strikes the right balance between usability in a mobile platform and reliability. 

How do you think we can make the C’reer assessment better or more predictive?

JS: Any assessment can be improved with additional analysis of data from new samples, which we have committed to doing over time with C’reer as more data are collected. Making it more predictive can best be accomplished by collecting data that links individuals’ assessment results to important and valued criteria such as satisfaction with the choices presented, and clarifying the purpose of the assessment

Vireo Labs hired the entire Alpha Omega Associates team of researchers to develop the vocational assessment that is running inside of C’reer. The team included:

  • Jane L. Swanson, Ph.D., Professor Emeritus
  • Joel T. Nadler, Ph.D., Associate Professor
  • Meghan R. Lowery, Ph.D., Organizational Consultant