Shrock and Coscarelli

Testing Human Competence

Turn-key Testing

We have worked with a wide range of organizations who have found the value of assessing and certifying human competence.  Some organizations have been as small as a regional bank wanting to provide a series of incentives for tellers to global corporations that were seeking to create tests as part of a hire/fire decision.  Obviously hire/fire decisions represent high stakes assessments that will place more rigorous demands on the test developer than career development path.  However, the processes we use for either organization are quite similar in concept and represent a package that provides turn-key test development expertise for the organization. 

A complete certification test is easily doable in one quarter of the year if properly planned in advance.  The total cost to the organization is usually under $30,000 including expenses. Generally speaking there are four major milestones in the criterion-reference test process: 

Here are the four milestones…

Stakeholder Meeting (One Day)
Any successful testing effort will require management buy-in and the formation of a team to develop and manage the testing process.  We have found it best to bring everyone together to confirm the test’s purpose, check assumptions about testing and the test development process, and identify the people and timelines that organization wants to work with.

CRTD Workshop (Two Days)
The most fundamental element of criterion-referenced test development is to make the test look like the job.  While everyone agrees with the idea, we have found that creating the tests that look like the job is much harder to do than we first imagined.  People tend to write tests that look like the tests they took in school; as a result most first drafts of tests are memory level items that don’t match the job.  Teaching subject-matter experts to raise the level of their tests from memory to application takes time and practice and feedback.  We have found no substitute for having the test developers all together for training in both item writing and the test development process.

Off Site Support
Successful test development projects always have one person who understands testing and the corporate culture.  After the two-day workshop the items can be created and reviewed in light of the complete test development model.  This usually means we interact electronically to help support the process.

Standard Setting (One-Half to One Day)
Deciding who is a master and who is not is a fundamental outcome of the criterion-referenced testing process.  Arbitrary standards (“Passing is 80% because that is how we always do it.”) are not defensible professionally, ethically, or legally. 

There are three major classes of techniques for setting a defensible and meaningful cut-score.  Of these three techniques most organizations choose to use the Angoff Technique because of its reliance on subject-matter experts who are already engaged in the item writing process and are thus also readily available.  We have found there is a great deal of initial confusion about the process.  However, by placing the SMEs together physically and providing clear direction, practice, and feedback, we can usually bring the group up to speed by lunch.  Once a shared understanding is established about the Angoff Technique, additional scoring can be done via electronic meeting tools.