Interviewer Reliability Qualifications Reviews

Before the Interviewer Reliability Qualifications Reviews were introduced, existing phases of the Supports Intensity Scale (SIS) Train-the-Trainer program offered by the American Association on Intellectual and Developmental Disabilities (AAIDD) focused on a) training attendees to conduct face-to-face SIS interviews with individuals with an intellectual disability, and b) observing these trained SIS trainers teach others about the administration of SIS. Following the training, these AAIDD-trained trainers typically go on to independently train SIS interviewers within their own organization or state, and then establish the reliability of the interviewers, i.e., they ascertain that the scores obtained by SIS interviewers are accurate and reliable through a process that is devised internally. Now AAIDD offers trainers help with establishing this internal reliability process as well, based on several requests from attendees of past training workshops on getting more guidance during this stage of the SIS training process.

This feature of the SIS training program is designed to establish better consensus on use of the instrument by those who administer it, and ultimately benefit the service and funding outcomes for persons with intellectual disabilities.

So how does this new service offering work? Following the SIS train-the-trainer session, the AAIDD trainer observes the trained trainers working in pairs for demonstrated knowledge and competency in teaching the SIS orientation course to other interviewers. Next, the AAIDD trainer observes the trainer conduct the interviewer reliability qualifications review process with SIS interviewers to ensure that they have understood this procedure and that they score the interviewers correctly. While earlier, the trainer would conduct this process independently, now, an AAIDD senior trainer observes this process and follows up with in-depth critiques, a question and answer session, and a written evaluation of the process.

At the end of this training process, total raw scores on each of the 9 subscales on the SIS interview form obtained by the AAIDD trainer and the trainee are compared. In general, establishing inter-rater reliability with SIS interviewers means that the interviewer’s score should be equal to the AAIDD trainer’s score on at least 8 of the 9 subscales. Specifically, the trainee’s performance is evaluated by the AAIDD senior trainer in the areas of:

• Setting the scene: How well has the trained trainer created a supports-oriented mind set for the SIS respondents?
• Intent: Has the trainer clearly articulated the intent of each item? Has s/he provided a comprehensive description of the item?
• Decision making: Has the trainer demonstrated the ability to pull all the interview information together? Has s/he determined an appropriate rating?
• Agreement: The trained trainer is considered reliable at the sub-scale if 3 or fewer disagreements are noted.

As Michael Chapman, senior trainer of AAIDD says, “There are two ways we can help AAIDD-trained trainers evaluate the inter-rater reliability of their interviewers–we can teach them to use our own AAIDD forms and processes, or we can work with them to develop their own.” In the past, AAIDD-trained trainers have been creative in devising an inter-rater reliability protocol that suits individual styles of trainers. Chapman also adds that the process of establishing reliability becomes easier with practice.

The SIS assessment results are used to make important life decisions for people with intellectual disabilities. For the SIS assessment to be reliable, achieving inter-rater reliability in the administration process is critical. A recent study supports the finding that being properly trained in administering SIS plays a major role in obtaining reliable results from a SIS assessment. It is only right that administrators of SIS learn to use the instrument properly so that a person with an intellectual disability receives the supports and services s/he needs to lead a quality life.

For more information on SIS training, please contact AAIDD at books@aaidd.org or (202) 387-1968, Ext. 213. For testimonials on past training sessions, visit www.aaidd.org/sis/training.