November 2013 - An NCCCO Auditor Training Workshop held in Lincolnton, SC, earlier this month was the latest workshop held this year as part of NCCCO's mandatory annual training program for NCCCO Auditors.
The CCO Practical Exam Auditor Program is a key component of NCCCO's Quality Assurance Program designed to ensure that all certification examinations are delivered in a standardized manner that is fair to all candidates while remaining psychometrically valid and reliable in line with ANSI accreditation protocols.
Both this program and NCCCO’s Auditor selection process that works in conjunction with it, provide a systematic means of ensuring that CCO practical tests (now approaching 300,000 administered since testing began in 1999) are delivered in a consistent and fair manner, wherever and whenever they are administered.
During an audit, an Auditor checks all elements required for the Practical Exam, including the test site facilities and test course preparation. The Auditor also ensures that candidates are briefed with the required materials and that the test course meets the precise requirements of the test site layout (CAD) issued for that test.
Auditors check measurements at an NCCCO Practical Test Site. Standardization of test site layout is key to test integrity.
To verify that the Practical Examiner is conducting tests in conformance with NCCCO testing protocols, his (or her) performance is compared with a “shadow exam” conducted by the Auditor at the same time that the candidate is examined by the Practical Examiner. Any discrepancies between the Auditor’s shadow exam score and the candidate’s score sheet are noted. Auditors are authorized to stop tests and provide corrective action if they believe that a test’s integrity is being compromised.
Auditors are carefully selected from among NCCCO’s Practical Examiners in good standing based on such factors as the quality of their work and the professionalism they bring to test administrations. Those that qualify then receive thorough training in auditing techniques including examiner reliability calibration, shadow exam practicing, recognition techniques, inter-examiner reliability, and all pertinent NCCCO policies regarding the Practical Exam process. Auditors receive comprehensive practical, hands-on training on an approved crane and course layout at an official NCCCO test site.
High among Auditors' responsibilities is ensuring the practical test site course has been set up precisely to specifications.
Assignment of Auditors to test sites and/or examiners selected for audits is based on geographic region and alphabetical listing. Most audits are of two types: Announced, where the Examiner and the Test Site Coordinator are given prior notice of the intent to audit; or Unannounced, where no prior notice is given. However, NCCCO also conducts “secret shopper” audits where both the audit and the auditor are unannounced and where the examiner is unaware an audit is taking place.
Examiners and/or practical test sites are selected for audit based on established criteria, including:
- Unusually high or low pass rates as revealed by statistical analysis
- Anomalies, irregularities, or repeated nonconformities in the documentation submitted by examiners
- Complaints received from candidates, test site coordinators, or other examiners
- As a component to another investigation that may impact the site or examiner
- Random selection
In addition, all Practical Examiners in their first year of accreditation are considered “probationary” and are subject to audits without prior notice.
Completed audits are reviewed by NCCCO staff. Irregularities recorded during the audit are evaluated to determine whether any rise to the level where action should be taken. Examiners may be placed on suspension, or have their accreditation status revoked. While the quality of the NCCCO test process means, in practice, such action is rare, NCCCO’s due process also provides for appeals by sanctioned examiners to the duly appointed body within NCCCO’s committee structure. Analyses of audits conducted are submitted to the Practical Exam Management Committee for review on a semi-annual basis, and to the Board of Directors annually.
In order to verify that Practical Examiners are consistently applying the same scoring criteria across many exams (essential if the test is to be fair to each candidate), studies are conducted that compare Examiners’ scoring with that of specially trained Auditors. These so-called “inter-rater reliability studies” should show a high degree of correlation between the Practical Examiner and Auditor score sheets, as in this illustration.
To date, NCCCO’s team of more than two dozen trained and experienced Auditors have conducted several hundred audits at practical test sites across the country. All of them are aimed at providing effective and transparent oversight of NCCCO’s practical exam process to ensure that all candidates are examined in a consistently fair, valid, and reliable manner.