Compliance
Training
Andragogy blended learning
ADDIE
Analysis Design Development
Implementation Evaluation
assessment development
learning experience Training development
The Challenge:
In Spring 2021, only a very small fraction (0.36%) of the crewmembers passed the Spring 2021 Skills Validation assessment on their first attempt, and it took others a maximum of 3 attempts to pass, with an average score of 87%.
The assessment was required for approximately 5000 crewmembers to maintain their qualification. The test needed to be revised and redistributed in the Spring 2022 distance learning cycle. Several factors needed to be considered in the updates:
The test contained questions with specific regulatory details that needed to remain intact
Several diagrams, layouts, and images were used throughout the test
The Approach
To ensures the training remained relevant, effective, and aligned with the learners' needs, the ADDIE model was used to carefully identify the issues and strategically plan, execute, and improve the solutions.
Analyze
Analysis of the performance data and learner feedback revealed the most significant issue was the confusing layouts and mislabeled diagrams. The answer options to each multiple choice question were also alphabetically labeled causing confusion between diagram and manual codes.
Design
The diagrams and layouts were redesigned to match those used in the most recent manuals.
A more intuitive layout was used to enhance the user experience, and the test questions were reviewed against the learning objectives for relevance and clarity.
Development
The test was revised, providing clarity and free of errors. Diagrams were redesigned for better comprehension, and all links and coding issues were resolved. The LMS platform was tested thoroughly to ensure stable connectivity.
Implementation
The revised test was implemented with a small group to identify any remaining issues. Training sessions were conducted for crewmembers to familiarize them with the new format. Any additional feedback from the pilot group was used to make final adjustments before the full rollout.
Evaluation
Post-implementation, the test results from Spring 2022 were evaluated. Continuous evaluation of learner feedback and LMS performance ensured that the test remained effective and any new issues were promptly addressed.
The Result:
After the revision, an overwhelming majority (98.62%) passed on their first attempt in Spring 2022, although it took up to 8 attempts for the few who did not pass initially. The average score also improved to 91%.
The revisions made to the test were highly effective in improving the pass rates and overall scores. The increase in first-attempt passes and the improved average score demonstrate that the revised test was more aligned with the crewmembers' knowledge and abilities. The increased maximum attempts needed by the few who didn't pass initially, indicate that while the test became more accessible, it maintained a level of challenge for those who required multiple attempts.
