UHCL Cybercorps Project Evaluation Plan
The program evaluation will employ Extended-Term Mixed-Method Evaluation (ETMM) Design, which includes a long-term time-line;
an evaluation guided by project’s purposes; a deliberate incorporation of short-term, formative, and summative data collection and analysis; and rigorous quantitative and qualitative evidence incorporating formative data into processes of project refinement. Both formative and summative measures are outlined as the Logic Model.
Logical Model for Evaluation
| Performance Measures | Short-Term/Formative Evaluation | Summative Evaluation | Data Analysis |
|---|---|---|---|
| 1. Participating in the Cyber Defense Track will increase cybersecurity awareness. |
Pre-Survey Observations |
Post-Survey Focus Groups Interviews |
Descriptive Statistics Qualitative Coding |
| 2. Participating in the Cyber Defense Track will increase knowledge of cybersecurity, network security, network forensics, and database security. |
Pre-Assessment | Post-Assessment Focus Groups Interviews |
Descriptive Statistics Paired t-test Qualitative Coding |
| 3. Participating in the Cyber Defense Track will increase student interest to pursue a career in cybersecurity. | Pre-Survey | Post-Survey Focus Groups |
Descriptive Statistics Qualitative Coding |
| 4. Participating in the hands-on labs embedded within the course curriculum will increase student engagement and interest in the Cyber Defense Track. | Pre-Survey Observations |
Post-Survey Focus Groups |
Descriptive Statistics Qualitative Coding |
| 5. Were recommendations from Year 1’s formative evaluation results implemented appropriately with Year 2 and 3? | Observations |
Qualitative Coding |
Assessment
To assess student knowledge and application of cybersecurity, network security, network forensics, and database security, a researcher-constructed pre/post assessment, aligned to the objectives of the instructed curriculum, will be developed for each course.
Each assessment will be validated in 2018 by a panel of experts in cybersecurity and CS education prior to administration.
Short-Term/Formative Evaluation
The approach to short-term and formative evaluation will be to balance breadth and depth of information. Prior to exposure to the CDT curriculum (modules/coursework), all participants will complete a pre-survey to assess interest in cybersecurity-related coursework/careers, cybersecurity awareness, and attitudes toward active learning. During Year 2 and 3, a pre-assessment will be administered to all undergraduate students at the beginning of each piloted CDT course to
measure student knowledge and application of cybersecurity, network security, network forensics, and database security prior to instruction.
Summative Evaluation
All participants will complete a post-survey at the completion of each tutorial workshop, summer camp, and CDT course. Undergraduate students will be administered a post-assessment at the end of each CDT course. Comparisons between pre- and post-assessment will point to the nature of changes in perceptions and knowledge. At the completion of each workshop, summer camp, and CDT course, focus groups will be conducted with a minimum of 6-8 participants, last 40-60 minutes, and allow extra time if needed for probing and more in-depth responses. In addition, interviews will also be conducted with instructors at the completion of the CDT modules/courses, utilize a semi-structured interview protocol, and last 45-60 minutes.
Focus group and interview data will allow for a more thorough and in-depth exploration of the survey and assessment results.