12:00–1pm ET Friday, Jan. 31, 2025, online 

We evaluated the performance of five LLMs (Llama a, GPT-3.5-turbo, GPT-4, GPT-4O, and GPT-O1) on two cybersecurity concept inventories: Cybersecurity Concept Inventory (CCI) and Cybersecurity Curriculum Assessment (CCA). Using a zero-shot setting to minimize external influencing factors, we compared the performance of these LLMs with that of students previously studied, and we conducted a qualitative analysis of GPT-O1's output to examine if it exhibits misconceptions. Quantitative analysis reveals that, for the CCI and CCA, GPT-O1 significantly outperformed other models and students, correctly answering 92% of CCI and 72% of CCA test items. These results indicate GPT-O1’s strong proficiency in foundational topics (CCI) but reveal its limitations in addressing these concepts in more technically advanced scenarios (CCA). Qualitative analysis of GPT-O1’s reasoning patterns uncovered instances of insightful reasoning but also highlighted ways in which GPT-O1's answers reflect persistent student mistakes, such as biases, overgeneralizations, and logical inconsistencies. This work highlights the significant potential of GPT-O1 as a tool for introductory cybersecurity education in its ability to provide detailed explanations and structured reasoning for novice learners.

Shan Huang is a Ph.D. candidate in Computer Science at the University of Illinois Urbana-Champaign. She is broadly interested in how educational games can improve student learning. Current work includes improving student learning in cybersecurity with educational games and accessing student knowledge of cybersecurity concepts. Shan is also involved in various educational data mining projects.

Workshop sponsored by UMBC & Army Research Laboratory

Feb. 25-26, 2025 UMBC Training Centers, Columbia, MD & online

position paper deadline extended to 1/20/2025

The era of generative AI is upon us, and chatbots such as chatGPT are being used by programmers at all levels of experience to produce code.  Some generative AI systems, such as Gemini Code Assist, specialize in code generation.  Unfortunately, AI-generated code often contains errors in the form of functionality that fails to meet specifications or vulnerabilities that can be exploited by hackers.  People have been working on program verification and secure coding for sixty years, but even so, the skill needed to find such errors is possessed by only a fraction of software engineers, and these skills are not being passed on to student programmers as they should be.

The goal of this FREE workshop is to gather and produce actionable ideas and suggestions that may be of use to the IT profession.  The workshop will consist of invited speakers, panels, and open discussion. 

We invite would-be participants to submit short position papers offering comments, observations, experiences, and suggestions that pertain to any or all of the following workshop themes:

1. What is or could be done to make AI-generated code more trustworthy, from the perspective of functionality and/or cybersecurity?
2. How can we do better at instilling the ideas and tools of secure development into the software profession?
3. Being able to produce quality code, with or without the aid of AI, seems to be related to system skills in general. How can we do better at giving students these skills before (or as) they enter the workplace?

Position papers should limited to three pages and submitted according to this template.  Submit your position paper via email to codebot25@umbc.edu after registering for the workshop.

The organizing committee will select several papers for live presentation at the workshop. Selection will be based on relevance to the workshop themes, technical merit, and perceived interest to the audience.  Position papers that are mere marketing pieces will not be considered, but descriptions of hardware and software solutions tying into the themes described above are welcome. Limited travel support may be available for non-local speakers. Position papers and summaries of the discussions that follow will make up the core of the workshop report.

UMBC students, both graduate or undergraduate, are welcome to submit position papers that describe their own personal experience and observations with AI-generated code in their own words.  Students may include their resumes with position papers if they wish to have their work/resume circulated to other attendees.  Domestic and international students are welcome to participate in this workshop.

Important Dates:

  Position paper submission deadline: January 20, 2025

  Notice of acceptance: January 31, 2025
  Registration deadline: February 18, 2025
    (no registration fee, but space is limited)
  Workshop dates: February 25-26, 2025

The workshop will take place at UMBC Training Centers, 6996 Columbia Gateway Dr #100, Columbia, MD 21046

REGISTER @ https://forms.gle/CipmPbbBVBLfHc728

In-person space is limited, so register early! Based on RSVPs received, the organizing committee reserves the right to be selective in whom it selects to join the in-person meeting.

Instructions for virtual participation will be made available prior to the workshop.

Organizing Committee:
  Prajna Bhandary, UMBC
  Mike De Lucia, Army Research Laboratory
  Richard Forno, UMBC
  Lindsay Gaughan, UMBC Training Centers
  Cynthia Matuszek, UMBC
  Charles Nicholas, UMBC
  Steve Simske, Colorado State University
  Larry Wagoner, Dept. of Defense
  Linda Kidder Yarlott, UMBC
  Paul Yu, Army Research Laboratory

Questions? Send email to codebot25@umbc.edu