Irvine, California, August 25 2024: ASME International Mechanical Engineering Congress & Exposition (IMECE) has accepted a paper presentation by SimInsights representative Raj Jha. The paper is titled “Closing the Skills Gap With Generative AI, XR and Learning Engineering”. The conference will be held at Oregon Convention Center in Portland on Nov 17-21.
Presentation Abstract
Skills gap is a major problem in advanced manufacturing and many other industries. The problem is further exacerbated by shortage of skilled instructors, rapid technological advances and an aging workforce approaching retirement. Here, we review the landscape of AI, XR and Learning Engineering to outline a promising approach to addressing workforce training.
Generative AI (Gen-AI) technologies have developed rapidly on both research and commercialization fronts in recent years, as seen from the rapid launch of increasingly powerful models from OpenAI, Google, Nvidia, and many other smaller startups. Concurrently, VR/MR hardware industry has produced powerful devices at affordable prices. Combining these two technologies can advance the price-performance of both the stimulus-generation and response-evaluation aspects of training and assessment, ultimately realizing the promise of personalized and adaptive learning at scale and delivering outcomes approaching 1-1 expert-led tutoring and apprenticeship. However, a critical barrier is the high cost of code based immersive content creation which requires a multidisciplinary skilled team of coders, artists and game designers to create and maintain training content.
We present case studies showing that co-design and Gen-AI powered no-code tools are a winning combination for reducing the time, cost and skill required for high quality XR content creation and delivery. By enabling non-technical instructional designers and subject matter experts to rapidly create, edit, share and deliver quality XR content, it becomes possible to democratize high quality workforce training, while also delivering high return on investment (ROI), rapid time to value (TTV)) and functional (engagement, learning, retention and transfer) benefits.
Based on a review of the end to end needs of the training industry and the design patterns of no-code tools, we list a core set of key features desired in a general-purpose no-code tool for immersive interactive VR/AR simulations for training:
- Drag-and-Drop interface that allows users to create simulations by simply dragging and dropping pre-built components and assets which would eliminate the need for coding knowledge
- Asset library of pre-built 3D models, environments, audio clips, and other assets that users can choose from to build their simulations. The asset library should be diverse and customizable to accommodate various training scenarios including defense, manufacturing, healthcare and education.
- Simulation authoring to enable users to author interactive scenarios by defining the behavior of objects, characters, and environments using triggers, actions, animations, and interactions via visual interface.
- Cross platform support so that authored content can run on a wide range of devices, such as Oculus Rift, HTC Vive, Microsoft HoloLens, and mobile AR platforms like ARKit and ARCore.
- Real-time editing and preview to allow users to make changes to the simulation and see the results immediately, which would enable iterative design and rapid prototyping.
- Collaboration and sharing to allow multiple users to work together on the same simulation similar to Google Docs.
- Analytics and tracking to gather data on trainee interactions and performance within the simulation so that trainers and organizations can track progress, identify areas for improvement, and make data-driven decisions regarding the effectiveness of the training and conduct learning engineering to optimize the learners’ experiences.
- Integrations with other systems and tools commonly used in training, such as learning management systems (LMS), 3D CAD and other software, marketplaces etc., which would allow seamless integration of the simulations into existing training workflows.
- Scalability and performance to support complex simulations with a large number of objects and interactions which also optimizes performance to ensure smooth and immersive experiences for trainees regardless of the variation in client device capabilities.
In this presentation, we will outline high level architectures for no-code tools that serve common use cases in training such as rapid scenario creation, automated response scoring and feedback, and adaptive dialogue with AI agents. We will also present results from studies of efficiency of creation, AI-powered training, cross-platform deployment, co-design with subject matter experts and community members and creation by students to highlight the wide ranging impact of broader participation and collaboration on content quality, agility, learner experience and outcomes. Through these examples, we aim to demonstrate the opportunities afforded by co-design and no-code tools for delivering the same kind of price-performance improvements in XR content that we have witnessed in XR hardware and AI models.
About HyperSkill
HyperSkill is a no-code 3D simulation platform for both Virtual and Augmented Reality. HyperSkill was created to enable instructional designers to create immersive training content without having to learn programming. HyperSkill includes thousands of 3D assets and simulations that are accessible for free to users. HyperSkill enables non-programmers to author VR/AR content, publish it to various devices and audiences and collect and visualize experience data. HyperSkill has been developed with R&D funding from the National Science Foundation. To learn more, please visit https://siminsights.com/hyperskill/