Strategic Human Oversight Frameworks for AI-Enabled Training Microagents: Evidence from a Longitudinal Adoption Study

Abstract

Purpose: The purpose of this study is to examine how AI-enabled training outcomes evolve as an intervention transition from a supervised doctoral pilot to a scaled, longitudinal organizational deployment. The study focuses on learning adoption and learning efficiency in a frontline hospitality context, while explicitly examining the role of AI micro-agents operating within a human-in-the-loop governance framework.

Methods: The study adopts a longitudinal cohort extension design, building on a doctoral pilot conducted with 100 frontline employees during 2024 and extending into a scaled operational deployment during 2025. Objective learning-platform trace data from an AI-enabled training system were analyzed across two deployment phases. Learning adoption was measured using exposure-adjusted completion rates, while learning efficiency was assessed using assessment performance and time-on-task metrics. The analysis controls for workforce churn characteristic of frontline service environments.

Findings: Results show that completion rates normalized from 100% in the pilot phase to 86.82% under real-world scale, reflecting operational normalization rather than reduced effectiveness. Importantly, learning quality and efficiency improved over time: mean assessment scores increased, while average time-on-task declined significantly. These findings indicate faster mastery and deeper learning as AI-enabled training matured, rather than superficial compliance.

Implications: The findings demonstrate that AI-enabled training systems can sustain adoption and improve learning efficiency at scale when designed with constrained agency and supported by human oversight. For organizations in high-churn frontline environments, the results emphasize the importance of evaluating training effectiveness beyond pilot completion metrics and focusing on longitudinal learning quality, efficiency, and governance structures.

Originality: This study provides rare longitudinal, post-dissertation evidence on AI-enabled training effectiveness, directly linking a doctoral pilot to scaled organizational deployment. It advances technology management and digital learning research by introducing a churn-aware evaluation framework and empirically demonstrating how AI micro-agents, operating within human-in-the-loop governance, shape sustainable learning outcomes beyond pilot conditions.

  • Page Number : 87-102

  • Published Date : 2026-04-15

  • Keywords
    AI enabled training, Micro agents, AI governance, Longitudinal study, Technology adoption

  • DOI Number
    10.15415/jtmge/2025.162007

  • Authors
    Smrite Goudhaman

References

  • Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., ... & Horvitz, E. (2019, May). Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems (pp. 1-13). https://doi.org/10.1145/3290605.3300233
  • Aradhyula, G. (2024). Human-in-the-loop machine learning systems. World Journal of Advanced Engineering Technology and Sciences, 11(1), 514–521. https://doi.org/10.30574/wjaets.2024.11.1.0012
  • Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel psychology, 41(1), 63-105. https://doi.org/10.1111/j.1744-6570.1988.tb00632.x
  • Brynjolfsson, E., Rock, D., & Syverson, C. (2021). The productivity J-curve: How intangibles complement general purpose technologies. American Economic Journal: Macroeconomics, 13(1), 333-372. https://doi.org/10.1257/mac.20180386
  • Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 13(3), 319-340. https://doi.org/10.2307/249008
  • Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of experimental psychology: General, 144(1), 114. https://doi.org/10.1037/xge0000033
  • Dote Pardo, J. S. (2025). Artificial intelligence in organizational learning: a systematic review of applications, opportunities, and challenges. Development and Learning in Organizations: An International Journal, 1-6. https://doi.org/10.1108/DLO-06-2025-0228
  • Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2019). Cognitive challenges in human-AI collaboration: Investigating the path towards productive delegation (SSRN Scholarly Paper 3368813). Social Science Research Network. https://doi.org/10.2139/ssrn.3368813
  • Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025). Complementarity in human-AI collaboration: Concept, sources, and evidence. European Journal of Information Systems, 34(6), 979-1002. https://doi.org/10.1080/0960085X.2025.2475962
  • Muehlemann, S. (2025). Artificial intelligence adoption and workplace training. Journal of Economic Behavior & Organization, 238, 107206. https://doi.org/10.1016/j.jebo.2025.107206
  • Papagiannidis, E., Mikalef, P., & Conboy, K. (2025). Responsible artificial intelligence governance: A review and research framework. The Journal of Strategic Information Systems, 34(2), 101885. https://doi.org/10.1016/j.jsis.2024.101885
  • Pappas, I. O., Giannakos, M. N., & Sampson, D. G. (2019). Fuzzy set analysis as a means to understand users of 21st-century learning systems: The case of mobile learning and reflections on learning analytics research. Computers in Human Behavior, 92, 646-659. https://doi.org/10.1016/j.chb.2017.10.010
  • Passmore, J., & Daly, J. (2026). Are AI Coaching Agents Learning Friend or Foe?: A Qualitative Study of Learning and Development Leaders' Perceptions of AI Coaching Agents Using Unified Theory of Acceptance and Use of Technology 2. International Journal of Training and Development. https://doi.org/10.1111/ijtd.70028
  • Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of applied psychology, 88(5), 879. https://doi.org/10.1037/0021-9010.88.5.879
  • Raisch, S., & Krakowski, S. (2021). Artificial intelligence and management: The automation–augmentation paradox. Academy of management review, 46(1), 192-210. https://doi.org/10.5465/amr.2018.0072
  • Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management science, 46(2), 186-204. https://doi.org/10.1287/mnsc.46.2.186.11926
  • Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view1. MIS quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540
  • Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the Unified Theory of Acceptance and Use of Technology1. MIS quarterly, 36(1), 157-178. https://doi.org/10.2307/41410412