Liberty Munson is the Director of Psychometrics for the Microsoft Worldwide Learning organization. She is considered a thought leader in the certification industry, especially in areas related to accessibility and inclusivity of credentialing experiences and how technology can fundamentally change our approach to assessment design, development, delivery, and sustainment, and has proposed many innovative ideas related to the future of certification. She recently received the eAssessment Association’s Practitioner of the Year Award—Individual 2024.
Before Microsoft, she worked at Boeing in their Employee Selection Group and internal certification program and managed their Employee Survey. She received her BS in Psychology from Iowa State University and her MA and PhD in Industrial/Organizational Psychology with minors in Quantitative Psychology and Human Resource Management from the University of Illinois at Urbana-Champaign.
In a candid conversation with the CIO Magazine, Liberty shares her journey, insights into psychometric trends, and vision for the future of learning and credentialing, offering a glimpse into how Microsoft is shaping opportunities for individuals and organizations worldwide.
Could you share what initially drew you to psychometrics and how your career evolved from your academic pursuits to your role at Microsoft?
I have a PhD in industrial-organizational psychology, which is the science of creating more productive workforces. One way to do this is by hiring the right people through selection testing. Whether you realize it or not, an interview is a test. During graduate school, I did an internship at GTE, helping them develop an entry level selection process. I fell in love with assessment; being able to understand someone’s strengths and weaknesses with just a few well defined questions can change people’s lives for the better. Assessment helps people find the right careers and helps organizations find the right talent.
After graduate school, I went to work at Boeing in their Employee Selection Group. A few years later, I transitioned to lead their employee survey. Not long after that, I realized that I loved assessment more than surveys, so I started looking for a role that would get me back into the world of assessment.
At the time, Microsoft was hiring a psychometrician to oversee the measurement elements of their technical certification program. Although it wasn’t employee selection, it did get me back into something that I loved—assessing knowledge, skills, and abilities. So, more than 18 years ago, I took a leap to work for Microsoft as their sole psychometrician, ensuring that our certifications were valid, reliable, fair, and meaningful assessments of someone’s Microsoft technical skills. Over my career at Microsoft, I have easily managed the psychometrics of hundreds of different certifications and credentials. I have pushed the certification industry to think differently about how we assess skills; I have pushed Microsoft to be one of the most innovative IT certification programs, leading the way when it comes to the inclusivity and accessibility of our assessment experiences. I love the work that I do as Microsoft’s psychometrician so much that I never made it back to selection testing. I fell in love with certification and credentialing and don’t plan to leave.
While most people at Microsoft move from role to role, I have spent the 18 years (and counting) that I have been at Microsoft in the same role, ensuring the integrity of our technical credentialing program and building it into one of the most well respected, innovative IT certification programs in the world. I like to say that I have a unique set of skills and that’s why I have stayed in the same role at Microsoft for so long. But I truly love the work that I do every day and ensuring that Microsoft is creating opportunities for people to find jobs, get promotions, etc. through the power of credentialing.
Psychometrics is a critical field for validating skills in the rapidly evolving tech landscape. What are some of the most significant challenges you’ve faced in ensuring the reliability and validity of assessments? How have you addressed these challenges through innovation?
One challenge that all programs face, if their credentials have any value to organizations and candidates, is cheating in all its forms: braindump sites that sell exam content to candidates; proxy testing (having someone else take the exam whether they are physically in the exam location or somewhere else); leveraging tools like ChatGPT to answer test questions, among other activities. Staying on top of the new ways that people are attempting to cheat is critical to ensuring the integrity of the assessment process as well as the value of the credential.
Ensuring the security of certification is a challenge, and many programs are using AI to mitigate these threats by assisting proctors in detecting anomalies in test-taking behaviors. Programs are starting to use generative AI to create exam content, creating larger pools of items to make braindump sites less likely to give candidates an unfair advantage, and so on.
Another challenge that all cloud-based credentials face is the pace at which the technology that is the basis of our assessments is changing. How do we keep our exams and assessments up to date and relevant when the underlying technology can literally change daily? Microsoft has been doing this by refreshing exams on a quarterly basis. Because we have over 50 certifications, we have had to be innovative in our approach to how we develop, review, and update our exams. I leverage dynamic approaches to delivering exam content to gather psychometric information about the items, and our exam development vendors have taken innovative approaches to item development to streamline the process for quick turnaround time. I anticipate that the AI item generation tool that Microsoft is developing will make the process even more efficient. However, we often create exam questions on technologies that have very little documentation. This makes using generative AI to assist with item development tricky, if not impossible… you can’t create items using generative AI when there’s little to no content that can be used to generate those items–another challenge for IT certification programs building exams on new or constantly changing technologies, like AI itself.
The world of certifications and assessments has changed dramatically over the years. From your perspective, what are the emerging trends in psychometric assessments, and how do you see them reshaping the future of learning and credentialing?
Obviously, the biggest trend is in the way that AI is changing the way we think about assessment from design through delivery. For example, we are already seeing AI being used to generate items and improve test security. But it will also help us perform more effective job task analyses to really understand what people are doing in their jobs. Right now, we rely on subject matter experts, who may not be as reliable in recalling what they do on their job as we might like. With ML and AI, we can transform the digital exhaust (telemetry) that people leave behind into tasks that they perform for a more accurate representation of what they are doing, when, and how often.
Machine learning models can handle complex data sets, providing deeper insights into test results. This will allow psychometricians to uncover subtle relationships and predictors that might be missed with traditional statistical methods, increasing the power of the assessment process, if not changing assessment altogether.
One of the most promising emerging ideas is the possibility of AI being used to develop personalized assessments that adapt to the individual needs and abilities of test-takers. This can lead to more accurate measurements of abilities and traits, as well as a more engaging assessment experience.
Along those lines, AI-powered platforms can provide immediate feedback to test-takers, helping them understand their strengths and areas for improvement. This real-time feedback can be particularly valuable in educational and training contexts.
It may seem counterintuitive given the conversations around the bias in AI models, but I believe that AI can help ensure that assessments are fair and unbiased by identifying and mitigating potential biases in test items and scoring algorithms. This is crucial for maintaining the validity and reliability of psychometric assessments.
Beyond AI, technologies, such as mobile apps, games, virtual reality, enable real-time data collection and immediate feedback. This allows for more dynamic and interactive assessments, that I suspect test takers find more fun than the traditional multiple-choice exam.
The shift from traditional paper-and-pencil tests to digital formats has expanded the reach and accessibility of our assessments. Virtual reality (VR) and gamified assessments are emerging as innovative tools to evaluate cognitive and behavioral functions in more engaging and realistic environments.
Wearable technology can collect continuous physiological and behavioral data, offering new dimensions of assessment. For example, they can monitor stress levels, physical activity, and sleep patterns, which can be correlated with psychological states. While I’m not sure how I will leverage wearables in Microsoft’s credential program, I know that other assessment programs are exploring how to leverage this data to make more informed decisions.
My dream is what I have been calling “in the flow of work” assessment. For people who are already doing the job that is being certified, why do they have to take an exam to earn the certification? With advances in technology, there must be a way to assess them as they do their jobs and award the certification accordingly. This does introduce some interesting psychometric challenges and questions, but I would love to see the industry start to figure out how to make an assessment process like this work. (I do realize that this won’t work for all certifications, but I have the luxury of working at Microsoft where all my certifications are based on software solutions where something like this is possible.)
At the end of the day, the closer we can get our assessment process to what we are trying to certify (e.g., Azure administration, development, engineering AI solutions, etc.), the more candidates, hiring managers, and organization will accept the assessment process. And that’s the direction we need to head as an industry.
Your work involves collaboration across teams, from psychometrics to curriculum development and security. How do you foster effective communication and alignment across such diverse disciplines?
I’m not sure I have the secret sauce for this, but I leverage all the collaboration tools that Microsoft provides, especially Teams; Teams recap provides a great summary of the meeting, creates action items, and allows people to see when their name was mentioned in the transcript, making it easy for anyone who missed the meeting to catch up.
I establish clear, shared goals to ensure everyone is working towards the same outcomes and holding regular meetings to share progress, discuss challenges, align on priorities, and update those goals as necessary.
When we work with a team that doesn’t live and breathe assessment the way that I do, I’ll have 1:1 meetings with them to “onboard” them to the world of certification and credentialing, explaining key terminology and how its different from the tests they took in school and why that matters.
We often appoint “liaisons” within each team who are responsible for communication and coordination to streamline interactions and ensure that information flows smoothly between groups. We expect this liaison to communicate key information back to their teams and share any concerns that their team may have. This ensures that the key stakeholders have all the information without everyone having to attend every meeting. We also encourage feedback so that teams can voice concerns and suggest improvements, and we can adapt processes as needed.
With advancements in AI and adaptive learning, what opportunities and risks do you foresee for the future of technical certification and psychometrics?
I discussed many of the opportunities in my response above, but the greatest opportunity we have is changing what assessment looks like. I imagine a future where exams don’t feel like exams. We can harness the power of technology and AI to change the way we assess people through personalization, wearables, virtual reality, games, and work-based experiences where someone doesn’t have to leave their job to take a test to earn a credential (“in the flow of work”).
In addition, in the not-too-distant future, we should be able to use AI to predict the psychometric characteristics of items. If we’re using AI to generate more items and this can be done quickly and efficiently, we must address the downstream bottlenecks that slow down our ability to score those items on assessments/exams. One of those bottlenecks is understanding the psychometric properties of these new items, so if we can use AI to estimate those psychometric properties, we no longer have to “seed” questions (include them on the assessment as unscored while we collect the data needed for psychometrics); we can start scoring them immediately. In a world where piracy is a huge problem, having larger item pools is one important mitigating factor but only if we can leverage those items to score candidates as quickly as possible.
In terms of risks, the two biggest risks are the belief that AI is a silver bullet and making sure that we’re using AI ethically and responsibly. I think some programs will rush to use AI in ways that will be perceived to create efficiency but may not be more efficient or, worse, not in the best interests of the test taker. Currently, using AI for scoring in high-stakes situations may be the least ethical use, especially if humans aren’t in the loop (and may be extremely difficult under the EU AI Act), yet I don’t think many leaders see the risks associated with using it in this way.
We must continue to put validity, reliability, and fairness front and center of all decisions when using AI in exam development, delivery, and sustainment. I can see how it might be easy to fall into the “AI for everything” trap and forget that we still need to make sure that whatever we create is valid, reliable, and fair. The lure of the perceived power of AI to make everything more efficient is real, but even as an item generation tool, AI still has a long way to go.
What personal philosophy or life experience has guided you in your career and continues to inspire you in your work at Microsoft?
Stay curious and open-minded. My curiosity and the desire to learn from others have been key factors to the success I’ve had in my career. My insatiable curiosity has led to many interesting opportunities within Microsoft and beyond. For example, because I spoke with our candidates to learn more about their experiences, I have found pain points that could be mitigated in ways that don’t compromise the integrity of the assessment process. Beyond Microsoft, I have presented at conferences hundreds of times, participated as a Board member for a wide variety of associations, have published articles and book chapters, and have been the keynote speaker at numerous conferences. These experiences have allowed me to share my experiences at Microsoft with others so that they can learn from, and hopefully, be inspired by, what we’re doing. I am proud of what I have accomplished at Microsoft, and knowing that people are learning from that, I am inspired every day.