Matthew Garvin

Context

The Convergent Aeronautics Solutions project at NASA aims to push the boundaries of innovation in aviation by investing in improbable ideas that could solve problems in the industry. One such project involves developing infrastructure and strategy to improve regional and urban air mobility with eVTOL public and personal transportation (“flying cars”) and drones.

Role

As an innovation design researcher an UX thought leader on the team, my role on this project involved improving the user experience of internal meetings and tools to facilitate better communication among distributed teams. But secretly, I had a second goal my team didn’t know about.

My supervisor wanted me to discreetly evangelize UX to the engineers and make them more receptive to integrating qualitative data into their design. In this case study, I will describe my research, design, and development process, as well as the results and impact of the tool on the team’s productivity and performance.

Timeline

4 months

Research Overview

To understand the challenges of distributed teamwork and improve the integration of qualitative data by engineers, I conducted 20 semi-structured interviews with engineers on the team. I used this data to inform the survey design, which I then deployed as a pre-test to the engineers. Following the pre-test, I conducted three two-hour workshops with the engineers to test a new intervention, the Lightning Decision Jam. In the first hour of each workshop, the engineers conducted their meetings as usual. In the second hour, I step in to facilitate and introduce my intervention. Finally, I redistributed the survey as a post-test to assess the effectiveness of the intervention.

Analysis & synthesis

Here’s an overview of the analysis and synthesis process for my research:

1. Transcription: The first step was to transcribe your interviews and workshop sessions. This involved listening to recordings of the discussions and workshops and typing out the words that were said.
2. Coding: After transcribing the interviews and workshops, I developed a coding system to identify and label important themes and patterns that emerged from the data. This involves categorizing data based on commonalities and creating codes to represent these categories.
3. Theme Identification: After coding, I identified key themes from the data. This involves grouping codes into larger groups that capture the essence of the data.
4. Synthesis: With themes identified, I synthesized the data by drawing connections between themes and identifying patterns and insights. This involved looking at the data holistically and identifying trends that emerge from the data.
5. Visualization: I created visualizations to help communicate your findings. This could include creating graphs, charts, or other visual representations of your data highlighting the key themes and insights from your research.

Findings

Based on my analysis of the user research, I identified several pain points and opportunities for improvement in the way engineers were collaborating and integrating qualitative data. One of the key insights was that engineers had difficulty making sense of the vast amount of data available to them and often struggled to find the information they needed to make informed decisions. They also expressed frustration with the lack of structure in their meetings, which often led to unproductive discussions and wasted time.

Design

Through iterative design with increasing fidelity, I learned of some important changes that needed to be made to adapt this process to accommodate NASA engineers. This team is making decisions about life-critical systems rather than a mobile banking or food delivery app.

After incorporating the feedback from the engineers and making some initial changes to the LDJ process, I started with some low-fidelity wireframes of the process in Miro. I presented these to the team and received feedback on how the process could be further tailored to the needs of the NASA engineers.

Based on this feedback, I made adjustments to the wireframes and created a more refined prototype. I conducted some usability testing with a small group of engineers to get their feedback on the overall usability and effectiveness of the process. From this feedback, I continued to refine the prototype and made additional changes to the LDJ process. Some changes I made throughout this process include:

1. Incorporating technical specifications and constraints: You customized the problem statement and provided the engineers with technical specifications and constraints specific to the aviation industry, as well as NASA’s unique operational requirements.
2. Including relevant stakeholders: You made sure that all relevant stakeholders were included in the decision-making process. This allowed for a more diverse and inclusive perspective on the problem.
3. Focusing on data-driven decision making: You emphasized the importance of using data to make decisions. This was particularly important for NASA engineers, who rely on data to ensure the safety and reliability of their systems.
4. Providing a structured decision-making process: You created a step-by-step process that engineers could follow to make decisions. This helped to ensure that all relevant information was considered and that decisions were made in a timely and effective manner.
5. Emphasizing collaboration: You encouraged collaboration and teamwork throughout the decision-making process. This helped to break down silos between different teams and allowed for more effective communication and problem-solving.

I then created a high-fidelity prototype with the updated design and features, and conducted additional testing with a larger group of engineers. The feedback from this testing was used to make final refinements to the process and create a template that could be used by the team for their meetings moving forward.

Throughout this iterative design process, I focused on incorporating user feedback and making adjustments to the LDJ process to ensure it was tailored to the needs of the NASA engineers. The end result was a more effective and user-friendly process for decision-making in their meetings. Introducing, NASA CAS+ Critical Decision Storm.

During the workshops, I introduced this process and facilitated its implementation. After the workshops, I distributed the post-test survey to evaluate the effectiveness of the CAS+ Critical Decision Storm in improving the engineers’ ability to integrate qualitative data and make decisions more structured and efficient. A favorite component of mine with this activity, is the improvement over traditional brainstorming activities in design thinking. Often, we don’t come up with our best ideas under pressure in a timed activity, particularly if everyone is trying to shout over each other. Introverts and junior team members are less likely to speak up. Distinct work areas facilitate “working together, alone.”

The point of the activity is to make it short and sweet, rather than long and exhaustive. We quickly align on the goals for the meeting. Working together, alone, we quickly diverge and converge on what’s working and not working. We briefly review additional information for context (more detailed information is communicated beforehand and is expected to be understood by the team prior to the meeting). At this point the members list out their ideas for potential solutions. These solutions are voted on by the group, or further explained if necessary. The facilitator works with the group to place leading solutions onto a prioritization matrix to quickly decide which idea to follow up on. The beauty of this approach is that we then have several more potential solutions to revisit if this idea doesn’t work out.

Results

Based on these findings, I developed and introduced the CAS+ Critical Decision Storm, a structured process for team decision-making. During the workshops, the engineers were able to collaborate and contribute ideas and insights in a more equal and transparent manner. The structure of the Critical Decision Storm also helped ensure that each voice was heard and the team made decisions that were informed by all perspectives.

The post-test survey showed that the Lightning Decision Jam was successful in shifting the team’s approach from a discussion-oriented meeting to a decision-oriented one. Engineers reported feeling more confident that the decisions made during the workshops were informed by all perspectives, and that everyone had an equal opportunity to contribute. Overall, the Lightning Decision Jam helped improve the team’s collaboration and decision-making processes, resulting in a better user experience for the engineers involved.

Impact

The impact of my work was significant, as it led to a measurable increase in efficiency and improvement in the decision-making process of the engineering team. My variation was adopted as a regular part of the team’s meeting structure, and the team reported feeling more empowered and heard in the decision-making process. In addition, the use of qualitative data in decision-making increased, resulting in a more holistic and comprehensive approach to problem-solving. The team reported feeling more confident in their decisions and the safety of the systems they were working on. As a result of these improvements, the Convergent Aeronautics Solutions project made faster progress toward its goals, and NASA as a whole was able to increase its reputation as a leader in innovative and effective problem-solving in the urban air mobility space.

Conclusion

In conclusion, this case study demonstrates how user research can inform the design process and lead to improvements in user experience. By conducting interviews with engineers and analyzing their frustrations with decision-making, I was able to develop a new process, the CAS+ Critical Decision Storm, that addressed their needs and improved their workflows. The pre- and post-test surveys showed a significant increase in satisfaction with meeting outcomes and decision-making processes, indicating the success of the intervention. Through an iterative design process, I was able to refine the design of the Decision Jam and tailor it specifically to the needs of NASA engineers. Overall, this project was a great opportunity to use design thinking to address complex problems in a unique context, and I am proud of the impact my work had on the team and their ability to make informed decisions.