capstone_nasajpl_cover.png

NASA JPL

Working with the Ops Lab at NASA’s Jet Propulsion Laboratory to research and design ways to improve the efficiency of operations for the upcoming Europa Clipper mission

 
 

Sponsor

NASA Jet Propulsion Laboratory

Duration

18 weeks

Team

Daphne Liang
Will Oberleitner
Victoria Song

Role

Research Lead
Project Manager

Methods

Semi-structured interviews
Collaborative diagramming
Competitive analysis
Affinity mapping
Information visualization
Usability testing

 
 
 

Overview

For my graduate capstone project, my team and I were tasked by JPL's Ops Lab to design a concept for an interface to help improve the efficiency of operations for the upcoming Europa Clipper mission. Clipper will place a spacecraft in orbit around Jupiter to test the hypothesis that Europa, its icy moon, could support life. However, it is still in its early phases of planning and mission design and will not launch for at least another 4-5 years. The Ops Lab is currently working with systems engineers, mission planners, and other personnel to help design a suite of software tools to aid in operations. The goal of our project is to research the space of mission operations beyond just Europa Clipper to provide the team at JPL with design recommendations on how to improve the efficiency of planning and scheduling of spacecraft activity.

 

Concept of Clipper spacecraft design by Will Oberleitner

 

The Problem

Every new NASA mission tries new ways of improving the efficiency of operations, mainly because rules imposed by headquarters, most significant of which are tight budgets, constrain workforce size and dictate expected outcomes. All of this essentially means missions want to collect as much objective-relevant scientific data in the least amount of time and with as little extraneous effort as possible. The Europa Clipper mission has a requirement to uphold 1:1 uplink operations, which means that the ground team can only take as much time to plan, schedule, and uplink activities as it takes for the spacecraft to execute them. In the past, this ratio has been much higher - on the Cassini-Huygens mission to Saturn, it took at least 4 times as long to plan and schedule activities as the spacecraft took to execute them. Europa Clipper is working with a much tighter budget than Cassini, so various efforts are underway to improve operations efficiency as a primary means for reducing cost. One of these is an automated planning and scheduling tool for which my team was tasked with designing a concept. Because this was all the information we were given, we had a lot of research to do to uncover the complexities of and draw insights from the inner workings of JPL mission operations.

 

- Challenges -

Designing for the future

Clipper is still years away from launching, so it is still in the early phases of planning. Thus not much can be said definitively about how operations will be structured and subsequently unfold. In order to design for the future, we had to look into the past at similar orbiting missions like Cassini to learn what those missions have in common and what could be improved for Europa.
 

Bureaucratic limits

As a student group, we had to be careful not to overstep our boundaries. Working on a mission with so much uncertainty meant we had very little freedom in accessing research participants - in fact we were forbidden from taking time from anyone on the Europa Clipper mission unless explicit permission was granted. Navigating NASA's bureaucracy proved to be extremely limiting and required a lot of creativity to succeed.
 

Remote research

Most research activities were conducted remotely (mostly interviews and collaborative activities via Skype or Webex) as budget and bureaucratic constraints did not allow us to travel to JPL to conduct research such as contextual inquiries. This required being creative with what we could do via video and with other online resources.

 

- Design Challenge -

Throughout the process we had been considering how to narrow down the challenge given to us by JPL into a concrete, measurable design challenge. This required identifying problems contributed to decreased efficiency and which ones were feasible to solve. Our insights helped solidify a direction and ultimately informed our current challenge statement:

Though many factors contribute to inefficiencies in mission operations (it is an enormously complex sociotechnical system after all), we decided that attempting to promote cohesion across instrument teams would alleviate the conflicts caused by fragmentation of software tools. A planning and scheduling tool that facilitates cohesion could potentially help resolve issues early in the process and avoid the need for multiple iterations of conflict resolution.

 

Research

See the full research report for a more thorough overview of our research process and outcomes.

- Objectives -

1

Make sense of the organizational structure within JPL and the flow of interactions that define operations for orbiter missions

2

Learn about the intricacies of planning, scheduling, and sequencing involved in uplink operations

3

Understand perspectives on how software has been used on past missions to aid in planning and scheduling

 

- Participants -

Technically all of our interview/activity participants were experts, including all stakeholders. Due to each participant's unique experience, we used a template for an interview guide and customized it for each participant. See a sample interview guide here.

Overview of participants & main topics learned about through each

Instrument Scientists

Responsible for developing detailed plans for their instrument's data collection. Their primary goal is to make discoveries they can publish to contribute to discourse within the scientific community.

Science Planners

Responsible for ensuring instrument teams’ plans stay on track with mission objectives by integrating them and looking for any conflicts between instrument plans and with mission objectives. 

Systems Engineers

This is a fluid role that is generally responsible for coordinating among teams of scientists and engineers. Many are primarily responsible for integrating the plans of scientists and engineers, iteratively modeling them to find conflicts, and eventually sequencing commands that get uplinked to the spacecraft.

JPL Design Researchers

Researchers and designers mainly within the Ops Lab who are working to maintain the importance of human-centered design within the aerospace industry.

NASA Researchers

We spoke to various experts on automation and planning and scheduling problems who research and help design software tools to solve mission-critical problems.
 

- Methods -

 
 

Literature Review

Why? 

  • Familiarize ourselves with the work of our participants
  • Supplement/corroborate information learned in interviews
  • Understand how software works (making up for lack of contextual inquiry/other software evaluation methods)

How?

  • Read relevant papers written by interview participants
  • Tailor interview guide to their expertise
  • Synthesize information learned in papers into diagrams and share with rest of team

Outcomes

  • Preparation for interviews & the ability to contextualize information learned in interviews
  • Click here for a synthesis of the literature.

Semi-structured Interviews

Why?

  • Primary method for gaining insight into mission operations
  • Understand stakeholders' unique perspectives on operations
  • Identify pain points in stakeholders' experience
  • Fill in knowledge gaps left by high-level information available in literature

How?

  • 1 hour (30 minutes with activity)
  • Skype or Webex
  • Prepare interview guide in advance that aligns with research objectives and participants' expertise

Outcomes

  • Foundational data, primary insights, design principles

Collaborative Diagramming

Why?

  • Uncover more detail about the process for which we're designing
  • Difficult to get a complete picture from open-ended interview questions
  • Check assumptions
  • Identify pain points & possible points of intervention
  • Compare experiences across participants

How?

  • 15-30 minutes at the end of an interview
  • Remote collaboration via draw.io
  • Think-aloud protocol
    • Encourage participants to call out discrepancies
  • Iteratively edit the diagram after each interview

Outcomes

  • Detailed understanding of mission operations, stakeholders' experience, and clear points of intervention

Competitive Analysis

Why?

  • Supplement lack of access to JPL software
  • Identify features to consider & ones to avoid in related software

How?

  • Identify publicly available tools with desirable functionality:
    • Project management, version control, visualization
  • Establish guidelines/heuristics and tasks for evaluation
  • Walk through tasks to identify problems & features to consider

Outcomes

  • Features to consider and ones to avoid - incorporated into our concept map for the "ideal scheduling tool"
 
 
 

Synthesis

Because of the steady flow of information from interviews and literature and the complex nature of the problem space, many of our regular meetings were spent discussing what we learned and piecing together an actionable understanding of operations at JPL. Sense-making was constantly occurring throughout the research process.

- Diagramming -

In order to better understand all that we were reading in the literature and hearing from participants, we created diagrams of our understanding of various processes and organizational structures relevant to orbiter missions.

 
 

Methods of Planning

One of the first things we learned is how nuanced the science planning process is, even within a mission. This represents one method used to plan activities for the Cassini Solstice Mission extension.

Cassini Tour Design

Understanding how science drives every part of the mission - trajectories are designed far in advance of launch and involve lot of negotiation to maximize the opportunities available for relevant data collection. I created this diagram to represent one way that trajectories have been designed in the past.

Representing Shared Opportunities

In order to understand how instruments can work together to achieve shared science goals, I sketched my understanding of how orbital segments were allocated on Cassini.

 
 
 

Iterative Diagram

By far one of the most crucial methods for forming an understanding of the problem space and for identifying possible points of intervention, the diagram we built collaboratively with our participants grew from a rough framework of our mental model to a complex process flow. 

Early Stage

<! data-preserve-html-node="true"-- Generator: Sketch 49.3 (51167) - http://www.bohemiancoding.com/sketch --> Shape Created with Sketch.

<! data-preserve-html-node="true"-- Generator: Sketch 49.3 (51167) - http://www.bohemiancoding.com/sketch --> Shape Created with Sketch.

Late Stage

 

- Affinity Mapping -

Affinity mapping was crucial for piecing together all the information we'd learned during interviews and synthesizing it into actionable insights for a future design. We began coding before our first round of interviews had finished, so the whole process took over 2 weeks. We'd collected data on pretty much every major aspect of the mission, so identifying themes was highly challenging. In the end we had produced hundreds of pieces of data, over 36 themes, and over 12 insights spanning 3 blackboards, which we eventually narrowed down to 8 insights. With multiple iterations of rearranging and re-identifying themes, we were able to see clearly how our research objectives guided outcomes.

 
 

Objective 1: Organizational Structure

Objective 2: Planning Process

Objective 3: Software Tools

 
 
 

Findings & Outcomes

Overall, synthesis helped us piece together a complete understanding of our problem space and allowed us to focus on particular problems within that space. As mentioned above, our research uncovered many, so part of my role was to prioritize and identify which was most worth solving. We all agreed to focus on the fragmentation of software tools, since this issue had come up consistently across participants when asked about software usage in mission operations.

- Insights -

1

Missions are planned and simulated in as much detail as possible before launch to optimize the cost and efficiency of operations and reduce error, but they are also flexible enough to account for anomalies.

“You can make later updates to [the plan] but it was planned quite far out.” P1
"You think something is going to work but it doesn't quite once you get there." P4

2

Scientists' needs are grounded in instrument constraints, thus the more complex the constraints are, the more contentious negotiations of instrument and resource usage will be.

The Saturn Working Group “had a lot more … heated discussion from what I’ve heard because they’re the instruments that take photos of Saturn [so] they use a lot more data.” P12
“We have to constrain the amount of data we record far below [what] we could record to make sure to get data back.” P7

3

Science rules the mission, but because it can be put at risk by any problems with the spacecraft or its instruments, ensuring the health and safety of the spacecraft is paramount.

"There’s a huge amount of pressure not to declare the whole mission a failure - for political… for a lot of reasons.” Engineers will do everything possible to get the spacecraft functioning. P11
“Scientists can “really get off into the weeds” discussing possibilities. Engineers are there “to keep sanity.” P2

4

Mission operations are always adapting, learning from problems both in past missions and ongoing ones.

“They [Clipper team] are building the spacecraft very smartly.” P2

“We’re building models very early. We build models that run simulations and simulations help us discover if there might be opps or problems in our tour design. Then the model that is created will be continually updated during the mission... We’ll be able to put a plan in, run it against the planning tool and see if it makes sense.“ P13

5

Fragmentation of software tools across teams poses a challenge to science collaboration and contributes significantly to conflict.

“Instrument teams are not always aware of what other instruments are planning.” P6
“It was a combination of JPL not taking the right path in developing tools… they wanted to develop something for the uplink process but what we needed was a rudimentary planning tool that gave you a high level view of your planning process.” P1

6

Cross-pollination across diverse teams facilitates healthy collaboration and cohesion across disciplines because it ensures that no team’s individual needs dominate.

“It’s a good thing in my opinion to have scientists who are orthogonal: member of an instrument team and a Thematic Working Group.” P2
“The investigation scientists … would know what spacecraft things affect their instruments.” P1

7

Data return is emotionally charged because careers are at stake.

“People get emotionally attached to the work, the more time you work on something the more upset you get when someone comes in and says we’re going to do something different” P4

8

While automation is welcomed if scientists and engineers understand how and why decisions are made, it is still crucial that humans remain the arbiters in all aspects of decision-making.

“One of the reasons why the automated planning & scheduling wasn’t entirely successful was because it did not have the ability to accept the kind of preference information scientists wanted.” P5
“That’s why it [encoding of science objectives into software] typically hasn’t been done, and I’m not sure the scientists would give that up. They’d rather do it themselves.” P6
 

Moving Forward

We are now shifting from pure research mode into the design phase of the project - turning those insights and design principles into action to design an interface for a planning and scheduling tool for instrument scientists. So what's next?

 

Usability Evaluation

We are lacking a bit of specificity on what kinds of information stakeholders would expect/need in a planning and scheduling tool, so we need to make every effort to gain as much insight into current use of such software at JPL. One of our more recent participants referred us to a software tool, called SciBox, developed for similar planning purposes that has been used on missions such as MESSENGER and MRO. We plan on attempting to conduct remote usability evaluations of the tool with its developers and/or users by having them walk us through use cases and asking them questions to identify pain points or unmet needs. We also may be able to conduct a heuristic evaluation of SPIFe, a scheduling interface used on Mars missions, which has an open source version available.

 

Concept Generation

We already have some ideas about what features users might need in a planning and scheduling tool, based on all our exploratory research. We are currently working on sketches and storyboards of the concept for evaluation by former interview participants and our sponsors. 

 

Prototype Generation & Testing

Once we have iterated on our concepts to a point that is satisfactory to involved stakeholders and our sponsors, we will begin building low-fidelity prototypes for user testing and usability evaluations with at least 5 potential stakeholders. Following feedback on low-fidelity prototypes, we will begin the final stretch - developing high-fidelity prototypes for testing and refinement.