Learning in public, and what iteration should mean in healthcare

Modern software teams are taught to ship early, watch what happens and improve. It is good advice. Products built this way tend to fit people's needs better than products designed in full before anyone uses them.

Healthcare complicates it. Some things can be improved release by release without anyone coming to harm. Others cannot be treated as experiments at all. A team building health technology needs to know which is which before it starts, and needs to be honest, in public, about what it learns along the way.

This piece sets out how we think about that balance at Luna. We are early in the work. Obaa+, our first product, is still in development, so treat this as a statement of intent that we expect to test and refine.

Iteration is expected in good health services

Continuous improvement is not a startup habit that health services merely tolerate. It is written into their standards. The NHS service standard asks teams to "iterate and improve frequently", and to make sure they have the capacity, resources and technical flexibility to do so. It also asks them to "use agile ways of working".

The Principles for Digital Development, endorsed by more than 300 organisations working in development, make the same point. One of their nine principles is simply "Use evidence to improve outcomes".

Both point to the same discipline. Improvement should be a planned part of the work, with time and people set aside for it, not something a team gets to only when nothing else is urgent.

Decide what you will measure before you launch

Iteration only helps if a team knows what it is trying to improve. It is easy to measure what is convenient, such as downloads, sessions and time in the app, and to forget to ask whether anyone's health or experience of care got better.

WHO's practical guide to monitoring and evaluating digital health interventions is written for implementers, researchers and policymakers. It sets out stages for checking whether an intervention is being delivered as intended, and for evaluating its impact. That distinction is useful for product teams too. First, is the product working the way we designed it to work? Second, is it making the difference we hoped for?

WHO's 2019 guideline on digital health interventions raises the bar further. It says digital technologies need to show lasting improvements over traditional ways of delivering services if they are to be sustained and built into health systems. For a health product, the honest measure of success is that kind of improvement, not engagement alone.

In practice, we try to write down, before building a feature, what we expect it to change for the people using it and how we would know. Sometimes the answer is a number. Sometimes it is a pattern in conversations with users and with the midwives involved. Either way, deciding in advance makes it harder to declare success after the fact.

Some things are not for experimenting

A fast-moving team needs a clear line between what it can change freely and what it cannot. In health technology, we think a few areas sit firmly on the far side of that line.

• Clinical safety. The NHS standard asks teams to "make your service clinically safe". Content and features that touch health decisions need review by qualified people before release, not after feedback arrives.
• Privacy and data protection. The same standard asks for "a secure service which protects people's privacy". How a product handles personal health information should never be tested on users to see how they react.
• Honesty about limits. A product should never be made to sound more certain than it is because a confident tone performs better.

Everything else, from onboarding and navigation to how education is organised, can and should improve through testing with real people. Being strict about the first group is what makes it safe to move quickly on the second.

Being open about what we learn

The NHS standard asks teams to define what success looks like and to be open about how their service is performing. We think that openness is useful well beyond public services.

For a company like ours, learning in public means a few concrete things. We aim to explain on this journal what we are building and why, including the decisions we are unsure about. We aim to share what changes as Obaa+ moves through development and testing, including when something we expected to work does not. And we aim to describe the limits of our products as plainly as their features.

There is a practical reason for this as well as a principled one. Health technology depends on trust from many directions at once. Women deciding whether to use a product need it. So do the health professionals asked to work with it, and the organisations deciding whether to support it. Being open about how we work, and about what we still do not know, is one of the few ways a young company can earn that trust before it has a long track record.

Our working rhythm

Our approach follows a simple sequence. We start by understanding the healthcare problem and the people who experience and deliver care. We then design the experience and build the product. After that we learn from what happens and improve. Then the cycle starts again.

None of the steps is unusual. The discipline is in not skipping any of them, especially the last. If you are building, commissioning or evaluating digital health services and would like to compare how you approach iteration and evidence, we would welcome the conversation.