Research Area
Expert-in-the-loop machine learning

What’s important about machine learning has little to do with machines. It’s the learning that matters. Expert-in-the-loop machine learning is a framework to engage and learn in real-time from experts on a specific time and place. Familiar digital channels, natural language.

Beyond Algorithms

Not everything can – or should be – automated. Not everything can be machine learned from existing data. Large-scale data collection and aggregation is expensive and organizationally exhausting. Machine learning is perceived as focused solely on algorithms and automation; it can also be turned to how machines can learn from humans. Expert-in-the-loop machine learning (also called interactive learning) can markedly decrease the number of labelled data points necessary for prediction tasks, instead relying on domain expertise. EIL ML focuses on the use of human insight: to efficiently train machine learning models and access difficult-to-acquire information that can increase the accuracy and precision of predictive models. The Google prompt on your phone asking whether a cafe you’ve visited is crowded is an example. Interactive learning can be used for nobler aims.

Deploying at Scale

How can organizations deploy machine learning more rapidly and at greater scale? Learn from the experts. Make it simple for experts to share what they see and what they know. An expert can be a health worker, a surgeon, and someone who works in a warehouse: all have insights and information that when properly analyzed, can guide MACRO-EYES technology that learns to address problems before they occur.

The US Department of Defense has a strategy for operational intelligence built on the belief that “every soldier is a sensor” – a guiding concept for MACRO-EYES that we extend across domains. When every health worker, everyone who is part of a supply chain, every soldier becomes a sensor – existing infrastructure rapidly becomes resilient. Learn from the people who know. Learn from the people who are there now. There is also a phenomenology of AI to consider: use the system, and you understand it. Contribute to the system and you can learn its logic by tracing your input to its output.

The most radical element of our implementations of expert-in-the-loop machine learning is the capability to build systems for bio-surveillance / information infrastructure in months, rather than years at fraction of the conventional cost. Imagine being able to deploy, anywhere in the world – and within a few months - infrastructure to understand populations, the burden of disease, and the context for care. Satellites image every quadrant of the earth in increasing granularity and at decreasing cost. There are 4B smart phones on earth (and 5.7B adults); smart(ish) phones are present everywhere. The combination of expert-in-the-loop + insight machine learned from satellite imagery is extremely scalable. The ‘satellite + mobile’ dyad for learning at scale makes it easier to transition existing infrastructure to predictive systems, capable of anticipating risk and identifying improvements in care with time to prepare.


Health workers in Mozambique sent messages on what they deemed important. MACRO-EYES AI learned from the flow of messages to identify workforce sentiment and day-to-day facility capacity. When did health workers feel overwhelmed? When did the day feel under control?

When frontline sentiment is combined with routinely collected utilization, MACRO-EYES AI hones in on an elusive, critical measure of health system resiliency: which health facilities can care for more clients and which facilities cannot effectively manage the current population. No government in the world can see true capacity at the health facility level. We’re working to change that.


Our first customer was the Byers Eye Institute at Stanford [Stanford Medicine]. At Stanford, we tested and refined our technology for machine learning patient-to-patient similarity for clinical decision support. MACRO-EYES software was deployed on-premise behind the Stanford firewall. We couldn’t see or access the data that was analyzed by MACRO-EYES software and had to debug remotely. The software allows clinicians and scientists to respond to what the software deems a cohort of similar patients. The user can label a patient as particularly similar or dissimilar and explain why. The distance metrics that determine similarity then recalibrate based on this input, learning from the expert in real-time.