Josh, Panos & Foster have been working together on systems that tightly integrate human intelligence and AI/ML for a decade. They specialize in machine learning problems that have very rare positive examples and are not “self-revealing” – i.e., where historical data does not naturally reveal training data labels.

They designed and built the founding data science architecture for Integral Ad Science, originally founded as AdSafe Media. The AdSafe problem involved scouring the web to identify (among other things) web pages that contain ultra-rare objectionable content like hate speech, and blocking ads from appearing there.

The trio introduced a variety of methods for training that are stronger than traditional machine learning “labeling”, including new active learning strategies, active feature labeling, search-aided training, Beat the Machine, integrating expert knowledge with training data to do better than either, and integrating multiple sources of evidence—for example evidence from text and evidence from network interconnections.