Cross-post from http://idibon.com/the-fourth-generation-of-machine-learning-adaptive-learning/

The fourth generation of machine intelligence, adaptive learning, creates the first truly integrated human and machine learning environment. For text analytics, this has given us the most accurate analytics to date, allowing us to get actionable information in many areas for the first time. In the examples we will share here, we show that adaptive learning is 95% accurate in predicting people’s intention to purchase a car. Adaptive learning correlates with actual sales, unlike any previous approach to Machine Intelligence.

Adaptive learning combines the previous generations of rule-based, simple machine learning, and deep learning approaches to machine intelligence. Human analysts are optimally engaged in making the machine intelligence smarter, faster, and easier to interpret, building on a network of the previous generations of machine intelligence.

Four generations of machine intelligence

1st generation: Rules

The first generation of machine intelligence meant that people manually created rules. For example, in text analytics someone might create a rule that the word “Ford” followed by “Focus” meant that “Ford” referred to a car, and they would create a separate rule that “Ford” preceded by “Harrison” meant that “Ford” referred to a person.

The rule-based approach is very time consuming and not very accurate. Even after an analyst has exhausted all the words and phrases they can think of, there are always other contexts and new innovations that aren’t captured. For one of our clients, their experts analysts were only able to capture 11% of the documents they wanted to analyze using rules: this clearly is too limited.

2nd generation: Simple machine learning

The dominant form of machine intelligence today is simple machine learning. Simple machine learning uses statistical methods to make decisions about data processing. For example, a sentence might have the word “Ford” labeled as a car, and the machine learning algorithm will learn by itself that the following word “Focus” is evidence that “Ford” is a car in this context.

Simple machine learning can be fast, provided that you already have labeled examples for ‘supervised learning’. It also tends to be more accurate, because statistics are usually better than human intuition in deciding which features (like words and phrases) matter. The major drawback for supervised machine learning is that you need the labeled examples: if you have too few labels or the labels aren’t representative of the entire data set, then the accuracy is low or limited to a specific domain.

3rd generation: Deep learning

There has been a recent rise in the use of machine learning that learns more sophisticated relationships between features, known as deep learning. For example, if you had the sentence “We Will Let Harrison Ford Focus on Star Wars”, there is conflicting evidence between “Harrison” and “Focus” about whether “Ford” is a person or a car.

Deep learning can automatically learn how to use combinations of features when making a decision. For simple machine learning, a human has to tell the algorithm which combination of features to consider. Deep learning often cuts down on the amount of human time needed and typically gets up to 5% more accurate results than simple machine learning for text analytics–although only when applied to data from the same sources as it learned from.

4th generation: Adaptive learning

Adaptive learning brings human analysts into the process at every step. This is in contrast to rule-based, simple machine learning and deep learning approaches, where the humans only create rules and label data at the start of the process. For example, if you had the sentence “We Will Help Tom Ford Escape from New York”, and your system hadn’t seen any examples of “Tom Ford” or “Ford Escape”, you will need human input to build the knowledge.

Adaptive learning systems require the least human effort because they only require human input when it matters most and continually expand their knowledge when new information is encountered. As we show here, they are also the most accurate. They combine the three other types of machine intelligence, adding new types of ‘unsupervised machine learning’ and methods for optimizing the input from multiple, possibly disagreeing, humans.

Example of why it matters

Idibon’s solutions are the fourth generation of machine intelligence. We help organizations extract actionable information from text globally, including the largest manufacturers of consumer electronics, the creator of the world’s most popular games, many financial services companies, and the United Nations. Across all our clients, the information is most useful when it correlates with actual behavior in the world. Our adaptive learning system optimizes many aspects of text analytics with layers of machine learning, optimizing what an analyst sees for review (active learning), how they see it (human-computer interaction) and how that information is applied (data vs. rewriting its own code).

Here, we’ll use an example that everyone can understand: predicting car sales from social media communications. For example, how can we predict sales of Toyota Camrys from people saying things like “Thinking of getting a Camry”? We thank the engineers, data scientists, and statisticians at Edmunds.com, the world’s most visited car information site (and Idibon client), for feedback on an early version of these results.

Here are our results, compared to deep learning and simple machine learning systems from some of the most prominent machine intelligence companies in text analytics:

Accuracy in sentiment analysis about purchasing cars: Idibon and competitor systems

Idibon’s adaptive learning is 95% accurate, compared to 69% from deep learning and 60% from simple machine learning systems. This is a huge difference. To assess this accuracy, we created and manually labeled a few thousand items and then tested them. For Idibon, none of these items were used in the learning phase: our 95% accuracy is over completely novel social media communications. We couldn’t confirm this for our competitors, so it’s possible that some of our competitors had a slight advantage.

What does 95% accuracy get you?

For any analytics to be valuable, the output needs to be actionable. So for 14 car models, we looked at how well we could predict monthly sales from text analytics alone. Predicting sales in advance of the monthly sales figures is much more valuable than simply surfacing a selection of positive and negative tweets. It can help car manufacturers understand which features are resonating with buyers in new models, it can help make logistics decisions about the numbers of cars to ship to dealers each month, and it can help an investor predict how the share price of a car company will move before that company has made their end of month announcement of sales.

For 10 of the 14 models, Idibon’s 95% accurate analytics for predicting people buying a car correlate with actual sales. These correlations were calculated via Spearman’s rank correlation and confirmed by also analyzing the data using a variety of other statistical tests (e.g., Kendall’s tau).

Conclusion

Adaptive intelligence goes beyond the ~70% accurate sentiment analysis that we’re seeing across text analytics vendors, to a level of accuracy that genuinely predicts human behavior.

Just as importantly for Idibon’s clients, it also minimizes that amount of time that analysts need to spend. Here are the results from our examples about “Ford” above:

After just tens of minutes of analyst feedback, the system can disambiguate the use of “Ford” in social media more accurately than systems built on hundreds of hours of human intervention.

One downside of adaptive learning is that it’s not easy to package as a application that you can download and run on a single server. To help people who can only run single-server applications and need 70% accuracy, Idibon gives away a 70% accurate system for free. Our Idibon Public product is 69.9% accurate on this same data, equal to the best accuracy of our deep learning competitors who charge for the service.

When you can get to 95% accuracy, you can actually match what’s happening in the real world, and Idibon helps our clients predict changes and events in many market, including the car sales example here. We regularly show how much more accurate a system can be when it takes the humans-in-the-loop seriously.

Humans have always been part of Machine Intelligence systems–they create the unstructured data, they load it into systems, they tune statistical models, they interpret results. Adaptive learning optimizes people’s time and effort–making machines smarter to make people smarter. Really, it’s about making the most of human intelligence.

Robert Munro
September, 2015