Recommendation systems have impacted or even redefined our lives in many ways. One example of this impact is how our online shopping experience is being redefined. As we browse through products, the Recommendation system offer recommendations of products we might be interested in. Regardless of the perspective — business or consumer, Recommendation systems have been immensely beneficial. And big data is the driving force behind Recommendation systems. A typical Recommendation system cannot do its job without sufficient data and big data supplies plenty of user data such as past purchases, browsing history, and feedback for the Recommendation systems to provide relevant and effective recommendations. Keliais žodžiais, even the most advanced Recommenders cannot be effective without big data.
How does a Recommendation system work?
A Recommendation system works in well-defined, logical phases which are data collection, ratings, and filtering. These phases are described below.
Data collection
Let us assume that a user of Amazon website is browsing books and reading the details. Each time the reader clicks on a link, an event such as an Ajax event could be fired. The event type could vary depending on the technology used. The event then could make an entry into a database which usually is a NoSQL database. The entry is technical in content but in layman’s language could read something like “User A clicked Product Z details once”. That is how user details get captured and stored for future recommendations.
How does the Recommendation system capture the details? If the user has logged in, then the details are extracted either from an http session or from the system cookies. In case the Recommendation system depends on system cookies, then the data is available only till the time the user is using the same terminal. Events are fired almost in every case — a user liking a Product or adding it to a cart and purchasing it. So that is how user details are stored. But that is just one part of what Recommenders do.
The following paragraphs show how Amazon offers its product recommendations to a user who is browsing for books:
- As shown by the image below, when a user searched for the book Harry Potter and the Philosopher’s Stone, several recommendations were given.
Recommendation System
- In another example, a customer who searched Amazon for Canon EOS 1200D 18MP Digital SLR Camera (Black) was interestingly given several recommendations on camera accessories.
Recommendation System
Ratings
Ratings are important in the sense that they tell you what a user feels about a product. User’s feelings about a product can be reflected to an extent in the actions he or she takes such as likes, adding to shopping cart, purchasing or just clicking. Recommendation systems can assign implicit ratings based on user actions. The maximum rating is 5. For example, purchasing can be assigned a rating of 4, likes can get 3, clicking can get 2 and so on. Recommendation systems can also take into account ratings and feedback users provide.
Filtering
Filtering means filtering products based on ratings and other user data. Recommendation systems use three types of filtering: collaborative, user-based and a hybrid approach. In collaborative filtering, a comparison of users’ choices is done and recommendations given. For example, if user X likes products A, B, C, and D and user Y likes products A, B, C, D and E, the it is likely that user X will be recommended product E because there are a lot of similarities between users X and Y as far as choice of products is concerned.
Several reputed brands such as Facebook, Twitter, LinkedIn, Amazon, Google News, Spotify and Last.fm use this model to provide effective and relevant recommendations. In user-based filtering, the user’s browsing history, likes, purchases and ratings are taken into account before providing recommendations. This model is used by many reputed brands such as IMDB, Rotten Tomatoes and Pandora. Many companies also use a hybrid approach. Netflix is known to use a hybrid approach.
Role of big data
As stated earlier, big data drives what Recommenders do primarily. Recommenders cannot do a thing without the constant supply of data. However, the role of big data goes beyond just data. It is clear that the above operations require a high-capacity CPU which can work for hours. To realize this, Hadoop can be used. To reduce the manual work needed to code, identify right algorithms, similarity methods and other tasks, Mahout could be used.
Mahout is a library that comprises machine learning algorithms. It provides a set of options to choose recommendation algorithm, choosing n-nearest neighbors and similarity methods. Though it is a standard Java class, it operates purely on Hadoop.
To make your tasks even easier, you can use a tool known as PredictionIO which bundles both Mahout and Hadoop and what more, it provides a nice user interface.
So, the role of big data can be summed in providing meaningful, actionable data fast and providing necessary setup to quickly process the data. It is obvious that traditional technologies are not meant to process such large volumes of data so quickly. So, it will not suffice to just have big data in order to provide strong recommendations.
The Amazon use case
How Amazon uses the powerful duo of big data and Recommendation System is worth a study. Amazon has been in certain ways a pioneer of ecommerce but more important than that accolade is how it is driving its revenue up by providing more and more effective recommendations.
Buying can be both impulsive and planned and Amazon is smartly tapping into the impulsive shopper’s mind by providing relevant and useful product recommendations. For that, it is relentlessly working on making its Recommendation engine more powerful. Shopping has a connection with psychology. Shoppers buy for instant gratification, instant mood uplift, social esteem and reasons not even known to them clearly.
Amazon is smart enough to take these factors into account. And now, it is working on a system called predictive dispatch which means that its Recommendation engine can predict what the customer is going to buy and make arrangements for a speedy dispatch.
What makes Amazon’s achievements more creditable is the fact that unlike Facebook — which also relies a lot on big data — which knows a lot of details about its subscribers, all Amazon knows about its customers are the spending patterns.
Amazon has been cashing on this knowledge smartly in an attempt to get more out of your pockets. It is a difficult job to analyze spending patterns, likes, product preferences and provide effective recommendations just on that basis. And now, Amazon is trying to make available its tools and technologies that use big data and Recommendation systems so effectively for sale to other corporations that use big data. So, Amazon’s product ads will start to appear more frequently on other websites as well and that is going to drive up sales.
The following image shows how big companies have been using the power of big data and Recommendation engines.
Recommendations
Limits of Recommendation systems
For all their efficiencies, Recommendation Systems are not a full proof system. Recommenders have been known to suffer from the following limitations:
- Recommenders depend totally on data and their hirers must constantly supply them with large volumes of data. That is why; smaller firms are more disadvantaged then the bigger firms such as Google and Amazon.
- Recommenders may find it difficult to exactly identify user choice patterns if the user preferences tend to vary quickly, as in fashion. Recommenders depend a lot on historic data but that may not be suitable for certain product niches.
- Recommenders face problems with unpredictable items. For example, there are certain movie types that evoke extreme reactions such as love or hate. It is extremely difficult to provide recommendations for such items.
Summary
While big data and Recommendation engines have already proved an extremely useful combination for big corporations, it raises a question of whether companies with smaller budgets can afford such investments. It is encouraging for such companies that big data tools and technologies are relatively more affordable. Product recommendations are extremely important to provide a good user experience from the customer’s viewpoint. Also, from the company’s viewpoint, it takes into account unknown factors that can make a customer buy products which might seem unlikely. As the above image shows, the power of Recommenders is getting bigger.
