Creating a Better Youtube Recommendation System using Deep Neural Networks

YouTube is one of the largest pillars of the modern internet experience. If you want to find a video, or a song, or just waste a bit of your time away, YouTube is the place to do it.As such, it’s recommendation system is one of the largest and most sophisticated in existence. I recently had the pleasure of going through a paper on the improvements made to their recommendation system by deep learning. It is explained at a somewhat high level making it a bit unacessable to the standard crowd. As I believe it was a very well done paper that will undoubtably help me better navigate my own projects in the future, I would like to try sharing it with a larger audience by explaining some of the logic and process in more layman’s terms.

YouTube’s robust and effective recommendation system is what keeps so many users engaged on the platform where others video mediums fall short. However with major popularity comes major challenges for that system. Firstly, the massive scale. Solutions that would work perfectly fine on smaller platforms don’t when brought up to YouTube’s level. Secondly, keeping things fresh. YouTube has been around since 2005 and has the videos to prove it. According to Google it’s currently it’s sitting at about 150 Billion. It would not make sense for a recommendation system to pull from those early years as often as the 500 hours of vids uploaded every minute. And lastly, Noise. In this case noise is that tons of data that is not organized in a way that can be worked with, which is inevitable with the sheer amount of videos uploaded.

The solution? Deep learning. Deep learning is a machine learning method based on on artifical neural networks, which mimic the way humans learn.
It is often referred to as a “black box” method, meaning that it is not very transparent or easily understood. You give it problems, it spits out answers, and you smile and thank the machine without asking too many questions. Though this is not acceptable for ethical reasons in some fields, is is perfectably suitable for a recommendation system. It helps that the performance is fantastic if done properly.

The paper did a great job of explaining the system with a good level of detail while not overloading the reader with extraneous information. It gives a runthrough of the main structure of the system as well as explaining the logic and process behind it’s constiuent parts. However, I believe it was geared towards a technical audience with at least some experience with machine learning models. Even after having the lessons fresh in my head, I had to really pretty heavily on my friend Boogle(‘Alias’). What I will try to do is bring the technical stuff even farther down to the level of mere mortals and hope I do the original paper justice.

Below I’ve posted a visualization explaining the system borrow from the paper, as I could not have made a better one myself.

It is comprised of two neural networks, one for candidate generation and one for ranking. I’ll refer to them as just the candidates and ranker from now on. The user’s history is fed into the candidates which it uses to gather a subset of videos that it deems relevant to the user from the YouTube library.

That subset is then fed into ranking which rates the videos using a much larger set of factors. These inlude the user history and context as well as many more details on the video, user, and what the system has learned from other users in the past to assign relevance. I would like to go slightly deeper into the makeup of these systems as I think of simpler explanation for them.

I have plans for building recommendation systems in the future, which makes this paper a great find. Though not everything here is applicable to what I will be doing, the logic behind each step of the process and the solutions found to the various issues of building something at this scale is invaluable.

This should not matter only to me but to any company that relies on providing relevant recommendations to users, whether that be products or services. The scale at which these systems must work now completely changes the approach you need to build them properly. Recommendation systems not only help a company introduce people what they are most likely to pay for, but it helps users by saving them time looking for what they are actually interested in.

CREDITS

Related posts:

Many people are concerned that rising costs may eat into their savings as inflation reaches its greatest level in three decades. As the cost of everything from microchips to bed linens rises, the…

PyCon Hong Kong 2018 (23–24 Nov) is now calling for proposals, CFP will be closed at the end of this month (31 Aug). Welcome to submit proposals to PyCon HK this year.

I had stopped posting about my journey with COVID-19 because I kept on having the same symptoms recur over and over after them being gone for a few days. Let’s face it, it’s not very exciting to read…