Recommandation system with streaming data?

Hello and greetings ,
say i want to build a job recommender search engine for the most recent jobs based on the user skills .
Might someone give me information how can i deal with stream data, which the model could keep learning (online learning) ??

You may want to google about how to build a data pipeline. This topic isn’t covered in MLS, so I am moving it to the General discussion category for a broader audience who may discuss with you on your question or any of your findings.

Raymond

1 Like

Thank you for your help .

You are welcome @Rhayem_Bannouri. Your question indeed covers quite a lot of thing - could be the whole architecture of your modeling service. I suggest you to audit this specialization to see if something can catch your attention, adjust your question to focus on a specific part of your online learning service, or share with us how your current architecture can’t handle online learning.

Raymond

1 Like

Thank you…,
Indeed i just wondering what are the technologies used for building recommander system based on streaming data let’s say spark framework Is it a suitable tool for handling data used for recommender system?

My goal Is to keep my recommander system up to date for most recent jobs ,that’s why i need a streaming data.
Does this idea seem reasonable?

Keep streaming in new data, preprocessing them and storing them properly is ofcourse a reasonable idea. The timing to train the next version of your model and whether to adopt it are two of the other questions to consider. Spark can be a suitable framework - good for building pipeline, has MLlib, can run on GPU cluster, can work with tensorflow, and has matrix factorization packages in particular, but you may also need to consult your organization’s tech support for what’s available and supported. If you are in the proof-of-concept stage, you may try to build a minimum variable service with Spark and measure the performance at different scaling levels, to figure out as many issues as you can, before making any decision. This is not a simple yes/no question so I may have covered more than you are actually asking - though this coverage is still far from being enough.

1 Like

Okay , i will go with spark .
Thank you for you Time and consideration.
Best regards.:heart:

Try spark. Make notes of issues encountered, and take them into consideration when you decide whether you will use spark or not.

1 Like