Making RAG work with locally running LLM

Course Q&A Retrieval Augmented Generation
,
1

I am trying to implement the course exercises locally using my own Jupyter lab and LLM running on a local server. The course has the following endpoints configured.

  1. Coursera - url = os.path.join(‘https://proxy.dlai.link/coursera_proxy/together’, ‘v1/chat/completions’)
  2. Together - client = Together(api_key = together_api_key)

Model being used is : model: str = “meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo”,

Anyone has tried using the local LLM installation using Ollama or similar. I would appreciate if they could provide the sample code.

2

I’m not a mentor for this course. Does this help?

3

Thank you. I tinkered with this further and am now re-writing the LLM functions (generate_with_single_input etc) that will use ollama instead of Coursera or Together. Will be happy to share it with others if needed.