Building and Evaluating Advanced RAG Applications

Short Course Q&A Building and Evaluating Advanced RAG Applications
1

For the first notebook: Lesson 1: Advanced RAG Pipeline

There is a an import utils, bringing in a package with various functions such as get_openai_api_key() and get_prebuilt_trulens_recorder

Since this is not a standard python package, how do I replicate this functionality in my own environment (off platform)? Can this package be pip-installed from somewhere (not by “pip install utils” which apparently installs a totally different package).

Can any transparency be offered regarding this? Or is all this part in a black box?

Thanks,
Josh

2

Never mind; I guess it’s in the utils.py which is a local file on the platform. Sorry I’m not not familiar with this cloud platform.

3

Those are the custom functions written by the team and only work on the platform. The alternatives to those functions are as below:

OpenAI API Key setup: https://platform.openai.com/docs/quickstart?context=python

Trulens Recorder: You can use TruLlama instead of their custom function. However, TruLlama is not providing the required metrics such as Context and Answer Relevance, and Groundedness.
Check out TruLlama implementation here: https://github.com/truera/trulens/blob/releases/rc-trulens-eval-0.16.0/trulens_eval/examples/quickstart/llama_index_quickstart.ipynb

Hope this helps!

4

Ah OK,
Are you saying the functions in utils.py don’t work off platform? (My cursory glance suggests that they would.) Anyway, OpenAI API key setup is basically just a way to assign a password within the notebook, without it actually saving it in the notebook. ( import getpass is another alternative)

Regarding the more substantial TruLens package: So basically are you saying this tutorial is teaching how to evaluate RAGs using a proprietary tool (Trulens) as one of its key components? Or is TruLens open source? If it is, why couldn’t it be run off platform?

Sorry for so many questions. Just trying to understand how I can deploy what I learn in this tutorial after/outside the tutorial itself.

5

On my PC (Ubuntu/WSL2, trulens-eval 0.18.2 installed with pip), the notebooks run well & the utils.py files execute without errors.

6

I can’t find it.

so i just read utils.py file

# Add your utilities or helper functions to this file.

import os
from dotenv import load_dotenv, find_dotenv

# these expect to find a .env file at the directory above the lesson.                                                                                                                     # the format for that file is (without the comment)                                                                                                                                       #API_KEYNAME=AStringThatIsTheLongAPIKeyFromSomeService
def load_env():
    _ = load_dotenv(find_dotenv())

def get_openai_api_key():
    load_env()
    openai_api_key = os.getenv("OPENAI_API_KEY")
    return openai_api_key

def get_serper_api_key():
    load_env()
    openai_api_key = os.getenv("SERPER_API_KEY")
    return openai_api_key


# break line every 80 characters if line is longer than 80 characters
# don't break in the middle of a word
def pretty_print_result(result):
  parsed_result = []
  for line in result.split('\n'):
      if len(line) > 80:
          words = line.split(' ')
          new_line = ''
          for word in words:
              if len(new_line) + len(word) + 1 > 80:
                  parsed_result.append(new_line)
                  new_line = word
              else:
                  if new_line == '':
                      new_line = word
                  else:
                      new_line += ' ' + word
          parsed_result.append(new_line)
      else:
          parsed_result.append(line)
  return "\n".join(parsed_result)