Implementing Web-Retrieval with RAG AI LLMS.
Preface Note: (This is a part 2 to my original blog
and I’m hella tired)
As per my last post we talked about feeding an AI the book “Problem of Pain” written by CS Lewis. The issue with the last guide is that it is very static. If someone wants the AI to feed off a dynamic database then there isn’t much that can be done unless one re-codes their stuff and then re-deploys.
A solution behind this is to let the AI have the ability to parse certain websites which has the aforementioned data. For this tutorial we’ll examine the book, “The Four Loves” by C.S Lewis as well.
To set up a remote database one can create an AWS server but honestly that’s absolutely expensive and there’s an easier solution. Using this guy’s absolutely genius github repository
We can fork the repository, connect our Obsidian Files, then deploy the website to vercel. Here is my personal website I use for a lot of different purposes https://josephsliu.vercel.app/home. You can see if you go to the Four Loves page there’s the whole transcript of the “Four Loves” by CS Lewis in there.
We can render the data in through the following method
from langchain_community.document_loaders import WebBaseLoader
import bs4
loader = WebBaseLoader("https://josephsliu.vercel.app/attachments/Four%20Loves")
loader.load()This will load in the entire dataset. From here I have a more detailed explanation in my other post.
But you can just do the following code
import getpass
import dotenv
import os
dotenv.load_dotenv()
os.environ["Open_AI_API_Key"] = getpass.getpass()
import bs4
from langchain import hub
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(loader.load())
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())There is a small adjustment from my previous code, in this code we’ve adjusted it so that it references the sources as well.
# Retrieve and generate using the relevant snippets of the blog.
retriever = vectorstore.as_retriever()
prompt = hub.pull("rlm/rag-prompt")
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
rag_chain_from_docs = (
RunnablePassthrough.assign(context=(lambda x: format_docs(x["context"])))
| prompt
| llm
| StrOutputParser()
)
rag_chain_with_source = RunnableParallel(
{"context": retriever, "question": RunnablePassthrough()}
).assign(answer=rag_chain_from_docs)rag_chain_with_source.invoke("How does one know if one loves something or is obsessed by it?")Note: The sources are on the top and the answer is all the way at the very bottom
Returns:
{‘context’: [Document(page_content=’But in the Appreciative pleasures, even at their lowest, and more and more as they grow up into the full appreciation of all beauty, we get something that we can hardly help calling love and hardly help calling disinterested, towards the object itself. It is the feeling which would make a man unwilling to deface a great picture even if he were the last man left alive and himself about to die; which makes us glad of unspoiled forests that we shall never see; which makes us anxious that the garden or bean-field should continue to exist. We do not merely like the things; we pronounce them, in a momentarily God-like sense, “very good.” And now our principle of starting at the lowest without which “the highest does not stand” begins to pay a dividend. It has revealed to me a deficiency in our previous classification of the loves into those of Need and those of Gift. There is a third element in {32} Likings and Loves for the Sub-Human love, no less important than these, which is’, metadata={‘description’: ‘S.LEWIS THE FOUR LOVES That our affections kill us not, nor dye. DONNE CONTENTS I Introduction p. 11 II Likings and Loves for the Sub-human p. 25 HI Affect’, ‘language’: ‘en’, ‘source’: ‘https://josephsliu.vercel.app/attachments/Four%20Loves', ‘title’: ‘Four Loves | Linked Blog Starter’}),
Document(page_content=’in our previous classification of the loves into those of Need and those of Gift. There is a third element in {32} Likings and Loves for the Sub-Human love, no less important than these, which is foreshadowed by our Appreciative pleasures. This judgment that the object is very good, this attention (almost homage) offered to it as a kind of debt, this wish that it should be and should continue being what it is even if we were never to enjoy it, can go out not only to things but to persons. When it is offered to a woman we call it admiration; when to a man, heroworship; when to God, worship simply. Need-love cries to God from our poverty; Gift-love longs to serve, or even to suffer for, God; Appreciative love says: “We give thanks to thee for thy great glory.” Need-love says of a woman “I cannot live without her”; Gift-love longs to give her happiness, comfort, protection if possible, wealth; Appreciative love gazes and holds its breath and is silent, rejoices that such a wonder should’, metadata={‘description’: ‘S.LEWIS THE FOUR LOVES That our affections kill us not, nor dye. DONNE CONTENTS I Introduction p. 11 II Likings and Loves for the Sub-human p. 25 HI Affect’, ‘language’: ‘en’, ‘source’: ‘https://josephsliu.vercel.app/attachments/Four%20Loves', ‘title’: ‘Four Loves | Linked Blog Starter’}),
Document(page_content=’what we call “the love of nature” is a permanent and serious sentiment. I mean here that love of nature which cannot be adequately classified simply as an instance of our love for beauty. Of course many natural objects trees, flowers and animals are beautiful But the nature-lovers whom I have in mind are not very much concerned with individual beautiful objects of that sort. The man who is distracts them. An enthusiastic botanist is for them a dreadful companion on a ramble. He is always stopping to draw their attention to particulars. Nor are they looking for “views” or landscapes. Wordsworth, their spokesman, strongly deprecates this. It leads to “a comparison of scene with scene,” makes you “pamper” yourself with “meagre novelties of colour and proportion.” While you are busying yourself with this critical and discriminating activity you lose what really matters the “moods of time and season,” the “spirit” of the place. And of course Wordsworth is right. That is why, if you love’, metadata={‘description’: ‘S.LEWIS THE FOUR LOVES That our affections kill us not, nor dye. DONNE CONTENTS I Introduction p. 11 II Likings and Loves for the Sub-human p. 25 HI Affect’, ‘language’: ‘en’, ‘source’: ‘https://josephsliu.vercel.app/attachments/Four%20Loves', ‘title’: ‘Four Loves | Linked Blog Starter’}),
Document(page_content=’. For us at any rate the importance of the two sorts 27} THE FOUR LOVES of pleasure lies in the extent to which they foreshadow characteristics in our \’loves” (properly so called). The thirsty man who has just drunk off a tumbler of water may say, “By Jove, I wanted that.” So may the alcoholic who has just had his “nip.” The man who passes the sweet-peas in his morning walk is more likely to say, “How lovely the smell is.” The connoisseur after his first sip of the famous claret, may similarly say, “This is a great wine.” When Need-pleasures are in question we tend to make statements about ourselves in the past tense; when Appreciative pleasures are in question we tend to make statements about the object in the present tense. It is easy to see why. Shakespeare has described the satisfaction of a tyrannous lust as something Past reason hunted and, no sooner had, Past reason hated. But the most innocent and necessary of Needpleasures have about them something of the same character’, metadata={‘description’: ‘S.LEWIS THE FOUR LOVES That our affections kill us not, nor dye. DONNE CONTENTS I Introduction p. 11 II Likings and Loves for the Sub-human p. 25 HI Affect’, ‘language’: ‘en’, ‘source’: ‘https://josephsliu.vercel.app/attachments/Four%20Loves', ‘title’: ‘Four Loves | Linked Blog Starter’})],
‘question’: ‘How does one know if one loves something or is obsessed by it?’,
‘answer’: ‘One can know if they love something by feeling a sense of admiration, appreciation, and a wish for the object to continue existing even if they were never to enjoy it. This feeling of love is disinterested and goes beyond mere liking, involving a judgment that the object is “very good.” Appreciative love can extend not only to things but also to persons, reflecting a deep sense of gratitude and admiration.’}
