Compare - ox/Arxiv-Dive-RAG - ox/Arxiv-Dive-RAG

Comparing changes

Choose two branches to see what’s changed or to start a new merge request.

main

lilian

lilian is 50 commits ahead of main

These branches can be automatically merged

1 file added2 files removed2 files changed

50 Commits

remove description from gradio blocks

Gregcommitted 3 days ago

#5f14b6661d79fed9252479ed6a846926

add scripts to compute answer relevance and context relevance

Gregcommitted 3 days ago

#51dfc5aa79fbe224fe8661d73ff4d622

Updated "answer_relevance_question_embeddings" with text-embedding-3-small

Bessiecommitted 3 days ago

#f7e6526efa2fdcd042f1c78bc9b7d6fc

Generating original question embeddings

Bessiecommitted 3 days ago

#16e6867a8469760541f1d2e4d9c1b9e1

Merge commit bdf96082b9992b6048a684a7c5cbab36 into 270aff5a661ca307145dded7906f6c4d

Bessie Oxingtoncommitted 3 days ago

#232a5aa3034936c50c19a7adf8bd73ef

adding script to compute faithfulness and to extract questions from answer relevance

Bessie Oxingtoncommitted 3 days ago

#270aff5a661ca307145dded7906f6c4d

use data/ragas/section_embeddings.parquet for search

Bessie Oxingtoncommitted 3 days ago

#6e04e0ba8e91179912aae7cb08b99d07

update requirements.txt and where the rag pipeline points

Greg Schoeningercommitted 4 days ago

#d71e496c92d92c6d201db5294b28c837

add links to paper and notion doc

Bessie Oxingtoncommitted 4 days ago

#18903ba4667d70a53fada3b1c8cb7e01

Extracting relevant sentences from the context

Bessiecommitted 4 days ago

#46e1fec55be8afdfe1f661cbe48f41e3

Classifying if statement is faithful to context

Bessiecommitted 4 days ago

#d341f2fe906e4c1d0eacfd3e44f6e224

Generate faithfulness statements from questions and answer pairs.

Bessiecommitted 4 days ago

#f3db6b7ef21656651e964cc52e6c78ac

add section embeddings to ragas

Bessie Oxingtoncommitted 4 days ago

#536e8f994de66881b4b97e31902f972f

move data around in 100_questions

Bessie Oxingtoncommitted 4 days ago

#825ee6d45a5d20822fea387e44da7134

Generating answers from retrieved context

Oxcommitted 4 days ago

#337a8b48eeb71d3d6c1739d2366780a8

Merge commit 15c38078b3ab8bcf0b9326b4c1d80f3e into 5deeec38420f9d39381f70fbdd906b8d

Bessie Oxingtoncommitted 4 days ago

#df692686ecf03b8f0a59abaa57626fef

add search results

Bessie Oxingtoncommitted 4 days ago

#5deeec38420f9d39381f70fbdd906b8d

filter out unanswerable questions from the pipelien

Bessie Oxingtoncommitted 4 days ago

#923126e5b9a08f276e87a43cf7f4945c

add 100 answered questions

Bessie Oxingtoncommitted 4 days ago

#832e56b81210f75c7ca3185fce31568d

add 100 questions

Bessie Oxingtoncommitted 4 days ago

#f1d687cdf5a3c79ea3833c8d46291db3

Merge commit 196e12f90ed312f456dd409f58a47a49 into 1693168f05dc8a6fb13215ec5515685f

Bessie Oxingtoncommitted 4 days ago

#cd48a6a5feccb835f0a842967fa0fa2e

bump README

Bessie Oxingtoncommitted 4 days ago

#1693168f05dc8a6fb13215ec5515685f

add / to image paths

Bessie Oxingtoncommitted 5 days ago

#ce45fc3e75d07985066c3fe2f183c70

add flag to run demo without context and more images to readme

Bessie Oxingtoncommitted 5 days ago

#cce95e24414e6e1fe50b6cc357dce438

Merge commit a296eef936aa0b0ed681399002f2831e into 345341eb12e08788708f81c9895d74bc

Bessie Oxingtoncommitted 5 days ago

#78307ca4035bfaeb547c3928483b7153

add test.txt

Bessie Oxingtoncommitted 2 weeks ago

#345341eb12e08788708f81c9895d74bc

remove oxen columns from committed data/answerable_questions.parquet

Bessie Oxingtoncommitted 3 weeks ago

#a2022f5e9380532617fb4f255310d25d

filter down to answerable questions

Bessie Oxingtoncommitted 3 weeks ago

#2a630b989a83016f787969719c0de80b

Filtering questions that are not able to be answered by the context.

Oxcommitted 3 weeks ago

#819f370dba5e8704d302eb72a7de15fd

add generated data/questions.parquet

Bessie Oxingtoncommitted 4 weeks ago

#28fe82b1e249cd55806006873acba159

fix parsing of image tags in markdown docs

Bessie Oxingtoncommitted 4 weeks ago

#88b2a05d377660d2833ac32efd09a2b

update generate_questions script to take in new fields

Bessie Oxingtoncommitted 4 weeks ago

#42fe37541ff1f96e3269d4ee064e53cf

extract images from pdfs and consolidate title and abstract extraction

Bessie Oxingtoncommitted 4 weeks ago

#45e2193ebf9eb42905424a0844db7ab

add filter for if the chunks are interesting

Bessie Oxingtoncommitted 4 weeks ago

#bab824de2330719ec571e714a1b55f23

update readme

Bessie Oxingtoncommitted 4 weeks ago

#d1fc48344755789f88171b099defa359

add script to classify if chunks are interesting

Bessie Oxingtoncommitted 4 weeks ago

#c4c2890544c48ff5f295163475775a20

add script to generate questions based on the arxiv papers

Bessie Oxingtoncommitted 1 month ago

#69d2888d0fb2bda5d54513f992d3f122

app now collects data from 👍/👎

Bessie Oxingtoncommitted 1 month ago

#529b03e8d1c30b9422a45c0e1b54e255

Updated "title_embedding" with text-embedding-3-small

Bessiecommitted 1 month ago

#a8fc704ed33eff6f0fd060116c3d0ff9

Computing embeddings for paper sections

Bessiecommitted 1 month ago

#35d1174ddf91aa568d112f01479cc627

adding app

Bessie Oxingtoncommitted 1 month ago

#c85bf04a8e4f5f9bd29d316624828632

adding initial data

Bessie Oxingtoncommitted 1 month ago

#731ac85969ba1c92b3e296af95d2a4b6

Answering the questions with GPT-4o

Bessiecommitted 3 weeks ago

#a296eef936aa0b0ed681399002f2831e

Computing how faithful the context is to the statements about the answer

Bessiecommitted 4 days ago

#196e12f90ed312f456dd409f58a47a49

Generating statements in order to understand faithfulness.

Bessiecommitted 4 days ago

#191817f619936f4ed7f19c88b477b833

adding questions to data/ragas

Bessiecommitted 4 days ago

#81e66ec300bd332b36475e97b96fc7a6

Generating Answer Relevance Questions

Oxcommitted 4 days ago

#15c38078b3ab8bcf0b9326b4c1d80f3e

delete is_answerable column

oxcommitted 4 days ago

#e8d6cefdb7ec3f8002f7f5f4ca08d05e

switch back to prod for the app/rag.py

Bessie Oxingtoncommitted 4 days ago

#fc956e157449c2246582b162e1f4959

add link to pdf path

oxcommitted 3 days ago

#bdf96082b9992b6048a684a7c5cbab36