Repository evaluations - ox/Arxiv-Dive-RAG

Evaluations

Run models against your data

Introducing Evaluations, a powerful feature designed to enable you to effortlessly test and compare a selection of AI models against your datasets.

Whether you're fine-tuning models or evaluating performance metrics, Oxen evaluations simplifies the process, allowing you to quickly and easily run prompts through an entire dataset.

Once you're happy with the results, output the resulting dataset to a new file, another branch, or directly as a new commit.

Generate answer relevance question embeddings

5f4c7930-4cfd-4245-a25e-39ea7db5d329

300 rows completed

Bessie

3 days ago

Prompt: answer_relevance_questions

1 iteration 5563 tokens$ 0.0001

text → embeddings

OpenAI/Text Embedding 3 - Small

Source:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Target:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Generate question embeddings

95d7dfb7-0bf9-40ef-9853-e3f283451051

300 rows completed

Bessie

3 days ago

Prompt: question

2 iterations 6405 tokens$ 0.0001

text → embeddings

OpenAI/Text Embedding 3 - Small

Source:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Target:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Context Relevance - Extract Relevant Sentences

b26c70c9-929f-488a-851b-4f41ed3b7d59

100 rows completed

Bessie

4 days ago

Prompt: Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you're not allowed to make any changes to sentences from given context. Question: {question} Context: {rag_context}

2 iterations 123450 tokens$ 0.0230

text → text

OpenAI/GPT-4o mini

Source:

main

data/100_questions/search_results.parquet

Target:

main

data/100_questions/context_relevance.parquet

Answer Relevance Question Generation

33d672d8-e463-4132-b895-1426d1b3c589

100 rowserror

Bessie

4 days ago

Prompt: Generate 3 questions for the given the answer. Generate the questions in an ordered list: 1. 2. 3. Answer: {answer}

2 iterations 14131 tokens$ 0.0050

text → text

OpenAI/GPT-4o mini

Source:

main

data/100_questions/generated_answers.parquet

Target:

N/A

data/100_questions/answer_relevance_questions.parquet

Statement Faithfulness To Context

a2a8d567-ab5f-42da-9578-0ea3f2daecf9

100 rows completed

Bessie

4 days ago

Prompt: Consider the given context and following statements, then determine whether they are supported by the information present in the context. Provide a brief explanation for each statement before arriving at the final verdict (Yes/No). Provide a final vertict for each statement in order at the end in the given format. Do not deviate from the specified format. Context: {context} Statements: {faithfulness_statements}

3 iterations 38565 tokens$ 0.0166

text → text

OpenAI/GPT-4o mini

Source:

main

data/100_questions/faithfulness_statements.parquet

Target:

main

data/100_questions/faithfulness_statements.parquet

Faithfulness Statements

9246a591-d95b-4781-8eb4-8487c1a7dc63

100 rows completed

Bessie

4 days ago

Prompt: Given a question and an answer, create one or more statements from each sentence in the given answer. The statements should be in an ordered list such as 1. First Statement 2. Second Statement etc... question: {question} answer: {answer}

2 iterations 18912 tokens$ 0.0059

text → text

OpenAI/GPT-4o mini

Source:

main

data/100_questions/generated_answers.parquet

Target:

main

data/100_questions/faithfulness_statements.parquet

5235077a-645e-417d-9294-222971555232

847 rows completed

Bessie

4 days ago

2 iterations 712485 tokens$ 0.1946

text → text

OpenAI/GPT-4o mini

Source:

main

data/ragas/faithfulness.parquet

Target:

main

data/ragas/faithfulness.parquet

Generate statements

a0d21249-16f8-4ae5-a485-8590619cba79

847 rows completed

Bessie

4 days ago

4 iterations 145873 tokens$ 0.0451

text → text

OpenAI/GPT-4o mini

Source:

main

data/questions_answered.parquet

Target:

main

data/ragas/faithfulness.parquet

Answer Questions For Real

d70645d9-7bdd-4ff2-971c-b3b4680446b8

847 rows completed

Bessie

3 weeks ago

Prompt: Answer the following question as succinctly as possible given the context. If the question cannot be answered given the context, respond with not_answerable. Context: {section} Quetion: {question} Answer:

2 iterations 487127 tokens$ 1.45

text → text

OpenAI/GPT-4o

Source:

main

data/answerable_questions.parquet

Target:

main

data/questions_answered.parquet