Repository evaluations - ox/Arxiv-Dive-RAG

Evaluations

Run models against your data

Introducing Evaluations, a powerful feature designed to enable you to effortlessly test and compare a selection of AI models against your datasets.

Whether you're fine-tuning models or evaluating performance metrics, Oxen evaluations simplifies the process, allowing you to quickly and easily run prompts through an entire dataset.

Once you're happy with the results, output the resulting dataset to a new file, another branch, or directly as a new commit.

Generate markdown embeddings

70ef3b23-b9e1-4e4c-978a-dd0c2ff5553e

5 row sample completed

Bessie

1 week ago

Prompt: markdown_text

1 iteration$ 0.0000

text → embeddings

Google/Text Embedding 004

Source:

main

data/arxiv_markdown.parquet

Testing Summarize

59a4ee4a-3162-4ce3-ad3c-f5cfbc8cf2f1

5 row sample completed

Bessie

1 month ago

Prompt: Summarize the following text in a single sentence. {section}

2 iterations 2096 tokens$ 0.0004

text → text

OpenAI/GPT 4o mini

Source:

main

data/arxiv_markdown_chunks.parquet

Generate answer relevance question embeddings

5f4c7930-4cfd-4245-a25e-39ea7db5d329

300 rows completed

Bessie

1 month ago

Prompt: answer_relevance_questions

1 iteration 5563 tokens$ 0.0001

text → embeddings

OpenAI/Text Embedding 3 - Small

Source:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Target:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Generate question embeddings

95d7dfb7-0bf9-40ef-9853-e3f283451051

300 rows completed

Bessie

1 month ago

Prompt: question

2 iterations 6405 tokens$ 0.0001

text → embeddings

OpenAI/Text Embedding 3 - Small

Source:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Target:

main

data/100_questions/answer_relevance_questions_expanded.parquet

Context Relevance - Extract Relevant Sentences

b26c70c9-929f-488a-851b-4f41ed3b7d59

100 rows completed

Bessie

1 month ago

Prompt: Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you're not allowed to make any changes to sentences from given context. Question: {question} Context: {rag_context}

2 iterations 123450 tokens$ 0.0230

text → text

OpenAI/GPT 4o mini

Source:

main

data/100_questions/search_results.parquet

Target:

main

data/100_questions/context_relevance.parquet

Answer Relevance Question Generation

33d672d8-e463-4132-b895-1426d1b3c589

100 rowserror

Bessie

1 month ago

Prompt: Generate 3 questions for the given the answer. Generate the questions in an ordered list: 1. 2. 3. Answer: {answer}

2 iterations 14131 tokens$ 0.0050

text → text

OpenAI/GPT 4o mini

Source:

main

data/100_questions/generated_answers.parquet

Target:

N/A

data/100_questions/answer_relevance_questions.parquet

Statement Faithfulness To Context

a2a8d567-ab5f-42da-9578-0ea3f2daecf9

100 rows completed

Bessie

1 month ago

Prompt: Consider the given context and following statements, then determine whether they are supported by the information present in the context. Provide a brief explanation for each statement before arriving at the final verdict (Yes/No). Provide a final vertict for each statement in order at the end in the given format. Do not deviate from the specified format. Context: {context} Statements: {faithfulness_statements}

3 iterations 38565 tokens$ 0.0166

text → text

OpenAI/GPT 4o mini

Source:

main

data/100_questions/faithfulness_statements.parquet

Target:

main

data/100_questions/faithfulness_statements.parquet

Faithfulness Statements

9246a591-d95b-4781-8eb4-8487c1a7dc63

100 rows completed

Bessie

1 month ago

Prompt: Given a question and an answer, create one or more statements from each sentence in the given answer. The statements should be in an ordered list such as 1. First Statement 2. Second Statement etc... question: {question} answer: {answer}

2 iterations 18912 tokens$ 0.0059

text → text

OpenAI/GPT 4o mini

Source:

main

data/100_questions/generated_answers.parquet

Target:

main

data/100_questions/faithfulness_statements.parquet

5235077a-645e-417d-9294-222971555232

847 rows completed

Bessie

1 month ago

2 iterations 712485 tokens$ 0.1946

text → text

OpenAI/GPT 4o mini

Source:

main

data/ragas/faithfulness.parquet

Target:

main

data/ragas/faithfulness.parquet

Generate statements

a0d21249-16f8-4ae5-a485-8590619cba79

847 rows completed

Bessie

1 month ago

4 iterations 145873 tokens$ 0.0451

text → text

OpenAI/GPT 4o mini

Source:

main

data/questions_answered.parquet

Target:

main

data/ragas/faithfulness.parquet

Answer Questions For Real

d70645d9-7bdd-4ff2-971c-b3b4680446b8

847 rows completed

Bessie

2 months ago

Prompt: Answer the following question as succinctly as possible given the context. If the question cannot be answered given the context, respond with not_answerable. Context: {section} Quetion: {question} Answer:

2 iterations 487127 tokens$ 1.45

text → text

OpenAI/GPT 4o

Source:

main

data/answerable_questions.parquet

Target:

main

data/questions_answered.parquet