Skip to content
dotdock

Legal_RAG_Hallucinations

2024

Publication cover

Open publication workspace · Sign in to read the full PDF.

AI-generated summary

1) This study evaluates leading AI-driven legal research tools and demonstrates that despite vendor claims, retrieval-augmented generation (RAG) systems still hallucinate significantly.

2)
- Evaluates the accuracy and groundedness of Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI, and GPT-4 using a preregistered dataset of over 200 legal queries.
- Establishes a formal evaluation framework that decomposes legal hallucinations into correctness and groundedness dimensions.
- Categorizes primary RAG failure modes in a legal context, including naive retrieval, inapplicable authority, reasoning errors, and sycophancy.

3) Tags: Legal AI, Retrieval-Augmented Generation, Hallucinations, Legal Research, Large Language Models

Check the original publication for accuracy and context.