Haystack 2.1.0

Highlights

📊 New Evaluator Components

Haystack introduces new components for both with model-based, and statistical evaluation: AnswerExactMatchEvaluator, ContextRelevanceEvaluator, DocumentMAPEvaluator, DocumentMRREvaluator, DocumentRecallEvaluator, FaithfulnessEvaluator, LLMEvaluator, SASEvaluator

Here’s an example of how to use DocumentMAPEvaluator to evaluate retrieved documents and calculate mean average precision score:

from haystack import Document
from haystack.components.evaluators import DocumentMAPEvaluator

evaluator = DocumentMAPEvaluator()
result = evaluator.run(
    ground_truth_documents=[
        [Document(content="France")],
        [Document(content="9th century"), Document(content="9th")],
    ],
    retrieved_documents=[
        [Document(content="France")],
        [Document(content="9th century"), Document(content="10th century"), Document(content="9th")],
    ],
)

result["individual_scores"]
>> [1.0, 0.8333333333333333]
result["score"]
>> 0 .9166666666666666

To learn more about evaluating RAG pipelines both with model-based, and statistical metrics available in the Haystack, check out Tutorial: Evaluating RAG Pipelines.

🕸️ Support For Sparse Embeddings

Haystack offers robust support for Sparse Embedding Retrieval techniques, including SPLADE. Here’s how to create a simple retrieval Pipeline with sparse embeddings:

from haystack import Pipeline
from haystack_integrations.components.retrievers.qdrant import QdrantSparseEmbeddingRetriever
from haystack_integrations.components.embedders.fastembed import FastembedSparseTextEmbedder

sparse_text_embedder = FastembedSparseTextEmbedder(model="prithvida/Splade_PP_en_v1")
sparse_retriever = QdrantSparseEmbeddingRetriever(document_store=document_store)

query_pipeline = Pipeline()
query_pipeline.add_component("sparse_text_embedder", sparse_text_embedder)
query_pipeline.add_component("sparse_retriever", sparse_retriever)

query_pipeline.connect("sparse_text_embedder.sparse_embedding", "sparse_retriever.query_sparse_embedding")

Learn more about this topic in our documentation on Sparse Embedding-based Retrievers Start building with our new cookbook: 🧑‍🍳 Sparse Embedding Retrieval using Qdrant and FastEmbed.

🧐 Inspect Component Outputs

As of 2.1.0, you can now inspect each component output after running a pipeline. Provide component names with include_outputs_from key to pipeline.run:

pipe.run(data, include_outputs_from={"prompt_builder", "llm", "retriever"})

And the pipeline output should look like this:

{'llm': {'replies': ['The Rhodes Statue was described as being built with iron tie bars to which brass plates were fixed to form the skin. It stood on a 15-meter-high white marble pedestal near the Rhodes harbor entrance. The statue itself was about 70 cubits, or 32 meters, tall.'],
  'meta': [{'model': 'gpt-3.5-turbo-0125',
    ...
    'usage': {'completion_tokens': 57,
     'prompt_tokens': 446,
     'total_tokens': 503}}]},
 'retriever': {'documents': [Document(id=a3ee3a9a55b47ff651ae11dc56d84d2b6f8d931b795bd866c14eacfa56000965, content: 'Within it, too, are to be seen large masses of rock, by the weight of which the artist steadied it w...', meta: {'url': 'https://en.wikipedia.org/wiki/Colossus_of_Rhodes', '_split_id': 9}, score: 0.648961685430463),...]},
 'prompt_builder': {'prompt': "\nGiven the following information, answer the question.\n\nContext:\n\n    Within it, too, are to be seen large masses of rock, by the weight of which the artist steadied it while...
 ... levels during construction.\n\n\n\nQuestion: What does Rhodes Statue look like?\nAnswer:"}}

🚀 New Features

  • Add several new Evaluation components, i.e:

    • AnswerExactMatchEvaluator
    • ContextRelevanceEvaluator
    • DocumentMAPEvaluator
    • DocumentMRREvaluator
    • DocumentRecallEvaluator
    • FaithfulnessEvaluator
    • LLMEvaluator
    • SASEvaluator

  • Introduce a new SparseEmbedding class that can store a sparse vector representation of a document. It will be instrumental in supporting sparse embedding retrieval with the subsequent introduction of sparse embedders and sparse embedding retrievers.

  • Added a SentenceTransformersDiversityRanker. The diversity ranker orders documents to maximize their overall diversity. The ranker leverages sentence-transformer models to calculate semantic embeddings for each document and the query.

  • Introduced new HuggingFace API components, namely:

    • HuggingFaceAPIChatGenerator, which will replace the HuggingFaceTGIChatGenerator in the future.
    • HuggingFaceAPIDocumentEmbedder, which will replace the HuggingFaceTEIDocumentEmbedder in the future.
    • HuggingFaceAPIGenerator, which will replace the HuggingFaceTGIGenerator in the future.
    • HuggingFaceAPITextEmbedder, which will replace the HuggingFaceTEITextEmbedder in the future.
    • These components support different Hugging Face APIs:
      • free Serverless Inference API
      • paid Inference Endpoints
      • self-hosted Text Generation Inference

⚡️ Enhancement Notes

  • Compatibility with huggingface_hub>=0.22.0 for HuggingFaceTGIGenerator and HuggingFaceTGIChatGenerator components.

  • Adds truncate and normalize parameters to HuggingFaceTEITextEmbedder and HuggingFaceTEITextEmbedder to allow truncation and normalization of embeddings.

  • Adds trust_remote_code parameter to SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder for allowing custom models and scripts.

  • Adds streaming_callback parameter to HuggingFaceLocalGenerator, allowing users to handle streaming responses.

  • Adds a ZeroShotTextRouter that uses an NLI model from HuggingFace to classify texts based on a set of provided labels and routes them based on the label they were classified with.

  • Adds dimensions parameter to Azure OpenAI Embedders (AzureOpenAITextEmbedder and AzureOpenAIDocumentEmbedder) to fully support new embedding models like text-embedding-3-small, text-embedding-3-large and upcoming ones

  • Now the DocumentSplitter adds the page_number field to the metadata of all output documents to keep track of the page of the original document it belongs to.

  • Allows users to customise text extraction from PDF files. This is particularly useful for PDFs with unusual layouts, such as multiple text columns. For instance, users can configure the object to retain the reading order.

  • Enhanced PromptBuilder to specify and enforce required variables in prompt templates.

  • Set max_new_tokens default to 512 in HuggingFace generators.

  • Enhanced the AzureOCRDocumentConverter to include advanced handling of tables and text. Features such as extracting preceding and following context for tables, merging multiple column headers, and enabling single-column page layout for text have been introduced. This update furthers the flexibility and accuracy of document conversion within complex layouts.

  • Enhanced DynamicChatPromptBuilder’s capabilities by allowing all user and system messages to be templated with provided variables. This update ensures a more versatile and dynamic templating process, making chat prompt generation more efficient and customised to user needs.

  • Improved HTML content extraction by attempting to use multiple extractors in order of priority until successful. An additional try_others parameter in HTMLToDocument, True by default, determines whether subsequent extractors are used after a failure. This enhancement decreases extraction failures, ensuring more dependable content retrieval.

  • Enhanced FileTypeRouter with regex pattern support for MIME types. This powerful addition allows for more granular control and flexibility in routing files based on their MIME types, enabling the handling of broad categories or specific MIME type patterns with ease. This feature particularly benefits applications requiring sophisticated file classification and routing logic.

  • In Jupyter notebooks, the image of the Pipeline will no longer be displayed automatically. Instead, the textual representation of the Pipeline will be displayed. To display the Pipeline image, use the show method of the Pipeline object.

  • Add support for callbacks during pipeline deserialization. Currently supports a pre-init hook for components that can be used to inspect and modify the initialization parameters before the invocation of the component’s __init__ method.

  • pipeline.run() accepts a set of component names whose intermediate outputs are returned in the final pipeline output dictionary.

  • Refactor PyPDFToDocument to simplify support for custom PDF converters. PDF converters are classes that implement the PyPDFConverter protocol and have 3 methods: convert, to_dict and from_dict.

⚠️ Deprecation Notes

  • Deprecate HuggingFaceTGIChatGenerator, will be removed in Haystack 2.3.0. Use HuggingFaceAPIChatGenerator instead.
  • Deprecate HuggingFaceTEIDocumentEmbedder, will be removed in Haystack 2.3.0. Use HuggingFaceAPIDocumentEmbedder instead.
  • Deprecate HuggingFaceTGIGenerator, will be removed in Haystack 2.3.0. Use HuggingFaceAPIGenerator instead.
  • Deprecate HuggingFaceTEITextEmbedder, will be removed in Haystack 2.3.0. Use HuggingFaceAPITextEmbedder instead.
  • Using the converter_name parameter in the PyPDFToDocument component is deprecated. it will be removed in the 2.3.0 release. Use the converter parameter instead.

🐛 Bug Fixes

  • Forward declaration of AnalyzeResult type in AzureOCRDocumentConverter. AnalyzeResult is already imported in a lazy import block. The forward declaration avoids issues when azure-ai-formrecognizer>=3.2.0b2 is not installed.

  • Fixed a bug in the MetaFieldRanker: when the weight parameter was set to 0 in the run method, the component incorrectly used the default parameter set in the __init__ method.

  • Fixes Pipeline.run() logic so components with all their inputs with a default are run in the correct order.

  • Fix a bug when running a Pipeline that would cause it to get stuck in an infinite loop

  • Fixes on the HuggingFaceTEITextEmbedder returning an embedding of incorrect shape when used with a Text-Embedding-Inference endpoint deployed using Docker.

  • Add the @component decorator to HuggingFaceTGIChatGenerator. The lack of this decorator made it impossible to use the HuggingFaceTGIChatGenerator in a pipeline.

  • Updated the SearchApiWebSearch component with new search format and allowed users to specify the search engine via the engine parameter in search_params. The default search engine is Google, making it easier for users to tailor their web searches.