Issue Deploying Pipeline with LLMs in It

[Yuyang105]

I recently started experimenting with Haystack as a framework for RAG. During local testing, everything worked perfectly. However, when attempting to deploy a pipeline with a generator component using hayhooks deploy, I encountered an error related to the OpenAIGenerator.

Here is the pipeline file I'm using:

pipelines/test.yml   # same as @mpangrazzi  sample pipeline

components:
  llm:
    init_parameters:
      api_base_url: null
      api_key:
        env_vars:
        - OPENAI_API_KEY
        strict: true
        type: env_var
      generation_kwargs: {}
      model: gpt-4o-mini
      organization: null
      streaming_callback: null
      system_prompt: null
    type: haystack.components.generators.openai.OpenAIGenerator
  prompt_builder:
    init_parameters:
      required_variables: null
      template: "\nGiven the following information, answer the question.\n\nContext:\n\
        {% for document in documents %}\n    {{ document.content }}\n{% endfor %}\n\
        \nQuestion: {{question}}\nAnswer:\n"
      variables: null
    type: haystack.components.builders.prompt_builder.PromptBuilder
  retriever:
    init_parameters:
      document_store:
        init_parameters:
          bm25_algorithm: BM25L
          bm25_parameters: {}
          bm25_tokenization_regex: (?u)\b\w\w+\b
          embedding_similarity_function: dot_product
          index: 69eb366f-b05a-4811-befe-ed9d49c59a23
        type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
      filter_policy: replace
      filters: null
      return_embedding: false
      scale_score: false
      top_k: 10
    type: haystack.components.retrievers.in_memory.embedding_retriever.InMemoryEmbeddingRetriever
  text_embedder:
    init_parameters:
      batch_size: 32
      config_kwargs: null
      device:
        device: mps
        type: single
      model: sentence-transformers/all-MiniLM-L6-v2
      model_kwargs: null
      normalize_embeddings: false
      precision: float32
      prefix: ''
      progress_bar: true
      suffix: ''
      token:
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false
        type: env_var
      tokenizer_kwargs: null
      truncate_dim: null
      trust_remote_code: false
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
connections:
- receiver: retriever.query_embedding
  sender: text_embedder.embedding
- receiver: prompt_builder.documents
  sender: retriever.documents
- receiver: llm.prompt
  sender: prompt_builder.prompt
max_runs_per_component: 100
metadata: {}

And here is the error message I received:

(venv) (base) ➜  rag-biz git:(main) ✗ hayhooks deploy pipelines/test.yml
Error deploying pipeline: Couldn't deserialize component 'llm' of class 'OpenAIGenerator' with the following data: {'init_parameters': {'api_base_url': None, 'api_key': EnvVarSecret(_env_vars=('OPENAI_API_KEY',), _strict=True, _type=<SecretType.ENV_VAR: 'env_var'>), 'generation_kwargs': {}, 'model': 'gpt-4o-mini', 'organization': None, 'streaming_callback': None, 'system_prompt': None}, 'type': 'haystack.components.generators.openai.OpenAIGenerator'}. Possible reasons include malformed serialized data, mismatch between the serialized component and the loaded one (due to a breaking change, see https://github.com/deepset-ai/haystack/releases), etc.

Error deploying pipeline: Couldn't deserialize component 'llm' of class 'OpenAIGenerator' with the following data:
{
    'init_parameters': {
        'api_base_url': None,
        'api_key': EnvVarSecret(_env_vars=('OPENAI_API_KEY',), _strict=True, _type=<SecretType.ENV_VAR: 'env_var'>),
        'generation_kwargs': {},
        'model': 'gpt-4o-mini',
        'organization': None,
        'streaming_callback': None,
        'system_prompt': None
    },
    'type': 'haystack.components.generators.openai.OpenAIGenerator'
}.
Possible reasons include malformed serialized data, mismatch between the serialized component and the loaded one (due to a breaking change, see https://github.com/deepset-ai/haystack/releases), etc.

Steps to Reproduce

Install dependencies with the following versions:
haystack-ai==2.8.1rc2
hayhooks==0.0.19
FastAPI, Starlette, and other related packages (see detailed environment below).
Use hayhooks deploy pipelines/test.yml to deploy a pipeline containing an OpenAIGenerator component.
Encounter the error above.

Environment

Python version: 3.12.4
OS: macOS
Key libraries and versions:

(venv) (base) ➜  rag-biz git:(main) ✗ pip list
Package               Version
--------------------- -----------
annotated-types       0.7.0
anyio                 4.7.0
backoff               2.2.1
certifi               2024.12.14
charset-normalizer    3.4.1
click                 8.1.8
distro                1.9.0
fastapi               0.115.6
h11                   0.14.0
hayhooks              0.0.19
haystack-ai           2.8.1rc2
haystack-experimental 0.4.0
httpcore              1.0.7
httpx                 0.28.1
idna                  3.10
Jinja2                3.1.5
jiter                 0.8.2
lazy_imports          0.4.0
loguru                0.7.3
MarkupSafe            3.0.2
monotonic             1.6
more-itertools        10.5.0
networkx              3.4.2
numpy                 2.2.1
openai                1.58.1
pandas                2.2.3
pip                   24.3.1
posthog               3.7.4
pydantic              2.10.4
pydantic_core         2.27.2
python-dateutil       2.9.0.post0
python-multipart      0.0.20
pytz                  2024.2
PyYAML                6.0.2
requests              2.32.3
six                   1.17.0
sniffio               1.3.1
starlette             0.41.3
tenacity              9.0.0
tqdm                  4.67.1
typing_extensions     4.12.2
tzdata                2024.2
urllib3               2.3.0
uvicorn               0.34.0

Would it be possible to confirm if this is a version mismatch or if there’s something wrong with how the generator component or its parameters are defined?

Any help or guidance would be much appreciated!

[mpangrazzi]

Hi @Yuyang105 ! I've answered here about it. Can you please take a look? Thanks!