Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pinecone embedding store #64

Merged
merged 2 commits into from
Nov 29, 2023
Merged

Pinecone embedding store #64

merged 2 commits into from
Nov 29, 2023

Conversation

jmartisk
Copy link
Collaborator

No description provided.

@jmartisk jmartisk requested a review from a team as a code owner November 24, 2023 08:34
@jmartisk jmartisk force-pushed the pinecone branch 3 times, most recently from e22238f to fca573b Compare November 24, 2023 09:09
@geoand
Copy link
Collaborator

geoand commented Nov 24, 2023

CI seems to complain :)

@jmartisk jmartisk force-pushed the pinecone branch 3 times, most recently from ffacf08 to 04d26f9 Compare November 24, 2023 10:27
@jmartisk
Copy link
Collaborator Author

It's now failing because of an ambiguous dependency in the examples code.
An EmbeddingStore injection point can now have a Redis or Pinecone store. Do we really need to treat the examples as a valid Quarkus application a build it (

<goal>build</goal>
)? I fail to see what purpose this has

@geoand
Copy link
Collaborator

geoand commented Nov 24, 2023

Yeah, we want to sample applications to build as we point users to them all the time.

For the docs, you can just comment out the extension

@jmartisk jmartisk force-pushed the pinecone branch 3 times, most recently from 6a644d5 to 3221aca Compare November 28, 2023 07:56
@cescoffier
Copy link
Collaborator

It should be possible to inject the dedicated type (PineconeStore or whatever the name is) and avoid ambiguous dependency. In other parts of the doc, I mentioned that if there is only one bean of the given type in the application, you can inject it using DocumentStore.

@jmartisk
Copy link
Collaborator Author

jmartisk commented Nov 28, 2023

I've updated the docs to add Pinecone docs and resolve the ambiguities..
For the CI, the problem is that we only have one Pinecone index that is shared, so this is how I'm proposing it to work:

  • If we detect that secrets are enabled (it needs the PINECONE_API_KEY variable) then we set max-parallel to 1 to avoid running the test twice in parallel
  • If secrets are disabled (the workflow was triggered by a PR from a fork....) then the PINECONE_API_KEY variable doesn't exist and the test is skipped, but we keep max-parallel at a bigger number, because it's not necessary to limit the concurrency in this case
  • In the nightly build (build-against-langchain4j.yml) the test will run.. with a max-parallel of 1

"application.properties"));

@Inject
PineconeEmbeddingStore embeddingStore;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does @Inject EmbeddingStore work too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, @Inject EmbeddingStore works, but if #73 is OK I will update also it here to accept @Inject EmbeddingStore<TextSegment>, it's better than using the raw type

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, let's rebase this after that one is in

@jmartisk jmartisk merged commit 3df259f into main Nov 29, 2023
2 checks passed
@jmartisk jmartisk deleted the pinecone branch November 29, 2023 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants