Skip to content

Commit

Permalink
Upgrade protobuf from v3.20.0 to v4.25.3
Browse files Browse the repository at this point in the history
Upgrade protobuf from v3 to v4. This adds a number of performance
improvements in parsing / serialization as documented at
https://protobuf.dev/news/2022-05-06/#python-updates

This increases upsert() throughput by 3x (measured by upserting 1M 768
dimension indexes to a pod-based index in batches of 500):

Before:  880 vectors/sec
After:  2580 vectors/sec

As per the documentation, this results in an incompatible change with
the _generated_ Python code, so this depends on a related change to
pinecone-protos to change the version of protobuf used to generate the
Python code there.

This patch:

- Changes the version of protobuf package from v3 to current latest v4
  (v4.25.3).

- Updates the generated pb2 files from pinecone-protos to the protobuf
  v4 generated version.

- Switches from grpc-gateway-protoc-gen-openapiv2 to
  protoc-gen-openapiv2 as the support library needed to pull in the
  autogeneated openapi annotations dependancy - which changes format
  between v3 and v4.
  • Loading branch information
daverigby committed Apr 22, 2024
1 parent c5ba9ce commit e248553
Show file tree
Hide file tree
Showing 6 changed files with 861 additions and 2,074 deletions.
12 changes: 6 additions & 6 deletions .github/workflows/testing-dependency.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,12 @@ jobs:
# - 4.1.0
- 4.3.3
protobuf_version:
- 3.20.3
- 4.25.3
protoc-gen-openapiv2:
- 0.0.1
googleapis_common_protos_version:
- 1.53.0
- 1.62.0
grpc_gateway_protoc_gen_openapiv2_version:
- 0.1.0
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/test-dependency-grpc
Expand Down Expand Up @@ -92,12 +92,12 @@ jobs:
- 3.1.3
- 4.3.3
protobuf_version:
- 3.20.3
- 4.25.3
protoc-gen-openapiv2:
- 0.0.1
googleapis_common_protos_version:
- 1.53.0
- 1.62.0
grpc_gateway_protoc_gen_openapiv2_version:
- 0.1.0
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/test-dependency-grpc
Expand Down
Loading

0 comments on commit e248553

Please sign in to comment.