Merge branch 'main' into v1-27

authors.yml

@@ -195,3 +195,9 @@ akanksha:
   title: Product Management Intern
   url: https://www.linkedin.com/in/akankshasharmaiitg/
   image_url: /img/people/icon/akanksha.jpg
+deepti:
+  name: Deepti Naidu
+  title: Partner Solution Engineer
+  url: https://www.linkedin.com/in/deeptinaidu/
+  image_url: /img/people/icon/deepti.png
+

blog/2024-03-20-typescript-client-beta/index.mdx

@@ -337,7 +337,7 @@ The other is that the overall response object will indicate whether there were a
 
 The `cursor` API has been a popular tool for iterating through an entire collection, for example, to create manual backups or retrieve the entire set of IDs.
 
-We've simplified the API here with a more intuitive iterator, so the syntax to extract all objects from a collection is now as simple as:
+We've simplified the API here with a more intuitive iterator, so the syntax to extract all objects from a collection is now achievable with:
 
 ```typescript
 const articles = client.collections.get('Article')

blog/2024-10-10-wealingo-demo/index.mdx

@@ -0,0 +1,128 @@
+---
+title: Wealingo
+slug: wealingo-demo
+authors: [deepti, shukri]
+date: 2024-10-10
+tags: ['how-to', 'search']
+image: ./img/hero.png
+description: 'A language-learning app built using Confluent Cloud and Weaviate.'
+---
+
+![Wealingo](./img/hero.png)
+
+At [Confluent Current 2024](https://current.confluent.io/), we demoed Wealingo, a language-learning app built using Weaviate’s vector database and Confluent Cloud, a fully managed streaming data service based on Apache Kafka. The demo highlighted how **real-time data** and **vector embeddings** enable personalized learning by continuously adapting to user performance. Let’s dive into how Wealingo leverages these technologies to provide dynamic, tailored language lessons in real time.
+
+## App Overview
+
+Wealingo is a personalized language-learning app designed to help users learn Italian through a tailored experience. It stands out with two key innovations:
+
+1. **Semantic Search**: Users can quickly search for lessons across all topics, bypassing a rigid structure. This enables them to immediately find the most relevant content based on their current needs and context.
+2. **Personalized Question Generation**: The app generates custom questions based on individual user performance and preferences.
+
+The goal is to provide continuous, adaptive learning aligned with each user’s strengths, weaknesses, and interests.
+
+### Architecture
+
+<figure style={{ maxWidth: "90%", margin: "0 auto" }}>
+  <img
+    src={require('./img/image.png').default}
+    alt="alt"
+    style={{ width: "100%" }}
+  />
+</figure>
+
+Wealingo’s architecture consists of the following key components:
+
+1. Mobile Application: Where users interact with the app, search for phrases, and request personalized questions.
+2. [**PostgreSQL**](https://www.postgresql.org/): Acts as the system of record, storing all application and user data, including the question bank and user performance.
+3. [Debezium PostgreSQL Change Data Capture](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) (CDC): Captures any changes in the database and streams them to Confluent Cloud in real-time.
+4. [Confluent Cloud](https://www.confluent.io/confluent-cloud/): Processes real-time data streams, ensuring the app adapts to user progress.
+5. Large Language Model (LLM): Generates personalised questions
+6. Weaviate: Handles vector embeddings, semantic search and interacting with the LLM
+
+## How It Works
+
+### Semantic Search
+
+Semantic search in Wealingo allows users to find questions based on the meaning behind their queries, rather than being tied to exact keywords. This flexibility helps users quickly locate lessons that are most relevant to their current needs.
+
+For example, let’s say the user is interested in lessons about “how to buy flowers”. Doing a semantic search for this query will return results like:
+
+- I want to buy red roses
+- Can I buy a basket of yellow daisies?
+- Do you have hibiscus flowers in different colors?
+
+<figure style={{ maxWidth: "90%", margin: "0 auto" }}>
+  <img
+    src={require('./img/image1.png').default}
+    alt="alt"
+    style={{ width: "100%" }}
+  />
+</figure>
+
+By comparing vector embeddings of the user search queries with the translations already saved in the database, we can retrieve phrases that are analogous to what the user requested. With Weaviate, you can combine the power of vector and keyword searches by using the hybrid search API. Hybrid search fuses the result sets from vector and keyword search into one, based on the value of alpha in the api. An `alpha` of `1` is a pure vector search, while `0` is a pure keyword search. 
+
+```python
+search_query = "how to buy flowers"
+
+conversations = client.collections.get("ConversationalStatements")
+
+response = conversations.query.hybrid(
+    limit=10,
+    query=search_query,
+    query_properties=["question"],
+    return_properties=["question", "answer"],
+    return_metadata=MetadataQuery(score=True),
+)
+```
+
+### Personalized Question Generation
+
+Sometimes, users may need help with highly specific scenarios. For example, let’s assume a user wants to learn about “buying a bouquet of blue lilies.” This level of specificity is unlikely to be found in any pre-written lesson because the lessons are written by human teachers and no human could anticipate every scenario.
+
+This is where Generative AI steps in. Wealingo leverages **Weaviate** and **LLMs** to dynamically generate a personalized question that fits the user’s exact needs and tailored to their current skill level, helping them learn in context, no matter how niche the request.
+
+For example, if a user asks the AI to generate questions about  “buying a bouquet of blue lilies”:
+
+<figure style={{ maxWidth: "60%", margin: "0 auto" }}>
+  <img
+    src={require('./img/image2.png').default}
+    alt="alt"
+    style={{ width: "100%" }}
+  />
+</figure>
+
+Under the hood,  the app first gathers the query and relevant profile data, such as the user’s past mistakes, and sends this information to **Weaviate**. Weaviate uses these inputs to craft a prompt, which it then passes to an LLM (OpenAI in this case) via the [generative-openai](https://weaviate.io/developers/weaviate/model-providers/openai/generative) module. The LLM generates personalized questions that are sent back to the app and saved in the **PostgreSQL** database, which serves as the system of record. To capture the fluency of the user, Wealingo fetches the past unanswered or wrong questions and uses them in the context. 
+
+```python
+context = "Create 3 new questions based on fluency of user"
+generate_query = "buying a bouquet of blue lilies"
+
+response = conversations.generate.near_text(
+    query=generate_query,
+    limit=10,
+    grouped_task=context,
+)
+```
+
+To keep everything up to date, the Debezium PostgreSQL CDC connector streams any newly generated questions into a **Kafka** queue in real-time. From there, our [spark-connector](https://github.com/weaviate/spark-connector) consumes the questions from Kafka, ensuring that the next time a user searches for something similar e.g. “purchase bouquet of blue lilies”, the AI-generated content will already be in Weaviate, ready for retrieval:
+
+<figure style={{ maxWidth: "80%", margin: "0 auto" }}>
+  <img
+    src={require('./img/image3.png').default}
+    alt="alt"
+    style={{ width: "100%" }}
+  />
+</figure>
+
+This setup allows Wealingo’s question bank to continuously grow and evolve, according to real-world user needs.
+
+## Conclusion
+
+Wealingo demonstrates how real-time data streaming and AI can transform language learning into a personalized experience. With Weaviate’s seamless integration into Confluent Cloud and LLMs, the app dynamically generates tailored lessons and continuously evolves based on user interactions.
+
+If you're interested in exploring further, feel free to fork the [Wealingo code](https://github.com/weaviate/wealingo-demo) and experiment with it yourself.
+
+import WhatsNext from '/_includes/what-next.mdx';
+
+<WhatsNext />

developers/weaviate/concepts/vector-quantization.md

@@ -99,7 +99,9 @@ When SQ is enabled, Weaviate boosts recall by over-fetching compressed results.
 
 Weaviate over-fetches results and then re-scores them when you use SQ or BQ. This is because the distance calculation on the compressed vectors is not as accurate as the same calculation on the original vector embedding.
 
-When you run a query, Weaviate compares the `rescore limit` and the query limit. The query retieves compressed objects until the object count reaches whichever limit is greater. Then, Weaviate fetches the original, uncompressed vector embeddings that correspond to the compressed vectors. The uncompressed vectors are used to recalculate the query distance scores.
+When you run a query, Weaviate compares the query limit against a configurable `rescoreLimit` parameter.
+
+The query retrieves compressed objects until the object count reaches whichever limit is greater. Then, Weaviate fetches the original, uncompressed vector embeddings that correspond to the compressed vectors. The uncompressed vectors are used to recalculate the query distance scores.
 
 For example, if a query is made with a limit of 10, and a rescore limit of 200, Weaviate fetches 200 objects. After rescoring, the query returns top 10 objects. This process offsets the loss in search quality (recall) that is caused by compression.
 

developers/weaviate/config-refs/env-vars.md

@@ -36,6 +36,8 @@ All other values are interpreted as `false`.
 | `LIMIT_RESOURCES` | If `true`, Weaviate will automatically attempt to auto-detect and limit the amount of resources (memory & threads) it uses to (0.8 * total memory) and (number of cores-1). It will override any `GOMEMLIMIT` values, however it will respect `GOMAXPROCS` values. | `boolean` | `false` |
 | `LOG_FORMAT` | Set the Weaviate logging format <br/><br/>Default: Outputs log data to json. e.g.: `{"action":"startup","level":"debug","msg":"finished initializing modules","time":"2023-04-12T05:07:43Z"}` <br/>`text`: Outputs log data to a string. e.g. `time="2023-04-12T04:54:23Z" level=debug msg="finished initializing modules" action=startup` | `string` |  |
 | `LOG_LEVEL` | Sets the Weaviate logging level. Default: `info`<br/><br/>`panic`: Panic entries only. (new in `v1.24`) <br/>`fatal`: Fatal entries only. (new in `v1.24`)  <br/> `error`: Error entries only. (new in `v1.24`) <br/>`warning`: Warning entries only. (new in `v1.24`) <br/>`info`: General operational entries. <br/> `debug`: Very verbose logging. <br/>`trace`: Even finer-grained informational events than `debug`. | `string` | |
+| `MEMORY_READONLY_PERCENTAGE` | If memory usage is higher than the given percentage all shards on the affected node will be marked as `READONLY`, meaning all future write requests will fail. (Default: `80`) | `string - number` | `75` |
+| `MEMORY_WARNING_PERCENTAGE` | If memory usage is higher than the given percentage a warning will be logged by all shards on the affected node's disk. (Default: `0` - i.e. no limit) | `string - number` | `85` |
 | `MODULES_CLIENT_TIMEOUT` | Timeout for requests to Weaviate modules. Default: `50s` | `string - duration` | `5s`, `10m`, `1h` |
 | `ORIGIN` | Set the http(s) origin for Weaviate | `string - HTTP origin` | `https://my-weaviate-deployment.com` |
 | `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store | `string - file path` | `/var/lib/weaviate` <br/> Starting in v1.24, defaults to `./data`|
@@ -105,6 +107,12 @@ All other values are interpreted as `false`.
 | `RAFT_GRPC_MESSAGE_MAX_SIZE` | The maximum internal raft gRPC message size in bytes. Defaults to 1073741824 | `string - number` | `1073741824` |
 | `RAFT_JOIN` | Manually set Raft voter nodes. If set, RAFT_BOOTSTRAP_EXPECT needs to be adjusted manually to match the number of Raft voters. | `string` | `weaviate-0,weaviate-1` |
 | `RAFT_METADATA_ONLY_VOTERS` | If `true`, voter nodes only handle the schema. They do not accept any data. | `boolean` | `false` |
+| `REPLICATION_MINIMUM_FACTOR` | The minimum replication factor for all collections in the cluster. | `string - number` | `3` |
+
+<!-- Docs notes:
+MAINTENANCE_NODES: not documented, for internal use only
+ASYNC_BRUTE_FORCE_SEARCH_LIMIT: not documented, for internal use only
+-->
 
 ## Questions and feedback
 

developers/weaviate/configuration/replication.md

@@ -17,28 +17,17 @@ import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx';
 
 Replication is disabled by default. It can be enabled per collection in the [collection configuration](../manage-data/collections.mdx#replication-settings). This means you can set different replication factors per class in your dataset.
 
-To enable replication, set the replication factor:
-
-```yaml
-{
-  "class": "ExampleClass",
-  "properties": [
-    {
-      "name": "exampleProperty",
-      "dataType": [
-        "text"
-      ]
-    }
-  ],
-  # highlight-start
-  "replicationConfig": {
-    "factor": 3   # Integer, default 1. How many copies of this class will be stored.
-  }
-  # highlight-end
-}
-```
+To enable replication, you can set one or both of the following:
+- `REPLICATION_MINIMUM_FACTOR` environment variable for the entire Weaviate instance, or
+- `replicationFactor` parameter for a collection.
+
+### Weaviate-wide minimum replication factor
+
+The `REPLICATION_MINIMUM_FACTOR` environment variable sets the minimum replication factor for all collections in the Weaviate instance.
+
+If you set the [replication factor for a collection](#replication-factor-for-a-collection), the collection's replication factor overrides the minimum replication factor.
 
-The client code looks like this:
+### Replication factor for a collection
 
 import SchemaReplication from '/_includes/code/schema.things.create.replication.mdx';
 

developers/weaviate/manage-data/import.mdx

@@ -372,6 +372,10 @@ The Python clients have built-in batching methods to help you optimize import sp
 - [Python client `v4`](../client-libraries/python/index.md#batching)
 - [Python client `v3`](../client-libraries/python/python_v3.md#batching)
 
+### Async Python client and batching
+
+Currently, the [async Python client does not support batching](../client-libraries/python/async.md#bulk-data-insertion). To use batching, use the sync Python client.
+
 ## Stream data from large files
 
 If your dataset is large, consider streaming the import to avoid out-of-memory issues.

developers/weaviate/model-providers/openai/embeddings.md

@@ -48,6 +48,8 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
 
 </details>
 
+<!-- Docs note: the `OPENAI_ORGANIZATION` environment variable is not documented, as it is not the recommended way to provide the OpenAI organization parameter. -->
+
 ### API credentials
 
 You must provide a valid OpenAI API key to Weaviate for this integration. Go to [OpenAI](https://openai.com/) to sign up and obtain an API key.