data queries (#965)

THIRD_PARTY_LICENSES.md

@@ -7091,6 +7091,30 @@ THIS SOFTWARE.
 
 -----------
 
+The following npm package may be included in this product:
+
+ - zod-to-json-schema@3.24.1
+
+This package contains the following license:
+
+ISC License
+
+Copyright (c) 2020, Stefan Terdell
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+-----------
+
 The following npm package may be included in this product:
 
  - are-we-there-yet@2.0.0

docs/package.json

@@ -24,7 +24,7 @@
     "@astrojs/starlight": "^0.30.3",
     "astro": "^5.1.1",
     "rehype-mermaid": "^3.0.0",
-    "starlight-blog": "^0.16.0",
+    "starlight-blog": "^0.16.1",
     "starlight-links-validator": "^0.14.0",
     "starlight-package-managers": "^0.8.1",
     "typescript": "5.7.2",

docs/src/content/docs/reference/scripts/content-safety.mdx

@@ -12,15 +12,15 @@ GenAIScript has multiple built-in safety features to protect the system from mal
 
 The following safety prompts are included by default when running a prompt, unless the system option is configured:
 
--   [system.safety_harmful_content](/genaiscript/reference/scripts/system#systemsafety_harmful_content), safety prompt against Harmful Content: Hate and Fairness, Sexual, Violence, Self-Harm. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates.
--   [system.safety_jailbreak](/genaiscript/reference/scripts/system#systemsafety_jailbreak), safety script to ignore prompting instructions in code sections, which are created by the `def` function.
--   [system.safety_protected_material](/genaiscript/reference/scripts/system#systemsafety_protected_material) safety prompt against Protected material. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates
+- [system.safety_harmful_content](/genaiscript/reference/scripts/system#systemsafety_harmful_content), safety prompt against Harmful Content: Hate and Fairness, Sexual, Violence, Self-Harm. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates.
+- [system.safety_jailbreak](/genaiscript/reference/scripts/system#systemsafety_jailbreak), safety script to ignore prompting instructions in code sections, which are created by the `def` function.
+- [system.safety_protected_material](/genaiscript/reference/scripts/system#systemsafety_protected_material) safety prompt against Protected material. See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/safety-system-message-templates
 
 Other system scripts can be added to the prompt by using the `system` option.
 
--   [system.safety_ungrounded_content_summarization](/genaiscript/reference/scripts/system#systemsafety_ungrounded_content_summarization) safety prompt against ungrounded content in summarization
--   [system.safety_canary_word](/genaiscript/reference/scripts/system#systemsafety_canary_word) safety prompt against prompt leaks.
--   [system.safety_validate_harmful_content](/genaiscript/reference/scripts/system#systemsafety_validate_harmful_content) runs the `detectHarmfulContent` method to validate the output of the prompt.
+- [system.safety_ungrounded_content_summarization](/genaiscript/reference/scripts/system#systemsafety_ungrounded_content_summarization) safety prompt against ungrounded content in summarization
+- [system.safety_canary_word](/genaiscript/reference/scripts/system#systemsafety_canary_word) safety prompt against prompt leaks.
+- [system.safety_validate_harmful_content](/genaiscript/reference/scripts/system#systemsafety_validate_harmful_content) runs the `detectHarmfulContent` method to validate the output of the prompt.
 
 ## Azure AI Content Safety services
 
@@ -126,7 +126,9 @@ console.log(resf)
 }
 ```
 
-The [def](/genaiscript/reference/scripts/context) also supports setting a `detectPromptInjection` flag to apply the detection to each file.
+The [def](/genaiscript/reference/scripts/context)
+and [defData](/genaiscript/reference/scripts/context)
+functions supports setting a `detectPromptInjection` flag to apply the detection to each file.
 
 ```js
 def("FILE", env.files, { detectPromptInjection: true })
@@ -152,14 +154,14 @@ console.log(harms)
 
 ```json
 {
-  harmfulContentDetected: true,
-  categoriesAnalysis: [
+  "harmfulContentDetected": true,
+  "categoriesAnalysis": [
     {
-      category: 'Hate',
-      severity: 2
+      "category": "Hate'",
+      "severity": 2
     }, ...
  ],
-  chunk: 'you are a very bad person'
+  "chunk": "you are a very bad person"
 }
 ```
 

docs/src/content/docs/reference/scripts/context.md

@@ -257,6 +257,7 @@ The `defData` function also supports functions to slice the input rows and colum
 - `sliceTail`, number of rows to include from the end
 - `sliceSample`, number of rows to pick at random
 - `distinct`, list of column names to deduplicate the data based on
+- `query`, a [jq](https://jqlang.github.io/jq/) query to filter the data
 
 ```js
 defData("DATA", data, {

docs/src/content/docs/reference/scripts/parsers.md

@@ -16,7 +16,7 @@ The `parsers.json5` function parses the JSON5 format.
 In general, parsing a JSON file as JSON5 does not cause harm, but it might be more forgiving
 to syntactic errors. In addition to JSON5, [JSON repair](https://www.npmjs.com/package/jsonrepair) is applied if the initial parse fails.
 
--   JSON5 example
+- JSON5 example
 
 ```json5
 {
@@ -272,8 +272,8 @@ const fences = parsers.fences("...")
 Parses error, warning annotations in various formats
 into a list of objects.
 
--   [GitHub Actions](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions)
--   [Azure DevOps Pipeline](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#example-log-a-warning-about-a-specific-place-in-a-file)
+- [GitHub Actions](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions)
+- [Azure DevOps Pipeline](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#example-log-a-warning-about-a-specific-place-in-a-file)
 -
 
 ```js
@@ -322,6 +322,16 @@ used with `defData`.
 const d = parsers.tidyData(rows, { sliceSample: 100, sort: "name" })
 ```
 
+## GROQ
+
+Apply a [GROQ](GROQEvaluate) query to a JSON object.
+
+```js
+const d = parsers.GROQ(`*[completed == true && userId == 2]{
+  title
+}`, data)
+```
+
 ## hash
 
 Utility to hash an object, array into a string that is appropriate for hashing purposes.

docs/src/content/docs/reference/scripts/schemas.mdx

@@ -211,6 +211,10 @@ const schema = defSchema(...)
 $`Use ${schema} for the JSON schema.`
 ```
 
+## inference
+
+Use `JSONSchema.infer` to generate a JSON schema from a JSON object.
+
 ## Validation
 
 When a JSON/YAML payload is generated with the schema identifier,

docs/src/content/docs/reference/scripts/system.mdx

@@ -2555,7 +2555,8 @@ Plain text output
 `````js wrap title="system.output_plaintext"
 system({ title: "Plain text output" })
 $`## Plain Text Output
-Respond in plain text. Do not wrap result in Markdown code fences or XML tags.
+Respond in plain text. No yapping, no markdown, no code fences, no XML tags, no string delimiters
+wrapping it.
 `
 
 `````