Squashed commit of the following:

commit 9adde25 Author: Alban Dumouilla <alban@dust.tt> Date: Wed Nov 27 17:14:01 2024 +0100 AgentMessageFeedbacks API (#8879) * Feedbacks backend * Update sparkle * Update package-lock.json * Use resource * Move bodyvalidation * Undo change * Add cascade on workspace and message deletion * Truncate index name * Add workspaceId FK * Add destroy feedbacks in workspace scrub * lint * Destroy agentmessagefeedback * Add SET NULL on users FK * Fix msg id commit 04d68c9 Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 16:38:04 2024 +0100 feat(kw-search): pass title & mime when upserting tables from connectors (#8962) * feat(kw-search): pass title and mimetype when upserting tables from connectors * improve mimetype for notion DB --------- Co-authored-by: Henry Fontanier <henry@dust.tt> commit 551d655 Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 16:33:01 2024 +0100 fix(kw-search): delete datasource node then delete table (#8964) Co-authored-by: Henry Fontanier <henry@dust.tt> commit 6478c3e Author: Thomas Draier <thomas@dust.tt> Date: Wed Nov 27 16:30:52 2024 +0100 [core] Add folders endpoint (#8957) * folders endpoint * fixes commit e4eee59 Author: Alban Dumouilla <alban@dust.tt> Date: Wed Nov 27 16:28:44 2024 +0100 Revamping the extension whitelist (#8949) * Revamping the extension whitelist * Remove svg and sourcemaps * Remove csv/xml * Removing tfvars and tfstate to avoid syncing PII commit 44b6316 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 16:04:00 2024 +0100 Fix: counter-intuitive return values (#8961) commit 9a25d2d Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 15:56:40 2024 +0100 fix: upsert table before upserting datasource node (#8959) Co-authored-by: Henry Fontanier <henry@dust.tt> commit 591fe88 Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 15:55:35 2024 +0100 feat(kw-search): allow passible title and mimetype on front endpoints when upserting tables (#8955) Co-authored-by: Henry Fontanier <henry@dust.tt> commit 66d3d53 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:52:54 2024 +0100 Use if/else to avoid complex operators (#8960) commit 3e95e5c Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:47:46 2024 +0100 Fix: exit 0 in case not blocked, pass correct component name (#8958) commit 6a9a3b6 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:35:54 2024 +0100 Fix: confusing operators of bash (#8956) commit 5a84b75 Author: Daphné Popin <PopDaph@users.noreply.github.com> Date: Wed Nov 27 15:31:10 2024 +0100 Bump ext (#8954) commit 529fd50 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:22:18 2024 +0100 Strip the cc part (#8953) commit 6637b13 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:18:52 2024 +0100 Escape the topic (#8952) commit 8e5e0ca Author: Thomas Draier <thomas@dust.tt> Date: Wed Nov 27 15:07:17 2024 +0100 [extension] Better error handling in screen capture (#8944) * Better error handling * Update extension/app/background.ts Co-authored-by: Daphné Popin <PopDaph@users.noreply.github.com> * Update extension/app/background.ts Co-authored-by: Daphné Popin <PopDaph@users.noreply.github.com> * Update extension/app/background.ts Co-authored-by: Daphné Popin <PopDaph@users.noreply.github.com> * format --------- Co-authored-by: Daphné Popin <PopDaph@users.noreply.github.com> commit ecc53d6 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 15:06:39 2024 +0100 Remaining useless inputs (#8951) * Remaining useless inputs * Add: error to debug commit 4fe7961 Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 14:57:52 2024 +0100 Fix: action order (#8950) commit 113600a Author: Aubin <60398825+aubin-tchoi@users.noreply.github.com> Date: Wed Nov 27 14:57:42 2024 +0100 :recycle: remove an unused startFromTs parameter (#8947) commit cc11e95 Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 14:52:18 2024 +0100 feat(kw-search): create ds node for tables in core (#8945) * feat(kw-search): create ds node for tables in core * also delete ds node --------- Co-authored-by: Henry Fontanier <henry@dust.tt> commit 11dca3c Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 14:51:34 2024 +0100 Add: github action to prevent deployment based on channel topic (#8948) commit 9fd096c Author: Stanislas Polu <spolu@dust.tt> Date: Wed Nov 27 14:50:18 2024 +0100 ActionsDetails: finer grained action state (#8940) * ActionsDetails: finer grained action state * nit * memoize the spinner * nit/comment * tweak commit 3cadd72 Author: Daphné Popin <PopDaph@users.noreply.github.com> Date: Wed Nov 27 14:18:04 2024 +0100 Extension default shortcut to open panel (#8942) commit 68abb7c Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 13:12:56 2024 +0100 Add: dropdown to show all assistants used by a datasource and show modal on click (#8932) * Add: dropdown to show all assistants used by a datasource and show modal when click * Review fdbk commit aca995d Author: Daphné Popin <PopDaph@users.noreply.github.com> Date: Wed Nov 27 12:11:56 2024 +0100 Ext: various polish (#8941) commit 00eb9fc Author: Aubin <60398825+aubin-tchoi@users.noreply.github.com> Date: Wed Nov 27 12:01:53 2024 +0100 [connectors] enh(Zendesk) Fix misleading comments (#8939) * ♻️ * remove a misleading comment * 📝 * 📝 commit 51bde1a Author: Sebastien Flory <seb@dust.tt> Date: Wed Nov 27 11:56:16 2024 +0100 Relax columns count when uploading csv (#8934) * Relax columns count when uploading csv * Fix: missing correct handling for the other csv content type commit 8d80e5f Author: Philippe Rolet <pr@dust.tt> Date: Wed Nov 27 11:46:41 2024 +0100 [Favorites] Disable button while updating (#8936) * [Favorites] Disable button while updating Description --- Fixes #8933 Risks --- na Deploy --- front * hover full * better hover handling commit b86178d Author: Daphné Popin <PopDaph@users.noreply.github.com> Date: Wed Nov 27 11:38:17 2024 +0100 Ext: Various polish (#8937) commit ab30fa5 Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 11:22:15 2024 +0100 fix(zendesk): only fetch comment authors (#8935) Co-authored-by: Henry Fontanier <henry@dust.tt> commit b0af32e Author: Daphné Popin <PopDaph@users.noreply.github.com> Date: Wed Nov 27 10:25:07 2024 +0100 Ext: polish dropdown convo list (#8930) * Ext: polish dropdown convo list * Fix padding input bar commit 58e8f0b Author: Henry Fontanier <h.fontanier@gmail.com> Date: Wed Nov 27 10:24:08 2024 +0100 enh: improve viz prompt (#8929) Co-authored-by: Henry Fontanier <henry@dust.tt> commit 39d60fa Author: Jules Belveze <32683010+JulesBelveze@users.noreply.github.com> Date: Wed Nov 27 10:23:27 2024 +0100 [.github] - fix: trim trailing whitespace in workflow file (#8931) - Removed trailing spaces from k8s cloud build script arguments to avoid potential syntax issues in YAML commit 5380762 Author: Thomas Draier <thomas@dust.tt> Date: Wed Nov 27 10:19:33 2024 +0100 [extension] Explicit file names (#8927)
dust-tt · Nov 27, 2024 · f38b882 · f38b882
1 parent 42d3b68
commit f38b882
Show file tree

Hide file tree

Showing 55 changed files with 1,158 additions and 287 deletions.
diff --git a/.github/actions/slack-check-deployment-blocked/action.yml b/.github/actions/slack-check-deployment-blocked/action.yml
@@ -0,0 +1,65 @@
+name: 'Slack Check Deployment Blocked'
+description: 'Check the Slack #deployments channel title if deployment should be blocked'
+
+inputs:
+  component:
+    description: 'Component being deployed'
+    required: true
+  channel:
+    description: 'Slack channel'
+    required: true
+  slack_token:
+    description: 'Slack bot token'
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Get channel info
+      id: slack_channel_info
+      uses: slackapi/slack-github-action@v2.0.0
+      with:
+        method: conversations.info
+        token: ${{ inputs.slack_token }}
+        payload: |
+          channel: ${{ inputs.channel }}
+        errors: true
+
+    - name: Get channel topic
+      if: steps.slack_channel_info.outputs.ok == 'true'
+      id: slack_channel_topic
+      shell: bash
+      run: |
+        RESPONSE='${{ steps.slack_channel_info.outputs.response }}'
+        if ! TOPIC=$(echo "$RESPONSE" | jq -r '.channel.topic.value' | sed 's/cc.*$//'); then
+          echo "Failed to extract Slack channel topic" >&2
+          exit 0
+        fi
+      
+        echo "topic=$TOPIC" >> $GITHUB_OUTPUT
+
+    - name: Parse channel topic for blocked deployments
+      if: steps.slack_channel_topic.outputs.topic != ''
+      id: parse_channel_topic
+      shell: bash
+      run: |
+        is_blocked() {
+            local raw_component="$1"
+            local component=$(printf '%s' "$raw_component" | sed 's/[][\.*^$/]/\\&/g')
+
+            local status_string="$2"
+            
+            if echo "$status_string" | grep -q ":x:[ ]*${component}[ ]" || echo "$status_string" | grep -q ":x:[ ]*${component}\$" ; then
+                return 0 # Success
+            else
+                return 1 # Failure
+            fi
+        }
+
+        if is_blocked "${{ inputs.component }}" "${{ steps.slack_channel_topic.outputs.topic }}"; then
+            echo "${{ inputs.component}} is blocked based on channel topic"
+            exit 1
+        else
+            echo "${{ inputs.component}} is not blocked based on channel topic"
+            exit 0
+        fi
diff --git a/.github/workflows/deploy-front-edge.yml b/.github/workflows/deploy-front-edge.yml
@@ -19,6 +19,13 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@v3
 
+      - name: Slack Check Deployment Blocked
+        uses: ./.github/actions/slack-check-deployment-blocked
+        with:
+          component: "front-edge"
+          channel: ${{ secrets.SLACK_CHANNEL_ID }}
+          slack_token: ${{ secrets.SLACK_BOT_TOKEN }}
+
       - name: Get short sha
         id: short_sha
         run: echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

diff --git a/.github/workflows/deploy-front-qa.yml b/.github/workflows/deploy-front-qa.yml
@@ -42,9 +42,9 @@ jobs:
         run: |
           chmod +x ./k8s/cloud-build.sh
           ./k8s/cloud-build.sh \
-            --image-name=front-qa \ 
-            --dockerfile-path=./front/Dockerfile \ 
-            --working-dir=./ \ 
+            --image-name=front-qa \
+            --dockerfile-path=./front/Dockerfile \
+            --working-dir=./ \
             --dust-client-facing-url=https://front-qa.dust.tt
 
       - name: Generate a token

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -7,5 +7,6 @@
     "source.fixAll.eslint": "explicit"
   },
   "eslint.run": "onSave",
-  "typescript.preferences.importModuleSpecifier": "non-relative"
+  "typescript.preferences.importModuleSpecifier": "non-relative",
+  "github-actions.workflows.pinned.workflows": []
 }
diff --git a/connectors/src/connectors/github/lib/github_api.ts b/connectors/src/connectors/github/lib/github_api.ts
@@ -568,37 +568,95 @@ export async function getOctokit(connectionId: string): Promise<Octokit> {
 // Repository processing
 
 const EXTENSION_WHITELIST = [
+  // Programming Languages - General Purpose
   ".js",
   ".ts",
   ".tsx",
   ".jsx",
-  ".rb",
   ".py",
+  ".rb",
   ".rs",
   ".go",
   ".swift",
-  ".css",
-  ".html",
-  ".htm",
-  ".less",
-  ".sass",
-  ".scss",
-  ".php",
   ".java",
-  ".yaml",
-  ".yml",
-  ".md",
   ".c",
   ".h",
   ".cc",
   ".cpp",
   ".hpp",
+  ".php",
+
+  // .NET Ecosystem
+  ".cs",
+  ".csproj", // XML-based
+  ".sln", // Text-based solution file
+  ".cshtml", // Razor template
+  ".razor", // Razor component
+  ".resx", // XML-based resource
+  ".vb", // Visual Basic
+  ".fs", // F#
+  ".fsproj", // XML-based F# project
+  ".props", // MSBuild properties (XML)
+  ".targets", // MSBuild targets (XML)
+  ".nuspec", // NuGet specification (XML)
+
+  // Web Technologies
+  ".html",
+  ".htm",
+  ".css",
+  ".scss",
+  ".sass",
+  ".less",
+
+  // Data & Configuration
+  ".json",
+  ".yaml",
+  ".yml",
+  ".toml",
+  ".ini",
+  ".env",
+  ".conf",
+  ".config",
+
+  // Build & Dependencies
+  ".gradle",
+  ".lock", // Text-based lock files
+  ".mk", // Makefile
+  ".just", // Justfile
+  ".dockerfile",
+  ".editorconfig",
+
+  // Infrastructure as Code
+  ".tf", // Terraform
+  ".hcl", // HashiCorp Configuration Language
+  ".nix", // Nix expressions
+
+  // Documentation
+  ".md", // Markdown
+  ".rst", // ReStructured Text
+  ".adoc", // AsciiDoc
+  ".tex", // LaTeX
+  ".txt",
+
+  // Shell & Scripts
   ".sh",
   ".sql",
-  ".kt",
-  ".kts",
-  ".gradle",
-  ".xml",
+  ".kt", // Kotlin
+  ".kts", // Kotlin script
+
+  // Version Control
+  ".gitignore",
+  ".dockerignore",
+
+  // Testing
+  ".test.cs",
+  ".spec.cs",
+  ".tests.cs",
+
+  // Templates
+  ".liquid",
+  ".mustache",
+  ".handlebars",
 ];
 
 const SUFFIX_BLACKLIST = [".min.js", ".min.css"];

diff --git a/connectors/src/connectors/google_drive/temporal/spreadsheets.ts b/connectors/src/connectors/google_drive/temporal/spreadsheets.ts
@@ -83,6 +83,8 @@ async function upsertGdriveTable(
     truncate: true,
     parents: [tableId, ...parents],
     useAppForHeaderDetection: true,
+    title: `${spreadsheet.title} - ${title}`,
+    mimeType: "application/vnd.google-apps.spreadsheet",
   });
 
   logger.info(loggerArgs, "[Spreadsheet] Table upserted.");

diff --git a/connectors/src/connectors/microsoft/temporal/spreadsheets.ts b/connectors/src/connectors/microsoft/temporal/spreadsheets.ts
@@ -97,6 +97,9 @@ async function upsertMSTable(
     truncate: true,
     parents,
     useAppForHeaderDetection: true,
+    title: `${spreadsheet.name} - ${worksheet.name}`,
+    mimeType:
+      "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
   });
 
   logger.info(loggerArgs, "[Spreadsheet] Table upserted.");

diff --git a/connectors/src/connectors/notion/temporal/activities.ts b/connectors/src/connectors/notion/temporal/activities.ts
@@ -1812,6 +1812,8 @@ export async function renderAndUpsertPageFromCache({
               // We only update the rowId of for the page without truncating the rest of the table (incremental sync).
               truncate: false,
               parents,
+              title: parentDb.title ?? "Untitled Notion Database",
+              mimeType: "application/vnd.notion.database",
             }),
           localLogger
         );
@@ -2522,6 +2524,8 @@ export async function upsertDatabaseStructuredDataFromCache({
         // We overwrite the whole table since we just fetched all child pages.
         truncate: true,
         parents,
+        title: dbModel.title ?? "Untitled Notion Database",
+        mimeType: "notion/database",
       }),
     localLogger
   );

diff --git a/connectors/src/connectors/shared/file.ts b/connectors/src/connectors/shared/file.ts
@@ -78,6 +78,8 @@ export async function handleCsvFile({
       },
       truncate: true,
       parents,
+      title: fileName,
+      mimeType: "text/csv",
     });
   } catch (err) {
     localLogger.warn({ error: err }, "Error while parsing or upserting table");

diff --git a/connectors/src/connectors/zendesk/index.ts b/connectors/src/connectors/zendesk/index.ts
@@ -667,6 +667,7 @@ export class ZendeskConnectorManager extends BaseConnectorManager<null> {
 
   /**
    * Marks the connector as unpaused in db and restarts the workflows.
+   * Does not trigger full syncs, only restart the incremental and gc workflows.
    */
   async unpause(): Promise<Result<undefined, Error>> {
     const { connectorId } = this;
@@ -675,8 +676,8 @@ export class ZendeskConnectorManager extends BaseConnectorManager<null> {
       logger.error({ connectorId }, "[Zendesk] Connector not found.");
       return new Err(new Error("Connector not found"));
     }
-    // reset the cursor here to trigger a full resync
     await connector.markAsUnpaused();
+    // launch a gc and an incremental workflow (sync workflow without signals).
     return this.resume();
   }
 

diff --git a/connectors/src/connectors/zendesk/lib/sync_ticket.ts b/connectors/src/connectors/zendesk/lib/sync_ticket.ts
@@ -185,11 +185,7 @@ ${comments
       );
       author = null;
     }
-    return `
-[${comment?.created_at}] ${
-      author ? `${author.name} (${author.email})` : "Unknown User"
-    }:
-${comment.body}`;
+    return `[${comment?.created_at}] ${author ? `${author.name} (${author.email})` : "Unknown User"}:\n${comment.body}`;
   })
   .join("\n")}
 `.trim();

diff --git a/connectors/src/connectors/zendesk/temporal/activities.ts b/connectors/src/connectors/zendesk/temporal/activities.ts
@@ -434,16 +434,18 @@ export async function syncZendeskTicketBatchActivity({
     return { hasMore: false, nextLink: "" };
   }
 
-  const users = (await zendeskApiClient.users.list()) || [];
-
-  const comments = await concurrentExecutor(
+  const comments2d = await concurrentExecutor(
     tickets,
     async (ticket) => zendeskApiClient.tickets.getComments(ticket.id),
     { concurrency: 3, onBatchComplete: heartbeat }
   );
+  const userIds = _.uniq(
+    _.flatten(comments2d.map((comments) => comments.map((c) => c.author_id)))
+  );
+  const { result: users } = await zendeskApiClient.users.showMany(userIds);
 
   const res = await concurrentExecutor(
-    _.zip(tickets, comments),
+    _.zip(tickets, comments2d),
     async ([ticket, comments]) => {
       if (!ticket || !comments) {
         throw new Error(

diff --git a/connectors/src/connectors/zendesk/temporal/client.ts b/connectors/src/connectors/zendesk/temporal/client.ts
@@ -38,25 +38,19 @@ export function getZendeskGarbageCollectionWorkflowId(
 export async function launchZendeskSyncWorkflow(
   connector: ConnectorResource,
   {
-    startFromTs = null,
     brandIds = [],
     ticketsBrandIds = [],
     helpCenterBrandIds = [],
     categoryIds = [],
     forceResync = false,
   }: {
-    startFromTs?: number | null;
     brandIds?: number[];
     ticketsBrandIds?: number[];
     helpCenterBrandIds?: number[];
     categoryIds?: { brandId: number; categoryId: number }[];
     forceResync?: boolean;
   } = {}
 ): Promise<Result<undefined, Error>> {
-  if (startFromTs) {
-    throw new Error("[Zendesk] startFromTs not implemented yet.");
-  }
-
   const client = await getTemporalClient();
 
   const signals: (ZendeskUpdateSignal | ZendeskCategoryUpdateSignal)[] = [

diff --git a/connectors/src/lib/data_sources.ts b/connectors/src/lib/data_sources.ts
@@ -709,6 +709,8 @@ export async function upsertTableFromCsv({
   truncate,
   parents,
   useAppForHeaderDetection,
+  title,
+  mimeType,
 }: {
   dataSourceConfig: DataSourceConfig;
   tableId: string;
@@ -719,6 +721,8 @@ export async function upsertTableFromCsv({
   truncate: boolean;
   parents: string[];
   useAppForHeaderDetection?: boolean;
+  title: string;
+  mimeType: string;
 }) {
   const localLogger = logger.child({ ...loggerArgs, tableId, tableName });
   const statsDTags = [
@@ -754,6 +758,8 @@ export async function upsertTableFromCsv({
     truncate,
     async: true,
     useAppForHeaderDetection,
+    title,
+    mimeType,
   };
   const dustRequestConfig: AxiosRequestConfig = {
     headers: {