Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[content-nodes] Fill provider_visibility in data_source_nodes from connectors -> api/v1 #9988

Merged
merged 4 commits into from
Jan 16, 2025
Merged

[content-nodes] Fill provider_visibility in data_source_nodes from connectors -> api/v1 #9988

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
69318ef
pass the provider_visibility when upserting folders
aubin-tchoi Jan 14, 2025
8ede353
pass the provider_visibility in Slack activities
aubin-tchoi Jan 14, 2025
314400f
add a type for ProviderVisibility
aubin-tchoi Jan 14, 2025
beabc90
fix types in sdks js
aubin-tchoi Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions connectors/src/connectors/slack/lib/channels.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ export type SlackChannelType = {
slackId: string;
permission: ConnectorPermission;
agentConfigurationId: string | null;
private: boolean;
};

export async function updateSlackChannelInConnectorsDb({
Expand Down Expand Up @@ -60,6 +61,7 @@ export async function updateSlackChannelInConnectorsDb({
slackId: channel.slackChannelId,
permission: channel.permission,
agentConfigurationId: channel.agentConfigurationId,
private: channel.private,
};
}

Expand Down
1 change: 1 addition & 0 deletions connectors/src/connectors/slack/temporal/activities.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,7 @@ export async function syncChannel(
parents: [slackChannelInternalIdFromSlackChannelId(channelId)],
mimeType: MIME_TYPES.SLACK.CHANNEL,
sourceUrl: getSlackChannelSourceUrl(channelId, slackConfiguration),
providerVisibility: channel.private ? "private" : "public",
});
}

Expand Down
4 changes: 4 additions & 0 deletions connectors/src/lib/data_sources.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import type {
CoreAPIFolder,
CoreAPITable,
PostDataSourceDocumentRequestBody,
ProviderVisibility,
} from "@dust-tt/types";
import {
isValidDate,
Expand Down Expand Up @@ -1244,6 +1245,7 @@ export async function _upsertDataSourceFolder({
title,
mimeType,
sourceUrl,
providerVisibility,
}: {
dataSourceConfig: DataSourceConfig;
folderId: string;
Expand All @@ -1253,6 +1255,7 @@ export async function _upsertDataSourceFolder({
title: string;
mimeType: string;
sourceUrl?: string;
providerVisibility?: ProviderVisibility;
}) {
const now = new Date();

Expand All @@ -1265,6 +1268,7 @@ export async function _upsertDataSourceFolder({
parents,
mimeType,
sourceUrl: sourceUrl ?? null,
providerVisibility: providerVisibility || null,
});

if (r.isErr()) {
Expand Down
2 changes: 2 additions & 0 deletions front/pages/api/v1/w/[wId]/spaces/[spaceId]/data_sources/[dsId]/folders/[fId].ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ async function handler(
title,
mime_type,
source_url,
provider_visibility,
} = r.data;
if (parentId && parents && parents[1] !== parentId) {
return apiError(req, res, {
Expand Down Expand Up @@ -155,6 +156,7 @@ async function handler(
title: title,
mimeType: mime_type,
sourceUrl: source_url ?? null,
providerVisibility: provider_visibility,
});

if (upsertRes.isErr()) {
Expand Down
3 changes: 3 additions & 0 deletions sdks/js/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -852,6 +852,7 @@ export class DustAPI {
parents,
mimeType,
sourceUrl,
providerVisibility,
}: {
dataSourceId: string;
folderId: string;
Expand All @@ -861,6 +862,7 @@ export class DustAPI {
parents: string[];
mimeType: string;
sourceUrl: string | null;
providerVisibility: "public" | "private" | null;
}) {
const res = await this.request({
method: "POST",
Expand All @@ -874,6 +876,7 @@ export class DustAPI {
parents,
mime_type: mimeType,
source_url: sourceUrl,
provider_visibility: providerVisibility,
},
});

Expand Down
3 changes: 3 additions & 0 deletions sdks/js/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2145,13 +2145,16 @@ export type UpsertFolderResponseType = z.infer<
typeof UpsertFolderResponseSchema
>;

const ProviderVisibilitySchema = FlexibleEnumSchema<"public" | "private">();

export const UpsertDataSourceFolderRequestSchema = z.object({
timestamp: z.number(),
parents: z.array(z.string()).nullable().optional(),
parent_id: z.string().nullable().optional(),
title: z.string(),
mime_type: z.string(),
source_url: z.string().nullable().optional(),
provider_visibility: ProviderVisibilitySchema.nullable().optional(),
});
export type UpsertDataSourceFolderRequestType = z.infer<
typeof UpsertDataSourceFolderRequestSchema
Expand Down
3 changes: 2 additions & 1 deletion types/src/front/lib/connectors_api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ export type ConnectorType = {
*/
export type ConnectorPermission = "read" | "write" | "read_write" | "none";
export type ContentNodeType = "file" | "folder" | "database" | "channel";
export type ProviderVisibility = "public" | "private";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if you have a schema, the type should be derived from the schema

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the schema lives in sdks/js, it seems that types and sdks/js don't import stuff from one another and that we duplicate types (e.g. ConnectorProvider is here in sdks/js but also here in types)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kinda feel like we have to duplicate out of consistency

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed the SDK declares it's own types and they should not be used outside of the SDK itself : they define the public API contract.
By checking that internal types keeps matching the public types (because they are used as output of v1/ enpoints), we ensure that we don't break anything.

Does that make sense ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes very clear, thanks!


/*
* This constant defines the priority order for sorting content nodes by their type.
Expand Down Expand Up @@ -107,7 +108,7 @@ export interface ContentNode {
preventSelection?: boolean;
permission: ConnectorPermission;
lastUpdatedAt: number | null;
providerVisibility?: "public" | "private";
providerVisibility?: ProviderVisibility;
}

export type ContentNodeWithParentIds = ContentNode & {
Expand Down
4 changes: 4 additions & 0 deletions types/src/front/lib/core_api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import {
import { LightWorkspaceType } from "../../front/user";
import { LoggerInterface } from "../../shared/logger";
import { Err, Ok, Result } from "../../shared/result";
import { ProviderVisibility } from "./connectors_api";

export const MAX_CHUNK_SIZE = 512;

Expand Down Expand Up @@ -1547,6 +1548,7 @@ export class CoreAPI {
title,
mimeType,
sourceUrl,
providerVisibility,
}: {
projectId: string;
dataSourceId: string;
Expand All @@ -1557,6 +1559,7 @@ export class CoreAPI {
title: string;
mimeType: string;
sourceUrl?: string | null;
providerVisibility: ProviderVisibility | null | undefined;
}): Promise<CoreAPIResponse<{ folder: CoreAPIFolder }>> {
const response = await this._fetchWithError(
`${this._url}/projects/${projectId}/data_sources/${encodeURIComponent(
Expand All @@ -1575,6 +1578,7 @@ export class CoreAPI {
parents,
mime_type: mimeType,
source_url: sourceUrl,
provider_visibility: providerVisibility,
}),
}
);
Expand Down
Loading