Skip to content

Commit

Permalink
Improved walkDir performance (#1814)
Browse files Browse the repository at this point in the history
* Nate/control plane client (#1691)

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* fix(gui): ctx rendering w/ renderInlineAs: "" (#1541)

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* fix codeqwen autocomplete leading space

* add voyage rerank-1

* feat: `--noEmit` for tsc checks in CI (#1559)

* docs: update CustomContextProvider docs (#1557)

* add stop tokens to qwen prompt

* update docs to reflect 3.5 sonnet being best

* docs: comment out unused providers (#1561)

* import Handlebars

* feat: toast notification for config updates (#1560)

* feat: toast notification for config updates

* feat: only trigger toast on config.json save

* displayRawMarkdown option

* feat: open pane on install (#1564)

* feat: open pane on activation

* comment out testing code

* update to reflect 16 stop words limit for deepseek

* feat: only trigger config update toast in vscode (#1571)

* docs(prompt-files): fix typos + clarify (#1575)

* doc: prompt file typo + clarifications

* fix: add back correct docs

* chore: add telemetry for pageviews (#1576)

* feat: update onboarding w/ embeddings model (#1570)

* chore(gui): remove unused pages

* feat: add embeddings step

* feat: update styles

* feat: copy button updates

* fix: correct pull command for embed model

* fix: remove commented code

* fix: remove commented code

* feat: simplify copy btn props

* chore: rename onboarding selection event

* feat: add provider config

* fix: undo msg name

* remove dead code

* fix: invalid mode check

* fix: remove testing logic

* docs(telemetry): add pageviews to tracking list (#1581)

* Add reranker configuration options to codebase embedding docs (#1584)

- Introduce reranker concept
- List available reranker options
- Provide configuration instructions
- Update keywords to include "reranker"

* chore: update pr template with screenshots (#1590)

* Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586)

* Added instructions for running docs server locally (#1578)

- Added NPM script method
- Added VS Code task method
- Update contributing guidelines

* Update branch policy (#1577)

- Change PR target to `dev` branch
- Update `CONTRIBUTING.md` instructions

* Consolidate example configurations into the main configuration guide (#1579)

- Moved examples to configuration.md
- Deleted the separate examples.md file
- Updated sidebar order and links
- Improved readability and structure in configuration.md

* fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582)

* Update completionProvider.ts (warning tab-autocomplete models) (#1566)

* feat: enhanced IndexingProgressBar with blinking dot feature

- Integrated BlinkingDot component
- Added STATUS_COLORS for various states
- Replaced CircleDiv with BlinkingDot in UI
- Updated status messages and layout

* small UI tweaks

* feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595)

- add styling and adjustments to ModelCard
- update ModelProviderTag font size
- remove box shadow from Toggle component
- tweak icon styles in ModelCard
- improve alignment and spacing

* media query

* feat: add best experience onboarding

* fix: file rename

* stop movement on button hover by keeping same border thickness

* fix mistake in setting cursor: pointer

* fix when free trial option is shown

* Support Node.js versions below 20 for streaming response handling (#1591)

- Add fallback for Node < 20
- Implement toAsyncIterable for streaming
- Use TextDecoder for manual decoding
- Maintain existing streaming for Node 20+

* small fixes

* feat: add free trial card to onboarding (#1600)

* feat: add free trial card to onboarding

* add import

* fix hasPassedFTL

* fix /edit cancellation from UI

* feat: add `applyCodeBlock` experimental prop (#1601)

* feat: add new model styling improvements (#1609)

* feat: add new model styling improvements

* better gap size

* feat: update bug_report.yml (#1610)

* chore: update bug_report.yml

* typo fix

* feat: add labels to "Add docs" dialog (#1612)

* feat: add labels to "Add docs" dialog

* remove autofocus

* don't double load config

* small fixes

* speed up directory traversal, and use correct native path module

* option not to show config update toast

* merge air-gapped and recommended setup pages

* chore: add telemetry for full screen toggle (#1618)

* Fix headings in codebase-embeddings.md (#1617)

* mention jetbrains

* docs: update changie  (#1619)

* feat: updated changie config

* hide toc and autogenerate

* Update changelog.mdx

* link to deeper explanation of embeddings models

* ensure target="_blank" for all links in sidebar

* fix gif links in intellij README.md

* don't require rust in dependency installation

* chore: fix padding on gh button (#1620)

* chore: adjust button padding

* Update tasks.json

* escape colons in diff path

* smoother lancedb indexing reporting

* smooth progress updates for indexing

* fix tsc err

* rerank-lite-1

* remove doccs

* basic tests for VS Code extension

* improved testing of VS Code extension

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* 🚑 fix constant warnings when onboarding with Ollama

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* team analytics

* apply control plane settings

* workos auth

* ide protocol get session info

* UI for auth

* profile switching

* small fixes

* updates

* refresh tokens

* updates

* fix tsc errs

* model select in toolbar to make room for profile selector

* prod client id

* link to prod URL

* internal beta option

* profiles change listener

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com>
Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>

* temporary patch for type errs

* Dev (#1689)

* Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671)

On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this
change ⌘+K does not clear the terminal but instead iniates a chord
sequence and waits for the next stroke of the chord.

Co-authored-by: Rob Leidle <rleidle@tesla.com>

* Change treeSitter to cache the Language objects it loads from wasm (#1672)

Without this change, for a repository with 600 typescript files, the
indexer would fail to finish correctly and there would be many of the
following errors in the webview console log:

'Unable to load language for file ${path} RuntimeError: table index is out of bounds'

The following bash will create a repo that reproduces the problem:

current_path="."

for ((i=1; i<=20; i++)); do
    new_folder="folder-$i"
    mkdir -p "$current_path/$new_folder"
    current_path="$current_path/$new_folder"

    for ((a=1; a<=30; a++)); do
      head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts"
    done
done

Co-authored-by: Rob Leidle <rleidle@tesla.com>

* acknowledge sourcemap flag in esbuild.js

* don't run jetbrains-release.yaml on vscode releases

* further testing for walkDir

* chore: add telemetry to commands (#1673)

* test: Add basic unit test to baseLLM (#1668)

* update version

* test: Add basic unit test to baseLLM

---------

Co-authored-by: Nate Sesti <sestinj@gmail.com>
Co-authored-by: inimaz <imaz@ai-verse.com>

* feat: add Quick Actions CodeLens feature (#1674)

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* add voyage rerank-1

* import Handlebars

* feat: open pane on install (#1564)

* feat: open pane on activation

* comment out testing code

* chore: add telemetry for pageviews (#1576)

* feat: update onboarding w/ embeddings model (#1570)

* chore(gui): remove unused pages

* feat: add embeddings step

* feat: update styles

* feat: copy button updates

* fix: correct pull command for embed model

* fix: remove commented code

* fix: remove commented code

* feat: simplify copy btn props

* chore: rename onboarding selection event

* feat: add provider config

* fix: undo msg name

* remove dead code

* fix: invalid mode check

* fix: remove testing logic

* fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582)

* small UI tweaks

* media query

* feat: add best experience onboarding

* small fixes

* feat: add free trial card to onboarding (#1600)

* feat: add free trial card to onboarding

* add import

* chore: add telemetry for full screen toggle (#1618)

* rerank-lite-1

* remove doccs

* basic tests for VS Code extension

* improved testing of VS Code extension

* manually implement stop tokens for hf inference api

* chore: onboarding metrics (#1626)

* fix: pageview tracking

* feat: add onboarding telemetry

* create single `onboardingStatus` type

* improved var naming

* remove console logs

* fix windows performance issue

* rename vscodeExtension.ts

* migration of onboarding variables

* "stash" instead of "delete" in indexing progress

* fix preview.yaml

* also fix main.yaml

* Update troubleshooting.md (#1637)

* feat: add quick actions

* Update index.d.ts

* quick actions mvp

* update docs

* subscribe to vscode change settings

* Update commands.ts

* cleanup

* Update quick-actions.md

* Update VerticalPerLineCodeLensProvider.ts

* resolve feedback

---------

Co-authored-by: Nate Sesti <sestinj@gmail.com>
Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>

* chore: add `isCommandEvent` to command telemetry (#1675)

* chore: add `isCommandEvent` to command telemetry

* Update commands.ts

* Nate/better retrieval (#1677)

* deduplicatearray tests

* break out separate retrieval pipelines

* IConfigHandler

* tests for codebase indexer

* better .continueignore for continue

* indexing fixes

* ignore .gitignore and .continueignore when indexing

* retrieval pipeline improvements

* fix formatting err in out .continueignore

* add necessary filter to lance_db_cache

* update package.json version

* skip unused tests

* don't ignore .prompt files

* update version

* Update pull_request_template.md

* don't use multi-media format when there are multiple text items

* add free trial experience (#1685)

* fix: add code range for quick actions/fixes (#1687)

* fix: add code range for quick actions/fixes

* Update test.js

* add pathSep message type

* docs improvements

* jetbrains fix

* update package.json version

---------

Co-authored-by: Rob Leidle <rleidle@gmail.com>
Co-authored-by: Rob Leidle <rleidle@tesla.com>
Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com>
Co-authored-by: inimaz <imaz@ai-verse.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>

* update package.json version

* skip extension tests

* update package.json version

* handlebars import

* don't push package.json version change from CI

* fix: scroll issues w/ code blocks (#1688)

* fix: scroll issues w/ code blocks

* chore: name offset

* chore: remove docs start

* chore: update headings on troubleshooting.md (#1696)

* ignore .svn folder for indexing (#1699)

* chore: clearer naming for `useSuffix` (#1702)

* chore: clearer naming for `useSuffix`

* schema updates

* fix: cmd+shft+l closes sidebar if focused (#1638)

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* add voyage rerank-1

* import Handlebars

* feat: open pane on install (#1564)

* feat: open pane on activation

* comment out testing code

* chore: add telemetry for pageviews (#1576)

* feat: update onboarding w/ embeddings model (#1570)

* chore(gui): remove unused pages

* feat: add embeddings step

* feat: update styles

* feat: copy button updates

* fix: correct pull command for embed model

* fix: remove commented code

* fix: remove commented code

* feat: simplify copy btn props

* chore: rename onboarding selection event

* feat: add provider config

* fix: undo msg name

* remove dead code

* fix: invalid mode check

* fix: remove testing logic

* fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582)

* small UI tweaks

* media query

* feat: add best experience onboarding

* small fixes

* feat: add free trial card to onboarding (#1600)

* feat: add free trial card to onboarding

* add import

* chore: add telemetry for full screen toggle (#1618)

* rerank-lite-1

* remove doccs

* basic tests for VS Code extension

* improved testing of VS Code extension

* manually implement stop tokens for hf inference api

* chore: onboarding metrics (#1626)

* fix: pageview tracking

* feat: add onboarding telemetry

* create single `onboardingStatus` type

* improved var naming

* remove console logs

* fix windows performance issue

* rename vscodeExtension.ts

* migration of onboarding variables

* "stash" instead of "delete" in indexing progress

* fix preview.yaml

* also fix main.yaml

* Update troubleshooting.md (#1637)

* feat: close panel if main input is focused

* add skip param

* Update TipTapEditor.tsx

* merge dev

* Update commands.ts

---------

Co-authored-by: Nate Sesti <sestinj@gmail.com>
Co-authored-by: Nate Sesti <33237525+sestinj@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>

* bugfix: quick actions undefined array (#1704)

* feat: add rich quick pick for quick edit (#1706)

* feat: multistep quick pick

* fix: multi step for context provider

* Update test.js

* Create deepseek.md (#1708)

* style: add max height to model selector (#1712)

* update styling

* remove testing code

* remove comments

* feat: add tutorial card (#1716)

* feat: create tutorial card

* Update TutorialCard.tsx

* feat: add file search to quick edit (#1714)

* feat: multistep quick pick

* fix: multi step for context provider

* Update test.js

* add file search

* complete file search

* remove unused import

* add comments

* Update utils.ts

* update naming

* chore: move quick pick instantiation (#1723)

* Break out into NPM modules (#1726)

* config.json types

* export all schemas

* break out request logic into packages

* update openai adapters to export body types

* update packaging of npm modules

* update packages

* continue proxy

* mistral

* Fix typo in documentation (#1722)

In  set-up-codestral.md
"Trobleshooting" -> "Troubleshooting"

* update package.json version

* 🪄 model dropdown, control plane (#1692)

* Nate/control plane client (#1691)

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* fix(gui): ctx rendering w/ renderInlineAs: "" (#1541)

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* fix codeqwen autocomplete leading space

* add voyage rerank-1

* feat: `--noEmit` for tsc checks in CI (#1559)

* docs: update CustomContextProvider docs (#1557)

* add stop tokens to qwen prompt

* update docs to reflect 3.5 sonnet being best

* docs: comment out unused providers (#1561)

* import Handlebars

* feat: toast notification for config updates (#1560)

* feat: toast notification for config updates

* feat: only trigger toast on config.json save

* displayRawMarkdown option

* feat: open pane on install (#1564)

* feat: open pane on activation

* comment out testing code

* update to reflect 16 stop words limit for deepseek

* feat: only trigger config update toast in vscode (#1571)

* docs(prompt-files): fix typos + clarify (#1575)

* doc: prompt file typo + clarifications

* fix: add back correct docs

* chore: add telemetry for pageviews (#1576)

* feat: update onboarding w/ embeddings model (#1570)

* chore(gui): remove unused pages

* feat: add embeddings step

* feat: update styles

* feat: copy button updates

* fix: correct pull command for embed model

* fix: remove commented code

* fix: remove commented code

* feat: simplify copy btn props

* chore: rename onboarding selection event

* feat: add provider config

* fix: undo msg name

* remove dead code

* fix: invalid mode check

* fix: remove testing logic

* docs(telemetry): add pageviews to tracking list (#1581)

* Add reranker configuration options to codebase embedding docs (#1584)

- Introduce reranker concept
- List available reranker options
- Provide configuration instructions
- Update keywords to include "reranker"

* chore: update pr template with screenshots (#1590)

* Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586)

* Added instructions for running docs server locally (#1578)

- Added NPM script method
- Added VS Code task method
- Update contributing guidelines

* Update branch policy (#1577)

- Change PR target to `dev` branch
- Update `CONTRIBUTING.md` instructions

* Consolidate example configurations into the main configuration guide (#1579)

- Moved examples to configuration.md
- Deleted the separate examples.md file
- Updated sidebar order and links
- Improved readability and structure in configuration.md

* fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582)

* Update completionProvider.ts (warning tab-autocomplete models) (#1566)

* feat: enhanced IndexingProgressBar with blinking dot feature

- Integrated BlinkingDot component
- Added STATUS_COLORS for various states
- Replaced CircleDiv with BlinkingDot in UI
- Updated status messages and layout

* small UI tweaks

* feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595)

- add styling and adjustments to ModelCard
- update ModelProviderTag font size
- remove box shadow from Toggle component
- tweak icon styles in ModelCard
- improve alignment and spacing

* media query

* feat: add best experience onboarding

* fix: file rename

* stop movement on button hover by keeping same border thickness

* fix mistake in setting cursor: pointer

* fix when free trial option is shown

* Support Node.js versions below 20 for streaming response handling (#1591)

- Add fallback for Node < 20
- Implement toAsyncIterable for streaming
- Use TextDecoder for manual decoding
- Maintain existing streaming for Node 20+

* small fixes

* feat: add free trial card to onboarding (#1600)

* feat: add free trial card to onboarding

* add import

* fix hasPassedFTL

* fix /edit cancellation from UI

* feat: add `applyCodeBlock` experimental prop (#1601)

* feat: add new model styling improvements (#1609)

* feat: add new model styling improvements

* better gap size

* feat: update bug_report.yml (#1610)

* chore: update bug_report.yml

* typo fix

* feat: add labels to "Add docs" dialog (#1612)

* feat: add labels to "Add docs" dialog

* remove autofocus

* don't double load config

* small fixes

* speed up directory traversal, and use correct native path module

* option not to show config update toast

* merge air-gapped and recommended setup pages

* chore: add telemetry for full screen toggle (#1618)

* Fix headings in codebase-embeddings.md (#1617)

* mention jetbrains

* docs: update changie  (#1619)

* feat: updated changie config

* hide toc and autogenerate

* Update changelog.mdx

* link to deeper explanation of embeddings models

* ensure target="_blank" for all links in sidebar

* fix gif links in intellij README.md

* don't require rust in dependency installation

* chore: fix padding on gh button (#1620)

* chore: adjust button padding

* Update tasks.json

* escape colons in diff path

* smoother lancedb indexing reporting

* smooth progress updates for indexing

* fix tsc err

* rerank-lite-1

* remove doccs

* basic tests for VS Code extension

* improved testing of VS Code extension

* docs: add docs and schema for "OS" provider (#1536)

* ignore .env

* 🚑 fix constant warnings when onboarding with Ollama

* ✨ use and cache imports for autocomplete (#1456)

* ✨ use and cache imports for autocomplete

* fix tsc

* team analytics

* apply control plane settings

* workos auth

* ide protocol get session info

* UI for auth

* profile switching

* small fixes

* updates

* refresh tokens

* updates

* fix tsc errs

* model select in toolbar to make room for profile selector

* prod client id

* link to prod URL

* internal beta option

* profiles change listener

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com>
Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>

* temporary patch for type errs

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com>
Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>

* skip extension tests

* JetBrains status bar spinner for autocomplete

* Remove unused images

* partial accept jetbrains

* fix double-rendering of jetbrains completions

* fix offset of multi-line jetbrains completions

* enable/disable actions

* upgrade to version 0.0.54

* fix: wrap editor buttons when out of space (#1727)

* Add .continuerc file to config directory on activation to prevent it's indexing. (#1733)

* Updated the content links that were deprecated in the document (#1709)

Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com>

* Nate/dev (#1739)

* JetBrains status bar spinner for autocomplete

* Remove unused images

* partial accept jetbrains

* fix double-rendering of jetbrains completions

* fix offset of multi-line jetbrains completions

* enable/disable actions

* upgrade to version 0.0.54

* llm-info package

* small profile switching improvements

* instruct cmd+I not to leave placeholders

* @ files in context-providers docs

* onboarding fix

* update jetbrains ci to build gui

* feat: single default quick pick to edit (#1743)

* feat: single default quick pick to edit

* Update quick-actions.md

* Update quick-actions.md

* use defaultQuickAction

* quick fixes

* fix: handle line removal diff (#1744)

* fix: handle line removal diff

* remove comments

* Configurable `maxChunkSize` for embedding providers (#1746)

* Add `maxChunkSize` to `EmbeddingsProvider`

* Add `maxChunkSize` to schema

* add support for docs via config (#1594)

* mistral mamba (#1748)

* 🐍 mistral mamba

* autoscale retrieval based on context length

* autoscale retrieval based on context length

* fix input box movement

* update jetbrains build

* test: add myers diff tests (#1754)

* feat: enhance help center (#1755)

* feat: enhance help center

* Update help.tsx

* feat: add azure provider config (#1764)

* autocomplete reload fix

* logout of control plane

* Update QuickEditQuickPick.ts (#1772)

* Update QuickEditQuickPick.ts

* updates

* Update QuickEditQuickPick.ts

* pass refreshed access token to continue-proxy model

* test: add dir read check for walkDir (#1773)

* test: add dir read check for walkDir

* Update ignore.ts

* Update walkDir.test.ts

* Update walkDir.test.ts

* gpt-4o-mini

* gpt-4o-mini

* gpt4omini to ui selector

* fix: layout alignment (#1779)

* fix: delete old docs index on force re-index (#1778)

* add "Codebase Force Re-Index" command (#1757)

* Add `HuggingFaceTEIReranker` (#1711)

* Add `HuggingFaceTEIReranker`

* Add `huggingface-tei` to `config_schema.json`

* update package.json version

* update config schemas

* skip test failing only due to type error

* fix: convert `walkDir` to an async generator (#1783)

* walkdir to async generator

* add yields

* handle ignore files

* Update CodebaseIndexer.ts

* Update walkDir.ts

* chore: bump VS Code ext version

* fix completions support in ContinueProxy

* add test for walkDir

* IAnalyticsProvider

* feat: crawl `.mdx` docs

* update workspaces on session info change

* rename from profile to workspace (user-facing)

* update config schema files

* feat: improve chat thread ui for better readability (#1786)

* add "language" template variable for autocomplete

* Enable debug logs (#1800)

* Add an enableDebugLogs flag

* Send logs to VSCode

* Refresh index fixes (#1795)

* Use the resultType given to markComplete

This fixes an issue where if there were multiple result types in itemToAction such as an update with UpdateNewVersion and UpdateOldVersion, only the UpdateOldVersion would be hit (which is a noop) and the UpdateNewVersion would never be fired, causing us to never update the timestamp of the tag_catalog.

* Add a Compute and UpdateLastUpdated to refreshIndex

Removes the ability to insert duplicate rows if we're just computing the result. Instead we replace the existing row instead.

* Don't insert compute tags

They are REPLACEd in markComplete now.

* docs: change page heading to model providers (#1684)

* feat: allow JetBrains users to index docs (#1797)

* feat: allow JetBrains users to index docs

* update docs service

* update global context

* throw error popup on jetbrains + transformers

* update toast notifications

* update typings

* fix tsc errors

* Index on file change (#1785)

* Reload the submenu items when the context provider data is refreshed

* update version

* Just call refreshSubmenuItems

* remove log statements

* cleanup

* Remove the Promise declaration

* Copy currentFiles to a new dict since we're going to mutate it

Don't mutate the caller's currentFiles unexpectedly

* Remove reIndexFile and just call index/forceReIndex

* We MUST await the refreshCodebaseIndex call

* Expose send on core

* remove unused import

* Reindex the workspaces on save

* Call core.send

Ideally, we can implement externalRequest on the webview later so we don't have to expose messenger to the VSCode extension

* Always refreshSubmenuItems in refreshCodebaseIndex

* whitespace

* Change messenger back to private

* Add Pause Codebase On Start setting (#1788)

* Add a skipInitialSync function

* Add pauseInitialCodebaseIndex

* docs

* version bump

* Rename to pauseCodebaseIndexOnStart

* Un-shadow file context provider (#1801)

* allow FileContextProvider to handle files

* remove commented-out lines

* chore: add telemetry to alt-enter (#1805)

* Set the indexProgress indicator to paused on startup if pauseCodebaseIndexOnStart is set (#1804)

* Fix Input Toolbar layout (#1792)

* set to flex flex-column

* Remove fixed height and always show the StyledDivs content to hold its size

* Add a Delete Message button to the StepContainer (#1790)

* Add a delete button

* add a delete message action to the GUI

* Fix off by one error

* format

* This index should be less than length

* Update to trash icon

* Simplify walkDir(...) and improve performance by roughly 10x in larger repos (#1806)

The performance of WalkDir is improved by replacing the recursive DFS
with a stack based implementation (less memory in-use during peak
exploration). In addition, some unnecessary flags and options were
removed which allowed the code to be overall simplified. Much less rule
matching is occuring with the same files returned.

Some performance results on my linux host:

small repo:
- 649 files returned by walkDir()
- 9 .gitignore files
- walkDir() old vs new duration: 918 ms to 143 ms

medium sized repo:
- 9271 returned by walkDir()
- 35 .gitignore files
- walkDir() old vs new duration: 10,886 ms to 970ms

large sized repo:
- 197,900 files returned by walkDir()
- 929 .gitignore files
- walkDir() old vs new duration: 498,151 ms to 37,353 ms

Co-authored-by: Rob Leidle <rleidle@tesla.com>

* modified assistant indicator in UI

* update package.json version

* feat: add Llama 3.1 8B to cloudflare provider options (#1811)

Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com>

* increase gemini pro contextLength (#1809)

* fix indentation (#1808)

* Llama 3.1 405b model selection (#1813)

* add support for llama3 405b

* 3.1

* name mapping

* updates to model setup

* fix walkDir tests on windows

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
Co-authored-by: Priyash <38959321+priyashpatil@users.noreply.github.com>
Co-authored-by: Jonah Wagner <jonahwagner231@gmail.com>
Co-authored-by: YohannZe <99359799+YohannZe@users.noreply.github.com>
Co-authored-by: Dan Dascalescu <ddascalescu+github@gmail.com>
Co-authored-by: Rob Leidle <rleidle@gmail.com>
Co-authored-by: Rob Leidle <rleidle@tesla.com>
Co-authored-by: inimaz <49730431+inimaz@users.noreply.github.com>
Co-authored-by: inimaz <imaz@ai-verse.com>
Co-authored-by: Anaël Bonnafous <71386173+AnaelBonnafous@users.noreply.github.com>
Co-authored-by: Huy Tran <119860259+tranquochuy645@users.noreply.github.com>
Co-authored-by: Gabriel Gordbegli <ggordbegli@gmail.com>
Co-authored-by: Raven-1027 <83693755+Raven-1027@users.noreply.github.com>
Co-authored-by: Ihe Fan <ih.fan.1027@gmail.com>
Co-authored-by: Lukas Kreussel <65088241+LLukas22@users.noreply.github.com>
Co-authored-by: Josh Vera <josh@joshvera.com>
Co-authored-by: Peter Zaback <pzaback@gmail.com>
Co-authored-by: James Delorey <deloreyjmi@gmail.com>
Co-authored-by: jdelorey@cloudflare.com <jdelorey@cloudflare.com>
  • Loading branch information
20 people authored Jul 23, 2024
1 parent 82db97e commit 805984f
Show file tree
Hide file tree
Showing 18 changed files with 421 additions and 225 deletions.
5 changes: 0 additions & 5 deletions core/autocomplete/templates.ts
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,6 @@ const deepseekFimTemplate: AutocompleteTemplate = {
},
};

const deepseekFimTemplateWrongPipeChar: AutocompleteTemplate = {
template: "<|fim▁begin|>{{{prefix}}}<|fim▁hole|>{{{suffix}}}<|fim▁end|>",
completionOptions: { stop: ["<|fim▁begin|>", "<|fim▁hole|>", "<|fim▁end|>"] },
};

const gptAutocompleteTemplate: AutocompleteTemplate = {
template: `\`\`\`
{{{prefix}}}[BLANK]{{{suffix}}}
Expand Down
9 changes: 4 additions & 5 deletions core/indexing/walkDir.ts
Original file line number Diff line number Diff line change
Expand Up @@ -220,12 +220,11 @@ export async function walkDir(
for await (const e of dfsWalker.walk()) {
relativePaths.push(e);
}
if (options?.returnRelativePaths) {
return relativePaths;
}
const pathSep = await ide.pathSep();
const prefix = options.returnRelativePaths ? "" : path + pathSep;

if (pathSep === "/") {
return relativePaths.map((p) => path + pathSep + p);
return relativePaths.map((p) => prefix + p);
}
return relativePaths.map((p) => path + pathSep + p.split("/").join(pathSep));
return relativePaths.map((p) => prefix + p.split("/").join(pathSep));
}
1 change: 0 additions & 1 deletion core/llm/llms/Cloudflare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ export default class Cloudflare extends BaseLLM {
});

for await (const value of streamSse(resp)) {
console.log(value);
if (value.choices?.[0]?.delta?.content) {
yield value.choices[0].delta;
}
Expand Down
3 changes: 2 additions & 1 deletion core/llm/llms/FreeTrial.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,8 @@ class FreeTrial extends BaseLLM {
return [
"codestral-latest",
"claude-3-5-sonnet-20240620",
"llama3-70b",
"llama3.1-405b",
"llama3.1-70b",
"gpt-4o",
"gpt-3.5-turbo",
"claude-3-haiku-20240307",
Expand Down
3 changes: 3 additions & 0 deletions core/llm/llms/Groq.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ class Groq extends OpenAI {
gemma: "gemma-7b-it",
"llama3-8b": "llama3-8b-8192",
"llama3-70b": "llama3-70b-8192",
"llama3.1-8b": "llama-3.1-8b-instant",
"llama3.1-70b": "llama-3.1-70b-versatile",
"llama3.1-405b": "llama-3.1-405b-reasoning",
};
protected _convertModelName(model: string): string {
return Groq.modelConversion[model] ?? model;
Expand Down
3 changes: 3 additions & 0 deletions core/llm/llms/Ollama.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,9 @@ class Ollama extends BaseLLM {
"codellama-70b": "codellama:70b",
"llama3-8b": "llama3:8b",
"llama3-70b": "llama3:70b",
"llama3.1-8b": "llama3.1:8b",
"llama3.1-70b": "llama3.1:70b",
"llama3.1-405b": "llama3.1:405b",
"phi-2": "phi:2.7b",
"phind-codellama-34b": "phind-codellama:34b-v2",
"wizardcoder-7b": "wizardcoder:7b-python",
Expand Down
3 changes: 3 additions & 0 deletions core/llm/llms/Replicate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ class Replicate extends BaseLLM {
"llama2-13b": "meta/llama-2-13b-chat" as any,
"llama3-8b": "meta/meta-llama-3-8b-instruct" as any,
"llama3-70b": "meta/meta-llama-3-70b-instruct" as any,
"llama3.1-8b": "meta/meta-llama-3.1-8b-instruct" as any,
"llama3.1-70b": "meta/meta-llama-3.1-70b-instruct" as any,
"llama3.1-405b": "meta/meta-llama-3.1-405b-instruct" as any,
"zephyr-7b":
"nateraw/zephyr-7b-beta:b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931",
"mistral-7b":
Expand Down
3 changes: 3 additions & 0 deletions core/llm/llms/Together.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ class Together extends OpenAI {
"codellama-70b": "codellama/CodeLlama-70b-Instruct-hf",
"llama3-8b": "meta-llama/Llama-3-8b-chat-hf",
"llama3-70b": "meta-llama/Llama-3-70b-chat-hf",
"llama3.1-8b": "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
"llama3.1-70b": "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
"llama3.1-405b": "meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo",
"llama2-7b": "togethercomputer/llama-2-7b-chat",
"llama2-13b": "togethercomputer/llama-2-13b-chat",
"llama2-70b": "togethercomputer/llama-2-70b-chat",
Expand Down
7 changes: 7 additions & 0 deletions core/test/walkDir.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ async function expectPaths(
toNotExist: string[],
options?: WalkerOptions,
) {
// Convert to Windows paths
const pathSep = await ide.pathSep();
if (pathSep === "\\") {
toExist = toExist.map((p) => p.replace(/\//g, "\\"));
toNotExist = toNotExist.map((p) => p.replace(/\//g, "\\"));
}

const result = await walkTestDir(options);

for (const p of toExist) {
Expand Down
25 changes: 24 additions & 1 deletion docs/static/schemas/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,7 @@
"anyOf": [
{
"enum": [
"@cf/meta/llama-3.1-8b-instruct",
"@cf/meta/llama-3-8b-instruct",
"@hf/thebloke/deepseek-coder-6.7b-instruct-awq",
"@cf/deepseek-ai/deepseek-math-7b-instruct",
Expand Down Expand Up @@ -513,6 +514,9 @@
"anyOf": [
{
"enum": [
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"llama3-8b",
"llama3-70b",
"codellama-7b",
Expand Down Expand Up @@ -546,7 +550,8 @@
"enum": [
"gpt-4o",
"codestral-latest",
"llama3-70b",
"llama3.1-70b",
"llama3.1-405b",
"gpt-3.5-turbo",
"phind-codellama-34b",
"gemini-pro",
Expand Down Expand Up @@ -598,6 +603,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -748,6 +756,9 @@
"llama2-13b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"codellama-7b",
"codellama-13b",
"codellama-34b",
Expand Down Expand Up @@ -812,6 +823,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -860,6 +874,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phi-2",
"phind-codellama-34b",
"wizardcoder-7b",
Expand Down Expand Up @@ -947,6 +964,9 @@
"gemma",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"AUTODETECT"
]
}
Expand Down Expand Up @@ -1113,6 +1133,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down
24 changes: 23 additions & 1 deletion extensions/intellij/src/main/resources/config_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,9 @@
"anyOf": [
{
"enum": [
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"llama3-8b",
"llama3-70b",
"codellama-7b",
Expand Down Expand Up @@ -546,7 +549,8 @@
"enum": [
"gpt-4o",
"codestral-latest",
"llama3-70b",
"llama3.1-70b",
"llama3.1-405b",
"gpt-3.5-turbo",
"phind-codellama-34b",
"gemini-pro",
Expand Down Expand Up @@ -598,6 +602,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -748,6 +755,9 @@
"llama2-13b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"codellama-7b",
"codellama-13b",
"codellama-34b",
Expand Down Expand Up @@ -812,6 +822,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -860,6 +873,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phi-2",
"phind-codellama-34b",
"wizardcoder-7b",
Expand Down Expand Up @@ -947,6 +963,9 @@
"gemma",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"AUTODETECT"
]
}
Expand Down Expand Up @@ -1113,6 +1132,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down
24 changes: 23 additions & 1 deletion extensions/vscode/config_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,9 @@
"anyOf": [
{
"enum": [
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"llama3-8b",
"llama3-70b",
"codellama-7b",
Expand Down Expand Up @@ -546,7 +549,8 @@
"enum": [
"gpt-4o",
"codestral-latest",
"llama3-70b",
"llama3.1-70b",
"llama3.1-405b",
"gpt-3.5-turbo",
"phind-codellama-34b",
"gemini-pro",
Expand Down Expand Up @@ -598,6 +602,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -748,6 +755,9 @@
"llama2-13b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"codellama-7b",
"codellama-13b",
"codellama-34b",
Expand Down Expand Up @@ -812,6 +822,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down Expand Up @@ -860,6 +873,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phi-2",
"phind-codellama-34b",
"wizardcoder-7b",
Expand Down Expand Up @@ -947,6 +963,9 @@
"gemma",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"AUTODETECT"
]
}
Expand Down Expand Up @@ -1113,6 +1132,9 @@
"codellama-70b",
"llama3-8b",
"llama3-70b",
"llama3.1-8b",
"llama3.1-70b",
"llama3.1-405b",
"phind-codellama-34b",
"wizardcoder-7b",
"wizardcoder-13b",
Expand Down
Loading

0 comments on commit 805984f

Please sign in to comment.