Code

Add WASM/JS bindings#50

Code

Open

dqbd wants to merge 237 commits intoopenai:mainopenai/tiktoken:mainfrom dqbd:maindqbd/tiktoken:mainCopy head branch name to clipboard

+9,493-523

Commits

Commits on Feb 8, 2023

• refactor

  eisber

  committed
  6c49a41

  Copy full SHA for 6c49a41

Commits on Feb 9, 2023

• ported loading to Rust

  eisber

  committed
  09dada4

  Copy full SHA for 09dada4

• fixed?

  eisber

  committed
  4c6afcb

  Copy full SHA for 4c6afcb

• Propagate errors from split whitespace in bpe_merges

  jackgerrits

  committed
  502a6ec

  Copy full SHA for 502a6ec

• moving registry into rust

  eisber

  committed
  c685d99

  Copy full SHA for c685d99

• return ref

  jackgerrits

  committed
  6dcfa91

  Copy full SHA for 6dcfa91

Commits on Feb 10, 2023

• split into multiple libs

  eisber

  committed
  25306a5

  Copy full SHA for 25306a5

• fixed core

  eisber

  committed
  ba69bd9

  Copy full SHA for ba69bd9

• Fix python setup.py for package in workspace

  jackgerrits

  committed
  c98f824

  Copy full SHA for c98f824

• java bindings...

  eisber

  committed
  f248694

  Copy full SHA for f248694

• more TODO

  eisber

  committed
  2f371f6

  Copy full SHA for 2f371f6

Commits on Feb 11, 2023

• Initial VERY rough impl of JNI layer

  jackgerrits

  committed
  1d3f707

  Copy full SHA for 1d3f707

Commits on Feb 19, 2023

• Add wasm-bindgen, inline ranks

  dqbd

  committed
  95aef95

  Copy full SHA for 95aef95

• v0.2.1

  dqbd

  committed
  799df7f

  Copy full SHA for 799df7f

• Update README.md, polish API

  dqbd

  committed
  5a32b88

  Copy full SHA for 5a32b88

Commits on Feb 22, 2023

• improve error handling

  eisber

  committed
  97d8dea

  Copy full SHA for 97d8dea

Commits on Feb 23, 2023

• chore: add README

  dqbd

  committed
  3139d04

  Copy full SHA for 3139d04

• feat: add option to extend special tokens and to provide custom bfe

  dqbd

  committed
  02d132e

  Copy full SHA for 02d132e

• Merge pull request #2 from dqbd/extendability

  Show description for 46668df
  dqbd

  authored
  46668df

  Copy full SHA for 46668df

• Bump version, update README

  dqbd

  committed
  3a39f24

  Copy full SHA for 3a39f24

• Improve error handling, add support for parameters

  dqbd

  committed
  d0dc32b

  Copy full SHA for d0dc32b

• Fix any in TS files, add core tests

  dqbd

  committed
  90ee9f6

  Copy full SHA for 90ee9f6

Commits on Feb 24, 2023

• Update README.md, add tests, fix disallowed special bug

  dqbd

  committed
  f7fe717

  Copy full SHA for f7fe717

• Validate the values properly

  dqbd

  committed
  d38c936

  Copy full SHA for d38c936

• Improve error handling in JNI functions

  jackgerrits

  committed
  53628a4

  Copy full SHA for 53628a4

• Bump version to 0.4.0

  dqbd

  committed
  87603e9

  Copy full SHA for 87603e9

Commits on Feb 27, 2023

• Merge pull request #1 from eisber/error_handling

  Show description for 42548d8
  eisber

  authored
  42548d8

  Copy full SHA for 42548d8

• moved config into json

  eisber

  committed
  e3ab3f6

  Copy full SHA for e3ab3f6

• add github action

  eisber

  committed
  22584d4

  Copy full SHA for 22584d4

• add jar build

  eisber

  committed
  42efde4

  Copy full SHA for 42efde4

• fix rust build

  eisber

  committed
  484f15a

  Copy full SHA for 484f15a

• build jar

  eisber

  committed
  6c45bf6

  Copy full SHA for 6c45bf6

• fix path

  Show description for b883d0c
  eisber

  committed
  b883d0c

  Copy full SHA for b883d0c

• fix java

  eisber

  committed
  98070f1

  Copy full SHA for 98070f1

• Merge remote-tracking branch 'upstream/main' into main

  eisber

  committed
  3b39f6b

  Copy full SHA for 3b39f6b

Commits on Feb 28, 2023

• cleanup

  eisber

  committed
  4e70da0

  Copy full SHA for 4e70da0

• update groupid

  eisber

  committed
  381d32b

  Copy full SHA for 381d32b

• remove comments

  eisber

  committed
  f1560bd

  Copy full SHA for f1560bd

Commits on Mar 1, 2023

• Move to separate js folder

  dqbd

  committed
  01d4f9e

  Copy full SHA for 01d4f9e

• Merge branch 'eisber/main'

  dqbd

  committed
  4d8f9af

  Copy full SHA for 4d8f9af

• Make sure JS builds

  dqbd

  committed
  ef77b1a

  Copy full SHA for ef77b1a

Commits on Mar 2, 2023

• Attempt to fix sdist

  dqbd

  committed
  b04f0cf

  Copy full SHA for b04f0cf

• Match sdist

  dqbd

  committed
  bbcb591

  Copy full SHA for bbcb591

• Remove the _js suffix

  dqbd

  committed
  d1c4af2

  Copy full SHA for d1c4af2

• Update to newer build wheels

  dqbd

  committed
  33207e6

  Copy full SHA for 33207e6

Commits on Mar 5, 2023

• Fix wrong result for None

  dqbd

  committed
  2370284

  Copy full SHA for 2370284

• Add CI step to build and test

  dqbd

  committed
  98ac953

  Copy full SHA for 98ac953

• CI: install initialize wasm-pack

  dqbd

  committed
  d989c22

  Copy full SHA for d989c22

• Fix Python CI

  dqbd

  committed
  01bf979

  Copy full SHA for 01bf979

• Replace yum with apt

  dqbd

  committed
  82cd413

  Copy full SHA for 82cd413

• Try yum once again

  dqbd

  committed
  7db26cb

  Copy full SHA for 7db26cb

• debug ci

  dqbd

  committed
  08403a1

  Copy full SHA for 08403a1

• Try CI again

  dqbd

  committed
  12805e3

  Copy full SHA for 12805e3

• Fix CI again

  dqbd

  committed
  ae30c13

  Copy full SHA for ae30c13

• invalid -y command

  dqbd

  committed
  dabb296

  Copy full SHA for dabb296

• Revert "debug ci"

  Show description for 8e40682
  dqbd

  committed
  8e40682

  Copy full SHA for 8e40682

• Optimize performace, enable/disable features of core

  dqbd

  committed
  4866cf9

  Copy full SHA for 4866cf9

Commits on Mar 7, 2023

• Add custom initialisation, bundler changes

  dqbd

  committed
  33bb13d

  Copy full SHA for 33bb13d

Commits on Mar 8, 2023

• Flatten the structure

  dqbd

  committed
  03cdbb3

  Copy full SHA for 03cdbb3

• Run WASM build sequentially in CI for now

  dqbd

  committed
  7c8cd78

  Copy full SHA for 7c8cd78

• Rename TiktokenEmbedding to TiktokenEncoding

  Show description for fcb52cf
  dqbd

  committed
  fcb52cf

  Copy full SHA for fcb52cf

• Reverse order of default

  dqbd

  committed
  945d4f2

  Copy full SHA for 945d4f2

• Merge pull request #12 from dqbd/async-init-bundler

  Show description for 9ea421b
  dqbd

  authored
  9ea421b

  Copy full SHA for 9ea421b

• Update README.md

  dqbd

  committed
  680fbc5

  Copy full SHA for 680fbc5

• Merge remote-tracking branch 'upstream/main'

  dqbd

  committed
  7c75b04

  Copy full SHA for 7c75b04

• Add gpt-3.5-turbo support in types and matchers

  dqbd

  committed
  c1d11fb

  Copy full SHA for c1d11fb

• Add README.md

  dqbd

  committed
  268bc5c

  Copy full SHA for 268bc5c

• Merge pull request #13 from dqbd/gpt-3.5-turbo

  Show description for 9f9ad0d
  dqbd

  authored
  9f9ad0d

  Copy full SHA for 9f9ad0d

Commits on Mar 10, 2023

• Add caveats for CFW

  dqbd

  committed
  15dd0f2

  Copy full SHA for 15dd0f2

Commits on Mar 11, 2023

• Create a lite build which defers loading of weights to consumers

  dqbd

  committed
  2c7e0e4

  Copy full SHA for 2c7e0e4

• Fix README.md

  dqbd

  committed
  586d205

  Copy full SHA for 586d205

• Cleanup

  dqbd

  committed
  6d8c1dc

  Copy full SHA for 6d8c1dc

• Expose loading script

  dqbd

  committed
  64178e5

  Copy full SHA for 64178e5

• Add polyfill for Buffer.from

  dqbd

  committed
  17dd0ba

  Copy full SHA for 17dd0ba

• Merge pull request #15 from dqbd/lite-build

  Show description for 4f6745f
  dqbd

  authored
  4f6745f

  Copy full SHA for 4f6745f

• Fix exports for CJS (Node ESM)

  dqbd

  committed
  e6f0726

  Copy full SHA for e6f0726

• Remove bundler, as it is unnecessary

  dqbd

  committed
  1ee26c3

  Copy full SHA for 1ee26c3

• Add disclaimer for CFW

  dqbd

  committed
  8984ea7

  Copy full SHA for 8984ea7

Commits on Mar 14, 2023

• Add support for GPT-4

  dqbd

  committed
  9c8caec

  Copy full SHA for 9c8caec

Commits on Mar 15, 2023

• Expose model_to_encoding.json and registry.json

  dqbd

  committed
  103d010

  Copy full SHA for 103d010

• Bump to 1.0.0-alpha.5

  dqbd

  committed
  79ac036

  Copy full SHA for 79ac036

• Fix lite crash, add README.md

  dqbd

  committed
  f748754

  Copy full SHA for f748754

• Update README.md

  dqbd

  committed
  bfe3817

  Copy full SHA for bfe3817

• Bump to 1.0.0-alpha.6

  dqbd

  committed
  9847f4d

  Copy full SHA for 9847f4d

• Add JSON importable modules

  dqbd

  committed
  c01af19

  Copy full SHA for c01af19

• Bump to 1.0.0-alpha.7

  dqbd

  committed
  23bb57d

  Copy full SHA for 23bb57d

• Bump to 1.0.0-alpha.8

  dqbd

  committed
  a3baa6c

  Copy full SHA for a3baa6c

• Compress ranks

  dqbd

  committed
  6ac1a1a

  Copy full SHA for 6ac1a1a

• Use compressed version to make main WASM smaller

  dqbd

  committed
  7efba72

  Copy full SHA for 7efba72

• Bump to 1.0.0-alpha.10

  dqbd

  committed
  4d4b921

  Copy full SHA for 4d4b921

• Update README.md

  dqbd

  committed
  efe3728

  Copy full SHA for efe3728

• Bump to 1.0.0

  dqbd

  committed
  8476eca

  Copy full SHA for 8476eca

Commits on Mar 16, 2023

• Fix issues with duplicate initialization

  dqbd

  committed
  e1c4313

  Copy full SHA for e1c4313

• Bump to 1.0.1

  dqbd

  committed
  c4dfaad

  Copy full SHA for c4dfaad

• Clarifies usage

  Show description for 68efa86
  dqbd

  committed
  68efa86

  Copy full SHA for 68efa86

• Bump to 1.0.2

  dqbd

  committed
  481fb45

  Copy full SHA for 481fb45

Commits on Mar 19, 2023

• Update README.md with information about Create React App

  dqbd

  committed
  6823b8e

  Copy full SHA for 6823b8e

• Add custom exception when not initialized

  dqbd

  committed
  b9a03a7

  Copy full SHA for b9a03a7

Commits on Apr 4, 2023

• Add gpt-4 to model_to_encoding.json

  christophwitzko

  authored
  6bba615

  Copy full SHA for 6bba615

• Update lib.rs

  christophwitzko

  authored
  10e17c3

  Copy full SHA for 10e17c3

• Merge pull request #25 from christophwitzko/patch-1

  Show description for a3c88a5
  dqbd

  authored
  a3c88a5

  Copy full SHA for a3c88a5

Commits on Apr 5, 2023

• Merge remote-tracking branch 'upstream/main'

  dqbd

  committed
  ee98f78

  Copy full SHA for ee98f78

• Bump to 1.0.3

  dqbd

  committed
  7261f28

  Copy full SHA for 7261f28

Commits on Apr 8, 2023

• Fix Next.js + Webpack import issue due to missing exports entry

  dqbd

  committed
  ff230c8

  Copy full SHA for ff230c8

• Bump to 1.0.4

  dqbd

  committed
  83c2a85

  Copy full SHA for 83c2a85

Commits on Apr 11, 2023

• Reimplement node_modules detection for Next 13 appDir

  dqbd

  committed
  d40517f

  Copy full SHA for d40517f

• Bump to 1.0.5

  dqbd

  committed
  496168c

  Copy full SHA for 496168c

• Fix broken resolution when used in tests

  dqbd

  committed
  0c0cf00

  Copy full SHA for 0c0cf00

• Bump to 1.0.6

  dqbd

  committed
  26bf591

  Copy full SHA for 26bf591

Commits on Apr 28, 2023

• Use override for edge-light / NextJS appHandlers

  dqbd

  committed
  33405e9

  Copy full SHA for 33405e9

• Publish 1.0.7-alpha.0 build

  dqbd

  committed
  a54dd57

  Copy full SHA for a54dd57

• Bump to 1.0.7

  dqbd

  committed
  ea11d75

  Copy full SHA for ea11d75

Commits on May 8, 2023

• Remove Java bindings

  dqbd

  committed
  754a378

  Copy full SHA for 754a378

• Merge remote-tracking branch 'upstream/main'

  dqbd

  committed
  fe721cd

  Copy full SHA for fe721cd

• Replace README.md for clarity

  dqbd

  committed
  9390c03

  Copy full SHA for 9390c03

• Remove duplicate README.md

  dqbd

  committed
  b8e7817

  Copy full SHA for b8e7817

Commits on May 10, 2023

• Add weak refs

  dqbd

  committed
  83de511

  Copy full SHA for 83de511

• Add changesets folder

  dqbd

  committed
  243c061

  Copy full SHA for 243c061

Commits on May 12, 2023

• Move WASM bindings to wasm folder

  dqbd

  committed
  67d21c3

  Copy full SHA for 67d21c3

• Add JS port

  dqbd

  committed
  150cf05

  Copy full SHA for 150cf05

• Rename WASM workflows

  dqbd

  committed
  fdf6121

  Copy full SHA for fdf6121

Commits on May 13, 2023

• Generate ranks

  dqbd

  committed
  6b13f93

  Copy full SHA for 6b13f93

• Monorepo changes

  dqbd

  committed
  7e5ab35

  Copy full SHA for 7e5ab35

• Fix actions

  dqbd

  committed
  1bf31d3

  Copy full SHA for 1bf31d3

• More descriptive name

  dqbd

  committed
  1845548

  Copy full SHA for 1845548

• Add CDN hosting of ranks

  dqbd

  committed
  3debe21

  Copy full SHA for 3debe21

• Fix sdist

  dqbd

  committed
  990d892

  Copy full SHA for 990d892

• Rename js to tiktoken-js, rename wasm to tiktoken

  dqbd

  committed
  81bd689

  Copy full SHA for 81bd689

• Rename description

  dqbd

  committed
  f245e69

  Copy full SHA for f245e69

• Add publish CI step

  dqbd

  committed
  6b00942

  Copy full SHA for 6b00942

• Normalise workflows

  dqbd

  committed
  7c6a304

  Copy full SHA for 7c6a304

• Rename package once again

  dqbd

  committed
  975ed0f

  Copy full SHA for 975ed0f

Commits on May 14, 2023

• fix: typo

  rpidanny

  authored
  14f108c

  Copy full SHA for 14f108c

• Split into smaller files

  dqbd

  committed
  3240a8e

  Copy full SHA for 3240a8e

• Merge pull request #40 from rpidanny/fix-typo

  Show description for dd6a6ef
  dqbd

  authored
  dd6a6ef

  Copy full SHA for dd6a6ef

Commits on May 15, 2023

• Expose model to encoding mapping, fix types

  dqbd

  committed
  2d18fcc

  Copy full SHA for 2d18fcc

• Bump js-tiktoken to 1.0.1

  dqbd

  committed
  1da0b9d

  Copy full SHA for 1da0b9d

• Add missing main and types for compat

  dqbd

  committed
  621ae35

  Copy full SHA for 621ae35

• Add disallowed tokens

  dqbd

  committed
  de929d4

  Copy full SHA for de929d4

• Version Packages

  github-actions[bot]

  committed
  dba22a9

  Copy full SHA for dba22a9

• Merge pull request #42 from dqbd/changeset-release/main

  Show description for 97fe465
  dqbd

  authored
  97fe465

  Copy full SHA for 97fe465

• Expose ranks

  dqbd

  committed
  265e01d

  Copy full SHA for 265e01d

• Bump to 1.0.4

  dqbd

  committed
  7fbd899

  Copy full SHA for 7fbd899

• Bump js-tiktoken to 1.0.5

  Show description for 4d3f97f
  dqbd

  committed
  4d3f97f

  Copy full SHA for 4d3f97f

Commits on May 17, 2023

• Bump to 1.0.6

  dqbd

  committed
  1fcb052

  Copy full SHA for 1fcb052

Commits on Jun 2, 2023

• Add disclaimer about Svelte + Cloudflare Workers

  dqbd

  committed
  3f2c59c

  Copy full SHA for 3f2c59c

• Clarify the difference between js-tiktoken and tiktoken

  dqbd

  committed
  c4a6180

  Copy full SHA for c4a6180

• Update README.md

  dqbd

  committed
  c7faff0

  Copy full SHA for c7faff0

Commits on Jun 14, 2023

• Add new models

  Show description for ebfc814
  rodumani

  committed
  ebfc814

  Copy full SHA for ebfc814

Commits on Jun 15, 2023

• Merge remote-tracking branch 'upstream/main'

  dqbd

  committed
  a5fb1f4

  Copy full SHA for a5fb1f4

• Merge pull request #51 from rodumani/main

  Show description for d1f8f9f
  dqbd

  authored
  d1f8f9f

  Copy full SHA for d1f8f9f

• Merge remote-tracking branch 'origin/main'

  dqbd

  committed
  844343f

  Copy full SHA for 844343f

• Add changeset

  dqbd

  committed
  baca5b1

  Copy full SHA for baca5b1

• Version Packages

  github-actions[bot]

  committed
  a25079d

  Copy full SHA for a25079d

• Merge pull request #52 from dqbd/changeset-release/main

  Show description for 0d7e051
  dqbd

  authored
  0d7e051

  Copy full SHA for 0d7e051

Commits on Jun 16, 2023

• feat: Add missing models

  kdwkr

  authored
  5a7be3c

  Copy full SHA for 5a7be3c

• Update lib.rs

  kdwkr

  authored
  1bc9ada

  Copy full SHA for 1bc9ada

Commits on Jun 22, 2023

• Merge pull request #53 from kdwkr/patch-1

  Show description for b75f172
  dqbd

  authored
  b75f172

  Copy full SHA for b75f172

• Bump tiktoken to 1.0.9

  dqbd

  committed
  3310bfe

  Copy full SHA for 3310bfe

• Remove python CI steps for now, write .npmrc in actions

  dqbd

  committed
  15b8a8b

  Copy full SHA for 15b8a8b

• Use NODE_AUTH_TOKEN

  dqbd

  committed
  71db434

  Copy full SHA for 71db434

Commits on Jun 24, 2023

• Remove prefix from folder

  Show description for 8e776d3
  dqbd

  committed
  8e776d3

  Copy full SHA for 8e776d3

• Add changeset

  dqbd

  committed
  a691152

  Copy full SHA for a691152

• Merge pull request #57 from dqbd/prefix-fix

  Show description for b1260d9
  dqbd

  authored
  b1260d9

  Copy full SHA for b1260d9

• Version Packages

  github-actions[bot]

  committed
  41bb68a

  Copy full SHA for 41bb68a

• Merge pull request #58 from dqbd/changeset-release/main

  Show description for c86963f
  dqbd

  authored
  c86963f

  Copy full SHA for c86963f

Commits on Aug 5, 2023

• Add Electron setup instructions to README

  nikwen

  authored
  82d44e1

  Copy full SHA for 82d44e1

Commits on Aug 7, 2023

• Merge pull request #66 from nikwen/electron-readme

  Show description for 072dd12
  dqbd

  authored
  072dd12

  Copy full SHA for 072dd12

Commits on Sep 19, 2023

• feat: add gpt3.5-turbo-instruct model

  Show description for db9a804
  Prince-Mendiratta

  committed
  db9a804

  Copy full SHA for db9a804

Commits on Oct 1, 2023

• Merge remote-tracking branch 'upstream/main'

  dqbd

  committed
  4a91a81

  Copy full SHA for 4a91a81

Commits on Nov 15, 2023

• Add new encodings

  dqbd

  committed
  4cda379

  Copy full SHA for 4cda379

• Update WASM and JS to add new models

  dqbd

  committed
  32882f7

  Copy full SHA for 32882f7

• Update examples

  dqbd

  committed
  10302a2

  Copy full SHA for 10302a2

• Add new changeset

  dqbd

  committed
  bd3a360

  Copy full SHA for bd3a360

• Version Packages

  github-actions[bot]

  committed
  20d610f

  Copy full SHA for 20d610f

• Merge pull request #81 from dqbd/changeset-release/main

  Show description for 74c147e
  dqbd

  authored
  74c147e

  Copy full SHA for 74c147e

Commits on Jan 26, 2024

• Add new models

  risu729

  committed
  ff9c637

  Copy full SHA for ff9c637

Commits on Jan 28, 2024

• Merge pull request #86 from risu729/main

  Show description for e1b2472
  dqbd

  authored
  e1b2472

  Copy full SHA for e1b2472

• Merge pull request #71 from Prince-Mendiratta/main

  Show description for 5414d83
  dqbd

  authored
  5414d83

  Copy full SHA for 5414d83

• Fix build

  dqbd

  committed
  221a30e

  Copy full SHA for 221a30e

• Add support for new models, instruct models

  dqbd

  committed
  43e99b4

  Copy full SHA for 43e99b4

• Version Packages

  github-actions[bot]

  committed
  708f77c

  Copy full SHA for 708f77c

• Merge pull request #87 from dqbd/changeset-release/main

  Show description for a020c6a
  dqbd

  authored
  a020c6a

  Copy full SHA for a020c6a

• Fix invalid model

  dqbd

  committed
  de6d1c7

  Copy full SHA for de6d1c7

• Backport changes to @dqbd/tiktoken

  dqbd

  committed
  4491997

  Copy full SHA for 4491997

• Fix typings for wasm

  dqbd

  committed
  e3fcb12

  Copy full SHA for e3fcb12

• Fix typo

  dqbd

  committed
  f39912b

  Copy full SHA for f39912b

• Fix invalid model when requesting an instruct model

  dqbd

  committed
  d54f04c

  Copy full SHA for d54f04c

• Version Packages

  github-actions[bot]

  committed
  f6b2819

  Copy full SHA for f6b2819

• Merge pull request #89 from dqbd/changeset-release/main

  Show description for ffa8283
  dqbd

  authored
  ffa8283

  Copy full SHA for ffa8283

• Update development README.md

  Show description for 110eef4
  dqbd

  committed
  110eef4

  Copy full SHA for 110eef4

Commits on Apr 10, 2024

• Add GPT-4 Turbo

  akynau

  committed
  3768231

  Copy full SHA for 3768231

Commits on Apr 12, 2024

• Merge pull request #97 from akynau/add-gpt-4-turbo

  Show description for 1d8a762
  dqbd

  authored
  1d8a762

  Copy full SHA for 1d8a762

• Add changelog

  dqbd

  committed
  f386b28

  Copy full SHA for f386b28

• Version Packages

  github-actions[bot]

  committed
  f6165e7

  Copy full SHA for f6165e7

• Merge pull request #98 from dqbd/changeset-release/main

  Show description for 045f0e4
  dqbd

  authored
  045f0e4

  Copy full SHA for 045f0e4

Commits on May 13, 2024

• Add support for GPT-4-O, "Omni" model

  pkallos

  committed
  f2e1ac2

  Copy full SHA for f2e1ac2

• Merge pull request #104 from neonredwood/support-gpt4-omni

  Show description for 3c118a6
  dqbd

  authored
  3c118a6

  Copy full SHA for 3c118a6

• fix(js): specify exports in js-tiktoken

  dqbd

  committed
  ed9cc4f

  Copy full SHA for ed9cc4f

• Add entry to tsup

  dqbd

  committed
  e228b47

  Copy full SHA for e228b47

• Merge pull request #106 from dqbd/dqbd/o200k_base

  Show description for 5de6be5
  dqbd

  authored
  5de6be5

  Copy full SHA for 5de6be5

• Add changeset

  dqbd

  committed
  791cf62

  Copy full SHA for 791cf62

• Version Packages

  github-actions[bot]

  committed
  50f70dd

  Copy full SHA for 50f70dd

• Merge pull request #107 from dqbd/changeset-release/main

  Show description for a7cce99
  dqbd

  authored
  a7cce99

  Copy full SHA for a7cce99

Commits on Jul 18, 2024

• add support for gpt-4o-mini model

  mbarretol

  committed
  28ae6b3

  Copy full SHA for 28ae6b3

Commits on Jul 22, 2024

• Update js/src/core.ts

  Show description for 25bb43a
  

  mbarretol

  and

  Pederzh

  authored
  25bb43a

  Copy full SHA for 25bb43a

• Update tiktoken/model_to_encoding.json

  Show description for 396a8d0
  

  mbarretol

  and

  Pederzh

  authored
  396a8d0

  Copy full SHA for 396a8d0

• Update wasm/src/lib.rs

  Show description for be5db34
  

  mbarretol

  and

  Pederzh

  authored
  be5db34

  Copy full SHA for be5db34

• Update wasm/src/lib.rs

  Show description for f7e3df1
  

  mbarretol

  and

  Pederzh

  authored
  f7e3df1

  Copy full SHA for f7e3df1

Commits on Aug 10, 2024

• Update js/src/core.ts

  Show description for de6a052
  

  mbarretol

  and

  huytool157

  authored
  de6a052

  Copy full SHA for de6a052

• Update tiktoken/model_to_encoding.json

  Show description for fb1ca7b
  

  mbarretol

  and

  huytool157

  authored
  fb1ca7b

  Copy full SHA for fb1ca7b

• Update wasm/src/lib.rs

  Show description for 26ec6cc
  

  mbarretol

  and

  huytool157

  authored
  26ec6cc

  Copy full SHA for 26ec6cc

• Update wasm/src/lib.rs

  Show description for a32f0a4
  

  mbarretol

  and

  huytool157

  authored
  a32f0a4

  Copy full SHA for a32f0a4

Commits on Aug 15, 2024

• Fix unreachable code

  dqbd

  committed
  bba4ed6

  Copy full SHA for bba4ed6

• Add changeset

  dqbd

  committed
  4796f27

  Copy full SHA for 4796f27

• Add text-embedding-3-small and text-embedding-3-large

  Show description for 9d385cb
  dqbd

  committed
  9d385cb

  Copy full SHA for 9d385cb

• Add changeset

  dqbd

  committed
  831a7e8

  Copy full SHA for 831a7e8

• Merge pull request #112 from mbarretol/add-gpt4o-mini

  Show description for e77e339
  dqbd

  authored
  e77e339

  Copy full SHA for e77e339

• Version Packages

  github-actions[bot]

  committed
  ed2e785

  Copy full SHA for ed2e785

• Merge pull request #116 from dqbd/changeset-release/main

  Show description for 60ff9bb
  dqbd

  authored
  60ff9bb

  Copy full SHA for 60ff9bb

• fix(js): add missing rank files back to js-tiktoken

  Show description for a825d5f
  dqbd

  committed
  a825d5f

  Copy full SHA for a825d5f

• Changeset

  dqbd

  committed
  74e7870

  Copy full SHA for 74e7870

• Merge pull request #117 from dqbd/dqbd/js-tiktoken-missing-files

  Show description for 25f0656
  dqbd

  authored
  25f0656

  Copy full SHA for 25f0656

• Version Packages

  github-actions[bot]

  committed
  386acb8

  Copy full SHA for 386acb8

• Merge pull request #118 from dqbd/changeset-release/main

  Show description for f8aa455
  dqbd

  authored
  f8aa455

  Copy full SHA for f8aa455

Commits on Oct 3, 2024

• Add missing o1 models

  dqbd

  committed
  0ba7dbb

  Copy full SHA for 0ba7dbb

• Changeset

  dqbd

  committed
  53bd14a

  Copy full SHA for 53bd14a

• Merge pull request #121 from dqbd/dqbd/o1

  Show description for 503da21
  dqbd

  authored
  503da21

  Copy full SHA for 503da21

• Version Packages

  github-actions[bot]

  committed
  7e6cddc

  Copy full SHA for 7e6cddc

Commits on Oct 4, 2024

• Merge pull request #122 from dqbd/changeset-release/main

  dqbd

  authored
  b742a86

  Copy full SHA for b742a86

Commits on Dec 17, 2024

• Add new o1-2024-12-17 model

  lmg8

  committed
  f40be08

  Copy full SHA for f40be08

• Add new o1-2024-12-17 model

  lmg8

  committed
  c73f19e

  Copy full SHA for c73f19e

• Merge remote-tracking branch 'origin/main'

  lmg8

  committed
  8a20c2d

  Copy full SHA for 8a20c2d

Commits on Dec 19, 2024

• Add changeset

  dqbd

  committed
  919eb6b

  Copy full SHA for 919eb6b

• Merge pull request #124 from lmg8/main

  Show description for 9c95cc5
  dqbd

  authored
  9c95cc5

  Copy full SHA for 9c95cc5

• Version Packages

  github-actions[bot]

  committed
  7a8687d

  Copy full SHA for 7a8687d

• Merge pull request #125 from dqbd/changeset-release/main

  Show description for 638e6a2
  dqbd

  authored
  638e6a2

  Copy full SHA for 638e6a2