Code

2024OSPP——基于AI网关实现AI模型的轻量化部署#1

Code

Merged

johnlanni merged 11 commits intohigress-group:envoy-1.27higress-group/envoy:envoy-1.27from YJQ1101:yuanjq/envoy1.27/llm_inference_filterYJQ1101/envoy:yuanjq/envoy1.27/llm_inference_filterCopy head branch name to clipboardOct 8, 2024

+3,188-1

Commits

Commits on Aug 26, 2024

• feature: llm_inference_filter

  YJQ1101

  committed
  60ea17e

  Copy full SHA for 60ea17e

• add llama.cpp module

  YJQ1101

  committed
  d261d60

  Copy full SHA for d261d60

• //contrib/envoy/extensions/upstreams/http/dubbo_tcp/v3:pkg in place

  YJQ1101

  committed
  a65e32b

  Copy full SHA for a65e32b

• clear yaml file and add test file

  YJQ1101

  committed
  0d5f3bb

  Copy full SHA for 0d5f3bb

Commits on Aug 28, 2024

• apply context-shift if needed

  YJQ1101

  committed
  77ae6be

  Copy full SHA for 77ae6be

• Modify path matching to suffix matching and fix sendError bug

  YJQ1101

  committed
  a4eace8

  Copy full SHA for a4eace8

Commits on Sep 13, 2024

• Load the model based on filter configuration instead of router configuration

  YJQ1101

  committed
  9acc597

  Copy full SHA for 9acc597

Commits on Sep 22, 2024

• Remove locking operations, add use of system command log, fix a bug of request interrupt kvcache release twice

  YJQ1101

  committed
  c7fc2a7

  Copy full SHA for c7fc2a7

Commits on Sep 26, 2024

• Improve inference performance, Divide the load function into loadLLM and LoadEmbedding, Change LOG_INFO to ENVOY_LOG

  YJQ1101

  committed
  efc3e89

  Copy full SHA for efc3e89

Commits on Sep 28, 2024

• update test data

  YJQ1101

  committed
  9e5aae5

  Copy full SHA for 9e5aae5

• update README.md

  YJQ1101

  committed
  6d84641

  Copy full SHA for 6d84641