Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using metal crashes localAI #550

Closed
soleblaze opened this issue Jun 9, 2023 · 4 comments
Closed

Using metal crashes localAI #550

soleblaze opened this issue Jun 9, 2023 · 4 comments
Assignees
@mudler
Labels
bug Something isn't working

Comments

@soleblaze
Copy link

soleblaze commented Jun 9, 2023 •
edited
Loading

LocalAI version:
6bb5622

Environment, CPU architecture, OS, and Version:
Darwin c13.localdomain 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:43 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T8112 arm64

M2 Macbook Air

Describe the bug

Using metal fails to load the model properly. It appears that llama.cpp always thinks the model is GGML_TYPE_F32.

To Reproduce

make BUILD_TYPE=metal build
cp go-llama/llama.cpp/ggml-metal.metal .

Run local-ai with a model config that uses a Q4_0 model and sets gpu_layers: 1 in the model config. Send a curl command that uses that model

Expected behavior

I expect the model to run using metal and provide output back to the user

Logs

./local-ai --config-file ~/model/localai.yaml --models-path ~/model --debug 
Starting LocalAI using 4 threads, with models path: /Users/soleblaze/model
10:46PM DBG Model: 7B (config: {OpenAIRequest:{Model:ggml-model-q4_0.bin File: Language: ResponseFormat: Size: Prompt:<nil> Instruction: Input:<nil> Stop:<nil> Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0} Name:7B StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false Threads:0 Debug:false Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1 MMap:false MMlock:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[] InputToken:[]})
10:46PM DBG GPT4ALL_IMPLEMENTATIONS_PATH: /tmp/localai/backend_data/backend-assets/gpt4all

 ┌───────────────────────────────────────────────────┐
 │                   Fiber v2.46.0                   │
 │               http://127.0.0.1:8080               │
 │       (bound on host 0.0.0.0 and port 8080)       │
 │                                                   │
 │ Handlers ............ 24  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............. 29818 │
 └───────────────────────────────────────────────────┘

10:46PM DBG Request received: {"model":"7B","file":"","language":"","response_format":"","size":"","prompt":"A long time ago in a galaxy far, far away","instruction":"","input":null,"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0.7,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0}
10:46PM DBG `input`: &{Model:7B File: Language: ResponseFormat: Size: Prompt:A long time ago in a galaxy far, far away Instruction: Input:<nil> Stop:<nil> Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0.7 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0}
10:46PM DBG Parameter Config: &{OpenAIRequest:{Model:ggml-model-q4_0.bin File: Language: ResponseFormat: Size: Prompt:<nil> Instruction: Input:<nil> Stop:<nil> Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0.7 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0} Name:7B StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false Threads:4 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1 MMap:false MMlock:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[A long time ago in a galaxy far, far away] InputStrings:[] InputToken:[]}
10:46PM DBG Loading model 'ggml-model-q4_0.bin' greedly
10:46PM DBG [llama] Attempting to load
10:46PM DBG Loading model llama from ggml-model-q4_0.bin
10:46PM DBG Loading model in memory from file: /Users/soleblaze/model/ggml-model-q4_0.bin
llama.cpp: loading model from /Users/soleblaze/model/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 3615.71 MB
llama_model_load_internal: mem required  = 5407.71 MB (+ 2052.00 MB per state)
...................................................................................................
llama_init_from_file: kv self size  =  512.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '/Users/soleblaze/git/thirdparty/LocalAI/ggml-metal.metal'
ggml_metal_init: loaded kernel_add                            0x10fe07980
ggml_metal_init: loaded kernel_mul                            0x10fe080a0
ggml_metal_init: loaded kernel_mul_row                        0x10fe086d0
ggml_metal_init: loaded kernel_scale                          0x10fe08bf0
ggml_metal_init: loaded kernel_silu                           0x10fe09110
ggml_metal_init: loaded kernel_relu                           0x10fe09630
ggml_metal_init: loaded kernel_soft_max                       0x10fe09ce0
ggml_metal_init: loaded kernel_diag_mask_inf                  0x10fe0a340
ggml_metal_init: loaded kernel_get_rows_f16                   0x11fe08a90
ggml_metal_init: loaded kernel_get_rows_q4_0                  0x11fe09230
ggml_metal_init: loaded kernel_get_rows_q2_k                  0x11fe098b0
ggml_metal_init: loaded kernel_get_rows_q4_k                  0x11fe0a0a0
ggml_metal_init: loaded kernel_get_rows_q6_k                  0x11fe0a720
ggml_metal_init: loaded kernel_rms_norm                       0x11fe0add0
ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x11fe0b630
ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x11fe0be90
ggml_metal_init: loaded kernel_mul_mat_q2_k_f32               0x11fe0c6f0
ggml_metal_init: loaded kernel_mul_mat_q4_k_f32               0x11fe0cf50
ggml_metal_init: loaded kernel_mul_mat_q6_k_f32               0x11fe0d7b0
ggml_metal_init: loaded kernel_rope                           0x11fe0e210
ggml_metal_init: loaded kernel_cpy_f32_f16                    0x11fe0eaa0
ggml_metal_init: loaded kernel_cpy_f32_f32                    0x11fe0f330
ggml_metal_add_buffer: allocated 'data            ' buffer, size =  3615.72 MB
ggml_metal_add_buffer: allocated 'eval            ' buffer, size =   768.00 MB
ggml_metal_add_buffer: allocated 'kv              ' buffer, size =   514.00 MB
ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   512.00 MB
ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   512.00 MB
10:46PM DBG [llama] Loads OK
Asserting on type 0
GGML_ASSERT: ggml-metal.m:562: false && "not implemented"
SIGABRT: abort
PC=0x1a1dac724 m=5 sigcode=0
signal arrived during cgo execution

goroutine 22 [syscall]:
runtime.cgocall(0x10310ec74, 0x140004ee1e8)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/cgocall.go:157 +0x54 fp=0x140004ee1b0 sp=0x140004ee170 pc=0x102c62e34
github.com/go-skynet/go-llama%2ecpp._Cfunc_llama_predict(0x11fe0fe70, 0x110008200, 0x14000580000, 0x1)
        _cgo_gotypes.go:224 +0x38 fp=0x140004ee1e0 sp=0x140004ee1b0 pc=0x10306e1b8
github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict.func2(0x1400002c2a0?, 0x140004ee3e0?, {0x14000580000, 0x0?, 0x10?}, 0x103d2d1a0?)
        /Users/soleblaze/git/thirdparty/LocalAI/go-llama/llama.go:211 +0x90 fp=0x140004ee230 sp=0x140004ee1e0 pc=0x103070370
github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict(0x1400000ec00, {0x1400002c2a0, 0x29}, {0x14000130be0, 0xd, 0x0?})
        /Users/soleblaze/git/thirdparty/LocalAI/go-llama/llama.go:211 +0x26c fp=0x140004ee4f0 sp=0x140004ee230 pc=0x10307004c
github.com/go-skynet/LocalAI/api.ModelInference.func12()
        /Users/soleblaze/git/thirdparty/LocalAI/api/prediction.go:532 +0xb0 fp=0x140004ee810 sp=0x140004ee4f0 pc=0x1030e05a0
github.com/go-skynet/LocalAI/api.ModelInference.func14()
        /Users/soleblaze/git/thirdparty/LocalAI/api/prediction.go:574 +0x244 fp=0x140004ee8d0 sp=0x140004ee810 pc=0x1030e01a4
github.com/go-skynet/LocalAI/api.ComputeChoices({0x1400002c2a0, 0x29}, 0x14000000280, 0x14000138840, 0x14000244ab0?, 0x103d97dd8, 0x4?)
        /Users/soleblaze/git/thirdparty/LocalAI/api/prediction.go:598 +0x1c0 fp=0x140004ef170 sp=0x140004ee8d0 pc=0x1030e2830
github.com/go-skynet/LocalAI/api.completionEndpoint.func2(0x14000138580)
        /Users/soleblaze/git/thirdparty/LocalAI/api/openai.go:252 +0x764 fp=0x140004ef300 sp=0x140004ef170 pc=0x1030d92f4
github.com/gofiber/fiber/v2.(*App).next(0x14000178900, 0x14000138580)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:144 +0x184 fp=0x140004ef3b0 sp=0x140004ef300 pc=0x10303ab04
github.com/gofiber/fiber/v2.(*Ctx).Next(0x70?)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/ctx.go:913 +0x5c fp=0x140004ef3d0 sp=0x140004ef3b0 pc=0x10302850c
github.com/gofiber/fiber/v2/middleware/recover.New.func1(0x1?)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/middleware/recover/recover.go:43 +0xa4 fp=0x140004ef450 sp=0x140004ef3d0 pc=0x103046704
github.com/gofiber/fiber/v2.(*Ctx).Next(0x1400016cae0?)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/ctx.go:910 +0x48 fp=0x140004ef470 sp=0x140004ef450 pc=0x1030284f8
github.com/gofiber/fiber/v2/middleware/logger.New.func3(0x14000138580)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/middleware/logger/logger.go:121 +0x2d8 fp=0x140004efb00 sp=0x140004ef470 pc=0x1030417d8
github.com/gofiber/fiber/v2.(*App).next(0x14000178900, 0x14000138580)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:144 +0x184 fp=0x140004efbb0 sp=0x140004efb00 pc=0x10303ab04
github.com/gofiber/fiber/v2.(*App).handler(0x14000178900, 0x102fabb50?)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:171 +0x74 fp=0x140004efc10 sp=0x140004efbb0 pc=0x10303ace4
github.com/gofiber/fiber/v2.(*App).handler-fm(0x140000d6000?)
        <autogenerated>:1 +0x30 fp=0x140004efc30 sp=0x140004efc10 pc=0x10303f950
github.com/valyala/fasthttp.(*Server).serveConn(0x14000252000, {0x103da06d8?, 0x140004f6000})
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:2365 +0xdd0 fp=0x140004efea0 sp=0x140004efc30 pc=0x102fabba0
github.com/valyala/fasthttp.(*Server).serveConn-fm({0x103da06d8?, 0x140004f6000?})
        <autogenerated>:1 +0x3c fp=0x140004efed0 sp=0x140004efea0 pc=0x102fbab6c
github.com/valyala/fasthttp.(*workerPool).workerFunc(0x14000130aa0, 0x1400007a020)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:224 +0x70 fp=0x140004eff90 sp=0x140004efed0 pc=0x102fb6580
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:196 +0x38 fp=0x140004effd0 sp=0x140004eff90 pc=0x102fb6288
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140004effd0 sp=0x140004effd0 pc=0x102cc4f04
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:195 +0x220

goroutine 1 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x140004f3280 sp=0x140004f3260 pc=0x102c95d74
runtime.netpollblock(0x140004f3318?, 0x2d345d4?, 0x1?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/netpoll.go:527 +0x158 fp=0x140004f32c0 sp=0x140004f3280 pc=0x102c8f388
internal/poll.runtime_pollWait(0x10b9fd700, 0x72)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/netpoll.go:306 +0xa0 fp=0x140004f32f0 sp=0x140004f32c0 pc=0x102cbeea0
internal/poll.(*pollDesc).wait(0x14000154380?, 0x102c74e94?, 0x0)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x140004f3320 sp=0x140004f32f0 pc=0x102d2fc18
internal/poll.(*pollDesc).waitRead(...)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000154380)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/internal/poll/fd_unix.go:614 +0x250 fp=0x140004f33d0 sp=0x140004f3320 pc=0x102d346c0
net.(*netFD).accept(0x14000154380)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/net/fd_unix.go:172 +0x28 fp=0x140004f3490 sp=0x140004f33d0 pc=0x102db6838
net.(*TCPListener).accept(0x1400000e720)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/net/tcpsock_posix.go:148 +0x28 fp=0x140004f34c0 sp=0x140004f3490 pc=0x102dcbe68
net.(*TCPListener).Accept(0x1400000e720)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/net/tcpsock.go:297 +0x2c fp=0x140004f3500 sp=0x140004f34c0 pc=0x102dcafdc
github.com/valyala/fasthttp.acceptConn(0x14000252000, {0x103d9dc80, 0x1400000e720}, 0x140004f3718)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:1930 +0x44 fp=0x140004f35e0 sp=0x140004f3500 pc=0x102faa474
github.com/valyala/fasthttp.(*Server).Serve(0x14000252000, {0x103d9dc80?, 0x1400000e720})
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:1823 +0x4f0 fp=0x140004f3740 sp=0x140004f35e0 pc=0x102fa9c30
github.com/gofiber/fiber/v2.(*App).Listen(0x14000178900, {0x1032a8331?, 0x7?})
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/listen.go:82 +0xfc fp=0x140004f37a0 sp=0x140004f3740 pc=0x103032cdc
main.main.func1(0x1400024e160?)
        /Users/soleblaze/git/thirdparty/LocalAI/main.go:140 +0x4a4 fp=0x140004f39f0 sp=0x140004f37a0 pc=0x10310e544
github.com/urfave/cli/v2.(*Command).Run(0x1400024e160, 0x14000038980, {0x140000201e0, 0x6, 0x6})
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/command.go:274 +0x75c fp=0x140004f3c80 sp=0x140004f39f0 pc=0x1030fd8fc
github.com/urfave/cli/v2.(*App).RunContext(0x1400024a000, {0x103d9dfe8?, 0x140000260d8}, {0x140000201e0, 0x6, 0x6})
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/app.go:332 +0x604 fp=0x140004f3cf0 sp=0x140004f3c80 pc=0x1030fa6e4
github.com/urfave/cli/v2.(*App).Run(...)
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/app.go:309
main.main()
        /Users/soleblaze/git/thirdparty/LocalAI/main.go:144 +0xf38 fp=0x140004f3f70 sp=0x140004f3cf0 pc=0x10310e008
runtime.main()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:250 +0x248 fp=0x140004f3fd0 sp=0x140004f3f70 pc=0x102c95948
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140004f3fd0 sp=0x140004f3fd0 pc=0x102cc4f04

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005afa0 sp=0x1400005af80 pc=0x102c95d74
runtime.goparkunlock(...)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:387
runtime.forcegchelper()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:305 +0xb8 fp=0x1400005afd0 sp=0x1400005afa0 pc=0x102c95bb8
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005afd0 sp=0x1400005afd0 pc=0x102cc4f04
created by runtime.init.6
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:293 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005b760 sp=0x1400005b740 pc=0x102c95d74
runtime.goparkunlock(...)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgcsweep.go:319 +0x110 fp=0x1400005b7b0 sp=0x1400005b760 pc=0x102c82bb0
runtime.gcenable.func1()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:178 +0x28 fp=0x1400005b7d0 sp=0x1400005b7b0 pc=0x102c77668
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005b7d0 sp=0x1400005b7d0 pc=0x102cc4f04
created by runtime.gcenable
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:178 +0x74

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x1e7705a4?, 0x1e6708eb?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005bf50 sp=0x1400005bf30 pc=0x102c95d74
runtime.goparkunlock(...)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x1041672a0)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgcscavenge.go:400 +0x5c fp=0x1400005bf80 sp=0x1400005bf50 pc=0x102c809cc
runtime.bgscavenge(0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgcscavenge.go:633 +0xac fp=0x1400005bfb0 sp=0x1400005bf80 pc=0x102c80fac
runtime.gcenable.func2()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:179 +0x28 fp=0x1400005bfd0 sp=0x1400005bfb0 pc=0x102c77608
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005bfd0 sp=0x1400005bfd0 pc=0x102cc4f04
created by runtime.gcenable
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:179 +0xb8

goroutine 5 [finalizer wait]:
runtime.gopark(0x1a0?, 0x104167f00?, 0xc0?, 0x36?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005a580 sp=0x1400005a560 pc=0x102c95d74
runtime.runfinq()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mfinal.go:193 +0x10c fp=0x1400005a7d0 sp=0x1400005a580 pc=0x102c766fc
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005a7d0 sp=0x1400005a7d0 pc=0x102cc4f04
created by runtime.createfing
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mfinal.go:163 +0x84

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005c740 sp=0x1400005c720 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x1400005c7d0 sp=0x1400005c740 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005c7d0 sp=0x1400005c7d0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005cf40 sp=0x1400005cf20 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x1400005cfd0 sp=0x1400005cf40 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005cfd0 sp=0x1400005cfd0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005d740 sp=0x1400005d720 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x1400005d7d0 sp=0x1400005d740 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005d7d0 sp=0x1400005d7d0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x4e0c62e0e8d4?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x14000056740 sp=0x14000056720 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x140000567d0 sp=0x14000056740 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140000567d0 sp=0x140000567d0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x4e0c62dd8963?, 0x3?, 0xbf?, 0x65?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x140004d2740 sp=0x140004d2720 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x140004d27d0 sp=0x140004d2740 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140004d27d0 sp=0x140004d27d0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x4e0c62dd8a5d?, 0x3?, 0x4d?, 0x7f?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x14000056f40 sp=0x14000056f20 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x14000056fd0 sp=0x14000056f40 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x14000056fd0 sp=0x14000056fd0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x4e0c62dd869e?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x140004d2f40 sp=0x140004d2f20 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x140004d2fd0 sp=0x140004d2f40 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140004d2fd0 sp=0x140004d2fd0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x1043e6e40?, 0x1?, 0x92?, 0xfa?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400005df40 sp=0x1400005df20 pc=0x102c95d74
runtime.gcBgMarkWorker()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1275 +0xec fp=0x1400005dfd0 sp=0x1400005df40 pc=0x102c795ac
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400005dfd0 sp=0x1400005dfd0 pc=0x102cc4f04
created by runtime.gcBgMarkStartWorkers
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/mgc.go:1199 +0x28

goroutine 20 [select]:
runtime.gopark(0x140004d5708?, 0x2?, 0x0?, 0x0?, 0x140004d56dc?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x1400006bd40 sp=0x1400006bd20 pc=0x102c95d74
runtime.selectgo(0x1400006bf08, 0x140004d56d8, 0x0?, 0x0, 0x0?, 0x1)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/select.go:327 +0x690 fp=0x1400006be60 sp=0x1400006bd40 pc=0x102ca6050
github.com/go-skynet/LocalAI/api.(*galleryApplier).start.func1()
        /Users/soleblaze/git/thirdparty/LocalAI/api/gallery.go:103 +0xa8 fp=0x1400006bfd0 sp=0x1400006be60 pc=0x1030d79d8
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x1400006bfd0 sp=0x1400006bfd0 pc=0x102cc4f04
created by github.com/go-skynet/LocalAI/api.(*galleryApplier).start
        /Users/soleblaze/git/thirdparty/LocalAI/api/gallery.go:101 +0xa0

goroutine 21 [sleep]:
runtime.gopark(0x4e0b06fa6de0?, 0x102fb58cc?, 0xd0?, 0xa?, 0x0?)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/proc.go:381 +0xe4 fp=0x140004d5f40 sp=0x140004d5f20 pc=0x102c95d74
time.Sleep(0x2540be400)
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/time.go:195 +0x118 fp=0x140004d5f80 sp=0x140004d5f40 pc=0x102cc1b78
github.com/valyala/fasthttp.(*workerPool).Start.func2()
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:67 +0x54 fp=0x140004d5fd0 sp=0x140004d5f80 pc=0x102fb58b4
runtime.goexit()
        /opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/asm_arm64.s:1172 +0x4 fp=0x140004d5fd0 sp=0x140004d5fd0 pc=0x102cc4f04
created by github.com/valyala/fasthttp.(*workerPool).Start
        /Users/soleblaze/.local/share/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:59 +0xdc

r0      0x0
r1      0x0
r2      0x0
r3      0x0
r4      0x0
r5      0x16f1aa420
r6      0xa
r7      0x0
r8      0xef6fb4f378aaa284
r9      0xef6fb4f217b69284
r10     0x2
r11     0xfffffffd
r12     0x10000000000
r13     0x0
r14     0x0
r15     0x0
r16     0x148
r17     0x201904f60
r18     0x0
r19     0x6
r20     0x16f1c3000
r21     0xe03
r22     0x16f1c30e0
r23     0x0
r24     0x0
r25     0x3821bd980
r26     0x0
r27     0x600000a1d290
r28     0x20
r29     0x16f1aa3d0
lr      0x1a1de3c28
sp      0x16f1aa3b0
pc      0x1a1dac724
fault   0x1a1dac724

Additional context

I also filed this issue for go-llama.cpp: go-skynet/go-llama.cpp#91
It may be related, but once I got the go-llama.cpp example to compile with metal support it worked fine.
@soleblaze soleblaze added the bug Something isn't working label Jun 9, 2023
@nitrag
Copy link

nitrag commented Jun 9, 2023 •
edited
Loading

I can reproduce this as well (using vicuna-13b-1.1.ggmlv3.q4_0.bin)

M1 Pro 32gb

@soleblaze
Copy link
Author

The model config needs both gpu_layers: 1 and f16: true to work. Turns out I was missing f16: true.

@mudler
Copy link
Owner

mudler commented Jun 10, 2023

@soleblaze is the PR in go-llama.cpp needed or works currently with LocalAI?

@soleblaze
Copy link
Author

It currently works in localAI without that PR. I'm trying not to think too hard about that one. It sounds like the path to madness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mudler mudler

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nitrag @soleblaze @mudler