streaming server support?

[kevintanhongann]

Is there a way to run and expose an API streaming server compatible with OpenAI API specifications?

[tjake]

Probably, here's the current API call for chat

https://github.com/tjake/Jlama/blob/main/jlama-cli/src/main/java/com/github/tjake/jlama/cli/serve/GenerateResource.java

[phact]

I want this feature too

[geoand]

I am pretty sure that this would (at least) require Generator#generate to be enhanced with a callback that is called when the generation is complete.

[phact]

You mean for stream=false?

[geoand]

For both :)

[phact]

working PR here #23