Make llm serving template serverless

[ericl]

No description provided.

[ericl]

Add this to make it a bit more friendly out of the box

[gvspraveen]

Any update needed to readme to run serve run main:my_app?

[ericl]

It was already there, I just created the file so the user doesn't need to copy paste code from the markdown cell.

[shomilj]

💯

[shomilj]

https://github.com/anyscale/product/pull/26531 needs to land before this can merge

[akshay-anyscale]

@shomilj is multi-az stuff in serverless defaults

[shomilj]

@shomilj is multi-az stuff in serverless defaults

No, this is unrelated to serverless - this is something that should be set in the base advanced config if desired. So we should keep these around in the compute configs @ericl:

aws:
  TagSpecifications:
    - ResourceType: instance
      Tags:
        - Key: as-feature-enable-multi-az-serve
          Value: "true"
        - Key: as-feature-multi-zone
          Value: "true"

[gvspraveen]

No, this is unrelated to serverless - this is something that should be set in the base advanced config if desired.

@shomilj ooc, will serverless config only add worker_node_types?

[shomilj]

@gvspraveen yes, "serverless" is just "auto_select_worker_config" set to true - all it does is handle worker node types

[ericl]

No, this is unrelated to serverless - this is something that should be set in the base advanced config if desired. So we should keep these around in the compute configs @ericl:

@shomilj any reason not to enable this by default for all OA configs? It seems strange we have these configs that are required for the top CUJ that are off by default.

[shomilj]

https://github.com/anyscale/product/pull/26531 has been merged.

We will likely enable multi-zone by default for OA workloads -- will tackle that as a separate work item this upcoming sprint. So I would add back the aws / gcp advanced config blocks for now, but other than that this is ready to go :)