What should the manifest look like

[haz]

In planutils, we'll want to describe the full API, so that this planner-as-a-service package automagically exposes the "right stuff". What should that JSON manifest look like for both a generic planner endpoint and an arbitrary service.

CC: @camcunningham @nirlipo

Things to include:

• What the parameters would be for calling this service (note that it should be possible to expose multiple services)
• How to run the command line based on those parameters
• Files to return the contents of after running (flag indicating if it's just a string or json to be parsed)
• What the URL spec should be for the service

Note: If it is a generic planner service, then many things may be left out, and extra assumptions come into play:

• The URL will be appended to a plan or solve endpoint.
• Can assume domain and problem strings.
• Can assume extra_args for any planner config required
  • Format to run will be {planner} {domain} {problem} {extra_args}
• Still assume that file output stuff is defined (like the generic)

[camcunningham]

Example that I currently have going for lama-first:

{
"name": "LAMA-FIRST",
"description": "http://fast-downward.org/",
"install-size": "20K",
"dependencies": ["downward"],
"urlspec":"/package/lama-first",
"solver":true,
"services":{"solve":["domain", "problem"]}, -- For this, dict since we want support for multiple services within one executable
"extra_args":[""],
"return-files":["sas_plan"]
}

Early version, let me know what you think / what else I'm missing! @haz

[haz]

A great start, thanks! Slight re-write...

{
  "name": "LAMA-FIRST",
  "description": "http://fast-downward.org/",
  "install-size": "20K",
  "dependencies": ["downward"],


  "endpoint": {
    "type": "solver",
    "args": [
        {"name": "whatever",
         "description": "this is just a random argument added to show things"}
    ],
    "return": {
        "type": "generic",
        "file": "sas_plan"
    }
}

Further notes:

• Removed the urlspec since the default should place this at /packages/lama-first (latter taken from the package name)
• Ditto w/ the services section.
• Other type on the endpoint would be custom
• Existence of endpoint in the manifest means that it can be exposed via the API
• Other type of return is json -- in this case, you should try to parse the file as valid json.
• The standard output (and error) should be given in the return as well (always)
• args should come with domain and problem by default, since the endpoint type is solver.
• Worth putting in descriptions wherever we can.

An example of a more generic one:

{
  "name": "macq",
  "description": "A library for model acquisition",
  "install-size": "20M",
  "dependencies": [],


  "endpoint": {

    "type": "custom",

    "args": [

        {"name": "domain",
         "type": "file",
         "description": "domain file"},

        {"name": "problem",
         "type": "file",
         "description": "problem file"},

    ],

    "services": {

      "analyze": {
        "args": [
            {"name": "trace",
             "type": "file",
             "description": "number of plans to generate"},
        ],

        "call": "macq --analyze {trace} --out output.txt",
    
        "return": {
            "type": "generic",
            "file": "output.txt"
        }
      },

      "generate": {
        "args": [
          {"name": "num_plans",
           "type": "cmd-arg",
           "description": "number of plans to generate"},
        ],

        "call": "macq --domain {domain} --problem {problem} --num {num_plans} --out plans.json",
    
        "return": {
            "type": "json",
            "file": "plans.json"
        },
      }
    }
}

That make sense?

[camcunningham]

This looks really good. I'll implement this and start on the generic execution today.

[haz]

...updated the generic example a bit so we have multiple services.

[haz]

@nirlipo Any thoughts on this? General idea is that a manifest like the above in planutils is all that's required to get a service exposed with the planning-as-a-service. @camcunningham is making this magic happen, but design on the API would be useful.

@mikand You may want to have a look through as well. Again, not for your project to connect to the planning services (though you're more than welcome to), but to lock in on the design of the API. In particular, there will be a generic "I am a solver" API that saves us from writing much of the manifest details.

[haz]

Ah, well I dislike returning file parts for client-side processing -- especially if it's just json! We can parse and embed server side.

Why a dict mapping #'s to actions and not just a list of action objects? Also, a longstanding feature of the existing solver was that if the parsing works, you can get the ground action schema as part of it. Problem is that you need to know if it's a generic list of action strings or objects containing the schema. I think we can just have action objects, include a key "name" that works for both generic and enhanced versions, and include the extra (enhanced) details if it's available.

[nirlipo]

Ah, well I dislike returning file parts for client-side processing -- especially if it's just json! We can parse and embed server side.

I'm not sure I follow, what would you return as a response if not a json file/response message?

Why a dict mapping #'s to actions and not just a list of action objects?

#'s would be the id of the plan, and the list of actions the plan itself.

Also, a longstanding feature of the existing solver was that if the parsing works, you can get the ground action schema as part of it. Problem is that you need to know if it's a generic list of action strings or objects containing the schema. I think we can just have action objects, include a key "name" that works for both generic and enhanced versions, and include the extra (enhanced) details if it's available.

yes, that's a good way to do it. So you suggest returning a list of action objects as a plan, that refer to the ground action schema?

[haz]

I'm not sure I follow, what would you return as a response if not a json file/response message?

We're likely talking about the same thing, but my intent is that a single json payload be returned -- not json in addition to a file that needs to be pieced back together. So if some software generates a file plans.json, then the server should parse it, and include the contents as part of the json payload returned.

#'s would be the id of the plan, and the list of actions the plan itself.

Aha, I misread. Though the #'s were at the action level (so indicating which action). That makes sense -- sometimes the #'s mean nothing, but sometimes they do (like the progressively better plans from lama).

So you suggest returning a list of action objects as a plan, that refer to the ground action schema?

Yep! And then approaches that just need the action name could get it as resp['plans'][i][j]['name'] for the jth action of the ith plan. If grounding fails (we should use tarski so it fails less often), then the action names should still be captured. You can see the try/catch handling that does this currently here

[nirlipo]

Great, we are in the same page indeed! I agree with all 3 points.

We can return grounding schema + plans in the same response file.

[haz]

This something @YIDING-CODER is working on?

[YIDING-CODER]

Hi @haz and @nirlipo,

Based on your discussion and my understanding, I have made the following changes:

1. Unified solver type and custom type
2. Removed endpoint, solver/custom type and default args parts.
3. Use services to list all the available services.
4. Each service will have its own args, call, and return file.
5. For the solver-related service, it will return the plans.json. The plans.json may contain a single plan or multiple plans and log.

Reasons:
(1) and (2) make the manifest much simpler and reduce the development work to deal with different service types.
(3) and (4) are similar to the existing design, but I have added default args to each service.
(5) Client can get actions using "resp['plans'][i][j]['name']". I also add the log as an additional attribute.

Below is the new manifest template:

{
  "name": "Service Name",
  "description": "Service Description",
  "install-size": "Size",
  "dependencies": [
    "dependencies"
  ],
  "services": {
    "ServiceName": {
      "args": [
        {
          "name": "whatever",
          "type": "file|generic|json",
          "description": "this is just a random argument added to show things"
        }
      ],
      "call": "command line",
      "return": {
        "type": "json|generic",
        "file": "plans.json|output.txt"
      }
    }
  }
}

Example 1. For solver type service

Below is the example manifest for Lama-first:

{
  "name": "LAMA-FIRST",
  "description": "http://fast-downward.org/",
  "install-size": "20K",
  "dependencies": [
    "download"
  ],
  "services": {
    "solve": {
      "args": [
        {
          "name": "domain",
          "type": "file",
          "description": "domain file"
        },
        {
          "name": "problem",
          "type": "file",
          "description": "problem file"
        }
      ],
      "call": "lama-first {domain} {problem}",
      "return": {
        "type": "json",
        "file": "plans.json"
      }
    }
  }
}

Below is the example plans.json for Lama-first:

{
  "plans": [
    [
      {
        "action": "Pick a"
      },
      {
        "action": "Stack a b"
      },
      {
        "action": "Stack a b"
      },
    ]
  ],
  "log": "lama-first log"
}

Example 2. For generic type service:

Below is the example manifest for macq:

{
  "name": "macq",
  "description": "A library for model acquisition",
  "install-size": "20M",
  "dependencies": [],
  "services": {
    "analyze": {
      "args": [
        {
          "name": "domain",
          "type": "file",
          "description": "domain file"
        },
        {
          "name": "problem",
          "type": "file",
          "description": "problem file"
        },
        {
          "name": "trace",
          "type": "file",
          "description": "number of plans to generate"
        }
      ],
      "call": "macq --analyze {trace} --out output.txt",
      "return": {
        "type": "generic",
        "file": "output.txt"
      }
    },
    "generate": {
      "args": [
        {
          "name": "domain",
          "type": "file",
          "description": "domain file"
        },
        {
          "name": "problem",
          "type": "file",
          "description": "problem file"
        },
        {
          "name": "num_plans",
          "type": "cmd-arg",
          "description": "number of plans to generate"
        }
      ],
      "call": "macq --domain {domain} --problem {problem} --num {num_plans} --out plans.json",
      "return": {
        "type": "json",
        "file": "plans.json"
      }
    }
  }
}

[nirlipo]

Thanks Yi!

1- What's the difference between generate and solve endpoints? Do we need to make such a difference?
2- Re: "log": "lama-first log". Is this field left in case one wants to include the (std)output generated by the planner while solving?

[YIDING-CODER]

@nirlipo

1- What's the difference between generate and solve endpoints? Do we need to make such a difference?
(1) I believe they are the same? They are just the keywords used in the previous examples.
2- Re: "log": "lama-first log". Is this field left in case one wants to include the (std)output generated by the planner while solving?
(2) This is related to one of your comments "a service returns more than one file (plan & execution log)".

[haz]

Thanks! So the things that come to mind (and in no particular order):

• There will be a need to keep around a solve endpoint that doesn't reference a particular package. This is a major exception, and there only for backwards compatibility.
• Being a planner usually means you take a domain+problem and return a plan. This is why there was a service type discussed (or even included in some of the code?). We could get away with a mixin type of thing, where a service can optionally say "inherit": "planner", which would pre-populate the dictionary with a pre-set configuration (which can be overwritten as necessary). This would be extremely useful for generic planner manifests (avoiding copy/paste errors on planutils).
• Do we want to forgo any extra details on plans? Like cost, etc. Currently, as a list of lists, there's no natural place to put such info.
• I think solve|generate|analyze and plans.json|output.txt shouldn't be discrete options, but rather examples of any number of things that might be used.

[YIDING-CODER]

Thanks for your feedback. Please see my reply below.

• There will be a need to keep around a solve endpoint that doesn't reference a particular package. This is a major exception, and there only for backwards compatibility.

I am not sure whether I understand the comments correctly. My question is do we really need to use a keyword {type: solver} and {type: custom} to differentiate the planner and generic service? If we reserve the endpoint solve for the planner, so we know any service contains the solve endpoint is a planner? Vice versa, if a service doesn't contain solve endpoint, we know it is a generic service. I have updated the manifest slightly, I replaced the services keywords with endpoint, but the main structure is not changed. Can you please check whether it fit the requirements?

• Being a planner usually means you take a domain+problem and return a plan. This is why there was a service type discussed (or even included in some of the code?). We could get away with a mixin type of thing, where a service can optionally say "inherit": "planner", which would pre-populate the dictionary with a pre-set configuration (which can be overwritten as necessary). This would be extremely useful for generic planner manifests (avoiding copy/paste errors on planutils).

I really like the idea of inheritance. As you mentioned in the previous discussion, a default planner will contain domain, problem,optional extra args, and the format to call will be {planner} {domain} {problem} {extra_args}. I have provided an example of using inherent in the planner-related service below.

• Do we want to forgo any extra details on plans? Like cost, etc. Currently, as a list of lists, there's no natural place to put such info.

I think we now have two things to solve, one is the manifest of service and the other is the manifest of plans.json. For the plans.json, I have updated the template to include the plan level and action level details.

• I think solve|generate|analyze and plans.json|output.txt shouldn't be discrete options, but rather examples of any number of things that might be used.

No comments for this.

This reply is not completed, I will provide detailed examples tomorrow.

[haz]

So if you have the ability to "inherit" (not sure if this is the right term), then the planner vs generic discussion is moot. I don't think you should assume that an endpoint named solve is a planner (use the "inherit" functionality for that). Why? There may be some services that don't quite do planning that still want a solve endpoint. So let's leave it up to the services (and the manifest they decide on) to name their endpoints whatever they want.

I am not sure whether I understand the comments correctly.

My original comment refers to another aspect. There should be a package-free endpoint that's just /solve to allow for backwards compatibility. Corresponds roughly to this, but we'll need to refine it. Ultimately, it will just be a redirect to a pre-selected regular endpoint (e.g., using @nirlipo 's planner).

One final thing: we keep mentioning plans.json, but we should be clear that it's a full json payload that gets returned and not a file that contains json. So the server will read/parse plans.json, and return that for the the user.

[YIDING-CODER]

Hi @haz and @nirlipo ,

Thanks for your comments. I have separated my replies into three parts.

1. Introduce to the inherit keyword
2. Updated manifest and examples
3. Package-free solve endpoint

Part1: Introduce to the inherit keyword

1. inherit keyword

When the user use "inherit":"planner" value pair in the manifest, it will be replaced with a default planner service template when executing the code. Below is an example of the planner service template.

{
  "args": [
        {
          "name": "domain",
          "type": "file",
          "description": "domain file"
        },
        {
          "name": "problem",
          "type": "file",
          "description": "problem file"
        }
  ],
  "call": "{planner} {domain} {problem}",
  "return": {
    "type": "json",
    "file": "plans.json"
  }
}

2. Overwrite the default planner template

Users can overwrite the args, call, and return sections.

Below are examples of how to overwrite the default planner template.

{
  "generate": {
    "args": [
      {
        "name": "domain",
        "type": "file",
        "description": "domain file"
      },
      {
        "name": "problem",
        "type": "file",
        "description": "problem file"
      },
      {
        "name": "num_plans",
        "type": "cmd-arg",
        "description": "number of plans to generate"
      }
    ],
    "call": "macq --domain {domain} --problem {problem} --num {num_plans} --out plans.json",
    "return": {
      "type": "json",
      "file": "plans.json"
    }
  }
}

The above service can be overwritten to:

{
  "generate": {
    "inherit":"planner"
    "args": [ # only need to state the addtional args other than domain and problem
      {
        "name": "num_plans",
        "type": "cmd-arg",
        "description": "number of plans to generate"
      }
    ],
    # the call command need to be overwrite
    "call": "macq --domain {domain} --problem {problem} --num {num_plans} --out plans.json"
    
    # return is not affected
  }
}

[haz]

Precisely what I had in mind, thanks!

[haz]

Maybe we replace inherit with template? We don't really have natural inheritance going on here.

[YIDING-CODER]

Agree!

[YIDING-CODER]

Part2: Updated manifest and examples

1. updated manifest template with inherit

For all the planner, we can use "inherit":"planner" in the service so we know it's a planner related services.

{
  "name": "Package Name",
  "description": "Package Description",
  "install-size": "Size",
  "dependencies": [
    "dependencies"
  ],
  "endpoint": {
    "services": {
      "service_name": {
        "inherit": "planner|other_templates",
        "args": [
          {
            "name": "whatever",
            "type": "file|generic|json",
            "description": "this is just a random argument added to show things"
          }
        ],
        "call": "command line",
        "return": {
          "type": "json|generic",
          "file": "plans.json|output.txt"
        }
      }
    }
  }
}

2. Example of lama-first manifest

{
  "name": "LAMA-FIRST",
  "description": "http://fast-downward.org/",
  "install-size": "20K",
  "dependencies": [
    "download"
  ],
  "endpoint": {
    "services": {
      "solve": {
        "inherit": "planner"
      }
    }
  }
}

3. Example of macq

{
  "name": "macq",
  "description": "A library for model acquisition",
  "install-size": "20M",
  "dependencies": [],
  "endpoint": {
    "services": {
      "analyze": {
        "inherit": "planner",
        "args": [
          {
            "name": "trace",
            "type": "file",
            "description": "number of plans to generate"
          }
        ],
        "call": "macq --domain {domain} --problem {problem} --analyze {trace} --out output.txt",
        "return": {
          "type": "generic",
          "file": "output.txt"
        }
      },
      "generate": {
        "inherit": "planner",
        "args": [
          {
            "name": "num_plans",
            "type": "cmd-arg",
            "description": "number of plans to generate"
          }
        ],
        "call": "macq --domain {domain} --problem {problem} --num {num_plans} --out plans.json"
      }
    }
  }
}

[haz]

Ya, I suspect all is fine. Just useful to see the outputs and @ctpelok77 can confirm it's all good ;).

[ctpelok77]

Yup.
One suggestion: maybe worth testing with small-ish problems.
For K*, I am eager to port it to a more modern version of Fast Downward and use a modern consistent heuristic that actually works better than blind search on average.

[YIDING-CODER]

Hi @ctpelok77 ,
Thank you for your explanation.

As you mentioned these three planners(lama, cerberus-sat and kstar) are iterative planners, and I have to impose a time bond. However, I couldn't find the instruction on how to do it. Can you please write the working command for these three planners using the PDDL files in this folder?

Currently, I am using:

• lama domain.pddl problem.pddl
• kstar domain.pddl problem.pddl --search 'kstar(blind(),k=1)'
• cerberus-sat domain.pddl problem.pddl

I am looking forward to your response.

[ctpelok77]

Let me make two suggestions:

1. Use a smaller blocksworld problem. 10 blocks is more or less the limit for current optimal planners. 5 or 6 blocks should be enough for testing purposes.
2. While some planners may implement their own time limit, I would suggest to always use an external one. You can wrap your planner call with ulimit.

[YIDING-CODER]

@ctpelok77,
Thank you. I just tried the block world with 4 blocks and it's all working!

[YIDING-CODER]

Part3: Package-free solve endpoint

I assume there will be only one planner linked to the package-free solve endpoint. If lama-first is selected, we just need to state the mapping between solve endpoint to the lama-first package in the python configure/environment file? Currently, it is hardcoded.

solver GET method will return the manifest of lama-first package
solver Post method will return the plans.json payload generated by lama-first domain problem

[haz]

I think it's all being over-complicated. My thoughts on it...

• The generic /solve endpoint should eventually go away (barring some smart auto-detection of problem type, blablabla). It's there only for backwards compatibility.
• It can (and probably should) just redirect to a proper endpoint with the same payload. You can re-write the URL to call lama-first or whatever.
• I get that we want a drop-down on the online editor. We can do this editor side. How? Just recognize which of the services make sense to expose, and include the package names to the list on the editor. It can then reach out to the online planning-as-a-service and dynamically change the "Solve" interface to include extra parameters.
  • No need to specially designate editor-ready solvers in the manifest or online service.
• In the end, specifying a URL like we do now will be replaced by specifying the service end-point + list of services. It'll allow custom deploys, and integrate seamlessly.

[YIDING-CODER]

Thanks @haz,

Based on what we have been discussed, I am going to implement the following features.

Stage1

1. Implement logic to deal with the manifest structure listed in Part1 and Part2
2. Implement the logic to redirect the solve endpoint to the designated planner service
3. Deploy and test the planning-as-a-service

Stage2

1. Implement the logic to deal with the output payload
2. Implement the integration between planning editor and planning-as-a-service

Please let me know whether I can go ahead and implement it.

[haz]

Can you clarify what "logic to deal with the output payload" refers to?

Rest looks good to me!

[YIDING-CODER]

Hi @haz ,

It's referring to your comments below:

"
I think we should be passive in the expectations.
Attempt to parse the output as json if service says it should be so.
If it works, then use that as the json payload.
Inspect the payload to see if there's an action list like we'd expect.
Add a flag that indicates if it adheres to that action spec.
This way, if the service messes up, the check on the client-side can still understand if it can expect action lists or not. We can even have different levels of what it looks for (action names? ground actions as well? etc).
"

[haz]

Ah, ok. Well I think you can immediately go with part of that -- just pass through the file contents as json, wrap the json parse in a try-catch, and have your fallback pass back the raw file contents. Ok to punt "inspect for actions we'd like to see" for now.