Why can the PP-BT AzDevOps task not be pinned to a particular version?

[davidjenni]

This is a justified question that we've read several times in the reported issues. Some background is in order.

For broader context, we are switching from the legacy PowerShell-based task implementation to using the same pac CLI based architecture we've successfully use for our Github Actions.
Unifying this architecture will make it easier for us to ship new features faster and in better quality (since we no longer have to port between different code bases and technology stacks).

- Why not just use the per-task version?
The PP-BT tasks have interdependencies between tasks: all our tasks do depend on an initial ToolInstaller task. In the old implementation, that task will download several PS modules and nuget packages and make them available for successive tasks.
In the new pac CLI implementation, the ToolInstaller simply unzips the enclosed pac CLI nupkgs (one for Windows, one for Linux), thus obliviating the need for external network access, but also dramatically speeding up that task to < 1 sec.
It would have been relatively easy to introduce the new pac CLI based tasks all as task version "1", but that not only requires updates to existing pipelines (i.e. to explicitly opt-in), but also opens new error paths due to human mistakes: mix and match between e.g. a Tool Installer@0 and e.g. a Import Solution@1: the latter would expect a local pac CLI, but the former only initialized the PS runtime modules.

- The Azure DevOps task deployment model assumes backwards compatibility
Within the major version of a task, e.g. Import Solution@1, any update we deploy, e.g. going from 1.0.83 to 1.0.84, will automagically picked up when the next build/release job is queued after an update was deployed to VS Marketplace.
That auto-update is great to push fixes with no interaction required for pipeline authors. But it obviously also creates a challenge on our team to not mess up backwards compatibility.
In that AzDO deployment model, there simply is no way for a pipeline author to pin to a particular in-version variant, e.g. there is no way to prefer at large e.g. 1.0.41 over 1.0.81 for the whole task library. This is different to how e.g. our Github Actions work: there each action can be referenced as either microsoft/powerplatform-actions/import-solution@v0 or by a specific release, e.g. microsoft/powerplatform-actions/import-solution@v0.7.1. That would give our users a quick remediation path; but this is not available for Azure DevOps tasks

- Why not backwards compatibility testing?
We do take backwards compatibility very serious and do follow SDP (Safe Deployment Protocol) where each new release/build we do first is tested in our automation and our set of sample pipelines. We then make the release candidate available to the CI/CD systems of our internal, 1st party D365 application/solutions authors. Any discovered issues either result in a hotfix or rollback.
Only if a rel candidate "survives" in that first deployment station do we proceed to make it publicly available.
For this particular migration from the PS-stack to the pac CLI based implementation, we've had multiple release candidates deployed with our 1st party partner teams within MSFT, over a time of almost 8 weeks.
The challenge with backwards compatibility is the sheer amount of scenario permutations, both in the set of different input parameter permutations, but also the wide range of complexities of e.g. Dataverse solution and AppSource packages.

- Why not side by side?
We had considered a variation of the "version 0" vs "version 1" where we could have created a brand new set of tasks, each with a different identifier/name. But that would have meant a lot of migration work for existing users to update and validate their pipelines.

[remyblok]

A very nice write-up of the conundrum you are in. There are a few things I'd like to share on this. Hopefully it helps in the discussion en steps going forward.

- Per task version & Tool installer
The first question in my mind is: why is there a separate ToolInstaller Task. For the PS version I understand. It needs to download and install stuff. There are many reasons it could fail. It took a few minutes to complete etc.
As you describe the new version has everything internalized. This basically means that (for me) the main reasons for having a separate ToolsInstaller task are no longer there. Why not have the PAC tasks just kick off the install script just as the task runs. If it is just <1 sec then there is basically no impact in doing that. This then negates the need for a sepatrate ToolsInstaller for the PAC version. And thus you can 'mix and match' @0 and @1 versions of the tools. (if that is sensible is another question, but at at least you can)

- Custom dependencies on the PS tools
We created custom tasks/scripts that use the installed PowerShell scripts. We assume that the PowerShell tools are installed etc. Now the build breaks because the PowerShell stuff in no longer installed. Was it smart do do that? Maybe not. Will I be the only kid on the block who assumed in my PowerShell script that the ToolsInstaller installed certain PowerShell modules? No probably not.
This probably means that there are other build pipelines out there that fail. Not because there is a bug in the new version of the Tools. But because there is a definitive difference between how these versions behave. Basically a breaking change. Breaking changes most often mean new major version.

- Why Side by side!
I would think that reasons that you give are exactly why you should do Side by Side. I want to be in control of my pipelines. Now I have build failure galore. I have a customer going live, where the release pipeline now failed. The change was pretty much out of the blue. So I would be happy to have a v0 with a fat message "please switch to v1 at the earliest convenience, v0 is no longer supported". But at least I would be in control of the switch. I could plan the right moment and do the right testing for my setup and pipeline.

Another reason is that you can have two version, one stable and one new/beta. Users can gradually move over. When bugs are found they can be addressed while reverting to v0 resolves the imminent impact. The time it takes will be longer for users to move over. But at least they can do it in a controlled manner. Especially because, as you mentioned, there are so many different cases in which the tools are used. This give that control that we are all looking for.

Also, I can image you want to introduce breaking changed. I'm impressed that the tools are as backward compatible as they are! I wonder though, to what extend you some tasks be better it they were not backwards compatible. Maybe consistent naming of parameters between PAC parameters and the Tools as just an example. Would that not make the Tools as a whole better to understand? maybe not backward compatible between v0 and v1. But better when using v1 from scratch.

- Communication
Last but not least I would like to take a look at communication. I follow the Power Platform Blog (https://powerapps.microsoft.com/en-us/blog/) on a near daily basis. I would at least like to have seen a message there about these major changes, The PAC team publishes their changes there as well. Or similar to the previous paragraph, have a message indication in the v0 that major changes are coming soon.
Now everything just failed with no idea what was going on. Maybe it is mentioned somewhere. Maybe I missed the announcement. I haven't seen it though. In any case, some head-up would have been nice.

[davidjenni]

I appreciate the feedback, and I do agree we could have done better.

Given this and other feedback from July, we're changing our plan on how we move the PP-BT tasks to pac CLI:

• since we've in essence proven the obvious, that back-compat is hard when switching out an implementation stack as drastic as going from PS to dotnet/TS, we are rolling out the new pac CLI following proper semantic versioning, with the new pac CLI showing up as v2, leaving the PS side-by-side enabled as v0

• RE: Communication:
  We did bad on that one, our apologies, Comms are important. Earlier this week, information has gone out to MVP DLs and today to several blogs and updates to MSFT docs pages.
  E.g. here: https://aka.ms/pp-bt-migrate-to-v2

• RE: Side-by-Side:
  Agreed, pipeline owners need control to switch at their own pace. This is the approach we are now moving forward with.

• RE: ToolInstaller:
  When AzDevOps deploys tasks, it will NOT deploy the whole .vsix with all the tasks, instead, it optimizes to only bring the tasks referenced by the pipeline onto the agent. This in turn means, every task would need to contain the actual runtime, now pac CLI, in each and every task. This would cause our .vsix to explode in size considerably, which also then bumps into the upper .vsix that VS Marketplace accepts when publishing (currently around ~50 MB?). We've experimented with several other alternatives, like dynamically loading the pac CLI from nuget on first task requiring it. But that get's us back to depending on agent-external dependencies in pipelines. That has caused users and us many headaches in the past.
  We'll continue to explore alternatives to ultimately make ToolInstaller obsolete.

Thanks!

[remyblok]

Thank you for your reply. Also thanks for listening and making some changes to the approach. Great to see that you/we are moving forward with the new version of the tools. In the end using the pac CLI will be an improvement over the PowerShell version. Also interesting to learn how AzDevOps does the deployment optimization of tasks. I did not know that.
I'll be testing out the new version soon. If anything pops up I'll drop a comment/issue. Thanks for all the work and effort :)