Skip to content

Commit

Permalink
[build] Fix local workload template creation (#9250)
Browse files Browse the repository at this point in the history
The targets that set up the local workload have been moved into a
separate ConfigureLocalWorkload.targets, as the Directory.Build.targets
file was getting crowded.

The template pack is no longer deleted by DeleteExtractedWorkloadPacks,
allowing dotnet-local scripts to create new templates again.

The workload files and metadata created when installing the temporary
`android-deps` workload are now cleaned up, which should fix issues
where `bin/Debug/dotnet/dotnet` invocations could fail with:

    System.Collections.Generic.KeyNotFoundException: The given key 'android-deps' was not present in the dictionary.
       at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
       at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetManifestFromWorkload(WorkloadId workloadId)
       at Microsoft.DotNet.Cli.WorkloadCommandParser.ShowWorkloadsInfo(ParseResult parseResult, WorkloadInfoHelper workloadInfoHelper, IReporter reporter, String dotnetDir, Boolean showVersion)
       at Microsoft.DotNet.Cli.CommandLineInfo.PrintInfo()
       at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
       at Microsoft.DotNet.Cli.Program.Main(String[] args)

The temporary `android-deps` workload will now install all of the
runtime workloads that our workload extends, rather than a hard-coded
subset. This should allow the local workload setup to build projects
targeting both net9.0-android and net8.0-android.

The Windows build stage has been updated to use the local workload
configuration for all tests, including creating and building a template.
  • Loading branch information
pjcollins authored Aug 28, 2024
1 parent 47a3b33 commit 11fe204
Show file tree
Hide file tree
Showing 3 changed files with 157 additions and 89 deletions.
32 changes: 28 additions & 4 deletions build-tools/automation/yaml-templates/build-windows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,16 +53,25 @@ stages:
projects: Xamarin.Android.sln
arguments: '-c $(XA.Build.Configuration) -t:Prepare --no-restore -p:AutoProvision=true -bl:$(System.DefaultWorkingDirectory)\bin\Build$(XA.Build.Configuration)\dotnet-build-prepare.binlog'

# Build, pack .nupkgs, and extract workload packs to dotnet preview test directory
# Build Xamarin.Android and configure local workloads to test improved local build loop
- template: /build-tools/automation/yaml-templates/run-dotnet-preview.yaml
parameters:
project: Xamarin.Android.sln
arguments: >-
-t:BuildDotNet,PackDotNet -c $(XA.Build.Configuration) -v:n
-t:BuildDotNet -c $(XA.Build.Configuration) -v:n
-bl:$(System.DefaultWorkingDirectory)\bin\Build$(XA.Build.Configuration)\dotnet-build.binlog
displayName: Build Solution
continueOnError: false

- template: /build-tools/automation/yaml-templates/run-dotnet-preview.yaml
parameters:
project: build-tools/create-packs/Microsoft.Android.Sdk.proj
arguments: >-
-t:ConfigureLocalWorkload -c $(XA.Build.Configuration) -v:n -p:RunningOnCI=false
-bl:$(System.DefaultWorkingDirectory)\bin\Build$(XA.Build.Configuration)\local-workload.binlog
displayName: Run ConfigureLocalWorkload target
continueOnError: false

- template: /build-tools/automation/yaml-templates/install-dotnet-tool.yaml
parameters:
toolName: apkdiff
Expand All @@ -75,10 +84,25 @@ stages:
dotNetTestExtraArgs: --filter "TestCategory = SmokeTests"

- task: BatchScript@1
displayName: Test dotnet-local.cmd
displayName: Test dotnet-local.cmd - create template
inputs:
filename: dotnet-local.cmd
arguments: new android -o $(Build.StagingDirectory)/LocalWorkloadTest

- task: BatchScript@1
displayName: Test dotnet-local.cmd - build template
inputs:
filename: dotnet-local.cmd
arguments: build samples\HelloWorld\HelloWorld\HelloWorld.DotNet.csproj
arguments: build -v:n $(Build.StagingDirectory)/LocalWorkloadTest

# Pack .nupkgs and extract workload packs to dotnet preview test directory
- template: /build-tools/automation/yaml-templates/run-dotnet-preview.yaml
parameters:
project: Xamarin.Android.sln
arguments: >-
-t:PackDotNet -c $(XA.Build.Configuration) -v:n
-bl:$(System.DefaultWorkingDirectory)\bin\Build$(XA.Build.Configuration)\dotnet-pack.binlog
displayName: Test PackDotNet

- template: /build-tools/automation/yaml-templates/upload-results.yaml
parameters:
Expand Down
128 changes: 128 additions & 0 deletions build-tools/create-packs/ConfigureLocalWorkload.targets
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
<Project>

<ItemGroup>
<_FrameworkListInputs Include="$(MicrosoftAndroidRefPackDir)**" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidDefaultTargetDotnetApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidLatestStableApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidLatestUnstableApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_RuntimeListInputs Include="$(MicrosoftAndroidArmPackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidArm64PackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidx86PackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidx64PackDir)**" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidDefaultTargetDotnetApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidLatestStableApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidLatestUnstableApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_TemplatesInputs Include="$(XamarinAndroidSourcePath)src\Microsoft.Android.Templates\**" />
<_TemplatesOutputs Include="$(BuildOutputDirectory)lib\template-packs\microsoft.android.templates.$(AndroidPackVersion).nupkg" />
</ItemGroup>

<Target Name="CreateLocalRuntimeLists"
Inputs="$(MSBuildThisFile);@(_RuntimeListInputs)"
Outputs="@(_RuntimeListOutputs)">
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.Android.Runtime.proj"
Properties="FrameworkListFile=%(_RuntimeListOutputs.Identity);AndroidRID=%(_RuntimeListOutputs.AndroidRID)"
Targets="_GetRuntimePackItems;_GenerateFrameworkListFile"
/>
</Target>

<Target Name="CreateLocalFrameworkLists"
Inputs="$(MSBuildThisFile);@(_FrameworkListInputs)"
Outputs="@(_FrameworkListOutputs)">
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.Android.Ref.proj"
Properties="FrameworkListFile=%(_FrameworkListOutputs.Identity)"
Targets="_GetTargetingPackItems;_GenerateFrameworkListFile"
/>
</Target>

<Target Name="PackLocalTemplates"
Inputs="@(_TemplatesInputs)"
Outputs="@(_TemplatesOutputs)">
<ItemGroup>
<_PackProps Include="-v:n -c $(Configuration)" />
<_PackProps Include="-p:IncludeSymbols=False" />
<_PackProps Include="-p:OutputPath=$(BuildOutputDirectory)lib\template-packs" />
<_PackProps Include="-p:TemplatePackVersion=$(AndroidPackVersion)" />
<_PackProps Include="-p:PackageId=microsoft.android.templates" />
</ItemGroup>
<Exec Command="&quot;$(DotNetPreviewTool)&quot; pack @(_PackProps, ' ') &quot;$(XamarinAndroidSourcePath)src\Microsoft.Android.Templates\Microsoft.Android.Templates.csproj&quot;" />
</Target>

<UsingTask TaskName="GetAndroidWorkloadExtends" TaskFactory="RoslynCodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll">
<ParameterGroup>
<JsonFilePath ParameterType="System.String" Required="true" />
<ExtendsElement Output="true" ParameterType="System.String" />
</ParameterGroup>
<Task>
<Using Namespace="System" />
<Using Namespace="System.IO" />
<Code Type="Fragment" Language="cs">
<![CDATA[
var jsonContent = File.ReadAllText(JsonFilePath);
var startElement = "\"extends\" : [";
var startIndex = jsonContent.IndexOf(startElement);
var endIndex = jsonContent.IndexOf("]");
if (startIndex != -1) {
startIndex += startElement.Length;
endIndex = jsonContent.IndexOf("]", startIndex);
if (endIndex != -1) {
ExtendsElement = jsonContent.Substring(startIndex, endIndex - startIndex)?.Trim();
}
}
if (startIndex == -1 || endIndex == -1 || string.IsNullOrEmpty(ExtendsElement))
Log.LogError($"Failed to find extends element in workload json '{JsonFilePath}'");
]]>
</Code>
</Task>
</UsingTask>

<Target Name="InstallManifestAndDependencies"
DependsOnTargets="_GetDefaultPackageVersion">
<PropertyGroup>
<_LocalSdkManifestsFolder>$(BuildOutputDirectory)lib\sdk-manifests\$(DotNetSdkManifestsFolder)\</_LocalSdkManifestsFolder>
<_LocalAndroidManifestFolder>$(_LocalSdkManifestsFolder)microsoft.net.sdk.android\$(AndroidPackVersionLong)\</_LocalAndroidManifestFolder>
<_EmptyWorkloadDir>$(_LocalSdkManifestsFolder)android.deps.workload\0.0.1\</_EmptyWorkloadDir>
</PropertyGroup>

<MakeDir Directories="$(_LocalAndroidManifestFolder)" />
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.NET.Sdk.Android.proj"
Properties="WorkloadManifestJsonPath=$(_LocalAndroidManifestFolder)WorkloadManifest.json;WorkloadManifestTargetsPath=$(_LocalAndroidManifestFolder)WorkloadManifest.targets;WorkloadVersion=$(AndroidPackVersion)"
Targets="_GenerateXAWorkloadContent"
/>

<GetAndroidWorkloadExtends JsonFilePath="$(_LocalAndroidManifestFolder)WorkloadManifest.json">
<Output TaskParameter="ExtendsElement" PropertyName="AndroidWorkloadExtendsElement" />
</GetAndroidWorkloadExtends>

<PropertyGroup>
<_EmptyWorkloadJsonContent>
<![CDATA[
{"version": "0.0.1", "workloads": { "android-deps": { "extends" : [ $(AndroidWorkloadExtendsElement) ] } } }
]]>
</_EmptyWorkloadJsonContent>
</PropertyGroup>

<!-- Create empty workload to install dotnet/runtime dependencies, and then clean up the workload and metadata files from install -->
<Error Condition=" '$(AndroidWorkloadExtendsElement)' == '' " Text="Failed to find extends element in workload json" />
<MakeDir Directories="$(_EmptyWorkloadDir)" />
<WriteLinesToFile
File="$(_EmptyWorkloadDir)WorkloadManifest.json"
Lines="$(_EmptyWorkloadJsonContent)"
Overwrite="true"
/>
<Exec
Command="&quot;$(DotNetPreviewTool)&quot; workload install android-deps --configfile &quot;$(XamarinAndroidSourcePath)NuGet.config&quot; --skip-manifest-update --skip-sign-check --verbosity diag"
EnvironmentVariables="DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=$(BuildOutputDirectory)lib\sdk-manifests"
WorkingDirectory="$(XamarinAndroidSourcePath)"
/>
<RemoveDir Directories="$(_EmptyWorkloadDir)" />
</Target>

<PropertyGroup>
<ConfigureLocalWorkloadDependsOn Condition="'$(RunningOnCI)' != 'true'">CreateLocalFrameworkLists;CreateLocalRuntimeLists;InstallManifestAndDependencies;DeleteExtractedWorkloadPacks;PackLocalTemplates</ConfigureLocalWorkloadDependsOn>
</PropertyGroup>
<Target Name="ConfigureLocalWorkload" DependsOnTargets="$(ConfigureLocalWorkloadDependsOn)" />

</Project>
86 changes: 1 addition & 85 deletions build-tools/create-packs/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -213,90 +213,6 @@
</Target>

<!-- Targets for setting up a local workload test environment without needing to pack .nupkg files -->
<ItemGroup>
<_FrameworkListInputs Include="$(MicrosoftAndroidRefPackDir)**" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidDefaultTargetDotnetApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidLatestStableApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_FrameworkListOutputs Include="$(BuildOutputDirectory)lib\packs\Microsoft.Android.Ref.$(AndroidLatestUnstableApiLevel)\$(AndroidPackVersion)\data\FrameworkList.xml" />
<_RuntimeListInputs Include="$(MicrosoftAndroidArmPackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidArm64PackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidx86PackDir)**" />
<_RuntimeListInputs Include="$(MicrosoftAndroidx64PackDir)**" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidDefaultTargetDotnetApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidLatestStableApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_RuntimeListOutputs Include="@(AndroidSupportedTargetJitAbi->'$(BuildOutputDirectory)lib\packs\Microsoft.Android.Runtime.$(AndroidLatestUnstableApiLevel).%(AndroidRID)\$(AndroidPackVersion)\data\RuntimeList.xml')" AndroidRID="%(AndroidRID)" />
<_TemplatesInputs Include="$(XamarinAndroidSourcePath)src\Microsoft.Android.Templates\**" />
<_TemplatesOutputs Include="$(BuildOutputDirectory)lib\template-packs\microsoft.android.templates.$(AndroidPackVersion).nupkg" />
</ItemGroup>

<Target Name="CreateLocalRuntimeLists"
Inputs="$(MSBuildThisFile);@(_RuntimeListInputs)"
Outputs="@(_RuntimeListOutputs)">
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.Android.Runtime.proj"
Properties="FrameworkListFile=%(_RuntimeListOutputs.Identity);AndroidRID=%(_RuntimeListOutputs.AndroidRID)"
Targets="_GetRuntimePackItems;_GenerateFrameworkListFile"
/>
</Target>

<Target Name="CreateLocalFrameworkLists"
Inputs="$(MSBuildThisFile);@(_FrameworkListInputs)"
Outputs="@(_FrameworkListOutputs)">
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.Android.Ref.proj"
Properties="FrameworkListFile=%(_FrameworkListOutputs.Identity)"
Targets="_GetTargetingPackItems;_GenerateFrameworkListFile"
/>
</Target>

<Target Name="PackAndCopyTemplates"
Inputs="@(_TemplatesInputs)"
Outputs="@(_TemplatesOutputs)">
<ItemGroup>
<_PackProps Include="-v:n -c $(Configuration)" />
<_PackProps Include="-p:IncludeSymbols=False" />
<_PackProps Include="-p:OutputPath=$(DotNetPreviewPath)template-packs" />
<_PackProps Include="-p:TemplatePackVersion=$(AndroidPackVersion)" />
<_PackProps Include="-p:PackageId=microsoft.android.templates" />
</ItemGroup>
<Exec Command="&quot;$(DotNetPreviewTool)&quot; pack @(_PackProps, ' ') &quot;$(XamarinAndroidSourcePath)src\Microsoft.Android.Templates\Microsoft.Android.Templates.csproj&quot;" />
</Target>

<Target Name="InstallManifestAndDependencies"
DependsOnTargets="DeleteExtractedWorkloadPacks;_GetDefaultPackageVersion">
<PropertyGroup>
<_LocalSdkManifestsFolder>$(BuildOutputDirectory)lib\sdk-manifests\$(DotNetSdkManifestsFolder)\</_LocalSdkManifestsFolder>
<_LocalAndroidManifestFolder>$(_LocalSdkManifestsFolder)microsoft.net.sdk.android\$(AndroidPackVersionLong)\</_LocalAndroidManifestFolder>
<_EmptyWorkloadDir>$(_LocalSdkManifestsFolder)android.deps.workload\0.0.1\</_EmptyWorkloadDir>
<_EmptyWorkloadJsonContent>
<![CDATA[
{"version": "0.0.1", "workloads": { "android-deps": { "extends" : [ "microsoft-net-runtime-android", "microsoft-net-runtime-android-aot" ] } } }
]]>
</_EmptyWorkloadJsonContent>
</PropertyGroup>
<MakeDir Directories="$(_LocalAndroidManifestFolder)" />
<MSBuild
Projects="$(MSBuildThisFileDirectory)Microsoft.NET.Sdk.Android.proj"
Properties="WorkloadManifestJsonPath=$(_LocalAndroidManifestFolder)WorkloadManifest.json;WorkloadManifestTargetsPath=$(_LocalAndroidManifestFolder)WorkloadManifest.targets;WorkloadVersion=$(AndroidPackVersion)"
Targets="_GenerateXAWorkloadContent"
/>
<!-- Create empty workload to install dotnet/runtime dependencies -->
<MakeDir Directories="$(_EmptyWorkloadDir)" />
<WriteLinesToFile
File="$(_EmptyWorkloadDir)WorkloadManifest.json"
Lines="$(_EmptyWorkloadJsonContent)"
Overwrite="true"
/>
<Exec
Command="&quot;$(DotNetPreviewTool)&quot; workload install android-deps --configfile &quot;$(XamarinAndroidSourcePath)NuGet.config&quot; --skip-manifest-update --skip-sign-check --verbosity diag"
EnvironmentVariables="DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=$(BuildOutputDirectory)lib\sdk-manifests"
WorkingDirectory="$(XamarinAndroidSourcePath)"
/>
</Target>

<PropertyGroup>
<ConfigureLocalWorkloadDependsOn Condition="'$(RunningOnCI)' != 'true'">CreateLocalFrameworkLists;CreateLocalRuntimeLists;PackAndCopyTemplates;InstallManifestAndDependencies</ConfigureLocalWorkloadDependsOn>
</PropertyGroup>
<Target Name="ConfigureLocalWorkload" DependsOnTargets="$(ConfigureLocalWorkloadDependsOn)" />
<Import Project="ConfigureLocalWorkload.targets" />

</Project>

0 comments on commit 11fe204

Please sign in to comment.