[Xamarin.Android.Build.Tasks] Add @(AndroidPackagingOptionsInclude) (#…

…8459)

Context: #8456
Context: 2726a38
Context: xamarin/monodroid@43583d0

Commit 2726a386added the `@(AndroidPackagingOptionsExclude)` ItemGroup,
which is used to exclude certain files from the final apk.
xamarin/monodroid@43583d02 updated the `_BuildApkFastDev` target so
that `@(AndroidPackagingOptionsExclude)` was provided to `<BuildApk/>`.
`@(AndroidPackagingOptionsExclude)` was initially designed to filter
out Kotlin based meta data:

	<AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.kotlin_*'))" />

However, it turns out the default file glob of `*.kotlin_*` is too
restrictive, removing e.g. `kotlin/reflect/reflect.kotlin_builtins`,
which can result in runtime errors (see #8456):

	java.lang.AssertionError: Built-in class kotlin.Any is not found
	   at kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns$3.invoke(KotlinBuiltIns.java:93)

The "obvious" workaround is to `%(Remove)` the offending item:

	<ItemGroup>
	  <AndroidPackagingOptionsExclude Remove="$([MSBuild]::Escape('*.kotlin_*'))" />
	</ItemGroup>

Unfortunately this doesn't work, because of the order in which
`<ItemGroups/>`s are evaluated by MSBuild.  At the time the `.csproj`
is evaluated, `Xamarin.Android.Common.targets` has not yet been loaded,
and thus `@(AndroidPackagingOptionsExclude)` has not yet been set.

The "less obvious but works" workaround is to change the item group
within a target:

	<Target Name="_UpdateAndroidPackagingOptionsExclude" AfterTargets="AfterBuild">
	  <ItemGroup>
	    <AndroidPackagingOptionsExclude Remove="$([MSBuild]::Escape('*.kotlin_*'))" />
	    <AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.kotlin_m*'))" />
	  </ItemGroup>
	</Target>

This is not very intuitive.

Make this easier by moving the definition of the default
`@(AndroidPackagingOptionsExclude)` values into `AutoImport.props`,
which happens before the `.csproj` is read.  This allows the more
"obvious" `%(Remove)` usage to work as expected.

The next problem is that the default file glob of `*.kotlin_*` was
too expansive, excluding the `*.kotlin_builtins` files which we now
know are important.  We could try changing the default glob to
something else, such as the `*.kotlin_m*` value used by
`_UpdateAndroidPackagingOptionsExclude`, but this can mean that files
we didn't want to include are now included.

Partially rethink things by introducing a new
`@(AndroidPackagingOptionsInclude)` item group, which contains file
globs.  Any packaging artifact which matches the
`@(AndroidPackagingOptionsInclude)` file glob will be included, before
`@(AndroidPackagingOptionsInclude)` has a chance to exclude it.

`@(AndroidPackagingOptionsInclude)` will be public, and can be used
by NuGet packages such as [Xamarin.Kotlin.Reflect][0] to ensure that
specific files are included.  This can be done in the NuGet by
providing a `.targets` file which adds items to
`@(AndroidPackagingOptionsInclude)`.

[0]: https://www.nuget.org/packages/Xamarin.Kotlin.Reflect

Documentation/guides/building-apps/build-items.md

@@ -233,15 +233,62 @@ excluded from the final package. The default values are as follows
 	<AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.kotlin_*')" />
 </ItemGroup>
 ```
+
 Items can use file blob characters for wildcards such as `*` and `?`.
 However these Items MUST use URL encoding or '$([MSBuild]::Escape(''))'.
 This is so MSBuild does not try to interpret them as actual file wildcards.
 
+For example 
+
+```
+<ItemGroup>
+	<AndroidPackagingOptionsExclude Include="%2A.foo_%2A" />
+  <AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.foo')" />
+</ItemGroup>
+```
+
 NOTE: `*`, `?` and `.` will be replaced in the `BuildApk` task with the
-appropriate RegEx expressions.
+appropriate file globs.
+
+If the default file glob is too restrictive you can remove it by adding the 
+following to your csproj
+
+```
+<ItemGroup>
+	<AndroidPackagingOptionsExclude Remove="$([MSBuild]::Escape('*.kotlin_*')" />
+</ItemGroup>
+```
 
 Added in Xamarin.Android 13.1 and .NET 7.
 
+## AndroidPackagingOptionsInclude
+
+A set of file glob compatible items which will allow for items to be
+included from the final package. The default values are as follows
+
+```
+<ItemGroup>
+	<AndroidPackagingOptionsInclude Include="$([MSBuild]::Escape('*.kotlin_builtins')" />
+</ItemGroup>
+```
+
+Items can use file blob characters for wildcards such as `*` and `?`.
+However these Items MUST use URL encoding or '$([MSBuild]::Escape(''))'.
+This is so MSBuild does not try to interpret them as actual file wildcards.
+For example 
+
+```
+<ItemGroup>
+	<AndroidPackagingOptionsInclude Include="%2A.foo_%2A" />
+  <AndroidPackagingOptionsInclude Include="$([MSBuild]::Escape('*.foo')" />
+</ItemGroup>
+```
+
+NOTE: `*`, `?` and `.` will be replaced in the `BuildApk` task with the
+appropriate file globs.
+
+Added in .NET 9.
+
 ## AndroidResource
 
 All files with an *AndroidResource* build action are compiled into

src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/Sdk/AutoImport.props

@@ -61,4 +61,10 @@ https://github.com/dotnet/designs/blob/4703666296f5e59964961464c25807c727282cae/
     <ProguardConfiguration Include="**\proguard.txt" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder)" />
   </ItemGroup>
 
+  <ItemGroup>
+    <AndroidPackagingOptionsExclude Include="DebugProbesKt.bin" />
+    <AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.kotlin*'))" />
+    <AndroidPackagingOptionsInclude Include="$([MSBuild]::Escape('*.kotlin_builtins'))" />
+  </ItemGroup>
+
 </Project>

src/Xamarin.Android.Build.Tasks/Tasks/BuildApk.cs

@@ -76,6 +76,8 @@ public class BuildApk : AndroidTask
 
 		public string [] ExcludeFiles { get; set; }
 
+		public string [] IncludeFiles { get; set; }
+
 		public string Debug { get; set; }
 
 		public string AndroidSequencePointsMode { get; set; }
@@ -130,6 +132,8 @@ protected virtual void FixupArchive (ZipArchiveEx zip) { }
 
 		List<Regex> excludePatterns = new List<Regex> ();
 
+		List<Regex> includePatterns = new List<Regex> ();
+
 		void ExecuteWithAbi (string [] supportedAbis, string apkInputPath, string apkOutputPath, bool debug, bool compress, IDictionary<string, CompressedAssemblyInfo> compressedAssembliesInfo, string assemblyStoreApkName)
 		{
 			ArchiveFileList files = new ArchiveFileList ();
@@ -258,13 +262,22 @@ void ExecuteWithAbi (string [] supportedAbis, string apkInputPath, string apkOut
 							}
 							// check for ignored items
 							bool exclude = false;
-							foreach (var pattern in excludePatterns) {
-								if(pattern.IsMatch (path)) {
-									Log.LogDebugMessage ($"Ignoring jar entry '{name}' from '{Path.GetFileName (jarFile)}'. Filename matched the exclude pattern '{pattern}'.");
-									exclude = true;
+							bool forceInclude = false;
+							foreach (var include in includePatterns) {
+								if (include.IsMatch (path)) {
+									forceInclude = true;
 									break;
 								}
 							}
+							if (!forceInclude) {
+								foreach (var pattern in excludePatterns) {
+									if(pattern.IsMatch (path)) {
+										Log.LogDebugMessage ($"Ignoring jar entry '{name}' from '{Path.GetFileName (jarFile)}'. Filename matched the exclude pattern '{pattern}'.");
+										exclude = true;
+										break;
+									}
+								}
+							}
 							if (exclude)
 								continue;
 							if (string.Compare (Path.GetFileName (name), "AndroidManifest.xml", StringComparison.OrdinalIgnoreCase) == 0) {
@@ -310,6 +323,9 @@ public override bool RunTask ()
 			foreach (var pattern in ExcludeFiles ?? Array.Empty<string> ()) {
 				excludePatterns.Add (FileGlobToRegEx (pattern, RegexOptions.IgnoreCase | RegexOptions.Compiled));
 			}
+			foreach (var pattern in IncludeFiles ?? Array.Empty<string> ()) {
+				includePatterns.Add (FileGlobToRegEx (pattern, RegexOptions.IgnoreCase | RegexOptions.Compiled));
+			}
 
 			bool debug = _Debug;
 			bool compress = !debug && EnableCompression;

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs

@@ -585,6 +585,51 @@ public void CheckExcludedFilesAreMissing ()
 			}
 		}
 
+		[Test]
+		public void CheckExcludedFilesCanBeModified ()
+		{
+
+			var proj = new XamarinAndroidApplicationProject () {
+				IsRelease = true,
+			};
+			proj.PackageReferences.Add (KnownPackages.Xamarin_Kotlin_StdLib_Common);
+			using (var b = CreateApkBuilder ()) {
+				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
+				var apk = Path.Combine (Root, b.ProjectDirectory,
+					proj.OutputPath, $"{proj.PackageName}-Signed.apk");
+				string expected = $"Ignoring jar entry 'kotlin/Error.kotlin_metadata'";
+				Assert.IsTrue (b.LastBuildOutput.ContainsText (expected), $"Error.kotlin_metadata should have been ignored.");
+				using (var zip = ZipHelper.OpenZip (apk)) {
+					Assert.IsFalse (zip.ContainsEntry ("kotlin/Error.kotlin_metadata"), "Error.kotlin_metadata should have been ignored.");
+				}
+				proj.OtherBuildItems.Add (new BuildItem ("AndroidPackagingOptionsExclude") {
+					Remove = () => "$([MSBuild]::Escape('*.kotlin*'))",
+				});
+				Assert.IsTrue (b.Clean (proj), "Clean should have succeeded.");
+				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
+				using (var zip = ZipHelper.OpenZip (apk)) {
+					Assert.IsTrue (zip.ContainsEntry ("kotlin/Error.kotlin_metadata"), "Error.kotlin_metadata should have been included.");
+				}
+			}
+		}
+
+		[Test]
+		public void CheckIncludedFilesArePresent ()
+		{
+			var proj = new XamarinAndroidApplicationProject () {
+				IsRelease = true,
+			};
+			proj.PackageReferences.Add (KnownPackages.Xamarin_Kotlin_Reflect);
+			using (var b = CreateApkBuilder ()) {
+				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
+				var apk = Path.Combine (Root, b.ProjectDirectory,
+					proj.OutputPath, $"{proj.PackageName}-Signed.apk");
+				using (var zip = ZipHelper.OpenZip (apk)) {
+					Assert.IsTrue (zip.ContainsEntry ("kotlin/reflect/reflect.kotlin_builtins"), "reflect.kotlin_builtins should have been included.");
+				}
+			}
+		}
+
 		[Test]
 		[TestCase (1, -1)]
 		[TestCase (5, -1)]

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownPackages.cs

@@ -523,6 +523,10 @@ public static class KnownPackages
 			Id = "Xamarin.Kotlin.Stdlib.Common",
 			Version = "1.6.20.1"
 		};
+		public static Package Xamarin_Kotlin_Reflect = new Package {
+			Id = "Xamarin.Kotlin.Reflect",
+			Version = "1.9.10.2"
+		};
 		public static Package Acr_UserDialogs = new Package {
 			Id = "Acr.UserDialogs",
 			Version = "8.0.1",

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

@@ -358,11 +358,6 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
 	</AllowedReferenceRelatedFileExtensions>
 </PropertyGroup>
 
-<ItemGroup>
-	<AndroidPackagingOptionsExclude Include="DebugProbesKt.bin" />
-	<AndroidPackagingOptionsExclude Include="$([MSBuild]::Escape('*.kotlin_*'))" />
-</ItemGroup>
-
 <!-- Assets build properties -->
 <PropertyGroup>
 	<MonoAndroidAssetsDirIntermediate>$(IntermediateOutputPath)assets\</MonoAndroidAssetsDirIntermediate>
@@ -2136,6 +2131,7 @@ because xbuild doesn't support framework reference assemblies.
     CheckedBuild="$(_AndroidCheckedBuild)"
     RuntimeConfigBinFilePath="$(_BinaryRuntimeConfigPath)"
     ExcludeFiles="@(AndroidPackagingOptionsExclude)"
+    IncludeFiles="@(AndroidPackagingOptionsInclude)"
     ZipFlushFilesLimit="$(_ZipFlushFilesLimit)"
     ZipFlushSizeLimit="$(_ZipFlushSizeLimit)"
     UseAssemblyStore="$(AndroidUseAssemblyStore)">
@@ -2171,6 +2167,7 @@ because xbuild doesn't support framework reference assemblies.
       CheckedBuild="$(_AndroidCheckedBuild)"
       RuntimeConfigBinFilePath="$(_BinaryRuntimeConfigPath)"
       ExcludeFiles="@(AndroidPackagingOptionsExclude)"
+      IncludeFiles="@(AndroidPackagingOptionsInclude)"
       ZipFlushFilesLimit="$(_ZipFlushFilesLimit)"
       ZipFlushSizeLimit="$(_ZipFlushSizeLimit)"
       UseAssemblyStore="$(AndroidUseAssemblyStore)">