Skip to content

Commit

Permalink
dotnet-local & net8: local dotnet first in PATH (#8222)
Browse files Browse the repository at this point in the history
The .NET 8 `dotnet` command appears to require that `dotnet` be inThe .NET 8 `dotnet` command appears to require that `dotnet` be in
`$PATH`, lest it invoke a *different* `dotnet`.  This can result in
weird error messages:

	% ~/Developer/src/xamarin/xamarin-android/dotnet-local.sh build *.csproj 
	…
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : You must install or update .NET to run this application. 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :  
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : App: …/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/bincore/csc.dll 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : Architecture: x64 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : Framework: 'Microsoft.NETCore.App', version '8.0.0-rc.1.23371.3' (x64) 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : .NET location: /usr/local/share/dotnet/ 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :  
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : The following frameworks were found: 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :   6.0.16 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :   7.0.3 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :   7.0.4 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :   7.0.5 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :   7.0.9 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :  
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : Learn about framework resolution: 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : https://aka.ms/dotnet/app-launch-failed 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error :  
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : To install missing framework, download: 
	…/dotnet/sdk/8.0.100-rc.1.23373.1/Roslyn/Microsoft.CSharp.Core.targets(80,5): error : https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.0-rc.1.23371.3&arch=x64&rid=osx.13-x64 
	…

To focus on the weirdness: `dotnet-local.sh` uses the .NET 8 that we
provision into `bin/$(Configuration)/dotnet`, which is .NET 8, while
the error message states that it can only find .NET 6 and 7 from
`/usr/local/share/dotnet`, and *isn't* looking at `bin/*/dotnet`!

.NET 8 Roslyn apparently requires that the "executing" `dotnet` be in
`$PATH`:

	% PATH=…/xamarin-android/bin/Debug/dotnet:$PATH \
	    …/xamarin-android/dotnet-local.sh build *.csproj
	# no errors

Update `dotnet-local.sh` and `dotnet-local.cmd` so that `bin/*/dotnet`
is prepended to `$PATH`.  Additionally, for readability in
`dotnet-loca.sh`, explicitly export all `DOTNET*` environment
variables so that they don't all need to be on one line.

Update build scripts to use `-nodeReuse:false` instead of `-nr:false`
for readability.

…and some notes about `dotnet-local.cmd`: [`CMD.EXE`][0] environment
variable handling is "baroque".

By default, evironment variables added within a `.cmd` script are
exported to the caller.  This is similar to Unixy platforms
["source"][1]ing a script instead of executing a script.  To avoid
this behavior, the script should use [SETLOCAL][2].  As we want
`dotnet-local.cmd` to update `%PATH%`, add use of `SETLOCAL` so that
we don't update `%PATH%` on every `dotnet-local.cmd` invocation.

Environment variables are evalated *when a statement is read*,
***not*** when that statement is *executed*.  This can make for some
"weird" behavior:

	IF EXIST "%ROOT%\bin\Release\dotnet\dotnet.exe" (
	    SET XA_DOTNET_ROOT=%ROOT%\bin\Release\dotnet
	    echo XA_DOTNET_ROOT=%XA_DOTNET_ROOT%
	)

The above fragment will print `XA_DOTNET_ROOT=` because
`%XA_DOTNET_ROOT%` is evaluated when the entire `IF EXIST …` statment
is read, *not* when the `echo` is executed.  This has similar
"wierdness" if we instead attempt to update `PATH`:

	IF EXIST "%ROOT%\bin\Release\dotnet\dotnet.exe" (
	    SET PATH=%ROOT%\bin\Release\dotnet;%PATH%
	)

The above invariably fails with weirdness such as:

	\NSIS\ was unexpected at this time.

or

	\Microsoft was unexpected at this time.

The workaround is one of two options:

 1. Consider use "delayed environment variable expansion"; see the
    `SET /?` output [^0].

 2. Don't Do That™: don't `SET` an environment variable to include
    values which are also `SET` in the same statement, and `(…)`
    is evaluated as a single statement.

We opt to go with (2), drastically simplifying `dotnet-local.cmd` so
that the `COMMAND` within `IF EXIST filename COMMAND` *only* sets
environment variables which do not reference other environment
variables within the same `COMMAND`

[0]: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cmd
[1]: https://linuxize.com/post/bash-source-command/
[2]: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setlocal

[^0]: From `SET /?`:

	Delayed environment variable expansion is useful for getting around
	the limitations of the current expansion which happens when a line
	of text is read, not when it is executed.  The following example
	demonstrates the problem with immediate variable expansion:

	    set VAR=before
	    if "%VAR%" == "before" (
	        set VAR=after
	        if "%VAR%" == "after" @echo If you see this, it worked
	    )

	would never display the message, since the %VAR% in BOTH IF statements
	is substituted when the first IF statement is read, since it logically
	includes the body of the IF, which is a compound statement.  So the
	IF inside the compound statement is really comparing "before" with
	"after" which will never be equal.  Similarly, the following example
	will not work as expected:

	    set LIST=
	    for %i in (*) do set LIST=%LIST% %i
	    echo %LIST%

	in that it will NOT build up a list of files in the current directory,
	but instead will just set the LIST variable to the last file found.
	Again, this is because the %LIST% is expanded just once when the
	FOR statement is read, and at that time the LIST variable is empty.
	So the actual FOR loop we are executing is:

	    for %i in (*) do set LIST= %i

	which just keeps setting LIST to the last file found.

	Delayed environment variable expansion allows you to use a different
	character (the exclamation mark) to expand environment variables at
	execution time. …
  • Loading branch information
dellis1972 authored Aug 12, 2023
1 parent 7bf75ba commit 341a962
Show file tree
Hide file tree
Showing 8 changed files with 63 additions and 35 deletions.
11 changes: 4 additions & 7 deletions Documentation/building/windows/instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,18 @@ MSBuild version 15 or later is required.

5. In a [Developer Command Prompt][developer-prompt], prepare the project:

dotnet msbuild Xamarin.Android.sln -t:Prepare
dotnet msbuild Xamarin.Android.sln -t:Prepare -nodeReuse:false
dotnet msbuild external\Java.Interop\Java.Interop.sln -t:Prepare -nodeReuse:false

This will ensure that the build dependencies are installed, perform
`git submodule update`, download NuGet dependencies, and other
"preparatory" and pre-build tasks that need to be performed.

6. Build the project:

dotnet-local.cmd build Xamarin.Android.sln -m:1
dotnet-local.cmd build Xamarin.Android.sln -nodeReuse:false

7. In order to use the in-tree Xamarin.Android, build xabuild:

msbuild tools/xabuild/xabuild.csproj /restore

8. (For Microsoft team members only - Optional) In a [Developer Command
7. (For Microsoft team members only - Optional) In a [Developer Command
Prompt][developer-prompt], build external proprietary git
dependencies:

Expand Down
20 changes: 16 additions & 4 deletions build-tools/automation/azure-pipelines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ stages:
restoreNUnitConsole: false
updateMono: false
xaprepareScenario: EmulatorTestDependencies

- task: DownloadPipelineArtifact@2
inputs:
artifactName: $(TestAssembliesArtifactName)
Expand Down Expand Up @@ -140,7 +140,7 @@ stages:
arguments: -t:PrepareJavaInterop -c $(XA.Build.Configuration) --no-restore
displayName: prepare java.interop $(XA.Build.Configuration)
continueOnError: false

- template: yaml-templates/start-stop-emulator.yaml

- template: yaml-templates/apk-instrumentation.yaml
Expand Down Expand Up @@ -201,7 +201,7 @@ stages:
extraBuildArgs: -p:TestsFlavor=AotLlvm -p:EnableLLVM=true -p:AndroidEnableProfiledAot=false
artifactSource: bin/Test$(XA.Build.Configuration)/$(DotNetTargetFramework)-android/Mono.Android.NET_Tests-Signed.aab
artifactFolder: $(DotNetTargetFramework)-AotLlvm

- template: yaml-templates/apk-instrumentation.yaml
parameters:
configuration: $(XA.Build.Configuration)
Expand All @@ -220,13 +220,19 @@ stages:
artifactSource: bin/Test$(XA.Build.Configuration)/$(DotNetTargetFramework)-android/Xamarin.Android.JcwGen_Tests-Signed.apk
artifactFolder: $(DotNetTargetFramework)-FastDev_Assemblies_Dexes
extraBuildArgs: /p:AndroidFastDeploymentType=Assemblies:Dexes

- template: yaml-templates/run-nunit-tests.yaml
parameters:
testRunTitle: Xamarin.Android.Tools.Aidl-Tests - macOS
testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/$(DotNetStableTargetFramework)/Xamarin.Android.Tools.Aidl-Tests.dll
testResultsFile: TestResult-Aidl-Tests-macOS-$(XA.Build.Configuration).xml

- task: ShellScript@2
displayName: Test dotnet-local.sh
inputs:
scriptPath: dotnet-local.sh
args: build samples/HelloWorld/HelloWorld/HelloWorld.DotNet.csproj

- ${{ if ne(parameters.macTestAgentsUseCleanImages, true) }}:
- template: yaml-templates/start-stop-emulator.yaml
parameters:
Expand Down Expand Up @@ -276,6 +282,12 @@ stages:
dotNetTestExtraArgs: --filter "TestCategory = SmokeTests"
testResultsFile: TestResult-NETSmokeMSBuildTests-Linux-$(XA.Build.Configuration).xml

- task: ShellScript@2
displayName: Test dotnet-local.sh
inputs:
scriptPath: dotnet-local.sh
args: build samples/HelloWorld/HelloWorld/HelloWorld.DotNet.csproj

- template: yaml-templates/upload-results.yaml
parameters:
configuration: $(XA.Build.Configuration)
Expand Down
6 changes: 6 additions & 0 deletions build-tools/automation/yaml-templates/build-windows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,12 @@ stages:
testResultsFile: TestResult-SmokeMSBuildTests-WinDotnetBuild-$(XA.Build.Configuration).xml
dotNetTestExtraArgs: --filter "TestCategory = SmokeTests"

- task: BatchScript@1
displayName: Test dotnet-local.cmd
inputs:
filename: dotnet-local.cmd
arguments: build samples\HelloWorld\HelloWorld\HelloWorld.DotNet.csproj

- template: upload-results.yaml
parameters:
artifactName: ${{ parameters.buildResultArtifactName }}
Expand Down
1 change: 1 addition & 0 deletions build-tools/scripts/PrepareWindows.targets
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,6 @@
/>
<Exec Command="dotnet $(_XAPrepareExe) $(_XAPrepareStandardArgs) -a" WorkingDirectory="$(_TopDir)" />
<MSBuild Projects="$(_TopDir)\Xamarin.Android.BootstrapTasks.sln" Targets="Restore;Build" />
<CallTarget Targets="PrepareJavaInterop" />
</Target>
</Project>
16 changes: 7 additions & 9 deletions build.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,21 @@ IF ERRORLEVEL 1 CALL:FAILED_CASE
IF ERRORLEVEL 1 CALL :DEFAULT_CASE

:Prepare_CASE
dotnet build Xamarin.Android.sln -t:Prepare -nr:false
dotnet msbuild Xamarin.Android.sln -t:Prepare -nodeReuse:false
GOTO END_CASE
:PrepareExternal_CASE
dotnet build Xamarin.Android.sln -t:PrepareExternal -nr:false
dotnet build Xamarin.Android.sln -t:PrepareExternal -nodeReuse:false
GOTO END_CASE
:Build_CASE
dotnet-local.cmd build Xamarin.Android.sln -nr:false
dotnet-local.cmd build tools/xabuild/xabuild.csproj -nr:false
dotnet-local.cmd build Xamarin.Android.sln -nodeReuse:false
GOTO END_CASE
:Pack_CASE
dotnet-local.cmd build Xamarin.Android.sln -t:PackDotNet -nr:false
dotnet-local.cmd build Xamarin.Android.sln -t:PackDotNet -nodeReuse:false
GOTO END_CASE
:DEFAULT_CASE
dotnet build Xamarin.Android.sln -t:Prepare -nr:false
dotnet-local.cmd build Xamarin.Android.sln -nr:false
dotnet-local.cmd build tools/xabuild/xabuild.csproj -nr:false
dotnet-local.cmd build Xamarin.Android.sln -t:PackDotNet -nr:false
dotnet msbuild Xamarin.Android.sln -t:Prepare -nodeReuse:false
dotnet-local.cmd build Xamarin.Android.sln -nodeReuse:false
dotnet-local.cmd build Xamarin.Android.sln -t:PackDotNet -nodeReuse:false
GOTO END_CASE
:FAILED_CASE
echo "Failed to find an instance of Visual Studio. Please check it is correctly installed."
Expand Down
23 changes: 16 additions & 7 deletions dotnet-local.cmd
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
@echo off
SETLOCAL

SET ROOT=%~dp0

IF EXIST "%ROOT%\bin\Release\dotnet\dotnet.exe" (
SET DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=%ROOT%\bin\Release\lib\sdk-manifests
SET DOTNETSDK_WORKLOAD_PACK_ROOTS=%ROOT%\bin\Release\lib
call "%ROOT%\bin\Release\dotnet\dotnet.exe" %*
SET XA_CONFIG=Release
) ELSE IF EXIST "%ROOT%\bin\Debug\dotnet\dotnet.exe" (
SET DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=%ROOT%\bin\Debug\lib\sdk-manifests
SET DOTNETSDK_WORKLOAD_PACK_ROOTS=%ROOT%\bin\Debug\lib
call "%ROOT%\bin\Debug\dotnet\dotnet.exe" %*
SET XA_CONFIG=Debug
) ELSE (
echo "You need to run 'msbuild Xamarin.Android.sln /t:Prepare' first."
)
goto :exit
)

SET XA_DOTNET_ROOT=%ROOT%\bin\%XA_CONFIG%\dotnet
SET PATH=%XA_DOTNET_ROOT%;%PATH%
SET DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=%ROOT%\bin\%XA_CONFIG%\lib\sdk-manifests
SET DOTNETSDK_WORKLOAD_PACK_ROOTS=%ROOT%\bin\%XA_CONFIG%\lib

call "%XA_DOTNET_ROOT%\dotnet.exe" %*

:exit
19 changes: 12 additions & 7 deletions dotnet-local.sh
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
#!/bin/bash
ROOT="$(dirname "${BASH_SOURCE}")"
FULLROOT="$(cd "${ROOT}"; pwd)"
if [[ -x "${ROOT}/bin/Release/dotnet/dotnet" ]]; then
DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=${FULLROOT}/bin/Release/lib/sdk-manifests DOTNETSDK_WORKLOAD_PACK_ROOTS=${FULLROOT}/bin/Release/lib exec ${ROOT}/bin/Release/dotnet/dotnet "$@"
elif [[ -x "${ROOT}/bin/Debug/dotnet/dotnet" ]]; then
DOTNETSDK_WORKLOAD_MANIFEST_ROOTS=${FULLROOT}/bin/Debug/lib/sdk-manifests DOTNETSDK_WORKLOAD_PACK_ROOTS=${FULLROOT}/bin/Debug/lib exec ${ROOT}/bin/Debug/dotnet/dotnet "$@"
else
echo "You need to run 'make prepare' first."
fi
for config in Release Debug ; do
if [[ ! -x "${ROOT}/bin/${config}/dotnet/dotnet" ]] ; then
continue
fi
XA_DOTNET_ROOT="${FULLROOT}/bin/${config}/dotnet"
export PATH="${XA_DOTNET_ROOT}:${PATH}"
export DOTNETSDK_WORKLOAD_MANIFEST_ROOTS="${FULLROOT}/bin/${config}/lib/sdk-manifests"
export DOTNETSDK_WORKLOAD_PACK_ROOTS="${FULLROOT}/bin/${config}/lib"
exec "${ROOT}/bin/${config}/dotnet/dotnet" "$@"
done

echo "You need to run 'make prepare' first."
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ List<string> GetDefaultCommandLineArgs (string verb, string target = null, strin
$"/flp1:LogFile=\"{BuildLogFile}\";Encoding=UTF-8;Verbosity={Verbosity}",
$"/bl:\"{Path.Combine (testDir, $"{(string.IsNullOrEmpty (target) ? "msbuild" : target)}.binlog")}\"",
"-m:1",
"-nr:false",
"-nodeReuse:false",
"/p:_DisableParallelAot=true",
};
if (!string.IsNullOrEmpty (target)) {
Expand Down

0 comments on commit 341a962

Please sign in to comment.