[Swift] Add support for Windows Docker image #15071
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
cc: @compnerd |
|
This feels familiar 👀 (#13093) |
|
@shahmishal where is that script? I suspect that we should change that to run |
|
@grooverdan, @tianon thanks! I did poke a bit more at this - the problem is that this is running under MinGW (bash) and that is not supported. That tends to break pretty often and is not supported. Can we run this as a PowerShell script somehow? |
|
I have a fix for the test to make it work on windows. The solution is to just use Diff:--- a/test/tests/run-swift-in-container.sh
+++ b/test/tests/run-swift-in-container.sh
@@ -4,4 +4,15 @@ set -e
testDir="$(readlink -f "$(dirname "$BASH_SOURCE")")"
runDir="$(dirname "$(readlink -f "$BASH_SOURCE")")"
-source "$runDir/run-in-container.sh" "$testDir" "$1" swift ./container.swift
+image="$1"
+
+testName="$(basename "$testDir")"
+newImage="$("$runDir/image-name.sh" "librarytest/$testName" "$image")"
+"$runDir/docker-build.sh" "$testDir" "$newImage" <<EOD
+FROM $image
+COPY dir/container.swift /
+RUN swiftc /container.swift -o container
+CMD [ "/container" ]
+EOD
+
+docker run --rm "$newImage"Dockerfile review: The installers for Python I am also very intrigued by the size disparity between this python install layer and the one in the More space can be saved by cleaning up the |
|
Unfortunately, the python version is pinned and we cannot update that easily :-( (the lldb instance relies on the python matching since python has no stable ABI). I didn't know about the python base image, if that shaves size, I'm happy to swap to it. |
Well, since you need a specific, pinned version python, then the images we maintain won't help since we don't maintain old releases (and |
|
That is slightly ludicrous. Is there an explanation for why? I'm not against it, just completely flabbergasted at the results. It absolutely is worth the effort to do (unfortunately for older releases of Swift, like 5.8.x, this would not be possible since there is a dependency on install ordering). |
|
@yosifkit - that diff looks reasonable to me, do you need anything from our side? |
|
@tianon @yosifkit @grooverdan - anything that I can do to help push this along? |
|
Apologies for the delay. The Dockerfiles need a few updates. The installers for To get the size saving, I'd suggest installing The For cacheability optimization we suggest putting all This last part has me very confused. If I use the same python install arguments you have, but running it via powershell (same as the |
|
@yosifkit do you have examples for Windows that do that with sha? |
|
Thank you! I ended up with something similar (and tweaking the order as well for the space savings). |
|
@LaurentGoderre @yosifkit - seems that we cannot really do the hash checking on the VS installer as the bootstrapper seems to change. |
|
@compnerd isn't a CI creating the final installer? Would it change from that? |
|
@LaurentGoderre no, the VS installer is a dependency that we are installing that is not under our control and only has a "latest" version. |
|
@compnerd is it this step? https://github.com/apple/swift-docker |
|
@LaurentGoderre I think that I might have a version that works for now? Let me upload the version so I can indicate the step. |
Tested on Windows 22H2 (22621.2428). |
shahmishal
force-pushed
the
support-windows-swift
branch
from
548a2c9 to
07d2f55
Compare
This comment has been minimized.
This comment has been minimized.
|
@LaurentGoderre - seems that the bootstrapper has changed between yesterday and today. What do you recommend as the best way to deal with this? SHA256 yesterday: 96e08166034c5110cdb8361ba55f8a2010d0bad714b311a7c8f1c7c8f7c03bce |
|
@LaurentGoderre that is something on your side: #15071 (comment) --- a/test/tests/run-swift-in-container.sh
+++ b/test/tests/run-swift-in-container.sh
@@ -4,4 +4,15 @@ set -e
testDir="$(readlink -f "$(dirname "$BASH_SOURCE")")"
runDir="$(dirname "$(readlink -f "$BASH_SOURCE")")"
-source "$runDir/run-in-container.sh" "$testDir" "$1" swift ./container.swift
+image="$1"
+
+testName="$(basename "$testDir")"
+newImage="$("$runDir/image-name.sh" "librarytest/$testName" "$image")"
+"$runDir/docker-build.sh" "$testDir" "$newImage" <<EOD
+FROM $image
+COPY dir/container.swift /
+RUN swiftc /container.swift -o container
+CMD [ "/container" ]
+EOD
+
+docker run --rm "$newImage"@yosifkit had a patch for that. |
|
@compnerd yep, talked to him about it |
|
@LaurentGoderre what's the best way forward for this? Do we want to just push that change into this PR, have a separate PR? Or will @yosifkit push that? |
|
I can add it as an extra commit to this PR once we're ready. I'm just working on reviewing it once again. |
|
I haven't really realized this before (and I think some of our own images might have this bug), but it looks like just running # https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/start-process
$exitCode = (Start-Process executable-to-run -Wait -NoNewWindow -PassThru \
-ArgumentList @( \
'arg1', \
'arg2', \
'argn' \
) \
).ExitCode; \
if ($exitCode -ne 0) { \
Write-Host ('Message to print when the executable-to-run exited non-zero: {0}' -f $exitCode); \
# not required, but attempting to print some log files from TEMP can be helpful in debugging failing installs.
# this example might miss printing the correct log file if there are multiple with the same LastWrite time
Get-ChildItem $env:TEMP | Sort-Object -Descending -Property LastWriteTime | Select-Object -First 1 | Get-Content; \
# be sure to fail the build so that a partially installed dependency doesn't get committed to a layer
exit $exitCode; \
} \ |
|
Okay, redid some of the image setup; found that Microsoft has a pre-setup python image and I opted to use that and add the additional checking around the other installations. |
This comment has been minimized.
This comment has been minimized.
|
@yosifkit, the changes that you requested are now completed. The python installation ordering is important, and using the Microsoft provided python bundled image results in a smaller image overall (~13.2G vs ~14.8G). |
|
@yosifkit @tianon @LaurentGoderre - sorry to keep bugging you guys about this, but I really would like to get this PR resolved. |
|
From the failing "Naughty" test:
And then from the Official Images Readme:
Unfortunately, the |
|
@yosifkit - is there a way to get the Dockerfile for that image? It would be nice to do something similar - otherwise, we can ignore the error and just run the python installer I suppose. |
|
Depending on how the image was created, you can sometimes extract details about the Dockerfile by inspecting the image or interrogating the image's provenance, if provided. https://docs.docker.com/engine/reference/commandline/inspect/ |
|
I updated the SHA to pick up changes from @compnerd |
This comment has been minimized.
This comment has been minimized.
|
I think that we are at that point! |
|
It seem problematic that the Sha changes all the time. It would mean that if the Windows base image gets a security patch, a rebuild of this will fail because the Sha will have changed |
|
@LaurentGoderre - for now, it just records that the SHA for the VS build tools bootstrapper is mismatched, so it should rebuild. I don't see any way to control that as there is no pinned URL that Microsoft has. |
|
Just a couple review points:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
swift test-reorder 47f92e93d96f 2 minutes ago 12.9GB
swift test-new d3215fc02273 17 hours ago 13.8GB
$ docker history swift:test-reorder
IMAGE CREATED CREATED BY SIZE COMMENT
47f92e93d96f About an hour ago powershell -Command $ErrorActionPreference =… 41kB
f14b5afaf41b About an hour ago |8 GIT=https://github.com/git-for-windows/gi… 3.88GB
e3c9d85e859b About an hour ago powershell -Command $ErrorActionPreference =… 41kB
428846da6b5b About an hour ago powershell -Command $ErrorActionPreference =… 41kB
5be92c0cb314 2 hours ago |6 GIT=https://github.com/git-for-windows/gi… 4.49GB
4ff06a63cd2c 2 hours ago powershell -Command $ErrorActionPreference =… 41kB
25fc6976ebd4 2 hours ago powershell -Command $ErrorActionPreference =… 41kB
66f2d9d9e559 2 hours ago |4 GIT=https://github.com/git-for-windows/gi… 91.6MB
a1f55ed4f134 2 hours ago powershell -Command $ErrorActionPreference =… 41kB
3e1a580e48e0 2 hours ago powershell -Command $ErrorActionPreference =… 41kB
682acaf68a0c 21 hours ago |2 GIT=https://github.com/git-for-windows/gi… 379MB
5ae0ef8cdb2a 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
1c6169758db4 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
81addb6469a1 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
565570542ca6 8 days ago powershell -Command $ErrorActionPreference =… 41kB
72fdef526f1c 2 months ago cmd /S /C #(nop) LABEL description=Docker C… 41kB
e5bd6c080618 2 months ago cmd /S /C #(nop) LABEL maintainer=Swift Inf… 41kB
4742812abba1 3 months ago Install update 10.0.20348.1906 961MB
<missing> 4 months ago Apply image 10.0.20348.1787 3.06GB
$ docker history swift:test-new
IMAGE CREATED CREATED BY SIZE COMMENT
d3215fc02273 19 hours ago powershell -Command $ErrorActionPreference =… 41kB
fcdb5e22f47d 19 hours ago |8 GIT=https://github.com/git-for-windows/gi… 1.04GB
8c30f588be56 19 hours ago powershell -Command $ErrorActionPreference =… 41kB
29c7881674e7 19 hours ago powershell -Command $ErrorActionPreference =… 41kB
e69bc25b3a53 19 hours ago |6 GIT=https://github.com/git-for-windows/gi… 3.87GB
eca397835e43 20 hours ago powershell -Command $ErrorActionPreference =… 41kB
3d137b648862 20 hours ago powershell -Command $ErrorActionPreference =… 41kB
fecd9c6100fa 20 hours ago |4 GIT=https://github.com/git-for-windows/gi… 4.48GB
1209e34793e8 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
6249d947eaa2 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
682acaf68a0c 21 hours ago |2 GIT=https://github.com/git-for-windows/gi… 379MB
5ae0ef8cdb2a 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
1c6169758db4 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
81addb6469a1 21 hours ago powershell -Command $ErrorActionPreference =… 41kB
565570542ca6 8 days ago powershell -Command $ErrorActionPreference =… 41kB
72fdef526f1c 2 months ago cmd /S /C #(nop) LABEL description=Docker C… 41kB
e5bd6c080618 2 months ago cmd /S /C #(nop) LABEL maintainer=Swift Inf… 41kB
4742812abba1 3 months ago Install update 10.0.20348.1906 961MB
<missing> 4 months ago Apply image 10.0.20348.1787 3.06GB |
|
Swapped the order but I don't see a difference locally (its 13.1G either way). |
That's super strange. I re-pulled the windows base image to get it more up-to-date and tried again and still they are very different. I'm building on a Windows 11 host using Powershell and Docker Desktop. docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
swift test-reorder b5bfe34d3d86 54 minutes ago 13GB
swift test-new bb65ec56ae86 2 hours ago 13.9GB |
|
@yosifkit interesting. I think it might have something to do with a pretty old installation of Windows in my case. Can you help get the test changes applied so that we can get this merged? |
update swift windows commit: swiftlang/swift-docker@3fc6093
yosifkit
force-pushed
the
support-windows-swift
branch
from
053118d to
83d6a98
Compare
Diff for 83d6a98:diff --git a/_bashbrew-arches b/_bashbrew-arches
index e85a97f..09c6ecf 100644
--- a/_bashbrew-arches
+++ b/_bashbrew-arches
@@ -1,2 +1,3 @@
amd64
arm64v8
+windows-amd64
diff --git a/_bashbrew-cat b/_bashbrew-cat
index 892e6b8..af2f893 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -258,3 +258,9 @@ Tags: 5.9.1-slim, 5.9-slim, 5.9.1-jammy-slim, 5.9-jammy-slim, jammy-slim, slim
Architectures: amd64, arm64v8
GitCommit: df8e8e36df704e52b2fabf9e732a0f08fbb333e4
Directory: 5.9/ubuntu/22.04/slim
+
+Tags: 5.9.1-windowsservercore-ltsc2022, 5.9-windowsservercore-ltsc2022, windowsservercore-ltsc2022
+Architectures: windows-amd64
+GitCommit: 3fc609343f771e31c17140ed90d6cb2023e6e701
+Directory: 5.9/windows/LTSC2022
+Constraints: windowsservercore-ltsc2022
diff --git a/_bashbrew-list b/_bashbrew-list
index 15a24db..1b4c831 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -114,6 +114,7 @@ swift:5.9-jammy-slim
swift:5.9-rhel-ubi9
swift:5.9-rhel-ubi9-slim
swift:5.9-slim
+swift:5.9-windowsservercore-ltsc2022
swift:5.9.1
swift:5.9.1-amazonlinux2
swift:5.9.1-amazonlinux2-slim
@@ -126,6 +127,7 @@ swift:5.9.1-jammy-slim
swift:5.9.1-rhel-ubi9
swift:5.9.1-rhel-ubi9-slim
swift:5.9.1-slim
+swift:5.9.1-windowsservercore-ltsc2022
swift:amazonlinux2
swift:amazonlinux2-slim
swift:centos7
@@ -138,3 +140,4 @@ swift:latest
swift:rhel-ubi9
swift:rhel-ubi9-slim
swift:slim
+swift:windowsservercore-ltsc2022
diff --git a/_bashbrew-list-build-order b/_bashbrew-list-build-order
index 8c23f03..ab03fe4 100644
--- a/_bashbrew-list-build-order
+++ b/_bashbrew-list-build-order
@@ -56,3 +56,4 @@ swift:focal-slim
swift:rhel-ubi9
swift:rhel-ubi9-slim
swift:slim
+swift:windowsservercore-ltsc2022
diff --git a/swift_windowsservercore-ltsc2022/Dockerfile b/swift_windowsservercore-ltsc2022/Dockerfile
new file mode 100644
index 0000000..be0b6ff
--- /dev/null
+++ b/swift_windowsservercore-ltsc2022/Dockerfile
@@ -0,0 +1,156 @@
+FROM mcr.microsoft.com/windows/servercore:ltsc2022 AS windows
+
+LABEL maintainer="Swift Infrastructure <swift-infrastructure@forums.swift.org>"
+LABEL description="Docker Container for the Swift programming language"
+
+SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
+
+ENV PYTHONIOENCODING UTF-8
+
+# Install Git.
+# See: git-[version]-[bit].exe /SAVEINF=git.inf and /?
+ARG GIT=https://github.com/git-for-windows/git/releases/download/v2.42.0.windows.2/Git-2.42.0.2-64-bit.exe
+ARG GIT_SHA256=BD9B41641A258FD16D99BEECEC66132160331D685DFB4C714CEA2BCC78D63BDB
+RUN Write-Host -NoNewLine ('Downloading {0} ... ' -f ${env:GIT}); \
+ Invoke-WebRequest -Uri ${env:GIT} -OutFile git.exe; \
+ Write-Host '✓'; \
+ Write-Host -NoNewLine ('Verifying SHA256 ({0}) ... ' -f ${env:GIT_SHA256}); \
+ $Hash = Get-FileHash git.exe -Algorithm sha256; \
+ if ($Hash.Hash -eq ${env:GIT_SHA256}) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Hash.Hash); \
+ exit 1; \
+ } \
+ Write-Host -NoNewLine 'Installing git ... '; \
+ $Process = \
+ Start-Process git.exe -Wait -PassThru -NoNewWindow -ArgumentList @( \
+ '/SP-', \
+ '/VERYSILENT', \
+ '/SUPPRESSMSGBOXES', \
+ '/NOCANCEL', \
+ '/NORESTART', \
+ '/CLOSEAPPLICATIONS', \
+ '/FORCECLOSEAPPLICATIONS', \
+ '/NOICONS', \
+ '/COMPONENTS="gitlfs"', \
+ '/EditorOption=VIM', \
+ '/PathOption=Cmd', \
+ '/SSHOption=OpenSSH', \
+ '/CURLOption=WinSSL', \
+ '/UseCredentialManager=Enabled', \
+ '/EnableSymlinks=Enabled', \
+ '/EnableFSMonitor=Enabled' \
+ ); \
+ if ($Process.ExitCode -eq 0) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Process.ExitCode); \
+ exit 1; \
+ } \
+ Remove-Item -Force git.exe; \
+ Remove-Item -ErrorAction SilentlyContinue -Force -Recurse ${env:TEMP}\*
+
+# Install Python
+ARG PY39=https://www.python.org/ftp/python/3.9.13/python-3.9.13-amd64.exe
+ARG PY39_SHA256=FB3D0466F3754752CA7FD839A09FFE53375FF2C981279FD4BC23A005458F7F5D
+RUN Write-Host -NoNewLine ('Downloading {0} ... ' -f ${env:PY39}); \
+ Invoke-WebRequest -Uri ${env:PY39} -OutFile python-3.9.13-amd64.exe; \
+ Write-Host '✓'; \
+ Write-Host -NoNewLine ('Verifying SHA256 ({0}) ... ' -f ${env:PY39_SHA256});\
+ $Hash = Get-FileHash python-3.9.13-amd64.exe -Algorithm sha256; \
+ if ($Hash.Hash -eq ${env:PY39_SHA256}) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Hash.Hash); \
+ exit 1; \
+ } \
+ Write-Host -NoNewLine 'Installing Python ... '; \
+ $Process = \
+ Start-Process python-3.9.13-amd64.exe -Wait -PassThru -NoNewWindow -ArgumentList @( \
+ 'AssociateFiles=0', \
+ 'Include_doc=0', \
+ 'Include_debug=0', \
+ 'Include_lib=1', \
+ 'Include_tcltk=0', \
+ 'Include_test=0', \
+ 'InstallAllUsers=1', \
+ 'InstallLauncherAllUsers=0', \
+ 'PrependPath=1', \
+ '/quiet' \
+ ); \
+ if ($Process.ExitCode -eq 0) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Process.ExitCode); \
+ exit 1; \
+ } \
+ Remove-Item -Force python-3.9.13-amd64.exe; \
+ Remove-Item -ErrorAction SilentlyContinue -Force -Recurse ${env:TEMP}\*
+
+# Install Visual Studio Build Tools
+ARG VSB=https://aka.ms/vs/17/release/vs_buildtools.exe
+ARG VSB_SHA256=D4E08524CB0E5BD061A24F507928D1CFB91DCE192C5E12ED964B8343FC4CDEDD
+RUN Write-Host -NoNewLine ('Downloading {0} ... ' -f ${env:VSB}); \
+ Invoke-WebRequest -Uri ${env:VSB} -OutFile vs_buildtools.exe; \
+ Write-Host '✓'; \
+ Write-Host -NoNewLine ('Verifying SHA256 ({0}) ... ' -f ${env:VSB_SHA256}); \
+ $Hash = Get-FileHash vs_buildtools.exe -Algorithm sha256; \
+ if ($Hash.Hash -eq ${env:VSB_SHA256}) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Hash.Hash); \
+ } \
+ Write-Host -NoNewLine 'Installing Visual Studio Build Tools ... '; \
+ $Process = \
+ Start-Process vs_buildtools.exe -Wait -PassThru -NoNewWindow -ArgumentList @( \
+ '--quiet', \
+ '--wait', \
+ '--norestart', \
+ '--nocache', \
+ '--add', 'Microsoft.VisualStudio.Component.Windows11SDK.22000', \
+ '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64' \
+ ); \
+ if ($Process.ExitCode -eq 0 -or $Process.ExitCode -eq 3010) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Process.ExitCode); \
+ exit 1; \
+ } \
+ Remove-Item -Force vs_buildtools.exe; \
+ Remove-Item -ErrorAction SilentlyContinue -Force -Recurse ${env:TEMP}\*
+
+# Install Swift toolchain.
+ARG SWIFT=https://download.swift.org/swift-5.9.1-release/windows10/swift-5.9.1-RELEASE/swift-5.9.1-RELEASE-windows10.exe
+ARG SWIFT_SHA256=B219D472EEB87612E9977518D2253ED88FBA9CE2CD42CB8FEF3011583B7D1369
+RUN Write-Host -NoNewLine ('Downloading {0} ... ' -f ${env:SWIFT}); \
+ Invoke-WebRequest -Uri ${env:SWIFT} -OutFile installer.exe; \
+ Write-Host '✓'; \
+ Write-Host -NoNewLine ('Verifying SHA256 ({0}) ... ' -f ${env:SWIFT_SHA256}); \
+ $Hash = Get-FileHash installer.exe -Algorithm sha256; \
+ if ($Hash.Hash -eq ${env:SWIFT_SHA256}) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Hash.Hash); \
+ exit 1; \
+ } \
+ Write-Host -NoNewLine 'Installing Swift ... '; \
+ $Process = \
+ Start-Process installer.exe -Wait -PassThru -NoNewWindow -ArgumentList @( \
+ '/quiet', \
+ '/norestart' \
+ ); \
+ if ($Process.ExitCode -eq 0) { \
+ Write-Host '✓'; \
+ } else { \
+ Write-Host ('✘ ({0})' -f $Process.ExitCode); \
+ exit 1; \
+ } \
+ Remove-Item -Force installer.exe; \
+ Remove-Item -ErrorAction SilentlyContinue -Force -Recurse ${env:TEMP}\*
+
+# Default to powershell
+# FIXME: we need to grant ContainerUser the SeCreateSymbolicLinkPrivilege
+# privilege so that it can create symbolic links.
+# USER ContainerUser
+CMD ["powershell.exe", "-nologo", "-ExecutionPolicy", "Bypass"]Relevant Maintainers:
|
|
Squash & rebased, updated swift test, added updated swift-docker 5.9 Windows commit to library file: swiftlang/swift-docker@3fc6093 |
|
🎉 |
No description provided.