move linker input file parsing to the frontend #21700
Conversation
|
Nice, I like where this is headed. I will follow-up with a rewrite of handling dylibs (and dylib re-exports) for MachO since it's somewhat similar in spirit to GNU ld script processing in that parsing dylibs may add more dylibs into the pipeline. In MachO case though, I think this will never be an optional step due to the fact that Apple dylibs enforce this. What do you think? |
|
Doesn't |
|
I'm looking into moving library lookup into Compilation creation, which would address both of your comments and allow reverting the One possibility here is to guess the link mode heuristically, and then if the guess ends up being wrong, fail the compilation with an error message explaining the user needs to explicitly specify |
Hmm, I wonder how safe it would be to assume that |
Very common; |
|
This is why linkSystemLibrary provides these options:
This allows the user to have the traditional behavior, which allows for ambiguity in how libraries are linked, or to specify |
|
The "guess the link mode" heuristic idea doesn't work. This logic must go into the CLI - including dylibs if those require resolution with respect to library directories - in order for the cache system to function. The two possibilities are:
For example, if you have then you add a file and run the same CLI again: Currently, this works correctly. If library lookup moves later in the pipeline than CLI, it would have a false positive cache hit, and the changes to the cache system that would be necessary to correct this are undesirable. Note that in master branch, this exact example is broken if |
andrewrk
force-pushed
the
cli-lib-dirs
branch
3 times, most recently
from
a4d65a9
to
a903727
Compare
andrewrk
force-pushed
the
cli-lib-dirs
branch
from
a903727
to
668b73e
Compare
andrewrk
changed the title
andrewrk
force-pushed
the
cli-lib-dirs
branch
3 times, most recently
from
96991c4
to
fcc0f10
Compare
this allows it to be used by the frontend
other than a compile error, specifically
along with the relevant logic, making the libraries within subject to the same search criteria as all the other libraries. this unfortunately means doing file system access on all .so files when targeting ELF to determine if they are linker scripts, however, I have a plan to address this.
The compiler defaults this value to off so that users whose system shared libraries are all ELF files don't have to pay the cost of checking every file to find out if it is a text file instead. When a GNU ld script is encountered, the error message instructs users about the CLI flag that will immediately solve their problem.
* Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
By making it a field of link.Elf, it is now accessible without a data dependency on `files`, fixing a race condition with the codegen thread and linker thread.
and don't look for glibc files on windows
so we don't have to do an entire compilation unit for just this silly symbol
using name as tie-breaker.
it is incredible how many bad ideas glibc is bundled into one project.
unstable sort is always better if you have no ties
This test does not pass in master branch either if you flip the object order around.
If the "is darwin" check is moved below the libc_installation check below, error.LibCInstallationMissingCrtDir is returned from lci.resolveCrtPaths(). This should be revisited because it makes sense to check libc_installation first even on darwin. Anyway for now this more closely matches logic from master branch.
Unfortunately it's not a complete solution, so a follow-up commit will need to do something more drastic like not do the linker task queue at the same time as codegen task queue. From that point, it is possible to do more work at the same time but that should be a separate branch. This one has gotten big enough.
don't wait for AstGen and C source files to complete before starting to build compiler_rt and libfuzzer
these tasks have some shared data dependencies so they cannot be done simultaneously. Future work should untangle these data dependencies so that more can be done in parallel. for now this commit ensures correctness by making linker input parsing and codegen tasks part of the same queue.
The safety lock needs to happen after check()
which, in this branch causes a miscompilation because it would get sent to the linker.
andrewrk
force-pushed
the
cli-lib-dirs
branch
from
fcc0f10
to
4bdc2d3
Compare
andrewrk
added
the
release notes
|
break the cargo zigbuild |
|
@i18nsite please open an issue for that. |
Moves GNU ld script processing to the frontend to join the relevant library lookup logic, making the libraries within subject to the same search criteria as all the other libraries.
This change is necessary so that the compiler has knowledge of all linker inputs at the start of compilation, so that linking and compilation can begin at the same time. Finding out about linker inputs during flush() is too late. This branch fully removes
lib_dirsfrom being passed to the ELF linking code.This unfortunately means doing file system access on all .so files when targeting ELF to determine if they are linker scripts, so I introduced an opt-in CLI flag to enable .so scripts. When a GNU ld script is encountered, the error message instructs users about the CLI flag that will immediately solve their problem, which is passing
-fallow-so-scriptstozig build. This means that users who don't have funky libraries, or at least avoid linking against them, don't have to pay the cost of these file system accesses.This branch additionally moves all object file, archive, and shared object file parsing to happen during the compilation pipeline rather than in flush().