Revision 0d4c974adaf92e4646207eb4357efeb2265076bd authored by Keno Fischer on 01 April 2021, 17:17:14 UTC, committed by Keno Fischer on 14 April 2021, 01:31:50 UTC
This commit provides the ability to rebuild system images much faster. The key observation is that most of the time in sysimage build is spent in LLVM generating native code (serializing julia's data structures is quite fast). Thus if we can re-use the code already generated for the system image we're currently running, we'll save a fair amount of time. Unfortunately, this is not 100% straightforward since we were assuming that no linking happens in a number of places. This PR hacks around that, but it is not a particularly satisfying long term solution. That said, it should work fine, and I think it's worth doing, so that we can explore the workflow adjustments that would rely on this. With that said, here's how to use this (at the low level, of course PkgCompiler would just handle this) ```shell $ mkdir chained $ time ./usr/bin/julia --sysimage-native-code=chained --sysimage=usr/lib/julia/sys.so --output-o chained/chained.o.a -e 'Base.__init_build();' real 0m9.633s user 0m8.613s sys 0m1.020s $ cp ../usr/lib/julia/sys-o.a . # Get the -o.a from the old sysimage $ ar x sys-o.a # Extract it into text.o and data.o $ rm data.o # rm the serialized sysimg data $ mv text.o text-old.o $ llvm-objcopy --remove-section .data.jl.unique text-old.o # rm the link between the native code and the old sysimg data $ ar x chained.o.a # Extract new sysimage files $ gcc -shared -o chained.so text.o data.o text-old.o # Link everything $ ../julia --sysimage=chained.so ``` As can be seen, regenerating the system image took about 9s (the subsequent commands aren't timed here, but take less than a second total). This compares very favorably with a non-chained sysimg rebuild: ``` time ./usr/bin/julia --sysimage=usr/lib/julia/sys.so --output-o nonchained.o.a -e 'Base.__init_build();' real 2m42.667s user 2m39.211s sys 0m3.452s ``` Of course if you do load additional packages, the extra code does still need to be compiled, so e.g. building a system image for `Plots` goes from 3 mins to 1 mins (building all of plots, plus everything in base that got invalidated). That is still all in LLVM though - it should be relatively straightforward to multithread that after this PR (since linking the sysimg in multiple pieces is allowed). That part is not implemented yet though.
1 parent a84e765
File | Mode | Size |
---|---|---|
.devcontainer | ||
.github | ||
base | ||
cli | ||
contrib | ||
deps | ||
doc | ||
etc | ||
src | ||
stdlib | ||
test | ||
.clang-format | -rw-r--r-- | 3.3 KB |
.gitattributes | -rw-r--r-- | 65 bytes |
.gitignore | -rw-r--r-- | 283 bytes |
.mailmap | -rw-r--r-- | 11.0 KB |
CITATION.bib | -rw-r--r-- | 2.6 KB |
CONTRIBUTING.md | -rw-r--r-- | 19.3 KB |
HISTORY.md | -rw-r--r-- | 313.4 KB |
LICENSE.md | -rw-r--r-- | 5.0 KB |
Make.inc | -rw-r--r-- | 48.6 KB |
Makefile | -rw-r--r-- | 25.5 KB |
NEWS.md | -rw-r--r-- | 5.8 KB |
README.md | -rw-r--r-- | 6.8 KB |
VERSION | -rw-r--r-- | 10 bytes |
sysimage.mk | -rw-r--r-- | 4.0 KB |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...