Xcode 7.2 has a bug which results in compilation failure of project having +1500 Swift files. It seems that swiftc is buggy, but you can try to bypass that behaviour by using my wrapper script. Read the whole post if you are interested in what happens under the hood.
Background
I will start with a short story. I’m working on a complex project which has +1500 Swift files and mixes both Objective-C and Swift code. The project is written in Swift 1.2 and still uses Xcode 6.4. Migrating such a big project to Swift 2 is not a trivial task, automatic conversion provided by Xcode 7.2 (swift-update tool) takes several hours and it still can solve only basic issues. The rest has to be solved manually, so it is not only a time consuming task, but also brings higher regression risk. Surprisingly, even after fixing all migration related issues, the app still didn’t compile. It looked like the compiler didn’t generate a Swift umbrella header that was included by legacy Objective-C code to allow using Swift classes. Build procedure failed on copying header from project Intermediates directory, because it was simply not generated:
ditto: can't get real path for source '/Users/MyUsername/Library/Developer/Xcode/DerivedData/MyAppName-fwxrneawhopjkqcpeoykduytrgwv/Build/Intermediates/MyAppName.build/Debug-iphoneos/MyAppName.build/Objects-normal/arm64/MyAppName-Swift.h'
Command /usr/bin/ditto failed with exit code 1
In the beginning this issue sounded familiar and there were already several similar problems on StackOverFlow. Most of them suggested error in some Swift file which was not reported by Xcode and didn’t result in successful compilation and hence umbrella header was missing. After spending several hours on finding the problem with some “corrupted” Swift file, testing various compilation flags, deleting DerivedData and such, I was still not able to move forward.
Quite accidentally I’ve noticed that there was a difference when invoking project compilation with and without whole-module-optimization parameter.
The former resulted in almost instant compilation of Swift files, while the latter a very long compilation of one Swift file after another, resulting in generating object files for each Swift file, but still without generating any Objective-C umbrella header. I decided to check the first issue as I had a gut feeling that this could be a compiler bug.
It is worth to know that when turned on -whole-module-optimization swiftc is not compiling each Swift file after another, but takes all Swift files as an input and generates one merged binary object file. This allows the swiftc to perform better optimization based on all Swift files passed as an input, rather than individual file.
To confirm that this is a swiftc bug I decided to isolate the problem. I wrote a simple Python script that generated N pure Swift classes having one method and tried to compile them using command line version of swiftc. When reaching about 2000 Swift files I was able to reproduce the issue with -whole-module-optimization option. It definitely looked like a bug, as when swiftc got less than X Swift files to compile, it took a couple of minutes to compile them all and generate both umbrella header and object file. When reaching certain limit of files, swiftc was simply returning immediately, unfortunately without any error code.
Finding the solution
Inspired by recent open sourcing I decided to dive into swiftc source code. First I’ve downloaded latest version of swiftc from github repo and compiled it manually to check out if bug was not fixed recently. Unfortunately it didn’t help, the problem was still there.
After some debugging I found out that the culprit was posix_spawn invoked from Task::execute() (lib/Basic/Unix/TaskQueue.inc:142), it returned error code 7, which suggested [EFAULT] aka Path, argv, or envp point to an illegal address. I checked both argv and envp and they contained correct entries, didn’t look like corrupted. I think that reason for posix_spawn() failure could be too long list of arguments stored in argvp (about 5900 entries, which was much bigger than original list of input arguments passed to swiftc – about 2000 args). The situation could be solved by blocking posix_spawn (#undef HAVE_POSIX_SPAWN) and using fork function, but this would require swiftc recompilation and was not acceptable.
All of the above gave me some hint, which was that the problem was with parallel execution of compiler jobs and spawning the processes. I decided to check what happened if forcing swiftc to use one worker thread (-num-threads 1). This didn’t help, but what surprisingly worked and allowed skipping process spawning logic was setting -num-threads 0 in command line argument.
Knowing how to workaround the issue I decided to persuade Xcode to pass that argument to swiftc. It seems that by default Xcode inserts -num-threads equal to number of processor cores in the machine.
Setting:
defaults write com.apple.dt.Xcode IDEBuildOperationMaxNumberOfConcurrentCompileTasks 0
didn’t work, because default value was still passed to swiftc. Invoking xcodebuild with IDEBuildOperationMaxNumberOfConcurrentCompileTasks parameter worked for other values, but was not possible with 0. Adding -num-threads 0 to SWIFT COMPILER FLAGS in Xcode did not work, because Xcode inserted its version of -num-threads just after value from SWIFT COMPILER FLAGS and it got overwritten. I knew that I was very close, but I still could not get it working.
Bypassing the compiler
Finally I came up with a *brilliant* idea of wrapping swiftc with a Python script which would simply act as a proxy and append -num-threads 0 to all what was passed to it as input arguments.
I’ve noticed that Xcode uses swiftc from:
`
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xc/toolchain/usr/bin
`
which was a symbolic link to swift binary file in the same directory. I removed swiftc symlink from there and wrote a simple wrapper that was invoking swift. Unfortunately this didn’t work. I checked swift source code and found out that swift detects how it is executed. If it is executed through a symbolic link its argv[0] is named swiftc and command line version logic is invoked; when invoked directly from swift binary, Swift interpreter is started. The latter is what happened when I invoked swift from the wrapper script. I managed to solve this issue by creating a symbolic link in /usr/local/bin which pointed to swift in:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift
and invoked the compiler through my symbolic link.
My idea worked pretty well, all Swift files were compiled successfully, umbrella header with Swift classes for Objective-C code was generated and everything looked like I was going to see my app running in the simulator. Unfortunately, there was a linker that reported a problem, and this is when I discovered a second bug in Xcode, which can be reproduced easily when compiling the project with -whole-module-optimization.
The problem was that the linker expected object file to be generated for each Swift file. Because the project was compiled with -whole-module-optimization only one big object file with all symbols from Swift classes was generated. I checked how Xcode invokes the linker and I’ve noticed that it passes to it a file suffixed with *.LinkFileList which is stored inside DerivedData/../Intermediates/../Objects-normal/[ARCH]/. The file indeed contained a list of all object files, so it looked like it was wrongly generated.
I came up with another idea. Why not to reuse my Python wrapper and modify *.LinkFileList file? I replaced all missing object files from that file, with a path to a file being output from swiftc invoked with -whole-module-optimization. But how this could be automated? I could not truncate *.LinkFileList file and add merged object file path because it contained also objects from Objective-C compiler. Finally I came up with an idea of using input parameters from swiftc command and retrieve the names of Swift files from that argument list. After some parsing and adapting file names it worked pretty well. I was able to successfully compile Swift 2 version of my project.
I reported this problem to Apple and it seems that it affects more people. You can follow this issue here (SR-280)
Python script acting as a wrapper for swiftc is published on my github account
You may also want to ask why not dividing the project into frameworks and avoid all that problems? Unfortunately this is not possible as I have to still target iOS 7 devices which doesn’t support dynamic Swift frameworks.
If you like this post, please follow me on twitter @tgebarowski