Revision c9e32b8be95671340643c2e77a19af544af73266 authored by Ryan Lovelett on 12 January 2016, 22:07:27 UTC, committed by Ryan Lovelett on 13 January 2016, 01:56:33 UTC
Python 3 on Linux reads the system locale information to determine what
it should use as the default encoding for strings read from files (this
is different from OS X which is always UTF-8 by default [1]).

Since all the Swift gyb templates are UTF-8 encoded there is effectively
no reason to parse them as anything else. This patch forces the gyb
template parser to read the template using UTF-8 encoding. It accounts
for both reading and writing to a file as well as reading from stdin and
writing to stdout.

Two changes of note are that it now includes a __future__ import that
should make Python 2 behave a little closer to Python 3 in terms of
unicode support. Additionally Python 2 can no longer use cStringIO
because it does not support unicode [2].

To test this patch I ran these commands before and after the patch.
Note: that before the patch if the locale was set to something other
than UTF-8, ASCII for instance, the Python 3 runs would fail. See [3]
for example failure message.

Without stdin/stdout:

$ python2 utils/gyb -o Arrays.2.7.swift stdlib/public/core/Arrays.swift.gyb
$ python3 utils/gyb -o Arrays.3.5.swift stdlib/public/core/Arrays.swift.gyb
$ diff -u Arrays.2.7.swift Arrays.3.5.swift

With stdin/stdout:

$ cat stdlib/public/core/Arrays.swift.gyb | python2 utils/gyb > Arrays.2.7.stdin.stdout.swift
$ cat stdlib/public/core/Arrays.swift.gyb | python3 utils/gyb > Arrays.3.5.stdin.stdout.swift
$ diff -u Arrays.2.7.stdin.stdout.swift Arrays.3.5.stdin.stdout.swift

[1] https://docs.python.org/3/howto/unicode.html#unicode-filenames
[2] https://docs.python.org/2/library/stringio.html#cStringIO.StringIO
[3] https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160111/000780.html
1 parent 7026909
History
File Mode Size
_static
_templates
archive
proposals
scripts
tools
ABI.rst -rw-r--r-- 56.1 KB
ARCOptimization.rst -rw-r--r-- 21.2 KB
AccessControl.rst -rw-r--r-- 10.9 KB
AccessControlInStdlib.rst -rw-r--r-- 3.4 KB
Array.png -rw-r--r-- 68.3 KB
ArrayBridge.graffle -rw-r--r-- 42.3 KB
ArrayBridge.png -rw-r--r-- 160.3 KB
ArrayCast.graffle -rw-r--r-- 38.9 KB
ArrayCast.png -rw-r--r-- 125.0 KB
ArrayImplementation.graffle -rw-r--r-- 34.8 KB
ArrayImplementation.png -rw-r--r-- 272.6 KB
Arrays.rst -rw-r--r-- 10.0 KB
CMakeLists.txt -rw-r--r-- 5.1 KB
CallingConvention.rst -rw-r--r-- 56.6 KB
ContiguousArray.png -rw-r--r-- 39.0 KB
DebuggingTheCompiler.rst -rw-r--r-- 10.3 KB
Dependency Analysis.rst -rw-r--r-- 5.0 KB
DocumentationComments.md -rw-r--r-- 2.8 KB
DriverInternals.rst -rw-r--r-- 5.5 KB
DriverParseableOutput.rst -rw-r--r-- 4.9 KB
ErrorHandling.rst -rw-r--r-- 28.7 KB
ErrorHandlingRationale.rst -rw-r--r-- 89.8 KB
Failable Initializers.rst -rw-r--r-- 8.7 KB
GenerateTopLevelLitreCMakeLists.cmake -rw-r--r-- 206 bytes
Generics.rst -rw-r--r-- 35.5 KB
GitWorkflows.rst -rw-r--r-- 9.5 KB
HighLevelSILOptimizations.rst -rw-r--r-- 14.3 KB
Import.rst -rw-r--r-- 4.7 KB
IndexInvalidation.rst -rw-r--r-- 7.9 KB
Lexicon.rst -rw-r--r-- 8.4 KB
LibraryEvolution.rst -rw-r--r-- 45.2 KB
Literals.rst -rw-r--r-- 5.5 KB
LitreTemplate.cmake -rw-r--r-- 499 bytes
LogicalObjects.rst -rw-r--r-- 4.8 KB
Makefile -rw-r--r-- 5.4 KB
Modules.rst -rw-r--r-- 17.3 KB
MutationModel.rst -rw-r--r-- 9.5 KB
ObjectInitialization.rst -rw-r--r-- 20.1 KB
OptimizationTips.rst -rw-r--r-- 20.4 KB
OptimizerDesign.md -rw-r--r-- 10.8 KB
Pattern Matching.rst -rw-r--r-- 32.0 KB
Runtime.md -rw-r--r-- 14.2 KB
SIL.rst -rw-r--r-- 157.1 KB
SequencesAndCollections.rst -rw-r--r-- 13.5 KB
Serialization.rst -rw-r--r-- 7.1 KB
Slice.png -rw-r--r-- 86.4 KB
StdlibAPIGuidelines.rst -rw-r--r-- 7.3 KB
StdlibRationales.rst -rw-r--r-- 10.7 KB
StoredAndComputedVariables.rst -rw-r--r-- 7.7 KB
StringDesign.rst -rw-r--r-- 53.1 KB
Testing.rst -rw-r--r-- 15.1 KB
TextFormatting.rst -rw-r--r-- 15.4 KB
TransparentAttr.rst -rw-r--r-- 3.4 KB
TypeChecker.rst -rw-r--r-- 46.4 KB
conf.py -rw-r--r-- 8.8 KB
contents.rst -rw-r--r-- 372 bytes
doxygen.cfg.in -rw-r--r-- 50.8 KB
doxygen.css -rw-r--r-- 9.0 KB
doxygen.footer -rw-r--r-- 378 bytes
doxygen.header -rw-r--r-- 475 bytes
doxygen.intro -rw-r--r-- 873 bytes
favicon.ico -rw-r--r-- 0 bytes
toc.js -rw-r--r-- 4.2 KB
weak.rst -rw-r--r-- 50.0 KB

back to top