Revision c9e32b8be95671340643c2e77a19af544af73266 authored by Ryan Lovelett on 12 January 2016, 22:07:27 UTC, committed by Ryan Lovelett on 13 January 2016, 01:56:33 UTC
Python 3 on Linux reads the system locale information to determine what
it should use as the default encoding for strings read from files (this
is different from OS X which is always UTF-8 by default [1]).

Since all the Swift gyb templates are UTF-8 encoded there is effectively
no reason to parse them as anything else. This patch forces the gyb
template parser to read the template using UTF-8 encoding. It accounts
for both reading and writing to a file as well as reading from stdin and
writing to stdout.

Two changes of note are that it now includes a __future__ import that
should make Python 2 behave a little closer to Python 3 in terms of
unicode support. Additionally Python 2 can no longer use cStringIO
because it does not support unicode [2].

To test this patch I ran these commands before and after the patch.
Note: that before the patch if the locale was set to something other
than UTF-8, ASCII for instance, the Python 3 runs would fail. See [3]
for example failure message.

Without stdin/stdout:

$ python2 utils/gyb -o Arrays.2.7.swift stdlib/public/core/Arrays.swift.gyb
$ python3 utils/gyb -o Arrays.3.5.swift stdlib/public/core/Arrays.swift.gyb
$ diff -u Arrays.2.7.swift Arrays.3.5.swift

With stdin/stdout:

$ cat stdlib/public/core/Arrays.swift.gyb | python2 utils/gyb > Arrays.2.7.stdin.stdout.swift
$ cat stdlib/public/core/Arrays.swift.gyb | python3 utils/gyb > Arrays.3.5.stdin.stdout.swift
$ diff -u Arrays.2.7.stdin.stdout.swift Arrays.3.5.stdin.stdout.swift

[1] https://docs.python.org/3/howto/unicode.html#unicode-filenames
[2] https://docs.python.org/2/library/stringio.html#cStringIO.StringIO
[3] https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160111/000780.html
1 parent 7026909
History
File Mode Size
Inputs
complete_big_array.swift -rw-r--r-- 340 bytes
complete_cache.swift -rw-r--r-- 602 bytes
complete_constructor.swift -rw-r--r-- 172 bytes
complete_constructor.swift.response -rw-r--r-- 486 bytes
complete_crash1.swift -rw-r--r-- 162 bytes
complete_custom.swift -rw-r--r-- 3.3 KB
complete_filter.swift -rw-r--r-- 2.9 KB
complete_forbid_typecheck.swift -rw-r--r-- 723 bytes
complete_from_clang_module.swift -rw-r--r-- 626 bytes
complete_fuzzy.swift -rw-r--r-- 5.7 KB
complete_group_overloads.swift -rw-r--r-- 2.7 KB
complete_hide_low_priority.swift -rw-r--r-- 1.3 KB
complete_import_module_flag.swift -rw-r--r-- 1.1 KB
complete_inner.swift -rw-r--r-- 4.4 KB
complete_literals.swift -rw-r--r-- 1.5 KB
complete_member.swift -rw-r--r-- 2.8 KB
complete_member.swift.response -rw-r--r-- 1.4 KB
complete_moduleimportdepth.swift -rw-r--r-- 4.5 KB
complete_name.swift -rw-r--r-- 395 bytes
complete_open.swift -rw-r--r-- 2.8 KB
complete_operators.swift -rw-r--r-- 1.1 KB
complete_override.swift -rw-r--r-- 191 bytes
complete_override.swift.response -rw-r--r-- 9.3 KB
complete_playground_symlink.swift -rw-r--r-- 537 bytes
complete_popular_api.swift -rw-r--r-- 3.2 KB
complete_popular_api.swift.popular -rw-r--r-- 56 bytes
complete_popular_api.swift.unpopular -rw-r--r-- 38 bytes
complete_requestlimit.swift -rw-r--r-- 3.9 KB
complete_sort_order.swift -rw-r--r-- 2.2 KB
complete_stems.swift -rw-r--r-- 5.4 KB
complete_structure.swift -rw-r--r-- 3.9 KB
complete_test.swift -rw-r--r-- 1.5 KB
complete_type_match.swift -rw-r--r-- 1.7 KB
complete_typealias_different_file.swift -rw-r--r-- 191 bytes
complete_underscores.swift -rw-r--r-- 3.5 KB
complete_update.swift -rw-r--r-- 1.4 KB
complete_with_closure_param.swift -rw-r--r-- 688 bytes
metatype_in_expr.swift -rw-r--r-- 421 bytes

back to top