Skip to main content

Browse the archive

https://github.com/postgres/postgres
07 April 2024, 13:56:23 UTC
sort by:
Revision Author Date Message Commit Date
0ad348f Tom Lane 10 August 2020, 21:15:53 UTC Stamp 12.4. 10 August 2020, 21:15:53 UTC
f9ddc36 Tom Lane 10 August 2020, 19:35:46 UTC Last-minute updates for release notes. Security: CVE-2020-14349, CVE-2020-14350 10 August 2020, 19:35:46 UTC
515ee4a Noah Misch 10 August 2020, 16:22:54 UTC Document clashes between logical replication and untrusted users. Back-patch to v10, which introduced logical replication. Security: CVE-2020-14349 10 August 2020, 16:22:58 UTC
64a7106 Noah Misch 10 August 2020, 16:22:54 UTC Empty search_path in logical replication apply worker and walsender. This is like CVE-2018-1058 commit 582edc369cdbd348d68441fc50fa26a84afd0c1a. Today, a malicious user of a publisher or subscriber database can invoke arbitrary SQL functions under an identity running replication, often a superuser. This fix may cause "does not exist" or "no schema has been selected to create in" errors in a replication process. After upgrading, consider watching server logs for these errors. Objects accruing schema qualification in the wake of the earlier commit are unlikely to need further correction. Back-patch to v10, which introduced logical replication. Security: CVE-2020-14349 10 August 2020, 16:22:58 UTC
d4d0ec9 Noah Misch 10 August 2020, 16:22:54 UTC Move connect.h from fe_utils to src/include/common. Any libpq client can use the header. Clients include backend components postgres_fdw, dblink, and logical replication apply worker. Back-patch to v10, because another fix needs this. In released branches, just copy the header and keep the original. 10 August 2020, 16:22:58 UTC
3ba9670 Tom Lane 10 August 2020, 14:44:42 UTC Make contrib modules' installation scripts more secure. Hostile objects located within the installation-time search_path could capture references in an extension's installation or upgrade script. If the extension is being installed with superuser privileges, this opens the door to privilege escalation. While such hazards have existed all along, their urgency increases with the v13 "trusted extensions" feature, because that lets a non-superuser control the installation path for a superuser-privileged script. Therefore, make a number of changes to make such situations more secure: * Tweak the construction of the installation-time search_path to ensure that references to objects in pg_catalog can't be subverted; and explicitly add pg_temp to the end of the path to prevent attacks using temporary objects. * Disable check_function_bodies within installation/upgrade scripts, so that any security gaps in SQL-language or PL-language function bodies cannot create a risk of unwanted installation-time code execution. * Adjust lookup of type input/receive functions and join estimator functions to complain if there are multiple candidate functions. This prevents capture of references to functions whose signature is not the first one checked; and it's arguably more user-friendly anyway. * Modify various contrib upgrade scripts to ensure that catalog modification queries are executed with secure search paths. (These are in-place modifications with no extension version changes, since it is the update process itself that is at issue, not the end result.) Extensions that depend on other extensions cannot be made fully secure by these methods alone; therefore, revert the "trusted" marking that commit eb67623c9 applied to earthdistance and hstore_plperl, pending some better solution to that set of issues. Also add documentation around these issues, to help extension authors write secure installation scripts. Patch by me, following an observation by Andres Freund; thanks to Noah Misch for review. Security: CVE-2020-14350 10 August 2020, 14:44:42 UTC
6346761 Peter Eisentraut 10 August 2020, 13:21:18 UTC Translation updates Source-Git-URL: https://git.postgresql.org/git/pgtranslation/messages.git Source-Git-Hash: 444a6779aafc552ac452715caa65cfca0e723073 10 August 2020, 13:21:18 UTC
418414d Tom Lane 09 August 2020, 16:39:08 UTC Check for fseeko() failure in pg_dump's _tarAddFile(). Coverity pointed out, not unreasonably, that we checked fseeko's result at every other call site but these. Failure to seek in the temp file (note this is NOT pg_dump's output file) seems quite unlikely, and even if it did happen the file length cross-check further down would probably detect the problem. Still, that's a poor excuse for not checking the result of a system call. 09 August 2020, 16:39:08 UTC
65a6769 Tom Lane 09 August 2020, 00:01:41 UTC Release notes for 12.4, 11.9, 10.14, 9.6.19, 9.5.23. 09 August 2020, 00:01:41 UTC
85cb4ec Alvaro Herrera 08 August 2020, 16:31:55 UTC walsnd: Don't set waiting_for_ping_response spuriously Ashutosh Bapat noticed that when logical walsender needs to wait for WAL, and it realizes that it must send a keepalive message to walreceiver to update the sent-LSN, which *does not* request a reply from walreceiver, it wrongly sets the flag that it's going to wait for that reply. That means that any future would-be sender of feedback messages ends up not sending a feedback message, because they all believe that a reply is expected. With built-in logical replication there's not much harm in this, because WalReceiverMain will send a ping-back every wal_receiver_timeout/2 anyway; but with other logical replication systems (e.g. pglogical) it can cause significant pain. This problem was introduced in commit 41d5f8ad734, where the request-reply flag was changed from true to false to WalSndKeepalive, without at the same time removing the line that sets waiting_for_ping_response. Just removing that line would be a sufficient fix, but it seems better to shift the responsibility of setting the flag to WalSndKeepalive itself instead of requiring caller to do it; this is clearly less error-prone. Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Reported-by: Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com> Backpatch: 9.5 and up Discussion: https://postgr.es/m/20200806225558.GA22401@alvherre.pgsql 08 August 2020, 16:31:55 UTC
4f26932 Etsuro Fujita 07 August 2020, 05:45:02 UTC Fix yet another issue with step generation in partition pruning. Commit 13838740f fixed some issues with step generation in partition pruning, but there was yet another one: get_steps_using_prefix() assumes that clauses in the passed-in prefix list are sorted in ascending order of their partition key numbers, but the caller failed to ensure this for range partitioning, which led to an assertion failure in debug builds. Adjust the caller function to arrange the clauses in the prefix list in the required order for range partitioning. Back-patch to v11, like the previous commit. Patch by me, reviewed by Amit Langote. Discussion: https://postgr.es/m/CAPmGK16jkXiFG0YqMbU66wte-oJTfW6D1HaNvQf%3D%2B5o9%3Dm55wQ%40mail.gmail.com 07 August 2020, 05:45:02 UTC
a2e0cf4 Tom Lane 06 August 2020, 19:49:45 UTC First-draft release notes for 12.4. As usual, the release notes for other branches will be made by cutting these down, but put them up for community review first. 06 August 2020, 19:49:45 UTC
7c78040 Robert Haas 06 August 2020, 18:55:00 UTC Fix typo. Per report from Tom Lane. Previously fixed in master by commit f057980149ddccd4b862d2c6b3920ed498b0d7ec. 06 August 2020, 18:55:00 UTC
bcbc272 Robert Haas 06 August 2020, 17:58:23 UTC Fix minor problems with non-exclusive backup cleanup. The previous coding imagined that it could call before_shmem_exit() when a non-exclusive backup began and then remove the previously-added handler by calling cancel_before_shmem_exit() when that backup ended. However, this only works provided that nothing else in the system has registered a before_shmem_exit() hook in the interim, because cancel_before_shmem_exit() is documented to remove a callback only if it is the latest callback registered. It also only works if nothing can ERROR out between the time that sessionBackupState is reset and the time that cancel_before_shmem_exit(), which doesn't seem to be strictly true. To fix, leave the handler installed for the lifetime of the session, arrange to install it just once, and teach it to quietly do nothing if there isn't a non-exclusive backup in process. This was originally committed to master as 303640199d0436c5e7acdf50b837a027b5726594, but I did not back-patch at the time because the consequences were minor. However, now there's been a second report of this causing trouble with a slightly different test case than the one I reported originally, so now I'm back-patching as far as v11 where JIT was introduced. Patch by me, reviewed by Kyotaro Horiguchi, Michael Paquier (who preferred a different approach, but got outvoted), Fujii Masao, and Tom Lane, and with comments by various others. New problem report from Bharath Rupireddy. Discussion: http://postgr.es/m/CA+TgmobMjnyBfNhGTKQEDbqXYE3_rXWpc4CM63fhyerNCes3mA@mail.gmail.com Discussion: http://postgr.es/m/CALj2ACWk7j4F2v2fxxYfrroOF=AdFNPr1WsV+AGtHAFQOqm_pw@mail.gmail.com 06 August 2020, 17:58:23 UTC
f87f77e Bruce Momjian 05 August 2020, 21:12:10 UTC doc: clarify "state" table reference in tutorial Reported-by: Vyacheslav Shablistyy Discussion: https://postgr.es/m/159586122762.680.1361378513036616007@wrigleys.postgresql.org Backpatch-through: 9.5 05 August 2020, 21:12:10 UTC
f992da2 Tom Lane 05 August 2020, 19:38:55 UTC Fix matching of sub-partitions when a partitioned plan is stale. Since we no longer require AccessExclusiveLock to add a partition, the executor may see that a partitioned table has more partitions than the planner saw. ExecCreatePartitionPruneState's code for matching up the partition lists in such cases was faulty, and would misbehave if the planner had successfully pruned any partitions from the query. (Thus, trouble would occur only if a partition addition happens concurrently with a query that uses both static and dynamic partition pruning.) This led to an Assert failure in debug builds, and probably to crashes or query misbehavior in production builds. To repair the bug, just explicitly skip zeroes in the plan's relid_map[] list. I also made some cosmetic changes to make the code more readable (IMO anyway). Also, convert the cross-checking Assert to a regular test-and-elog, since it's now apparent that this logic is more fragile than one would like. Currently, there's no way to repeatably exercise this code, except with manual use of a debugger to stop the backend between planning and execution. Hence, no test case in this patch. We oughta do something about that testability gap, but that's for another day. Amit Langote and Tom Lane, per report from Justin Pryzby. Oversight in commit 898e5e329; backpatch to v12 where that appeared. Discussion: https://postgr.es/m/20200802181131.GA27754@telsasoft.com 05 August 2020, 19:38:55 UTC
55ffd61 Tom Lane 04 August 2020, 19:20:31 UTC Increase hard-wired timeout values in ecpg regression tests. A couple of test cases had connect_timeout=14, a value that seems to have been plucked from a hat. While it's more than sufficient for normal cases, slow/overloaded buildfarm machines can get a timeout failure here, as per recent report from "sungazer". Increase to 180 seconds, which is in line with our typical timeouts elsewhere in the regression tests. Back-patch to 9.6; the code looks different in 9.5, and this doesn't seem to be quite worth the effort to adapt to that. Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2020-08-04%2007%3A12%3A22 04 August 2020, 19:20:31 UTC
8d5c632 Tom Lane 03 August 2020, 17:11:16 UTC Doc: fix obsolete info about allowed range of TZ offsets in timetz. We've allowed UTC offsets up to +/- 15:59 since commit cd0ff9c0f, but that commit forgot to fix the documentation about timetz. Per bug #16571 from osdba. Discussion: https://postgr.es/m/16571-eb7501598de78c8a@postgresql.org 03 August 2020, 17:11:16 UTC
76b2b3e Thomas Munro 03 August 2020, 00:39:15 UTC Fix rare failure in LDAP tests. Instead of writing a query to psql's stdin, use -c. This avoids a failure where psql exits before we write, seen a few times on the build farm. Thanks to Tom Lane for the suggestion. Back-patch to 11, where the LDAP tests arrived. Reviewed-by: Noah Misch <noah@leadboat.com> Discussion: https://postgr.es/m/CA%2BhUKGLFmW%2BHQYPeKiwSp5sdFFHtFViCpw4Mh6yAgEx74r5-Cw%40mail.gmail.com 03 August 2020, 00:44:27 UTC
16c9779 Peter Geoghegan 31 July 2020, 22:34:25 UTC Restore lost amcheck TOAST test coverage. Commit eba77534 fixed an amcheck false positive bug involving inconsistencies in TOAST input state between table and index. A test case was added that verified that such an inconsistency didn't result in a spurious corruption related error. Test coverage from the test was accidentally lost by commit 501e41dd, which propagated ALTER TABLE ... SET STORAGE attstorage state to indexes. This broke the test because the test specifically relied on attstorage not being propagated. This artificially forced there to be index tuples whose datums were equivalent to the datums in the heap without the datums actually being bitwise equal. Fix this by updating pg_attribute directly instead. Commit 501e41dd made similar changes to a test_decoding TOAST-related test case which made the same assumption, but overlooked the amcheck test case. Backpatch: 11-, just like commit eba77534 (and commit 501e41dd). 31 July 2020, 22:34:25 UTC
70248d8 Tom Lane 31 July 2020, 15:43:12 UTC Fix recently-introduced performance problem in ts_headline(). The new hlCover() algorithm that I introduced in commit c9b0c678d turns out to potentially take O(N^2) or worse time on long documents, if there are many occurrences of individual query words but few or no substrings that actually satisfy the query. (One way to hit this behavior is with a "common_word & rare_word" type of query.) This seems unavoidable given the original goal of checking every substring of the document, so we have to back off that idea. Fortunately, it seems unlikely that anyone would really want headlines spanning all of a long document, so we can avoid the worse-than-linear behavior by imposing a maximum length of substring that we'll consider. For now, just hard-wire that maximum length as a multiple of max_words times max_fragments. Perhaps at some point somebody will argue for exposing it as a ts_headline parameter, but I'm hesitant to make such a feature addition in a back-patched bug fix. I also noted that the hlFirstIndex() function I'd added in that commit was unnecessarily stupid: it really only needs to check whether a HeadlineWordEntry's item pointer is null or not. This wouldn't make all that much difference in typical cases with queries having just a few terms, but a cycle shaved is a cycle earned. In addition, add a CHECK_FOR_INTERRUPTS call in TS_execute_recurse. This ensures that hlCover's loop is cancellable if it manages to take a long time, and it may protect some other TS_execute callers as well. Back-patch to 9.6 as the previous commit was. I also chose to add the CHECK_FOR_INTERRUPTS call to 9.5. The old hlCover() algorithm seems to avoid the O(N^2) behavior, at least on the test case I tried, but nonetheless it's not very quick on a long document. Per report from Stephen Frost. Discussion: https://postgr.es/m/20200724160535.GW12375@tamriel.snowman.net 31 July 2020, 15:43:12 UTC
a63fbd3 Tatsuo Ishii 30 July 2020, 22:48:23 UTC Doc: fix high availability solutions comparison. In "High Availability, Load Balancing, and Replication" chapter, certain descriptions of Pgpool-II were not correct at this point. It does not need conflict resolution. Also "Multiple-Server Parallel Query Execution" is not supported anymore. Discussion: https://postgr.es/m/20200726.230128.53842489850344110.t-ishii%40sraoss.co.jp Author: Tatsuo Ishii Reviewed-by: Bruce Momjian Backpatch-through: 9.5 30 July 2020, 22:48:23 UTC
7de22d2 Michael Paquier 30 July 2020, 06:48:56 UTC doc: Mention index references in pg_inherits Partitioned indexes are also registered in pg_inherits, but the description of this catalog did not reflect that. Author: Dagfinn Ilmari Mannsåker Discussion: https://postgr.es/m/87k0ynj35y.fsf@wibble.ilmari.org Backpatch-through: 11 30 July 2020, 06:48:56 UTC
6ed3464 David Rowley 28 July 2020, 10:52:51 UTC Doc: Improve documentation for pg_jit_available() Per complaint from Scott Ribe. Based on wording suggestion from Tom Lane. Discussion: https://postgr.es/m/1956E806-1468-4417-9A9D-235AE1D5FE1A@elevated-dev.com Backpatch-through: 11, where pg_jit_available() was added 28 July 2020, 10:52:51 UTC
62c4a77 Etsuro Fujita 28 July 2020, 02:00:02 UTC Fix some issues with step generation in partition pruning. In the case of range partitioning, get_steps_using_prefix() assumes that the passed-in prefix list contains at least one clause for each of the partition keys earlier than one specified in the passed-in step_lastkeyno, but the caller (ie, gen_prune_steps_from_opexps()) didn't take it into account, which led to a server crash or incorrect results when the list contained no clauses for such partition keys, as reported in bug #16500 and #16501 from Kobayashi Hisanori. Update the caller to call that function only when the list created there contains at least one clause for each of the earlier partition keys in the case of range partitioning. While at it, fix some other issues: * The list to pass to get_steps_using_prefix() is allowed to contain multiple clauses for the same partition key, as described in the comment for that function, but that function actually assumed that the list contained just a single clause for each of middle partition keys, which led to an assertion failure when the list contained multiple clauses for such partition keys. Update that function to match the comment. * In the case of hash partitioning, partition keys are allowed to be NULL, in which case the list to pass to get_steps_using_prefix() contains no clauses for NULL partition keys, but that function treats that case as like the case of range partitioning, which led to the assertion failure. Update the assertion test to take into account NULL partition keys in the case of hash partitioning. * Fix a typo in a comment in get_steps_using_prefix_recurse(). * gen_partprune_steps() failed to detect self-contradiction from strict-qual clauses and an IS NULL clause for the same partition key in some cases, producing incorrect partition-pruning steps, which led to incorrect results of partition pruning, but didn't cause any user-visible problems fortunately, as the self-contradiction is detected later in the query planning. Update that function to detect the self-contradiction. Per bug #16500 and #16501 from Kobayashi Hisanori. Patch by me, initial diagnosis for the reported issue and review by Dmitry Dolgov. Back-patch to v11, where partition pruning was introduced. Discussion: https://postgr.es/m/16500-d1613f2a78e1e090%40postgresql.org Discussion: https://postgr.es/m/16501-5234a9a0394f6754%40postgresql.org 28 July 2020, 02:00:02 UTC
5bd087e Michael Paquier 27 July 2020, 06:58:59 UTC Fix corner case with 16kB-long decompression in pgcrypto, take 2 A compressed stream may end with an empty packet. In this case decompression finishes before reading the empty packet and the remaining stream packet causes a failure in reading the following data. This commit makes sure to consume such extra data, avoiding a failure when decompression the data. This corner case was reproducible easily with a data length of 16kB, and existed since e94dd6a. A cheap regression test is added to cover this case based on a random, incompressible string. The first attempt of this patch has allowed to find an older failure within the compression logic of pgcrypto, fixed by b9b6105. This involved SLES 15 with z390 where a custom flavor of libz gets used. Bonus thanks to Mark Wong for providing access to the specific environment. Reported-by: Frank Gagnepain Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: Tom Lane Discussion: https://postgr.es/m/16476-692ef7b84e5fb893@postgresql.org Backpatch-through: 9.5 27 July 2020, 06:58:59 UTC
61a4a3a Michael Paquier 27 July 2020, 01:29:11 UTC Fix handling of structure for bytea data type in ECPG Some code paths dedicated to bytea used the structure for varchar. This did not lead to any actual bugs, as bytea and varchar have the same definition, but it could become a trap if one of these definitions changes for a new feature or a bug fix. Issue introduced by 050710b. Author: Shenhao Wang Reviewed-by: Vignesh C, Michael Paquier Discussion: https://postgr.es/m/07ac7dee1efc44f99d7f53a074420177@G08CNEXMBPEKD06.g08.fujitsu.local Backpatch-through: 12 27 July 2020, 01:29:11 UTC
bdaa84e Amit Kapila 25 July 2020, 05:08:46 UTC Fix buffer usage stats for nodes above Gather Merge. Commit 85c9d347 addressed a similar problem for Gather and Gather Merge nodes but forgot to account for nodes above parallel nodes. This still works for nodes above Gather node because we shut down the workers for Gather node as soon as there are no more tuples. We can do a similar thing for Gather Merge as well but it seems better to account for stats during nodes shutdown after completing the execution. Reported-by: Stéphane Lorek, Jehan-Guillaume de Rorthais Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com> Reviewed-by: Amit Kapila Backpatch-through: 10, where it was introduced Discussion: https://postgr.es/m/20200718160206.584532a2@firost 25 July 2020, 05:08:46 UTC
3d4a778 Tom Lane 23 July 2020, 21:19:37 UTC Fix ancient violation of zlib's API spec. contrib/pgcrypto mishandled the case where deflate() does not consume all of the offered input on the first try. It reset the next_in pointer to the start of the input instead of leaving it alone, causing the wrong data to be fed to the next deflate() call. This has been broken since pgcrypto was committed. The reason for the lack of complaints seems to be that it's fairly hard to get stock zlib to not consume all the input, so long as the output buffer is big enough (which it normally would be in pgcrypto's usage; AFAICT the input is always going to be packetized into packets no larger than ZIP_OUT_BUF). However, IBM's zlibNX implementation for AIX evidently will do it in some cases. I did not add a test case for this, because I couldn't find one that would fail with stock zlib. When we put back the test case for bug #16476, that will cover the zlibNX situation well enough. While here, write deflate()'s second argument as Z_NO_FLUSH per its API spec, instead of hard-wiring the value zero. Per buildfarm results and subsequent investigation. Discussion: https://postgr.es/m/16476-692ef7b84e5fb893@postgresql.org 23 July 2020, 21:20:02 UTC
63b2297 Peter Eisentraut 23 July 2020, 15:13:00 UTC doc: Document that ssl_ciphers does not affect TLS 1.3 TLS 1.3 uses a different way of specifying ciphers and a different OpenSSL API. PostgreSQL currently does not support setting those ciphers. For now, just document this. In the future, support for this might be added somehow. Reviewed-by: Jonathan S. Katz <jkatz@postgresql.org> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> 23 July 2020, 18:38:31 UTC
8bf4e69 Thomas Munro 23 July 2020, 09:10:49 UTC Fix error message. Remove extra space. Back-patch to all releases, like commit 7897e3bb. Author: Lu, Chenyang <lucy.fnst@cn.fujitsu.com> Discussion: https://postgr.es/m/795d03c6129844d3803e7eea48f5af0d%40G08CNEXMBPEKD04.g08.fujitsu.local 23 July 2020, 09:17:47 UTC
e30a63f Michael Paquier 22 July 2020, 23:29:18 UTC Revert "Fix corner case with PGP decompression in pgcrypto" This reverts commit 9e10898, after finding out that buildfarm members running SLES 15 on z390 complain on the compression and decompression logic of the new test: pipistrelles, barbthroat and steamerduck. Those hosts are visibly using hardware-specific changes to improve zlib performance, requiring more investigation. Thanks to Tom Lane for the discussion. Discussion: https://postgr.es/m/20200722093749.GA2564@paquier.xyz Backpatch-through: 9.5 22 July 2020, 23:29:18 UTC
bba2e66 Michael Paquier 22 July 2020, 05:52:46 UTC Fix corner case with PGP decompression in pgcrypto A compressed stream may end with an empty packet, and PGP decompression finished before reading this empty packet in the remaining stream. This caused a failure in pgcrypto, handling this case as corrupted data. This commit makes sure to consume such extra data, avoiding a failure when decompression the entire stream. This corner case was reproducible with a data length of 16kB, and existed since its introduction in e94dd6a. A cheap regression test is added to cover this case. Thanks to Jeff Janes for the extra investigation. Reported-by: Frank Gagnepain Author: Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/16476-692ef7b84e5fb893@postgresql.org Backpatch-through: 9.5 22 July 2020, 05:52:46 UTC
171633f Tom Lane 21 July 2020, 23:40:44 UTC neqjoinsel must now pass through collation to eqjoinsel. Since commit 044c99bc5, eqjoinsel passes the passed-in collation to any operators it invokes. However, neqjoinsel failed to pass on whatever collation it got, so that if we invoked a collation-dependent operator via that code path, we'd get "could not determine which collation to use for string comparison" or the like. Per report from Justin Pryzby. Back-patch to v12, like the previous commit. Discussion: https://postgr.es/m/20200721191606.GL5748@telsasoft.com 21 July 2020, 23:40:44 UTC
43ef3c4 Tom Lane 21 July 2020, 16:38:08 UTC Assert that we don't insert nulls into attnotnull catalog columns. The executor checks for this error, and so does the bootstrap catalog loader, but we never checked for it in retail catalog manipulations. The folly of that has now been exposed, so let's add assertions checking it. Checking in CatalogTupleInsert[WithInfo] and CatalogTupleUpdate[WithInfo] should be enough to cover this. Back-patch to v10; the aforesaid functions didn't exist before that, and it didn't seem worth adapting the patch to the oldest branches. But given the risk of JIT crashes, I think we certainly need this as far back as v11. Pre-v13, we have to explicitly exclude pg_subscription.subslotname and pg_subscription_rel.srsublsn from the checks, since they are mismarked. (Even if we change our mind about applying BKI_FORCE_NULL in the branch tips, it doesn't seem wise to have assertions that would fire in existing databases.) Discussion: https://postgr.es/m/298837.1595196283@sss.pgh.pa.us 21 July 2020, 16:38:08 UTC
b7103bb Tom Lane 21 July 2020, 15:40:46 UTC Avoid direct C access to possibly-null pg_subscription_rel.srsublsn. This coding technique is unsafe, since we'd be accessing off the end of the tuple if the field is null. SIGSEGV is pretty improbable, but perhaps not impossible. Also, returning garbage for the LSN doesn't seem like a great idea, even if callers aren't looking at it today. Also update docs to point out explicitly that pg_subscription.subslotname and pg_subscription_rel.srsublsn can be null. Perhaps we should mark these two fields BKI_FORCE_NULL, so that they'd be correctly labeled in databases that are initdb'd in the future. But we can't force that for existing databases, and on balance it's not too clear that having a mix of different catalog contents in the field would be wise. Apply to v10 (where this code came in) through v12. Already fixed in v13 and HEAD. Discussion: https://postgr.es/m/732838.1595278439@sss.pgh.pa.us 21 July 2020, 15:40:46 UTC
798b4fa Tom Lane 20 July 2020, 19:54:24 UTC Kluge slot_compile_deform() to ignore incorrect attnotnull markings. Since we mustn't force an initdb in released branches, there is no simple way to correct the markings of pg_subscription.subslotname and pg_subscription_rel.srsublsn as attnotnull in existing pre-v13 installations. Fortunately, released branches don't rely on attnotnull being correct for much. The planner looks at it in relation_excluded_by_constraints, but it'd be difficult to get that to matter for a query on a system catalog. The only place where it's really problematic is in JIT's slot_compile_deform(), which can produce incorrect code that crashes if there are NULLs in an allegedly not-null column. Hence, hack up slot_compile_deform() to be specifically aware of these two incorrect markings and not trust them. This applies to v11 and v12; the JIT code didn't exist before that, and we've fixed the markings in v13. Discussion: https://postgr.es/m/229396.1595191345@sss.pgh.pa.us 20 July 2020, 19:54:24 UTC
71e561b Tom Lane 20 July 2020, 17:40:16 UTC Fix construction of updated-columns bitmap in logical replication. Commit b9c130a1f failed to apply the publisher-to-subscriber column mapping while checking which columns were updated. Perhaps less significantly, it didn't exclude dropped columns either. This could result in an incorrect updated-columns bitmap and thus wrong decisions about whether to fire column-specific triggers on the subscriber while applying updates. In HEAD (since commit 9de77b545), it could also result in accesses off the end of the colstatus array, as detected by buildfarm member skink. Fix the logic, and adjust 003_constraints.pl so that the problem is exposed in unpatched code. In HEAD, also add some assertions to check that we don't access off the ends of these newly variable-sized arrays. Back-patch to v10, as b9c130a1f was. Discussion: https://postgr.es/m/CAH2-Wz=79hKQ4++c5A060RYbjTHgiYTHz=fw6mptCtgghH2gJA@mail.gmail.com 20 July 2020, 17:40:16 UTC
ab5ad0c Michael Paquier 18 July 2020, 13:43:45 UTC doc: Refresh more URLs in the docs This updates some URLs that are redirections, mostly to an equivalent using https. One URL referring to generalized partial indexes was outdated. Author: Kyotaro Horiguchi Discussion: https://postgr.es/m/20200717.121308.1369606287593685396.horikyota.ntt@gmail.com Backpatch-through: 9.5 18 July 2020, 13:43:45 UTC
aee6729 Michael Paquier 18 July 2020, 01:42:50 UTC doc: Fix description of \copy for psql The WHERE clause introduced by 31f3817 was not described. While on it, split the grammar of \copy FROM and TO into two distinct parts for clarity as they support different set of options. Author: Vignesh C Discussion: https://postgr.es/m/CALDaNm3zWr=OmxeNqOqfT=uZTSdam_j-gkX94CL8eTNfgUtf6A@mail.gmail.com Backpatch-through: 12 18 July 2020, 01:42:50 UTC
6623222 Tom Lane 17 July 2020, 17:03:50 UTC Cope with data-offset-less archive files during out-of-order restores. pg_dump produces custom-format archive files that lack data offsets when it is unable to seek its output. Up to now that's been a hazard for pg_restore. But if pg_restore is able to seek in the archive file, there is no reason to throw up our hands when asked to restore data blocks out of order. Instead, whenever we are searching for a data block, record the locations of the blocks we passed over (that is, fill in the missing data-offset fields in our in-memory copy of the TOC data). Then, when we hit a case that requires going backwards, we can just seek back. Also track the furthest point that we've searched to, and seek back to there when beginning a search for a new data block. This avoids possible O(N^2) time consumption, by ensuring that each data block is examined at most twice. (On Unix systems, that's at most twice per parallel-restore job; but since Windows uses threads here, the threads can share block location knowledge, reducing the amount of duplicated work.) We can also improve the code a bit by using fseeko() to skip over data blocks during the search. This is all of some use even in simple restores, but it's really significant for parallel pg_restore. In that case, we require seekability of the input already, and we will very probably need to do out-of-order restores. Back-patch to v12, as this fixes a regression introduced by commit 548e50976. Before that, parallel restore avoided requesting out-of-order restores, so it would work on a data-offset-less archive. Now it will again. Ideally this patch would include some test coverage, but there are other open bugs that need to be fixed before we can extend our coverage of parallel restore very much. Plan to revisit that later. David Gilman and Tom Lane; reviewed by Justin Pryzby Discussion: https://postgr.es/m/CALBH9DDuJ+scZc4MEvw5uO-=vRyR2=QF9+Yh=3hPEnKHWfS81A@mail.gmail.com 17 July 2020, 17:03:50 UTC
39a068c Tom Lane 17 July 2020, 16:14:28 UTC Remove manual tracking of file position in pg_dump/pg_backup_custom.c. We do not really need to track the file position by hand. We were already relying on ftello() whenever the archive file is seekable, while if it's not seekable we don't need the file position info anyway because we're not going to be able to re-write the TOC. Moreover, that tracking was buggy since it failed to account for the effects of fseeko(). Somewhat remarkably, that seems not to have made for any live bugs up to now. We could fix the oversights, but it seems better to just get rid of the whole error-prone mess. In itself this is merely code cleanup. However, it's necessary infrastructure for an upcoming bug-fix patch (because that code *does* need valid file position after fseeko). The bug fix needs to go back as far as v12; hence, back-patch that far. Discussion: https://postgr.es/m/CALBH9DDuJ+scZc4MEvw5uO-=vRyR2=QF9+Yh=3hPEnKHWfS81A@mail.gmail.com 17 July 2020, 16:14:28 UTC
a40733d Tom Lane 17 July 2020, 15:03:55 UTC Ensure that distributed timezone abbreviation files are plain ASCII. We had two occurrences of "Mitteleuropäische Zeit" in Europe.txt, though the corresponding entries in Default were spelled "Mitteleuropaeische Zeit". Standardize on the latter spelling to avoid questions of which encoding to use. While here, correct a couple of other trivial inconsistencies between the Default file and the supposedly-matching entries in the *.txt files, as exposed by some checking with comm(1). Also, add BDST to the Europe.txt file; it previously was only listed in Default. None of this has any direct functional effect. Per complaint from Christoph Berg. As usual for timezone data patches, apply to all branches. Discussion: https://postgr.es/m/20200716100743.GE3534683@msg.df7cb.de 17 July 2020, 15:04:30 UTC
dafa153 Peter Eisentraut 17 July 2020, 13:16:13 UTC Fix whitespace 17 July 2020, 13:16:27 UTC
cd113a0 Michael Paquier 16 July 2020, 06:52:58 UTC Switch pg_test_fsync to use binary mode on Windows pg_test_fsync has always opened files using the text mode on Windows, as this is the default mode used if not enforced by _setmode(). This fixes a failure when running pg_test_fsync down to 12 because O_DSYNC and the text mode are not able to work together nicely. We fixed the handling of O_DSYNC in 12~ for the tool by switching to the concurrent-safe version of fopen() in src/port/ with 0ba06e0. And 40cfe86, by enforcing the text mode for compatibility reasons if O_TEXT or O_BINARY are not specified by the caller, broke pg_test_fsync. For all versions, this avoids any translation overhead, and pg_test_fsync should test binary writes, so it is a gain in all cases. Note that O_DSYNC is still not handled correctly in ~11, leading to pg_test_fsync to show insanely high numbers for open_datasync() (using this property it is easy to notice that the binary mode is much faster). This would require a backpatch of 0ba06e0 and 40cfe86, which could potentially break existing applications, so this is left out. There are no TAP tests for this tool yet, so I have checked all builds manually using MSVC. We could invent a new option to run a single transaction instead of using a duration of 1s to make the tests a maximum short, but this is left as future work. Thanks to Bruce Momjian for the discussion. Reported-by: Jeff Janes Author: Michael Paquier Discussion: https://postgr.es/m/16526-279ded30a230d275@postgresql.org Backpatch-through: 9.5 16 July 2020, 06:52:58 UTC
9292747 Michael Paquier 15 July 2020, 06:17:36 UTC Fix handling of missing files when using pg_rewind with online source When working with an online source cluster, pg_rewind gets a list of all the files in the source data directory using a WITH RECURSIVE query, returning a NULL result for a file's metadata if it gets removed between the moment it is listed in a directory and the moment its metadata is obtained with pg_stat_file() (say a recycled WAL segment). The query result was processed in such a way that for each tuple we checked only that the first file's metadata was NULL. This could have two consequences, both resulting in a failure of the rewind: - If the first tuple referred to a removed file, all files from the source would be ignored. - Any file actually missing would not be considered as such. While on it, rework slightly the code so as no values are saved if we know that a file is going to be skipped. Issue introduced by b36805f, so backpatch down to 9.5. Author: Justin Pryzby, Michael Paquier Reviewed-by: Daniel Gustafsson, Masahiko Sawada Discussion: https://postgr.es/m/20200713061010.GC23581@telsasoft.com Backpatch-through: 9.5 15 July 2020, 06:17:36 UTC
de797e8 Tom Lane 14 July 2020, 22:56:49 UTC Fix bitmap AND/OR scans on the inside of a nestloop partition-wise join. reparameterize_path_by_child() failed to reparameterize BitmapAnd and BitmapOr paths. This matters only if such a path is chosen as the inside of a nestloop partition-wise join, where we have to pass in parameters from the outside of the nestloop. If that did happen, we generated a bad plan that would likely lead to crashes at execution. This is not entirely reparameterize_path_by_child()'s fault though; it's the victim of an ancient decision (my ancient decision, I think) to not bother filling in param_info in BitmapAnd/Or path nodes. That caused the function to believe that such nodes and their children contain no parameter references and so need not be processed. In hindsight that decision looks pretty penny-wise and pound-foolish: while it saves a few cycles during path node setup, we do commonly need the information later. In particular, by reversing the decision and requiring valid param_info data in all nodes of a bitmap path tree, we can get rid of indxpath.c's get_bitmap_tree_required_outer() function, which computed the data on-demand. It's not unlikely that that nets out as a savings of cycles in many scenarios. A couple of other things in indxpath.c can be simplified as well. While here, get rid of some cases in reparameterize_path_by_child() that are visibly dead or useless, given that we only care about reparameterizing paths that can be on the inside of a parameterized nestloop. This case reminds one of the maxim that untested code probably does not work, so I'm unwilling to leave unreachable code in this function. (I did leave the T_Gather case in place even though it's not reached in the regression tests. It's not very clear to me when the planner might prefer to put Gather below rather than above a nestloop, but at least in principle the case might be interesting.) Per bug #16536, originally from Arne Roland but with a test case by Andrew Gierth. Back-patch to v11 where this code came in. Discussion: https://postgr.es/m/16536-2213ee0b3aad41fd@postgresql.org 14 July 2020, 22:56:49 UTC
1231a0b David Rowley 14 July 2020, 05:03:12 UTC Fix timing issue with ALTER TABLE's validate constraint An ALTER TABLE to validate a foreign key in which another subcommand already caused a pending table rewrite could fail due to ALTER TABLE attempting to validate the foreign key before the actual table rewrite takes place. This situation could result in an error such as: ERROR: could not read block 0 in file "base/nnnnn/nnnnn": read only 0 of 8192 bytes The failure here was due to the SPI call which validates the foreign key trying to access an index which is yet to be rebuilt. Similarly, we also incorrectly tried to validate CHECK constraints before the heap had been rewritten. The fix for both is to delay constraint validation until phase 3, after the table has been rewritten. For CHECK constraints this means a slight behavioral change. Previously ALTER TABLE VALIDATE CONSTRAINT on inheritance tables would be validated from the bottom up. This was different from the order of evaluation when a new CHECK constraint was added. The changes made here aligns the VALIDATE CONSTRAINT evaluation order for inheritance tables to be the same as ADD CONSTRAINT, which is generally top-down. Reported-by: Nazli Ugur Koyluoglu, using SQLancer Discussion: https://postgr.es/m/CAApHDvp%3DZXv8wiRyk_0rWr00skhGkt8vXDrHJYXRMft3TjkxCA%40mail.gmail.com Backpatch-through: 9.5 (all supported versions) 14 July 2020, 05:03:12 UTC
330410e Michael Paquier 14 July 2020, 04:17:34 UTC Fix comments related to table AMs Incorrect function names were referenced. As this fixes some portions of tableam.h, that is mentioned in the docs as something to look at when implementing a table AM, backpatch down to 12 where this has been introduced. Author: Hironobu Suzuki Discussion: https://postgr.es/m/8fe6d672-28dd-3f1d-7aed-ac2f6d599d3f@interdb.jp Backpatch-through: 12 14 July 2020, 04:17:34 UTC
d3b642a Tom Lane 14 July 2020, 00:38:21 UTC Cope with lateral references in the quals of a subquery RTE. The qual pushdown logic assumed that all Vars in a restriction clause must be Vars referencing subquery outputs; but since we introduced LATERAL, it's possible for such a Var to be a lateral reference instead. This led to an assertion failure in debug builds. In a non-debug build, there might be no ill effects (if qual_is_pushdown_safe decided the qual was unsafe anyway), or we could get failures later due to construction of an invalid plan. I've not gone to much length to characterize the possible failures, but at least segfaults in the executor have been observed. Given that this has been busted since 9.3 and it took this long for anybody to notice, I judge that the case isn't worth going to great lengths to optimize. Hence, fix by just teaching qual_is_pushdown_safe that such quals are unsafe to push down, matching the previous behavior when it accidentally didn't fail. Per report from Tom Ellis. Back-patch to all supported branches. Discussion: https://postgr.es/m/20200713175124.GQ8220@cloudinit-builder 14 July 2020, 00:38:21 UTC
e8b22a8 Tom Lane 13 July 2020, 15:57:55 UTC Fix bugs in libpq's management of GSS encryption state. GSS-related resources should be cleaned up in pqDropConnection, not freePGconn, else the wrong things happen when resetting a connection or trying to switch to a different server. It's also critical to reset conn->gssenc there. During connection setup, initialize conn->try_gss at the correct place, else switching to a different server won't work right. Remove now-redundant cleanup of GSS resources around one (and, for some reason, only one) pqDropConnection call in connectDBStart. Per report from Kyotaro Horiguchi that psql would freeze up, rather than successfully resetting a GSS-encrypted connection after a server restart. This is YA oversight in commit b0b39f72b, so back-patch to v12. Discussion: https://postgr.es/m/20200710.173803.435804731896516388.horikyota.ntt@gmail.com 13 July 2020, 15:57:55 UTC
5fea14f Tom Lane 11 July 2020, 17:36:50 UTC Avoid trying to restore table ACLs and per-column ACLs in parallel. Parallel pg_restore has always supposed that ACL items for different objects are independent and can be restored in parallel without conflicts. However, there is one case where this fails: because REVOKE on a table is defined to also revoke the privilege(s) at column level, we can't restore per-column ACLs till after we restore any table-level privileges on their table. Failure to honor this restriction can lead to "tuple concurrently updated" errors during parallel restore, or even to the per-column ACLs silently disappearing because the table-level REVOKE is executed afterwards. To fix, add a dependency from each column-level ACL item to its table's ACL item, if there is one. Note that this doesn't fix the hazard for pre-existing archive files, only for ones made with a corrected pg_dump. Given that the bug's been there quite awhile without field reports, I think this is acceptable. This requires changing the API of pg_dump's dumpACL() function. To keep its argument list from getting even longer, I removed the "CatalogId objCatId" argument, which has been unused for ages. Per report from Justin Pryzby. Back-patch to all supported branches. Discussion: https://postgr.es/m/20200706050129.GW4107@telsasoft.com 11 July 2020, 17:36:50 UTC
f4ae676 Alexander Korotkov 11 July 2020, 00:21:00 UTC Forbid numeric NaN in jsonpath SQL standard doesn't define numeric Inf or NaN values. It appears even more ridiculous to support then in jsonpath assuming JSON doesn't support these values as well. This commit forbids returning NaN from .double(), which was previously allowed. NaN can't be result of inner-jsonpath computation over non-NaNs. So, we can not expect NaN in the jsonpath output. Reported-by: Tom Lane Discussion: https://postgr.es/m/203949.1591879542%40sss.pgh.pa.us Author: Alexander Korotkov Reviewed-by: Tom Lane Backpatch-through: 12 11 July 2020, 00:21:57 UTC
3ec5f6b Alexander Korotkov 11 July 2020, 00:20:46 UTC Improve error reporting for jsonpath .double() method When jsonpath .double() method detects that numeric or string can't be converted to double precision, it throws an error. This commit makes these errors explicitly express the reason of failure. Discussion: https://postgr.es/m/CAPpHfdtqJtiSXkP7tOXez18NxhLUH_-75bL8%3DOce4Ki%2Bbv7V6Q%40mail.gmail.com Author: Alexander Korotkov Reviewed-by: Tom Lane Backpatch-through: 12 11 July 2020, 00:21:53 UTC
a5f18e8 Tom Lane 10 July 2020, 17:16:00 UTC Doc: update or remove dead external links. Re-point comp.ai.genetic FAQ link to a more stable address. Remove stale links to AIX documentation; we don't really need to tell AIX users how to use their systems. Remove stale links to HP documentation about SSL. We've had to update those twice before, making it increasingly obvious that HP does not intend them to be stable landing points. They're not particularly authoritative, either. (This change effectively reverts bbd3bdba3.) Daniel Gustafsson and Álvaro Herrera, per a gripe from Kyotaro Horiguchi. Back-patch, since these links are just as dead in the back branches. Discussion: https://postgr.es/m/20200709.161226.204639179120026914.horikyota.ntt@gmail.com 10 July 2020, 17:16:00 UTC
ca5001a Alvaro Herrera 10 July 2020, 00:13:25 UTC Remove WARNING message from brin_desummarize_range This message was being emitted on the grounds that only crashed summarization could cause it, but in reality even an aborted vacuum could do it ... which makes it way too noisy, particularly since it shows up in regression tests and makes them die. Reported by Tom Lane. Discussion: https://postgr.es/m/489091.1593534251@sss.pgh.pa.us 10 July 2020, 00:13:25 UTC
a3cfb10 Tom Lane 09 July 2020, 21:38:52 UTC Tighten up Windows CRLF conversion in our TAP test scripts. Back-patch commits 91bdf499b and ffb4cee43, so that all branches agree on when and how to do Windows CRLF conversion. This should close the referenced thread. Thanks to Andrew Dunstan for discussion/review. Discussion: https://postgr.es/m/412ae8da-76bb-640f-039a-f3513499e53d@gmx.net 09 July 2020, 21:38:52 UTC
2564e2d Tom Lane 09 July 2020, 20:02:23 UTC Fix pg_current_logfile() to not emit a carriage return on Windows. Due to not having our signals straight about CRLF vs. LF line termination, the output of pg_current_logfile() included a trailing \r on Windows. To fix, force the file descriptor it uses into text mode. While here, move a couple of local variable declarations to make the function's logic clearer. In v12 and v13, also back-patch the test added by 1c4e88e2f so that this function has some test coverage. However, the 004_logrotate.pl test script doesn't exist before v12, and it didn't seem worth adding to older branches just for this. Per report from Thomas Kellerer. Back-patch to v10 where this function was added. Discussion: https://postgr.es/m/412ae8da-76bb-640f-039a-f3513499e53d@gmx.net 09 July 2020, 20:02:23 UTC
0df3f9c Fujii Masao 09 July 2020, 04:31:33 UTC doc: Correct the description about the length of pg_stat_activity.query. pg_stat_activity.query text is truncated at 1024 bytes. But previously the document described that it's truncated at 1024 characters. This was not accurate when considering multibyte characters. Back-patch to v10 where this inaccurate description was added. Author: Atsushi Torikoshi Reviewed-by: Daniel Gustafsson, Fujii Masao Discussion: https://postgr.es/m/cd5b49a5a14e887542f5f569c1c6bde2@oss.nttdata.com 09 July 2020, 04:35:56 UTC
939db2c Michael Paquier 05 July 2020, 10:36:18 UTC doc: Fix incorrect reference to textout in plpgsql examples This error has survived for 22 years, and has been introduced by da63386. Reported-by: Erwin Brandstetter Discussion: https://postgr.es/m/CAGHENJ57wogGOvGXo5LgWYcqswxafLck8ELqHDR+zrkTPgs_OQ@mail.gmail.com Backpatch-through: 9.5 05 July 2020, 10:36:18 UTC
b615b23 Joe Conway 04 July 2020, 17:47:21 UTC Fix "ignoring return value" complaints from commit 96d1f423f9 The cfbot and some BF animals are complaining about the previous read_binary_file commit because of ignoring return value of ‘fread’. So let's make everyone happy by testing the return value even though not strictly needed. Reported by Justin Pryzby, and suggested patch by Tom Lane. Backpatched to v11 same as the previous commit. Reported-By: Justin Pryzby Reviewed-By: Tom Lane Discussion: https://postgr.es/m/flat/969b8d82-5bb2-5fa8-4eb1-f0e685c5d736%40joeconway.com Backpatch-through: 11 04 July 2020, 17:47:21 UTC
015e899 Joe Conway 04 July 2020, 10:28:44 UTC Read until EOF vice stat-reported size in read_binary_file read_binary_file(), used by SQL functions pg_read_file() and friends, uses stat to determine file length to read, when not passed an explicit length as an argument. This is problematic, for example, if the file being read is a virtual file with a stat-reported length of zero. Arrange to read until EOF, or StringInfo data string lenth limit, is reached instead. Original complaint and patch by me, with significant review, corrections, advice, and code optimizations by Tom Lane. Backpatched to v11. Prior to that only paths relative to the data and log dirs were allowed for files, so no "zero length" files were reachable anyway. Reviewed-By: Tom Lane Discussion: https://postgr.es/m/flat/969b8d82-5bb2-5fa8-4eb1-f0e685c5d736%40joeconway.com Backpatch-through: 11 04 July 2020, 10:28:44 UTC
153c14c Tom Lane 03 July 2020, 23:01:21 UTC Clamp total-tuples estimates for foreign tables to ensure planner sanity. After running GetForeignRelSize for a foreign table, adjust rel->tuples to be at least as large as rel->rows. This prevents bizarre behavior in estimate_num_groups() and perhaps other places, especially in the scenario where rel->tuples is zero because pg_class.reltuples is (suggesting that ANALYZE has never been run for the table). As things stood, we'd end up estimating one group out of any GROUP BY on such a table, whereas the default group-count estimate is more likely to result in a sane plan. Also, clarify in the documentation that GetForeignRelSize has the option to override the rel->tuples value if it has a better idea of what to use than what is in pg_class.reltuples. Per report from Jeff Janes. Back-patch to all supported branches. Patch by me; thanks to Etsuro Fujita for review Discussion: https://postgr.es/m/CAMkU=1xNo9cnan+Npxgz0eK7394xmjmKg-QEm8wYG9P5-CcaqQ@mail.gmail.com 03 July 2020, 23:01:21 UTC
8b83e01 Tom Lane 03 July 2020, 21:01:34 UTC Fix temporary tablespaces for shared filesets some more. Commit ecd9e9f0b fixed the problem in the wrong place, causing unwanted side-effects on the behavior of GetNextTempTableSpace(). Instead, let's make SharedFileSetInit() responsible for subbing in the value of MyDatabaseTableSpace when the default tablespace is called for. The convention about what is in the tempTableSpaces[] array is evidently insufficiently documented, so try to improve that. It also looks like SharedFileSetInit() is doing the wrong thing in the case where temp_tablespaces is empty. It was hard-wiring use of the pg_default tablespace, but it seems like using MyDatabaseTableSpace is more consistent with what happens for other temp files. Back-patch the reversion of PrepareTempTablespaces()'s behavior to 9.5, as ecd9e9f0b was. The changes in SharedFileSetInit() go back to v11 where that was introduced. (Note there is net zero code change before v11 from these two patch sets, so nothing to release-note.) Magnus Hagander and Tom Lane Discussion: https://postgr.es/m/CABUevExg5YEsOvqMxrjoNvb3ApVyH+9jggWGKwTDFyFCVWczGQ@mail.gmail.com 03 July 2020, 21:01:34 UTC
28a862e Magnus Hagander 03 July 2020, 13:09:06 UTC Fix temporary tablespaces for shared filesets A likely copy/paste error in 98e8b480532 from back in 2004 would cause temp tablespace to be reset to InvalidOid if temp_tablespaces was set to the same value as the primary tablespace in the database. This would cause shared filesets (such as for parallel hash joins) to ignore them, putting the temporary files in the default tablespace instead of the configured one. The bug is in the old code, but it appears to have been exposed only once we had shared filesets. Reviewed-By: Daniel Gustafsson Discussion: https://postgr.es/m/CABUevExg5YEsOvqMxrjoNvb3ApVyH+9jggWGKwTDFyFCVWczGQ@mail.gmail.com Backpatch-through: 9.5 03 July 2020, 13:10:33 UTC
6425d6f Michael Meskes 30 June 2020, 15:31:08 UTC Fix ecpg crash with bytea and cursor variables. Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 30 June 2020, 16:35:21 UTC
d7fe24e Bruce Momjian 30 June 2020, 16:26:51 UTC doc: clarify that storage parameter values are optional In a few cases, the documented syntax specified storage parameter values as required. Reported-by: galiev_mr@taximaxim.ru Discussion: https://postgr.es/m/159283163235.684.4482737698910467437@wrigleys.postgresql.org Backpatch-through: 9.5 30 June 2020, 16:26:51 UTC
35d5f69 Bruce Momjian 30 June 2020, 15:55:53 UTC doc: change pg_upgrade wal_level to be not minimal Previously it was specified to be only replica. Discussion: https://postgr.es/m/20200618180058.GK7349@momjian.us Backpatch-through: 9.5 30 June 2020, 15:55:53 UTC
a5e169a Noah Misch 28 June 2020, 05:05:04 UTC Fix documentation of "must be vacuumed within" warning. Warnings start 10M transactions before xidStopLimit, which is 11M transactions before wraparound. The sample WARNING output showed a value greater than 11M, and its HINT message predated commit 25ec228ef760eb91c094cc3b6dea7257cc22ffb5. Hence, the sample was impossible. Back-patch to 9.5 (all supported versions). 28 June 2020, 05:05:07 UTC
257ee34 Bruce Momjian 25 June 2020, 22:33:28 UTC doc: mention trigger helper functions in CREATE TRIGGER docs Reported-by: petermpallesen@gmail.com Discussion: https://postgr.es/m/159195294959.673.5752624528747900508@wrigleys.postgresql.org Backpatch-through: 9.5 25 June 2020, 22:33:28 UTC
ed3af2c Bruce Momjian 25 June 2020, 22:22:44 UTC docs: clarify that CREATE DATABASE does not copy db permissions That is, those database permissions set by GRANT. Diagnosed-by: Joseph Nahmias Discussion: https://postgr.es/m/20200614072613.GA21852@nahmias.net Backpatch-through: 9.5 25 June 2020, 22:22:44 UTC
ff28f35 Tom Lane 24 June 2020, 19:47:30 UTC Fix compiler warning induced by commit d8b15eeb8. I forgot that INT64_FORMAT can't be used with sscanf on Windows. Use the same trick of sscanf'ing into a temp variable as we do in some other places in zic.c. The upstream IANA code avoids the portability problem by relying on <inttypes.h>'s SCNdFAST64 macro. Once we're requiring C99 in all branches, we should do likewise and drop this set of diffs from upstream. For now, though, a hack seems fine, since we do not actually care about leapseconds anyway. Discussion: https://postgr.es/m/4e5d1a5b-143e-e70e-a99d-a3b01c1ae7c3@2ndquadrant.com 24 June 2020, 19:47:54 UTC
013d324 Alvaro Herrera 24 June 2020, 18:00:37 UTC Add parens to ConvertToXSegs macro The current definition is dangerous. No bugs exist in our code at present, but backpatch to 11 nonetheless where it was introduced. Author: Álvaro Herrera <alvherre@alvh.no-ip.org> 24 June 2020, 18:00:37 UTC
d3d8755 Tom Lane 22 June 2020, 15:46:41 UTC Undo double-quoting of index names in non-text EXPLAIN output formats. explain_get_index_name() applied quote_identifier() to the index name. This is fine for text output, but the non-text output formats all have their own quoting conventions and would much rather start from the actual index name. For example in JSON you'd get something like "Index Name": "\"My Index\"", which is surely not desirable, especially when the same does not happen for table names. Hence, move the responsibility for applying quoting out to the callers, where it can go into already-existing special code paths for text format. This changes the API spec for users of explain_get_index_name_hook: before, they were supposed to apply quote_identifier() if necessary, now they should not. Research suggests that the only publicly available user of the hook is hypopg, and it actually forgot to apply quoting anyway, so it's fine. (In any case, there's no behavioral change for the output of a hook as seen in non-text EXPLAIN formats, so this won't break any case that programs should be relying on.) Digging in the commit logs, it appears that quoting was included in explain_get_index_name's duties when commit 604ffd280 invented it; and that was fine at the time because we only had text output format. This should have been rethought when non-text formats were invented, but it wasn't. This is a fairly clear bug for users of non-text EXPLAIN formats, so back-patch to all supported branches. Per bug #16502 from Maciek Sakrejda. Patch by me (based on investigation by Euler Taveira); thanks to Julien Rouhaud for review. Discussion: https://postgr.es/m/16502-57bd1c9f913ed1d1@postgresql.org 22 June 2020, 15:46:41 UTC
396a44a Alexander Korotkov 20 June 2020, 14:34:51 UTC Fix masking of SP-GiST pages during xlog consistency check spg_mask() didn't take into account that pd_lower equal to SizeOfPageHeaderData is still valid value. This commit fixes that. Backpatch to 11, where spg_mask() pg_lower check was introduced. Reported-by: Michael Paquier Discussion: https://postgr.es/m/20200615131405.GM52676%40paquier.xyz Backpatch-through: 11 20 June 2020, 14:35:55 UTC
5b52008 Alvaro Herrera 19 June 2020, 20:46:07 UTC Ensure write failure reports no-disk-space A few places calling fwrite and gzwrite were not setting errno to ENOSPC when reporting errors, as is customary; this led to some failures being reported as "could not write file: Success" which makes us look silly. Make a few of these places in pg_dump and pg_basebackup use our customary pattern. Backpatch-to: 9.5 Author: Justin Pryzby <pryzby@telsasoft.com> Author: Tom Lane <tgl@sss.pgh.pa.us> Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com 19 June 2020, 20:46:07 UTC
b22ca76 Tom Lane 19 June 2020, 17:55:21 UTC Future-proof regression tests against possibly-missing posixrules file. The IANA time zone folk have deprecated use of a "posixrules" file in the tz database. While for now it's our choice whether to keep supplying one in our own builds, installations built with --with-system-tzdata will soon be needing to cope with that file not being present, at least on some platforms. This causes a problem for the horology test, which expected the nonstandard POSIX zone spec "CST7CDT" to apply pre-2007 US daylight savings rules. That does happen if the posixrules file supplies such information, but otherwise the test produces undesired results. To fix, add an explicit transition date rule that matches 2005 practice. (We could alternatively have switched the test to use some real time zone, but it seems useful to have coverage of this type of zone spec.) While at it, update a documentation example that also relied on "CST7CDT"; use a real-world zone name instead. Also, document why the zone names EST5EDT, CST6CDT, MST7MDT, PST8PDT aren't subject to similar failures when "posixrules" is missing. Back-patch to all supported branches, since the hazard is the same for all. Discussion: https://postgr.es/m/1665379.1592581287@sss.pgh.pa.us 19 June 2020, 17:55:21 UTC
6cc2866 Andres Freund 08 June 2020, 23:50:37 UTC Fix deadlock danger when atomic ops are done under spinlock. This was a danger only for --disable-spinlocks in combination with atomic operations unsupported by the current platform. While atomics.c was careful to signal that a separate semaphore ought to be used when spinlock emulation is active, spin.c didn't actually implement that mechanism. That's my (Andres') fault, it seems to have gotten lost during the development of the atomic operations support. Fix that issue and add test for nesting atomic operations inside a spinlock. Author: Andres Freund Discussion: https://postgr.es/m/20200605023302.g6v3ydozy5txifji@alap3.anarazel.de Backpatch: 9.5- 18 June 2020, 21:13:06 UTC
008c119 Andres Freund 08 June 2020, 23:36:51 UTC Add basic spinlock tests to regression tests. As s_lock_test, the already existing test for spinlocks, isn't run in an automated fashion (and doesn't test a normal backend environment), adding tests that are run as part of a normal regression run is a good idea. Particularly in light of several recent and upcoming spinlock related fixes. Currently the new tests are run as part of the pre-existing test_atomic_ops() test. That perhaps can be quibbled about, but for now seems ok. The only operations that s_lock_test tests but the new tests don't are the detection of a stuck spinlock and S_LOCK_FREE (which is otherwise unused, not implemented on all platforms, and will be removed). This currently contains a test for more than INT_MAX spinlocks (only run with --disable-spinlocks), to ensure the recent commit fixing a bug with more than INT_MAX spinlock initializations is correct. That test is somewhat slow, so we might want to disable it after a few days. It might be worth retiring s_lock_test after this. The added coverage of a stuck spinlock probably isn't worth the added complexity? Author: Andres Freund Discussion: https://postgr.es/m/20200606023103.avzrctgv7476xj7i@alap3.anarazel.de 18 June 2020, 21:06:26 UTC
3b8210d Tom Lane 18 June 2020, 20:27:18 UTC Doc: document POSIX-style time zone specifications in full. We'd glossed over most of this complexity for years, but it's hard to avoid writing it all down now, so that we can explain what happens when there's no "posixrules" file in the IANA time zone database. That was at best a tiny minority situation till now, but it's likely to become quite common in the future, so we'd better explain it. Nonetheless, we don't really encourage people to use POSIX zone specs; picking a named zone is almost always what you really want, unless perhaps you're stuck with an out-of-date zone database. Therefore, let's shove all this detail into an appendix. Patch by me; thanks to Robert Haas for help with some awkward wording. Discussion: https://postgr.es/m/1390.1562258309@sss.pgh.pa.us 18 June 2020, 20:27:41 UTC
16f4312 Michael Paquier 18 June 2020, 07:35:32 UTC Fix oldest xmin and LSN computation across repslots after advancing Advancing a replication slot did not recompute the oldest xmin and LSN values across replication slots, preventing resource removal like segments not recycled at checkpoint time. The original commit that introduced the slot advancing in 9c7d06d never did the update of those oldest values, and b0afdca removed this code. This commit adds a TAP test to check segment recycling with advancing for physical slots, enforcing an extra segment switch before advancing to check if the segment gets correctly recycled after a checkpoint. Reported-by: Andres Freund Reviewed-by: Alexey Kondratov, Kyptaro Horiguchi Discussion: https://postgr.es/m/20200609171904.kpltxxvjzislidks@alap3.anarazel.de Backpatch-through: 11 18 June 2020, 07:35:32 UTC
812a84d Tom Lane 17 June 2020, 22:29:29 UTC Sync our copy of the timezone library with IANA release tzcode2020a. This absorbs a leap-second-related bug fix in localtime.c, and teaches zic to handle an expiration marker in the leapseconds file. Neither are of any interest to us (for the foreseeable future anyway), but we need to stay more or less in sync with upstream. Also adjust some over-eager changes in the README from commit 957338418. I have no intention of making changes that require C99 in this code, until such time as all the live back branches require C99. Otherwise back-patching will get too exciting. For the same reason, absorb assorted whitespace and other cosmetic changes from HEAD into the back branches; mostly this reflects use of improved versions of pgindent. All in all then, quite a boring update. But I figured I'd get it done while I was looking at this code. 17 June 2020, 22:29:44 UTC
b91cfaa Andres Freund 08 June 2020, 22:25:49 UTC spinlock emulation: Fix bug when more than INT_MAX spinlocks are initialized. Once the counter goes negative we ended up with spinlocks that errored out on first use (due to check in tas_sema). Author: Andres Freund Reviewed-By: Robert Haas Discussion: https://postgr.es/m/20200606023103.avzrctgv7476xj7i@alap3.anarazel.de Backpatch: 9.5- 17 June 2020, 19:51:12 UTC
dab8561 Tom Lane 16 June 2020, 20:41:11 UTC Doc: fix copy-and-pasteo in ecpg docs. The synopsis for PGTYPESinterval_free() used the wrong name. Discussion: https://postgr.es/m/159231203030.679.3061023914894071953@wrigleys.postgresql.org 16 June 2020, 20:41:23 UTC
28ee126 Thomas Munro 16 June 2020, 01:50:56 UTC Fix buffile.c error handling. Convert buffile.c error handling to use ereport. This fixes cases where I/O errors were indistinguishable from EOF or not reported. Also remove "%m" from error messages where errno would be bogus. While we're modifying those strings, add block numbers and short read byte counts where appropriate. Back-patch to all supported releases. Reported-by: Amit Khandekar <amitdkhan.pg@gmail.com> Reviewed-by: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Ibrar Ahmed <ibrar.ahmad@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CA%2BhUKGJE04G%3D8TLK0DLypT_27D9dR8F1RQgNp0jK6qR0tZGWOw%40mail.gmail.com 16 June 2020, 05:00:21 UTC
8e93359 Bruce Momjian 16 June 2020, 00:59:40 UTC pg_upgrade: set vacuum_defer_cleanup_age to zero Non-zero vacuum_defer_cleanup_age values cause pg_upgrade freezing of the system catalogs to be incomplete, or do nothing. This will cause the upgrade to fail in confusing ways. Reported-by: Laurenz Albe Discussion: https://postgr.es/m/7d6f6c22ba05ce0c526e9e8b7bfa8105e7da45e6.camel@cybertec.at Backpatch-through: 9.5 16 June 2020, 00:59:40 UTC
eb12864 Thomas Munro 14 June 2020, 23:33:13 UTC Doc: Add references for SI and SSI. Our documentation failed to point out that REPEATABLE READ is really snapshot isolation, which might be important to some users. Point to the standard reference paper for this complicated topic. Likewise, add a reference to the VLDB paper about PostgreSQL SSI, for technical information about our SSI implementation and how it compares to S2PL. While here, add a note about catalog access using a lower isolation level, per recent user complaint. Back-patch to all releases. Reported-by: Kyle Kingsbury <aphyr@jepsen.io> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Peter Geoghegan <pg@bowt.ie> Reviewed-by: Tatsuo Ishii <ishii@sraoss.co.jp> Discussion: https://postgr.es/m/db7b729d-0226-d162-a126-8a8ab2dc4443%40jepsen.io Discussion: https://postgr.es/m/16454-9408996bb1750faf%40postgresql.org 15 June 2020, 01:13:11 UTC
874372a Tom Lane 13 June 2020, 17:43:24 UTC Fix behavior of float aggregates for single Inf or NaN inputs. When there is just one non-null input value, and it is infinity or NaN, aggregates such as stddev_pop and covar_pop should produce a NaN result, because the calculation is not well-defined. They used to do so, but since we adopted Youngs-Cramer aggregation in commit e954a727f, they produced zero instead. That's an oversight, so fix it. Add tests exercising these edge cases. Affected aggregates are var_pop(double precision) stddev_pop(double precision) var_pop(real) stddev_pop(real) regr_sxx(double precision,double precision) regr_syy(double precision,double precision) regr_sxy(double precision,double precision) regr_r2(double precision,double precision) regr_slope(double precision,double precision) regr_intercept(double precision,double precision) covar_pop(double precision,double precision) corr(double precision,double precision) Back-patch to v12 where the behavior change was accidentally introduced. Report and patch by me; thanks to Dean Rasheed for review. Discussion: https://postgr.es/m/353062.1591898766@sss.pgh.pa.us 13 June 2020, 17:43:24 UTC
81cd796 Bruce Momjian 11 June 2020, 22:19:25 UTC doc: remove xreflabels from commits 75fcdd2ae2 and 85af628da5 xreflabels prevent references to the chapter numbers of sections id's. It should only be used in specific cases. Discussion: https://postgr.es/m/8315c0ca-7758-8823-fcb6-f37f9413e6b6@2ndquadrant.com Backpatch-through: 9.5 11 June 2020, 22:19:25 UTC
4284e11 Tom Lane 11 June 2020, 21:38:42 UTC Fix mishandling of NaN counts in numeric_[avg_]combine. When merging two NumericAggStates, the code missed adding the new state's NaNcount unless its N was also nonzero; since those counts are independent, this is wrong. This would only have visible effect if some partial aggregate scans found only NaNs while earlier ones found only non-NaNs; then we could end up falsely deciding that there were no NaNs and fail to return a NaN final result as expected. That's pretty improbable, so it's no surprise this hasn't been reported from the field. Still, it's a bug. I didn't try to produce a regression test that would show the bug, but I did notice that these functions weren't being reached at all in our regression tests, so I improved the tests to at least exercise them. With these additions, I see pretty complete code coverage on the aggregation-related functions in numeric.c. Back-patch to 9.6 where this code was introduced. (I only added the improved test case as far back as v10, though, since the relevant part of aggregates.sql isn't there at all in 9.6.) 11 June 2020, 21:38:42 UTC
e620a38 Peter Geoghegan 11 June 2020, 17:09:43 UTC Avoid update conflict out serialization anomalies. SSI's HeapCheckForSerializableConflictOut() test failed to correctly handle conditions involving a concurrently inserted tuple which is later concurrently updated by a separate transaction . A SELECT statement that called HeapCheckForSerializableConflictOut() could end up using the same XID (updater's XID) for both the original tuple, and the successor tuple, missing the XID of the xact that created the original tuple entirely. This only happened when neither tuple from the chain was visible to the transaction's MVCC snapshot. The observable symptoms of this bug were subtle. A pair of transactions could commit, with the later transaction failing to observe the effects of the earlier transaction (because of the confusion created by the update to the non-visible row). This bug dates all the way back to commit dafaa3ef, which added SSI. To fix, make sure that we check the xmin of concurrently inserted tuples that happen to also have been updated concurrently. Author: Peter Geoghegan Reported-By: Kyle Kingsbury Reviewed-By: Thomas Munro Discussion: https://postgr.es/m/db7b729d-0226-d162-a126-8a8ab2dc4443@jepsen.io Backpatch: All supported versions 11 June 2020, 17:09:43 UTC
b7ed1d9 Amit Kapila 11 June 2020, 08:40:43 UTC Fix typos. Reported-by: John Naylor Author: John Naylor Backpatch-through: 9.5 Discussion: https://postgr.es/m/CACPNZCtRuvs6G+EYqejhVJgBq2AKeZdXRVJsbX4syhO9gn5SNQ@mail.gmail.com 11 June 2020, 08:56:17 UTC
17d8cf2 Peter Eisentraut 10 June 2020, 09:57:41 UTC Update description of parameter password_encryption The previous description string still described the pre-PostgreSQL 10 (pre eb61136dc75a76caef8460fa939244d8593100f2) behavior of selecting between encrypted and unencrypted, but it is now choosing between encryption algorithms. 10 June 2020, 11:40:43 UTC
980a3cd Andres Freund 09 June 2020, 02:52:19 UTC Avoid need for valgrind suppressions for pg_atomic_init_u64 on some platforms. Previously we used pg_atomic_write_64_impl inside pg_atomic_init_u64. That works correctly, but on platforms without 64bit single copy atomicity it could trigger spurious valgrind errors about uninitialized memory, because we use compare_and_swap for atomic writes on such platforms. I previously suppressed one instance of this problem (6c878edc1df), but as Tom reports that wasn't enough. As the atomic variable cannot yet be concurrently accessible during initialization, it seems better to have pg_atomic_init_64_impl set the value directly. Change pg_atomic_init_u32_impl for symmetry. Reported-By: Tom Lane Author: Andres Freund Discussion: https://postgr.es/m/1714601.1591503815@sss.pgh.pa.us Backpatch: 9.5- 09 June 2020, 03:02:52 UTC
72766ad Thomas Munro 08 June 2020, 01:57:24 UTC Fix locking bugs that could corrupt pg_control. The redo routines for XLOG_CHECKPOINT_{ONLINE,SHUTDOWN} must acquire ControlFileLock before modifying ControlFile->checkPointCopy, or the checkpointer could write out a control file with a bad checksum. Likewise, XLogReportParameters() must acquire ControlFileLock before modifying ControlFile and calling UpdateControlFile(). Back-patch to all supported releases. Author: Nathan Bossart <bossartn@amazon.com> Author: Fujii Masao <masao.fujii@oss.nttdata.com> Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Thomas Munro <thomas.munro@gmail.com> Discussion: https://postgr.es/m/70BF24D6-DC51-443F-B55A-95735803842A%40amazon.com 08 June 2020, 01:58:35 UTC
b944b1d Thomas Munro 08 June 2020, 01:20:46 UTC Doc: Update example symptom of systemd misconfiguration. In PostgreSQL 10, we stopped using System V semaphores on Linux systems. Update the example we give of an error message from a misconfigured system to show what people are most likely to see these days. Back-patch to 10, where PREFERRED_SEMAPHORES=UNNAMED_POSIX arrived. Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://postgr.es/m/CA%2BhUKGLmJUSwybaPQv39rB8ABpqJq84im2UjZvyUY4feYhpWMw%40mail.gmail.com 08 June 2020, 01:21:50 UTC
fb2641f Noah Misch 07 June 2020, 23:27:13 UTC MSVC: Avoid warning when testing a TAP suite without PROVE_FLAGS. Commit 7be5d8df1f74b78620167d3abf32ee607e728919 surfaced the logic error, which had no functional implications, by adding "use warnings". The buildfarm always customizes PROVE_FLAGS, so the warning did not appear there. Back-patch to 9.5 (all supported versions). 07 June 2020, 23:27:17 UTC
2edf14f Tom Lane 07 June 2020, 17:44:13 UTC Try to read data from the socket in pqSendSome's write_failed paths. Even when we've concluded that we have a hard write failure on the socket, we should continue to try to read data. This gives us an opportunity to collect any final error message that the backend might have sent before closing the connection; moreover it is the job of pqReadData not pqSendSome to close the socket once EOF is detected. Due to an oversight in 1f39a1c06, pqSendSome failed to try to collect data in the case where we'd already set write_failed. The problem was masked for ordinary query operations (which really only make one write attempt anyway), but COPY to the server would continue to send data indefinitely after a mid-COPY connection loss. Hence, add pqReadData calls into the paths where pqSendSome drops data because of write_failed. If we've lost the connection, this will eventually result in closing the socket and setting CONNECTION_BAD, which will cause PQputline and siblings to report failure, allowing the application to terminate the COPY sooner. (Basically this restores what happened before 1f39a1c06.) There are related issues that this does not solve; for example, if the backend sends an error but doesn't drop the connection, we did and still will keep pumping COPY data as long as the application sends it. Fixing that will require application-visible behavior changes though, and anyway it's an ancient behavior that we've had few complaints about. For now I'm just trying to fix the regression from 1f39a1c06. Per a complaint from Andres Freund. Back-patch into v12 where 1f39a1c06 came in. Discussion: https://postgr.es/m/20200603201242.ofvm4jztpqytwfye@alap3.anarazel.de 07 June 2020, 17:44:13 UTC
a00222f Peter Eisentraut 07 June 2020, 11:18:36 UTC doc: Clean up title case use 07 June 2020, 11:19:25 UTC
8414f41 Peter Eisentraut 07 June 2020, 09:16:51 UTC doc: Fix incorrect link target 07 June 2020, 09:17:13 UTC
back to top