0.15.0

Merged PRs

doltgresql

• 1035: Tests and schema fixes for dolt.status, dolt.merge_status and dolt_constraint_violations_*
• 1033: switch DoltgresType to be used as pointer
• 1029: Fix schemas for dolt system tables with boolean columns
  Dolt PR: dolthub/dolt#8634
• 1014: Reworked DoltgresType function access
  This reworks how functions are accessed within DoltgresType, in addition to adding a faster path for functions that fall under specific criteria.
• 1004: Add tests for dolt_statistics
  GMS: dolthub/go-mysql-server#2762
  Dolt: dolthub/dolt#8589
• 1003: Add diff tests for dolt.docs
• 998: implement to_char for timestamp (mostly)
  This PR adds support for most of the date format specifiers mentioned here: https://www.postgresql.org/docs/current/functions-formatting.html
• 996: Support dolt.rebase, add tests
• 995: Tests for dolt_ignore per schema
  Dolt PR: dolthub/dolt#8581
• 994: Tests and fixes for dolt_rebase and dolt_ignore
  Dolt PR: https://github.com/dolthub/dolt/pull/8578/files
• 988: Added a mini sysbench test that runs on PRs
  This adds a new workflow that runs on every PR, similar to the regression tests. This runs a small version of the sysbench tests, primarily to detect large performance changes that a PR may unexpectedly introduce. For now, the variance is set to ±10%. That is, if it's under 10% in either direction, it'll just display as a white color. If it's greater, then it'll display as green or red respectively to indicate a possible performance regression. There's variability in the tests, that we'll want to account for, but perhaps the 10% window is too large. Or maybe we should run the tests for longer.
• 986: Revert "Merge pull request #904 from dolthub/jennifer/type"
  This reverts commit 40dba4f, reversing changes made to 165c30e.
  Reverting this change because of performance regression.
• 983: Major improvements to auth
  This adds a myriad of tests, in addition to:
  • Schema-Wide Table Privileges
  • Schema Privileges
  • Database Privileges
  • Role Memberships
    There are also many smaller changes, such as disallowing non-superusers from modifying superusers, and many more.
• 981: support insert DEFAULT VALUES
  This is postgres special syntax that allows you to just insert all default values.
  This is equivalent to insert into <tbl> values () in MySQL.
• 979: support EXCEPT set operation
• 978: support EXISTS subquery
  This PR adds support for EXISTS <subquery> clause.
  Additionally, we have to add a bunch of skips namely in the setup queries in the testing/go/regression/results directory; these queries encounter performance issues for joins over pg_catalog tables discussed here: #940
• 977: update Regression test to display 50 tests each at most with total number of tests
• 970: catch more unsupported partition of queries
  These were panicking before, but now we throw unsupported
• 965: Fix DEFAULT insert for NULL or unspecified column default values
• 964: Add dolt_schema and dolt_procedure tests
• 963: TestUpdate and TestDelete
• 949: Release v0.14.1
  Created by the Release workflow to update DoltgreSQL's version
• 945: support with
  somewhat related: #928
• 904: create DoltgresType struct used for all types
  This PR replaces current DoltgresType interface with DoltgresType struct. All built-in types will have pre-defined values for this struct. User-defined types will populate the struct fields with appropriate values.
  Depends on dolthub/go-mysql-server#2736

Closed Issues

• 985: pull requests need #benchmark so latency regressions can be caught before they're merged.
• 943: Unable to set statement_timeout
• 937: doltgres 0.14.0 regression test failure
• 843: Error on restore into doltgres (pg_restore)

0.14.1

Merged PRs

doltgresql

• 947: Fix parameter context for statement_timeout and recovery_target_name
  The statement_timeout parameter had the wrong context configured, which prevented users from being able to change its value. I did a quick skim of Doltgres' parameters and compared them to pg_catalog.pg_settings and found one other parameter context (recovery_target_name) that didn't match.
  Fixes: #943
• 942: Removed doltgres user and default database
  This removes the default doltgres user, so that we're more in-line with a standard Postgres installation. In addition, this also removes the default doltgres database, and fully replaces it with a postgres database. Postgres creates this database by default, so it's best that we mimic it.
  In addition, Postgres uses the username as the database name if a database name is not given, and I'm sure many users simply use the postgres user without even thinking about what database they're connecting to, so this allows for that workflow to work as expected.
• 941: throw unsupported error for PARTITION OF statements
  Catch PARTITION OF statements and return error instead of panicking.
• 940: Add caching for pg_catalog tables
  While investigating performance on some skipped tests for pg_catalog tables, I found a few areas to optimize around how we deal with OIDs. The biggest source of latency I found was that pg_catalog tables must iterate through every object in a database (e.g. schemas, tables, indexes, types) to generate OIDs and create the rows for each pg_catalog table. When multiple pg_catalog tables are joined together, this starts to consume a significant amount of time, with some queries triggering this database iteration process hundreds of thousands of times.
  This change creates a new cache in the session object that stores the data for pg_catalog tables so that it can be reused within the same query. The effect is that several queries that timed out before will now complete. Some of these queries are still slow (e.g. multiple seconds to execute) and can be further optimized. The biggest source of latency now seems to be from the join operations, since the data is currently not eligible for point lookups and has to be scanned repeatedly to stitch rows together.
  Another area for future optimization is the regclass/regtype/regproc IO_Output and IO_Input functions, which also trigger iterations over database objects to turn a relation name into an OID, and to turn an OID into a relation name.
  Longer term, instead of relying on iterating database objects and caching OID information, we eventually need to store this information so it can be quickly loaded, and keep it in sync as database objects are created/modified/deleted, including changes made through merges, resets, and other database version control changes.
• 939: Unskip many InsertInto engine tests
  This adds several new features / bug fixes found during testing:
  • DROP INDEX support
  • Support for ON CONFLICT UPDATE
  • Panic during insert of some UNION statements
  • Fixed case sensitivity bug in column names for INSERT statements
• 938: parse explain select and explain options
• 936: Release v0.14.0
  Created by the Release workflow to update DoltgreSQL's version
• 935: implement multiple table inherits
  This PR supports most of the functionality around PostgreSQL INHERITs.
  We just translate create table t1 (...) inherits (t2, t3, ...); to create table t1 like (t2, t3, ...);
  Note: MySQL does not support multiple LIKE statements
  gms logic: dolthub/go-mysql-server#2738
  syntax: dolthub/vitess#375

Closed Issues

• 943: Unable to set statement_timeout

0.14.0

Merged PRs

doltgresql

• 934: support VALUES()
  Add support for select ... from (values (...), (...), ...) sqa statements
• 921: more partition no-ops
• 919: support multi-alter statements
  We support multi alter statements, so we shouldn't error.
• 918: partitions are no-ops
  We parse, but ignore, PARTITION BY clauses in CREATE TABLE statements in gms, so do the same thing here.
  Docs noting this here: dolthub/docs#2392
• 909: have rootFinalizerIter implement sql.MutableRowIter
  We've moved rules out of analyzer into FinalizeIters so the applyRowUpdateAccumulator is applied afterwards.
  However, this seems to be a special sql.RowIter, so we need to expose the child iterators through the sql.CustomRowIter interface to apply that rule
• 908: Tests for system tables with _$tablename
  Dolt PR: dolthub/dolt#8527
• 901: More system table updates, add and unskip some tests
• 898: more unskipped engine tests
• 887: Bug fix: Make db.Ping(ctx) work correctly with Doltgres
  Implementations of db.Ping(ctx) from Go's database/sql package typically send some sort of empty query to test the liveness of a database connection. For example, the pq library sends ; and the pgx library sends -- ping.
  Before this change, Doltgres' parser would return an empty statement, instead of Vitess' ErrEmpty error, which caused GMS to try and execute a nil analyzed plan for these statements, return an error, and the ping check would fail. This change makes the Doltgres parser return the same Vitess ErrEmpty error that the Vitess parser throws, sending the same signal to GMS to handle an empty statement without returning an error to the client. The related PR dolthub/go-mysql-server#2716 updates the Parser interface documentation to explicitly call out the requirement to return ErrEmpty in order for empty statements to be handled correctly.
  For testing, I've added some no-op statements to the smoke tests, but since these tests go through as prepared statements, they don't follow the exact code path as db.Ping(ctx), so I've also added calls to db.Ping(ctx) in the test framework, which do reproduce this error. I've also added a unit test for Doltgres' Parser implementation that explicitly tests empty statement parsing.
  Fixes: #884
  Related to: dolthub/go-mysql-server#2716
• 886: /testing/logictest/harness/doltgres_server_harness.go: skip db.Ping call
• 882: drop domain support
  Added:
  • merge functionality for domain types
  • check for domain usage in tables for dropping domain
    TODO:
  • creating domain type creates instance of array type of this domain type
  • unique OID generated for each domain created
  • add domains to pg_type
  • ALTER TABLE (e.g.: modify existing column to domain type column)
  • use domain type as underlying type to another domain
• 881: Update first batch of dolt system tables to use dolt schema
  • dolt_status -> dolt.status
  • dolt_branches -> dolt.branches
  • dolt_log -> dolt.log
  • dolt_tags -> dolt.tags
  • dolt_docs -> dolt.docs
    These changes are mostly backwards compatible, unless you have an existing doc in dolt_docs, which will need to be recreated after updating
• 879: Allow nextval() to take a regclass instance
  Allows the nextval() function to take a regclass parameter. Also changes the output function of regclass so that the returned relation name is schema-qualified if the schema is not on search_path.
  This change also moves the GetCurrentSchemas() function into a new settings package to break a package import cycle. The new settings package is intended for low-level functions that access settings, without dependencies on other packages.
  Fixes: #850
• 878: tests for cross-schema foreign keys
  Depends on:
  dolthub/go-mysql-server#2713
  dolthub/dolt#8479
• 877: Function: set_config()
  Adds support for the set_config() function.
  Example Usage:

  SELECT set_config('mynamespace.foo', 'bar', false);
  set_config
  ------------
  bar
  SELECT current_setting('mynamespace.foo');
  current_setting
  -----------------
  bar

  Fixes: #852
• 872: rule changes
• 868: More ALTER TABLE Support
  Adds support for additional ALTER TABLE syntax:
  • Adding a UNIQUE constraint
  • Adding a CHECK constraint
  • Dropping constraints
    Adding support for check constraints triggered an issue with string literal value quoting that affects check constraints and column defaults. The fix was to make expression.Literal.String() match the behavior of GMS' expression.Literal.String() method and quote string literals. This required fixing another spot where we had been adding in quotes for string literals, as well as a small change in GMS (dolthub/go-mysql-server#2710).
    Fixes:
  • #799
  • #800
     
    Regresions Report:
    The regressions listed below are a little tricky to read, but everything seems to be working correctly as far as I can tell. In the first regression listed (INSERT INTO inhg VALUES ('foo');), this query now fails, because a previous statement now executes correctly to add a check constraint to a table, but our CREATE TABLE LIKE logic incorrectly copies over check constraints.
    The rest of the regressions listed seem to actually be working correctly and I'm unable to repro problems with them, and they aren't reporting any errors in the regression report. For example, I've confirmed that the regression reported for ALTER TABLE inhx add constraint foo CHECK (xx = 'text'); actually executes correctly without error now, while on main it returns the error: ALTER TABLE with unsupported constraint definition type *tree.AlterTableAddConstraint.
• 867: Release v0.13.0
  Created by the Release workflow to update DoltgreSQL's version
• 863: Added GRANT, REVOKE, Privilege Checking, Ownership, and persisting changes
  This adds the majority of the core functionality that was missing for users, authorization, and privilege checking. This is missing rigorous testing, but that will be added in a separate PR (both engine and bats). All changes that will accompany those tests (such as missing statement support) will also be added in separate PRs.
• 845: add create domain and domain usage support
  Depends on: dolthub/go-mysql-server#2697
  TODO:
  • add merge functionality for domains
  • creating domain type creates instance of array type of this domain type
  • unique OID generated for each domain created
  • track domain usage in table for dropping domain
  • add domains to pg_type
  • ALTER TABLE (e.g.: modify existing column to domain type column)
  • use domain type as underlying type to another domain

Closed Issues

• 884: db.Ping results in error
• 850: Column default value expression error
• 852: Support set_config() function
• 799: Support ALTER TABLE ... DROP CONSTRAINT
• 800: Support adding unique constraints through ALTER TABLE

0.13.0

Merged PRs

doltgresql

• 862: Foreign key tests
  Depends on dolthub/dolt#8461
• 858: Fix merge bug, add tests
  Fixes #855
  Dolt PR: dolthub/dolt#8459
• 856: fix type for coalesce regression test
• 851: Bug fixes for double-quoted relation names
  While investigating #843, several statements were failing due to double-quoted identifiers. This PR fixes COPY statements and use of nextval() and setval() to properly parse double-quoted identifiers.
• 834: Label contribution PRs
  Adds a GitHub Actions workflow to label PRs from outside the team with the "contribution" label. This is the same workflow we use for dolt/go-mysql-server/etc.
• 833: Test and fix for detached head error for pg_catalog tables
  Dolt PR: dolthub/dolt#8434
• 832: Bug fixes for DESCRIBE and implemented SHOW TABLES
  Fixes #830
• 829: Support for using TEXT fields in secondary indexes and UNIQUE constraints
  Allows TEXT columns in Doltgres to work when used in secondary indexes and in unique constraints. PostgreSQL does not have a concept of prefix lengths and Dolt requires prefix lengths for TEXT columns. This now works in Doltgres because a new Doltgres analyzer rule implicitly fills in a default prefix length in these cases.
  Depends on:
  • dolthub/go-mysql-server#2703
• 828: Support for DESCRIBE with AS OF
• 824: Add information_schema tests for checkout and detached head
• 820: Unskip tests for dolt_revert
  Stacked on #818
• 818: Support for per-table AS OF clauses
• 816: Unskip COPY ... DELIMITER test
• 815: Tests for dolt_history table
• 813: Added CREATE USER, ALTER USER, DROP USER
  This adds support for CREATE USER, ALTER USER, and DROP USER. This also adds the related ROLE statements too, since the USER statements are essentially aliases for them. Next step will be to look into persistence and bats testing.
• 812: fix: fix slice init length
  The intention here should be to initialize a slice with a capacity of length rather than initializing the length of this slice.
  The online demo: https://go.dev/play/p/q1BcVCmvidW
• 809: Unskip dolt_patch test
  Dolt PR: dolthub/dolt#8415
• 804: add column check constraint support for create table
  depends on: dolthub/go-mysql-server#2683
• 801: Fix diff functions, return schema.table for table_name columns
  Dolt PR: dolthub/dolt#8411
• 790: Support for setting custom session settings in user namespaces
  e.g.:

  SET mynamespace.var to 'true';
  SHOW mynamespace.var;
  SELECT current_setting('mynamespace.var');

  Postgres allows any configuration parameter to be set this way as long as it's not in the default (empty) namespace.
  This functionality lets us exercise Dolt engine tests that have user variables in them.
• 789: Add support for ALTER COLUMN TYPE, DEFAULT, and NOT NULL statements
  Depends on:
  • dolthub/go-mysql-server#2681
  • dolthub/vitess#367
• 788: Diff tests for changes in multiple schemas
• 787: add functions translate and quote_ident and add pg_operator table
  Also:
  • Removed pg_stat_io view as it is supported for Postgres versions 16 and later.
  • Fixed parsing date type input - fixes #773
• 786: Add diff tests for single schema
• 785: ALTER TABLE support for ADD COLUMN, DROP COLUMN, and RENAME COLUMN
• 783: Test fixes for merge and related operations
• 782: Release v0.12.0
  Created by the Release workflow to update DoltgreSQL's version
• 779: enable using postgres unix socket file path
• 778: Feature: delimiter support for COPY FROM STDIN
  Allows the DELIMITER copy option to be specified for CSV and TEXT imports.
  Also adds initial support for the legacy copy option syntax described in the Compatibility section of COPY docs (e.g. COPY <table> FROM STDIN CSV, HEADER; instead of COPY <table> FROM STDIN WITH (FORMAT CSV, HEADER true);).
• 775: Added RFC5802 for user client authentication
  For the most part, this implements the following portions of RFC 5802:
  • Section 2.2
  • Section 3
  • Validation portion of Section 5
    This also bases the stored data from:
  • https://www.postgresql.org/docs/15/catalog-pg-authid.html
    This PR finally lets the server do the full authentication routine with the client. For now, I've created a mock database full of mock users, which are actually being tested by all of our engine tests since we do supply a username and password when creating and setting up the server. The pgx library is handling the client-side authentication for these tests.
    The next step is to handle basic CREATE USER and DROP USER statements. With those in, I'll create a full battery of tests (bats tests, unit tests, dedicated engine tests, etc.). Unit tests are not included in this PR since I may make further changes with the next PR, and I'd prefer to do a big testing pass at once since none of this is in production just yet.
    After the aforementioned statements and testing, I'll move on to designing the storage portion, since the mock user and mock database are stand-ins for an actual implementation.
• 746: fix value types used for datetime types and add timezone function

Closed Issues

• 855: dolt_merge stomps working set
• 844: Not picking up a commit?
• 830: DESCRIBE [table] does not work for tables in other schemas
• 822: information_schema tables do not work in detached head state
• 808: Support DELIMITER in COPY...FROM
• 33: Default socket file is named mysql.sock
• 773: Support PostgreSQL datetime literal formats
• 728: Support socket interface

0.12.0

Merged PRs

doltgresql

• 780: Don't count failure to send standby messages as terminal failures dur…
  …ing replication
• 770: Tests for dolt_schemas
  Dolt PR: dolthub/dolt#8401
• 768: Add workbench tests for docs
  Dolt PR: dolthub/dolt#8398
• 766: More workbench tests for schema, tags, views
• 763: Wrap column default expressions in ParenExprto match MySQL's requirements
  PostgreSQL's syntax does not require column default expressions to be wrapped in parens, but MySQL's does, so when we translate the column default value expressions to the vitess AST, we need to wrap them in parens so that they execute in GMS without triggering an error.
  Fixes #751
• 762: Add support for ALTER TABLE ADD FOREIGN KEY
  Helps with data imports, since it's common to add FKs at the end of an import script.
  The regression below seems to come from the FK being added, but we don't yet support removing an FK, so the drop table call now fails.
  Related to #724
• 761: Implement information_schema views and tables, add as schema
  GMS: dolthub/go-mysql-server#2678
  Dolt: dolthub/dolt#8391
• 752: More unskipped tests
  Better test harness allowing to unskip about half of the dolt merge tests (after various bug fixes in dolt, already in main)
• 747: Smoke tests for dolt diff functions/tables
  Dolt PR: dolthub/dolt#8381
• 742: Fixed panic for some index joins
  Fixes:
  • #730
    For some index joins, the analyzer will create a specific type of plan that creates MySQL ranges rather than Doltgres ranges. It appears as though there may be divergent branches for the join logic, so I attempted to look for the source of the divergence, however I came up short.
    For now, rather than chasing this down and delaying a PR (since Tim needs this fixed asap), we can pass the lookup to the internal Dolt table. This will return incorrect results in some situations, but it won't panic for now, so I'll follow up with a better search through GMS at a later date to merge the index join paths.
• 741: Support automatic transaction management with COPY FROM STDIN
• 740: Add workbench tests for COPY FROM
• 739: Tests for unique indexes
• 736: Expanding tests for the steps in the Doltgres Getting Started Guide
  Added more test coverage over the Doltgres Getting Started Guide and pulled them out into their own file.
• 733: Initial Authentication Protocol
  This implements the initial portion of the authentication protocol.
  Postgres Reference Documentation:
  • https://www.postgresql.org/docs/15/protocol-flow.html
  • https://www.postgresql.org/docs/15/sasl-authentication.html
    Primarily, this implements SASL SCRAM-SHA-256, which appears to be the primary form of authentication used in modern Postgres. It has been built by following the RFC specification:
  • https://datatracker.ietf.org/doc/html/rfc5802
    There are no tests since the implementation is incomplete. It cannot truly be tested until we have passwords and such that it can verify against (as the results must be sent back to the client for verification, so it can't be faked), however I have tested it up through what has been written, and what exists works as it should.
    Surprisingly, there aren't any libraries that we could really leverage for this. Most SASL libraries don't implement SCRAM. The closest was the following:
  • https://codeberg.org/mellium/sasl
    However, I couldn't really find a way to integrate it using raw messages and the eventual Doltgres user backend, so this is all custom-written using the RFC as a guideline (along with capturing packets using the regression capture tool to ensure that Postgres follows the RFC's implementation). For now, the logic is hidden behind a bogus parameter check so that the static analyzer is happy, and the next step is to make a mock in-memory database of users and passwords so I can fully test the entire workflow.
• 732: Initial support for ALTER TABLE, starting with adding a primary key
  Adding initial support for converting ALTER TABLE statements. This first iteration only supports ALTER TABLE t ADD PRIMARY KEY (...);.
  Related to #724
• 731: fix return limited row issue
• 727: Enabled use mydb/main without quoting and implemented the IF function
  Most of this PR is changes to the doltgres engine testing harness to make it pass more tests.
  • Correctly validates most DML statements
  • Matches commit hashes more reliably
  • Handles more types
    Also include parser support for unquoted db identifiers with a / in a USE statement, and implements the IF function (apparently a cockroach extension).
• 722: Feature: COPY support for HEADER option
  Adds support for using the HEADER option in COPY statements.
  In this first iteration, we only support specifying HEADER or HEADER true. This form causes the tabular data loader and CSV data loader to skip over the initial, header line in import data. In addition to this form, COPY also supports a HEADER MATCH option, where the server asserts that the columns in the import data exactly match the name and the order of the columns in the destination table.
  (Note: this PR is based off of #700 to help split up the changes to make them easier to review)
• 717: Moved enginetests to their own testing CI workflow, don't use -race
  This works around the problem described here:
  #718
• 711: Fix Regression Tests Workflow
• 707: use pgproto3 for handling server connection messages
• 705: Workbench tests for log, merge, table details
• 700: Feature: COPY FROM STDIN support for CSV files
  Support for loading data via COPY FROM STDIN using CSV data.
• 699: Unskip some engine tests
  dolt_reset now works correctly
• 694: Tests for version control operations on new schemas / tables in new schemas
  Code changes are all in Dolt:
  dolthub/dolt#8343
• 691: Convert ANALYZE statements
  Adds support for converting Postgres' ANALYZE statement for a single table and running it through the GMS SQL engine. There are stills lots of unsupported options in Postgres' ANALYZE statement, but this change allows Doltgres to process the simplest form – where a single table is being analyzed.
  Since it's common to run ANALYZE at the end of data load scripts (example), this change is intended to make it easier to load dumps into Doltgres.
• 689: COPY FROM STDIN support
  Adds support for COPY ... FROM STDIN. When copying from STDIN, the COPY FROM statement starts a multi-message flow between the server and client – the client will send COPY DATA messages until all the data has been sent, and then send a COPY DONE (or COPY FAIL) message to finalize the transfer and let the server process more queries.
  This PR adds a new TabularDataLoader type, with the idea that we can create a DataLoader interface for that when we extend this to add support for loading data from CSV data files, too.
  This PR also depends on a GMS change to allow us to create a new sql.Context instance: dolthub/go-mysql-server#2652
• 686: Added COPY FROM
  This was pulled from:
  • #672
    Separating the COPY FROM portion from the regression tests.
• 681: use ParameterObjectIDs in Parse message for binding variable types
  Note: the prepared tests don't utilize this field in the Parse message, so it needs to extract the binding value types from analyzed plan of the query.
• 680: Prevent premature decay for IN tuple
  This removes IN prematurely decaying, since it's only necessary for index filters. To complement this, I've implemented SplitConjunction and SplitDisjunction, so that they're aware of Doltgres expression types. The GMS versions will see *pgexprs.GMSCast and do nothing, since we actually care about the child, but GMS is unaware of that.
• [677](https://github...

0.11.1

Merged PRs

doltgresql

• 640: current_schema function return non "$user" schema
• 636: add Postgres default access methods in pg_am table
  This allows displaying index information on TablePlus
• 635: More bug fixes for subqueries
  Fixes column resolution problems when using subqueries in joins.
  Partially addresses #554
• 632: Support for IN with a subquery
  Tuples won't work until we have a doltgres tuple type, but should work for everything else.
  New expression type is a combination of InTuple from Doltgres and InSubquery from GMS. GMS code couldn't really be reused usefully.
• 631: Unskip some subquery tests, add more workbench tests, allow cross-database references for regclass
• 624: Bug fix for subqueries
  The context finalizer node was implemented incorrectly in terms of the semantics required for subquery logic to function -- the top-level child was never being considered in the row-prepend process.
  IN clauses with subqueries still do not work, that's next.
• 621: add test for non default schema qualified view
  depends on dolthub/go-mysql-server#2633
• 620: Allow varchar columns as primary keys
  The problem: the varchar type in pgtypes was declared as Text, not Varchar. This triggered validation checks in both GMS and Dolt when such a column was used in a primary key. I originally tried adding additional index / length info in the CREATE TABLE transformation, but with the varchar type being interpreted as Text, there's no combination of params that can satisfy all the validation checks.
  The error before this fix is e.g.

  "ERROR: blob/text column 't' used in key specification without a key length"
  

  Given that we have a lot of validation logic with key prefix lengths related to BLOB / TEXT types, it seems like this type really can't be declared as TEXT, but open to other ideas.
  Collation / charset support is still fake and will need to be fixed, but no worse than it was before.
• 617: use unknown type for string literal
  This PR also adds some build in functions:
  • pg_get_indexdef - parses
  • pg_indexes_size - parses
  • pg_tables_size - parses
  • pg_total_relation_size - parses
• 614: Implement information_schema.schemata
  GMS PR: dolthub/go-mysql-server#2632

Closed Issues

• 41: TablePlus can Connect and run queries but does not show Table Metadata
• 14: TablePlus connection kind of works
• 16: show create table and show tables should not work
• 64: CREATE SCHEMA not yet supported
• 171: current_schema() not yet supported
• 599: querying a schema qualified view from different schema fails

0.11.0

Merged PRs

doltgresql

• 596: Implemented DISCARD
  Fixes #591
• 585: support INTERVAL type and AGE() function
• 584: add more pg functions and extract() function
  Added temporary resolution for handling ENUM type as TEXT type until we support CREATE TYPE statement.
  Added functions:
  • extract - partially works
  • pg_get_function_identity_arguments - parses
  • pg_get_partkeydef - parses
  • pg_postmaster_start_time - parses
• 562: Add partially implemented unnest function
• 560: add some of pg_... functions
  Added functions:
  • array_to_string - works
  • pg_encoding_to_char - parses
  • pg_function_is_visible - parses
  • pg_get_functiondef - parses
  • pg_get_triggerdef - parses
  • pg_get_userbyid - parses
  • pg_get_viewdef - works
  • pg_table_is_visible - works
• 552: Add pg_get_expr for nil values, pg_attrdef table
• 550: unskip and add tests for querying pg_catalog table without schema defined
  Depends on dolthub/dolt#8174
• 544: Implement pg_get_constraintdef function
• 530: Fixed function resolution for NULLs
  We had a workaround to allow function resolution to handle NULL expressions (NULL values worked fine since they still resolved to a non-NULL type). This adds the necessary logic so that we no longer need the workaround, and the NULL expression is properly resolved.
• 526: Implement information_schema.columns for doltgres
  Relies on GMS PR dolthub/go-mysql-server#2600
• 523: Convert all dolt_ procedures to functions so they can return values
  This PR also adds basic support for a Composite type, which needs to be fully fleshed out. Its values are sql.Row and its mysql wire type is Tuple.
  This works, in that you can now run e.g.

  select dolt_commit('-am', 'mymessage');

  I'm looking for feedback on this approach before I spend any more time polishing it up and adding more tests.
  We will also want to support this syntax:

  select * from dolt_commit(...);

  For functions that return composite types, the first syntax will return a single formatted column result, while the latter will return a the same number of columns, in the same types, as the underlying composite type. The latter is probably how most people will use this functionality if they want to examine the result of these function calls. Support for that syntax will be in a future PR.
• 520: Implement ANY/SOME expression
• 519: Fix "char" and array types for system functions and tables
• 517: Add format_type function
• 512: Fix using table names in columns, add skipped test for pg_catalog query
• 508: Add tests for information_schema.columns
  Relies on dolthub/go-mysql-server#2596
• 506: Add byteacat to binary concat
• 502: Add anytextcat and textanycat functions to binary concat operator
• 501: Add regtype type
• 500: Implement binary concat operator for text types
• 497: Add comment information system functions
  These will return an empty string until we support comments
• 495: Fix column name for functions
  Fixes #486
• 493: Fix prepared statements for Name type
• 492: Added regclass and regproc
  This adds regclass and regproc, in addition to everything needed to support their full functionality. The hashing approach we previously used for OIDs were not compatible as-is, and reverse hashing would require extensive modifications to track all side-effects (table renames, etc.) in addition to needing some way to resolve hash collisions, so an approach similar to the one I proposed in the OID document was used.
  OIDs are now treated as indexes into arrays, and all database elements (tables, views, etc.) are iterated in a deterministic way, such that we can imagine that all tables are just a single large array containing every table from every schema, all views are a single large array, etc. This let's us get a definitive OID for any database element, and also when given any OID, we are able to grab the relevant database element.
  This is done by splitting an OID's bits into three sections.
  • Section: This is a header that specifies the search space. For example, tables have their own search space since we can iterate over all of them using database.GetTableNames and database.GetTableInsensitive. Each item has its own section, since they all have their own search spaces.
  • SchemaIndex: This is the index of the relevant schema. We order schemas by sorting them by their names (ascending), so the index gives us exactly the schema to use.
  • DataIndex: This is the index of whatever data the section represents. Some elements exist across multiple elements, such as the set of all indexes. To represent an index for those, we sort all tables by their name (ascending) and iterate over each table's indexes as though it were a single contiguous array.
    The size of each bit section is malleable, so I've made some guesses at what sensible starting values may be. In addition, I've disabled the ability to save tables with regclass and regproc, even though Postgres officially supports them. Since we do not yet have a permanent OID solution, I felt that it wouldn't be wise to allow persisting such values to disk, since changing the OID algorithm would break all such databases.
• 487: Add more information_schema tests
  Depends on dolthub/go-mysql-server#2590
• 484: Fix version function so that it returns Postgres version
  Relies on GMS PR: dolthub/go-mysql-server#2588
• 482: /.github/scripts/performance-benchmarking/get-postgres-doltgres-job-json.sh: add additional tests to results
• 481: Added IN to indexes
  select_random_points is performing slower than expected, and this is due to IN not using indexes. This just adds IN to the index logic, so that select_random_points will now actually use indexes. This logic does not properly reflect how PostgreSQL should really behave, but currently none of the index logic behaves properly due to GMS requiring exact type matches, rather than relying on comparison operators, so there's no additional harm in adding this.
  No new tests are added, as we already have pre-existing tests for IN using indexes. They just didn't actually use the index, but still returned the correct result, and now they're using the index as well.
• 475: Added function support for polymorphic types
  This adds the types anyelement and anynonarray, in addition to adding support for defining functions that use polymorphic types.
  https://www.postgresql.org/docs/15/extend-type-system.html#EXTEND-TYPES-POLYMORPHIC
  Besides the changes to function resolution, this also adds a new parameter so that functions can see which types are passed in. This also cleans up the function interface, so that function definitions are a little more consistent and slightly less prone to typos (such as changing function parameters to an array from a slice, so that the length must be defined).
• 471: Release v0.10.0
  Created by the Release workflow to update DoltgreSQL's version
• 461: Fixed quoting in enginetest queries
  Automatically fix quoting in engine test query strings
• 440: Update get-postgres-doltgres-job-json.sh

Closed Issues

• 591: Support DISCARD statements
• 590: Support INTERVAL type
• 513: Should be able to select from pg_catalog tables without specifying pg_catalog
• 442: select and drop on tables with the same name as pg_catalog table name without schema qualifier refers to pg_catalog table by default
• 281: using table-qualified column name in SELECT stmt doesn't work
• 511: Should be able to specify table name in column in SELECT
• 486: SELECT function() returned column name is incorrect
• 443: Brok...

0.10.0

Merged PRs

doltgresql

• 468: Add pg_constraint table, fix index names
• 463: Workaround for current_database and current_schema
• 462: Unskip pg_database order by tests
• 460: Add OIDs to the rest of the partially implemented pg_catalog tables
• 459: Generate OIDs for pg_namespace and pg_class
• 454: add pg_views table to pg_catalog and views displayed on \d psql cmd
  Depends on dolthub/dolt#8086
• 452: Fixed secondary indexing
  • Companion PR: dolthub/dolt#8081
    The companion PR fixes two issues regarding the creation of the secondary index map, which resulted in the now-unskipped tests either panicking or returning incorrect results. With the merger of the aforementioned PR, both of these issues will be fixed.
• 450: Fixed panic during handling of another panic
  Also improved the error output when attempting to use geometry types
• 449: Update get-postgres-doltgres-job-json.sh
• 447: add current_schemas(bool) function
• 445: Implement part of pg_tables
• 441: add pg_namespace names
• 438: pg_attribute and pg_class should only show data for current database
• 436: Fixed some index issues and IN comparisons
  Companion PR: dolthub/dolt#8073
  Created indexes in Doltgres were not actually added to the table, which was fixed in the above PR. In addition, IN was corrected to use comparison functions (IN predates their creation) and SUM was excluded from the list of cast functions since GMS does not search children when looking for certain aggregate functions.
  In addition, IN was overhauled so that comparison functions are resolved once. More details may be found in the accompanying comments.
• 434: Add some pg_index and pg_indexes columns
• 432: Implement some pg_class and pg_attribute columns
• 426: use table_schema from information_schema.tables for schema name
  of \d and \dt commands. This allows these commands not to display pg_catalog tables
  Depends on dolthub/go-mysql-server#2569 and dolthub/dolt#8062
• 424: Ignore new performance_schema database
  Updating doltgresql from changes to GMS in dolthub/go-mysql-server#2568
• 422: Add the rest of the pg_catalog table schemas
• 419: Enable partial index support
  Companion PR: dolthub/go-mysql-server#2566
  This enables partial index support for a constrained set of comparisons.
• 414: Enginetest stubs
  All the dolt enginetests, all skipped for now. The harness works, but no single test method passes entirely, so they're all skipped.
  Next steps:
  • Document categories of test failures for things that should work on Postgres but don't, prioritize work to fix
  • Document categories of test failures for things that don't work on Postgres, translate tests
• 411: add all pg_catalog views as tables
• 410: Even even more pg_catalog table schemas with empty row iters
• 408: Even more pg_catalog table schemas with empty row iters
• 407: add partially populated pg_catalog.pg_type table
• 406: More pg_catalog table schemas with empty row iters
• 405: Add more pg_catalog schemas
• 404: Add schemas with empty row iters for some pg_catalog tables
  Moves over already defined schemas from this PR.
  Includes pg_am, pg_attribute, pg_class, pg_constraint, pg_event_trigger, pg_index, pg_namespace, pg_proc, pg_trigger, and pg_type
• 403: : update doltgres/postgres harness to support context methods
• 400: Implement partial pg_database
  Adds the full pg_catalog.pg_database schema with a few populated columns
• 399: Implement current_schema, current_catalog and current_database functions
  Will follow up with the corresponding table functions in another PR
• 398: add null checks in type cast methods
• 395: Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2
  Bumps github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2.
  Changelog

  Sourced from github.com/jackc/pgx/v4's changelog.

  4.18.2 (March 4, 2024)

  Fix CVE-2024-27289

  SQL injection can occur when all of the following conditions are met:

  1. The non-default simple protocol is used.
  2. A placeholder for a numeric value must be immediately preceded by a minus.
  3. There must be a second placeholder for a string value after the first placeholder; both must be on the same line.
  4. Both parameter values must be user-controlled.

  Thanks to Paul Gerste for reporting this issue.

  Fix CVE-2024-27304

  SQL injection can occur if an attacker can cause a single query or bind message to exceed 4 GB in size. An integer overflow in the calculated message size can cause the one large message to be sent as multiple messages under the attacker's control.

  Thanks to Paul Gerste for reporting this issue.

  • Fix *dbTx.Exec not checking if it is already closed

  Commits

  • 14690df Update changelog
  • 779548e Update required Go version to 1.17
  • 80e9662 Update github.com/jackc/pgconn to v1.14.3
  • 0bf9ac3 Fix erroneous test case
  • f94eb0e Always wrap arguments in parentheses in the SQL sanitizer
  • 826a892 Fix SQL injection via line comment creation in simple protocol
  • 7d882f9 Fix *dbTx.Exec not checking if it is already closed
  • 1d07b8b go mod tidy
  • See full diff in compare view
  
  [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
  Dependabot commands and options

  
  You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any m...

0.9.0

Merged PRs

doltgresql

• 368: /.github/workflows/cd-release.yaml: fix sed command for server version
• 365: add smoke tests for perl, php and ruby
• 362: Perform function analysis once
  Functions did their entire overload resolution during Eval, meaning if a function was called 1000 times, then we did overload resolution 1000 times for the exact same function. This changes it so that we only do it once. This optimization was made while investigating our performance benchmarks.
  Of note, errors found during overload resolution are stashed for later. This ensures that other steps in the pipeline retain their error priority. Previously, when this was all done in Eval, stashing wasn't needed.
• 355: Added correct timeouts to server
  Fixes #348
  Idle connections were timing out in 60s before this change.
• 350: Reworked config
  This change reworks how the data dir, config file and related initialization values are loaded. The main effect is that it's now possible to run doltgres without a config.yaml file.
  The sever config is now resolved in this order
  1. --config flag
  2. config.yaml file if present
  3. Built-in default config values
     The data-dir for this config is then overridden as necessary with a similar pattern:
  4. --data-dir flag
  5. The value in a config.yaml file if, if a file was loaded
  6. The env var DOLTGRES_DATA_DIR
  7. The default ($home/doltgres/databases)
• 343: support for drop table without a schema name
  This PR has the tests. The fix for the behavior is in dolthub/dolt#7952
• 339: Release v0.8.0
  Created by the Release workflow to update DoltgreSQL's version

Closed Issues

• 348: disabling the session after executing the request

0.8.0

Merged PRs

doltgresql

• 336: Added string literal to array interpretation
  This adds functionality so that string literals such as '{1,2,3}' can be interpreted as arrays of any type. This builds off of the #333 PR, which enables this to "just work" for all contexts (includes casting, inserting into columns, automatic operator resolution, etc.). This also includes additional tests for a few bugs that weren't caught in the previous PR.
• 333: Removed sql.Convert usage, moved everything to casts
  This removes our reliance on sql.Convert and changed everything to make use of casts. Casts have three contexts: explicit (value::type), assignment (INSERT and UPDATE statements), and implicit. In addition to moving to casts, I've also added assignment casts.
  This move also comes with a massive change to how function overloads are handled. Overload resolution is handled through available casts, so by removing the hacky behavior that was in place, we're now able to unskip a number of tests that just could not be handled through the previous logic. There were other tests that were also incorrect, and those have been fixed.
  Overall, this has resulted in a fairly substantial decrease in code throughout the codebase, in addition to allowing for types to finally have "full" support.
  Companion PR:
  • dolthub/go-mysql-server#2523
• 325: Update get-doltgres-correctness-job-json.sh
• 320: Update schema path in scripts
• 319: Added comparisons for JSONB
  This adds comparisons for jsonb. In the original JSON PR, I mistakenly only checked if json had comparisons, and that type does not. jsonb, however, does have comparisons, so this implements those. In addition, I've added a massive collection of rows that it properly sorts over. Test results were lifted straight from a Postgres 15 server to ensure accuracy.
• 317: /utils/doltgres_builder/run.go: fix doltgres builder paths
• 302: Added JSON types
  This adds JSON types, and also fixes a bug where the text types would incorrectly persist an empty string as a NULL.
• 288: Release v0.7.5
  Created by the Release workflow to update DoltgreSQL's version
• 287: Added SERIAL types along with remaining SEQUENCE functionality
  Companion PRs:
  • dolthub/vitess#347
  • dolthub/go-mysql-server#2504
    This adds the SERIAL types, along with the remaining functionality for sequences that was not present in the first PR. In addition, this adds our first uses of injected analyzer steps, and introduces custom nodes used for the sequence statements (instead of stored procedures).
• 269: New tests for schema support
  Also changes expected errors to match an error string. Not all tests are passing yet.
• 185: Update README.md

Closed Issues

• 280: Release 0.7.4 for windows not Working
• 90: doltgres version shows the previous version
• 279: Doltgresql does not yet support PostgreSQL's JDBC connection