v0.31.0-rc.1

v0.31.0-rc.1 is the second release candidate of the upcoming v0.31.0.

$ docker pull graphprotocol/graph-node:v0.31.0-rc.1

Edit 2023-06-08: v0.31.0-rc.1 has been promoted to v0.31.0.

v0.31.0

$ docker pull graphprotocol/graph-node:v0.31.0

What's new

• Fulltext searches can now be combined with where filtering, further narrowing down search results. #4442
• Derived fields loader support. This feature requires subgraph authors to upgrade graph-cli to v0.51.0. #4434, graph-cli release notes
• Tweaked how RPC provider limiting rules are interpreted from configurations. In particular, node IDs that don't match any rules of a provider won't have access to said provider instead of having access to it for an unlimited number of subgraphs. Read the docs for more information. #4353
• Introduced WASM host function store.get_in_block, which is a much faster variant of store.get limited to entities created or updated in the current block. #4540
• The subgraph_deploy JSON-RPC method now accepts a history_blocks parameter, which indexers can use to set default amounts of history to keep. #4564
• IPFS requests for polling file data sources are not throttled anymore (also known as concurrency or burst limiting), only rate-limited. #4570, #4649
• Exponential requests backoff when retrying failed subgraphs is now "jittered", smoothing out request spikes. #4476
• Indexers can use thee new feature GRAPH_EXPERIMENTAL_SUBGRAPH_SETTINGS to define matching rules on subgraph names for reducing amounts of history stored. #4633
• RPC provider responses that decrease the chain head block number (non-monotonic) are now ignored, increasing resiliency against inconsistent provider data. #4354
• It's now possible to to have a Firehose-only chain with no RPC provider at all in the configuration. #4508, #4553
• The materialized views in the info schema (table_sizes, subgraph_sizes, and chain_sizes) that provide information about the size of various database objects are now automatically refreshed every 6 hours. #4461
• Subgraph error logs now contain the transaction hash when available. #4635
• Adapter selection now takes error rates into account, preferring adapters with lower error rates. #4468
• Removed support for GRAPH_ETHEREUM_IS_FIREHOSE_PREFERRED, REVERSIBLE_ORDER_BY_OFF, and GRAPH_STORE_CONNECTION_TRY_ALWAYS env. variables. #4375, #4436
• New env. var. GRAPH_ETH_CALL_NO_GAS which is a comma-separated list of chains' net_version for which to disable eth_call gas limits. The default value (421613) disables gas limits for Arbitrum Goerli. #4609

Performance improvements

• Batched writes: write operations to the database when indexing subgraphs are now accumulated into in-memory batches before persisting them, resulting in noticeably faster indexing and lower database CPU load. In case of unexpected issues, this feature can be turned off setting GRAPH_STORE_WRITE_BATCH_SIZE=0. #4606, #4632
• The entity cache that graph-node keeps around is much more efficient, meaning more cache entries fit in the same amount of memory resulting in a performance increase under a wide range of workloads. #4485, #4624
• Indexing speed improvements for subgraphs with more than 10k data sources. #4631
• BRIN indexes in PostgreSQL now use ..._minmax_multi_ops instead of _minmax_ops if available (PostgreSQL 14+), often resulting in more selective indexes. #4629
• GIN indexes are not created anymore for numeric arrays. #4628

Substreams

• The substreams protocol has been updated to sf.substreams.rpc.v2.Stream/Blocks. #4556
• Added support for adapter balancing a.k.a. provider failover to substreams. #4578
• Fixed faulty startBlock selection logic in substreams. #4463
• Several bug fixes were applied after breakage caused by an internal refactor of how entities are stored in-memory. #4604, #4572, #4612, #4664

Bug fixes

• Fixed a bug that would cause subgraphs to fail with a subgraph writer poisoned by previous error message following certain database errors. #4533
• Fixed a bug that would cause subgraphs to fail with a store error: no connection to the server message when database connection e.g. gets killed. #4435
• The subgraph_reassign JSON-RPC method doesn't fail anymore when multiple deployment copies are found: only the active copy is reassigned, the others are ignored. #4395
• Fixed a bug that would cause on_sync handlers on copied deployments to fail with the message Subgraph instance failed to run: deployment not found [...]. #4396
• Fixed a bug that would cause the copying or grafting of a subgraph while pruning it to incorrectly set earliest_block in the destination deployment. #4502
• Handler timeouts would sometimes be reported as deterministic errors with the error message Subgraph instance failed to run: Failed to call 'asc_type_id' with [...] wasm backtrace [...]; this error is now nondeterministic and recoverable. #4475
• Fixed faulty exponential request backoff behavior after many minutes of failed requests, caused by an overflow. #4421
• json.fromBytes and all BigInt operations now require more gas, protecting against malicious subgraphs. #4594, #4595
• Fixed a Rust panic on program startup when mixing Firehose and RPC providers. #4680

Graphman

• graphman rewind now requires block-number and block-hash to be passed as flags instead of arguments. #4400
• You can now use the new flag --start-block in graphman rewind to rewind a subgraph to the startBlock set in manifest, or to the genesis block if no startBlock is set. #4400
• The behavior for graphman prune has changed: running just graphman prune will mark the subgraph for ongoing pruning in addition to performing an initial pruning. To avoid ongoing pruning, use graphman prune --once (docs). #4429
• The env. var. GRAPH_STORE_HISTORY_COPY_THRESHOLD –which serves as a configuration setting for graphman prune– has been renamed to GRAPH_STORE_HISTORY_REBUILD_THRESHOLD. #4505
• You can now list all existing deployments via graphman info --all. #4347
• The command graphman chain call-cache remove now requires --remove-entire-cache as an explicit flag, protecting against accidental destructive command invocations. #4397
• graphman copy create accepts two new flags, --activate and --replace, which make moving of subgraphs across shards much easier. #4374
• The log level for graphman is now set via GRAPHMAN_LOG or command line instead of GRAPH_LOG. #4462
• graphman reassign now emits a warning when it suspects a typo in node IDs. #4377

Metrics and logging

• Subgraph syncing time metric deployment_sync_secs now stops updating once the subgraph has synced. #4489
• New endpoint_request metric to track error rates of different providers. #4490, #4504, [#4430](...

v0.31.0-rc.0

v0.31.0-rc.0 is the first release candidate of the upcoming v0.31.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.31.0-rc.0

Edit 2023-06-08: v0.31.0-rc.0 is obsolete; v0.31.0-rc.1 is the latest release candidate.

v0.30.0-rc.0

v0.30.0-rc.0 is the first release candidate for the upcoming v0.30.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.30.0-rc.0

Edit 2023-03-01: v0.30.0-rc.0 has been promoted to v0.30.0.

v0.30.0

$ docker pull graphprotocol/graph-node:v0.30.0

Database locale change

New graph-node installations now mandate PostgreSQL to use C locale and UTF-8 encoding. The official docker-compose.yml template has been updated accordingly. Pre-existing graph-node installations are not concerned with this change, but local development scripts and CI pipelines may have to adjust database initialization parameters. This can be done with initdb -E UTF8 --locale=C. #4163, #4151, #4201, #4340

What's new

• AND/OR filters. AND/OR logical operators in where filters have been one of graph-node's most awaited features. They do exactly what you would expect them to do, and are very powerful. #579, #4080, #4171
• IPFS file data sources. IPFS file data sources allow subgraph developers to query offchain information from IPFS directly in mappings. This feature is the culmination of much community and development efforts (GIP here). A future iteration of this feature will also include a so-called "Availability Chain", allowing IPFS file data sources to contribute to Proofs of Indexing. At the moment, entity updates that originate from these data sources' handlers do not contribute to PoIs. #4147, #4162, and many others!
• Sorting by child entities (a.k.a. nested sorting). You can now orderBy properties of child entities. #4058, #3737, #3096
• Added support for a Firehose-based block ingestor. Indexers that use the new Firehose-based block ingestor cannot automatically switch back to RPC. In order to downgrade, indexers must manually delete all blocks accumulated by Firehose in the database. For this reason, we suggest caution when switching over from RPC to Firehose. #4059, #4204, #4216
• Fields of type Bytes can now use less than and greater than filters. #4285
• "userinfo" is now allowed in IPFS URLs (e.g. https://foo:bar@example.com:5001/). #4252
• The default for GRAPH_IPFS_TIMEOUT is now 60 seconds instead of 30. #4324
• Forking options can now be set via env. vars. (GRAPH_START_BLOCK, GRAPH_FORK_BASE, GRAPH_DEBUG_FORK). #4308
• Allow retrieving GraphQL query tracing over HTTP if the env. var. GRAPH_GRAPHQL_TRACE_TOKEN is set and the header X-GraphTraceQuery is included. The query traces' JSON is the same as returned by graphman query. #4243
• Lots of visual and filtering improvements to #4232
• More aggressive in-memory caching of blocks close the chain head, potentially alleviating database load. #4215
• New counter Prometheus metric query_validation_error_counter, labelled by deployment ID and error code. #4230
  graph_elasticsearch_logs_sent
• Turned "Flushing logs to Elasticsearch" log into a Prometheus metric (graph_elasticsearch_logs_sent) to reduce log noise. #4333
• New materialized view info.chain_sizes, which works the same way as the already existing info.subgraph_sizes and info.table_sizes. #4318
• New graphman stats subcommands set-target and target to manage statistics targets for specific deployments (i.e. how much data PostgreSQL samples when analyzing a table). #4092

Fixes

• graph-node now has PID=1 when running inside the official Docker image. #4217
• More robust ipfs.cat logic during grafted subgraphs' manifest lookup. #4284
• Fixed a bug that caused some large multi-entity inserts to fail because of faulty chunk size calculation. #4250
• Subgraph pruning now automatically cancels ongoing autovacuum, to avoid getting stuck. #4167
• ens.getNameByHash now fails nondeterministically if ENS rainbow tables are not available locally. #4219
• Some kinds of subgraph failures were previously wrongly treated as unattestable (value parsing, enum and scalar coercion), i.e. nondeterministic. These subgraph failure modes are now flagged as fully-deterministic. #4278

v0.29.0

$ docker pull graphprotocol/graph-node:v0.29.0

Upgrade notes

• This release includes a determinism fix that affect a very small number of subgraphs on the network (we counted 2): if a subgraph manifest had one data source with no contract address, listening to the same events or calls of another data source that has a specified address, then the handlers for those would be called twice. After the fix, this will happen no more, and the handler will be called just once like it should.

  Affected subgraph deployments:

  • Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  • Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg

  Here's an example manifest, taking a look at the data sources of name ERC721 and CryptoKitties, both listen to the Transfer(...) event. Considering a block where there's only one occurrence of this event, graph-node would duplicate it and call handleTransfer twice. Now this is fixed and it will be called only once per event/call that happened on chain.

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0xd75c2a2412396b6b2b387323d84b9c19313dbba35ccb1493d94976fedb0b3a23 5774644 Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  graphman rewind 0xf23c45dd2e1ed77fce1919cdcc9df2de51e821763e9691f8adbb79a55ddb28a4 3914495 Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg
  

  See #4055 for more information.

• This release fixes another determinism bug that affects a handful of subgraphs. The bug affects all subgraphs which have an apiVersion older than 0.0.5 using call handlers. While call handlers prior to 0.0.5 should be triggered by both failed and successful transactions, in some cases failed transactions would not trigger the handlers. This resulted in nondeterministic behavior. With this version of graph-node, call handlers with an apiVersion older than 0.0.5 will always be triggered by both successful and failed transactions. Behavior for apiVersion 0.0.5 onward is not affected.

  The affected subgraphs are:

  • QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  • QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  • QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  • QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  • QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  • Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0x8aed5a76735b0890d4c0a9e752c88cba4f3d9d0e4e1dad70d6bd913cc279e4b8 7913573 QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  graphman rewind 0x15da49d2177dde3012b67f12d25d9ae4cb56c3638c2ec011298e509be27be77a 7303699 QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  graphman rewind 0x1be8b52bd6b3c29b1ead63f6a8bef83d626d8b6795f6c2d89c302cbe32d59830 10613640 QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  graphman rewind 0xef547ca70b451cc9fedbc039edcfd4a35de80fce76ed6ad31bc15a47850dfefd 10736242 QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  graphman rewind 0x8fb3744b077831856203bbfe465e9cb681d0ed0c67db297faee54f0542f045d3 5533890 Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K
  
  # Note: the following subgraph (https://github.com/Data-Nexus/721-Marketplace) is extremely
  # write-heavy, and your node may OOM while rewinding. If you run into this issue suggest
  # you either rewind in smaller chunks of blocks, or simply redeploy it.
  graphman rewind 0xf9bc13c7864c997aaed501715a0627a25e748c2df4889cc9cbf4d077e7bca34a 4944642 QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  

  See #4149 for more information.

What's new

• Grafted subgraphs can now add their own data sources. #3989, #4027, #4030
• Add support for filtering by nested interfaces. #3677
• Add support for message handlers in Cosmos #3975
• Dynamic data sources for Firehose-backed subgraphs. #4075
• Various logging improvements. #4078, #4084, #4031, #4144, #3990
• Some DB queries now have GCP Cloud Insight -compliant tags that show where the query originated from. #4079
• New configuration variable GRAPH_STATIC_FILTERS_THRESHOLD to conditionally enable static filtering based on the number of dynamic data sources. #4008
• New configuration variable GRAPH_STORE_BATCH_TARGET_DURATION. #4133

Docker image

• The official Docker image now runs on Debian 11 "Bullseye". #4081
• We now ship envsubst with the official Docker image, allowing you to easily run templating logic on your configuration files. #3974

Graphman

We have a new documentation page for graphman, check it out here!

• Subgraph pruning with graphman! #3898, #4125, #4153, #4152, #4156, #4041
• New command graphman drop to hastily delete a subgraph deployment. #4035
• New command graphman chain call-cache for clearing the call cache for a given chain. #4066
• Add --delete-duplicates flag to graphman check-blocks by @tilacog in #3988

Performance

• Restarting a node now takes much less time because postgres_fdw user mappings are only rebuilt upon schema changes. If necessary, you can also use the new commands graphman database migrate and graphman database remap to respectively apply schema migrations or run remappings manually. #4009, #4076
• Database replicas now won't fall behind as much when copying subgraph data. #3966 #3986
• Block handlers optimization with Firehose >= 1.1.0. #3971
• Reduced the amount of data that a non-primary shard has to mirror from the primary shard. #4015
• We now use advisory locks to lock deployments' tables against concurrent writes. #4010

Bug fixes

• Fixed a bug that would cause some failed subgraphs to never restart. #3959
• Fixed a bug that would cause bad POIs for Firehose-backed subgraphs when processing CREATE calls. #4085
• Fixed a bug which would cause failure to redeploy a subgraph immediately after deletion. #4044
• Firehose connections are now load-balanced. #4083
• Determinism fixes. See above. #4055, #4149

Dependency updates

Dependency	updated to
anyhow	1.0.66
base64	0.13.1
clap	3.2.23
env_logger	0.9.1
iana-time-zone	0.1.47
itertools	0.10.5
jsonrpsee	0.15.1
num_cpus	1.14.0
openssl	0.10.42
pretty_assertions	1.3.0
proc-macro2	1.0.47
prometheus	0.13.3
protobuf-parse	3.2.0
semver	1.0.14
serde_plain	1.0.1
sha2	0.10.6
structopt	removed
tokio-stream	0.1.11
tokio-tungstenite	0.17.2
tower-test	d27ba65
url	2.3.1

v0.29.0-rc.0

v0.29.0-rc.0 is the first release candidate for the upcoming v0.29.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.29.0-rc.0

Edit 2022-12-06: v0.29.0-rc.0 has been promoted to v0.29.0.

v0.28.2

Indexers are advised to migrate to v0.28.2 and entirely bypass v0.28.0 and v0.28.1.

Fixed a bug which would cause subgraphs to stop syncing under some graph-node deployment configurations. #4046, #4051

v0.28.1

Yanked. Please migrate to v0.28.2.

v0.28.0

This is the v0.28.0 release.

Upgrade notes

• New DB table for dynamic data sources.
  For new subgraph deployments, dynamic data sources will be recorded under the sgd*.data_sources$ table, rather than subgraphs.dynamic_ethereum_contract_data_source. As a consequence new deployments will not work correctly on earlier graph node versions, so downgrading to an earlier graph node version is not supported.
  See issue #3405 for other details.

What's new

• The filepath which "too expensive qeueries" are sourced from is now configurable. You can use either the GRAPH_NODE_EXPENSIVE_QUERIES_FILE environment variable or the expensive_queries_filename option in the TOML configuration. #3710
• The output you'll get from graphman query is less cluttered and overall nicer. The new options --output and --trace are available for detailed query information. #3860
• docker build will now --target the production build stage by default. When you want to get the debug build, you now need --target graph-node-debug. #3814
• Node IDs can now contain any character. The Docker start script still replaces hyphens with underscores for backwards compatibility reasons, but this behavior can be changed with the GRAPH_NODE_ID_USE_LITERAL_VALUE environment variable. With this new option, you can now seamlessly use the K8s-provided host names as node IDs, provided you reassign your deployments accordingly. #3688
• You can now use the conn_pool_size option in TOML configuration files to configure the connection pool size for Firehose providers. #3833
• Index nodes now have an endpoint to perform block number to canonical hash conversion, which will unblock further work towards multichain support. #3942
• _meta.block.timestamp is now available for subgraphs indexing EVM chains. #3738, #3902
• The deployment_eth_rpc_request_duration metric now also observes eth_getTransactionReceipt requests' duration. #3903
• New Prometheus metrics query_parsing_time and query_validation_time for monitoring query processing performance. #3760
• New command graphman config provider, which shows what providers are available for new deployments on a given network and node. #3816
  E.g. $ graphman --node-id index_node_0 --config graph-node.toml config provider mainnet
• Experimental support for GraphQL API versioning has landed. #3185
• Progress towards experimental support for off-chain data sources. #3791
• Experimental integration for substreams. #3777, #3784, #3897, #3765, and others

Bug fixes

• graphman stats now complains instead of failing silently when incorrectly setting account-like optimizations. #3918
• Fixed inconsistent logic in the provider selection when the limit TOML configuration option was set. #3816
• Fixed issues that would arise from dynamic data sources' names clashing against template names. #3851
• Dynamic data sources triggers are now processed by insertion order. #3851, #3854
• When starting, the Docker image now replaces the bash process with the graph-node process (with a PID of 1). #3803
• Refactor subgraph store tests by @evaporei in #3662
• The ethereum_chain_head_number metric doesn't get out of sync anymore on chains that use Firehose. #3771, #3732
• Fixed a crash caused by bad block data from the provider. #3944
• Fixed some minor Firehose connectivity issues via TCP keepalive, connection and request timeouts, and connection window size tweaks. #3822, #3855, #3877, #3810, #3818
• Copying private data sources' tables across shards now works as expected. #3836

Performance improvements

• Firehose GRPC stream requests are now compressed with gzip, if the server supports it. #3893
• Memory efficiency improvements within the entity cache. #3594
• Identical queries now benefit from GraphQL validation caching, and responses are served faster. #3759

Other

• Avoid leaking some sensitive information in logs. #3812