Author: @Craig Tutterow, Edge & Node Data Science, craig@edgeandnode.com

Implementation

A team of core developers has produced an oracle that:

1. posts aggregated quality and cost of service data from available gateways to IPFS every 5 minutes
2. posts the IPFS file metadata to Gnosis chain via a DataEdge contract
3. and indexes the IPFS files in a subgraph for consumption by protocol stakeholders

The community can use this data to aggregate quality and cost of service data in different ways and build a variety of analytics and monitoring dashboards related to quality and cost of service at the Indexer and subgraph deployment levels.

Below are details on the published IPFS data

• Chain: gnosis

• DataEdge Contract: 0x5b4293b4c0f36cb5d4448950830bc777759b6c4f (gnosisscan)

• Schema for oracle data edge events (published using a virtual submitQoSPayload function):

  {
  	"topic": string // Topic of the oracle message
    "hash" : string // IPFS hash of the oracle data file in json format
    "timestamp": int // timestamp of the time interval end time in seconds (UTC) 
  }
  

• Current production QoS data topic names:

  • gateway_indexer_attempt_qos_5_minutes_prod_v3 : contains data on quality and cost of service at the Indexer and subgraph level based on individual requests from the gateways to Indexers
    • Fields
      • indexer_wallet: the ETH wallet address of the Indexer
      • indexer_url: the url of the Indexer’s subgraph endpoint
      • subgraph_deployment_ipfs_hash: the IPFS hash of the subgraph deployment
      • chain: the chain that the subgraph indexes
      • gateway_id: the id of the gateway (e.g. mainnet, testnet, mainnet-arbitrum, etc.)
      • start_epoch: the start of a 5 minute period over which the QoS stats were observed (in seconds, UTC time). This is unrelated to protocol epochs
      • end_epoch: the end of a 5 minute period over which the QoS stats were observed (in seconds, UTC time). This is unrelated to protocol epochs
      • avg_query_fee: the average query fee requested by the gateways in queries sent to the Indexer
      • max_query_fee: the max query fee requested by the gateways in queries sent to the Indexer
      • total_query_fees: the total amount of query fees requested from queries that minute
      • query_count: number of queries that minute
      • avg_indexer_latency_ms: the average latency of the Indexer’s query responses in milliseconds
      • max_indexer_latency_ms: the maximum latency of the Indexer’s query responses in milliseconds
      • num_indexer_200_responses: the number of 200 status code responses from queries sent to the Indexer
      • proportion_indexer_200_responses: the proportion of 200 responses of queries sent to the Indexer (between 0 and 1)
        • Note: some 200s may not result in data being returned to client, so shouldn’t be considered as success rate, though the two are highly correlated
      • avg_indexer_blocks_behind: the average number of blocks behind an Indexer was in indexing a subgraph when responding to queries
      • max_indexer_blocks_behind: the maximum number of blocks behind an Indexer was in indexing a subgraph when responding to queries
      • stdev_indexer_latency_ms: the standard deviation in milliseconds of gateway query responses to a client
    • Example file: https://api.thegraph.com/ipfs/api/v0/cat?arg=QmbnsBrKEENmgKKH8a1ZjnSCRqkWVNk4KrJnkrBo8Z4ViV
  • gateway_query_result_qos_5_minutes_prod_v3 : contains data on quality and cost of service at the Indexer and subgraph level from the perspective of the response returned to the client from the gateways (accounts for multiple Indexer attempts and retries from the gateways to produce a valid response)
    • Columns:
      • subgraph_deployment_ipfs_hash: the IPFS hash of the subgraph deployment
      • chain: the chain that the subgraph indexes
      • gateway_id: the id of the gateway (e.g. mainnet, testnet, mainnet-arbitrum, etc.)
      • start_epoch: the start of the 5 minute period over which the QoS stats were observed (in seconds, UTC time). This is unrelated to protocol epochs
      • end_epoch: the end of the 5 minute period over which the QoS stats were observed (in seconds, UTC time). This is unrelated to protocol epochs
      • avg_query_fee: the average query fee requested in queries of the subgraph
      • max_query_fee: the maximum query fee of a query of the subgraph
      • total_query_fees: the total amount of query fees requested in queries of the subgraph that minute
      • query_count: the number of queries of the subgraph
      • most_recent_query_ts: the epoch timestamp (in seconds, UTC time) of the most recent query of the subgraph (accounting for retries to multiple Indexers)
      • gateway_query_success_rate: the success rate of queries in the gateway, net user attributed errors (described below). User attributed errors are counted as successfully handled queries from the gateway’s point of view under this definition, but can be split out using the following field.
      • user_attributed_error_rate: the proportion of queries that resulted in an error attributed to users. Possible reasons for user attributed errors are:
        • client request came from unauthorized domain
        • insufficient GRT in billing account
        • an invalid API key
        • invalid query syntax, or
        • requesting a block prior to the minimum start block specified in the subgraph manifest
      • avg_gateway_latency_ms: the average latency in milliseconds of gateway query responses to a client (accounting for retries to multiple Indexers)
      • max_gateway_latency_ms: the maximum latency in milliseconds of gateway query responses to a client (accounting for retries to multiple Indexers)
      • stdev_gateway_latency_ms: the standard deviation in milliseconds of gateway query responses to a client
    • Example file: https://api.thegraph.com/ipfs/api/v0/cat?arg=QmTnwqRF9qkfz4QVtCm1vRrWdjcSv6G9mpKV7YcRNKPT1s
  • Any subgraph logic should filter using the above topic names to retrieve only data from production pipelines. Staging, test and unrelated pipelines may be deployed periodically under different topic names.
  • Note: This data is based on a number of off-chain dependencies, so is nondeterministic and not guaranteed to be accurate or timely. It is being provided as a public service. Please discount appropriately and contact us on Discord if you notice any issues. Please read the disclaimers at the bottom of the document for more details.

• Example subgraph indexing Oracle IPFS hash payloads (does not index or process underlying data)

  • parsed quality and cost of service subgraph using IPFS files
    • https://thegraph.com/hosted-service/subgraph/graphprotocol/gateway-mips-qos-oracle
    • Github: https://github.com/juanmardefago/gateway-qos-oracle-example-subgraph
  • Shell subgraph
    • https://thegraph.com/hosted-service/subgraph/craigtutterow/gateway-qos-oracle
    • Github: https://github.com/edgeandnode/gateway-qos-oracle-example-subgraph

Architecture

The subgraph oracle is a data pipeline based on the following components:

• Log messages from gateways are streamed to a Redpanda cluster. Redpanda is a high performance, Kafka compatible streaming platform that allows us to process a high volume of log data in a cost efficient manner using internal hardware. At peak, our Redpanda and Materialize instances are handling 300mb per seconds of data throughput.
• Log messages are consumed and aggregated into analytical tables in a Materialize database instance. Materialize is a streaming database that supports Postgres SQL and calculates OLAP results incrementally based on streaming inputs.