Rollup-node P2P interface
The rollup node has an optional peer-to-peer (P2P) network service to improve the latency between the view of sequencers and the rest of the network by bypassing the L1 in the happy case, without relying on a single centralized endpoint.
This also enables faster historical sync to be bootstrapped by providing block headers to sync towards, and only having to compare the L2 chain inputs to the L1 data as compared to processing everything one block at a time.
The rollup node will always prioritize L1 and reorganize to match the canonical chain. The L2 data retrieved via the P2P interface is strictly a speculative extension, also known as the "unsafe" chain, to improve the happy case performance.
This also means that P2P behavior is a soft-rule: nodes keep each other in check with scoring and eventual banning of malicious peers by identity or IP. Any behavior on the P2P layer does not affect the rollup security, at worst nodes rely on higher-latency data from L1 to serve.
In summary, the P2P stack looks like:
- Discovery to find peers: Discv5
- Connections, peering, transport security, multiplexing, gossip: LibP2P
- Application-layer publishing and validation of gossiped messages like L2 blocks.
This document only specifies the composition and configuration of these network libraries. These components have their own standards, implementations in Go/Rust/Java/Nim/JS/more, and are adopted by several other blockchains, most notably the L1 consensus layer (Eth2).
Table of Contents
P2P configuration
Identification
Nodes have a separate network- and consensus-identity.
The network identity is a secp256k1
key, used for both discovery and active LibP2P connections.
Common representations of network identity:
PeerID
: a LibP2P specific ID derived from the pubkey (through protobuf encoding, typing and hashing)NodeID
: a Discv5 specific ID derived from the pubkey (through hashing, used in the DHT)Multi-address
: an unsigned address, containing: IP, TCP port, PeerIDENR
: a signed record used for discovery, containing: IP, TCP port, UDP port, signature (pubkey can be derived) and L2 network identification. Generally encoded in base64.
Discv5
Structure
The Ethereum Node Record (ENR) for an Optimism rollup node must contain the following values, identified by unique keys:
- An IPv4 address (
ip
field) and/or IPv6 address (ip6
field). - A TCP port (
tcp
field) representing the local libp2p listening port. - A UDP port (
udp
field) representing the local discv5 listening port. - An OpStack (
opstack
field) L2 network identifier
The opstack
value is encoded as a single RLP bytes
value, the concatenation of:
- chain ID (
unsigned varint
) - fork ID (
unsigned varint
)
Note that DiscV5 is a shared DHT (Distributed Hash Table): the L1 consensus and execution nodes, as well as testnet nodes, and even external IOT nodes, all communicate records in this large common DHT. This makes it more difficult to censor the discovery of node records.
The discovery process in Optimism is a pipeline of node records:
- Fill the table with
FINDNODES
if necessary (Performed by Discv5 library) - Pull additional records with searches to random Node IDs if necessary
(e.g. iterate
RandomNodes()
in Go implementation) - Pull records from the DiscV5 module when looking for peers
- Check if the record contains the
opstack
entry, verify it matches the chain ID and current or future fork number - If not already connected, and not recently disconnected or put on deny-list, attempt to dial.
LibP2P
Transport
TCP transport. Additional transports are supported by LibP2P, but not required.
Dialing
Nodes should be publicly dialable, not rely on relay extensions, and able to dial both IPv4 and IPv6.
NAT
The listening endpoint must be publicly facing, but may be configured behind a NAT. LibP2P will use PMP / UPNP based techniques to track the external IP of the node. It is recommended to disable the above if the external IP is static and configured manually.
Peer management
The default is to maintain a peer count with a tide-system based on active peer count:
- At "low tide" the node starts to actively search for additional peer connections.
- At "high tide" the node starts to prune active connections, except those that are marked as trusted or have a grace period.
Peers will have a grace period for a configurable amount of time after joining. In emergency, when memory runs low, the node should start pruning more aggressively.
Peer records can be persisted to disk to quickly reconnect with known peers after restarting the rollup node.
The discovery process feeds the peerstore with peer records to connect to, tagged with a time-to-live (TTL). The current P2P processes do not require selective topic-specific peer connections, other than filtering for the basic network participation requirement.
Peers may be banned if their performance score is too low, or if an objectively malicious action was detected.
Banned peers will be persisted to the same data-store as the peerstore records.
TODO: the connection gater does currently not gate by IP address on the dial Accept-callback.
Transport security
Libp2p-noise, XX
handshake, with the the secp256k1
P2P identity, as popularized in Eth2.
The TLS option is available as well, but noise
should be prioritized in negotiation.
Protocol negotiation
Multistream-select 1.0 (/multistream/1.0.0
) is an interactive protocol
used to negotiate sub-protocols supported in LibP2P peers. Multistream-select 2.0 may be used in the future.
Identify
LibP2P offers a minimal identification module to share client version and programming language. This is optional and can be disabled for enhanced privacy. It also includes the same protocol negotiation information, which can speed up initial connections.
Ping
LibP2P includes a simple ping protocol to track latency between connections. This should be enabled to help provide insight into the network health.
Multiplexing
For async communication over different channels over the same connection, multiplexing is used.
mplex (/mplex/6.7.0
) is required, and yamux (/yamux/1.0.0
) is recommended but optional
GossipSub
GossipSub 1.1 (/meshsub/1.1.0
, i.e. with peer-scoring extension) is a pubsub protocol for mesh-networks,
deployed on L1 consensus (Eth2) and other protocols such as Filecoin, offering lots of customization options.
Content-based message identification
Messages are deduplicated, and filtered through application-layer signature verification.
Thus origin-stamping is disabled and published messages must only contain application data,
enforced through a StrictNoSign
Signature Policy
This provides greater privacy, and allows sequencers (consensus identity) to maintain multiple network identities for redundancy.
Message compression and limits
The application contents are compressed with snappy single-block-compression (as opposed to frame-compression), and constrained to 10 MiB.
Message ID computation
Same as L1, with recognition of compression:
- If
message.data
has a valid snappy decompression, setmessage-id
to the first 20 bytes of theSHA256
hash of the concatenation ofMESSAGE_DOMAIN_VALID_SNAPPY
with the snappy decompressed message data, i.e.SHA256(MESSAGE_DOMAIN_VALID_SNAPPY + snappy_decompress(message.data))[:20]
. - Otherwise, set
message-id
to the first 20 bytes of theSHA256
hash of the concatenation ofMESSAGE_DOMAIN_INVALID_SNAPPY
with the raw message data, i.e.SHA256(MESSAGE_DOMAIN_INVALID_SNAPPY + message.data)[:20]
.
Heartbeat and parameters
GossipSub parameters:
D
(topic stable mesh target count): 8D_low
(topic stable mesh low watermark): 6D_high
(topic stable mesh high watermark): 12D_lazy
(gossip target): 6heartbeat_interval
(interval of heartbeat, in seconds): 0.5fanout_ttl
(ttl for fanout maps for topics we are not subscribed to but have published to, in seconds): 24mcache_len
(number of windows to retain full messages in cache forIWANT
responses): 12mcache_gossip
(number of windows to gossip about): 3seen_ttl
(number of heartbeat intervals to retain message IDs): 130 (= 65 seconds)
Notable differences from L1 consensus (Eth2):
seen_ttl
does not need to cover a full L1 epoch (6.4 minutes), but rather just a small window covering latest blocksfanout_ttl
: adjusted to lower thanseen_ttl
mcache_len
: a larger number of heartbeats can be retained since the gossip is much less noisy.heartbeat_interval
: faster interval to reduce latency, bandwidth should still be reasonable since there are far fewer messages to gossip about each interval than on L1 which uses an interval of 0.7 seconds.
Topic configuration
Topics have string identifiers and are communicated with messages and subscriptions.
/optimism/chain_id/hardfork_version/Name
chain_id
: replace with decimal representation of chain IDhardfork_version
: replace with decimal representation of hardfork, starting at0
Name
: topic application-name
Note that the topic encoding depends on the topic, unlike L1, since there are less topics, and all are snappy-compressed.
Topic validation
To ensure only valid messages are relayed, and malicious peers get scored based on application behavior, an extended validator checks the message before it is relayed or processed. The extended validator emits one of the following validation signals:
ACCEPT
valid, relayed to other peers and passed to local topic subscriberIGNORE
scored like inactivity, message is dropped and not processedREJECT
score penalties, message is dropped
Gossip Topics
blocks
The primary topic of the L2, to distribute blocks to other nodes faster than proxying through L1 would.
Block encoding
A block is structured as the concatenation of:
signature
: Asecp256k1
signature, always 65 bytes,r (uint256), s (uint256), y_parity (uint8)
payload
: A SSZ-encodedExecutionPayload
, always the remaining bytes.
The topic uses Snappy block-compression (i.e. no snappy frames): the above needs to be compressed after encoding, and decompressed before decoding.
Block signatures
The signature
is a secp256k1
signature, and signs over a message:
keccak256(domain ++ chain_id ++ payload_hash)
, where:
domain
is 32 bytes, reserved for message types and versioning info. All zero for this signature.chain_id
is a big-endian encodeduint256
.payload_hash
iskeccak256(payload)
, wherepayload
is the SSZ-encodedExecutionPayload
The secp256k1
signature must have y_parity = 1 or 0
, the chain_id
is already signed over.
Block validation
An extended-validator checks the incoming messages as follows, in order of operation:
[REJECT]
if the compression is not valid[REJECT]
if the block encoding is not valid[REJECT]
if thepayload.timestamp
is older than 60 seconds in the past (graceful boundary for worst-case propagation and clock skew)[REJECT]
if thepayload.timestamp
is more than 5 seconds into the future[REJECT]
if theblock_hash
in thepayload
is not valid[REJECT]
if more than 5 different blocks have been seen with the same block height[IGNORE]
if the block has already been seen[REJECT]
if the signature by the sequencer is not valid- Mark the block as seen for the given block height
The block is signed by the corresponding sequencer, to filter malicious messages. The sequencer model is singular but may change to multiple sequencers in the future. A default sequencer pubkey is distributed with rollup nodes and should be configurable.
Note that blocks that a block may still be propagated even if the L1 already confirmed a different block. The local L1 view of the node may be wrong, and the time and signature validation will prevent spam. Hence, calling into the execution engine with a block lookup every propagation step is not worth the added delay.
Block processing
A node may apply the block to their local engine ahead of L1 availability, if it ensures that:
- The application of the block is reversible, in case of a conflict with delayed L1 information
- The subsequent forkchoice-update ensures this block is recognized as "unsafe"
(see
engine_forkchoiceUpdatedV1
)
Block topic scoring parameters
TODO: GossipSub per-topic scoring to fine-tune incentives for ideal propagation delay and bandwidth usage.
Req-Resp
The op-node implements a similar request-response encoding for its sync protocols as the L1 ethereum Beacon-Chain. See L1 P2P-interface req-resp specification and Altair P2P update.
However, the protocol is simplified, to avoid several issues seen in L1:
- Error strings in responses, if there is any alternative response, should not need to be compressed or have an artificial global length limit.
- Payload lengths should be fixed-length: byte-by-byte uvarint reading from the underlying stream is undesired.
<context-bytes>
are relaxed to encode auint32
, rather than a beacon-chainForkDigest
.- Payload-encoding may change per hardfork, so is not part of the protocol-ID.
- Usage of response-chunks is specific to the req-resp method: most basic req-resp does not need chunked responses.
- Compression is encouraged to be part of the payload-encoding, specific to the req-resp method, where necessary: pings and such do not need streaming frame compression etc.
And the protocol ID format follows the same scheme as L1, except the trailing encoding schema part, which is now message-specific:
/ProtocolPrefix/MessageName/SchemaVersion/
The req-resp protocols served by the op-node all have /ProtocolPrefix
set to /opstack/req
.
Individual methods may include the chain ID as part of the /MessageName
segment,
so it's immediately clear which chain the method applies to, if the communication is chain-specific.
Other methods may include chain-information in the request and/or response data,
such as the ForkDigest
<context-bytes>
in L1 beacon chain req-resp protocols.
Each segment starts with a /
, and may contain multiple /
, and the final protocol ID is suffixed with a /
.
payload_by_number
This is an optional chain syncing method, to request/serve execution payloads by number. This serves as a method to fill gaps upon missed gossip, and sync short to medium ranges of unsafe L2 blocks.
Protocol ID: /opstack/req/payload_by_number/<chain-id>/0/
/MessageName
is/block_by_number/<chain-id>
where<chain-id>
is set to the op-node L2 chain ID./SchemaVersion
is/0
Request format: <num>
: a little-endian uint64
- the block number to request.
Response format: <response> = <res><version><payload>
<res>
is a byte code describing the result.0
on success,<version><payload>
should follow.1
if valid request, but unavailable payload.2
if invalid request3+
if other error- The
>= 128
range is reserved for future use.
<version>
is a little-endianuint32
, identifying the type ofExecutionPayload
(fork-specific)<payload>
is an encoded block, read till stream EOF.
The input of <response>
should be limited, as well as any generated decompressed output,
to avoid unexpected resource usage or zip-bomb type attacks.
A 10 MB limit is recommended, to ensure all blocks may be synced.
Implementations may opt for a different limit, since this sync method is optional.
<version>
list:
0
: SSZ-encodedExecutionPayload
, with Snappy framing compression, matching theExecutionPayload
SSZ definition of the L1 Merge, L2 Bedrock and L2 Regolith versions.- Other versions may be listed here with future network upgrades, such as the L1 Shanghai upgrade.
The request is by block-number, enabling parallel fetching of a chain across many peers.
A res = 0
response should be verified to:
- Have a block-number matching the requested block number.
- Have a consistent
blockhash
w.r.t. the other block contents. - Build towards a known canonical block.
- This can be verified by checking if the parent-hash of a previous trusted canonical block matches that of the verified hash of the retrieved block.
- For unsafe blocks this may be relaxed to verification against the parent-hash of any previously trusted block:
- The gossip validation process limits the amount of blocks that may be trusted to sync towards.
- The unsafe blocks should be queued for processing, the latest received L2 unsafe blocks should always override any previous chain, until the final L2 chain can be reproduced from L1 data.
A res > 0
response code should not be accepted. The result code is helpful for debugging,
but the client should regard any error like any any other unanswered request, as the responding peer cannot be trusted.