SSZ upgrades, HTTP Rest API Standardisation and v0.6.1 spec implementation progress.
Eth 2.0 development at Lighthouse has been steady over the last fortnight, seeing hefty updates to some core aspects of Lighthouse as new networking components are defined and the specification is updated.
In summary form:
- SSZ updated to include SOS-style offsets (SOS is the name given to the encoding scheme devised by @karalabe from the geth team).
- A new parser has been built for parsing test formats defined by the Ethereum Foundation and our
cached_tree_hashcrates are passing all specified tests.
- Further development towards
- Standardisation towards a RESTful HTTP API service to be provided by all beacon nodes.
- Further development on bringing DiscoveryV5 (discv5) to life inside rust-libp2p.
- Complete the update to spec
- Migrate the current gRPC interface between the beacon node and validator client to the specified RESTful HTTP API.
- Complete the development of the discv5 protocol in rust-libp2p and integrate into Lighthouse.
What you need to know
For the interested readers, this section provides further details about the development progress/discussions happening inside the Lighthouse project and the broader Eth 2.0 ecosystem.
Design decisions and the road ahead
Recently, an Eth 2.0 interoperability workshop was held in New York where various client teams and researches discussed design choices and a path towards client interoperability. A live stream was available, however the link appears to be down at the time of writing. We will update this post when a new link is available.
This article will discuss a few topics that were covered in the workshop, however we recommend watching the recorded live stream for further information.
RESTful HTTP API
There has been quite a bit of discussion amongst the client teams and researchers about how to best interface with a beacon node (the software that connects to the chain, the Eth 2.0 analogue of geth or parity-ethereum). Currently, Ethereum clients use JSON-RPC which is widely supported, however there have been some drawbacks which we have an opportunity to address in the design of Eth 2.0 clients. A discussion of the pros/cons of using JSON-RPC can be found in this issue.
There are some interesting advantages to using a RESTful HTTP API in place of a JSON-RPC interface. In addition to being widely supported by most Internet applications, its stateless design allows for easier load-balancing, easier firewall configuration (specific HTTP requests can be rate-limited or blocked), and the API can be easily standardised using a single YAML file.
The Lighthouse team agree with making the switch to support a RESTful HTTP API and as such have started efforts towards standardising such an API. Luke has created PR-1069 which proposes an API that clients will ideally support when Ethereum 2.0 goes live. With a standard interface amongst clients, this could also lead to interoperability between validator clients and beacon nodes from different client implementations.
Ethereum 2.0 is based on a peer-to-peer network. In such networks, nodes need a mechanism to find each other without relying on some central service. This is typically referred to as a discovery mechanism, of which there are a variety of approaches. P2P networks (such as BitTorrent) typically use a DHT (Distributed Hash Table) to track other peers in the network. This forms a global collection/database of peer addresses where small subsets of of the global collection are stored individually on each node of the network. A common protocol that facilitates this is Kademlia.
Clients that support present-Ethereum use an adaptation of this protocol known as DiscoveryV4 (discv4). A number of proposed improvements to discv4 has lead to a new protocol known as DiscoveryV5 (discv5). In summary, it adds encryption, topic advertisement, and a few other nice features that we'd like to see in a next generation Ethereum client. Security considerations with discovery mechanisms are very important in P2P networks since significant attack vectors can arise as a direct result of a poorly designed mechanism. The eclipse attack is a common example of this, where nodes can be lead to believe in a false head of a blockchain.
We believe that discv5 is a good fit for Ethereum 2.0, however, no implementations are currently able to perform tests and benchmarks using this protocol. Since Eth 2.0 clients will be using the libp2p networking stack, it would be ideal to have this mechanism built into libp2p. To this end, we have been working on a rust-libp2p discv5 implementation and expect to have an initial working version by the next development update.
SSZ Spec-related decisions
As mentioned earlier,
v0.6.x of the specification includes significant
modifications to SSZ. During these updates, the research team has decided to not
include an explicit definition of SSZ de-serialisation. Instead, they leave
it to the reader to derive their own method based upon the supplied definition
of SSZ serialisation.
This is completely reasonable and supports the concept of a succinct specification. However, it does necessitate that implementers think carefully about edge-cases and attacks which may not be obvious at first glance. During our implementation we found several such cases and produced a HackMD document to assist other implementations and security researchers.
Lighthouse-specific development details
In a effort lead by @protolambda, the new
specification came with a vastly more comprehensive test suite. Passing these
tests is a critical stepping-stone for a multi-client testnet and we have been
hard at work implementing these tests.
The tests are delivered in YAML files and are consistent enough for us to build a framework that only requires a small amount of unique code for each test. We have successfully implemented the generic framework and have implemented generic SSZ tests in less than 100 lines of unique code.
There are many (>30) "constants" that shape an Ethereum 2.0 chain, some of the
most significant being
SHARD_COUNT. The specification
is defined across a fixed set of constants, however it is possible to alter
these constants to achieve a chain with different properties. For example, we
have been using
SHARD_COUNT = 8 and
SLOTS_PER_EPOCH = 8 to create chains
for testing that run comfortably with small validator counts (e.g. 16
validators in total).
It is a matter of choice for each implementation as to whether or not these "constants" are indeed compile-time constants or if they are really variables defined dynamically at runtime.
With Lighthouse, we have elected to avoid fixing a single set of constants at compile time so that we can support running chains with different constants without requiring a re-compile. This is consistent with how existing, current-Ethereum clients may switch between Ropsten and mainnet without having to recompile.
Whilst we had originally intended to support any arbitrary set of constants declared at runtime (e.g. from a JSON file), we found this added significant complexity to our codebase. As a result, we struck a compromise that we will support any arbitrary set of constants, as long as they are defined at compile-time; however the set that is used can be selected dynamically at runtime. We can declare sets of constants very easily (1-10 lines of code per set), so we may define many sets of constants which will achieve code-simplicity as well as providing flexibility to our users.
Lighthouse's immediate future
In the next fortnight, Lighthouse will undertake significant core library development. As discussed in this post, we are focusing on upgrading our spec-related libraries, overhauling our networking stack to include discv5, and converting our current gRPC beacon-node API to a RESTful HTTP API.