Introduction

Welcome to the bittide documentation book! Bittide is a inter-chip communication link that allows cycle-level deterministic communication between chips with no in-band overhead.

The benefits include:

• eliminating tail latency
• allowing compilers to statically schedule workloads across physical chips
• scaling to an arbitrary number of nodes without in-band loss

This book serves as an introduction to bittide and its concepts. Our goal is to provide the necessary information to develop and deploy experiments on bittide-based systems.

Key terms

Clock drift - The gradual deviation of two physical clocks from each other

Logical synchrony - Distributed computing to be coordinated as tightly as in synchronous systems without the distribution of a global clock or any reference to universal time

Latency-deterministic hardware - Computer hardware whose latencies and computation graph can be known ahead of time, independent of the data. Examples of non latency-deterministic hardware are branch predictors and cache prefetching, since the computation time cannot be known at compile time. Non latency-deterministic hardware can still use bittide, but loses most of the benefit of bittide’s cycle-accurate communication.

The problem with inter-chip communication

Modern computation workloads are growing larger. To accomodate this growth, workloads are being split between multiple physical processors. The real-world scaling of these workloads does not match the theoretically achievable scaling.

This disconnect is caused by non-determinism in the underlying physical clocks of each chip causing clock drift. Due to physical phenomena including manufacturing defects, heat, or vibrations, these clocks do not run perfectly in sync. Most chips have an input buffer to compensate, but this buffer can be overrun. To handle this, chips must be able to apply backpressure, or a signal to wait before sending more data. The problem also scales exponentially with the number of chips in a system, effectively limiting the scaling of systems.

As you scale up the number of connected processors, the problem compounds. More chips mean more independent clocks, more buffering, and more backpressure, which together place practical limits on system scaling.

Bittide’s Core Idea

Bittide uses a decentralized hardware algorithm to prevent clock drift. The algorithm is based on the following observations:

1. The clock drift between any two nodes can be known as the difference between the number of data frames sent out by a node and the number of data frames received by a node
2. If you adjust the clock frequency based on this signal, you can keep two nodes in logical synchrony
3. If this algorithm is carefully scaled up to an arbitrary number of nodes¹, an entire system can be held in logical synchrony

If we remove all non-determinism from inter-chip communication, then we get the benefit of scaling logic as if we were using a bigger chip, but the physical scaling of adding more chips to the cluster.

For an in-depth description of the bittide system, please see this paper.

Benefits of bittide

By removing non-determinism from inter-chip communication, bittide allows distributed systems to scale as if they were a single, larger chip. From the perspective of software and compilers, communication becomes a deterministic operation with known cost, enabling more aggressive scheduling, static placement of data, and compile-time reasoning about performance.

In effect, bittide shifts complexity away from runtime mechanisms and into design-time guarantees. This tradeoff enables large-scale systems that preserve the simplicity and predictability traditionally associated with small, tightly integrated hardware designs.

Who might be interested in bittide

The bittide system represents a novel approach to inter-chip communication that guarantees determinism. Certain workloads and compute architectures are better suited to take advantage of this property than others.

We believe those engineers are:

• Hardware engineers who work on latency-deterministic hardware
• Compiler engineers who are interested in optimizing the mapping of computation onto distributed hardware
• Software engineers who need a fixed latency output for their workload

The requirements of a bittide system

Any engineering implementation requires tradeoffs. For bittide,

• a small amount of die space to handle input buffers and clock control
• an adjustable clock source
• some startup time on power up to synchronize clocks before starting workload
• a compiler that is able to take advantage of a bittide-based system

Timeline of project

The bittide project is Apache 2.0 licensed and is being developed by QBayLogic and Google DeepMind.

Timeline

Mar 2025 - Paper on 8 node full bittide setup published (arXiv)

Aug 2024 - 8 node demo extended to handle logical latency and scheduling between nodes

June 2023 - 8 node proof-of-concept demo completed for clock synchronization

Jan 2023 - 2 node proof-of-concept demo completed for clock synchronization

Sep 2021 - Paper on bittide system theory published (arXiv)

Aug 2021 - Project start

1. See this paper ↩

Architecture Overview

Bittide as a communication link is inherently scalable. Bittide nodes can be connected point-to-point between nodes, boxes, and racks without loss of per-cycle accuracy. The only parameters to a bittide network are link topology and link latency. Notably, link latency DOES NOT affect per-cycle accuracy, but it does affect inter-node latency.

Here’s an example bittide network

Any Processing Element can be added to a bittide network, with the requirements that

1. the Processing Element is able to run on the bittide clock domain, or some PLL multiple
2. the Processing Element dedicates a small amount of die or FPGA space for the bittide interface
3. …that’s about it, actually

The rest of this chapter is devoted to how bittide achieves logical synchrony on a per-node basis. For scheduling a computation over the network and failure recovery, see a further chapter.

Bittide Node bringup sequence

For actually using a bittide network, these details are largely irrelevant. Still, it’s useful to have a general understanding of how bittide works in practice.

Step 1: Booting the bittide clock

Contrary to what one might expect, there are actually TWO clocks in a bittide node: an adjustable clock (called the bittide clock) and a static clock. For most of the bittide boot and all of the Processing Element functioning, the bittide clock is used. However, since the adjustable clock is actually a somewhat complex piece of silicon, it itself needs to be set up. To do this, we have a Boot CPU running on the static clock, which has two jobs:

1. set up bittide clock by configuring clock registers and setting the initial nominal frequency
2. bring parts of the SerDes out of reset

Once the bringup sequence has moved to the next step, the Boot CPU and static clock are no longer used.

Step 2: Achieving clock syntony(ish)

At this point in the bringup sequence, all bittide clocks have been coordinated with reference to local, inperfect, static clocks. The goal of this step is to align every bittide clock on the network, creating syntony.

Each bittide node starts sending a pseudorandom binary sequence over every link. The content of the data is not important, merely that data is being sent. The SerDes of every link locks onto the remote clock frequency embedded in the link in order to deserialize the data.

In typical networks, once the data is deserialized and converted to the local frequency, this remote clock information is discarded. In bittide, the remote clock frequency is stored locally in a register that counts every tick of the remote clock. Bittide also stores a counter for the local bittide clock. Collectively, we call these registers the Domain Difference Counter (or DDC). The Clock Control CPU reads in these counters and adjusts (FINC or FDEC) the local bittide clock depending on whether it detects the local clock is too fast or too slow.

Over time, we can show bittide clocks settle to a common network frequency (up to a small delta). Once this state is achieved, these clock control circuits stay active to maintain a common frequency despite changes in heat, etc.

Step 3: Actually achieving clock syntony

So far, we have achieved a common bittide network frequency. The dutiful engineer will have noticed we only guaranteed that frequency with a small delta. This seems counter to bittide’s promise of cycle-level accuracy. And if we left it there, you would be right. Cycle-ish-level accuracy doesn’t get us much.

To absorb small wobbles between the remote bittide clocks and local bittide clock, a small Elastic Buffer (EB) is inserted. Notably, this buffer can be made much smaller than the corresponding input buffer used by most networking interfaces, because bittide guarantees a small delta of clock drift. The bittide system tries to keep the elastic buffer filled halfway at all times, so that it can absorb a wobble of (buffer size)/2 clock cycles.

We can now say the bittide system has achieved cycle-level latency. But we have a new problem: no bittide node knows what that latency actually is between itself and its neighbors (nor even who its neighbors are).

Step 4: Determining logical latencies

Each node will send out a message of the current clock cycle. Notably, this message sending does not need to be coordinated in any way. Each neighbor will receive that clock cycle and record the local clock cycle it was received on. This number defines the one way domain mapping, also called logical latency.

How this “recording” happens in practice is an open technical discussion on bittide. We present the two main solutions below, both of which have been tested on the bittide hardware.

Option 1: Hard UGN capture

Hard UGN capture has two parts:

1. For each node, the first Bittide Word it writes to the network is the local clock counter value
2. Each node also has a hardware component, called the UGN capture, that sits between the EB and the RB on the receiving end. Its sole job is to wait for the first valid piece of bittide data and save it with the local clock cycle at that time. It lets all data thereafter through to the RB

These two pieces, together, ensure the UGNs are captured between each node. The Management Unit (MU) can then read the UGN values and use them. Below we have an example of a UGN capture in practice.

Example Hard UGN capture

◀ ▶

Option 2: Soft UGN capture

Soft UGN capture is done by using the Management Unit (MU) CPU to read the Ring Buffer (RB). The benefit of this approach is we can re-use an existing component instead of creating a new one, saving space on the hardware.

However, the CPU approach comes with a major limitation - unlike the UGN Capture component, the MU cannot inspect every bittide word the same cycle it comes in.¹ Therefore, we need to do two things:

1. let the MU know which element in the Rx RB corresponds to the start of the Tx RB.
2. have the neighbor node send the clock cycle sometime at the start of the Tx RB.

This way, the MU does not need to inspect every element in the RB for the clock cycle, it just needs to inspect the one entry it knows the clock cycle will eventually be in. For more detail, see the Ring Buffer alignment section.

Once the relationship has been mapped, the sending node can send a “UGN event” (5 bittide words), which will be read by the receiving MU.

Step 5: Handover to the processing element

Once logical latency has been established, the bittide network guarantees these latencies until reboot. Control is handed over to the Processing Element. The Processing Element can operate as normal, without any knowledge of the bittide network. However, it now has guaranteed latency with all other nodes, allowing it to schedule and execute computations.

Glossary for bittide-specific terms

Bittide word - The smallest unit of the bittide network in a clock cycle. The word size of bittide is 64 bits.

Clock Control CPU (CC) - The CPU that reads in the domain differences between each neighbor node and the local node and sends a signal to the clock to speed up or slow down based on some function.

Boot CPU - The CPU that boots the adjustable (bittide) clock and configures its registers via SPI. It also brings SerDes and Handshake out of reset, which negotiates the 8b10b link with every other node.

Domain difference counter (DDC) -

Domain mapping - Previously called the Uninterpretable Garbage Number (UGN), the domain mapping is the observation we make to obtain the logical latency. It consists of transmit timestamp expressed in clock cycles of your link partner. When this timestamp arrives in your own domain, it is stored together with the receive timestamp. This pair of counters is the domain mapping because it maps the transmit cycle of your link partner to your receive cycle. Both of these are natural numbers (each represented as u64)

Elastic buffer (EB) -

Processing element (PE) - The computation element that is connected to the bittide network via bittide. The computational elements can be anything (hence the general term), but is often considered to be an ASIC or similar.

Handshake (needs better name) - A step function that sends out PRBS until a link is negotiated. Then simply passes data through.

Logical latency - Integer that is derived from the domain mapping, we can use this to predict the exact clock cycle when a message will arrive at our link partner.

Logic layer -

Management Unit (MU) - The CPU that performs elastic buffer centering and UGN capturing.

Nominal frequency -

Pseudorandom binary sequence (PRBS) -

Physical layer -

(Aligned) Ring buffer

Roundtrip time - Natural number that represents the number of clock cycles it takes for a message to make a roundtrip from node A to node B and back. It is the sum of the logical latencies l a -> b and l b -> a.

Static clock (SCLK) - A reference clock that is not adjustable. Its only purpose is to provide a clock for the Boot CPU. Once the Boot CPU is finished, the static clock is no longer needed.

Glossary for non-bittide technical terms

Comma symbol - An alignment symbol in 8b10b link negotiation

1. This difference occurs because the Ring Buffer (RB) only supports accessing one address per cycle. The UGN Capture sits before the Ring Buffer (RB) in the data pipeline, while the MU CPU sits behind the RB. So UGN capture can inspect every new word, while the MU needs to know which RB element to inspect. If the MU were to scan the entire RB, it would find the right element, but it would then not know on which clock cycle the element was put into the RB. ↩

Hardware-in-the-Loop (HITL) Platform

This chapter describes the specific hardware setup used to realize a bittide system in our lab environment. The HITL platform is designed to implement the principles of bittide using real hardware components for experimentation, development, and validation for topologies up to 8 nodes.

Platform Overview

Requires diagram

The HITL platform consists of the following main components:

• Host Computer: Executes experiments on the bittide system.
• FPGA Boards: Implement clock control, bittide routing, and compute fabric.
• Clock Adjustment Boards: Provide single controllable clock source for an FPGA.
• High-Speed Interconnects: Serial links (such as SFP+ or QSFP) connect the FPGAs for low-latency, high-bandwidth communication.
• JTAG/UART dongles: Provide JTAG and UART interface for each FPGA to the host computer.
• Ethernet connections: Each FPGA has an ethernet connection from a dedicated RJ45 port to the host computer.
• SYNC_IN / SYNC_OUT: Ring connection through all FPGAs to get a sense of global time. This is used for starting / stopping experiments and mapping measurement data from all FPGAs to a single time line. This is not part of the bittide architecture itself.

Experiments

This chapter describes the existing infrastructure to support the creation and execution of experiments on the example bittide system.

Required components

• Example design: Contains all bittide related FPGA logic.
• Programs: The binary files that will be loaded into the CPUs that exist in the example design.
• Driver: Runs on the host PC and communicates with the testbench.
• Testbench: Contains the example design and other necessary logic such as ILA’s.

Relevant infrastructure

Names which tools are relevant and where they live

Execution

Describes how tests are executed as part of a CI/CD pipeline.

Writing programs

Describes how to write new programs for the management unit, general purpose processing element or clock control.

Writing a driver

Describes how to write a driver for the experiment.

Adding your experiment to CI/CD

Describes how to add your experiment to the CI/CD pipeline.

Components

This section provides an overview of the main hardware components in the Bittide system. Each component plays a specific role in enabling efficient, synchronized, and flexible operation of the hardware platform.

Available Components

• Transmit Ring Buffer
• Receive Ring Buffer

Transmit Ring Buffer

Transmit Ring Buffer

The Transmit Ring Buffer is a hardware component designed to transmit data frames over a Bittide link. It uses a single memory buffer and a free-running hardware read counter that wraps around the buffer, forming a ring buffer. The CPU can write to the buffer via the Wishbone interface.

Architecture

The Transmit Ring Buffer consists of:

• Memory Buffer: A single memory buffer written by the CPU and read by hardware.
• Free-Running Read Counter: A hardware counter that increments each cycle, wrapping around the buffer. When the ring buffer is enabled, data at the address indicated by this counter is transmitted over the Bittide link.
• Enable Register: When enabled, the buffer contents are transmitted to the network. When disabled, zeroes are transmitted instead, but the counter continues to increment to maintain alignment.
• Wishbone Interface: Allows the CPU to write data to be transmitted.

Operation

1. Data Transmission:

   • Each cycle, if the ring buffer is enabled, the data at the address indicated by the free-running read counter is read from the buffer and sent over the Bittide link.
   • The read counter increments and wraps around the buffer.

2. CPU Access:

   • The CPU writes data to the buffer using the Wishbone interface.
   • Because the hardware continuously reads from the buffer, the CPU must ensure it writes data before the hardware’s read counter reaches that address. See Ring Buffer Alignment and Asynchronous Communication for protocols that handle this.

Receive Ring Buffer

Receive Ring Buffer

The Receive Ring Buffer is a hardware component designed to receive data frames from a Bittide link and store them into a local memory. It uses a single memory buffer and a free-running hardware write counter that wraps around the buffer, forming a ring buffer. The CPU can read from the buffer via the Wishbone interface.

Architecture

The Receive Ring Buffer consists of:

• Memory Buffer: A single memory buffer written by hardware and read by the CPU.
• Free-Running Write Counter: A hardware counter that increments each cycle, wrapping around the buffer. When the ring buffer is enabled, incoming frames are written at the address indicated by this counter.
• Enable Register: When enabled, incoming frames from the network are written to the buffer. When disabled, incoming frames are ignored, but the counter continues to increment to maintain alignment.
• Clear-at-Count Register: Resets the write counter to zero when it reaches the configured value. This is used during the Ring Buffer Alignment procedure to synchronize the TX and RX counters.
• Wishbone Interface: Allows the CPU to read received data.

Operation

1. Data Reception:

   • Each cycle, if the ring buffer is enabled and a valid frame arrives from the Bittide link, the frame is written to the memory at the address indicated by the free-running write counter.
   • The write counter increments and wraps around the buffer.

2. CPU Access:

   • The CPU reads data from the buffer using the Wishbone interface.
   • Because the hardware continuously overwrites the buffer, the CPU must read data before it is overwritten. See Ring Buffer Alignment and Asynchronous Communication for protocols that handle this.

Software UGN Demo

This chapter describes the specific hardware setup used to perform a test where each node is able to determine the UGNs to its neighbors without communicating with a host PC that is able to read all of the information from each of the nodes in the system. This is accomplished through the firmware on the management unit (MU).

Architecture

Initialization sequence

1. “Boot” CPU
   1. Gets programmed by the host
   2. Programs the clock boards
   3. Gets transceiver block out of reset (enabling the bittide domain / other CPUs)
   4. Activates each transceiver channel and waits until they’ve negotiated a link with their neighbors.
   5. Prints “all done” message to UART.
2. Clock control CPU
   1. Gets programmed by the host
   2. Calibrates clocks
   3. Prints “all done” message to UART.
   4. (Keeps calibrating clocks.)
3. Management unit CPU
   1. Gets programmed by the host
   2. Sets the channels to “user” mode. This makes the channels accept data from the outside world instead of their negotiation state machinery.
   3. Waits for hardware UGN to be captured
   4. Start auto-centering elastic buffers
   5. Waits for clocks to be considered stable
   6. Stops auto-centering the elastic buffers
   7. Prints UGNs captured by hardware component to UART.
   8. Aligns the transmit/receive ring buffers.
   9. Calls the “c_main” and runs the software UGN discovery protocol

Domain related

Components:

• Boot CPU (BOOT)
• Transceivers
• Domain difference counters
• Clock control (CC)
• Elastic buffers (one per incoming transceiver link)
• Hardware UGN capture (for comparison)

Node related

Components:

• Management unit (MU)
• 7 transmit and receive ring buffers, one of each per link

Management unit

Connected components:

• Timer
• UART (for debugging)
• FPGA DNA register

The management unit is connected to the bittide interconnect network via transmit/receive ring buffers. In this demo the management unit runs a distributed protocol to discover the Uninterpretable Garbage Numbers (UGNs) of the network links. The protocol uses the aligned ring buffers to exchange timestamped messages with neighbors, calculating the propagation delays in software. For a detailed description of the procedure, see Software UGN Discovery Procedure.

To change the binary run on this CPU, one may either:

• Edit bittide-instances/src/bittide/Instances/Hitl/SoftUgnDemo/Driver.hs, line 215 (at time of writing) to use another binary instead of soft-ugn-mu
• Edit the source files in firmware-binaries/soft-ugn-mu/ to change the binary pre-selected by the driver function

Debugging related

• UART arbiter
• JTAG interconnect
• Integrated logic analyzers
• SYNC_IN/SYNC_OUT

Running tests

One may specifically run the software UGN demo test by making a .github/synthesis/debug.json with the following contents:

[
  {"top": "softUgnDemoTest",       "stage": "test", "cc_report": true}
]

Tests are configured to run the following binaries on the system’s CPUs:

• Boot CPU: wire-demo-boot (firmware-binaries/demos/wire-demo-boot)
• Clock control CPU: clock-control (firmware-binaries/demos/clock-control)
• Management unit: soft-ugn-mu (firmware-binaries/demos/soft-ugn-mu)

One may change this by either:

1. Changing the driver function so that it loads different binaries onto the CPUs. This may be accomplished by changing which binary name is used with each of the initGdb function calls.
2. Changing the source code for the binaries. The locations for them are listed above.

Wire Demo

The wire demo demonstrates the “wire” behavior of bittide: threading a path through an 8-node FPGA network, treating each link as a dedicated point-to-point connection. Rather than using calendars to do time multiplexing on every link, each link is connected to the PE through a single mux. This mux can either be set to serve the MU or the PE.

The host driver uses the UGNs to calculates a route through all nodes and programs each node’s mux and PE configuration accordingly. Each PE reads a value from one link, XORs it with its own FPGA DNA, and writes the result to another link — chaining all nodes together. The final node’s result should be the XOR of all device DNAs, which serves as verification that the path was correctly established.

Architecture

Initialization sequence

1. “Boot” CPU
   1. Gets programmed by the host
   2. Programs the clock boards
   3. Gets transceiver block out of reset (enabling the bittide domain / other CPUs)
   4. Activates each transceiver channel and waits until they’ve negotiated a link with their neighbors.
   5. Prints “all done” message to UART.
2. Clock control CPU
   1. Gets programmed by the host
   2. Calibrates clocks
   3. Prints “all done” message to UART.
   4. (Keeps calibrating clocks.)
3. Management unit CPU
   1. Gets programmed by the host
   2. Sets the channels to “user” mode. This makes the channels accept data from the outside world instead of their negotiation state machinery.
   3. Waits for hardware UGN to be captured
   4. Start auto-centering elastic buffers
   5. Waits for clocks to be considered stable
   6. Stops auto-centering the elastic buffers
   7. Prints UGNs captured by hardware component to UART.

Domain related

Components:

• Boot CPU (BOOT)
• Transceivers
• Domain difference counters
• Clock control (CC)
• Elastic buffers (one per incoming transceiver link)
• Hardware UGN capture (for comparison)

Node related

Components:

• Management unit (MU)
• 7 transmit and receive ring buffers, one of each per link
• Programmable mux
• WireDemoPe, a processing element specific for this demo
• WireDemoPeConfig, a bus accessible device with a writable configuration for the processing element

Management unit

Connected components:

• Timer
• UART (for debugging)
• FPGA DNA register

The management unit has access to and is responsible for all transmit/receive ring buffers in the node. In this demo these are not used.

To change the binary run on this CPU, one may either:

• Edit bittide-instances/src/bittide/Instances/Hitl/WireDemo/Driver.hs, line 215 (at time of writing) to use another binary instead of wire-demo-management-unit
• Edit the source files in firmware-binaries/demos/wire-demo-management-unit/ to change the binary pre-selected by the driver function

Wire Demo processing element

PE (wireDemoPe)

Takes two configuration values (from “PE config”):

• What link to read from
• What link to write to

The PE is active for two cycles after reset:

• First cycle: read from link indicated by config (or source static “0”).
• Second cycle: write value_read_in_first_cycle XOR fpga_dna_lsbs to link indicated by config (or do nothing)

PE Config (wireDemoPeConfig)

Has 3 bus accessible registers:

• read_link :: Maybe (Index 7)
• write_link :: Maybe (Index 7)
• written_data :: Maybe (BitVector 64)

The first 2 registers are used to configure the PE. The third register stores the data written by the PE over the configured link. This is used to verify the demo works. The data in the last PE Config should be the XOR of all DNAs.

Debugging related

• UART arbiter
• JTAG interconnect
• Integrated logic analyzers
• SYNC_IN/SYNC_OUT

Running tests

One may specifically run the software UGN demo test by making a .github/synthesis/debug.json with the following contents:

[
  {"top": "wireDemoTest",       "stage": "test", "cc_report": true}
]

At the time of writing, the clock control CPU stabilizes system. The driver running on the host (bittide-instances/src/bittide/Instances/Hitl/WireDemo/Driver.hs) then releases the reset of the management unit CPU. In turn, this CPU will center the elastic buffers and print out the UGNs captured using the hardware UGN capture component over UART.

The host driver then calculates a schedule to thread a path through all nodes and programs all PE Configs and programmable muxes. After waiting for a static time the host driver reads the written_data register in each PE Config device and verifies it is equal to the XOR of all device DNAs up to that node. The last node should therefore have stored the XOR of all device DNAs.

Tests are configured to run the following binaries on the system’s CPUs:

• Boot CPU: wire-demo-boot (firmware-binaries/demos/wire-demo-boot)
• Clock control CPU: clock-control (firmware-binaries/demos/clock-control)
• Management unit: wire-demo-management-unit (firmware-binaries/demos/wire-demo-management-unit)

One may change this by either:

1. Changing the driver function so that it loads different binaries onto the CPUs. This may be accomplished by changing which binary name is used with each of the initGdb function calls.
2. Changing the source code for the binaries. The locations for them are listed above.

Ring Buffer Alignment Protocol

Definitions

• Transmitting Ring Buffer (TX): Local memory written by CPU, read by hardware using a free-running wrapping counter. Has an enable register: when enabled, the hardware transmits buffer contents; when disabled, zeroes are transmitted, but the counter continues to increment.
• Receiving Ring Buffer (RX): Local memory written by hardware using a free-running wrapping counter, read by CPU. Has an enable register (incoming frames are ignored when disabled, but the counter keeps incrementing) and a clear_at_count register that resets the write counter to zero when it reaches the configured value.

Context

In a bittide system, nodes operate in a globally synchronous manner despite being asynchronous devices with unknown start times. Communication occurs via ring buffers. When TX and RX ring buffers are the same size, the address mapping between them is constant, determined by (logical) network latency and the start time difference between nodes.

Because nodes start at different times, data written to TX index 0 by one node may arrive at an arbitrary RX index on the neighbor. The alignment protocol measures this offset and uses the hardware clear_at_count register to realign the RX write counter so that TX index 0 maps to RX index 0.

Objective

Synchronize the TX and RX hardware counters on each link so that corresponding indices map to each other, enabling reliable asynchronous communication without software-level offset tracking.

Alignment Algorithm

The alignment is implemented as a per-link state machine (AlignedReceiveBuffer::align_step) with four phases after the initial setup. Both sides of a link run the same algorithm simultaneously.

Phase 0 — Unaligned (setup)

1. Clear the TX buffer (write zeroes to all positions).
2. Write the marker ALIGNMENT_ANNOUNCE (0xBADC0FFEE) to TX index 0.
3. Enable both the TX and RX ring buffers.
4. Transition to FindingAlignment.

Phase 1 — FindingAlignment

Scan the entire RX buffer for a non-zero marker (ALIGNMENT_ANNOUNCE or ALIGNMENT_ACKNOWLEDGE).

• Marker found at RX index 0: The counters are already aligned. Write ALIGNMENT_ACKNOWLEDGE (0xDEADABBA) to TX index 0 and transition to AcknowledgingAlignment.
• Marker found at RX index n (n ≠ 0): Write n to the clear_at_count register. The next time the hardware write counter reaches n, it resets to 0, shifting the marker to RX index 0 on a subsequent scan. Stay in FindingAlignment.
• No marker found: Stay in FindingAlignment (keep scanning).

Phase 2 — AcknowledgingAlignment

Poll RX index 0:

• When the value is ALIGNMENT_ACKNOWLEDGE (partner sent its acknowledgement) or 0 (partner already disabled TX): disable our TX (set_enable(false)) and transition to WaitingForZeroes.

The invariant is that we only disable our TX after confirming the partner has seen our marker and acknowledged.

Phase 3 — WaitingForZeroes

Poll RX index 0:

• When the value is 0 (partner has also disabled TX): disable our RX (set_enable(false)) and transition to Aligned. Alignment is complete.

This two-step shutdown (AcknowledgingAlignment → WaitingForZeroes) prevents a race where one side disables TX before the partner has seen the acknowledgement.

Summary

Resulting Interface: Aligned Receive Buffer

Upon successful alignment, the hardware counters are synchronized: TX index i on one node maps to RX index i on the neighbor. No software-level offset tracking is needed. The AlignedReceiveBuffer type tracks the alignment state and exposes the underlying RX buffer for direct indexed access once aligned.

After alignment completes, both TX and RX are disabled. Higher-level code re-enables them as needed for normal operation.

Communication Challenges

While the aligned ring buffer provides logical connectivity, the physical link remains unreliable due to the interaction between the read/write counters and asynchronous CPU access:

1. Continuous Hardware Operation: The hardware continuously cycles through the ring buffers at the network link speed.
2. Asynchronous CPU Access: The CPU operates asynchronously and often slower than the network link.

This leads to specific failure modes:

• Data Corruption (Pointer Overtaking):
  • TX Side: If the hardware’s read pointer overtakes the CPU’s write pointer during a write, a torn frame is sent.
  • RX Side: If the hardware’s write pointer overtakes the CPU’s read pointer during a read, the message is corrupted.
• Data Loss: If the CPU does not read from the RX ring buffer every iteration, the hardware will overwrite unread data.
• Data Duplication: If the CPU does not write to the TX ring buffer every iteration, the hardware will resend old data.

Reliable communication requires a higher-level protocol to handle these errors. See the Asynchronous Communication Protocol for a proposed solution using the smoltcp library to implement a reliable TCP/IP layer over the aligned ring buffers.

Asynchronous Communication Protocol

Context

In a bittide system, we need asynchronous communication between nodes, particularly during the boot phase. The Ring Buffer Alignment Protocol provides an AlignedRing Buffer abstraction that allows for packet exchange.

However, as described in that protocol’s documentation, the raw AlignedRing Buffer link is unreliable, subject to packet corruption and loss due to hardware/software speed mismatches.

Objective

Establish a reliable, asynchronous, point-to-point communication channel between nodes over the potentially unreliable AlignedRing Buffer links.

Proposed Solution

Leverage the TCP/IP protocol suite to handle error detection, retransmission, and flow control. We will use the smoltcp library, a lightweight TCP/IP stack designed for embedded systems, to implement this layer. Note that future versions of bittide will probably use a bespoke network stack for asynchronous communication.

Implementation Strategy

1. Network Interface (smoltcp::phy::Device)

We will implement the smoltcp::phy::Device trait for the Aligned Ring Buffer.

• Medium: Use Medium::Ip to minimize overhead (no Ethernet headers required for point-to-point).
• MTU: Set to 1500 bytes (standard Ethernet size) to accommodate typical payloads.

2. Framing & Alignment

• The underlying Aligned Ring Buffer abstraction ensures packets are read from the correct aligned memory location.
• Packet Boundaries: The length of each packet will be derived directly from the IP Header length field.

3. Addressing

• Topology: Initially restricted to Peer-to-Peer links.
• IP Assignment: Placeholder IPs, if necessary we use static addressing derived from unique hardware identifiers (e.g., FPGA DNA or Port ID) to avoid the complexity of DHCP.

4. Demo Application

Develop a proof-of-concept application that:

1. Initializes the Aligned Ring Buffer.
2. Sets up a smoltcp interface.
3. Establishes a TCP connection between two nodes.
4. Transfers data to verify reliability against induced packet loss/corruption.

Assumptions

• An Aligned Ring Buffer abstraction exists that provides a read/write interface for single aligned packets.
• The ringbuffer size is sufficient to hold at least one MTU-sized packet plus overhead.