---
gitea: none
include_toc: true
---

# Development manual

This document gives an overview of the internal structure of the PuzzleFW firmware package.
It also describes the process to build the FPGA firmware and the embedded software.


## FPGA design overview

The FPGA captures ADC samples and digital input signals.
ADC samples go into the analog acquisition chain.
The acquisition chain handles triggering and optional downsampling.
A configurable number of samples are collected following each trigger event.
Trigger events and collected samples are transferred to a buffer in DDR memory via the AXI slave interface of the Zynq processing system.

Digital signals go into a timetagger subsystem.
The timetagger detects edges on selected signals and assigns timestamps to such events.
Timestamped events are transferred to a buffer in DDR memory via the AXI slave interface.

All synchronous elements are clocked at 125 MHz, derived from the ADC sample clock.

![FPGA design schematic](fpga_design_schematic.png)

### Block design

A Vivado block design contains the Zynq processing system (ARM), AXI interconnect, and AXI-to-APB bridge.
This block design is instantiated in the top-level VHDL file.

![Block design](fpga_block_design.png)

### PS-to-PL interfaces

The following interfaces are used between the PS (ARM) and PL (FPGA):

- M\_AXI\_GP0 <br>
  Used by embedded software to read and write registers in the FPGA via AXI and APB bus.
  This interface is mapped to addresses 0x43000000 to 0x431fffff in the PS address map.
- S\_AXI\_HP0 <br>
  Used by FPGA firmware to read and write to DDR memory.
- GPIO <br>
  The PS controls a few specific digital signals in the FPGA via GPIO lines.
  One of these lines is a global reset signal for the FPGA firmware.
- SPI0 <br>
  Routed via EMIO through the FPGA to the SPI bus of the ADCs (only on 4-input boards).
- IRQ\_F2P <br>
  Used by FPGA firmware to trigger interrupts in the PS.
- FCLK\_CLK0 <br>
  Fixed 200 MHz clock generated by the PS.
  On 4-input boards, the FPGA uses this clock as reference for IODELAY primitives.
  On 2-input boards, this clock is not used.

### ADC input timing

The ADC transfers samples via a parallel source-synchronous interface.
A 125 MHz clock is sent along with the samples.
This is also the main clock source of the FPGA design.

On 2-input Red Pitaya boards, the ADC is hardwired to operate in _full rate CMOS mode_.
Each analog channel has 14 parallel data signals from ADC to FPGA.
These data signals transition on the falling clock edge, such that the FPGA can capture on the rising clock edge.
The FPGA buffers the 125 MHz ADC clock through a PLL.
Data signals are captured in the FPGA in IOB flip-flops, clocked on the rising edge of the buffered 125 MHz clock.
It turns out that a phase shift of 90 degrees in the PLL provides near-optimal timing for capturing the data signals.
This follows from synthesizer timing reports and has been confirmed experimentally by testing different PLL phase shift settings.

On 4-input Red Pitaya boards, only 7 parallel data signals per analog channel are connected from the ADCs to the FPGA.
The ADCs must be programmed (via SPI) to operate in _double data rate CMOS mode_.
In this mode, the ADC still outputs a 125 MHz clock, but data lines transition on both edges of the clock.
The data signals are routed through IDELAY components in the FPGA before being captured in IDDR registers.
The IDDR are clocked on the 125 MHz clock from the ADC that produces the samples (without PLL).
The IDELAY components use a fixed delay, tuned to 2.34 ns to optimize data capture timing.
Timing reports indicate that this provides sufficient margin.
This has been confirmed experimentally by phase-shifting the ADC output clock.
The main FPGA design runs on a 125 MHz clock, derived by a PLL from the ADC that samples channels 1 and 2.
Transferring samples from channels 3 and 4 to the main clock is slightly tricky because the two ADC clocks may be skewed with respect to each other.
A few mysterious timing constraints were added to deal with this.


## Embedded software overview

The purpose of the embedded software is to make the system remote accessible via the network.
The software collects data from the FPGA and transmits it via TCP.
Similarly, the software accepts remote control commands via TCP and executes these by accessing registers in the FPGA.

### Operating system

The embedded software runs within an embedded Linux system on the Red Pitaya.

The core operating system is based on [Buildroot](https://buildroot.org/), a framework that builds an embedded Linux system from scratch.
Buildroot automatically downloads and builds many additional open source packages that are needed to construct the embedded system.
Among these packages is a cross-compilation toolchain.

The kernel for the embedded system is built from the [Xilinx distribution of the Linux kernel](https://github.com/Xilinx/linux-xlnx).
It is necessary to use the Xilinx variant of Linux.
The vanilla Linux kernel can run on the Red Pitaya, but the Xilinx kernel provides important features that are missing from the vanilla kernel, such as programming the FPGA.

The shell and system utilities of the embedded system are based on Busybox.
Dropbear is installed as SSH server, but disabled by default.

The set of packages to be installed in the embedded system, along with their compile-time settings, are specified in the Buildroot configuration.
Further tweaking of the embedded system is done via a _filesystem overlay_: a set of files that are injected into the embedded root filesystem as a final step.
The overlay mechanism is used to install boot scripts and configuration files.
It is also used to install custom user space software.

### Boot flow

[U-Boot](https://source.denx.de/u-boot/u-boot) is used as boot loader and as _secondary program loader_ (SPL).
Building U-Boot requires a proper device tree for the board.

Building the U-Boot SPL image also requires the file `ps7_init_gpl.c`.
The contents of this file depend on the configuration of the Zynq programming system which is specified in the Vivado block design.
To generate this file, the FPGA design must be _exported_ from Vivado in the form of an XSA file.
The XSA file is actually a ZIP archive which contains `ps7_init_gpl.c` and the corresponding header file.
The U-Boot build script (`12_build_uboot.sh`) copies these files to the correct location in the U-Boot source tree.

The boot process of the Red Pitaya is as follows:

- The Zynq PS boots from ROM.
  The ROM program reads `boot.bin` from the SD card and runs it.

- `boot.bin` is the U-Boot SPL image.
  It initializes many subsystems in the Zynq PS, including the DDR memory controller.
  It then reads `u-boot.img` from the SD card and runs it.

  Note that we don't use a Xilinx FSBL at all.
  Many Zynq tutorials explain how to build `boot.bin` using Xilinx tools.
  That requires a non-free FSBL image which must be built in a Vitis project.
  The PuzzleFW package avoids that whole mess by using U-Boot SPL instead.

- `u-boot.img` is the main U-Boot program image.
  It reads and executes `boot.scr`, a script which describes the Linux boot process.
  The script instructs U-Boot to read the Linux kernel `uImage`,
  the device tree `devicetree.dtb` and the root filesystem image `rootfs.cpio.uboot`
  from the SD card.
  It then runs the Linux kernel.

- `uImage` is the Linux kernel image.
  The kernel initializes drivers and subsystems.
  The root filesystem is unpacked and mounted as a RAM filesystem.
  The filesystem is writeable, but any changes remain in RAM and will be forgotten on the next reboot.

- A set of boot scripts are invoked to initialize the embedded system and start services.
  Most of these scripts are prepared by Buildroot, but some scripts have been tweaked or specially created for PuzzleFW.
  These scripts take care of the following:
    - Read the MAC address from the EEPROM on the Red Pitaya board, and assign it to the Ethernet interface.
    - Set a unique hostname `rp-XXXXXX` based on the MAC address.
    - Obtain an IPv4 address via DHCP or a preconfigured static address.
      This is controlled by a file in the configuration partition of the SD card.
    - Start an SSH server, if enabled. By default, this is disabled.
    - Read a BIT file from the SD card and program the FPGA.
      Information in the EEPROM is used to determine which FPGA image must be used.
    - Configure ADCs on the Red Pitaya board via SPI or GPIO.
    - Load the Linux kernel driver to access the FPGA firmware.
    - Run a user space program that accepts remote control commands via TCP.

### Linux kernel driver

A custom Linux kernel module is used to access the FPGA firmware.
This module is called `puzzlefw.ko`.

The driver is based on the Linux UIO framework.
The kernel module itself has minimal functionality.
Its only functions are:

 - mapping the register address range of the FPGA to user space via mmap
 - mapping the DMA buffer in DDR RAM to user space via mmap
 - handling FPGA interrupts and notifying user space via UIO

All non-trivial interactions with the FPGA are done in user space.
This includes managing the data flow via DMA.

### User space software

The FPGA firmware is controlled by a user space program.
This program accepts remote control commands via TCP.
Commands are handled by reading and writing registers in the FPGA.
The program also collects data from the FPGA via DMA, and transmits these data through the network via TCP.

The program is written in C++.
It uses Boost Asio, an asynchronous I/O framework.

### Configuration partition

While the embedded software runs, the root filesystem is located in RAM.
The filesystem is writable, but any changes will be lost on the next boot.

In order to store persistent settings, the SD card has a separate configuration partition.
This is `/dev/mmcblk0p2`, formatted as Ext4.
The following files are stored there:

- `network.conf`: IP address configuraton (DHCP or static IP address)
- `calibration.conf`: analog calibration coefficients
- `start_ssh.conf`: enables or disables starting the SSH server
- SSH host key

The configuration partition is temporarily mounted read-only during boot, to read the configuration files.
Otherwise, the partition is only accessed when the configuration is modified.
In that case, the partition is temporarily mounted to write the updated files, then unmounted again.
As a result, writes to the SD card occur only when the user applies a change in the persistent configuration of the system.
This avoids unnecessary wear of the SD card.


## Build procedure

A Linux PC (x86-64) is required to build the system.
Development was done with Debian 12 (bookworm).
Other Linux distributions will probably also work.

The build process consists of two main steps.
These steps must be done in the following order:

 1. Build the FPGA firmware for the Zynq PL.
 2. Build the embedded software for the Zynq PS.


## Building FPGA firmware

Building the FPGA firmware requires Vivado release 2020.2.
Later releases will probably work, but may require adjustments.

There are two ways to build the firmware: as a Vivado project, or in Vivado non-project mode.
In most cases, non-project mode will be easier.

### Preparing the build environment

**Install Vivado** <br>
Download and install Vivado 2020.2.
The build scripts assume that Vivado is installed in `/opt/Xilinx/Vivado/2020.2`.
If a different location is used, it must be configured in the script environment file.

**Configure paths** <br>
If necessary, edit the file `redpitaya-puzzlefw/fpga/script_env` and make sure that `VIVADO_DIR` points to the directory where Vivado is installed.

**Install board files** <br>
When running Vivado in project mode, it requires board definition files
for the Red Pitaya.
These files are available from the original Red Pitaya FPGA repository.
Use the following steps to download and install these files.

- Go to directory `redpitaya-puzzlefw/fpga`
- Run `./01_get_redpitaya.sh`
- Copy the complete folder `redpitaya-puzzlefw/fpga/RedPitaya-FPGA/brd/redpitaya`
  to `/opt/Xilinx/Vivado/2020.2/data/boards/board_files`

### Building FPGA firmware in non-project mode

There are two top-level designs, each targeting a specific Red Pitaya board type.
These designs are built in separate design runs:

 - `puzzlefw_top` for the normal (2-input) Red Pitaya
 - `puzzlefw_top_4ch` for the 4-input Red Pitaya

The following steps invoke Vivado in non-project mode to handle synthesis, implementation and bitfile generation of each top-level design.

- Go to directory `redpitaya-puzzlefw/fpga`
- Run `./11_build_bitfile.sh`
- Run `./12_build_bitfile_4ch.sh`

It is normal for the build process to display a large number of messages and warnings.
The build process produces the following output files:

- `puzzlefw_top.bit.bin` <br>
  This file is required to program the FPGA.
- `redpitaya_puzzlefw.xsa` <br>
  This file contains a description of the system configuration.
  It is necessary to build U-boot.
- Similar files for the 4-input design.

The build process also creates various logs and report files in directory `vivado/output` (or `output_4ch`).
It is recommended to review the contents of `vivado/output/post_route_timing.rpt` to make sure that it does not report any timing violations and contains the line `All user specified timing constraints are met`.

### Building FPGA firmware in project mode

The repository contains a Vivado project file to build the design for the normal (2-input) Red Pitaya.
There is currently no project file for the 4-input board.

The following steps may be used to build the design in Vivado in project mode:

- Start Vivado, typically by running something like <br>
  `source /opt/Xilinx/Vivado/2020.1/settings64.sh ; vivado`
- Open the project file `redpitaya-puzzlefw/fpga/vivado/redpitaya_puzzlefw.xpr`.
- In the *Flow Navigator* panel on the left, click on "Run Synthesis", then click "Ok".
  It may appear that very little is happening, but synthesis should be running in the background.
  The status is displayed in the *Design Runs* panel at the bottom of the screen.
- When synthesis completes successfully, choose "Run Implementation" and click "Ok".
- When implementation completes successfully, choose "Generate Bitstream" and click "Ok".
- When bitstream generation completes successfully, choose "View Reports".
- In the *Project Summary* panel, check that timing is ok (Number of Falling Endpoints should be 0).
- In the main menu bar, click *File*, *Export*, *Export Hardware*.
  Select "Pre-synthesis".
  Change XSA file name and export folder to write the XSA file to
  `redpitaya-puzzlefw/fpga/redpitaya_puzzlefw.xsa`.
  Then click "Finish".

The build process produces a BIT file `puzzlefw_top.bit` containing the FPGA design.
The Zynq software needs this file in a different format (`.bit.bin`) which can not be created by the Vivado IDE.
Use the following steps to convert the BIT file:

- Go to directory `redpitaya-puzzlefw/fpga`
- Run `./make_binfile.sh vivado/redpitaya_puzzlefw.runs/impl_1/puzzlefw_top.bit`

### Editing the block design

Most source files for the FPGA firmware can be edited with any text editor.
The VHDL source files are in directory `redpitaya-puzzlefw/fpga/rtl`.
Constraint files are in `redpitaya-puzzlefw/fpga/constraints`.

The only exception is the block design that links the Zynq processing system to the FPGA logic.
The block design is stored as `redpitaya-puzzlefw/fpga/vivado/redpitaya_puzzlefw.srcs/sources_1/bd/puzzlefw/puzzlefw.bd`.
This file can only be edited through Vivado, as follows:

- Start Vivado as described above.
- Open the project file `redpitaya-puzzlefw/fpga/vivado/redpitaya_puzzlefw.xpr`.
- In the *Flow Navigator* panel on the left, click on "Open Block Design".
- Make changes as needed.
- In the main menu bar, click *File*, *Save Block Design*.

Both top-level designs (2-input and 4-input boards) use the same block design file.
This is a tricky situation because the block design file targets a specific FPGA part (e.g. `xc7z010clg400-1`), but the two boards have different FPGA parts.
When using the non-project build flow, we find that this just seems to work without issues.
During the build flow, Vivado quietly modifies the block design file to match the target part of the design run.

### Adding VHDL files to the project

When VHDL files are added to the project, both the Vivado project and the non-project build script must be updated.

To update the Vivado project:

- Open the project file `redpitaya-puzzlefw/fpga/vivado/redpitaya_puzzlefw.xpr`.
- In the *Flow Navigator* panel on the left, click "Add Sources".
- Add VHDL files directly from the directory `redpitaya-puzzlefw/fpga/rtl`.
  Do not select "Copy sources into project".
- Vivado may get a confused because, by default, it does not support VHDL-2008.
  To fix this, in the *Sources* panel, click "Libraries" below the hierachy tree.
  Right-click each new VHDL file, choose "Set File Type" and change the file type to "VHDL 2008".

Vivado stores all project settings in the XPR file.
Changes to that file must be tracked in the Git repository.

To update the non-project build script:

- Open `redpitaya-puzzlefw/fpga/vivado/nonproject.tcl` in a text editor.
- Add `read_vhdl` statements to load the new VHDL files.


## Building embedded software

### Preparing the build environment

Install the following Debian packages (or equivalent packages on other Linux distributions):
<br>
`build-essential`,
`bc`,
`bzip2`,
`cpio`,
`dosfstools`,
`file`,
`git`,
`libncurses-dev`,
`mtools`,
`patch`,
`perl`,
`rsync`,
`unzip`,
`wget`.

### Downloading software

Use the following steps to download required software packages:

- Go to directory `redpitaya-puzzlefw/sw`
- Run `./01_get_buildroot.sh` to download [Buildroot](https://buildroot.org).
- Run `./02_get_uboot.sh` to download [U-Boot](https://source.denx.de/u-boot/u-boot).
- Run `./03_get_kernel.sh` to download the [Xilinx distribution of the Linux kernel](https://github.com/Xilinx/linux-xlnx).

### Building embedded software

Use the following steps to build the embedded software:

- Go to directory `redpitaya-puzzlefw/sw`
- Run `./11_build_buildroot.sh` to configure and build Buildroot.
  <br>
  One of the outcomes of this step is a cross-compilation toolchain for the ARM architecture.
  Other build steps need this toolchain, therefore this must be the first step of the build flow.
- Run `./12_build_uboot.sh` to configure and build U-Boot.
  <br>
  This step requires the file `redpitaya-puzzlefw/fpga/redpitaya_puzzlefw.xsa` from the FPGA build flow.
- Run `./13_build_kernel.sh` to configure and build the Linux kernel.
- Run `./14_build_devicetree.sh` to build the device tree.
- Run `./15_build_driver.sh` to build the Linux kernel module that provides access to the FPGA firmware.
- Run `./16_build_userspace.sh` to build user space applications that will run on the Red Pitaya.
- Run `./21_rebuild_rootfs.sh` to re-build the embedded root filesystem, including the kernel driver and user space tools that were built during previous steps.
- Run `./22_prepare_sdcard.sh` to collect the files that must be installed on the SD card for the Red Pitaya.
- Run `./23_sdcard_image.sh` to build an image file which can be written to the SD card.

### Changing the Buildroot configuration

The Buildroot configuration for this project is stored in the folder `config`.
To change the configuration, start by applying the existing settings, then run the Buildroot configuration tool, then copy the new settings to the `config` folder.
Use the following steps:

- Copy `sw/config/buildroot_puzzlefw_defconfig` to `sw/buildroot-2023.02.08/.config`
- Go to `sw/buildroot-2023.02.8`
- Run `make olddefconfig`
- Run `make nconfig`
- Use the menu to change the configuration as needed.
- Run `make savedefconfig`
- Copy `sw/buildroot-2023.02.08/defconfig` to `sw/config/buildroot_puzzlefw_defconfig`

After changing the Buildroot configuration, rerun step `11_build_buildroot.sh`.

### Changing the U-Boot configuration

The U-Boot configuration for this project is stored in the folder `config`.
To change the configuration, start by applying the existing settings, then run the U-boot configuration tool, then copy the new settings to the `config` folder.
Use the following steps:

- Copy `sw/config/uboot_redpitaya_puzzlefw_defconfig` to `sw/u-boot/configs/redpitaya_puzzlefw_defconfig`
- Go to `sw/u-boot`
- Run `make redpitaya_puzzlefw_defconfig`
- Run `make nconfig`
- Use the menu to change the configuration as needed.
- Run `make savedefconfig`
- Copy `sw/u-boot/defconfig` to `sw/config/uboot_redpitaya_puzzlefw_defconfig`

After changing the Buildroot configuration, rerun step `12_build_uboot.sh`.

### Changing the Linux kernel configuration

The Linux kernel configuration for this project is stored in the folder `config`.
To change the configuration, start by applying the existing settings, then run the kernel configuration tool, then copy the new settings to the `config` folder.
Use the following steps:

- Copy `sw/config/linux_redpitaya_puzzlefw_defconfig` to `sw/linux-xlnx/.config`
- Go to `sw/linux-xlnx`
- Run `make ARCH=arm olddefconfig`
- Run `make ARCH=arm nconfig`
- Use the menu to change the configuration as needed.
- Run `make ARCH=arm savedefconfig`
- Copy `sw/linux-xlnx/defconfig` to `sw/config/linux_redpitaya_puzzlefw_defconfig`

After changing the Buildroot configuration, rerun step `13_build_kernel.sh`.

### Preparing the SD card

This section explains how to set up the SD card manually.
It is not normally necessary to do this.
An easier way to set up the SD card is by building an image file and writing it to the card.

The SD card contains two partitions:

- `/dev/mmcblk0p1` is a 256 MB partition with FAT filesystem
- `/dev/mmcblk0p2` is a 256 MB partition with an EXT4 filesystem

The FAT partition contains the following files:

| File                       | Description |
|----------------------------|-------------|
| `boot.bin`                 | first stage boot loader for the Zynq (built by U-Boot) |
| `u-boot.img`               | main boot loader program (U-Boot) |
| `boot.scr`                 | boot script for U-Boot |
| `devicetree.dtb`           | device tree for the Linux kernel |
| `uImage`                   | Linux kernel image |
| `rootfs.cpio.uboot`        | root filesystem image |
| `puzzlefw_top.bit.bin`     | FPGA firmware image for 2-input board |
| `puzzlefw_top_4ch.bit.bin` | FPGA firmware image for 4-input board |

The EXT4 partition can be left empty initially.
Configuration files will be written to this partition by the embedded system.

The following commands may be used to format the two partitions:

    mkdosfs -F 16 /dev/mmcblk0p1
    mke2fs -t ext4 -b 4096 /dev/mmcblk0p2