Document timetagger

doc/fpga_firmware.md

@@ -8,6 +8,20 @@ The PuzzleFW firmware provides the following functionality:
 * Transfer ADC data and timetagger data to the Zynq PS via DMA.
 
 
+# Status LEDs
+
+LED0 to LED7 are the first 8 yellow LEDs from left to right on the side of the Red Pitaya.
+
+LED0 blinks at a rate of 1 Hz when the PuzzleFW firmware is active.
+Its purpose is to provide a minimal indication that the FPGA is active.
+
+LED1 is on when the analog acquisition chain is enabled (register `ACQUISITION_EN`).
+
+LED2 is on when the analog acquisition chain is waiting for a trigger.
+
+LED3 is on when the timetagger is active (at least one event type is enabled).
+
+
 # Analog acquisition chain
 
 TODO ...
@@ -29,27 +43,83 @@ TODO : LEDs
 
 # Timetagger
 
-TODO ...
+The timetagger has 4 digital input signals.
+If a rising or falling edge occurs on one of these signals, a timestamp is assigned to that event and a message is emitted and transferred via DMA.
 
-TODO : DMA data format
+A 4-cycle glitch filter is applied to the digital input signals.
+This filter rejects digital pulses shorter than 4 clock cycles (32 ns).
 
-TODO : LEDs
+Messages are only emitted for _enabled_ event types.
+Rising edges and falling edges can be separately enabled or disabled for each digital input channel.
+This is done via register `TIMETAGGER_EN`.
+
+The messages are queued in a FIFO buffer inside the FPGA before being transferred via DMA.
+This FIFO is necessary to hold messages for a short time while the DMA engine sets up a DMA transfer.
+It has room for 4096 messages.
+
+## Output data format
+
+The output from the timetagger is a sequence of 64-bit messages.
+Three types of messages are used:
+
+* Event messages, emitted when a digital input event occurs that it enabled for time tagging.
+* Marker messages, emitted on explicit request from software.
+* Overflow messages, emitted when the internal data buffer overflows.
+
+### Timetagger event message
+
+| Bits    | Field name    | Description |
+|---------|---------------|-------------|
+| 63 : 60 | msg_type      | Fixed value 0x2 |
+| 59 : 57 | channel       | Index of digital input channel where the event occurred. |
+| 56      | falling_edge  | '1' if a falling edge occurred, '0' if a rising edge occurred. |
+| 51 : 48 | input_state   | New state of digital input signals after the event. |
+| 47 : 0  | timestamp     | Timestamp in units of 8 ns |
+
+An event message is emitted when a digital input event occurs that is enabled for time tagging.
+
+Note that multiple events can occur simultaneously.
+If this happens, separate messages are emitted for all events that occurred.
+These messages all have the same timestamp, and the same `input_state` field.
+
+### Timetagger marker messages
+
+| Bits    | Field name    | Description |
+|---------|---------------|-------------|
+| 63 : 56 | msg_type      | Fixed value 0x30 |
+| 51 : 48 | input_state   | State of digital input signals. |
+| 47 : 0  | timestamp     | Timestamp in units of 8 ns |
+
+A marker message is emitted when software writes value 1 to register `TIMETAGGER_MARK`.
+This may be used to establish synchronization between digital input events and some other type of events that are triggered or detected by software.
+
+### Timetagger overflow message
+
+| Bits    | Field name    | Description |
+|---------|---------------|-------------|
+| 63 : 56 | msg_type      | Fixed value 0x40 |
+
+An overflow message is emitted when the internal data buffer overflows.
+It marks the point in the message sequence where an unknown number of messages have been discarded.
 
 
 # Registers
 
-The behaviour of the PuzzleFW firmware is controlled via a set of registers.
+The behaviour of the FPGA is controlled via a set of registers.
 These registers are accessible from the Zynq PS via memory mapped I/O.
 
 The registers are located in a 2 MB region with address 0x43000000 to 0x431fffff in the PS address map.
 
+Register access is implemented via an internal APB bus inside the FPGA.
+This APB bus is connected via an intermediate AXI bus to port `M_AXI_GP0` of the Zynq PS-to-PL interface.
+
 Within the register address space, the first 1 MB area is intended for various types of control and status registers that are accessed by user space software.
 The second 1 MB area is intended for privileged registers that are only accessed by system software such as the Linux kernel driver.
 The difference is that writing incorrect values to privileged registers can corrupt or crash the PS system, while writing to user registers can not.
 
 The complete set of registers is listed below.
 Access to undefined addresses within the register area must be avoided.
-If such access occurs, the firmware may either ignore it or map it to an existing register in a way that may be unexpected.
+If such access occurs, the FPGA may either ignore it or map it to an existing register in a way that may be unexpected.
 
 ## Register list
 
@@ -124,9 +194,9 @@ Fields are marked as _RW_ or _RO_ or _WC_.
 * RW fields can be modified by writing to the register.
   Reading from the register returns the most recent value written to the field.
 * RO fields are read-only. Writes to these fields are ignored.
-  The value of these fields typically changes spontaneously to reflect the status of the firmware.
+  The value of these fields typically changes spontaneously to reflect the status of the FPGA.
 * WC fields are write-only and auto-clearing.
-  Writing a 1 bit to the field triggers an action in the firmware.
+  Writing a 1 bit to the field triggers an action in the FPGA.
   Writing a 0 bit has no effect.
   WC fields are returned as 0 when reading from the register.
 
@@ -153,7 +223,7 @@ Enable PL-to-PS interrupts.
 |---------|---------------|--------|-------------|
 | 0       | irq_enable    | RW     | '1' to enable interrupts, '0' to disable |
 
-This register acts as a master switch for interrupts from the firmware.
+This register acts as a master switch for interrupts from the FPGA.
 Specific interrupt conditions can be separately enabled or disabled via other registers.
 
 ### 0x000014: IRQ_PENDING
@@ -173,7 +243,7 @@ Enable DMA transfers.
 |---------|---------------|--------|-------------|
 | 0       | dma_en        | RW     | '1' to enable DMA transfers, '0' to disable |
 
-This register acts as a master switch for the DMA engine in the PuzzleFW firmware.
+This register acts as a master switch for the DMA engine in the FPGA.
 
 Disabling DMA will prevent new bursts from starting, but allows ongoing bursts to complete.
 Under normal circumstances, the completion of ongoing bursts can take at most a few microseconds.
@@ -194,7 +264,7 @@ If a DMA error occurs, no further DMA bursts will be started until the error is
 The failed DMA burst will not be acknowledged to the subsystem that initiated it (analog or timetagger).
 To recover from this situation, it will typically be necessary to stop and re-initialize all ongoing DMA data streams, clear the error, then restart DMA data streams.
 
-An address error typically occurs when the DMA address range for the analog subsystem or the timetagger exceeds the size of the DMA window specified via `DMA_BUF_SIZE`.
+An address error typically occurs when the DMA address range for the analog subsystem or the timetagger exceeds the size of the DMA buffer specified via `DMA_BUF_SIZE`.
 
 ### 0x000108: DMA_CLEAR
 
@@ -643,20 +713,92 @@ Size of DMA buffer in RAM.
 
 | Bits    | Field name    | Access | Description |
 |---------|---------------|--------|-------------|
-| 31 : 12 |               | RW     | DMA buffer size as number of 4 kB pages. |
+| 31 : 12 |               | RW     | DMA buffer size. |
+
+If this register is set to zero, no DMA access is allowed.
 
 
 # DMA
 
-TODO ...
+The FPGA acts as an AXI bus master to transfer bulk data to system RAM.
+This data transfer pattern is commonly referred to as DMA.
+Note that the Zynq DMA controller (DMAC) is not involved in these data transfers in any way.
+
+DMA access is implemented via an internal AXI bus master inside the FPGA, connected to port `S_AXI_HP0` of the Zynq PS-to-PL interface.
+
+## Data transfer pattern
+
+The AXI HP data bus is 64 bits wide.
+All DMA transfers use the full width of the bus on every beat.
+Therefore, the smallest unit of data transferred via DMA is a 64-bit word.
+The size of every DMA transfer is a multiple of 8 bytes, and the address of every transfer is aligned to a multiple of 8 bytes.
+
+(This is an implementation choice in the firmware.
+It is possible in principle to transfer smaller amounts of data via the AXI bus, but the PuzzleFW firmware is designed to transfer 64-bit words in all cases.)
+
+Data are temporarily queued in a FIFO RAM block inside the FPGA until the DMA engine is ready to start a transfer.
+This is necessary because DMA operates in bursts, and it may take some time before the DMA engine can initiate a burst.
+The two DMA data streams (analog acquisition, timetagger) each have a separate FIFO memory.
+
+If sufficient data are queued up inside the FPGA, the DMA engine transfers 16-beat bursts (blocks of 128 bytes).
+This is the most efficient way to use the AXI bus.
+If the amount of available data is insufficient for a full burst, the DMA engine waits briefly (2 microseconds) for more data to arrive.
+After waiting, if there are still insufficient data for a full burst, the available data are transferred via single-beat bursts (blocks of 8 bytes).
+
+The highest sustainable data transfer rate is approximately 105,000,000 64-bit words per second, or 800 MiB per second.
+
+## Global DMA buffer
+
+A global address window determines the range of allowed target addresses for DMA transfers.
+This window is configured through the pair of registers `DMA_BUF_ADDR` and `DMA_BUF_SIZE`.
+
+Whenever the FPGA initiates a DMA transfer, it first checks that the target address is within the configured buffer.
+If the transfer falls outside the buffer, it fails and triggers an _address error_.
+
+The start and size of the DMA buffer must be aligned to 4 kB page boundaries.
+
+## Per-stream buffer segments
+
+For each of the two DMA data streams (analog acquisition, timetagger) a dedicated segment may be allocated within the global DMA buffer.
+These segments are used as circular buffers.
+
+All addresses that specify the start or end of a buffer segment, or any location within a buffer, are specified as offsets relative to the start of the global DMA buffer `DMA_BUF_ADDR`.
+
+For the analog aqcuisition data stream, the start and end of the buffer segment are configured through registers `ACQ_ADDR_START` and `ACQ_ADDR_END`, with `ACQ_ADDR_END` pointing to the first byte past the end of the segment.
+The start and end addresses must be aligned to 128-byte boundaries.
+
+The _write pointer_ of the DMA stream is stored in register `ACQ_ADDR_PTR`.
+It indicates the destination address of the next DMA transfer.
+It also signals to software that write transfers have completed for the region up to (but excluding) the write pointer.
+The write pointer is read-only for software.
+The FPGA automatically updates the pointer after each data transfer.
+If the write pointer reaches the end of the segment, it immediately wraps back to the start.
+The write pointer is always aligned to an 8-byte boundary.
+
+If the write pointer reaches the address configured in `ACQ_ADDR_LIMIT`, no further DMA transfers will occur until the limit is updated.
+Software can use this limit to prevent the DMA engine from overwriting parts of the buffer that have not yet been procecessed.
+The limit address must be aligned to a 128-byte boundary.
+
+If the write pointer reaches the address configured in `ACQ_ADDR_INTR`, an interrupt is triggered.
+This happens only if the corresponding interrupt is enabled in registers `IRQ_ENABLE` and `ACQ_INTR_CTRL`.
+The interrupt address must be aligned to an 8-byte boundary.
+
+The timetagger data stream works in the same way as the analog acquisition data stream, but uses a separate set of registers.
+
+## Cache coherence
+
+Keep in mind that the Zynq architecture does not provide transparent cache coherence.
+When the FPGA writes data to system RAM via DMA, the cache of the ARM processor may still contain obsolete data for the same address range.
+
+Software running on the Zynq PS that wants to read from the DMA buffer, must either take explicit steps to invalidate the cache, or it must configure the DMA buffer as non-cacheable memory.
 
 
 # Interrupts
 
-The PuzzleFW firmware can send interrupts to the Zynq PS to notify the software of specific conditions.
+The FPGA can send interrupts to the Zynq PS to notify the software of specific conditions.
 Interrupts are routed via `IRQ_F2P[0]`, which corresponds to interrupt 61 in the Zynq Generic Interrupt Controller (GIC).
 
-There are two types of conditions that can trigger an interrupt:
+Two types of conditions can trigger an interrupt:
 
 * DMA transfers for analog acquisition have progressed to a configurable point.
 * DMA transfers for the timetagger have progressed to a configurable point.
@@ -670,9 +812,9 @@ For example, interrupts may be used to wait for a certain amount of data from th
 * Write register `ACQ_INTR_CTRL` to clear any pending interrupt and enable interrupts on DMA progress.
 * Read register `ACQ_ADDR_PTR` to check whether DMA progress has already passed the configured threshold.
   This is necessary to avoid a race condition where software keeps waiting for a condition thas has already happened.
-* Write register `IRQ_ENABLE` to enable firmware interrupts.
+* Write register `IRQ_ENABLE` to enable interrupts.
 * Wait for the interrupt or do other stuff in the mean time.
-* When the interrupt occurs, the interrupt handler may write register `IRQ_ENABLE` to disable firmware interrupts until the software has time to handle the cause of the interrupt.
+* When the interrupt occurs, the interrupt handler may write register `IRQ_ENABLE` to disable interrupts until the software has time to handle the cause of the interrupt.
 * Optionally read register `IRQ_PENDING` to confirm that the interrupt was caused by analog data acquisition.
 * Write register `ACQ_INTR_CTRL` to clear the interrupt and disable further interrupts of this type.
 * Process the data that arrived in the DMA buffer.