Event Camera Project

The goal of this project was to develop and deploy a software platform that will provide user access to the event stream with graphical & memory-mapped interface.

Project Description

If you are reading this post, this and this might be useful.
Link to GitHub
More code, Vivado Project and Altium files can be requested via email.

Event Camera, PCB Connector & ULTRA96

The Project provides user access to the event stream and records the event stream. FPGA design interfaces the event camera generates the timestamp for individual events and saves data to the DDR memory. jAER software (provided by the camera manufacturer) enables the parallel output. PCB connector up translates the signals from the ULTRA96 board (1.8V -> 3.3V). 

Project Block Diagram
1Vivado ProjectIt generates the FPGA design (.tcl file and .bit file).
2Python Script & LibraryPrograms the FPGA. Reads event stream data from the memory. Displays event stream data.
3C++ ScriptReads the event stream from the memory. Deletes repeated events. Writes the event stream (~10 sec) to the .csv file.
4PCB ConnectorPCB connector up translates the signals from the ULTRA96 board (1.8V -> 3.3V). Altium project is provided.
5Matlab ScriptIt generates a video from the recorded events.

How does it work?

The FPGA design block diagram is shown below.

FPGA Design Block Diagram

FPGA design building blocks are described in the table below. DAVIS240C processing block will be described in a separate section.

Data packager MUX
1Data PackagerData packager acts as a data multiplexer.The Switch is flipped at the negative edge of “Write Ready” and the positive edge of “New Transmission”.Switch = 0 -> Data Stream = EventsSwitch = 1 -> Data Stream = Timestamp There is a counter inside this block. The counter is triggered at the negative edge of “Write Ready” and the positive edge of “New Transmission”.If counter = 1021 -> Data Stream = TX Counter (Switch don’t care).
2Transmission Init.After FPGA is programmed and the clock is running the block will count 2 seconds and initiate the first transmission. 
3Transmission GeneratorThis block will generate a “New transmission” pulse at the clock edge of the “Start TX” signal or “Transmission Done” signal.
NOTE: for successful “New Transmission” signal detection by the DDR Memory writer, it should be extended by 4 clock cycles.
4Transmission Counter32-bit counter. Used to count transmissions.
5DDR Memory WriterState Machine that performs memory writing.  

On the CPU side the event stream can be read from the set memory location with a small Python or C script. The event stream can be visualised with a MATLAB script.

DAVIS240C Processing Block
The processing block is the updated version of this.
NOTE: Single Y address can be followed by multiple X addresses.
DAVIS240C Processing Block is implemented as a clock driven state machine .

Final State Machine

There are 3 configurable parameters in this design:
1. X Delay = 20 (in clock cycles).
2. Y Delay = 20 (in clock cycles).
3. ACK Hold time 4 (in clock cycles).
X&Y delays are introduced to make sure that the bus is stable and the data is ready. Extra ACK hold time is introduced to make sure that the camera captures the acknowledge signal every time (200Mhz single clock pulse is too short for the camera to latch).
NOTE: These parameters were defined from the experiment. Delays were set to the smallest possible number with the design being able to work.

How to use it?

Connecting Hardware

  1. Connect your ULTRA96 board to the PC with a micro USB cable.
  2. Turn on the ULTRA96 board. Wait for the blue LED to light up.
  3. Connect your DAVIS240C to the PC with a micro USB cable.
  4. Open jAER -> HW Configuration -> DVS Config.
  5. Disable “DVS.Run”. Enable “DVS.ExternalAERControl”
  6. Attach the PCB Connector to the camera (Figure 3).
  7. Attach the PCB connector to the board (Figure 3).
  8. Make sure that the blue LED is still on.

Running Software

Camera Simulator

Camera Simulator is implemented as a Vivado project. There are 2 blocks in the simulation: Event Stream Processing Block, Camera Simulator.

To run the simulation open Vivado project and press “Run Behavioral Simulation” Right-click on “clk” and set 5ns period.
NOTE: Running the simulation under different scenarios and with different delays and clocks can help to understand the operation of the camera interface.

Event Camera Simulator

Event Stream Access with Jupiter Notebooks

If you have never worked with PYNQ -> http://www.pynq.io

This software product should be used only for FPGA programming and checking that the camera read is established. 
NOTE: Program execution in “Jupiter Notebooks” is slower than in the Terminal. Do not attempt to record the event stream with “Jupiter Notebooks”.

To access the event stream with the Python Shell:
1. Connect the Hardware.
2. To open “Jupiter Notebooks” type “” in your web browser.
3. Go to -> Software_Platform -> Event_Camera -> Event_Camera.ipynb 
The first block of code will use the Python library (DVS_LIB) to run the automated events quality check . The third block of code will display the part of the event stream with the corresponding timestamp. It is the recommended way of checking that the camera interface is operating as expected:
1. “TX” (Transmission counter) is incrementing.
2. The timestamp is incrementing.
3. Events are changing.

Event Checker Access with Python Shell

Event Stream Access with Terminal

This method allows reading the event stream at high speed and recording data to the .csv file. To access the event stream with the C++ script:
1. Connect the hardware.
2. Program FPGA with the Jupiter Notebooks.
3. Open terminal with PuTTY (COM6 -> 115200).
4. Go to /home/Xilinx/C_CODE/1_EVENT_PRINT_FSM.
5. Run “g++ main.cpp –o event_recorder”.
6. Run “sudo ./event_recorder”. The program will record unique events for ~10 sec.

Event Recorder Access with PuTTY

Copy generated “data.csv” file on your PC for further processing

SD card access with windows

Event Stream Visualisation

Link to MATLAB Script

To generate a video from the events that were recorded with the C++ script.
1. Save .csv in the same folder with the MATLAB script.
2. Open Matlab and run: “fpga_loaddata” & “test_delay_USB_plugin”.
3. Wait for the plot to open.
4. Close Matlab.
5. You can find the video in the folder with the script.
NOTE: “test_delay_USB_plugin” can produce an error. Ignore it.