A Survey on FPGA-Based Sensor Systems: Towards Intelligent and Reconfigurable Low-Power Sensors for Computer Vision, Control and Signal Processing

Abstract

The current trend in the evolution of sensor systems seeks ways to provide more accuracy and resolution, while at the same time decreasing the size and power consumption. The use of Field Programmable Gate Arrays (FPGAs) provides specific reprogrammable hardware technology that can be properly exploited to obtain a reconfigurable sensor system. This adaptation capability enables the implementation of complex applications using the partial reconfigurability at a very low-power consumption. For highly demanding tasks FPGAs have been favored due to the high efficiency provided by their architectural flexibility (parallelism, on-chip memory, etc.), reconfigurability and superb performance in the development of algorithms. FPGAs have improved the performance of sensor systems and have triggered a clear increase in their use in new fields of application. A new generation of smarter, reconfigurable and lower power consumption sensors is being developed in Spain based on FPGAs. In this paper, a review of these developments is presented, describing as well the FPGA technologies employed by the different research groups and providing an overview of future research within this field.

1. Introduction

Processing capabilities in sensor nodes are typically based on Digital Signal Processors (DSPs) or programmable microcontrollers. However, the use of Field Programmable Gate Arrays (FPGAs) provides specific hardware technology, which can also be reprogrammable thus providing a reconfigurable sensor system. The partial reconfiguration is the process of modifying only sections of the logic that is implemented in an FPGA. Thus, the corresponding circuit can be modified to adapt its functionality to perform different tasks. This adaptation capability allows the implementation of complex applications by using the partial reconfigurability with very low power consumption. This last feature also represents an important aspect when FPGAs are applied in sensor systems. Nowadays, the sensor systems are required to provide an increasing accuracy, resolution, and precision while decreasing the size and consumption. Additionally, FPGAs and their partial reconfigurability allow us to provide sensor systems with additional properties such as processing capabilities, interfaces, testing, configuration, etc. These sensors are typically referred to as smart sensors.

The current capabilities of FPGA architectures allow not only implementation of simple combinational and sequential circuits, but also the inclusion of high-level soft processors. The use of integrated processors holds many exceptional advantages for the designer, including customization, obsolescence mitigation, component and cost reduction and hardware acceleration. FPGA embedded processors use FPGA logic elements to build internal memory units, data and control busses, internal and external peripheral and memory controllers. Both ** and low redesign effort. For that end, dynamic reconfigurable devices such as FPGAs play an important role because they can add flexibility to the sensor node. In this context, an important approach developed in Spain is the Cookie platform [4,19,66,67], a modular device that has an innovative WSN node architecture. This modular platform is divided into four functional layers: communication, processing, power supply and sensing/actuating layer. The heart of this platform, the processing layer, includes a microcontroller and an FPGA device, giving more processing power and flexibility to the platform. This layer carries out the processing of all the information given by the sensors. On the one hand, the microcontroller usually deals with the communication control. On the other hand, the FPGA processes the signals coming from sensors. This platform has been tested in processing, power consumption, communication, and encryption with successful results [19,67].

Another relevant issue for WSNs is to compute the location of the mobile nodes connected to the network. For outdoors, location technologies such as GPS or Galileo can be used. However, for indoor environments, technologies based on RFID, image recognition or ultrasonic must be employed. In this context, in [68], a low cost ultrasonic-based location system for mobile nodes is presented using FPGA devices. This location system is employed to obtain the maximum reachable precision. On the one hand, an FPGA device is used to excite the ultrasonic transmitter, and on the other hand, another FPGA device is employed in the mobile node to identify the time difference between the obtained measurements.

2.4. Signal Processing

Embedded signal processing is another topic of interest in the use of FPGAs. Until the appearance of FPGAs in the electronic world, DSPs were the key devices for signal processing. Currently, for highly demanding tasks, FPGAs have superseded DSPs due to the high efficiency given by their architectural flexibility (parallelism, on-chip memory, etc.) [69], reconfigurability [70] and massive performance in the development of algorithms [71]. This subsection provides a brief explanation about the main Spanish approaches in the use of FPGAs for signal processing in sensor systems.

In most cases, FPGAs are used for the implementation of sensor data processing. In this context, in [14], the design of WSNs to get the data of a set of pulse oximeters is presented. In this paper, pulse and oxygen values are processed in the FPGA and the obtained values are sent in real time to the Database Server via a WSN. Another contribution to mention is the presented in [72], where some spike-based band-pass filters have been synthesized for FPGA devices.

Low-level processing of ultrasonic signals is another issue which is being implemented with FPGA devices in order to increase scan rate, precision, and reliability [15,20,73,74]. In this context, using Time-Of-Flight (TOF) measurements given by the transducers, some drawbacks such as cross-talk problems, specular reflection and echo discrimination can arise and generate errors in the distance computation. In order to solve these problems, multimode techniques such as Golay sequences [15] are employed. The implementation of this algorithm in an FPGA device permits adaptation to the distance of the reflector in the environment, simultaneous emissions and simultaneous reception in all transducers being able to discriminate the emitter of the echo.

2.5. Other Sensors

Data compression is a technique that often improves the data bandwidth requirement of any sensor system. FPGAs can be used to implement different kinds of data compression [7,75]. The design of a compression scheme depends, obviously, on the application. In [18,76] a compression scheme over an FPGA is described. The works presented in these papers can be applied to any multi-sensor that must send and receive data simultaneously from different independent sources. Parallelism properties of FPGAs schemes are exploited to implement different units in charge of each of the signals emitted. Sending more than a data source at the same time is the main idea of these papers. This idea was firstly developed by Hernandez et al. in [74], where an ultrasonic sensor is improved by sending data simultaneously. Using both, an FPGA and a DSP, the system is able to receive and process this data in real-time. FPGAs can also contribute to decide which kind of image compression fits better with a given image, as in [77], where the authors apply their compression selector in an FPGA embedded on a satellite. An FPGA is a device that really improves the behavior of parallel algorithms. There are also applications exploiting this parallelism like the FPGA-based web servers shown in [78].

Another application where an FPGA may help is in reconfigurable data acquisition systems [79]. Data acquisition systems are used in vast range of tasks. In [80], an FPGA is used as a thermal sensor to measure the behavior of ring oscillators over different voltages. This sensor measures whether a given device dissipates excessive power in relation to the input voltage. The program is implemented through a Microblaze microprocessor over a ** operations such as determining object features or region labeling. These tasks are again characterized by local data access, but more complex pixel operations. Finally, high-level vision tasks are more decision-oriented, such as object recognition, face recognition or scene recognition. These tasks involve non-local data access and non-deterministic and complex algorithms. The same task can often be referred as any of the three levels in the literature. However, in this paper the different works about vision and FPGA have been divided into these three categories following the next rules: if the primary purpose task is image enhancement, the task is categorized as low-level; the tasks that operate on the pixels to produce features in the image are mid-level tasks; and finally, decision-making stage is classified as a high-level task.

3.1. Low-Level Vision Tasks

FPGAs are ideal for image processing, particularly for low-level and mid-level tasks where parallelism is exploited [9]. Most of the works found in the literature related to computer vision and FPGAs describe a parallelism version of a classical sequential computer vision algorithm [9,10]. For a pipelined architecture, a different hardware block is built for each image processing operation. The block implementing the image processing operation passes its processed data to the next block, which performs a different operation. When the system is not synchronous, intermediate buffers between operations are required. These buffers handle the variations in the data flow. As stated before, building multiple copies of implemented operations and assigning different partitions of the image to each copy can exploit spatial parallelism. A full spatial parallelism can be achieved by building a processor for each pixel. In practice, high image resolution of modern cameras makes this unlikely.

Logical parallelism is the overall parallelism contained in a program, i.e., all the computations that may, according to the semantics of the programming language, be executed in parallel. The logical parallelism within an image processing operation fits into an implementation on the FPGA. This is where most of the image processing algorithms can significantly improve performance. To do so, inner loops are unrolled. Thus, operations are performed in parallel hardware instead of sequentially.

Figure 3 depicts a scheme of a low-level to mid-level vision task implemented over an FPGA. Parallel skills have effects on the construction of the vision system [9]. Implementing a pipelined architecture in an FPGA permits operating at the same frequency pixels are served. Given that power consumption is directly related to the clock frequency, a lower frequency implies a lower power demand by the system. The vision task described in Figure 3 is a typical FPGA approximation to an image processing task.

Normally, the image data goes serially, which fits perfectly in a hardware implementation, especially if it is possible to interface directly to the camera. Anyway, a block (represented in Figure 3 as an I/O interface directly connected to the camera) performs the communication with the camera to receive the flow of pixels from the sensor. This block is responsible for implementing the required protocol (I2C, Camera link, etc.) to communicate with the capture device, configure it and get the image stream. Once configured and initiated the transmission of data, the flow of pixels is driven into the basic image processing block (Point operations green block in Figure 3). This block represents a low-level vision task block. Point operations have widely used in terms of contrast enhancement, segmentation, color filtering, change detection, masking and many other applications. These operations contain the peculiarity that the output pixel depends only on the value of the input's pixel. Figure 4 depicts an example of this kind of module, where a simple contrast enhancement operation to the input image is performed. The constants a and b with two simple math operations over input pixel value provide a new luminance value. This value may exceed the range of representable values. Thus, the result must be clipped. In Figure 4, this clip** operation is performed over the output value. Operating with the input value may improve the performance in a parallel scheme because both, math operations and logic comparisons can be processed concurrently with two processors. The result of this module can be stored in some kind of device memory (DDR2 RAM in Figure 3). This last step is not strictly necessary. A buffer storage is required only for system synchronization.

Point operations are just the basic low-level vision tasks. Normally, from the enhanced image obtained by a point operation module, the computer vision system performs other low-level operations like an image average filter. Filters or blob tracking operations have in common that they need more information besides the value of the pixel being processed. To do so, providing with the necessary architecture to obtain such information (vector structures, intermediate buffers, etc.) is essential. Figure 5 shows some iterations of an image filter computed in an FPGA. On the left the input image is represented for each iteration. The red grid remarks the convolution mask employed to compute the central point in the correspondent iteration, whereas row buffers are depicted in a darker blue and green. Row buffers values are also shown in the right scheme of each iteration. The window mask buffers are represented in orange. Row buffers and window buffers are updated by iterating over the image stream. Parallelism is exploited thanks to these buffers. From window buffer, the simple average can be computed at each iteration. The outcome is a valid pixel value of the output filtered image.

One of the basic low-level vision tasks is an image convolution. This was also one of the first processing image issues to be implemented in an FPGA [6,33,85]. In this work, images provided by a high-resolution sensor were passed to the FPGA. Then, the program embedded on the FPGA applied a convolution with a mask over the image and afterwards transmitted that preprocessed image to a PC. Recently, this basic operation was used to obtain object's edges of an image provided by a spiking system [86]. A spike system also called Address-Event-Representation systems (AER) is a camera sensor that computes internally the movement of the objects in the scene. When a pixel changes its luminance, an event is generated and this is the information transmitted by the camera to the computer vision system. In this paper, Linares-Barranco et al. present two FPGA implementations of AER-based convolution processors. In [87] a design based of FPGA device is described, used in spiking systems for real time image processing. In this case, the AER device described is a synthetic AER retina emulator, used to simulate spiking retina behavior getting as video source a standard video composite source. This design has been synthesized into synchronous and asynchronous FPGA devices to compare their capabilities. Another project related to AER sensors that uses an FPGA is the described in [45]. In this project, the FPGA can perform five different functions: turn a sequence of frames into AER in real time; histogram AER events into sequences of frames in real time; remap addresses using lookup tables; capture and time-stamp events for offline analysis; and reproduce time-stamped sequences of events in real time. In [88] an FPGA is used to develop a real-time high-definition Bayer to RGB converter. Two image processing operations were parallelized in order to obtain this converter: bilinear interpolation and a new median filter scheme that does not require extra memory and is able to work in real time.

Motion estimation represents a highly descriptive visual cue that can be used for applications such as time interpolation of image sequences, video compression, segmentation from motion or tracking. Optical-flow algorithms have been widely employed for motion estimation using FPGAs [32,89–91]. Different approaches to the subject include image block-matching, gradient constraints, phase conservation, and energy models. In [55] Botella et al. present a work developed over a ** a real-time FPGA-based controller. Therefore, the complexity has been reduced, while kee** a great degree of parallelism. Other works use the computational power of the Spartan III FPGAs to achieve vision and image processing tasks [88,121,122].

The Virtex series of FPGAs integrate features that include FIFO logic, DSP blocks, PCI-Express controllers, Ethernet MAC blocks, and high-speed transceivers. In addition to FPGA logic, the Virtex series include embedded fixed function hardware for commonly used functions such as multipliers, memories, serial transceivers and microprocessor cores. ** tool into an impressive solution for those system designs that require a very high level of accuracy, powerful computational capabilities and real parallel execution. From a technological perspective, the industry of FPGA devices has made great strides from simple FPGA chips for prototy** purposes only that included a few hundred logic cells and small blocks of memory to FPGAs with the 28 nm process technology. These last FPGAs include more than two million logic cells, several types of memories and peripheral interfaces. This is the case of the Virtex-7 FPGA devices family from **linx and the Stratix-V from Altera.

As described throughout the paper, research on FPGA-based sensor systems in Spain is well established. The capabilities of the new FPGAs allow providing the sensor systems with different functions such as self-diagnosis, signal processing, communications in a WSN, etc. Currently, the term smart sensor is employed to refer these sensor systems with some kind of “intelligence”. Furthermore, we have described interesting research related with these FPGA-based sensor systems. Within this research, we can mention several works like the implementation of real time signal processing from the obtained sensory information, the use of parallel architectures to process a great quantity of information, to implement and optimize sensor-based controllers in embedded systems, the use of WSNs to process sensory information using different FPGA-based network nodes, etc. Computer vision systems have specially been enhanced by the use of FPGAs. From low-level vision tasks like basic point pixel operations or simple filtering image processing, through mid-level tasks that compute visual features like image moments, until high-level vision tasks where the FPGA allows taking important decisions by processing an image, camera sensors have been considerably enhanced. Different works employ FPGA in sensor systems to implement parallel algorithms in order to process data in a low-power consumption device. These algorithms use FPGA architectures not only to implement simple combinational and sequential circuits, but also to include high-level operations in embedded systems. The optimal implementation of these algorithms using the capabilities of the new FPGAs will suppose an important research field in the near future.