Energy saving for OpenFlow switch on the NetFPGA platform based on queue engineering
© Tran et al.; licensee Springer. 2015
Received: 7 July 2014
Accepted: 23 December 2014
Published: 6 February 2015
Data centers play an important role in our daily activities. The increasing demand on data centers in both scale and size has led to huge energy consumption that rises the cost of data centers. Besides, environmental impacts also increase considerably due to a large amount of carbon emissions. In this paper, we present a design aimed at green networking by reducing the power consumption for routers and switches. Firstly, we design the Balance Switch on the NetFPGA platform to save consumed energy based on Queue Engineering. Secondly, we design the test-bed system to precisely measure the consumed energy of our switches. Experimental results show that energy saving of our switches is about 30% - 35% of power consumption according to variation of input traffic compared with normal Openflow Switch. Finally, we describe performance evaluations.
One of the most important issues that many researches of today society are very concerned for is to save energy consumption in data centers. A research, the Datacenter Dynatmics 2012 Global census shown that the consumed power in data centers between 2011 and 2012 rose significantly to 63% with 38GW (Sverdlik et al. 2011); in which the network devices consumes around from 20% to 30% of this energy (Heller et al. 2010; The Green Grid). On the other hand, the cost of consumed energy on data centers was about 44% of total costs (U.S. Environmental Protection Agency’s Data Center Report to Congress). At the same time, along with huge energy consumption, data centers also emitted a large amount of carbon dioxide.
We design a new module - Power Manager (PM) for Openflow Switch. This function can manage and automatically control to change operating modes of switches including IDLE mode, WORKING mode and SLEEP mode. So our method does not drop packets compared with these methods in (Lombardo et al. 2012) and (Meng et al. 2012) that can drop packets.
We also design a new module - Clock Controller (CC) to control the frequence of switches. Based on the control signals of the PM block, at WORKING mode and IDLE mode, CC maintains the operating frequence at 125 MHz; whereas the frequence is reduced to 0 MHz at SLEEP mode in order to save the consumed energy of Openflow Switch.
We build a test-bed system and precisely measure the consumed energy of the Openflow Switch, based on PCIEXT-64UB kit (Product Specifications and User Manual of PCI-EXT64U/UB and PCIeEXT-16HOT). Experimental results show that energy saving is around 30-35%.
The rest of this paper is organized into following main parts: Section II – Design of Power Manager Block for the OpenFlow Switch. Section III – Design of Clock Controller Block for the Openflow Switch. Section IV – Description of Experimental Results. Section V - Description of Evaluation Testbed System. Finally, section VI - Conclusions.
Design of power manager block for the openflow switch
The method to change the frequence based on queue engineering
Where Tworking and Tsleep denote the time spent in the working mode and sleep mode, respectively. Pworking and Psleep represent the power consumption in each mode.
In our method, when the queue length or the number of packets at the input queue or wait time of a first packet equals to their thresholds (Max Queue Length, Max Packet Number, Wait Timeout, respectively), the switch changes to its normal state with operating frequency of 125 MHz.
Describe the power manager block on the openflow switch
We would like to analyze contributions of different functional blocks embedded in the FPGA chip on the Openflow Switch (Figure 2) (Naous et al. 2008a) (OpenFlow Switch Specification Version 1.1.0 Implemented (Wire Protocol 0x02)). Speaking to the OpenFlow Switch designed in FPGA, NF2_TOP is the top level of switch architecture, where all entities are described and linked to each other. It is highlighted that the main functional block in this design is NF2_CORE; the outside of this block only contains digital clock manager units while the inside has the following implemented functional blocks: CPCI Bus, NF2 DMA, CPU Queue, NF2 Reg grp, User Data Path, NF2 MDIO, and NF2 Mac etc. These blocks are connected to four Ethernet ports through one NIC of Broadcom BCM5464SR that controls the operation of these ports. Besides, the Openflow Switch also includes some other components such as the power modules, memory device, PCI link, CPCI Bridge chip, and so forth.
CPCI Bus block connects to CPCI Chip (Chip spartan 2) ((2014) “NetFPGA-1G-CML™ Board Reference Manual” Revised January 28 2014) by Bus CPI in order that PC can communicate with registers on the Openflow Switch.
NF2 DMA block also connects to CPCI Chip by Bus DMA in order to transmit data to and receive them (packets through Ethernet/IP) from CPU Queue.
CPU Queue: This block includes CPU RX Queue and CPU TX Queue. Packets shall be transferred from the CPU RX queue in the NetFPGA chip to the CPCI chip. The other operation mode, packets shall be transferred from the CPCI chip to the CPU TX queue in the NetFPGA chip, using the pins of the CPCI chip and the NetFPGA chip.(Naous et al. 2008b)
User Data Path (UDP): Includes blocks Input Arbiter, VLAN Remover, Watchdog, Output Port Lookup, VLAN Adder and Output Queue. This block function is to process and forward packets. It collects the packets from 8 inputs (4 CPU queue, and 4 DMA queue) (Moore et al. 2010) and transfers packets to 8 outputs respectively.
NF2 MAC: This block communicates with MAC to Chip PHY to transmit and receive data through Ethernet ports.
Detailed description of the input and output signals on the power manager
Our design contains six small functional blocks: System States, Power Manager Register, Queue Condition, Packets Manager, Registers Manager and Core Clock Enable. The details of these blocks are designed as follows:
Four signal groups
work_udp_grp0 = udp_in_wr| vlan_remover_out_wr| (!opl_in_fifo_empty);
Report UDP is processing and forwarding packets.
work_udp_grp1 = vlan_adder_out_wr | udp_out_wr;
work_cputx_grp = cpu_q_dma_wr_pkt_vld | cpu_q_dma_wr;
Report NF2 DMA is sending data to CPU DMA Queue.
work_cpurx_grp = cpu_q_dma_pkt_avail | cpu_q_dma_rd_rdy;
Report CPU Queue is receiving packets.
Based on signals from Table 1, the System States will report a signal - working_state to inform active states of the User Data Path and the CPU Queue block. If working_state = 1, the system is working and the switch still operates normally. If working_state = 0, packets have finished the packet processing and then we can reduce the operating frequency to 0 MHz to save energy.
Power manager registers
➢ Max Queue Length is the maximum of queue length. When a queue length is equal to this threshold, a switch automatically changes to its normal operating status in order to forward packets.
➢ Max Packet Number is the largest number of packets on the input queue. When the number of packets at the input queue is equal to this threshold, the switch also changes to its normal operating state.
➢ Idle Timeout is the idle time of a switch after completing the packet processing. During this course, there is no traffic flowing through the switch and then it will sleep to save energy.
➢ Wait Timeout is a wait time threshold of packets in input queue. Wait time of a flow starts counting when a first packet received completely on the input queue. If the wait time is equal to the threshold - Wait timeout then the switch also changes to its normal operating state.
This is a main block of the Power Manager. The function of this block is to manage the switch operating states. An input signal of this block named dma_vld_c2n reports the switch will receive packets via the DMA bus. Moreover, other inputs include the mac_grp_core_en that requires from the Queue Condition block, the working_state that informs active states of User Data Path and CPU Queue, and the Idle Timeout that is provided to Power Manager Registers.
WORKING mode: This mode is the normal state of a switch. The operating frequence is provided at 125 MHz in order that the switch can receive and transmit data to other devices on the network.
IDLE mode: This mode is activated when there is no traffic for processing. However, the operating frequence is still maintained at 125 MHz to wait next packets.
SLEEP mode: This mode is activated when there is not any traffic going through the switch. After idle time of the switch equals to a threshold – idle timeout then the operating frequence shall be reduced to 0 MHz in order to save energy.
This block is responsible for reporting the state of registers system on a switch. It receives signals from the registers such as the work_reg_grp signal reports queues of Nf2 Reg Group are processing and the cpci_bus_dv signal reports the switch will receive packets via the PCI bus. Either work_reg_grp = 1 or cpci_bus_dv = 1 Register Manager block will report an output signal – core_clk_reg_en to require the switch to operate normally at 125 MHz.
Core clock enable
Design of clock controller block for the openflow switch
Gtx_clk (125 MHz): Clock used to transmit data to four Ethernet ports. The orange blocks in Figure 10 use this clock.
Grx_clk (125 MHz): Clock used to receive data to four Ethernet ports. The blue block – RX Queue in Figure 10 uses this clock.
Cpci_clk (62.5 MHz): Clock used for communication blocks between NetFPGA and PC through the PCI Bus. The violet blocks in Figure 10 use this clock.
Core_clk: The main clock used for mosts of the functional blocks of the Openflow Switch. The frequence of this clock is 125 MHz to guarantee that the bandwidth on each Ethernet port is 1Gbps. In this paper, we intently impact on this clock to save energy.
DCM provides a wide range of powerful clock management features: Clock De-Skew, Frequency Synthesis, Phase Shifting and General Control Signals. The DCM contains a delay-locked loop (DLL) that can completely eliminate clock distribution delays, hence deskewing the DCM’s output clocks with respect to the input clock. The DLL contains delay elements (individual small buffers) and control logic. The control logic contains a phase detector and a delay line selector. The phase detector compares the incoming clock signal (CLKIN) against a feedback input (CLKFB) and steers the delay line selector, essentially adding delay to the output of DCM until the CLKIN and CLKFB coincide.
BUFGMUX can switch between two unrelated, even asynchronous clocks. With the output signal of Power Manager - core_clk_en, BUFGMUX selects operating frequencies such as 0 MHz or 125 MHz of core_clk for blocks inside a switch. Basically, a High on core_clk_en selects the I0 input (core_clk_in = 125 MHz), and a Low on core_clk_en selects the I1 input (core_clk_in = 0 MHz).
BUFG is a global clock buffer with one clock input (CLK125) and one clock output (CLKFB), driving a low skew clock distribution network.
When the switch is running at the sleep mode, Clock Controller does not provide frequence - core_clk for the inside blocks, which are the green blocks shown in Figure 10. Therefore, these blocks will not operate and sleep to save the consumed energy.
In our method, we can reduce the operating frequence to 0 MHz because we always maintain the performance of Power Manager and Clock Controller. The clock of these blocks is Gtx_clk. This is effective in saving consumed energy of the Openflow Switch.
Design test-bed system
We also use PC1 to generate packets on links with different throughputs. PC2 is connected to the Openflow Switch and receives packets to PC1.
Power Measure Board and Circuit Display are used to read ADC value at test-points 3.3 V and 5.0 V via PCIEXT-64UB (Product Specifications and User Manual of PCI-EXT64U/UB and PCIeEXT-16HOT) and then calculate and display the consumed energy of the Openflow Switch.
Define two modes for balance switch
We can propose and set many different modes for different requirements by changing thresholds such as the Idle Timeout, Max Queue Length, Max Number packet and Wait Timeout. In our experiment, we define two modes for the Balance Switch that is Low Power and Save Power as below:
Low Power mode
The purpose of this mode is to wake up the switch after receiving completely a packet on the input queue. We set thresholds includes: Idle Timeout = 5 clocks = 40 ns, Max Queue Length = 2000 bytes, Max Number packet = 1 packet, Wait Timeout = 12500 clocks = 100us.
Save Power mode
The purpose of this mode is to set parameters as the Idle Timeout, Max Queue Length, Max Number packet and Wait Timeout with the largest thresholds in order to save the largest power. We set these thresholds: Idle Timeout = 5 clocks = 40 ns, Max Queue Length = 5120 bytes, Max Number packet = 127 packets and Wait Timeout = 12500000 clocks = 100 ms.
Measure consumed power of balance switch depending on throughput
Consumed power of normal switch and balance switch
Normal Switch (W)
Low Power mode (W)
Save Power mode (W)
Measure consumed energy of balance switch with different input traffic
Consumed and saved energy of balance switch
Consumed energy (J/15 min)
Saved energy (%)
As shown in Table 3, you can see that the saved energy of switch depends on the input flow. In experiment, the saved energy of Save Power mode is larger than that of Low Power mode. There is around from 30% to 35% energy saved in comparison with Normal switch. In (Thanh et al. 2012) (Heller) shown that the traffic through switches and routers varies according to time the input traffic reaches the peak during the day and falls at night. Therefore, the input traffic has lower level within one day (from 0 hour to 6 hour everyday), energy is saved considerably.
Evaluation testbed system
Evaluate processing time of our method
If interpacket gap TIPG (k) > Tprocess (k) + Idle Timeout then our system will sleep to save energy. Normally, (Tprocess (k) + Idle Timeout) is about μs that is effective in saving energy.
Evaluate QoS for balance switch
Delay time of a packet
➢ Low Power
➢ Save Power
In this case, we can reduce the idle timeout to proper the quality of service.
Packet dropping probability
In our switch, we control operating modes of switches based on queue size, when queue length equals the threshold –Max Queue Lengh then the switch starts in processing and transmiting packets to receivers. Besides, the value of Max Queue Lengh at the Low Power mode and Save Power mode is set at 2000 bytes and 5120 bytes, respectively. While size of an input queue (RX Queue) is 8096 bytes that is larger than the Max Queue Lengh. On the other hand, in (Naous et al. 2008a) presents implementation of the Openflow Switch can hold more than 32,000 exact-match flow entries and is capable of running at line-rate across the four NetFPGA ports. Thus, when the switch is turned on, it runs at line-rate. Our method also does not drop packets compared with the normal switch.
In this paper, we have designed and implemented Balance Switch to reduce power consumption. We have proposed two modes: Low Power and Save Power for the Balance Switch. Additionally, we have also built the test-bed system and have precisely measured the energy of the whole switch saving of about 30-35%. As a result, the saved energy at the Save Power is larger than the Low Power mode. Besides, we have also assessed and shown that our switch do not drop packets compared with the normal Openflow Switch. The results present in this paper can be readily used in reducing energy consumption on data centers. Moreover, the results also contribute to develop on green networking.
In the future, however, we will design other test-bed to study and propose some thresholds such as Idle Timeout, Max Queue Length, Max Number packet and Wait Timeout for different services. Besides, we will also combine with previous methods in (Vu et al. 2014) to measure and evaluate power saving for a large network.
- (2011) “OpenFlow Switch Specification Version 1.1.0 Implemented (Wire Protocol 0x02)”. http://archive.openflow.org/
- (2013) “Product Specifications and User Manual of PCI-EXT64U/UB and PCIeEXT-16HOT” v1r08, Ultraview Corporation. http://ultraviewcorp.com/manuals.php
- (2014) “NetFPGA-1G-CML™ Board Reference Manual” Revised January 28, 2014. http://www.digilentinc.com/
- “Virtex-II Pro and Virtex-II Pro X FPGA User Guide”, UG012 (v4.2) 5 November 2007 in Xilinx company. http://www.xilinx.com/
- Anderson JH, Najm FN “Power Estimation Techniques for FPGAs” (2004) IEEE Transactions on very large scale integration (VLSI) systems, vol. 12, No. 10, October 2004.Google Scholar
- Gibb G, Lockwood JW, Naous J, Hartke P, McKeown N (2008) “NetFPGA – An Open Platform for Teaching How to Build Gigabit-rate Network Switches and Routers” IEEE Transactions on Education.Google Scholar
- Hanay YS, Li W, Tessier R, Wolf T (2012) Saving Energy and Improving TCP Throughput with Rate Adaptation in Ethernet”. In: IEEE International Conference on Communications (ICC), Ottawa, CanadaGoogle Scholar
- Heller B, Seetharaman S, Mahadevan P, Yiakoumis Y, Sharma P, Banerjee S, McKeown N (2010a) ElasticTree: Saving Energy in Data Center Networks. In: NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation, 28 April 2010Google Scholar
- Heller B, Seetharaman S, Mahadevan P, Yiakoumis Y, Sharma P, Banerjee S, McKeown N “ElasticTree: Saving Energy in Data Center Networks” (2010). In: NSDI’10 Proceedings of the 7th USENIX conference on Networked systems design and implementation, USENIX Association Berkeley, CA, USAGoogle Scholar
- Naous J, Gibb G, Bolouk S, McKeown N “NetFPGA: Reusable Router Architecture for Experimental Research” (2008). In PRESTO ’08: Proceedings of the ACM workshop on Programmable routers for extensible services of tomorrow, New York, NY, USA, p 1-7Google Scholar
- Lombardo A, Panarello C, Reforgiato D, Schembra G (2012) Power control and management in the NetFPGA Gigabit Router. Conf. FutureNetw, Berlin, pp 1–8Google Scholar
- Meng W, Wang Y, Hu C, He K, Li J, Liu B (2012) Greening the Internet using Multi-Frequency Scaling Scheme. In: 26th IEEE International Conference on Advanced Information Networking and Applications. IEEE, Fukuoka, JapanGoogle Scholar
- Moore AW, Miller D, Žádník M (2010) Day 2: NetFPGA Cambridge spring school module development and testing. In: NetFPGA Cambridge Spring School. Cambridge University, UKGoogle Scholar
- Naous J, Erickson D, Covington GA, Appenzeller G, McKeown N (2008b) Implementing an OpenFlow switch on the NetFPGA platform. ANCS '08 Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems, New York, NY, USA, pp 1–9Google Scholar
- Thanh NH, Nam PN, Huong TT, Hung NT, Doanh LK, Rastin P (2012) Enabling experiments for energy-efficient data center networks on openflow-based platform. In: The 4th International Conference on Communications and Electronics 2012 (ICCE 2012). IEEE, Hue, VietnamGoogle Scholar
- The Green Grid, “Guidelines for Energy-Efficient Data Centers” (2007). http://www.thegreengrid.org
- U.S. Environmental Protection Agency’s Data Center Report to Congress. Available. http://tinyurl.com/2jz3ft
- Vu TH, Thanh T, Trong VQ, Thanh NH, Nam PN (2014) Energy saving for OpenFlow switch on the NetFPGA platform using multi-frequency. Int J Comput Netw Technol 1:9–15Google Scholar
- Sverdlik Y “Global data center energy use to grow by 19% in 2012” (2011), in DatacenterDynamics. www.datacenterdynamics.com/focus/archive/2011/09/global-data-center-energy-use-grow-19-2012
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.