text:
Open access peer-reviewed chapter

# Linking a Developed Decision Support System with Advanced Methodologies for Optimized Agricultural Water Delivery

By Kristoph-Dietrich Kinzli, David Gensler and Ramchand Oad

Submitted: October 15th 2010Reviewed: March 1st 2011Published: September 6th 2011

DOI: 10.5772/16877

## 1. Introduction

Water is the lifeblood of the American West and the foundation of its economy, but it remains its scarcest resource. The explosive population growth in the Western United States, the emerging additional need for water for environmental uses, and the national importance of the domestic food production are driving major conflicts between these competing water uses. Irrigated agriculture in particular is by far the largest water user of diverted water – 80% country wide and 90% in the Western U.S – and since it is perceived to be a comparatively inefficient user, it is frequently asked to decrease its water consumption. The case of the Middle Rio Grande illustrates the problem very well. The Rio Grande is the ecological backbone of the Chihuahuan Desert region in the western United States, and supports its dynamic and diverse ecology, including the fish and wildlife habitat. The Rio Grande Silvery Minnow is a federally listed as endangered species, and irrigated agriculture in the Middle Rio Grande has come under increasing pressure to reduce its water consumption, while maintaining the desired level of service to its water users.

Irrigated agriculture in the Western United States has traditionally been the backbone of the rural economy. The climate in the American West with low annual rainfall of 10 to 14 inches is not conducive to dry land farming. Topography in the West is characterized by the Rocky Mountains which accumulate significant snowfall, and the peaks of the snowmelt hydrograph are stored in reservoirs allowing for irrigation throughout the summer crop growing season. Of the total available surface water irrigated agriculture uses roughly 80 to 90% (Oad and Kullman, 2006). The combined demands of agriculture, urban, and industrial sectors in the past have left little water for fish and wildlife. Since irrigated agriculture uses roughly 80 to 90% of surface water in the West, it is often targeted to decrease diversions. Due to wildlife concerns and demands from an ever growing urban population, the pressure for flow reductions on irrigated agriculture increases every year. In order to sustain itself and deal with external pressure for reduced river diversions irrigated agriculture has to become more efficient in its water consumption. This chapter focuses on research regarding improving water delivery operations in the Middle Rio Grande Conservancy District system through the use of a decision support system linked with advanced infrastructure, methodologies, and technology.

The Middle Rio Grande (MRG) Valley runs north to south through central New Mexico from Cochiti Reservoir to the headwaters of Elephant Butte Reservoir, a distance of approximately 175 miles. The MRG Valley is displayed in Figure 1.

The valley is narrow, with the majority of water use occurring within five miles on either side of the river. The bosque,or riverside forest of cottonwood and salt cedar, is supported by waters of the Rio Grande; the bosque being surrounded by widespread irrigated farming. The Cities of Albuquerque, Rio Rancho, Belen and several smaller communities are located in and adjacent to the MRG Valley. Although the valley receives less than 10 inches of rainfall annually, it supports a rich and diverse ecosystem of fish and wildlife and is a common outdoor resource for communities in the region. Water supply available for use in the MRG Valley includes: native flow of the Rio Grande and its tributaries, allocated according to the Rio Grande Compact of 1938; San Juan-Chama (SJC) project water, obtained via a trans-mountain diversion from the Colorado River system; and groundwater. Water is fully appropriated in the MRG Valley and its utilization is limited by the Rio Grande Compact, which sets forth a schedule of deliveries of native Rio Grande water from Colorado to New Mexico and from New Mexico to Texas (Rio Grande Compact Commission, 1997). Water demand in the MRG Valley includes irrigated agriculture in the Middle Rio Grande Conservancy District (MRGCD) Indian Lands, and municipal and industrial consumption. In addition to these demands, there are significant consumptive uses associated withriparian vegetation, reservoir evaporation, and river flow targets associated with two federally-listed endangered species, the Rio Grande silvery minnow (Hybognathus amarus), and the southwestern willow fly catcher (Empidonax traillii extimus) (USFWS, 2003). Figure 2 displays the Rio Grande silvery minnow and the southwestern willow fly catcher.

This chapter will focus on research for improving water delivery operations in irrigation systems through the innovative use of water delivery decision support systems (DSS) linked with SCADA technology and infrastructure modernization.The chapter will present decision support modeling of irrigation systems in a broad sense and present the model development and structure of a decision support system developed specifically for the Middle Rio Grande Conservancy District to provide for more efficient and effective management of water supplies, and more timely delivery of irrigation water to agricultural users. The developed DSS will be presented in detail and all three modules that function together to create water delivery schedules will be explained and examined. The chapter will address the development of the DSS and will also present the utility of linking the developed DSS to the SCADA (Supervisory Control And Data Acquisition) system and automated structure network that the Middle Rio Grande Conservancy District utilizes to manage water delivery. Linking the DSS and SCADA allows water managers to implement DSS water delivery schedules at the touch of a button by remotely controlling water delivery structures. Linking the DSS water distribution recommendations with the MRGCD SCADA provides a simple and effective medium for managers to implement DSS recommended water delivery schedules. Additionally, the combination of both programs allowed for real-time management that converged river diversions and the water required by the DSS. As the demand for water increases globally throughout the next decades, water managers will be faced with significant challenges. The use of decision support systems linked with SCADA and automated structures presents an advanced, efficient, and innovative approach for meeting water management challenges in the future.

## 2.Middle Rio Grande Conservancy District

The Middle Rio Grande Conservancy District (MRGCD) may be one of the oldest operating irrigation systems in North America (Gensler et al. 2009). Prior to Spanish settlement in the 1600s the area was being flood irrigated by the native Pueblo Indians. At the time of Albuquerque’s founding in 1706 the ditches that now constitute the MRGCD were in already existence and were operating as independent acequia (tertiary canal) associations (Gensler et al. 2009). Acequias consisted of farmer groups that maintained individual irrigation canals. The acequia system was introduced to the MRG Valley by Spanish settlers. In acequia communities, each farmer was responsible for maintaining a certain length of canal and would in return receive irrigation water. The use of irrigation water was managed by an elected mayordomo (Gensler et al. 2009).

Irrigated agriculture in the MRG Valley reached its greatest extent in the 1880s, but thereafter underwent a significant decline partially caused by an overabundance of water. By the early 1920s inadequate drainage and periodic flooding resulted in water logging throughout the MRG Valley. Swamps, seeps, and salinization of agricultural lands were the result. In 1925, the State of New Mexico passed the Conservancy Act, which allowed for the creation of the MRGCD, by combining 79 independent acequia associations into a single entity (Gensler et al. 2009; Shah, 2001). Over the next twenty years the MRGCD provided benefits of irrigation, drainage, and flood control; however, by the late 1940’s, the MRGCD was financially unstable and further rehabilitation of structures was required. In 1950, the MRGCD established a 50-year contract termed the Middle Rio Grande Project with the USBR to provide financial assistance, system rehabilitation, and system improvement. System improvements and oversight from the USBR continued until 1975 when the MRGCD resumed operation and maintenance of the system. The loan from the USBR to the MRGCD for improvements and operational expenses was repaid in 1999 (Shah, 2001). Currently the MRGCD operates and maintains nearly 1,200 miles of canals and drains throughout the valley in addition to nearly 200 miles of levees for flood protection.

Water use in the MRG Valley has not been adjudicated but the MRGCD holds various water rights and permits for irrigation (Oad and Kullman, 2006). Some users in the MRGCD hold vested water rights that are surface rights claimed by land owners who irrigated prior to 1907 (SSPA, 2002). Most water users in the MRGCD receive water through state permits held by the MRGCD. In 1930, the MRGCD filed two permits (#0620 and #1690) with the Office of the State Engineer that allow for storage of water in El Vado reservoir (180,000 acre feet capacity), release of the water to meet irrigation demand, and diversion rights from the Rio Grande to irrigate lands served by the MRGCD. The permits allow the MRGCD to irrigate 123,000 acres although only about 70,000 acres are currently served (MRGCD, 2007). This acreage includes roughly 10,000 acres irrigated by pueblo farmers. The MRGCD charges water users an annual service charge per acre to operate and maintain the irrigation system. In 2000 the MRGCD charged $28 per acre per year for the right to irrigate land within the district (Barta, 2003).The MRGCD services irrigators from Cochiti Reservoir to the Bosque del Apache National Wildlife Refuge. An overview map of the MRGCD is displayed in Figure 3. Irrigation structures managed by the MRGCD divert water from the Rio Grande to service agricultural lands that include both small urban landscapes and large scale production of alfalfa, corn, vegetable crops such as chile, orchards, and grass pasture. The majority of the planted acreage, approximately 85%, consists of alfalfa, grass hay, and corn. In the period from 1991 to 1998, USBR crop production and water utilization data indicate that the average irrigated acreage in the MRGCD, excluding pueblo lands, was 53,400 acres (21,600 ha) (SSPA, 2002). Analysis from 2003 through 2009 indicates that roughly 50,000 acres (20,200 ha) are irrigated as non-pueblo or privately owned lands and 10,000 acres (4,000 ha) are irrigated within the six Indian Pueblos (Cochiti, San Felipe, Santo Domingo, Santa Ana, Sandia, and Isleta). Agriculture in the MRGCD is a$142 million a year industry (MRGCD, 2007). Water users in the MRGCD include large farmers, community ditch associations, six Native American pueblos, independent acequia communities and urban landscape irrigators. The MRGCD supplies water to its four divisions -- Cochiti, Albuquerque, Belen and Socorro -- through Cochiti Dam and Angostura, Isleta and San Acacia diversion weirs, respectively (Oad et al. 2009; Oad et al. 2006; Oad and Kinzli, 2006). In addition to diversions, all divisions except Cochiti receive return flow from upstream divisions.

Return flows are conveyed through interior and riverside drains. From the drains, excess water is diverted into main canals in the downstream divisions for reuse or eventual return to the Rio Grande. Drains were originally designed to collect excess irrigation water and drain agricultural lands, but are currently used as interceptors of return flow and as water conveyance canals that allow for interdivisional supply.

During the later part of the irrigation season, the MRGCD operates using released storage water from the high mountain reservoirs of El Vado, Heron, and Abiquiu. Water stored in these reservoirs consists of snowmelt runoff captured during the early summer months and water from the San Juan-Chama trans-mountain diversion. These reservoirs are located 98 river miles upstream and water delivery is associated with a significant time lag, which can approach seven days to reach the southern portion of the district.

The Cochiti Division consists primarily of Native American pueblo land. The pueblo and non-pueblo lands in the Cochiti Division are managed by four MRGCD ditch-riders and represent 4800 acres. The Albuquerque Division services many small urban irrigators, but also provides irrigation water to pueblo irrigators at the northern and southern boundaries of the division. The Albuquerque Division is managed by one water master and 12 ditch-riders to oversee the complex irrigation scheme. The Albuquerque Division acreage is 10,300acres. The Belen Division is the largest in terms of overall service area with a total irrigated acreage of 31,400 acres. Irrigation in the Belen Division is comprised of large farms, pueblo irrigators, and urban water users. In Belen the MRGCD employs one water master and 11 ditch-riders. The Socorro Division consists of mostly large parcel irrigators with a total irrigated acreage of 13,600 acres. Water distribution in Socorro is straightforward when compared to the Albuquerque and Belen Division, and is managed by one water master and four ditch-riders. Water availability in Socorro can become problematic since the division depends on return flows from upstream users.

Water in the MRGCD is delivered in hierarchical fashion; first, it is diverted from the river into a main canal, then to a secondary canal or lateral, and eventually to an acequia or small ditch. Figure 4 displays the organization of water delivery in the MRGCD. Conveyance canals in the MRGCD are primarily earthen canals but concrete lined canals exist in areas where bank stability and seepage are of special concern. After water is conveyed through laterals or acequias it is delivered to the farm turnouts, with the aid of check structures when necessary. Once water passes the farm turnout it is the responsibility of individual farmers to apply water and it is applied to fields generally using basin or furrow irrigation techniques.

Water in the MRGCD is delivered to users through management and administration provided at a central office and four divisional offices. The central office in Albuquerque provides oversight of the four divisional offices and assesses service charges for water use. Each division office includes administrative, field and equipment maintenance, and water operations personnel. Water operations in each division are managed by a division manger, a water master, and ditch-riders in each division. The division managers oversee all aspects of the division, and water masters coordinate ditch-rider operations. Ditch-riders are responsible for managing water delivery in a particular service area. Ditch-riders have anywhere from 250 to 900 irrigators they are responsible for in their service area. Check structures and head gates are controlled by ditch-riders to deliver irrigation water in their service area to meet user demand. Water delivery and water use conditions are monitored by ditch-riders through the physical riding of ditches and through communication with water users. Ditch-riders generally cover all of the ditches in their service area twice a day and are in constant contact with water users via cellular phones. Ditch-riders are on call 24 hours a day to deal with emergencies and water disputes, in addition to daily operations.

Water delivery in the MRGCD is not metered at individual farm turnouts. To determine water delivery the ditch-riders estimate the time required for irrigation. The historic practice in the MRGCD was to operate main canals and laterals as full as possible throughout the entire irrigation season. This practice provided for flexible and reliable water delivery with minimal managerial and financial ramifications; also known as on-demand water delivery. On-demand or continuous water delivery, however resulted in large diversions from the Rio Grande. During the past decade, the MRGCD has voluntarily reduced river diversions by switching to scheduled water delivery. The drawback to this approach is the increased managerial involvement and the overall cost of water delivery. To aid with the operational and managerial challenges posed by scheduled water delivery, the MRGCD has developed and implemented a Decision Support System (DSS) to aid in facilitating scheduled water delivery. Additionally, the MRGCD has begun to replace aging water delivery infrastructure with automated control gates that allow for precise control of canal flow rates, a requirement for scheduled water delivery.

## 3. Decision support system for the Middle Rio Grande Conservancy District

The New Mexico Interstate Stream Commission and the MRGCD, in cooperation with Colorado State Univeristy have sponsored a research project from 2003 to 2010 to develop a decision support system (DSS), to model and assist implementation of scheduled water delivery for efficiency improvements in the MRGCD’s service area. A DSS is a logical arrangement of information including engineering models, field data, GIS and graphical user interfaces, and is used by managers to make informed decisions. In irrigation systems, a DSS can organize information about water demand in the service area and then schedule available water supplies to efficiently fulfill the demand.

The conceptual problem addressed by a DSS for an irrigation system, then, is: how best to route water supply in a main canal to its laterals so that the required river water diversion is minimized. The desirable solution to this problem should be “demand-driven”, in the sense that it should be based on a realistic estimation of water demand. The water demand in a lateral canal service area, or for an irrigated parcel, can be predicted throughout the season through analysis of information on the irrigated area, crop type and soil characteristics. The important demand concepts are: When is water supply needed to meet crop demand (Irrigation Timing), How longis the water supply needed during an irrigation event (Irrigation Duration), and How often must irrigation events occur for given service area (Frequency of Irrigation). Decision support systems have found implementation throughout the American West and are mostly used to regulate river flow. Decision support systems on the river level are linked to gauging stations and are used to administer water rights at diversions points. Although decision support systems have proved their worth in river management, few have been implemented for modeling irrigation canals and laterals and improving water delivery (NMISC, 2006). The followingsection will focus on developing a decision support system capable of modeling flow on a canal and lateral level, with the overall goal of efficient irrigation water delivery.

The DSS has been formulated to model and manage water delivery in the MRGCD. The DSS was designed to optimize water scheduling and delivery to meet crop water demand, and, specifically, to aid in the implementation of scheduled water delivery (Oad et al. 2009; NMISC, 2006). The DSS consists of three elements, or modules:

• A water demand module that calculates crop consumptive use and soil moisture storage, aggregated by lateral service area;

• A water supply network module that represents the layout of the conveyance system, main canal inflow, conveyance system physical properties, and the relative location of diversions for lateral service area; and,

• A scheduling module that routes water through the supply network to meet irrigation demand using a mass-balance approach and based on a ranking system that depends on the existing water deficit in the root-zone.

A Graphical User Interface (GUI) was designed to link the three modules of the DSS together allowing users to access data and output for the system. A schematic of the three modules and the way that they relate within the DSS framework is shown in Figure 5. Figure 6 displays a simplified view of the DSS. GIS information and data obtained from the MRGCD were used to develop input for both the water demand and the supply network modules. Some of the input is directly linked through the GUI and some is handled externally (NMISC, 2006).

The DSS has two modes of operation: planning mode and operation mode. In planning mode, the user inputs an anticipated cropping pattern for the season and other related data, and the model calculates the required main canal diversions as a function of time based on the calculated demand. In operation mode, the user inputs the available main canal flows, and the model recommends a water delivery schedule for the lateral canal service areas within the main canal that optimizes the use of available water (Oad et al. 2009; NMISC, 2006).

### 3.1. Model structure

The MRGCD DSS was developed using three modules. The three modules that comprise the MRGCD DSS are the water demand module, the supply network module, and the irrigation scheduling module. The following subsections describe the three modules in detail.

#### 3.1.1. Water demand module

The water demand module of the MRGCD DSS is implemented through the Integrated Decision Support Consumptive Use (IDSCU) model code or the ET Toolbox. The IDSCU model was developed over a period of years by Colorado State University. The IDSCU code consists of a GUI written in Visual C++ and program calculations implemented with FORTRAN (Oad et al. 2009; NMISC, 2006). The ET Toolbox was developed specifically for the Middle Rio Grande and calculates the crop consumptive use using a system of weather stations throughout the valley. Using either the IDSCU or the ET Toolbox, the following variables are determined in the water demand module:

• Crop consumptive use (CU);

• Crop irrigation requirement (CIR); and,

• Readily available (soil) moisture (RAM), as a capacity.

CIR and RAM as a capacity, are used in the supply network module. Required data for the water demand module, the source of these data, and the spatial unit for which the information is aggregated, are shown in Table 1.

### Table 1.

Required Data for Water Demand Module (NMISC, 2006)

The Penman-Monteith method suggested by FAO is used to determine the crop consumptive use for the system when using the IDSCU. The Penman-Monteith equation is discussed in detail in Chapter 3. When using the IDSCU, weather data required for the calculation are obtained from the USBR ET Toolbox (USBR, 2003). The USBR ET Toolbox obtains weather data from 16 weather stations throughout the MRG Valley. In the water demand module, crop coefficients using growing degree days are combined with the Penman-Monteith based ET to obtain a consumptive use for each crop type throughout the growing season. Crop coefficients using growing degree days were obtained from the New Mexico Climate Center and are based on work done by (King et al. 2000). The water demand module performs calculations to obtain a spatially-averaged CU at the lateral service area level using the distribution of crop types within each service area. When the ET Toolbox is used in the demand module, the consumptive use is already calculated within the ET Toolbox using grid cells throughout the valley.

The Penman-Monteith-based CU is adjusted to account for effective precipitation, and is used to obtain a CIR for each lateral service area. Effective precipitation is calculated according to the Soil Conservation Service Method and is subtracted from the Penman-Monteith-based CU. The CIR is calculated on a daily basis, corresponding to the water needed to directly satisfy crop needs for all acres in a lateral service area. The CIR for each lateral is subsequently passed to the supply network module, where it is divided by an efficiency factor to obtain a lateral delivery requirement or LDR (Lateral Delivery Requirement)(NMISC, 2006).

RAM is determined by first calculating the total available water (TAW) per unit area, which is the available water-holding capacity (AWC) for a given soil type, multiplied by the root-zone depth. The calculation of TAW is displayed in Equation 1 (Oad et al. 2009; NMISC, 2006).

TAW = AWC * root zone depthE1

Once the TAW is calculated, a management allowed depletion factor (MAD) for each crop type is applied to determine the RAM. The values employed for MAD in the Water Demand Module are crop specific and range from 30% to 60% (NMISC, 2006; FAO, 1998). RAM is calculated using Equation 2.

Based on acreage, crop type, and soil type within each lateral service area, a value for RAM is calculated. The RAM calculated in this context represents a storage capacity to be filled and depleted over several irrigation cycles during the course of the irrigation season (NMISC, 2006). During each irrigation event, it is expected that an amount of water equal to the RAM will be stored in soils. Then, as crops utilize water, the RAM will become depleted (Oad et al. 2009; NMISC, 2006).

#### 3.1.2. Supply network module

The supply network module consists of data relating to water supply, water demand, and physical information relating to the conveyance network. It represents the layout of the conveyance system, its physical properties, and the relative location of diversions from the network to individual lateral service areas (Oad et al. 2009; NMISC, 2006). The layout of the conveyance system is specified through a user-designed link-node network. Through the DSS GUI, a user can drag and drop in different types of nodes such as inflows, demands, and return flow nodes into the program (NMISC, 2006). Developing the link-node network is straightforward and can be completed by ditch-riders and water master after some short training. The link-node network represents the connections between canals or laterals and water demands for each service area. The GUI also contains information on ditch-rider service area, and includes photographs and locations of all major water control structures. Figure 7 provides a view of the link-node network.

The supply network module obtains the CIR and the RAM from the water demand module, and then associates these parameters with a demand node on the link-node network (Oad et al. 2009; NMISC, 2006). An efficiency factor is applied to the CIR to account for on-farm application efficiency and to account for conveyance losses within the service area. This results in a lateral delivery requirement (LDR) that applies to all acreage served by a given lateral. The efficiency factor can be specified by the user for each lateral service area (NMISC, 2006).

The link-node network representing the layout of the conveyance system within the supply network module consists of inflow nodes, stream nodes, and return-flow nodes, connected by links that represent canals and laterals (NMISC, 2006). Inflow nodes contain information about inflow volume. The user may either type the inflow values into the inflow node GUI, or import them from an inflow table. Stream nodes require the user to provide information about individual lateral service areas, including turnout capacity, the number of days needed to completely irrigate the service area, the lateral service area efficiency factor, which days of the week the service area can be irrigated, the minimum flow required in the canal on a daily basis, and to which “sub-system” a given stream segment belongs (NMISC, 2006). Sub-systems are used to preferentially rank laterals that should be irrigated together.

Return flow nodes are used to capture and route drain flow. The percentage of water applied to farms that can be captured by drains is set by the user for the entire project. Each drain has a daily schedule that indicates how much of that return flow is available as inflow in the successive days (NMISC, 2006). Using the information specified, the supply network module calculates a demand-based flow rate associated with each diversion to a lateral service area. The flow rate is calculated as the LDR divided by the irrigation duration. However, if the calculated flow rate exceeds the lateral capacity specified by the user, then the flow rate is set at the user-input lateral capacity. In either case, when irrigation occurs, as determined by the scheduling module, the amount of water removed from the stream link and delivered to the lateral service area for the daily time step is set equal to the volume of water that would be delivered at this flow rate over a one-day period (NMISC, 2006). This volume of water, or time-adjusted portion of LDR, is then reduced by the efficiency factor and added to the daily RAM for the service area. If the LDR is not fully delivered in the one-day irrigation, the irrigation may continue into subsequent days depending on the remaining need within the given lateral, the need of other laterals, and the assigned ranking system in the scheduler (NMISC, 2006).

#### 3.1.3. Irrigation scheduling module

The irrigation scheduling module uses the information provided by the water demand and supply network modules to schedule water deliveries to meet crop demand at the lateral level. The irrigation scheduling module calculates and displays a schedule for the laterals on a given main canal. This schedule indicates how many laterals can be run at a time, how long each lateral should run, and how often (NMISC, 2006). Figure 8 displays an irrigation schedule created using the DSS. The module can also create monthly water delivery calendars for individual laterals.

The module is set to run on a daily time step. The irrigation scheduling module calculates the daily irrigation schedule using mass balance equations and a linear programming solver. The module writes out an input file for the solver, executes the solver, and reads the solver output. In the present model version, the module displays results in tabular form. Mass balance calculations used to schedule irrigation timing and duration for lateral canal service areas are based on the consideration that the farm soil root-zone is a reservoir for water storage, for which irrigation applications are inflows and CIR is an outflow (NMISC, 2006). The mass balance approach is displayed in Equation 3:

RAMt+1=It+1Ot+1+RAMtE3

Where I is inflow, O is outflow (which includes return flow), and t is time.

A linear programming approach is used to calculate flows to the service areas by posing the problem as a minimum cost flow optimization. The model uses the projected number of days until the soil moisture storage is depleted using a reverse-ranking system to prioritize the need for irrigation among service areas (NMISC, 2006). Based on observations of the water delivery operations, it appears that water delivery can be changed from one set of laterals to another set within one day.

### 3.2. Model programming

Model programming for scheduling water supplies to lateral service areas is described in the following section. To obtain the optimum irrigation schedule a linear programming approach is utilized. Linear programming is a method for optimizing a quantity that is defined with a mathematical expression or objective function. Constraints on variables within the objective function are also specified and must be satisfied in determining the optimum solution. This process favors water delivery to laterals with more immediate water needs, and minimizes delivery to laterals that have sufficient water (Oad et al. 2009; NMISC, 2006).

For illustration purposes, the scheduling problem is described using a hypothetical network. Figure 9 shows a simple irrigation network with a main supply canal and a number of laterals that represent crop water demand. The problem is similar to a transportation problem, where the service areas are demand nodes and the inflows are supply nodes (NMISC, 2006). Links are created between nodes where water can be routed. In a transportation problem, the supply needs to equal the demand; in this case, however, both under-supply (excess demand) and excess supply are possible. Therefore, to ensure that the system balances, a “dummy” source node is added that computes the water shortage in the event the system is water-short. Note that in a water-rich scenario, the dummy node is not used because it calculates only water shortage (NMISC, 2006)

The scheduling problem is cast as a minimization problem, for which the goal is to provide water to the nodes with the greatest need for water (NMISC, 2006). This is achieved through the use of a ranking system based on water need, the use of water delivery from the dummy supply, and a set of constraints that capture mass balance conditions through the stream network. The objective function is displayed by Equation 4.

Minimize Z = MPD0XD0+ MPD1XD1+ MPD2XD2+ MPD3XD3E4

In this equation Z is the sum of a modified priority (MP) multiplied by the amount of supply (X) from the dummy supply to each demand node. The subscripts refer to the nodal points between which flow occurs, i.e., X D-1 refers to flow from the Dummy supply to Check 1, and MP D-1 refers to the modified priority of demand to be satisfied at Check 1 from the Dummy supply node. The MP value reflects the need-based ranking system where demand nodes with lower available soil moisture are favored for irrigation (NMISC, 2006). The objective function is solved in conjunction with a system of mass balance equations representing the actual water (and dummy water) delivered to demand nodes, along with other physically-based constraints.

The variables in the objective function represent the links in the network between the dummy supply and the demand nodes. The coefficient of each variable represents the flow “cost” of that link. In other words, delivery of water to a node without a need for water results in a higher “cost”. As further discussed below, the ranking system has been assigned such that minimization of this objective function will result in minimization of water delivery to demand nodes that already have sufficient RAM (NMISC, 2006).

Constraints on the objective function solution reflect the mass balance relationships throughout the link-node network and the capacity limits on flow (NMISC, 2006). A mass-balance constraint is created for each node (including the dummy) that establishes the inflow and outflow to that node. The coefficients of the variables for each constraint (each row) are represented as a matrix, with a column for every variable in the objective function and a row for every node (NMISC, 2006). Inflows are represented as negative values and outflows as positive values. Outflow coefficients are always one, and inflow coefficients equal the conveyance loss of the connection.

The objective function is subject to the following constraints:

X01<= C01   X12<= C12  X23<= C23   All Xij>= 0E5

The variables used are:

• I is the total available inflow

• Xi-j is the flow in a canal reach between points i and j

• Ci-j is the maximum capacity of the canal reach between points i and j

• D refers to a dummy supply node that is used to force the demands and supplies to balance. The subscript 0 refers to the inflow node, and subscripts 1, 2, 3, … refer to nodal points, typically located at check structures

• Li is the conveyance loss between in the canal reach

• Ri is the demand (water requirement) at the nodal point indicated by the subscript (can be zero if not associated with a lateral diversion point)

For example, the third row refers to activity at check 1. There is an inflow from the headgate (-L1X0-1), and it is given a negative sign since by convention all inflows are negative. The conveyance loss is represented by the coefficient L1. There is an outflow to check 2 (+ X1-2) (positive sign, since by convention all outflows are positive). To ensure that the system balances, there is also an inflow from the dummy source (- XD-1). Because this node represents a demand, the solution for this row is constrained to be exactly the demand (R1). If a node represented a source, then the solution for the row would be constrained to fall between zero and the inflow. That allows the use of less than the total amount of water available if the demands are less than the supplies, or if at some point in the network the capacity is insufficient to route the inflow (NMISC, 2006). The first row in the constraint equations represents this type of node.

The conveyance loss factor specified in the supply network module is a fractional value of flow per mile. The conveyance loss (L) to be applied in the mass balance equation is calculated by subtracting the fractional value from one and raising it to the power of the number of miles of the canal segment between nodes. For example, a 3-mile reach with a 0.015 conveyance loss factor would have a loss of [1 – (1-0.015)3], or a loss of 0.0443 of the in-stream flow to this reach.

The ranking system used to derive the modified priority (MP) values for the objective function is a two-step process, involving assignment of a priority (P) based on the irrigation need at demand nodes, and then a modified priority that effectively reverses the ranking so that nodes with the least need are the preferred recipients for dummy water (NMISC, 2006). This results in the actual available water being delivered to the demand nodes with highest irrigation need. First, a priority (P) is assigned to each of the demand nodes, with smaller values indicating higher needs for irrigation. The priority is based on the number of days until the service area runs out of RAM (NMISC, 2006). If the service area is not being irrigated, 100 is added to the priority, which forces the system to favor areas being irrigated until the RAM is full again. Subsystems were added to give priority to remaining canals within a group on the assumption that if one canal service area in a subsystem is being irrigated then it is desirable that the remaining canal services areas in the same group be irrigated as well. If a service area is not being irrigated, but is in a subsystem that is being irrigated, 50, rather than 100, is added to the priority. This makes it a higher priority than service areas that are not being irrigated but are not in the subsystem. Normally a service area is irrigated only once during a schedule. However, when excess water is available, service areas in need of water are added back into the scheduling algorithm with a higher priority. The ranking system is implemented by modifying the priorities with respect to the dummy connections, effectively reversing the priorities (NMISC, 2006). Currently the modified priority (MP) for the “dummy -> node x” connection is 100,000/Px. For example, if the node has a priority of 105, then the priority assigned to the connection is 100,000/105 or 952.38. This will force dummy water to be delivered first to the lower priority nodes, leaving real water for the higher priority nodes. The modified priority (MP) values are represented by the MP variables in the objective function. The linear programming software utilized in the DSS is a package called GLPK (GNU Linear Programming Kit).

## 4. Supervisory Control and Data Acquisition (SCADA) technology and infrastructure modernization

One of the most critical components to realizing schedule water delivery and utilizing real time recommendations from the DSS to deliver water was the MRGCD SCADA network and infrastructure modernization. The MRGCD SCADA system can be broken into the following components:

• Flow Measurement Structures

• Automated Control Structures

• Instrumentation

• Telemetry

• Software

### 4.1. Flow measurement structures

Water measurement is the single most important component of the MRGCD's SCADA system, since all operational decisions require sound knowledge of available water supplies and the demand throughout the system. In 1996, crisis struck the MRGCD in the form of drought, endangered species flow requirements, and development of municipal water supplies. At the time, the MRGCD was operating only 15 gauges on 1200 miles of canals. The following year, MRGCD officially embarked upon itsmodernization program. The construction of new flow gauges was the first step in this program. New gauges were constructed at key points in the canal system, notably at diversion structures and at return flow points. Along with the increase in numbers of gauging stations, efforts were made to improve the quality of measurement. Open channel gauging sites with no control structures gave way to site specific measuring structures. A variety of flow measurement structures were built in the MRGCD and include sharp crested weirs, broad crested weirs, adjustable weirs and Parshall flumes. Currently, the MRGCD is operating over 100 gauges. Figure 10 displays a broad crested with telemetry utilized for flow measurement

### 4.2. Automated control structures

With the advent of better data collection, it became apparent that automated control was necessary. Data from gauges revealed that many operational problems occurred because canal operators could not be physically present at all times. Automation followed shortly thereafter with an experimental effort at a wasteway that had been fitted with an automated Langemann gate for water measurement, and was therefore a practical starting point. The MRGCD built the electronic controller and created the control software for this first automated gate, borrowing heavily from Bureau of Reclamation experience in Utah. Success with the first automated structure led to installation of over 60 additional automated structures. Most of MRGCD’s recent automation efforts have involved the installation of Langemann overshot gates (Aqua Systems, 2006). The majority of these can be easily retrofitted to existing structures, though some involve the construction of new check or heading structures. The Langemann Gate has the capability to maintain a constant upstream water level as a check structure or it can provide a constant flow rate to downstream users (Figure 11). The Langemann gate is equipped with solar panels to power both gate operation and telemetry units. The gates employ integrated electronic controllers built around IC Tech radio terminal units (RTU’s) and Aqua Systems 2000 software. Langemann gates in the MRGCD are used as checks, turnouts, spillways, and diversion structures.

### 4.3. Instrumentation

Flow measurement and automated control must include some level of instrumentation. In the 1930’s, a float in a stilling well driving a pen across a revolving strip of paper was adequate. In fact, at the beginning of modernization efforts, the MRGCD was still using 15 Stevens A-71 stage recorders. Diversions into the canal system were only known after the strip charts were collected and reduced at the end of the irrigation season. Modernization meant a device was needed to generate an electrical or electronic output that could be digitally stored or transmitted. Initially, shaft encoders were used for this purpose, providing input for electronic data loggers. Experimentation with submersible pressure sensors soon followed, and these have been adopted, although a number of shaft encoders are still in use. Sonar sensors have been used satisfactorily at a number of sites and recently compressed air bubblers have been utilized. Different situations call for specific sensor types and sensors are selected for applications where they are most appropriate.

### 4.4. Telemetry

Data from electronic data-loggers was initially downloaded manually and proved to be only a minimal improvement over strip chart recording, though processing was much faster. To address data downloading concerns telemetry was adopted to bring the recorded data back to MRGCD headquarters at regular intervals (Figure 12). MRGCD's initial exposure to telemetry was through the addition of GOES satellite transmitters to existing electronic data loggers. This method worked, but presented limitations. Data could only be transmitted periodically, and at regularly scheduled intervals. Of greater consequence was that the GOES system, at least as used by MRGCD, was a one-way link. Data could be received from gauging stations, but not sent back to them. As experiments with automation progressed, it was clear that reliable 2-way communication would be a necessity. To address the rising cost of phone service, experiments with FM radio telemetry were conducted. These began as a way to bring multiple stream gage sites to a central data logger, which would then be relayed via GOES to MRGCD. First attempts with FM radio were not encouraging; however a successful system was eventually developed. As this use of FM radio telemetry (licensed 450 mHz) expanded, and knowledge of radio telemetry grew, it was soon realized that data could be directly transmitted to MRGCD headquarters without using the GOES system. Full conversion to FM radio based telemetry also resulted in very low recurring costs.

The shift to FM radio produced what is one of the more unique features of the MRGCD SCADA system. The data link proved so reliable, that there was no longer a need to store data on site, and the use of data loggers was mostly discontinued, the exception being weather stations. In effect, a single desktop computer at the MRGCD headquarters has become the data-logger for the entire stream gauge and gate system, being connected to sensors in the field through the FM radio link. Three repeater sites are used to relay data up and down the length of the valley, with transmission being up to 75 miles. Also, this has the benefit of being a 2-way link, so various setup and control parameters can be transmitted to devices along the canals. This 2-way link allowed for the system to be linked to the DSS in order to improve water delivery operations. The MRGCD telemetry network consists exclusively of Control Design RTU’s. Several different types of these units are used, depending on the application. The simplest units contain only a modem and radio, and transmit collected and processed weather station data from Campbell Scientific CR10X dataloggers.

The majority of the RTU’s contain a modem, radio, and an input/output (I/O) board packaged into a single unit. Sensors can be connected directly to these and read remotely over the radio link. A variety of analog (4-20ma, 0-20ma, 0-5v) and digital (SDI-12, RS-485) output devices can be accommodated this way. Another type includes a programmable (RP-52 BASIC) controller in the modem/radio/(I/O) unit. This style is used for all automatic control sites and places where unusual processing of sensor outputs such as averaging values, combining values, or timed functions, are required. At the present time, the MRGCD telemetry network gathers data from over 100 stream flow gages and 18 ag-met stations, and controls 60 automated gates (Figure 12).

### 4.5. Software

Measurement, automation, and telemetry components were developed simultaneously, but largely independent of one another. While each component functioned as expected, components did not exist as a harmonious whole, or what could truly be called a SCADA system. The missing component was software to tie all the processes together. There are a variety of commercially available software packages for such use and MRGCD experimented with several. Ultimately, the MRGCD chose to purchase the commercial software package Vsystem and to employ the vendor Vista Controls to develop new features specific to the control of a canal network. Installation and setup was done by the MRGCD.

This system, known as the Supervisory Hydro-data Acquisition and Handling System (SHAHS), gathers data from RTU's on a regular basis. With the capability to define both timed and event driven poll routines, and specify a virtually unlimited number of RTU’s and MODBUS registers to collect, virtually any piece of information can be collected at any desired time. The Vsystem software can process data through a myriad of mathematical functions, and combine outputs from multiple stations. Vsystem also incorporates the ability to permanently store data in its own internal database, MS Sequel databases, or export data in other formats. Data can be displayed in a user-created graphical user interface (GUI) which MRGCD water operations personnel use to monitor water movement. The screens can also execute scripts to generate data, control parameters, control gate set points, and monitor alarm conditions for automated control structures. Finally, the GUI’s can be used to control automated structures by transmitting new parameters and setpoints.

One of the key components to improving water delivery efficiency was linking the DSS water delivery recommendations to the MRGCD SCADA network. This was done in 2009 so that water operations personnel could closely monitor the actual canal diversions and compare those diversions to the DSS recommendations on a real-time basis. The overall goal was to match the diversions from the Rio Grande to the real-time crop water requirement calculated using the DSS.

Incorporating the DSS with the MRGCD SCADA system involved three distinct steps. First, the DSS was installed on the MRGCD main water operations computer. Second, the DSS output was converted to a format that the Vista software could recognize. The Vista software uses the SHEF.A. file format for data coming into the SCADA system, and the entire MRGCD canal network is set to function on this format. The data for each individual gate or measuring site are characterized by a distinct data stream in the Vista Software, the data and are linked to the appropriate node in the SCADA GUI. The data stream for each node in the GUI is user specified; and therefore, nodes were created that display DSS recommended flow rates. In order to link the DSS to this SCADA software it was necessary to create DSS output files in SHEF.A. format. This was accomplished through cooperative work between Colorado State University, the MRGCD, and Vista Systems. A subroutine in the MRGCD SCADA Vsystem software converts the output from the DSS into the SHEF.A. format. This subroutine allows for the creation of separate data streams from the DSS schedule for each lateral canal service area. These data streams are the same data streams that are used throughout the MRGCD SCADA network.

The third step in linking the DSS was to create a node for the DSS recommended flowrate for each lateral service area in the SCADA GUI that also contains actual flowrate data. This consisted of creating nodes in the SCADA GUI that display the actual flow passing into a lateral on the left side of the node and the recommended flowrate from the DSS in parentheses. The DSS recommended flow was linked to the correct data stream from the DSS output in SHEF.A format. For the 2009 irrigation season actual flow and the DSS recommended flow were displayed side by side for each lateral on the Peralta Main Canal in the SCADA GUI. Figure 13 displays the revised SCADA GUI with the DSS recommendations.

Whenever the recommendations for DSS flowrates are updated the DSS is run and the scheduler output saves as a CSV file in Excel. This file is then converted using the Vsystems subroutine and available as a SHEF.A. data stream for each individual lateral. Linking the DSS water distribution recommendations with the MRGCD SCADA provides a simple and effective medium for managers to implement and monitor DSS water delivery schedules. Additionally, the combination of both programs allows for real-time management and the convergence of river diversions and the water required based on crop demand.

## 6. Results

The DSS linked with SCADA has multiple benefits for the MRGCD. These benefits include providing a method for predicting anticipated water use, sustaining efficient agriculture in the valley, minimizing the difficulty of meeting flow requirements for environmental concerns, and improved reservoir operations. The DSS in concert with SCADA also provides the MRGCD with a powerful tool to manage water during periods of drought.

One of the main benefits that the DSS has for the MRGCD is that the DSS provides a tool to determine anticipated water use. The need for this arises out of the fact that water delivery to farms in the MRGCD is not metered and scheduling is inconsistent. This makes it difficult, if not impossible, to place the appropriate amount of water in a canal at the proper time. The MRGCD dealt with this in the past by typically operating all canals at maximum capacity throughout the irrigation season. This provided a high level of convenience for water users, and made the lack of scheduling unimportant. But this practice has had significant negative consequences. Not least of these consequences was the public perception of irrigated agriculture. The MRGCD practice of operating canals at maximum capacity resulted in diversion rates from the Rio Grande that were large. Through the creation of a thorough and systematic database in the DSS of cropping patterns, irrigated acreage, automated processing of climate data into ET values, and incorporation of a flow-routing component incorporating the physical characteristics of the MRGCD canals, it became possible to predict how much water users will need. The DSS, with the myriad of calculations it performs, provides the MRGCD water operations manager a method to determine water requirements in advance and manage water using SCADA. Problems of timing and crop stress are addressed and the MRGCD can operate at reduced diversion levels, while serving its water users.

Use of the DSS linked with SCADA also has the benefit that it can aid the MRGCD in sustaining irrigated agriculture in the Middle Rio Grande Valley. The agricultural tradition in the Middle Rio Grande Valley dates back over 500 years and one of the main goals of MRGCD is to sustain the agricultural culture, lifestyle and heritage of irrigation. The problem facing the MRGCD is how to sustain irrigated agriculture amidst drought and increased demands for water from the urban and environmental sectors. These demands for water from other sectors will increase as the population in the Middle Rio Grande Valley grows and expands. Additionally, the MRGCD will be faced with dealing with periodic drought and climate change with the possibility of reduced snow melt runoff in the future. The concept of scheduled water delivery, implemented through the use of the DSS linked with SCADA provides the MRGCD with the ability to sustain irrigated agriculture with reduced river diversions. Scheduled water delivery that is based on crop demand calculated using the DSS and delivered through a highly efficient modernized system will allow the MRGCD to continue supplying farmers with adequate water for irrigation, even though the available water supply may be reduced due to natural or societal constraints.

Scheduled water delivery implemented through the use of the DSS and SCADA will also benefit the MRGCD by easing competition with environmental water needs. Due to the Endangered Species Act, water operations and river maintenance procedures have been developed in the Middle Rio Grande Valley to ensure the survival and recovery of the Rio Grande silvery minnow. These procedures include timing of flow requirements to initiate spawning and continuous flow throughout the year to provide suitable habitat. Additionally, the entire Rio Grande in the MRGCD has been designated as critical habitat for the RGSM. The use of the DSS, linked with SCADAprovides the MRGCD with the ability to reduce river diversions at certain times during the irrigation season. Reduced river diversions from the MRGCD main canals may at times leave more water in the Rio Grande for the benefit of the RGSM with credit toward the MRGCD for providing the flow requirements for the recovery of the species. The DSS may also be useful in providing deliveries of water specifically for the RGSM. Since the listing of the RGSM, the MRGCD canal system has been used to deliver a specific volume of water to points along the Rio Grande to meet flow requirements for the species. At certain times of the year, this is the only efficient way to maintain RGSM flow targets. While not presently incorporated in the DSS, it would be straightforward to specify delivery volumes at specific points in the MRGCD system. These delivery volumes for the RGSM would be scheduled and routed in a similar fashion to agricultural deliveries. Depending on the outcome of the current process of developing RGSM management strategies, this could someday become a very important component and benefit of the DSS.

One of the significant benefits of using the DSS with SCADA in the MRGCD is improved reservoir operations. The storage reservoirs in the MRGCD are located in the high mountains and it takes up to seven days for released water to reach irrigators. Prior to scheduled water delivery utilizing the DSS, the MRGCD water manager had to guess at what the demand for a main canal would be in advance and then release stored water to meet the assumed demand. Through the use of the DSS and SCADA the water operations manager can utilize historical climate data to predict what the agricultural demand will be in the future. This allows for an accurate calculation of the required release from reservoir storage and minimizes superfluous releases. These reduced reservoir releases have significant benefits for the MRGCD. Since less water is released from storage reservoirs during the irrigation season it allows for increased storage throughout the season. This allows the MRGCD to stretch the limited storage further and minimize the impacts of drought. Decreases in reservoir releases also have the added benefit of providing more carryover storage for the following irrigation season, providing greater certainty for water users in subsequent years. Larger carryover storage also translates to less empty space to fill during spring runoff. This leads to three benefits for the MRGCD. The first is that reservoirs can still be filled, even in a year when runoff is below average. The second benefit is that in above average runoff years the reservoirs will fill quickly and much of the runoff will go downstream, mimicking the hydrograph before the construction of upstream storage reservoirs. This is a subtle but significant environmental benefit to the Middle Rio Grande and RGSM as peaks in the hydrograph induce spawning, provide river channel equilibrium and affect various other ecosystem processes. The third benefit is that increased downstream movement of water during the spring will aid the state of New Mexico in meeting Rio Grande compact obligations to Texas. Overall, scheduled water delivery in the MRGCD provides significant benefits to reservoir operations and will allow the MRGCD to reduce reservoir releases, provide more reliable deliveries, increase certainty of full deliveries, and sustain irrigated agriculture.

In times of surplus water the MRGCD may not utilize the DSS for scheduled water delivery because of the associated increase in management intensity and higher operational costs. In drought years the available water supply will be lower and result in tighter and more stringent management requirements to equitably distribute water to users. It is during times of drought and shortage that the DSS linked with SCADA will provide the necessary management to distribute water based on crop demand. In the future the DSS will be a powerful tool that the MRGCD can utilize to equitably distribute water and sustain irrigated agriculture through times of low water supply resulting from drought.

## 7. Conclusions

Water delivery through the use of a DSS linked with SCADA has applications and benefits that can be realized throughout the arid regions of the world. The main benefit of scheduled water delivery utilizing a DSS and SCADA is that diversions can more closely match crop water requirements eliminating spills and drainage problems. In the American West urban growth and environmental concerns have forced irrigated agriculture to reduce diversions. In many irrigated systems in the United States that traditionally used surface application, agriculture has opted to improve water use efficiency by changing water application methods to sprinkler and drip irrigation systems. Irrigated agriculture throughout most of the world still relies on surface application and cannot afford to upgrade to systems such as sprinkler or drip irrigation. Therefore, water delivery utilizing a DSS with SCADA has the potential to reduce diversions and sustain agriculture.

The DSS, SCADA, and scheduled water delivery also have the potential of meeting future agricultural demands in developing regions throughout the world. As the world population continues to grow there will be an increased demand for food production and in many cases water resources available for agriculture are already fully utilized. Utilizing a DSS linked to SCADA would allow water users in developing countries with surface application systems to conserve water from their current practices and apply the saved water to increased food production.

The DSS could also be used to refine water delivery scheduling. Many arid regions have been dealing with water shortages for decades and have already implemented scheduled water delivery. In most cases, water delivery schedules are based on a set interval of time and do not coincide with crop demand. In areas where this type of scheduling is practiced the DSS could be used to refine scheduling protocols to include crop demand. Scheduling water deliveries based on crop demands would provide additional saving in areas where scheduled water delivery is already implemented. The developed DSS could be utilized in any irrigation system worldwide that practices surface irrigation techniques. Through scheduling based on crop demand, overall diversions could be significantly reduced. Reduced diversions could help irrigators deal with drought, and climate change by allowing for increased utilization of stored water. Additionally, reduced diversions could be utilized to grow supplementary crops to supply the needs of a growing population in the future.

## Acknowledgments

The authors would like to thank Subhas Shah, the MRGCD Board of Directors and staff, the New Mexico Interstate Stream Commission, the New Mexico Office of the State Engineer, the Middle Rio Grande Endangered Species Act Collaborative Program, the United States Army Corp of Engineers, and the United States Bureau of Reclamation for the assistance and the financial support to undertake this project. Also, the exceptional support of Jim Conley at Control Design, Gerald Robinson and Lee Allen at Aqua Systems 2000, and Cathy Laughlin and Peter Clout of Vista Control Systems is graciously recognized.The authors would also like to acknowledge the support of the National Science Foundation

## How to cite and reference

### Cite this chapter Copy to clipboard

Kristoph-Dietrich Kinzli, David Gensler and Ramchand Oad (September 6th 2011). Linking a Developed Decision Support System with Advanced Methodologies for Optimized Agricultural Water Delivery, Efficient Decision Support Systems - Practice and Challenges in Multidisciplinary Domains, Chiang Jao, IntechOpen, DOI: 10.5772/16877. Available from:

### chapter statistics

1Crossref citations

### Related Content

Next chapter

#### Estimating the Impact on Water Quality under Alternate Land Use Scenarios: A Watershed Level BASINS-SWAT Modeling in West Georgia, United States

By Gandhi Bhattarai, Diane Hite and Upton Hatch

First chapter

#### Semantic Knowledge Representations for Soft Data Fusion

By Claire Laudy

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

View all Books


text:
# Using Python to rename multiple csv files in Windows

I need to rename a large list of csv files that are generated from 3 different servers. The files are produced with a date stamp as the extension, which I need to move in each file name to retain the date stamp.

The file name format before editing is as:

billing.pps-svr01.csv.2015-09-01

billing.pps-svr02.csv.2015-09-01

billing.pps-svr03.csv.2015-09-01

The file name format after editing should be:

billing.pps-svr01.2015-09-01.csv

billing.pps-svr02.2015-09-01.csv

billing.pps-svr03.2015-09-01.csv

My question is in regard to code efficiency and best practice. The following code seems to work in testing, however I'm very new to Python and programming in general and I'd like to know if there are any other ways to solve this problem that are more efficient...for example, could this be accomplished in a one-liner or something else? I should also note that I intend to incorporate this into a much larger script that parses these files after they've been renamed. Any feedback and/or suggestions would be great.

import os

os.chdir(r'C:\Users\Extract')

for filename in os.listdir('.'):
if filename.startswith('billing.pps-svr'):
name = filename.split('.')
name = name+[name.pop(0-2)]
new_name = '.'.join(name)
os.rename(filename, new_name)

## 4 Answers

Here are my thoughts:

• You don't do any error handling if any of the file operation fails. Not to be recommended
• Your code will flip the latter two parts, all the time. It does not care if the filename has already been fixed or not!
• The code name + [name.pop(0-2)] troubles me. You are concatenating the name list with the popped value, but in order for this work you need for the popping to happen before the first part to be joined. Scary stuff...
• Here are some pythonic ways to do stuff with lists:
• name[:-2] – Get everything but the two last elements
• name[-2:] – Get only the last two elements
• name[::-1] – Reverse the element list

Here is some coding displaying the flaw in your original rename code, and two options for how to handle it correctly.

for filename in (
'billing.pps-svr01.2014-09-01.csv',
'billing.pps-svr02.2014-09-01.csv',
'billing.pps-svr01.csv.2015-09-01',
'billing.pps-svr02.csv.2015-09-01',
):

print('\nTesting {}:'.format(filename))

name = filename.split('.')
name = name + [name.pop(0-2)]
new_name = '.'.join(name)

print '    old rename: {} to {}'.format(filename, new_name)

filename_parts = filename.split('.')
first_part = filename_parts[:-2]
last_part = filename_parts[-2:]

if last_part[-1] != 'csv':
new_name = '.'.join(first_part + last_part[::-1])
print '    new rename: {} to {}'.format(filename, new_name)
else:
print '    no rename needed'

if filename_parts[-2] == 'csv':
new_name = '.'.join(filename_parts[:-2] + filename_parts[-2:][::-1])
print '    alt rename: {} to {}'.format(filename, new_name)
else:
print '    no alternate rename needed'

The output from this are as follows:

Testing billing.pps-svr01.2014-09-01.csv:
old rename: billing.pps-svr01.2014-09-01.csv to billing.pps-svr01.csv.2014-09-01
no rename needed
no alternate rename needed

Testing billing.pps-svr02.2014-09-01.csv:
old rename: billing.pps-svr02.2014-09-01.csv to billing.pps-svr02.csv.2014-09-01
no rename needed
no alternate rename needed

Testing billing.pps-svr01.csv.2015-09-01:
old rename: billing.pps-svr01.csv.2015-09-01 to billing.pps-svr01.2015-09-01.csv
new rename: billing.pps-svr01.csv.2015-09-01 to billing.pps-svr01.2015-09-01.csv
alt rename: billing.pps-svr01.csv.2015-09-01 to billing.pps-svr01.2015-09-01.csv

Testing billing.pps-svr02.csv.2015-09-01:
old rename: billing.pps-svr02.csv.2015-09-01 to billing.pps-svr02.2015-09-01.csv
new rename: billing.pps-svr02.csv.2015-09-01 to billing.pps-svr02.2015-09-01.csv
alt rename: billing.pps-svr02.csv.2015-09-01 to billing.pps-svr02.2015-09-01.csv

Notice how the two first files would have gotten a wrongly rename use your original code.

## Code refactor (added)

To accomodate for your question regarding building this into a larger script, and to give example of error handling, I've refactor your code into the following (using the tip from Janne Karila on using rsplit):

import os

def rename_csv_files(directory, required_start):
"""Rename files in <directory> starting with <required_start> to csv files

Go to <directory> and read through all files, and for those
starting with <required_start> and ending with something like
*.csv.YYYY-MM-DD and rename these to *.YYYY-MM-DD.
"""
try:
os.chdir(directory)
except OSError, exception:
print('IOError when changing directory - {}'.format(exception))
return

try:
for filename in os.listdir('.'):
if filename.startswith(required_start):

base, ext, date = filename.rsplit('.', 2)
new_filename = '.'.join((base, date, ext))
if ext == 'csv' and not os.path.exists(new_filename):
try:
os.rename(filename, new_filename)
print 'Renamed: {}'.format(new_filename)

except OSError, exception:
print('Failed renaming file - dir: {}, original file:  {}, new file: {} - {}'.format(
directory, filename, new_filename, exception))

elif ext != 'csv':
print('Skipped: {}'.format(filename))

else:
print('Skipped: {} - Renamed version already exists'.format(filename))

except OSError, exception:
print('Failed traversing directory - dir: {} - {}'.format(directory, exception))

def main():
rename_csv_files('./test_data', 'billing.pps-svr')

if __name__ == '__main__':
main()

Running this script against the following test-data:

\$ ls -1d test_data/* | sort -n
test_data/billing.pps-svr01.2014-09-01.csv
test_data/billing.pps-svr01.csv.2015-09-01
test_data/billing.pps-svr02.2014-09-01.csv
test_data/billing.pps-svr02.2015-09-01.csv
test_data/billing.pps-svr02.csv.2015-09-01
test_data/original_files.tar

Gives the following output:

Skipped: billing.pps-svr01.2014-09-01.csv
Renamed: billing.pps-svr01.2015-09-01.csv
Skipped: billing.pps-svr02.2014-09-01.csv
Skipped: billing.pps-svr02.2015-09-01.csv
Skipped: billing.pps-svr02.csv.2015-09-01 - Renamed version already exists

This code now handles error handling for at least the following cases:

• Directory not existing, or read or execution permission faults
• Errors when traversing directory or renaming files
• The logical error of renaming a file into an already existing file
• This is great feedback, thank you! One minor note - I hit an NameError using your refactored code because main() was not defined...had to move the IF statement after the function was defined to clear it. Otherwise works great! – erns Oct 5 '15 at 18:26
• @erns, That is correct. The if needs to be after... Have corrected code – holroy Oct 5 '15 at 20:00

The string manipulation becomes more readable using sequence unpacking:

base, ext, date = name.rsplit('.', 2)
new_name = '.'.join((base, date, ext))

I should also note that I intend to incorporate this into a much larger script that parses these files after they've been renamed

If so, you want modularity and customization possibilities, both of which can be achieved by wrapping your code in a function:

def sensible_name_for_this_renaming_process(directory, required_start):
os.chdir(directory)

for filename in os.listdir('.'):
if filename.startswith(required_start):
name = filename.split('.')
name = name+[name.pop(0-2)]
new_name = '.'.join(name)
os.rename(filename, new_name)

It's slightly neater to avoid nesting by inverting the if condition at the start:

for filename in os.listdir('.'):
if not filename.startswith('billing.pps-svr'):
continue

name = filename.split('.')
name = name+[name.pop(0-2)]
new_name = '.'.join(name)
os.rename(filename, new_name)

format: plain


text:
# An OSI layer model for the 21st century

The Internet protocol suite is wonderful, but it was designed before the advent of modern cryptography and without the benefit of hindsight. On the modern Internet, cryptography is typically squeezed into a single, incredibly complex layer, Transport Layer Security (TLS; formerly known as Secure Sockets Layer, or SSL). Over the last few months, 3 entirely unrelated (but equally catastrophic) bugs have been uncovered in 3 independent TLS implementations (Apple SSL/TLS, GnuTLS, and most recently OpenSSL, which powers most “secure” servers on the Internet), making the TLS system difficult to trust in practice.

What if cryptographic functions were spread out into more layers? Would the stack of layers become too tall, inefficient, and hard to debug, making the problem worse instead of better? On the contrary, I propose that appropriate cryptographic protocols could replace most existing layers, improving security as well as other functions generally not thought of as cryptographic, such as concurrency control of complex data structures, lookup or discovery of services and data, and decentralized passwordless login. Perhaps most importantly, the new architecture would enable individuals to internetwork as peers rather than as tenants of the telecommunications oligopoly, putting net neutrality directly in the hands of citizens and potentially enabling a drastically more competitive bandwidth market.

Current OSI model In practice Proposed update
8 (none) Application Application
7 Application HTTP Transactions
6 Presentation SSL/TLS (Non-)Repudiation
5 Session TCP Confidentiality
4 Transport Availability
3 Network IP Integrity
2 Data Link e-UTRA (LTE), 802.11 (WiFi), 802.3 (Ethernet), etc. Data Link
1 Physical Physical

Of course, the layers I propose will doubtless introduce new problems of their own, but I’d like to start this conversation with some concrete ideas, even if I don’t have a final answer. (Please feel free to email me your comments or tweet @davidad.)

Descriptions follow for each of the five new layers I suggest, four of which are named after common information security requirements, and one of which (Transactions) is borrowed from database requirements (and also vaguely suggestive of cryptocurrency).

General disclaimer for InfoSec articles: Reading this article does not qualify you to design secure systems. Writing this article does not qualify me to design secure systems. In fact, nobody is qualified to design secure systems. A system should not be considered secure unless it has been reviewed by multiple security experts and resisted multiple serious attempts to violate its security claims in practice. The information contained in this article is offered “as is” and without warranties of any kind (express, implied, and statutory), all of which the author expressly disclaims to the fullest extent permitted by law.

## Integrity layer

We would like a received payload to self-evidently be the same payload which was sent. Although the Data Link layer is supposed to provide such an assurance, various kinds of attacks on the system might invalidate this assumption. Integrity protocols mitigate these attacks:

Paranoia Level Attacks Mitigation Common Implementation My Preferred Implementation
1 Thermal noise, cosmic rays checksum hash TCP Checksum CRC-32C
2 Deliberate corruption cryptographic hash SHA-1 BLAKE2b
3 Spoofing of trusted contacts keyed hash HMAC-SHA1 SipHash
4 Spoofing of strangers public-key signature of cryptographic hash SHA-1 + RSA BLAKE2b + Ed25519

Integrity protocols are fairly simple: the appropriate verification material is placed at the beginning of every Data Link payload. The Integrity layer exposes the same kind of “transmit queue and receive queue” interface as the Data Link layer, but the payload which can be passed to the Integrity layer must be somewhat smaller, so that there is room for the verification material and the Integrity payload together to fit into 1280 octets. Overhead ranges from 4 octets for a CRC-32C checksum to 96 octets for an Ed25519 signature.

In the keyed hash case, some state is necessary at the Integrity protocol level: each API customer must be able to add “trusted contacts” to its “address book” by specifying a symmetric key corresponding to a given endpoint name (which may have been negotiated at a higher protocol level, or simply out-of-band entirely). Since some advanced higher-level protocols may define symmetric authentication keys that are only good for a single use (e.g. Axolotl ratcheting after the handshake phase), “address book entries” should be single-use by default, with renewal explicitly required after each payload received from a given contact.

## Availability layer

We would like networked endpoints to be available to receive packets from other endpoints in a way that is robust to unannounced changes in network topology. This layer conceptually takes the place of the Network layer in the original model, as it will be responsible for routing packets. Significantly, in this proposal, there are no “hosts” or “ports”: only “endpoints”, identified by public keys. This is simply taking the end-to-end principle one step further, by considering the “host” merely part of the network infrastructure which makes applications available.

A fully implemented Availability layer should provide unicast (deliver to a unique endpoint authenticated by a given public key, wherever it may be), anycast (deliver to nearest endpoint authenticated by a given public key), and multicast (a.k.a. pub/sub: route to all endpoints who have asked to subscribe to a given ID, and provide a subscription method).

Routing Semantics Current Reliability New Implemenation
Overlay on existing Internet Native Mesh
Multicast awful S/Kademlia message broker Straightforward extension of unicast
Anycast decent No advantage over load balancers Possible extension of unicast
Unicast excellent Special case of multicast Electric Routing

I believe the Electric Routing algorithm2 is up to the challenge of replacing unicast3, and that it could be extended to provide multicast and even anycast, but other algorithms could be developed at this protocol layer as well. The first real-world implementation of the system I’m describing will very likely be developed as an overlay network on top of IP, in which case multicast can be implemented simply atop S/Kademlia, with unicast as a special case, and anycast can be emulated with standard load-balancing techniques.

The tradeoff here is that routers have a lot more work to do, since there are no “addresses” corresponding directly to geographic location. But, it means that every node on the network can participate as a router, so there is a lot more capacity to do that work. In addition, the endpoints-only scheme has many potentially desirable properties with respect to features like pseudonymity, NAT transparency, redundancy, and decentralization of the telecommunications market (especially in densely settled areas).

## Confidentiality layer

Ideally, we would like to not transmit any information to anything other than the destination endpoint(s). This ideal is not in general achievable on a public network, but some types of mitigation are possible:

Paranoia Level Attacks Mitigation Common Implementation My Preferred Implemenation
1 Sniffing payloads to trusted contacts symmetric encryption AES ChaCha
2 Sniffing payloads to strangers public-key encryption RSA RSA
3 Chosen plaintext attacks key agreement + symmetric encryption ECDH + AES Curve25519 + ChaCha
4 Key compromise ephemeral key agreement + symmetric encryption ECDHE + AES Axolotl ratchet with Curve25519, SipHash, PBKDF2, ChaCha

In cases 3 and 4, this layer has to maintain some state, holding session keys or message keys, and the Axolotl ratchet is a little complicated; but this layer does not have to worry about the verification of identity (which will be provided on a higher layer, by services such as keybase.io or using pronounceable hash fingerprints) or integrity (which will be provided by a lower layer).

## Non-Repudiation and/or Repudiation layer

We would like for a receiver to be sure that a message they receive was sent by a given sender, and we would like for a sender to be sure that a given message was successfully received. Sometimes, we would also like for a receiver to be unaware of the location a message was sent from. The result is three related but orthogonal protocol types, which may be nested:

Repudiation Property Meaning Protocol
Non-Repudiation of Sending Recipient knows immediate sender Sender includes a hash of their public key in the message. To understand why this is necessary given the Integrity layer, read this excellent article
Non-Repudiation of Receipt Sender knows message was received Recipient must send a signed acknowledgement for every message. This also implements “reliable delivery”
Repudiation of Origin Message is difficult to trace Onion Routing

## Transactions layer

We would like for sets of nodes which wish to maintain common mutable state variables to be able to do so, even in the presence of various types of adversaries. This is a common abstraction for the requirements of git, cryptocurrencies, and distributed databases (i.e. ACID MVCC). I propose that (borrowing most directly from git, but also from Clojure’s concurrent data structures) changes in large or complex mutable states be represented as changes to the root of a Merkle tree, thus reducing the state subject to transactional semantics to single-packet size4.

To make it obvious what I’m intending to refer to, the owner of a particular “domain name” or a particular “coin” (or, generally, any cryptographically controlled resource) is an example of a mutable state. But so is, for instance, the contents of any social media profile, email inbox, hypertext page, or source code repository. These things could all be managed without reference to central authorities or single points of failure.

Paranoia Level Attacks Mitigation
1 Asynchrony; node failure/disconnection D1HT tracker
2 Sybil attacks; eclipse attacks; churn attacks S/Kademlia tracker
3 Malicious trackers Leaderless Byzantine Paxos or Byzantine gossip
4 Any attack that Bitcoin can survive Block-chain protocol

Many (including myself) have claimed that the core contribution of Bitcoin, the block-chain protocol, is a novel solution to the Byzantine Generals Problem, but it turns out this is somewhat misleading. Although the block-chain protocol is Byzantine-fault-tolerant in a novel way, there has been plenty of research on Byzantine protocols over the years, and it seems probably unnecessary to constantly “mine,” i.e. solve cryptopuzzles, to achieve Byzantine fault tolerance. The main reason to introduce cryptopuzzles is to reduce the efficacy of Sybil attacks, in which one malicious actor fabricates arbitrarily many identities in order to exceed the Byzantine fault tolerance threshold and control the system. However, these attacks can also be mitigated by requiring crypto-puzzles only for joining the network (as in S/Kademlia), and by blacklisting nodes which behave suspiciously (the latter being how most attacks on Bitcoin are stopped in practice).

## Application layer

In such an environment, applications (or application components!) are essentially just maps from one mutable state to another, in functional reactive programming style. In the same way that you might encode packet filters into a kernel’s TCP/IP stack today, you might encode entire applications into a kernel’s “mesh” stack in the future. Various search functions, including full-text search, could be provided using the OneSwarm approach or potentially by distributed Bloom filters implemented atop this platform (an idea due to Andrée Monette). Resource control and access control can be provided by means of cryptographic capabilities.

But, in general, this layer is completely open for all sorts of applications. Essentially, any end-user service that runs on a network (and what doesn’t, these days?) would fit here.

# Conclusion

I’ve outlined some radical ideas for how to re-build the Internet protocol stack in a way that is ultimately more coherent with Internet cultural values (freedom of expression, pseudonymity, reduced potential for abuses of power). This outline still needs quite a bit of work and thought before being turned into implementations, but I feel like I’ve reached a turning point in making my ideas about next-generation architectures concrete, and at a timely moment with respect to conversations about TLS and net neutrality. If you would like to see these concepts made into working code, please reach out and let me know.

1. This number is cribbed from the IPv6 RFC.

2. coauthored by Petar Maymounkov, who also coauthored Kademlia, the DHT powering BitTorrent

3. Electric routing does need some extensions to mitigate various attacks, but I believe the countermeasures from S/Kademlia are readily adapted to meet these needs.

4. This is similar in principle to the trick used by most practical public-key cryptosystems, which use the actual public-key algorithm only to encrypt a key from some symmetric cryptosystem, and then encrypt arbitrarily large content using a stream cipher. The common principle is that you can do the hard security algorithm on a small piece of data, and use easier security algorithms to apply those hard security properties to large chunks of data.


text:
# Laves Graph

This picture by Greg Egan shows the Laves graph, a structure discovered by the crystallographer Fritz Laves in 1932. It is also called the ‘$\mathrm{K}_4$ crystal’, since is an embedding of the maximal abelian cover of the complete graph on 4 vertices in 3-dimensional Euclidean space. It is also called the ‘triamond’, since it is a theoretically possible — but never yet seen — crystal structure for carbon.

# Diamond Cubic

This picture by Greg Egan shows the pattern of carbon atoms in a diamond, called the diamond cubic. Each atom is bonded to four neighbors. This pattern is found not just in carbon but also other elements in the same column of the periodic table: silicon, germanium, and tin.

# Balaban 11-Cage

This picture shows part of a graph called the Balaban 11-cage. A (3,11)-graph is a simple graph where every vertex has 3 neighbors and the shortest cycle has length 11. A (3,11)-cage is a (3,11)-graph with the least possible number of vertices. The Balaban 11-cage is the unique (3,11)-cage.

# Heawood Graph

This is the Heawood graph. This graph can be drawn on a torus with no edges crossing in such a way that it divides the torus into 7 hexagons, each pair of which shares an edge. In 1890, Percy John Heawood proved that for any map drawn on a torus, it takes at most 7 colors to ensure that no two countries sharing a common boundary have the same color. The Heawood graph proves that the number 7 is optimal.

# Petersen Graph

Suppose you have a set with 5 elements. There are 10 ways to choose a 2-element subset. Form a graph with these 10 choices as vertices, and with two vertices connected by an edge precisely when the corresponding subsets are disjoint. You get the graph shown here, called the Petersen graph.


text:
# Rig Models¶

## Coordinate Systems¶

Rig are defined by a fixed assembly of cameras that are triggered at the same instant. The following terms define such assembly and capture in OpenSfM terminology :

• A RigCamera is a camera of the rig assembly defined as a combination of an existing camera model (it refers only to its ID) and its pose wrt. the rig assembly coordinate frame. RigCamera are defined in the rig_cameras.json as the following:

{

“RIG_CAMERA_ID”: {

“translation”: translation of the rig frame wrt. the RigCamera frame “rotation”: rotation bringing a point from rig frame to the RigCamera frame “camera”: camera model ID of this RigCamera

• A RigInstance is a list of Shots, each of which correspond to a RigCamera of the RigModel and the actual pose of the RigModel in the world : it’s indeed an instanciation of the RigModel by combining Shots. These instances are defined in the rig_assignments.json file as follows:

{
[
[
"FILENAME",
"RIG_CAMERA_ID1"
],
[
"FILENAME",
"RIG_CAMERA_ID2"
],
...
[
"FILENAME",
"RIG_CAMERA_IDn"
]
],
[
[
"FILENAME",
"RIG_CAMERA_ID1"
],
[
"FILENAME",
"RIG_CAMERA_ID2"
],
...
[
"FILENAME",
"RIG_CAMERA_IDn"
]
],
...

A picture is often worth many words :

### Usage¶

Given the above, one can either define manually the rig_assignments.json and the rig_cameras.json, or use the OpenSfM create_rig command. This commands will take a JSON string as input to help it defines the rig instances based on the filenames, such as:

{
"RIG_CAMERA_ID1": "PATTERN1",
"RIG_CAMERA_ID2": "PATTERN2",
...

Where “PATTERN” is the following :
• A camera model ID if the method for assigning is camera. A given camera model ID correspond to one RigCamera

• A REGEX with the form (.*) where the part in parenthesis identifies the camera models, when the method is pattern. For example, it would be “(RED)” or “(GREEN)” for multispectral data.

Based on this instances, it then run SfM on a small subset on the data and infers some averaged rig cameras poses, which are then written to rig_cameras.json.


text:
## Precalculus (6th Edition)

$\color{blue}{\dfrac{25p^2}{9}}$
Use the rule $\dfrac{a}{b} \div \dfrac{c}{d} = \dfrac{a}{b} \times \dfrac{d}{c}$ to obtain: $\dfrac{15p^3}{9p^2} \times \dfrac{10p^2}{6p}$ Cancel common factors, then multiply to obtain: $\require{cancel} =\dfrac{\cancel{15}^5p^3}{\cancel{9}^3\cancel{p^2}} \times \dfrac{10\cancel{p^2}}{6p} \\=\dfrac{5p^3}{3} \times \dfrac{10}{6p} \\=\dfrac{5\cancel{p^3}p^2}{3} \times \dfrac{\cancel{10}5}{\cancel{6}3\cancel{p}} \\=\color{blue}{\dfrac{25p^2}{9}}$