Supported resampling operations
CIS is intended to facilitate a number of oft repeated operations on scientific data. Before analysis, the user is likely to want to subset and aggregate their dataset and possibly collocate it with another dataset. Once this is done, visualisation and statistical analysis help the user make sense of the dataset(s).
A subset of the original dataset is created by restricting its spatio-temporal extent. This command can be used to select e.g. all data for June 2007; all data at the surface or all data over a longitude/latitude box encompassing the Sahara. Restrictions can be combined e.g. all data for 2007 at the surface in a longitude/latitude box encompassing the Sahara. The subset is written to file in NetCDF format.
The original dataset is aggregated by spatio-temporal averaging. This can serve several purposes: it regularizes the data grid; it reduces overall data size; it smoothes the data; it makes the data more representative of large length and/or time-scales. Spatial and temporal averaging can be applied individually or together. This command can be used e.g. to aggregate satellite measurements over 1 by 1 degree grid-boxes or to produce yearly, zonal model averages. The aggregated result is written to file in NetCDF format.
One dataset is resampled to the spatio-temporal grid of another dataset. Depending on the grid structures of these datasets, different resampling methods are available: Nearest Neighbour selection, linear interpolation or area averages. For the latter method, the user may specify a weighting function. Using collocation, satellite data can be collocated with ground site data for evaluation of the satellite product or model data can be collocated with satellite data for evaluation of the model. The collocated result is written to file in NetCDF format.
Supported visualization and analysis types
Arbitrary mathematical functions can be applied to the contents of one or more datasets. This command assumes all data is on the same spatio-temporal grid (either originally, or as a result of collocation). It can be used to calculate e.g. the difference or ratio of two datasets. It also allows the user to mask out data based on other data. The evaluated result is written to file in NetCDF format.
Comprehensive summary statistics for variables in one or more datasets can be calculated. The command is meant to compare two variables on the same spatio-temporal grid (either originally, or as a result of collocation). The statistics include individual means and standard deviations, mean and standard deviation of the difference (both relative and absolute) as well as correlation coefficients and linear regression parameters. The evaluated result is written to screen and may also be written to file in NetCDF format.
The plot command is by far the most versatile CIS command as it allows for a large ranges of plot types: line graphs, scatter plots, heat maps, contour plots and all with arbitrarily specified axes. Plots may be overlaid on top of one another. This command may be used to plot e.g. a global map of yearly averaged model data; a time-series of station data; a scatter plot to compare two collocated datasets; The plot will be produced in a new window but can also be written to a number of graphical file formats (including png, jpg, gif and eps).
Obtaining information about a file
We will examine the contents of the data file, and in particular we will learn the names of the stored variables.
cis info NAerosol_mrg60_NP3_20020510_R0.ict.nc
UTC_mid GpsLat GpsLon GpsAlt PAlt AmbTemp StaticPrs RelHumidity nnucl npm1 ncoarse ntot
Obtaining help about a CIS command
We will request help on the plot command. This will list some general information on its use as well as a list of possible options that a user can pass along. All CIS commands can be queried in this way. The help function is not exhaustive (some features may not be mentioned, see the manual for more information).
Cis plot –help
usage: cis plot [-h] [-o [Output filename]] [--type [Chart type]] [--xlabel [X axis label]] [--ylabel [Y axis label]] [--cbarlabel [Colour bar label]] [--xtickangle [X tick angle]] [--ytickangle [Y tick angle]] [--title [Chart title]] [--itemwidth [Item width]] [--fontsize [Font size]] [--cmap [Colour map]] [--height [Plot height]] [--width [Plot width]] [--xmin [Minimum x]] [--xmax [Maximum x]] [--xstep [X step]] [--ymin [Minimum y]] [--ymax [Maximum y]] [--ystep [Y step]] [--vmin [Minimum value]] [--vmax [Maximum value]] [--vstep [X value]] [--xbinwidth [Histogram x axis bin width]] [--ybinwidth [Histogram y axis bin width]] [--cbarorient [Colour bar orientation]] [--nocolourbar [Hides the colour bar]] [--logx [Log base 10) scale on X axis]] [--logy [Log (base 10) scale on Y axis]] [--logv [Log (base 10 scale for values]] [--grid [Show grid]] [--xaxis [Variable on x axis]] [--yaxis [Variable on y axis]] [--coastlinescolour [Coastlines Colour]] [--nasabluemarble [NASA Blue Marble background]] [--plotwidth [Width of the plot in inches]] [--plotheight [Height of the plot in inches]] [--cbarscale [A scaling for the color bar]] Input datagroups [Input datagroups ...] positional arguments: Input datagroups The datagroups to be plotted, in the format 'variable:filenames[:options]', where options are entered in a comma separated list of the form 'keyword=value'. Available options are color, edgecolor, itemstylem, label and product. Colour is any valid html colour and product is one of the options listed below. For example 'cis plot var1:file:product=NetCDF_CF_Gridded,colour=red'. Products: ['ASCII_Hyperpoints', 'Aeronet', 'Aerosol_CCI', 'Caliop_L1', 'Caliop_L2', 'CloudSat', 'Cloud_CCI', 'MODIS_L2', 'MODIS_L3', 'NCAR_NetCDF_RAF', 'NetCDF_Gridded', 'cis', 'default_NetCDF', 'eumetsat_metop', 'eumetsat_metop'] optional arguments: -h, --help show this help message and exit -o [Output filename], --output [Output filename] The filename of the output file for the plot image --type [Chart type] The chart type, one of: ['contourf', 'heatmap', 'comparativescatter', 'overlay', 'histogram3d', 'line', 'histogram2d', 'scatteroverlay', 'contour', 'scatter'] --xlabel [X axis label] The label for the x axis --ylabel [Y axis label] The label for the y axis --cbarlabel [Colour bar label] The label for the colour bar --xtickangle [X tick angle] The angle (in degrees) of the ticks on the x axis --ytickangle [Y tick angle] The angle (in degrees) of the ticks on the y axis --title [Chart title] The title for the chart --itemwidth [Item width] The width of an item. Unit are points in the case of a line, and point^2 in the case of a scatter point. --fontsize [Font size] The size of the font in points --cmap [Colour map] The colour map used, e.g. RdBu --height [Plot height] The height of the plot in inches --width [Plot width] The width of the plot in inches --xmin [Minimum x] The minimum x value to plot --xmax [Maximum x] The maximum x value to plot --xstep [X step] The step of the x axis --ymin [Minimum y] The minimum y value to plot --ymax [Maximum y] The maximum y value to plot --ystep [Y step] The step of the y axis --vmin [Minimum value] The minimum value to plot --vmax [Maximum value] The maximum value to plot --vstep [X value] The step of the colour bar --xbinwidth [Histogram x axis bin width] The width of the bins on the x axis of a histogram --ybinwidth [Histogram y axis bin width] The width of the bins on the y axis of a histogram --cbarorient [Colour bar orientation] The orientation of the colour bar, either horizontal or vertical --nocolourbar [Hides the colour bar] Does not show the colour bar --logx [Log (base 10) scale on X axis] Uses a log scale (base 10) on the x axis --logy [Log (base 10) scale on Y axis] Uses a log scale (base 10) on the y axis --logv [Log (base 10) scale for values] Uses a log scale (base 10) on the colour bar --grid [Show grid] Shows grid lines on the plot --xaxis [Variable on x axis] Name of variable to use on the x axis --yaxis [Variable on y axis] Name of variable to use on the y axis --coastlinescolour [Coastlines Colour] The colour of the coastlines on a map. Any valid html colour (e.g. red) --nasabluemarble [NASA Blue Marble background] Add the NASA 'Blue Marble' image as the background to a map, instead of coastlines --plotwidth [Width of the plot in inches] Set the width of the plot when outputting to file --plotheight [Height of the plot in inches] Set the height of the plot when outputting to file --cbarscale [A scaling for the color bar] Scale the color bar, use when color bar does not match plot size
Plotting flight tracks for a full campaign
We will plot the flight tracks for all the flights in a campaign. Each flight is stored in a separate file. Instead of using a list of filenames, wildcards can be used, or one can point to a directory. We will also modify the plot a little bit, in particular select the total plot area and the size of the plotted symbol. Note that cis will select defaults if no options are passed. In particular, plot area will be determined from the file contents.
cis plot GpsAlt: NAerosol_mrg60_NP3*.nc --itemwidth 10 --ymin 27 --ymax 54 --xmin -90 --xmax -56 --cbarscale=0.85
2015-06-03 16:21:51,597 - WARNING - Identified 237 point(s) which were missing values for some or all coordinates - these points have been removed from the data.
Graphics window output:
Install using Conda
A pre-packaged version of CIS is available for installation using conda for 64-bit Linux, Mac OS X and Windows.
If you don’t already have conda, you must first download and install it. Anaconda is a free conda package that includes Python and many common scientific and data analysis libraries, and is available here. Further documentation on using Anaconda and the features it provides can be found here.
Once conda is installed, you can easily install CIS with the following command:
conda create -c conda-forge -n cis_env cis
This will install CIS into a clean environment named 'cis_env' which will preserve the versions of any other packages you may have installed, and ensure you get the latest version of the CIS packages. You can activate this environment by just typing:
source activate cis_env
Or, on Windows just:
See the conda documentation for more details. If you already have CIS installed updating to the latest version is just as easy. Simply type:
conda -c conda-forge update cis
Note that if you have a previous environment created using our 'cistools' channel you may need to delete it and create a new one to avoid conflicting packages from conda-forge. More details for installing CIS from source, and other package sources, can be found below.
Installing from source
Installing CIS from source is simply a matter of downloading and unzipping the source code from github, and then typing:
python setup.py install
from the download location.
CIS has a number of dependencies on other open-source packages which must be present before it can be installed from source however, and these are detailed below. They are all available through conda, or via the respective websites.
CIS has a number of build and run-time dependencies which are detailed here. Some of these packages have their own dependencies, but many of them are available pre-packaged through official repositories.
- python 2.7 or later (http://www.python.org/) CIS requires Python 2.7 or later, and is not currently compatible with Python 3.
- numpy 1.6 or later (http://numpy.scipy.org/) Python package for large numerical, N-dimensional operations
- scipy 0.10 or later (http://www.scipy.org/) Python package for scientific computing.
- iris 1.7.3 or later (http://scitools.org.uk/iris/) Python library providing an abstract interface onto CF conformant data.
- matplotlib 1.2.0 (http://matplotlib.sourceforge.net/) Python package for 2D plotting.
- nose 1.1.2 or later (http://nose.readthedocs.org/en/latest/) Python package for software testing.
- netcdf4-python 0.9.9 or later (http://netcdf4-python.googlecode.com/) Python interface to the netCDF version 4 C library.
- pyhdf 0.8.3 (http://pysclint.sourceforge.net/pyhdf/index.html) Python interface to the C HDF libraries
- basemap 1.0.7 or later (http://matplotlib.org/basemap/) Python library to support plotting maps of different projections in matplotlib
Specific instructions for installation of these dependencies on Ubuntu and Red Hat based systems are given in the following section.
Binary distributions of CIS exist for Ubuntu and Red Hat based Linux distributions which making make installing CIS and all of its dependencies very straightforward. For Ubunu a binary distribution of CIS available on Launchpad (https://launchpad.net/), and for Centos and RedHat 6.0 based distributions an .rpm of the latest version of CIS is available in the Centre for Environmental Data Archival (CEDA) repository.
64-bit and 32-bit Ubuntu Trusty (14.04 LTS)
- First add the community repository, and update the list of packages available:
sudo add-apt-repository ppa:cis/stable sudo apt-get update
- Next, install the CIS package:
sudo apt-get install python-cis
64-bit Red Hat Enterprise Linux 6.4
- Configure the EPEL 6 repository. Use the RPM below or see the [EPEL Home Page - http://fedoraproject.org/wiki/EPEL] for available mirrors.
$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
- Configure the CEDA repository. (See the CEDA Yum Repository for the latest configuration RPM.)
$ rpm -Uvh http://dist.ceda.ac.uk/yumrepo/RPMS/ceda-yumrepo-0.1-1.ceda.el6.noarch.rpm
Install CIS and basemap:
$ yum install python27-basemap python27-jasmin_cis.noarch
64-bit Centos 6.x
- Configure repositories as above:
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm rpm -Uvh http://dist.ceda.ac.uk/yumrepo/RPMS/ceda-yumrepo-0.1-1.ceda.el6.noarch.rpm
There is an issue in the EPEL repository which causes issues selecting the right grib_api version. To fix this edit the /etc/yum.repos.d/epel.repo file, adding line:
exclude=grib_api*to end of the [epel] section
Yum install http://proj.badc.rl.ac.uk/cedaservices/raw-attachment/ticket/670/armadillo-3.800.2-1.el6.x86_64.rpm
Install CIS and basemap:
yum install python27-basemap python27-jasmin_cis.noarch