stlabdict – Data structures and matrices

Module providing data structures for data collection and analysis

The main features provided in this module are the stlabmtx class and the stlabdict class as well as the framearr_to_mtx function.

stlabutils.stlabdict.checkEqual1(iterator)

Check if all elements in iterator are equal or is empty

Returns

True if iterator empty or has the same value for all elements. False otherwise.

Return type

bool

stlabutils.stlabdict.dictarr_to_mtx(data, key, rangex=None, rangey=None, xkey=None, ykey=None, xtitle=None, ytitle=None, ztitle=None)

Converts an array of dicts (or stlabdicts) to an stlabmtx object

Takes an array of dict-like (dict, OrderedDict, stlabdict), typically from a measurement file, and selects the appropriate columns for conversion into an stlabmtx that allows spyview like operations and processing.

If neither ranges or titles are given, some defaults are filled in. The chosen data column from each data array element will be placed as a line in the final matrix sequentially.

Parameters

• data (array of dict) – Input array of data dicts. The dicts are expected to contain a series of arrays of floats with the same length.

• key (str) – Index of the appropriate column of each dict for the data axis of the final matrix (data values for each pixel)

• ykey (xkey,) – Columns to use to calculate the desired x and y ranges for the final matrix. If these are proviced they are also used as the x and y titles. x runs across the matrix columns and y along the rows. This means that if x is the “slow” variable in the measurement file, the output matrix will be transposed to accomodate this. The ranges are assumed to be the same for all lines.

• rangey (rangex,) – If provided, they override the xkey and ykey assingnment. They should contain arrays of the correct length for use on the axes. These ranges will be saved along with the data (can be unevenly spaced). The ranges are assumed to be the same for all lines.

• ytitle, ztitle (xtitle,) – Titles for the x, y and z axes. If provided, they override the titles provided in xkey, ykey and key.

Returns

Resulting stlabmtx.

Return type

stlabmtx

stlabutils.stlabdict.framearr_to_mtx(data, key, rangex=None, rangey=None, xkey=None, ykey=None, xtitle=None, ytitle=None, ztitle=None)

Converts an array of pandas DataFrame to an stlabmtx object

Takes an array of pandas.DataFrame, typically from a measurement file, and selects the appropriate columns for conversion into an stlabmtx that allows spyview like operations and processing. Is essentially the same as dictarr_to_mtx but adapted for pandas DataFrame.

If neither ranges or titles are given, some defaults are filled in. The chosen data column from each data array element will be placed as a line in the final matrix sequentially.

Parameters

• data (array of dict) – Input array of frames.

• key (str) – Index of the appropriate column of each frame for the data axis of the final matrix (data values for each pixel)

• ykey (xkey,) – Columns to use to calculate the desired x and y ranges for the final matrix. If these are proviced they are also used as the x and y titles. x runs across the matrix columns and y along the rows. This means that if x is the “slow” variable in the measurement file, the output matrix will be transposed to accomodate this. The ranges are assumed to be the same for all lines.

• rangey (rangex,) – If provided, they override the xkey and ykey assingnment. They should contain arrays of the correct length for use on the axes. These ranges will be saved along with the data (can be unevenly spaced). The ranges are assumed to be the same for all lines.

• ytitle, ztitle (xtitle,) – Titles for the x, y and z axes. If provided, they override the titles provided in xkey, ykey and key.

Returns

Resulting stlabmtx.

Return type

stlabmtx

class stlabutils.stlabdict.stlabdict(*args, **kwargs)

Class to hold a data table with multiple lines and columns

This class is DEPRECATED in favor of pandas DataFrame. They serve the same function as an stlabdict but have much more functionality (and documentation…).

This class is essentialy an ordered_dict (is a child of) with a few convenience methods included. Each element of the dict has an index that labels the column and contains an array of numbers with the column data. It is basically a matrix where the column index are string constants instead of numbers (to more explicitly keep track or what each column contains). Can also be indexed by column number.

__getitem__(key)

Overloaded indexing of the dict

Reimplements the getting of items from the dict to allow for indexing by column position as well as by label

Parameters

key (str or int) – Desired column index or position. If a int is given, the method first checks if it is already an index. If it is not, it returns the column given by the index position.

__init__(*args, **kwargs)

Init method for stlabdict

Simply calls the ordered_dict constructor

addparcolumn(colname, colval)

Adds a parameter column

A parameter column is typically a column with a constant value for all lines (i.e. power in a vna trace). Simply repeats the same value in an array of the same length as the other columns. Does not work if there are no existing columns.

Parameters

• colname (str) – Column title for the new parameter column

• colval (float) – Value to fill the parameter column

line(nn)

Gets a line from the table

Takes a line from the stlabdict given by index. While getting a column can be done by simply taking mystlabdict[myindex], getting a line requires iterating over the dict and pulling out the desired line.

Parameters

nn (int) – Line number to be extracted

Returns

ret – New stlabdict with only the desired line (each element is labelled by the same column name as before but only contains a single float in each).

Return type

stlabdict

matrix()

Converts entire table into a numpy matrix.

Returns

Matrix containing the same data as the stlabdict. Loses column titles.

Return type

numpy.matrix

ncol()

Get the number of columns

Returns

Number of columnms in stlabdict

Return type

int

nline()

Get the number of lines in dict

Checks that all columns have the same number of lines

Returns

Number of lines in first column (should be the same for any column)

Return type

int

class stlabutils.stlabdict.stlabmtx(mtx, xtitle='xtitle', ytitle='ytitle', ztitle='ztitle')

stlabmtx class for spyview-like operations

This class implements a matrix in the form of a pandas DataFrame and contains methods analogous to those present in spyview.

mtx

Original dataframe before any processing. The dataframe indexes are considered the x and y ranges on the final matrix

Type

pandas.DataFrame

pmtx

Processed dataframe (filters applied)

Type

pandas.DataFrame

processlist

List strings specifying the applied filters (in order)

Type

array of str

xtitle, ytitle, ztitle

Titles for the x,y and z (data) axes

Type

str

xtitle0, ytitle0, ztitle0

Initial Titles for the x,y and z (data) axes (so, in case they are changed, reset can recover them)

Type

str

__init__(mtx, xtitle='xtitle', ytitle='ytitle', ztitle='ztitle')

stlab mtx initialization

Takes an input DataFrame and sets up the object

Parameters

• mtx (pandas.DataFrame) – Intup Dataframe

• ytitle, ztitle (xtitle,) – Title for x,y and z axes

absolute()

Absolute value filter

Applies np.abs to all elements of the matrix. Process string abs.

applyprocessfile(filename)

Apply all steps in a process list file

Takes in input file containing a process list and applies them to the data.

Parameters

filename (str) – Process file name

applyprocesslist(pl)

Apply all steps in array of process strings

Takes in input list of strings descibing filters to be applied to the data and runs them.

Parameters

line (str) – String describing the desired filter to be applied

applystep(line)

Apply step from a process list string

Takes in input string descibing one filter application and applies it to the data

Parameters

line (str) – String describing the desired filter to be applied

crop(left=None, right=None, up=None, low=None)

Crop filter

Crops data matrix to the given extents. Process string crop left,right,up,low

Parameters

• left (int or None, optional) – New first column of cropped array. If None, is assumed to be the first column of the whole set (no crop)

• right (int or None, optional) – New last column of cropped array. If None, is assumed to be the last column of the whole set (no crop). When given a value, the actual index specified is not included in the crop

• up (int or None, optional) – New first row of the cropped array. If None, is assumed to be the first line of the whole set (no crop) When given a value, the actual index specified is not included in the crop

• low (int or None, optional) – New first row of the cropped array. If None, is assumed to be the last line of the whole set (no crop)

delstep(ii)

Removes a filter from the current process list by index

Parameters

ii (int) – Index of filter to be removed from applied filters

detrend()

Detrend filter

Removes linear trend from data. Process string detrend`. This can be useful for phase signals.

flip(x=False, y=False)

Flip filter

Reverses x and/or y axis. Process string flip x,y (0 is false, 1 is true).

Parameters

y (x,) – If True, x or y is flipped

getextents()

Get the extents of the matrix

Returns a tuple containing (xmin, xmax, ymin, ymax) from the axis ranges, typically to correctly scale the axes when plotting with matplotlib.pyplot.imshow

Returns

Four element tuple containing (xmin,xmax,ymin,ymax)

Return type

tuple of float

insertstep(ii, line)

Inserts new filter into process list

Adds a new filter at a specific position in the process list

Parameters

• ii (int) – Index for the position of the new filter

• line (str) – Process string for the new filter

loadmtx(filename)

Load matrix from an existing Spyview mtx file

Parameters

filename (string) – Name of the mtx file to open

log()

Natural log filter

Applies log_e to all elements in the matrix. Process string log

log10()

Log10 filter

Applies log_10 to all elements in the matrix. Process string log10

logx(x)

Logx filter

Applies log_n to all elements in the matrix. Process string logx x

lowpass(x=0, y=0)

Low Pass filter

Applies a gaussian filter to the data with given pixel widths. Other filters are yet to be implemented. Process string lowpass x,y

Parameters

x,y (int, optional) – Width of the filter in the x and y direction

nan_greater(thres)

NaN for values greater than

Changes all values greater than thres to np.nan. Process string nan_greater thres.

Parameters

thres (float, optional) – Threshold value

nan_smaller(thres)

NaN for values smaller than

Changes all values smaller than thres to np.nan. Process string nan_smaller thres.

Parameters

thres (float, optional) – Threshold value

neg()

Negative filter

Multiplies matrix by -1. Process string neg

norm_cbc()

Stretch the contrast of each column to full scale

Each column gets normalized. Process string norm_cbc

norm_lbl()

Stretch the contrast of each line to full scale

Each line gets normalized. Process string norm_lbl

offset(x=0)

Offset filter

Offsets data values by adding given value. Process string offset x

Parameters

x (float, optional) – Value to add to all data values

offset_axes(x=0, y=0)

Axes offset filter

Offset axis values. Process string offset_axes x,y

Parameters

y (x,) – Values to add to the axes values of the matrix

outlier(line, vertical=1)

Outlier filter

Drop a line or column from the data. Process string outlier line,vertical

Parameters

• line (int) – Line or column number to drop

• vertical ({1,0}, optional) – If 1, drops a column. If 0, drops a line

pixel_avg(nx=0, ny=0, center=0)

Pixel average filter

Performs pixel averaging on matrix. Process string pixel_avg nx,ny,center

Parameters

• nx,ny (int, optional) – Width and height of averaging window

• center ({0,1}, optional) – I don’t know what this does… Looks like it omits the center point of each averaging window from the average?

power(x=1)

Power filter

Applies np.power to all elements in the matrix. Process string power x

Parameters

x (float,optional) –

reset()

Reset filters

Resets all filters and returns matrix to its initial state

rotate_ccw()

Rotate counter-clockwise filter

Rotates matrix and axes counter-clockwise. Process string rotate_ccw

rotate_cw()

Rotate clockwise filter

Rotates matrix and axes clockwise. Process string rotate_cw

save(name='output')

Save matrix to file

Pickels the object and saves it to given file.

Parameters

name (str) – Base filename to be used. “.mtx.pkl” will be appended to given filename

savemtx(filename='./output')

Save to Spyview mtx format

Saves current processed matrix to a spyview mtx file

Parameters

filename (str) – Name of the new mtx file. “.mtx” will be appended.

saveprocesslist(filename='./process.pl')

Save applied filter list

Saves the applied filters and parameters to a text file (process.pl in the current folder by default)

Parameters

filename (str) – Name of the new file to save the list in.

scale_data(factor=1.0)

Scale filter

Scales all data by given factor. Process string scale x

Parameters

factor (float, optional) – Value to scale the data by

sub_cbc(lowp=40, highp=40, low_limit=-1e+99, high_limit=1e+99)

Subtract column by column filter

Same as sub_lbl but done on a column by column basis. Process string sub_cbc lowp,highp,low_limit,high_limit

sub_lbl(lowp=40, highp=40, low_limit=-1e+99, high_limit=1e+99)

Substract line by line filter

The average value of each line is substracted from the data. Parts of each line cut can be excluded using the high and low percentile options. The idea is that all points are sorted in increasing order and a percentage from the back and front of the list is rejected for the average calculation. Process string sub_lbl lowp,highp,low_limit,high_limit

Parameters

• lowp (float) – Percentage of points to be rejected from the averaging on the low side.

• highp (float) – Percentage of points to be rejected from the averaging on the high side.

• low_limit (float) – Absolute value below which points are ignored for the average (and percentile calculations)

• low_limit – Absolute value above which points are ignored for the average (and percentile calculations)

sub_linecut(pos, horizontal=1)

Subtract lincut filter

Selects a line or column and subtracts it from all othe lines or columns in the matrix. Process string sub_linecut pos,horizontal

Parameters

• pos (int) – Index of line or column to be subtracted

• horizontal ({1,0}) – If 1, a line is subtrcted. If 0 a column is subtracted

transpose()

Transpose filter

Transposes the data matrix (and axes). Process string transpose

unwrap()

Unwrap filter

Unwraps the phase of data. Process string unwrap`. This can be useful for phase signals.

vi_to_iv(vmin, vmax, nbins)

vi to iv filter

Reverses the data axis with the y axis of the matrix. For example, if the data contains the voltage and the axis the current this filter replaces the voltage data with the corresponding current data and the axis with the voltage (I think…). Since the axes are expected to be ordered, this is not an immediate operation and may not be possible in many cases (repeated data values?).

If one desires to do this with the x axis instead of the y, the matrix must first be transposed. After the filter is applied the transpose can be undone.

Process string vi_to_iv vmin,vmax,nbins

Parameters

• vmin (float) – Lower end of the new y axis

• vmax (float) – Upper end of the new y axis

• nbins (int) – Number of points in the new axis

xderiv(direction=1)

X derivative filter

Apply a derivative along the lines of the matrix. Process string xderiv direction

Parameters

direction ({1,-1}) – Direction for derivative. 1 by default (normal diff derivative)

yderiv(direction=1)

Y derivative filter

Apply a derivative along the columns of the matrix. Process string yderiv direction

Parameters

direction ({1,-1}) – Direction for derivative. 1 by default (normal diff derivative)

stlabutils.stlabdict.stlabmtx_pd

alias of stlabutils.stlabdict.stlabmtx