mannkendall package

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

Submodules

mannkendall.mk_hardcoded module

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains the hardcoded parameters for the mannkendall package.

mannkendall.mk_hardcoded.VALID_PW_METHODS = ['pw', 'tfpw_y', 'tfpw_ws', 'vctfpw', '3pw']

supported pre-whitening methods “tags”

Type: list

mannkendall.mk_hardcoded.PROB_MK_N = array([[ nan, nan, nan, 6.25e-01, 5.92e-01, nan, nan, 5.48e-01, 5.40e-01, nan], [ nan, nan, nan, nan, nan, 5.00e-01, 5.00e-01, nan, nan, 5.00e-01], [ nan, nan, nan, 3.75e-01, 4.08e-01, nan, nan, 4.52e-01, 4.60e-01, nan], [ nan, nan, nan, nan, nan, 3.60e-01, 3.86e-01, nan, nan, 4.31e-01], [ nan, nan, nan, 1.67e-01, 2.42e-01, nan, nan, 3.60e-01, 3.81e-01, nan], [ nan, nan, nan, nan, nan, 2.35e-01, 2.81e-01, nan, nan, 3.64e-01], [ nan, nan, nan, 4.20e-02, 1.17e-01, nan, nan, 2.74e-01, 3.06e-01, nan], [ nan, nan, nan, nan, nan, 1.36e-01, 1.91e-01, nan, nan, 3.00e-01], [ nan, nan, nan, nan, 4.20e-02, nan, nan, 1.99e-01, 2.38e-01, nan], [ nan, nan, nan, nan, nan, 6.80e-02, 1.19e-01, nan, nan, 2.42e-01], [ nan, nan, nan, nan, 8.30e-03, nan, nan, 1.38e-01, 1.79e-01, nan], [ nan, nan, nan, nan, nan, 2.80e-02, 6.80e-02, nan, nan, 1.90e-01], [ nan, nan, nan, nan, nan, nan, nan, 8.90e-02, 1.30e-01, nan], [ nan, nan, nan, nan, nan, 8.30e-03, 3.50e-02, nan, nan, 1.46e-01], [ nan, nan, nan, nan, nan, nan, nan, 5.40e-02, 9.00e-02, nan], [ nan, nan, nan, nan, nan, 1.40e-03, 1.50e-02, nan, nan, 1.08e-01], [ nan, nan, nan, nan, nan, nan, nan, 3.10e-02, 6.00e-02, nan], [ nan, nan, nan, nan, nan, nan, 5.40e-03, nan, nan, 7.80e-02], [ nan, nan, nan, nan, nan, nan, nan, 1.60e-02, 3.80e-02, nan], [ nan, nan, nan, nan, nan, nan, 1.40e-03, nan, nan, 5.40e-02], [ nan, nan, nan, nan, nan, nan, nan, 7.10e-03, 2.20e-02, nan], [ nan, nan, nan, nan, nan, nan, 2.00e-04, nan, nan, 3.60e-02], [ nan, nan, nan, nan, nan, nan, nan, 2.80e-03, 1.20e-02, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 2.30e-02], [ nan, nan, nan, nan, nan, nan, nan, 8.70e-04, 6.30e-03, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 1.40e-02], [ nan, nan, nan, nan, nan, nan, nan, 1.90e-04, 2.90e-03, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 8.30e-03], [ nan, nan, nan, nan, nan, nan, nan, 2.50e-05, 1.20e-03, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 4.60e-03], [ nan, nan, nan, nan, nan, nan, nan, nan, 4.30e-04, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 2.30e-03], [ nan, nan, nan, nan, nan, nan, nan, nan, 1.20e-04, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 1.10e-03], [ nan, nan, nan, nan, nan, nan, nan, nan, 2.50e-05, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 4.70e-04], [ nan, nan, nan, nan, nan, nan, nan, nan, 2.80e-06, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 1.80e-04], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 5.80e-05], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 1.50e-05], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 2.80e-06], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 2.80e-07]])

MannKendall probability array

Type: ndarray

mannkendall.mk_main module

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains the core routines for the mannkendall package.

mannkendall.mk_main.prob_3pw(p_pw, p_tfpw_y, alpha_mk)

Estimate the probability of the MK test and its statistical significance.

Estimates the probability of the MK test with the 3PW method. To have the maximal certainty, P is taken as the maximum of P_PW and P_TFPW_Y.

Estimates the statistical significance of the MK test as a function of the given confidence level alpha_MK.

Parameters

p_pw (float) – probability computed from the PW prewhitened dataset
p_tfpw_y (float) – probability computed from the TFPW_Y prewhitened dataset
alpha_mk (float) – confidence level in % for the MK test.

Returns

(float, int) – P, ss

Todo

improve this docstring

mannkendall.mk_main.compute_mk_stat(obs_dts, obs, resolution, alpha_mk=95, alpha_cl=90)

Compute all the components for the MK statistics.

Parameters

obs_dts (ndarray of datetime.datetime) – a list of observation datetimes.
obs (ndarray of floats) – the data array. Must be 1-D.
resolution (float) – delta value below which two measurements are considered equivalent.
alpha_mk (float, optional) – confidence level for the Mann-Kendall test in %. Defaults to 95.
alpha_cl (float, optional) – confidence level for the Sen’s slope in %. Defaults to 90.

Returns

(dict, int, float, float) – result, s, vari, z

mannkendall.mk_main.mk_temp_aggr(multi_obs_dts, multi_obs, resolution, pw_method='3pw', alpha_mk=95, alpha_cl=90, alpha_xhomo=90, alpha_ak=95)

Applies the Mann-Kendall test and the Sen slope on the given time granularity for a data set split into different temporal aggregations.

Five prewhitening methods can be chosen:

3PW (Collaud Coen et al., 2020): 3 prewhitening methods are applied (PW and TFPW_Y to determine the statistic significance (ss) of the MK test and the VCTFPW method to compute the Sen’s slope.

PW (prewhitened, Kulkarni and von Storch, 1995)

TFPW_Y(trend free PW,Yue et al., 2001)

TFPW_WS (trend free PW, Wang and Swail, 2001)

VCTFPW (variance corrected trend free PW, Wang et al., 2015)

For the PW method, only ss autocorrelation are taken into account. The default ss for the MK test is taken at 95% confidence limit. The default ss for upper and lower confidence limits is 90% of the all intervals differences distribution.

The default ss for the autocorrelation coefficient is 95%. The default ss for the homogeneity test between temporal aggregation of the MK test is 90%. If seasonal Mann-Kendall is applied, the yearly trend is assigned only if the results of the seasonal test are homogeneous. The default ss for the homogeneity test between temporal aggregation of the seasonal MK test is 90%.

Parameters

multi_obs_dts (list of 1-D ndarray of datetime.datime) – the observation times. Each array defines a new season.
multi_obs (list of 1-D ndarray) – the observations. Each array defines a new season.
resolution (float) – interval to determine the number of ties. It should be similar to the resolution of the instrument.
pw_method (str) – must be one of [‘3pw’, ‘pw, ‘tfpw_y’, ‘tfpw_ws’, ‘vctfpw’]. Defaults to ‘3pw’.
alpha_mk (float, optional) – confidence limit for Mk test in %. Defaults to 95.
alpha_cl (float, optional) – confidence limit for the Sen’s slope in %. Defaults to 90.
alpha_xhomo (float, optional) – confidence limit for the homogeneity between seasons in %. Defaults to 90.
alpha_ak (float, optional) – confidence limit for the first lag autocorrelation in %. Defaults to 95.

Returns

dict of dict – n+1 entries, where n= number of temporal aggregation. The last item corresponds to the yearly trend. Each item comprises the following fields:

’p’ (float): probability for the statistical significance. If 3PW is applied, P= max(P_PW, P_TFPW_Y);

’ss’ (float): statistical significance:

alpha_MK if the test is ss at the alpha confidence level. Defaults to 95.

0 if the test is not ss at the alpha_MK confidence level.

-1 if the test is a TFPW_Y false positive at alpha_MK confidence level

-2 if the test is a PW false positive at alpha_MK confidence level

’slope’ (float): Sen’s slope in units/y

’ucl’ (float): upper confidence level in units/y

’lcl’ (float): lower confidence level in units/y

Note

Sources

Collaud Coen et al., Effects of the prewhitening method, the time granularity and the time segmentation on the Mann-Kendall trend detection and the associated Sen’s slope, Atmos. Meas. Tech. Discuss, https://doi.org/10.5194/amt-2020-178, 2020.

Sirois, A.: A brief and biased overview of time-series analysis of how to find that evasive trend, WMO/EMEP Workshop on Advanced Statistical Methods and Their Application to Air Quality Data Sets, Annex E., Global Atmosphere Watch No. 133, TD- No. 956, World Meteorological Organization, Geneva, Switzerland, 1998. annexe E, p. 26.

Gilbert, R.: Statistical Methods for Environmental Pollution Monitoring, Van Nostrand Reinhold Company, New York, 1987.

The explanations about MULTMK/PARTMK de C. Libiseller.

mannkendall.mk_stats module

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains the core statistical routines for the package.

mannkendall.mk_stats.std_normal_var(s, var_s)

Compute the normal standard variable Z.

From Gilbert (1987).

Parameters

s (int) – S statistics of the Mann-Kendall test computed from the S_test.
k_var (float) – variance of the time series taking into account the ties in values and time. It should be computed by Kendall_var().

Returns

float – S statistic weighted by the variance.

mannkendall.mk_stats.sen_slope(obs_dts, obs, k_var, alpha_cl=90.0)

Compute Sen’s slope.

Specifically, this computes the median of the slopes for each interval:

(xj-xi)/(j-i), j>i

The confidence limits are computed with an interpolation that is important if the number of data point is small, such as for yearly averages for a 10 year trend.

Parameters

obs_dts (ndarray of datetime.datetime) – an array of observation times. Must be 1-D.
obs (ndarray of floats) – the data array. Must be 1-D.
k_var (float) – Kendall variance, computed with Kendall_var.
confidence (float, optional) – the desired confidence limit, in %. Defaults to 90.

Returns

(float, float, float) – Sen’s slope, lower confidence limit, upper confidence limit.

Note

The slopes are returned in units of 1/s.

mannkendall.mk_stats.s_test(obs, obs_dts)

Compute the S statistics (Si) for the Mann-Kendall test.

From Gilbert (1987).

Parameters

obs (ndarray of floats) – the observations array. Must be 1-D.
obs_dts (ndarray of datetime.datetime) – a list of observation datetimes.

Returns

(float, ndarray) –

S, n.: S (float) = double sum on the sign of the difference between data pairs (Si). n (ndarray of int) = number of valid data in each year of the time series

mannkendall.mk_tools module

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains useful tools for the package.

mannkendall.mk_tools.de_sort(vals, inds)

De-sort an array of values vals that were sorted according to the indices inds.

Parameters

vals (ndarray) – a 1-D array to de-sort.
inds (ndarray of int) – the sorting indices.

Returns

ndarray – the de-sorted array.

mannkendall.mk_tools.dt_to_s(time_deltas)

A convenience function that converts an array of datetime.timedeltas into an array of floats corresponding to the total_seconds() of each elements.

Parameters: time_deltas (ndarray of datetime.timedelta) – array of timedeltas.
Returns: ndaray of flot – the same array with all elements converted to total_seconds().

Note

Adapted from SO. Reply by prgao.

mannkendall.mk_tools.nb_tie(data, resolution)

Compute the number of data point considered to be equivalent (and to be treated as “ties”).

Parameters

data (ndarray of floats) – the data array. Must be 1-D.
resolution (float) – delta value below which two measurements are considered equivalent.

Returns

ndarray of int – amount of ties in the data.

Todo

adjust docstring to better describe the function.

mannkendall.mk_tools.kendall_var(data, t, n)

Compute the variance with ties in the data and ties in time.

Parameters

data (ndarray of floats) – the data array. Must be 1-D.
t (ndarray of int) – number of elements in each tie. Must be 1-D.
n (ndarray of int) – number of non-missing data for each year. Must be 1-D.

Returns

float – the variance.

Note

Source: Eq. 4.20, GAW report 133 (A. Sirois), p.30 of annex D.

mannkendall.mk_tools.nanautocorr(obs, nlags, r=0)

Compute the Pearson R autocoreelation coefficient for an array that contains nans.

Also compute the confidence bounds b following Bartlett’s formula.

Parameters

obs (ndarray of float) – the data array. Must be 1-D.
nlags (int) – number of lags to compute.
r (int, optional) – number of lags until the model is supposed to have a significant autocorrelation coefficient. Must be < nlags. Defaults to 0.

Returns

(ndarray, float) – the autocorrelation coefficients, and the confidence bounds b.

Note

Adapted from Fabio (2020), Autocorrelation and Partial Autocorrelation with NaNs, https://www.mathworks.com/matlabcentral/fileexchange/43840-autocorrelation-and-partial-autocorrelation-with-nans, MATLAB Central File Exchange. Retrieved August 26, 2020.

mannkendall.mk_tools.levinson(r, n)

Adapts the levinson() routine from matlab.

Basically re-arranges the outputs from statsmodels.tsa.stattools.levinson_durbin() to match the matlab outputs. Includes a sign change and swapping a “1”.

For more info, see https://ch.mathworks.com/help/signal/ref/levinson.html?s_tid=srchtitle.

Todo

fix this docstring

mannkendall.mk_version module

Copyright (c) 2020 MeteoSwiss, contributors of the original Matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains the version of the code. It is used by the code itself, as well as setup.py.

mannkendall.mk_white module

Copyright (c) 2020 MeteoSwiss, contributors of the original matlab version of the code listed in ORIGINAL_AUTHORS. Copyright (c) 2020 MeteoSwiss, contributors of the Python version of the code listed in AUTHORS.

Distributed under the terms of the BSD 3-Clause License.

SPDX-License-Identifier: BSD-3-Clause

This file contains the different pre-whiteneing routines for the mannkendall package.

mannkendall.mk_white.nanprewhite_arok(obs, alpha_ak=95)

Compute the first lag autocorrelation coefficient to prewhite data as an AR(Kmax) function.

Parameters

obs (ndarray of floats) – the data array. Must be 1-D.
alpha_ak (float) – confidence level for the first lag autocorrelation in %. Defaults to 95.

Returns

(float, ndarray, int) –

ak_lag, data_prewhite, ak_ss.: ak_lag (float): first lag autocorrelation coefficient. data_prewhite (ndarray): data after removing of the first lag autorcorrelation if ak_lag is ss, original data otherwise. ak_ss (int): statistical significance of the first lag autocorrelation. alpha_ak is ss at the alpha_ak level, zero otherwise

Todo

fix the docstring

mannkendall.mk_white.prewhite(obs, obs_dts, resolution, alpha_ak=95)

Compute the necessary prewhitened datasets to assess the statistical significance, and to compute the Sen slope for each of the prewhitening method, including 3PW.

Parameters

obs (ndarray of floats) – the data array. Must be 1-D.
obs_dts (ndarray of datetime.datetime) – a list of observation datetimes.
resolution (float) – delta value below which two measurements are considered equivalent. It is used to compute the number of ties.
alpha_ak (float, optional) – statistical significance in % for the first lag autocorrelation. Defaults to 95.

Returns

(dict) –

data_pw, that contains 5 PW timeseries:

’pw’: PW with the first lag autocorrelation of the data
’pw_cor’: PW corrected with 1/(1-ak1)
’tfpw_ws’: PW with the first lag autocorrelation of the data after detrending
computed from PW data (see Wang & Swail)
’tfpw_y’: method of Yue et al 2002, not 1/1-ak1) correction, detrend on original
data
’vctfpw’: PW with the first lag autocorrelation of the data after detrending +
correction of the PW data for the variance (see Wang 2015)

Todo

fix this docstring