Linear regression with Eustatic Sea-level data

If you are running this notebook locally, make sure you’ve followed steps here to set up the environment. (This environment.yml file specifies a list of packages required to run the notebooks)

---

What we do in this notebook

Here we demonstrate use of CoFI on a real dataset linear regression problem, where we fit a polynomial function to Eustatic Sea-level heights.

• by solution of a linear system of equations,

• by optimization of a data misfit function

• by Bayesian sampling of a Likelihood multiplied by a prior.

---

Data set is from “Sea level and global ice volumes from the Last Glacial Maximum to the Holocene” K. Lambeck, H. Rouby, A. Purcell, Y. Sun, and M. Sambridge, 2014. Proc. Nat. Acad. Sci., 111, no. 43, 15296-15303, doi:10.1073/pnas.1411762111.

# Environment setup (uncomment code below)

# !pip install -U cofi

Remember to uncomment and run the code cell below as well, as we are going to load some data from GitHub.

# !git clone https://github.com/inlab-geo/cofi-examples.git
# %cd cofi-examples/examples/linear_regression

Linear regression

Lets start with some (x,y) data.

import numpy as np
import matplotlib.pyplot as plt

#
def load_data_xy(filename):

    f = open(filename, 'r')
    header = f.readline()
    lines = f.readlines()

    x = np.array([])
    y = np.array([])
    sx = np.array([])
    sy = np.array([])
    for line in lines:
        columns = line.split()
        x = np.append(x,float(columns[0]))
        y = np.append(y,float(columns[1]))
        sx = np.append(sx,float(columns[2])/2.0)
        sy = np.append(sy,float(columns[3])/2.0)

    d = x,y, sy                                   # Combine into a single data structure

    return d

def load_data_ref(filename):

    f = open(filename, 'r')
    lines = f.readlines()
    dx = np.array([]) # Age data
    dy = np.array([]) # ESL height
    dz = np.array([]) # derivative of ESL w.r.t. age
    for line in lines:
        columns = line.split()
        dx = np.append(dx,float(columns[0]))
        dy = np.append(dy,float(columns[1]))
    datavals = np.column_stack((dx,dy))  # Stack data

    return datavals

data_x,data_y,sy = load_data_xy("ESL-ff11-sorted.txt")  # Load x,sx,y,sy ESL data (x=time, Y=ESL)

maxtime = 20.
ndata = np.where(data_x>maxtime)[0][0]

data_x,data_y,sy = data_x[:ndata],data_y[:ndata],sy[:ndata]

And now lets plot the data.

def plot_data(x=data_x,y=data_y,sigma=sy,title=None):
    fig, axes = plt.subplots(figsize=(9,6))
    plt.errorbar(x, y, yerr=sy, fmt='.',color="lightcoral",ecolor='lightgrey',ms=2)
    plt.xlabel(' Time before present (ka)')
    plt.ylabel(' ESL height (m)')
    if(title != None): plt.title(title)
plot_data(title='Eustatic sea-level')

Problem description

To begin with, we will work with polynomial curves,

\[y(x) = \sum_{j=0}^M m_j x^j\,.\]

Here, \(M\) is the ‘order’ of the polynomial: if \(M=1\) we have a straight line with 2 parameters, if \(M=2\) it will be a quadratic with 3 parameters, and so on. The \(m_j, (j=0,\dots M)\) are the ‘model coefficients’ that we seek to constrain from the data.

For this class of problem the forward operator takes the following form:

\[\begin{split}\left(\begin{array}{c}y_0\\y_1\\\vdots\\y_N\end{array}\right) = \left(\begin{array}{ccc}1&x_0&x_0^2&x_0^3\\1&x_1&x_1^2&x_1^3\\\vdots&\vdots&\vdots\\1&x_N&x_N^2&x_N^3\end{array}\right)\left(\begin{array}{c}m_0\\m_1\\m_2\end{array}\right)\end{split}\]

This clearly has the required general form, \(\mathbf{d} =G{\mathbf m}\).

where:

• \(\textbf{d}\) is the vector of data values, (\(y_0,y_1,\dots,y_N\));

• \(\textbf{m}\) is the vector of model parameters, (\(m_0,m_1,m_2\));

• \(G\) is the basis matrix (or design matrix) of this linear regression problem (also called the Jacobian matrix for this linear problem).

We have a set of noisy data values, \(y_i (i=0,\dots,N)\), measured at known locations, \(x_i (i=0,\dots,N)\), and wish to find the best fit degree 3 polynomial.

The function that generated our data is assumed to have independent Gaussian random noise, \({\cal N}(0,\Sigma)\), with \((\Sigma)_{ij} = \delta_{ij}/\sigma_i^2\), where the variance of the noise on each datum, \(\sigma_i^2 (i=1,\dots,N)\), differs between observations and is given.

We now build the Jacobian/G matrix for this problem and define a forward function which simply multiplies \(\mathbf m\) by \(G\).

nparams = 5 # Number of model parameters to be solved for

def jacobian(x=data_x, n=nparams):
    return np.array([x**i for i in range(n)]).T

def forward(model):
    return jacobian().dot(model)

def Cd_inv(sigma=sy):
    factor= 10                                   # factor to inflate observational errors
    return np.diag(1./sy*1./sy)/(factor**2)

Define a reference model for later.

# Reference model for plotting
ESLref = load_data_ref("ESL-f11_yonly.txt") # Load x, y, z reference model and estimated derivative (x=time, Y=ESL, z=dESL/dt)
ndata2 = np.where(ESLref.T[0]>maxtime)[0][0]
ESLref = ESLref[:ndata2]
ref_x,ref_y = ESLref.T[0],ESLref.T[1]

Now lets plot the data with the reference curve

# Some plotting utilities
def plot_model(x,y, label, color=None,lw=0.5):
    plt.plot(x, y, color=color or "green", label=label,lw=lw)
    #plt.xlabel("X")
    #plt.ylabel("ESL")
    plt.legend()

def plot_models(models, label="Posterior samples", color="seagreen", alpha=0.1,lw=0.5):
    G = jacobian(data_x)
    plt.plot(data_x, G.dot(models[0]), color=color, label=label, alpha=alpha,lw=lw)
    for m in models:
        plt.plot(data_x, G.dot(m), color=color, alpha=alpha,lw=lw)
    plt.legend()

plot_data(title="Eustatic sea-level")
plot_model(ref_x,ref_y, "Reference model")

Now we have the data and the forward model we can start to try and estimate the coefficients of the polynomial from the data.

The structure of CoFI

In the workflow of cofi, there are three main components: BaseProblem, InversionOptions, and Inversion.

• BaseProblem defines the inverse problem including any user supplied quantities such as data vector, number of model parameters and measure of fit between model predictions and data. python     inv_problem = BaseProblem()     inv_problem.set_objective(some_function_here)     inv_problem.set_jacobian(some_function_here)     inv_problem.set_initial_model(a_starting_point) # if needed, e.g. we are solving a nonlinear problem by optimization

• InversionOptions describes details about how one wants to run the inversion, including the backend tool and solver-specific parameters. It is based on the concept of a method and tool.

  inv_options = InversionOptions()
  inv_options.suggest_solving_methods()
  inv_options.set_solving_method("matrix solvers")
  inv_options.suggest_tools()
  inv_options.set_tool("scipy.linalg.lstsq")
  inv_options.summary()
  

• Inversion can be seen as an inversion engine that takes in the above two as information, and will produce an InversionResult upon running.

  inv = Inversion(inv_problem, inv_options)
  result = inv.run()
  

Internally CoFI decides the nature of the problem from the quantities set by the user and performs internal checks to ensure it has all that it needs to solve a problem.

1. Linear system solver

from cofi import BaseProblem, InversionOptions, Inversion

Step 1. Define CoFI BaseProblem

inv_problem = BaseProblem()
inv_problem.set_data(data_y)
inv_problem.set_jacobian(jacobian())
inv_problem.set_data_covariance_inv(Cd_inv())

Step 2. Define CoFI InversionOptions

inv_options = InversionOptions()

Using the information supplied, we can ask CoFI to suggest some solving methods.

inv_options.suggest_solving_methods()

The following solving methods are supported:
{'sampling', 'optimization', 'matrix solvers'}

Use `suggest_tools()` to see a full list of backend tools for each method

We can ask CoFI to suggest some specific software tools as well.

inv_options.suggest_tools()

Here's a complete list of inversion tools supported by CoFI (grouped by methods):
{
    "optimization": [
        "scipy.optimize.minimize",
        "scipy.optimize.least_squares",
        "torch.optim"
    ],
    "matrix solvers": [
        "scipy.linalg.lstsq",
        "cofi.simple_newton"
    ],
    "sampling": [
        "emcee"
    ]
}

inv_options.set_solving_method("matrix solvers") # lets decide to use a matrix solver.
inv_options.summary()

=============================
Summary for inversion options
=============================
Solving method: matrix solvers
Use `suggest_solving_methods()` to check available solving methods.
-----------------------------
Backend tool: `scipy.linalg.lstsq (by default)` - SciPy's wrapper function over LAPACK's linear least-squares solver, using 'gelsd', 'gelsy' (default), or 'gelss' as backend driver
References: ['https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html', 'https://www.netlib.org/lapack/lug/node27.html']
Use `suggest_tools()` to check available backend tools.
-----------------------------
Solver-specific parameters: None set
Use `suggest_solver_params()` to check required/optional solver-specific parameters.

# below is optional, as this has already been the default tool under "linear least square"
inv_options.set_tool("scipy.linalg.lstsq")

Step 3. Define CoFI Inversion and run

Our choices so far have defined a linear parameter estimation problem (without any regularization) to be solved within a least squares framework. In this case the selection of a matrix solvers method will mean we are calculating the standard least squares solution

\[m = (G^T C_d^{-1} G)^{-1} G^T C_d^{-1} d\]

and our choice of backend tool scipy.linalg.lstsq, means that we will employ scipy’s linalg package to perform the numerics.

Lets run CoFI.

inv = Inversion(inv_problem, inv_options)
inv_result = inv.run()

print(f"The inversion result from `scipy.linalg.lstsq`: {inv_result.model}\n")
inv_result.summary()

The inversion result from `scipy.linalg.lstsq`: [ 1.44051039 -3.11381469  1.412872   -0.20910136  0.00653572]

============================
Summary for inversion result
============================
SUCCESS
----------------------------
model: [ 1.44051039 -3.11381469  1.412872   -0.20910136  0.00653572]
sum_of_squared_residuals: []
effective_rank: 5
singular_values: [1.72339368e+09 1.35569994e+06 3.54123758e+03 1.10384400e+02
 7.16903392e+00]
model_covariance: [[ 6.70145209e-02 -6.28173834e-02  1.67266578e-02 -1.60867740e-03
   4.77025592e-05]
 [-6.28173834e-02  7.43365995e-02 -2.23488563e-02  2.32179303e-03
  -7.20214235e-05]
 [ 1.67266578e-02 -2.23488563e-02  7.38371078e-03 -8.20763652e-04
   2.65025504e-05]
 [-1.60867740e-03  2.32179303e-03 -8.20763652e-04  9.62437921e-05
  -3.21300646e-06]
 [ 4.77025592e-05 -7.20214235e-05  2.65025504e-05 -3.21300646e-06
   1.10114955e-07]]

Lets plot the solution.

plot_data(title="Eustatic sea-level")
plot_model(data_x,jacobian(data_x).dot(inv_result.model), "linear system solver", color="seagreen")
plot_model(ref_x,ref_y, "Reference model", color="darkorange")

2. Optimizer

The same overdetermined linear problem, \(\textbf{d} = G\textbf{m}\), with Gaussian data noise can also be solved by minimising the squares of the residual of the linear equations, e.g. \(\textbf{r}^T \textbf{C}_d^{-1}\textbf{r}\) where \(\textbf{r}=\textbf{d}-G\textbf{m}\). The above matrix solver solution gives us the best data fitting model, but a direct optimisation approach could also be used, say when the number of unknowns is large and we do not wish, or are unable to provide the Jacobian function.

So we use a plain optimizer scipy.optimize.minimize to demonstrate this ability.

######## CoFI BaseProblem - provide additional information
inv_problem.set_initial_model(np.ones(nparams))
#inv_problem.set_initial_model(inv_result.model)
inv_problem.set_forward(forward)
inv_problem.set_data_misfit("squared error")

# inv_problem.set_objective(your_own_misfit_function)    # (optionally) if you'd like to define your own misfit
# inv_problem.set_gradient(your_own_gradient_of_misfit_function)    # (optionally) if you'd like to define your own misfit gradient

######## CoFI InversionOptions - set a different tool
inv_options_2 = InversionOptions()
inv_options_2.set_tool("scipy.optimize.minimize")
inv_options_2.set_params(method="Nelder-Mead")

######## CoFI Inversion - run it
inv_2 = Inversion(inv_problem, inv_options_2)
inv_result_2 = inv_2.run()

######## CoFI InversionResult - check result
print(f"The inversion result from `scipy.optimize.minimize`: {inv_result_2.model}\n")
inv_result_2.summary()

The inversion result from `scipy.optimize.minimize`: [ 4.02211771 -7.6352002   3.47983694 -0.52237961  0.02043681]

============================
Summary for inversion result
============================
FAILURE
----------------------------
fun: 396.30180936007037
nit: 598
nfev: 1000
status: 1
message: Maximum number of function evaluations has been exceeded.
final_simplex: (array([[ 4.02211771, -7.6352002 ,  3.47983694, -0.52237961,  0.02043681],
       [ 4.02227491, -7.63498804,  3.47977907, -0.52238912,  0.02043829],
       [ 4.02218792, -7.63507241,  3.47979137, -0.5223722 ,  0.02043577],
       [ 4.02245903, -7.63488993,  3.47966829, -0.52235893,  0.02043602],
       [ 4.02257488, -7.63525136,  3.47962954, -0.52233062,  0.02043424],
       [ 4.02215782, -7.63502984,  3.47981924, -0.5223883 ,  0.02043736]]), array([396.30180936, 396.30181593, 396.30181983, 396.30182837,
       396.30183836, 396.30184074]))
model: [ 4.02211771 -7.6352002   3.47983694 -0.52237961  0.02043681]

plot_data()
plot_model(data_x,jacobian(data_x).dot(inv_result_2.model), "optimization solution", color="cornflowerblue")
plot_model(ref_x,ref_y, "Reference model", color="darkorange")

The optimization fails to convergence for this problem (with default settings).

---

Challenge - Change the polynomial degree

Try and replace the 3rd order polynomial with a 2nd order polynomial (i.e. \(M=2\)) by adding the required commands below. What does the plot looks like?

Start from code below:

inv_problem = BaseProblem()
inv_problem.set_data(data_y)
inv_problem.set_jacobian(jacobian(n=<CHANGE ME>))
inv_problem.set_data_covariance_inv(Cd_inv())
inv_options.set_solving_method("matrix solvers") # lets decide to use a matrix solver.
inv = Inversion(inv_problem, inv_options)
inv_result = inv.run()

print("Inferred curve with n = <CHANGE ME> ")
plot_data()
plot_model(data_x,jacobian(x,n=<CHANGE ME>).dot(inv_result.model), "optimization solution", color="cornflowerblue")
plot_model(ref_x,ref_y, "Reference model", color="darkorange")

# Copy the template above, Replace <CHANGE ME> with your answer

#@title Solution

inv_problem = BaseProblem()
inv_problem.set_data(data_y)
inv_problem.set_jacobian(jacobian(n=3))
inv_problem.set_data_covariance_inv(Cd_inv())
inv_options.set_solving_method("matrix solvers") # lets decide to use a matrix solver.
inv = Inversion(inv_problem, inv_options)
inv_result = inv.run()

print("Inferred curve with n = 3 ")
plot_data()
plot_model(data_x,jacobian(data_x,n=3).dot(inv_result.model), "optimization solution", color="cornflowerblue")
plot_model(ref_x,ref_y, "Reference model", color="darkorange")

Inferred curve with n = 3

Changing to a second order polynomial does converge but gives a poor fit.

---

3. Bayesian sampling

Likelihood

Since data errors follow a Gaussian in this example, we can define a Likelihood function, \(p({\mathbf d}_{obs}| {\mathbf m})\).

\[p({\mathbf d}_{obs} | {\mathbf m}) \propto \exp \left\{- \frac{1}{2} ({\mathbf d}_{obs}-{\mathbf d}_{pred}({\mathbf m}))^T C_D^{-1} ({\mathbf d}_{obs}-{\mathbf d}_{pred}({\mathbf m})) \right\}\]

where \({\mathbf d}_{obs}\) represents the observed y values and \({\mathbf d}_{pred}({\mathbf m})\) are those predicted by the polynomial model \(({\mathbf m})\). The Likelihood is defined as the probability of observing the data actually observed, given a model. In practice we usually only need to evaluate the log of the Likelihood, \(\log p({\mathbf d}_{obs} | {\mathbf m})\). To do so, we require the inverse data covariance matrix describing the statistics of the noise in the data, \(C_D^{-1}\) . For this problem the data errors are independent with identical standard deviation in noise for each datum. Hence \(C_D^{-1} = \frac{1}{\sigma^2}I\) where \(\sigma=1\).

Here we artificially increase the observational errors on the data so that the spread of the posterior samples are visible.

Cdinv = Cd_inv()                 # inverse data covariance matrix

def log_likelihood(model):
    y_synthetics = forward(model)
    residual = data_y - y_synthetics
    return -0.5 * residual @ (Cdinv @ residual).T

Note that the user could specify any appropriate Likelihood function of their choosing here.

Prior

Bayesian sampling requires a prior probability density function. A common problem with polynomial coefficients as model parameters is that it is not at all obvious what a prior should be. Here we choose a uniform prior with specified bounds

\[\begin{split}\begin{align} p({\mathbf m}) &= \frac{1}{V},\quad l_i \le m_i \le u_i, \quad (i=1,\dots,M)\\ \\ &= 0, \quad {\rm otherwise}, \end{align}\end{split}\]

where \(l_i\) and \(u_i\) are lower and upper bounds on the \(i\)th model coefficient.

Here use the uniform distribution with \({\mathbf l}^T = (-10.,-10.,-10.,-10.)\), and \({\mathbf u}^T = (10.,10.,10.,10.)\).

m_lower_bound = np.ones(nparams) * (-10.)             # lower bound for uniform prior
m_upper_bound = np.ones(nparams) * 10                 # upper bound for uniform prior

def log_prior(model):    # uniform distribution
    for i in range(len(m_lower_bound)):
        if model[i] < m_lower_bound[i] or model[i] > m_upper_bound[i]: return -np.inf
    return 0.0 # model lies within bounds -> return log(1)

Note that the user could specify any appropriate Prior PDF of their choosing here.

Bayesian sampling

In this aproach we sample a probability distribution rather than find a single best fit solution. Bayes’ theorem tells us the the posterior distribution is proportional to the Likelihood and the prior.

\[p(\mathbf{m}|\mathbf{d}) = K p(\mathbf{d}|\mathbf{m})p(\mathbf{m})\]

where \(K\) is some constant. Under the assumptions specified \(p(\mathbf{m}|\mathbf{d})\) gives a probability density of models that are supported by the data. We seek to draw random samples from \(p(\mathbf{m}|\mathbf{d})\) over model space and then to make inferences from the resulting ensemble of model parameters.

In this example we make use of The Affine Invariant Markov chain Monte Carlo (MCMC) Ensemble sampler Goodman and Weare 2010 to sample the posterior distribution of the model. (See more details about emcee).

Starting points for random walkers

Now we define some hyperparameters (e.g. the number of walkers and steps), and initialise the starting positions of walkers. We start all walkers in a small ball about a chosen point \((0, 0, 0, 0)\).

nwalkers = 32
ndim = nparams
nsteps = 10000
walkers_start = np.zeros(nparams) + 1e-4 * np.random.randn(nwalkers, ndim)

Add the information and run with CoFI

######## CoFI BaseProblem - provide additional information
inv_problem.set_log_prior(log_prior)
inv_problem.set_log_likelihood(log_likelihood)
inv_problem.set_model_shape(ndim)

######## CoFI InversionOptions - get a different tool
inv_options_3 = InversionOptions()
inv_options_3.set_tool("emcee")      # Here we use to Affine Invariant McMC sampler from Goodman and Weare (2010).
inv_options_3.set_params(nwalkers=nwalkers, nsteps=nsteps, progress=True, initial_state=walkers_start)

######## CoFI Inversion - run it
inv_3 = Inversion(inv_problem, inv_options_3)
inv_result_3 = inv_3.run()

######## CoFI InversionResult - check result
print(f"The inversion result from `emcee`:")
inv_result_3.summary()

  0%|          | 0/10000 [00:00<?, ?it/s]
  0%|          | 1/10000 [00:00<19:28,  8.56it/s]
  0%|          | 3/10000 [00:00<15:26, 10.79it/s]
  0%|          | 7/10000 [00:00<09:41, 17.17it/s]
  0%|          | 18/10000 [00:00<03:46, 44.12it/s]
  0%|          | 31/10000 [00:00<02:24, 69.17it/s]
  0%|          | 39/10000 [00:00<02:24, 69.00it/s]
  0%|          | 47/10000 [00:00<02:54, 57.16it/s]
  1%|          | 54/10000 [00:01<02:48, 59.09it/s]
  1%|          | 69/10000 [00:01<02:02, 81.08it/s]
  1%|          | 84/10000 [00:01<01:41, 97.88it/s]
  1%|          | 97/10000 [00:01<01:34, 105.08it/s]
  1%|1         | 112/10000 [00:01<01:25, 115.73it/s]
  1%|1         | 127/10000 [00:01<01:19, 123.41it/s]
  1%|1         | 142/10000 [00:01<01:16, 128.92it/s]
  2%|1         | 157/10000 [00:01<01:14, 132.79it/s]
  2%|1         | 171/10000 [00:01<01:13, 133.53it/s]
  2%|1         | 185/10000 [00:02<01:14, 132.33it/s]
  2%|2         | 200/10000 [00:02<01:12, 134.81it/s]
  2%|2         | 215/10000 [00:02<01:11, 136.99it/s]
  2%|2         | 229/10000 [00:02<01:11, 137.21it/s]
  2%|2         | 244/10000 [00:02<01:10, 138.63it/s]
  3%|2         | 259/10000 [00:02<01:09, 139.62it/s]
  3%|2         | 274/10000 [00:02<01:09, 139.91it/s]
  3%|2         | 289/10000 [00:02<01:09, 140.44it/s]
  3%|3         | 304/10000 [00:02<01:08, 140.71it/s]
  3%|3         | 319/10000 [00:02<01:08, 141.12it/s]
  3%|3         | 334/10000 [00:03<01:08, 141.34it/s]
  3%|3         | 349/10000 [00:03<01:08, 141.53it/s]
  4%|3         | 364/10000 [00:03<01:07, 141.75it/s]
  4%|3         | 379/10000 [00:03<01:07, 141.86it/s]
  4%|3         | 394/10000 [00:03<01:07, 141.28it/s]
  4%|4         | 409/10000 [00:03<01:07, 141.47it/s]
  4%|4         | 424/10000 [00:03<01:07, 141.65it/s]
  4%|4         | 439/10000 [00:03<01:07, 141.80it/s]
  5%|4         | 454/10000 [00:03<01:07, 141.90it/s]
  5%|4         | 469/10000 [00:04<01:07, 141.74it/s]
  5%|4         | 484/10000 [00:04<01:13, 130.02it/s]
  5%|4         | 498/10000 [00:04<01:15, 126.59it/s]
  5%|5         | 513/10000 [00:04<01:12, 130.74it/s]
  5%|5         | 528/10000 [00:04<01:10, 133.92it/s]
  5%|5         | 543/10000 [00:04<01:09, 136.27it/s]
  6%|5         | 558/10000 [00:04<01:08, 137.80it/s]
  6%|5         | 573/10000 [00:04<01:07, 139.06it/s]
  6%|5         | 587/10000 [00:04<01:07, 139.13it/s]
  6%|6         | 602/10000 [00:05<01:07, 140.01it/s]
  6%|6         | 617/10000 [00:05<01:06, 140.60it/s]
  6%|6         | 632/10000 [00:05<01:07, 139.22it/s]
  6%|6         | 647/10000 [00:05<01:06, 140.00it/s]
  7%|6         | 662/10000 [00:05<01:06, 140.61it/s]
  7%|6         | 677/10000 [00:05<01:06, 141.03it/s]
  7%|6         | 692/10000 [00:05<01:05, 141.31it/s]
  7%|7         | 707/10000 [00:05<01:05, 141.50it/s]
  7%|7         | 722/10000 [00:05<01:05, 141.64it/s]
  7%|7         | 737/10000 [00:05<01:05, 141.15it/s]
  8%|7         | 752/10000 [00:06<01:06, 139.96it/s]
  8%|7         | 767/10000 [00:06<01:05, 140.60it/s]
  8%|7         | 782/10000 [00:06<01:05, 141.04it/s]
  8%|7         | 797/10000 [00:06<01:05, 141.35it/s]
  8%|8         | 812/10000 [00:06<01:04, 141.58it/s]
  8%|8         | 827/10000 [00:06<01:04, 141.75it/s]
  8%|8         | 842/10000 [00:06<01:04, 141.82it/s]
  9%|8         | 857/10000 [00:06<01:04, 141.89it/s]
  9%|8         | 872/10000 [00:06<01:04, 141.91it/s]
  9%|8         | 887/10000 [00:07<01:04, 141.95it/s]
  9%|9         | 902/10000 [00:07<01:04, 142.00it/s]
  9%|9         | 917/10000 [00:07<01:03, 142.04it/s]
  9%|9         | 932/10000 [00:07<01:03, 142.09it/s]
  9%|9         | 947/10000 [00:07<01:11, 126.00it/s]
 10%|9         | 962/10000 [00:07<01:09, 130.43it/s]
 10%|9         | 977/10000 [00:07<01:07, 133.73it/s]
 10%|9         | 992/10000 [00:07<01:06, 136.16it/s]
 10%|#         | 1007/10000 [00:07<01:05, 137.90it/s]
 10%|#         | 1021/10000 [00:08<01:04, 138.17it/s]
 10%|#         | 1036/10000 [00:08<01:04, 139.27it/s]
 11%|#         | 1051/10000 [00:08<01:03, 140.19it/s]
 11%|#         | 1066/10000 [00:08<01:03, 140.87it/s]
 11%|#         | 1081/10000 [00:08<01:03, 141.35it/s]
 11%|#         | 1096/10000 [00:08<01:03, 140.96it/s]
 11%|#1        | 1111/10000 [00:08<01:02, 141.35it/s]
 11%|#1        | 1126/10000 [00:08<01:02, 141.68it/s]
 11%|#1        | 1141/10000 [00:08<01:02, 141.84it/s]
 12%|#1        | 1156/10000 [00:08<01:02, 141.94it/s]
 12%|#1        | 1171/10000 [00:09<01:02, 142.04it/s]
 12%|#1        | 1186/10000 [00:09<01:02, 142.09it/s]
 12%|#2        | 1201/10000 [00:09<01:03, 139.39it/s]
 12%|#2        | 1216/10000 [00:09<01:02, 140.25it/s]
 12%|#2        | 1231/10000 [00:09<01:02, 140.88it/s]
 12%|#2        | 1246/10000 [00:09<01:01, 141.22it/s]
 13%|#2        | 1261/10000 [00:09<01:01, 141.55it/s]
 13%|#2        | 1276/10000 [00:09<01:01, 141.79it/s]
 13%|#2        | 1291/10000 [00:09<01:01, 141.74it/s]
 13%|#3        | 1306/10000 [00:10<01:01, 141.79it/s]
 13%|#3        | 1321/10000 [00:10<01:01, 141.81it/s]
 13%|#3        | 1336/10000 [00:10<01:01, 141.94it/s]
 14%|#3        | 1351/10000 [00:10<01:00, 142.08it/s]
 14%|#3        | 1366/10000 [00:10<01:00, 142.13it/s]
 14%|#3        | 1381/10000 [00:10<01:00, 142.06it/s]
 14%|#3        | 1396/10000 [00:10<01:00, 142.02it/s]
 14%|#4        | 1411/10000 [00:10<01:00, 142.11it/s]
 14%|#4        | 1426/10000 [00:10<01:00, 142.16it/s]
 14%|#4        | 1441/10000 [00:10<01:00, 141.80it/s]
 15%|#4        | 1456/10000 [00:11<01:00, 141.14it/s]
 15%|#4        | 1471/10000 [00:11<01:00, 141.43it/s]
 15%|#4        | 1486/10000 [00:11<01:00, 141.71it/s]
 15%|#5        | 1501/10000 [00:11<00:59, 141.94it/s]
 15%|#5        | 1516/10000 [00:11<00:59, 142.11it/s]
 15%|#5        | 1531/10000 [00:11<00:59, 142.09it/s]
 15%|#5        | 1546/10000 [00:11<00:59, 142.21it/s]
 16%|#5        | 1561/10000 [00:11<00:59, 142.29it/s]
 16%|#5        | 1576/10000 [00:11<00:59, 142.29it/s]
 16%|#5        | 1591/10000 [00:12<00:59, 142.30it/s]
 16%|#6        | 1606/10000 [00:12<00:58, 142.31it/s]
 16%|#6        | 1621/10000 [00:12<00:58, 142.33it/s]
 16%|#6        | 1636/10000 [00:12<00:58, 142.28it/s]
 17%|#6        | 1651/10000 [00:12<00:58, 142.25it/s]
 17%|#6        | 1666/10000 [00:12<00:58, 142.29it/s]
 17%|#6        | 1681/10000 [00:12<00:58, 142.28it/s]
 17%|#6        | 1696/10000 [00:12<00:58, 142.30it/s]
 17%|#7        | 1711/10000 [00:12<00:58, 142.28it/s]
 17%|#7        | 1726/10000 [00:12<00:58, 142.31it/s]
 17%|#7        | 1741/10000 [00:13<00:58, 142.31it/s]
 18%|#7        | 1756/10000 [00:13<00:57, 142.38it/s]
 18%|#7        | 1771/10000 [00:13<00:57, 142.14it/s]
 18%|#7        | 1786/10000 [00:13<00:57, 142.08it/s]
 18%|#8        | 1801/10000 [00:13<00:57, 141.40it/s]
 18%|#8        | 1816/10000 [00:13<00:57, 141.65it/s]
 18%|#8        | 1831/10000 [00:13<00:57, 141.80it/s]
 18%|#8        | 1846/10000 [00:13<00:57, 141.97it/s]
 19%|#8        | 1861/10000 [00:13<00:57, 142.03it/s]
 19%|#8        | 1876/10000 [00:14<00:57, 142.11it/s]
 19%|#8        | 1891/10000 [00:14<00:57, 142.18it/s]
 19%|#9        | 1906/10000 [00:14<00:56, 142.21it/s]
 19%|#9        | 1921/10000 [00:14<00:56, 142.19it/s]
 19%|#9        | 1936/10000 [00:14<00:56, 141.63it/s]
 20%|#9        | 1951/10000 [00:14<00:56, 141.75it/s]
 20%|#9        | 1966/10000 [00:14<00:56, 141.87it/s]
 20%|#9        | 1981/10000 [00:14<00:56, 141.85it/s]
 20%|#9        | 1996/10000 [00:14<00:56, 141.25it/s]
 20%|##        | 2011/10000 [00:15<00:56, 141.53it/s]
 20%|##        | 2026/10000 [00:15<00:56, 141.78it/s]
 20%|##        | 2041/10000 [00:15<00:56, 141.91it/s]
 21%|##        | 2056/10000 [00:15<00:55, 142.04it/s]
 21%|##        | 2071/10000 [00:15<00:55, 142.10it/s]
 21%|##        | 2086/10000 [00:15<00:55, 142.17it/s]
 21%|##1       | 2101/10000 [00:15<00:55, 142.17it/s]
 21%|##1       | 2116/10000 [00:15<00:55, 142.23it/s]
 21%|##1       | 2131/10000 [00:15<00:55, 142.23it/s]
 21%|##1       | 2146/10000 [00:15<00:55, 142.27it/s]
 22%|##1       | 2161/10000 [00:16<00:55, 142.23it/s]
 22%|##1       | 2176/10000 [00:16<00:54, 142.27it/s]
 22%|##1       | 2191/10000 [00:16<00:54, 142.29it/s]
 22%|##2       | 2206/10000 [00:16<00:54, 142.31it/s]
 22%|##2       | 2221/10000 [00:16<00:54, 142.34it/s]
 22%|##2       | 2236/10000 [00:16<00:54, 142.35it/s]
 23%|##2       | 2251/10000 [00:16<00:54, 141.20it/s]
 23%|##2       | 2266/10000 [00:16<00:54, 141.44it/s]
 23%|##2       | 2281/10000 [00:16<00:54, 141.74it/s]
 23%|##2       | 2296/10000 [00:17<00:54, 141.92it/s]
 23%|##3       | 2311/10000 [00:17<00:54, 142.06it/s]
 23%|##3       | 2326/10000 [00:17<00:53, 142.15it/s]
 23%|##3       | 2341/10000 [00:17<00:53, 142.19it/s]
 24%|##3       | 2356/10000 [00:17<00:53, 142.26it/s]
 24%|##3       | 2371/10000 [00:17<00:53, 142.27it/s]
 24%|##3       | 2386/10000 [00:17<00:53, 142.29it/s]
 24%|##4       | 2401/10000 [00:17<00:53, 142.30it/s]
 24%|##4       | 2416/10000 [00:17<00:53, 142.33it/s]
 24%|##4       | 2431/10000 [00:17<00:53, 142.34it/s]
 24%|##4       | 2446/10000 [00:18<00:53, 142.34it/s]
 25%|##4       | 2461/10000 [00:18<00:52, 142.36it/s]
 25%|##4       | 2476/10000 [00:18<00:52, 142.39it/s]
 25%|##4       | 2491/10000 [00:18<00:52, 142.32it/s]
 25%|##5       | 2506/10000 [00:18<00:52, 141.66it/s]
 25%|##5       | 2521/10000 [00:18<00:52, 141.42it/s]
 25%|##5       | 2536/10000 [00:18<00:52, 141.65it/s]
 26%|##5       | 2551/10000 [00:18<00:52, 141.84it/s]
 26%|##5       | 2566/10000 [00:19<01:05, 112.78it/s]
 26%|##5       | 2581/10000 [00:19<01:01, 120.04it/s]
 26%|##5       | 2594/10000 [00:19<01:07, 109.96it/s]
 26%|##6       | 2606/10000 [00:19<01:09, 106.90it/s]
 26%|##6       | 2621/10000 [00:19<01:03, 116.08it/s]
 26%|##6       | 2636/10000 [00:19<00:59, 123.14it/s]
 27%|##6       | 2651/10000 [00:19<00:57, 128.29it/s]
 27%|##6       | 2666/10000 [00:19<00:55, 132.26it/s]
 27%|##6       | 2681/10000 [00:19<00:54, 135.15it/s]
 27%|##6       | 2696/10000 [00:20<00:53, 136.94it/s]
 27%|##7       | 2711/10000 [00:20<00:52, 138.34it/s]
 27%|##7       | 2726/10000 [00:20<00:52, 139.00it/s]
 27%|##7       | 2741/10000 [00:20<00:51, 139.90it/s]
 28%|##7       | 2756/10000 [00:20<00:51, 140.61it/s]
 28%|##7       | 2771/10000 [00:20<00:51, 141.05it/s]
 28%|##7       | 2786/10000 [00:20<00:51, 141.41it/s]
 28%|##8       | 2801/10000 [00:20<00:55, 130.12it/s]
 28%|##8       | 2815/10000 [00:20<01:02, 115.69it/s]
 28%|##8       | 2827/10000 [00:21<01:28, 80.65it/s]
 28%|##8       | 2842/10000 [00:21<01:16, 93.18it/s]
 29%|##8       | 2855/10000 [00:21<01:11, 100.11it/s]
 29%|##8       | 2870/10000 [00:21<01:04, 110.48it/s]
 29%|##8       | 2883/10000 [00:21<01:18, 91.24it/s]
 29%|##8       | 2896/10000 [00:21<01:11, 99.40it/s]
 29%|##9       | 2908/10000 [00:22<01:28, 80.20it/s]
 29%|##9       | 2920/10000 [00:22<01:22, 85.65it/s]
 29%|##9       | 2931/10000 [00:22<01:18, 89.63it/s]
 29%|##9       | 2941/10000 [00:22<01:18, 89.71it/s]
 30%|##9       | 2955/10000 [00:22<01:09, 101.22it/s]
 30%|##9       | 2970/10000 [00:22<01:02, 112.17it/s]
 30%|##9       | 2985/10000 [00:22<00:58, 120.43it/s]
 30%|###       | 3000/10000 [00:22<00:55, 126.53it/s]
 30%|###       | 3015/10000 [00:22<00:53, 130.89it/s]
 30%|###       | 3030/10000 [00:23<00:51, 134.12it/s]
 30%|###       | 3045/10000 [00:23<00:50, 136.47it/s]
 31%|###       | 3060/10000 [00:23<00:50, 138.17it/s]
 31%|###       | 3075/10000 [00:23<00:49, 139.36it/s]
 31%|###       | 3090/10000 [00:23<00:49, 140.15it/s]
 31%|###1      | 3105/10000 [00:23<00:49, 140.03it/s]
 31%|###1      | 3120/10000 [00:23<00:48, 140.56it/s]
 31%|###1      | 3135/10000 [00:23<00:48, 141.04it/s]
 32%|###1      | 3150/10000 [00:23<00:50, 135.79it/s]
 32%|###1      | 3165/10000 [00:23<00:49, 137.69it/s]
 32%|###1      | 3180/10000 [00:24<00:49, 138.94it/s]
 32%|###1      | 3195/10000 [00:24<00:48, 139.87it/s]
 32%|###2      | 3210/10000 [00:24<01:08, 98.76it/s]
 32%|###2      | 3222/10000 [00:24<01:15, 90.00it/s]
 32%|###2      | 3234/10000 [00:24<01:10, 95.76it/s]
 32%|###2      | 3245/10000 [00:24<01:16, 87.74it/s]
 33%|###2      | 3255/10000 [00:25<01:27, 77.10it/s]
 33%|###2      | 3264/10000 [00:25<02:15, 49.63it/s]
 33%|###2      | 3278/10000 [00:25<01:44, 64.17it/s]
 33%|###2      | 3293/10000 [00:25<01:24, 79.37it/s]
 33%|###3      | 3308/10000 [00:25<01:11, 93.12it/s]
 33%|###3      | 3323/10000 [00:25<01:03, 104.81it/s]
 33%|###3      | 3336/10000 [00:26<01:33, 71.21it/s]
 33%|###3      | 3346/10000 [00:26<01:56, 56.92it/s]
 34%|###3      | 3361/10000 [00:26<01:32, 71.40it/s]
 34%|###3      | 3376/10000 [00:26<01:17, 85.23it/s]
 34%|###3      | 3390/10000 [00:26<01:09, 95.45it/s]
 34%|###4      | 3405/10000 [00:26<01:01, 106.47it/s]
 34%|###4      | 3418/10000 [00:27<01:43, 63.39it/s]
 34%|###4      | 3428/10000 [00:27<01:36, 67.85it/s]
 34%|###4      | 3449/10000 [00:27<01:09, 93.74it/s]
 35%|###4      | 3470/10000 [00:27<00:55, 117.08it/s]
 35%|###4      | 3488/10000 [00:27<00:49, 131.45it/s]
 35%|###5      | 3504/10000 [00:27<00:48, 134.31it/s]
 35%|###5      | 3520/10000 [00:27<00:47, 135.64it/s]
 35%|###5      | 3535/10000 [00:28<00:47, 137.19it/s]
 36%|###5      | 3550/10000 [00:28<00:46, 138.58it/s]
 36%|###5      | 3565/10000 [00:28<00:46, 139.61it/s]
 36%|###5      | 3580/10000 [00:28<00:45, 140.35it/s]
 36%|###5      | 3595/10000 [00:28<00:45, 140.17it/s]
 36%|###6      | 3610/10000 [00:28<00:45, 140.73it/s]
 36%|###6      | 3625/10000 [00:28<00:45, 141.13it/s]
 36%|###6      | 3640/10000 [00:28<00:44, 141.47it/s]
 37%|###6      | 3655/10000 [00:28<00:44, 141.62it/s]
 37%|###6      | 3670/10000 [00:29<00:44, 141.72it/s]
 37%|###6      | 3685/10000 [00:29<00:44, 141.83it/s]
 37%|###7      | 3700/10000 [00:29<00:44, 141.84it/s]
 37%|###7      | 3715/10000 [00:29<00:44, 142.01it/s]
 37%|###7      | 3730/10000 [00:29<00:44, 142.14it/s]
 37%|###7      | 3745/10000 [00:29<00:43, 142.17it/s]
 38%|###7      | 3760/10000 [00:29<00:43, 142.15it/s]
 38%|###7      | 3775/10000 [00:29<00:43, 142.25it/s]
 38%|###7      | 3790/10000 [00:29<00:43, 142.36it/s]
 38%|###8      | 3805/10000 [00:29<00:43, 142.09it/s]
 38%|###8      | 3820/10000 [00:30<00:43, 142.05it/s]
 38%|###8      | 3835/10000 [00:30<00:43, 142.19it/s]
 38%|###8      | 3850/10000 [00:30<00:43, 142.28it/s]
 39%|###8      | 3865/10000 [00:30<00:43, 142.26it/s]
 39%|###8      | 3880/10000 [00:30<00:42, 142.33it/s]
 39%|###8      | 3895/10000 [00:30<00:42, 142.15it/s]
 39%|###9      | 3910/10000 [00:30<00:42, 142.10it/s]
 39%|###9      | 3925/10000 [00:30<00:42, 142.21it/s]
 39%|###9      | 3940/10000 [00:30<00:42, 142.26it/s]
 40%|###9      | 3955/10000 [00:31<00:42, 142.35it/s]
 40%|###9      | 3970/10000 [00:31<00:42, 142.40it/s]
 40%|###9      | 3985/10000 [00:31<00:42, 142.45it/s]
 40%|####      | 4000/10000 [00:31<00:42, 142.48it/s]
 40%|####      | 4015/10000 [00:31<00:42, 142.47it/s]
 40%|####      | 4030/10000 [00:31<00:41, 142.51it/s]
 40%|####      | 4045/10000 [00:31<00:41, 142.48it/s]
 41%|####      | 4060/10000 [00:31<00:41, 142.52it/s]
 41%|####      | 4075/10000 [00:31<00:41, 141.76it/s]
 41%|####      | 4090/10000 [00:31<00:41, 141.85it/s]
 41%|####1     | 4105/10000 [00:32<00:41, 141.99it/s]
 41%|####1     | 4120/10000 [00:32<00:41, 142.12it/s]
 41%|####1     | 4135/10000 [00:32<00:41, 142.21it/s]
 42%|####1     | 4150/10000 [00:32<00:41, 142.24it/s]
 42%|####1     | 4165/10000 [00:32<00:40, 142.32it/s]
 42%|####1     | 4180/10000 [00:32<00:40, 142.34it/s]
 42%|####1     | 4195/10000 [00:32<00:40, 142.37it/s]
 42%|####2     | 4210/10000 [00:32<00:40, 142.43it/s]
 42%|####2     | 4225/10000 [00:33<01:12, 79.81it/s]
 42%|####2     | 4237/10000 [00:33<01:26, 66.40it/s]
 43%|####2     | 4252/10000 [00:33<01:12, 79.72it/s]
 43%|####2     | 4267/10000 [00:33<01:02, 92.21it/s]
 43%|####2     | 4282/10000 [00:33<00:55, 103.38it/s]
 43%|####2     | 4297/10000 [00:33<00:50, 112.78it/s]
 43%|####3     | 4311/10000 [00:34<01:35, 59.63it/s]
 43%|####3     | 4326/10000 [00:34<01:18, 72.43it/s]
 43%|####3     | 4341/10000 [00:34<01:06, 85.18it/s]
 44%|####3     | 4356/10000 [00:34<00:58, 96.99it/s]
 44%|####3     | 4371/10000 [00:34<00:52, 107.31it/s]
 44%|####3     | 4385/10000 [00:34<00:48, 114.95it/s]
 44%|####3     | 4399/10000 [00:35<01:30, 61.83it/s]
 44%|####4     | 4414/10000 [00:35<01:14, 74.85it/s]
 44%|####4     | 4429/10000 [00:35<01:03, 87.52it/s]
 44%|####4     | 4444/10000 [00:35<00:56, 99.08it/s]
 45%|####4     | 4457/10000 [00:35<00:54, 102.00it/s]
 45%|####4     | 4472/10000 [00:35<00:49, 111.83it/s]
 45%|####4     | 4487/10000 [00:36<00:46, 119.66it/s]
 45%|####5     | 4502/10000 [00:36<00:43, 125.70it/s]
 45%|####5     | 4517/10000 [00:36<00:42, 130.29it/s]
 45%|####5     | 4532/10000 [00:36<00:40, 133.61it/s]
 45%|####5     | 4547/10000 [00:36<00:40, 136.04it/s]
 46%|####5     | 4562/10000 [00:36<00:39, 137.82it/s]
 46%|####5     | 4577/10000 [00:36<00:39, 139.01it/s]
 46%|####5     | 4592/10000 [00:36<00:38, 139.87it/s]
 46%|####6     | 4607/10000 [00:36<00:38, 140.42it/s]
 46%|####6     | 4622/10000 [00:37<00:38, 140.65it/s]
 46%|####6     | 4637/10000 [00:37<00:38, 140.96it/s]
 47%|####6     | 4652/10000 [00:37<00:37, 141.37it/s]
 47%|####6     | 4667/10000 [00:37<00:37, 141.59it/s]
 47%|####6     | 4682/10000 [00:37<00:37, 141.71it/s]
 47%|####6     | 4697/10000 [00:37<00:37, 141.87it/s]
 47%|####7     | 4712/10000 [00:37<00:37, 141.95it/s]
 47%|####7     | 4727/10000 [00:37<00:37, 142.04it/s]
 47%|####7     | 4742/10000 [00:37<00:37, 142.11it/s]
 48%|####7     | 4757/10000 [00:37<00:37, 141.21it/s]
 48%|####7     | 4772/10000 [00:38<00:36, 141.49it/s]
 48%|####7     | 4787/10000 [00:38<00:36, 141.62it/s]
 48%|####8     | 4802/10000 [00:38<00:36, 141.92it/s]
 48%|####8     | 4817/10000 [00:38<00:36, 142.09it/s]
 48%|####8     | 4832/10000 [00:38<00:36, 141.48it/s]
 48%|####8     | 4847/10000 [00:38<00:36, 141.79it/s]
 49%|####8     | 4862/10000 [00:38<00:36, 142.02it/s]
 49%|####8     | 4877/10000 [00:38<00:36, 142.16it/s]
 49%|####8     | 4892/10000 [00:38<00:35, 142.12it/s]
 49%|####9     | 4907/10000 [00:39<00:35, 142.05it/s]
 49%|####9     | 4922/10000 [00:39<00:35, 142.06it/s]
 49%|####9     | 4937/10000 [00:39<00:35, 142.12it/s]
 50%|####9     | 4952/10000 [00:39<00:35, 142.07it/s]
 50%|####9     | 4967/10000 [00:39<00:35, 142.07it/s]
 50%|####9     | 4982/10000 [00:39<00:35, 142.05it/s]
 50%|####9     | 4997/10000 [00:39<00:35, 141.92it/s]
 50%|#####     | 5012/10000 [00:39<00:35, 141.97it/s]
 50%|#####     | 5027/10000 [00:39<00:35, 141.97it/s]
 50%|#####     | 5042/10000 [00:39<00:34, 141.76it/s]
 51%|#####     | 5057/10000 [00:40<00:34, 141.53it/s]
 51%|#####     | 5072/10000 [00:40<00:34, 141.60it/s]
 51%|#####     | 5087/10000 [00:40<00:34, 141.73it/s]
 51%|#####1    | 5102/10000 [00:40<00:34, 141.80it/s]
 51%|#####1    | 5117/10000 [00:40<00:34, 141.89it/s]
 51%|#####1    | 5132/10000 [00:40<00:34, 141.94it/s]
 51%|#####1    | 5147/10000 [00:40<00:34, 141.91it/s]
 52%|#####1    | 5162/10000 [00:40<00:34, 142.11it/s]
 52%|#####1    | 5177/10000 [00:40<00:33, 142.25it/s]
 52%|#####1    | 5192/10000 [00:41<00:33, 142.28it/s]
 52%|#####2    | 5207/10000 [00:41<00:33, 142.28it/s]
 52%|#####2    | 5222/10000 [00:41<00:33, 141.99it/s]
 52%|#####2    | 5237/10000 [00:41<00:33, 142.16it/s]
 53%|#####2    | 5252/10000 [00:41<00:33, 142.28it/s]
 53%|#####2    | 5267/10000 [00:41<00:33, 142.41it/s]
 53%|#####2    | 5282/10000 [00:41<00:33, 142.47it/s]
 53%|#####2    | 5297/10000 [00:41<00:32, 142.55it/s]
 53%|#####3    | 5312/10000 [00:41<00:32, 142.64it/s]
 53%|#####3    | 5327/10000 [00:41<00:32, 142.65it/s]
 53%|#####3    | 5342/10000 [00:42<00:32, 141.78it/s]
 54%|#####3    | 5357/10000 [00:42<00:32, 141.85it/s]
 54%|#####3    | 5372/10000 [00:42<00:32, 142.10it/s]
 54%|#####3    | 5387/10000 [00:42<00:32, 142.19it/s]
 54%|#####4    | 5402/10000 [00:42<00:32, 142.28it/s]
 54%|#####4    | 5417/10000 [00:42<00:32, 142.33it/s]
 54%|#####4    | 5432/10000 [00:42<00:32, 142.41it/s]
 54%|#####4    | 5447/10000 [00:42<00:31, 142.48it/s]
 55%|#####4    | 5462/10000 [00:42<00:31, 142.52it/s]
 55%|#####4    | 5477/10000 [00:43<00:31, 142.61it/s]
 55%|#####4    | 5492/10000 [00:43<00:31, 142.51it/s]
 55%|#####5    | 5507/10000 [00:43<00:31, 142.54it/s]
 55%|#####5    | 5522/10000 [00:43<00:31, 142.53it/s]
 55%|#####5    | 5537/10000 [00:43<00:31, 142.58it/s]
 56%|#####5    | 5552/10000 [00:43<00:31, 141.87it/s]
 56%|#####5    | 5567/10000 [00:43<00:31, 141.99it/s]
 56%|#####5    | 5582/10000 [00:43<00:31, 142.18it/s]
 56%|#####5    | 5597/10000 [00:43<00:30, 142.14it/s]
 56%|#####6    | 5612/10000 [00:43<00:30, 142.34it/s]
 56%|#####6    | 5627/10000 [00:44<00:30, 142.42it/s]
 56%|#####6    | 5642/10000 [00:44<00:30, 142.49it/s]
 57%|#####6    | 5657/10000 [00:44<00:30, 142.46it/s]
 57%|#####6    | 5672/10000 [00:44<00:30, 142.54it/s]
 57%|#####6    | 5687/10000 [00:44<00:30, 142.53it/s]
 57%|#####7    | 5702/10000 [00:44<00:30, 142.58it/s]
 57%|#####7    | 5717/10000 [00:44<00:30, 142.63it/s]
 57%|#####7    | 5732/10000 [00:44<00:29, 142.67it/s]
 57%|#####7    | 5747/10000 [00:44<00:29, 141.96it/s]
 58%|#####7    | 5762/10000 [00:45<00:30, 140.12it/s]
 58%|#####7    | 5777/10000 [00:45<00:29, 140.85it/s]
 58%|#####7    | 5792/10000 [00:45<00:29, 141.36it/s]
 58%|#####8    | 5807/10000 [00:45<00:29, 141.75it/s]
 58%|#####8    | 5822/10000 [00:45<00:29, 141.98it/s]
 58%|#####8    | 5837/10000 [00:45<00:29, 142.25it/s]
 59%|#####8    | 5852/10000 [00:45<00:29, 142.35it/s]
 59%|#####8    | 5867/10000 [00:45<00:29, 142.46it/s]
 59%|#####8    | 5882/10000 [00:45<00:28, 142.62it/s]
 59%|#####8    | 5897/10000 [00:45<00:28, 142.76it/s]
 59%|#####9    | 5912/10000 [00:46<00:28, 142.84it/s]
 59%|#####9    | 5927/10000 [00:46<00:28, 142.93it/s]
 59%|#####9    | 5942/10000 [00:46<00:28, 143.02it/s]
 60%|#####9    | 5957/10000 [00:46<00:28, 143.03it/s]
 60%|#####9    | 5972/10000 [00:46<00:28, 143.03it/s]
 60%|#####9    | 5987/10000 [00:46<00:28, 142.98it/s]
 60%|######    | 6002/10000 [00:46<00:27, 143.00it/s]
 60%|######    | 6017/10000 [00:46<00:27, 143.01it/s]
 60%|######    | 6032/10000 [00:46<00:27, 143.02it/s]
 60%|######    | 6047/10000 [00:47<00:27, 143.02it/s]
 61%|######    | 6062/10000 [00:47<00:27, 143.01it/s]
 61%|######    | 6077/10000 [00:47<00:27, 143.01it/s]
 61%|######    | 6092/10000 [00:47<00:27, 143.03it/s]
 61%|######1   | 6107/10000 [00:47<00:27, 143.04it/s]
 61%|######1   | 6122/10000 [00:47<00:27, 142.96it/s]
 61%|######1   | 6137/10000 [00:47<00:27, 142.88it/s]
 62%|######1   | 6152/10000 [00:47<00:26, 142.91it/s]
 62%|######1   | 6167/10000 [00:47<00:26, 142.92it/s]
 62%|######1   | 6182/10000 [00:47<00:26, 142.95it/s]
 62%|######1   | 6197/10000 [00:48<00:26, 142.85it/s]
 62%|######2   | 6212/10000 [00:48<00:26, 142.84it/s]
 62%|######2   | 6227/10000 [00:48<00:26, 142.82it/s]
 62%|######2   | 6242/10000 [00:48<00:26, 142.87it/s]
 63%|######2   | 6257/10000 [00:48<00:26, 142.22it/s]
 63%|######2   | 6272/10000 [00:48<00:26, 142.36it/s]
 63%|######2   | 6287/10000 [00:48<00:26, 142.51it/s]
 63%|######3   | 6302/10000 [00:48<00:25, 142.67it/s]
 63%|######3   | 6317/10000 [00:48<00:25, 142.80it/s]
 63%|######3   | 6332/10000 [00:49<00:25, 142.86it/s]
 63%|######3   | 6347/10000 [00:49<00:25, 142.92it/s]
 64%|######3   | 6362/10000 [00:49<00:25, 142.95it/s]
 64%|######3   | 6377/10000 [00:49<00:25, 142.93it/s]
 64%|######3   | 6392/10000 [00:49<00:25, 142.91it/s]
 64%|######4   | 6407/10000 [00:49<00:25, 142.68it/s]
 64%|######4   | 6422/10000 [00:49<00:25, 142.45it/s]
 64%|######4   | 6437/10000 [00:49<00:24, 142.58it/s]
 65%|######4   | 6452/10000 [00:49<00:24, 142.66it/s]
 65%|######4   | 6467/10000 [00:49<00:24, 142.57it/s]
 65%|######4   | 6482/10000 [00:50<00:24, 142.58it/s]
 65%|######4   | 6497/10000 [00:50<00:24, 142.64it/s]
 65%|######5   | 6512/10000 [00:50<00:24, 142.72it/s]
 65%|######5   | 6527/10000 [00:50<00:24, 142.81it/s]
 65%|######5   | 6542/10000 [00:50<00:24, 142.82it/s]
 66%|######5   | 6557/10000 [00:50<00:24, 142.81it/s]
 66%|######5   | 6572/10000 [00:50<00:23, 142.85it/s]
 66%|######5   | 6587/10000 [00:50<00:23, 142.85it/s]
 66%|######6   | 6602/10000 [00:50<00:23, 142.90it/s]
 66%|######6   | 6617/10000 [00:51<00:23, 142.93it/s]
 66%|######6   | 6632/10000 [00:51<00:23, 142.94it/s]
 66%|######6   | 6647/10000 [00:51<00:23, 143.01it/s]
 67%|######6   | 6662/10000 [00:51<00:23, 143.06it/s]
 67%|######6   | 6677/10000 [00:51<00:23, 143.08it/s]
 67%|######6   | 6692/10000 [00:51<00:23, 143.02it/s]
 67%|######7   | 6707/10000 [00:51<00:23, 142.74it/s]
 67%|######7   | 6722/10000 [00:51<00:22, 142.75it/s]
 67%|######7   | 6737/10000 [00:51<00:22, 142.75it/s]
 68%|######7   | 6752/10000 [00:51<00:22, 142.72it/s]
 68%|######7   | 6767/10000 [00:52<00:22, 142.60it/s]
 68%|######7   | 6782/10000 [00:52<00:22, 142.58it/s]
 68%|######7   | 6797/10000 [00:52<00:22, 142.68it/s]
 68%|######8   | 6812/10000 [00:52<00:22, 142.76it/s]
 68%|######8   | 6827/10000 [00:52<00:22, 142.77it/s]
 68%|######8   | 6842/10000 [00:52<00:22, 142.74it/s]
 69%|######8   | 6857/10000 [00:52<00:22, 142.66it/s]
 69%|######8   | 6872/10000 [00:52<00:21, 142.70it/s]
 69%|######8   | 6887/10000 [00:52<00:21, 142.60it/s]
 69%|######9   | 6902/10000 [00:53<00:21, 142.64it/s]
 69%|######9   | 6917/10000 [00:53<00:21, 142.36it/s]
 69%|######9   | 6932/10000 [00:53<00:21, 142.35it/s]
 69%|######9   | 6947/10000 [00:53<00:21, 142.44it/s]
 70%|######9   | 6962/10000 [00:53<00:21, 142.49it/s]
 70%|######9   | 6977/10000 [00:53<00:21, 142.16it/s]
 70%|######9   | 6992/10000 [00:53<00:21, 142.22it/s]
 70%|#######   | 7007/10000 [00:53<00:21, 142.32it/s]
 70%|#######   | 7022/10000 [00:53<00:20, 142.47it/s]
 70%|#######   | 7037/10000 [00:53<00:20, 142.58it/s]
 71%|#######   | 7052/10000 [00:54<00:20, 142.61it/s]
 71%|#######   | 7067/10000 [00:54<00:20, 142.70it/s]
 71%|#######   | 7082/10000 [00:54<00:20, 142.71it/s]
 71%|#######   | 7097/10000 [00:54<00:20, 142.78it/s]
 71%|#######1  | 7112/10000 [00:54<00:20, 142.78it/s]
 71%|#######1  | 7127/10000 [00:54<00:20, 142.74it/s]
 71%|#######1  | 7142/10000 [00:54<00:20, 142.71it/s]
 72%|#######1  | 7157/10000 [00:54<00:19, 142.63it/s]
 72%|#######1  | 7172/10000 [00:54<00:19, 141.92it/s]
 72%|#######1  | 7187/10000 [00:55<00:19, 142.05it/s]
 72%|#######2  | 7202/10000 [00:55<00:19, 142.08it/s]
 72%|#######2  | 7217/10000 [00:55<00:19, 142.19it/s]
 72%|#######2  | 7232/10000 [00:55<00:19, 142.36it/s]
 72%|#######2  | 7247/10000 [00:55<00:19, 138.18it/s]
 73%|#######2  | 7262/10000 [00:55<00:19, 139.46it/s]
 73%|#######2  | 7277/10000 [00:55<00:19, 140.48it/s]
 73%|#######2  | 7292/10000 [00:55<00:19, 141.19it/s]
 73%|#######3  | 7307/10000 [00:55<00:19, 141.57it/s]
 73%|#######3  | 7322/10000 [00:55<00:18, 141.81it/s]
 73%|#######3  | 7337/10000 [00:56<00:18, 142.07it/s]
 74%|#######3  | 7352/10000 [00:56<00:18, 142.29it/s]
 74%|#######3  | 7367/10000 [00:56<00:18, 142.50it/s]
 74%|#######3  | 7382/10000 [00:56<00:18, 142.57it/s]
 74%|#######3  | 7397/10000 [00:56<00:18, 142.52it/s]
 74%|#######4  | 7412/10000 [00:56<00:18, 142.42it/s]
 74%|#######4  | 7427/10000 [00:56<00:18, 142.46it/s]
 74%|#######4  | 7442/10000 [00:56<00:17, 142.55it/s]
 75%|#######4  | 7457/10000 [00:56<00:17, 142.54it/s]
 75%|#######4  | 7472/10000 [00:57<00:17, 142.66it/s]
 75%|#######4  | 7487/10000 [00:57<00:17, 142.64it/s]
 75%|#######5  | 7502/10000 [00:57<00:17, 142.68it/s]
 75%|#######5  | 7517/10000 [00:57<00:17, 142.74it/s]
 75%|#######5  | 7532/10000 [00:57<00:17, 142.79it/s]
 75%|#######5  | 7547/10000 [00:57<00:17, 142.83it/s]
 76%|#######5  | 7562/10000 [00:57<00:17, 142.87it/s]
 76%|#######5  | 7577/10000 [00:57<00:16, 142.90it/s]
 76%|#######5  | 7592/10000 [00:57<00:16, 142.94it/s]
 76%|#######6  | 7607/10000 [00:57<00:16, 142.83it/s]
 76%|#######6  | 7622/10000 [00:58<00:16, 142.75it/s]
 76%|#######6  | 7637/10000 [00:58<00:16, 142.77it/s]
 77%|#######6  | 7652/10000 [00:58<00:16, 142.75it/s]
 77%|#######6  | 7667/10000 [00:58<00:16, 142.74it/s]
 77%|#######6  | 7682/10000 [00:58<00:16, 142.39it/s]
 77%|#######6  | 7697/10000 [00:58<00:16, 142.38it/s]
 77%|#######7  | 7712/10000 [00:58<00:16, 142.01it/s]
 77%|#######7  | 7727/10000 [00:58<00:15, 142.31it/s]
 77%|#######7  | 7742/10000 [00:58<00:15, 142.52it/s]
 78%|#######7  | 7757/10000 [00:59<00:15, 141.90it/s]
 78%|#######7  | 7772/10000 [00:59<00:15, 142.23it/s]
 78%|#######7  | 7787/10000 [00:59<00:15, 141.80it/s]
 78%|#######8  | 7802/10000 [00:59<00:15, 142.14it/s]
 78%|#######8  | 7817/10000 [00:59<00:15, 142.31it/s]
 78%|#######8  | 7832/10000 [00:59<00:15, 142.22it/s]
 78%|#######8  | 7847/10000 [00:59<00:15, 142.08it/s]
 79%|#######8  | 7862/10000 [00:59<00:15, 142.19it/s]
 79%|#######8  | 7877/10000 [00:59<00:14, 142.46it/s]
 79%|#######8  | 7892/10000 [00:59<00:14, 142.31it/s]
 79%|#######9  | 7907/10000 [01:00<00:14, 142.36it/s]
 79%|#######9  | 7922/10000 [01:00<00:14, 142.45it/s]
 79%|#######9  | 7937/10000 [01:00<00:14, 142.50it/s]
 80%|#######9  | 7952/10000 [01:00<00:14, 142.50it/s]
 80%|#######9  | 7967/10000 [01:00<00:14, 142.66it/s]
 80%|#######9  | 7982/10000 [01:00<00:14, 142.73it/s]
 80%|#######9  | 7997/10000 [01:00<00:14, 142.75it/s]
 80%|########  | 8012/10000 [01:00<00:13, 142.67it/s]
 80%|########  | 8027/10000 [01:00<00:13, 142.68it/s]
 80%|########  | 8042/10000 [01:01<00:13, 142.73it/s]
 81%|########  | 8057/10000 [01:01<00:13, 142.68it/s]
 81%|########  | 8072/10000 [01:01<00:13, 142.79it/s]
 81%|########  | 8087/10000 [01:01<00:13, 142.84it/s]
 81%|########1 | 8102/10000 [01:01<00:13, 142.82it/s]
 81%|########1 | 8117/10000 [01:01<00:13, 142.83it/s]
 81%|########1 | 8132/10000 [01:01<00:13, 142.71it/s]
 81%|########1 | 8147/10000 [01:01<00:12, 142.61it/s]
 82%|########1 | 8162/10000 [01:01<00:12, 142.66it/s]
 82%|########1 | 8177/10000 [01:01<00:12, 142.72it/s]
 82%|########1 | 8192/10000 [01:02<00:12, 142.80it/s]
 82%|########2 | 8207/10000 [01:02<00:12, 142.76it/s]
 82%|########2 | 8222/10000 [01:02<00:12, 142.76it/s]
 82%|########2 | 8237/10000 [01:02<00:12, 142.80it/s]
 83%|########2 | 8252/10000 [01:02<00:12, 142.77it/s]
 83%|########2 | 8267/10000 [01:02<00:12, 142.77it/s]
 83%|########2 | 8282/10000 [01:02<00:12, 142.51it/s]
 83%|########2 | 8297/10000 [01:02<00:11, 142.65it/s]
 83%|########3 | 8312/10000 [01:02<00:11, 142.61it/s]
 83%|########3 | 8327/10000 [01:03<00:11, 142.67it/s]
 83%|########3 | 8342/10000 [01:03<00:11, 142.63it/s]
 84%|########3 | 8357/10000 [01:03<00:11, 142.70it/s]
 84%|########3 | 8372/10000 [01:03<00:11, 142.78it/s]
 84%|########3 | 8387/10000 [01:03<00:11, 142.81it/s]
 84%|########4 | 8402/10000 [01:03<00:11, 142.24it/s]
 84%|########4 | 8417/10000 [01:03<00:11, 142.38it/s]
 84%|########4 | 8432/10000 [01:03<00:11, 142.54it/s]
 84%|########4 | 8447/10000 [01:03<00:10, 142.55it/s]
 85%|########4 | 8462/10000 [01:03<00:10, 142.66it/s]
 85%|########4 | 8477/10000 [01:04<00:10, 142.71it/s]
 85%|########4 | 8492/10000 [01:04<00:10, 142.81it/s]
 85%|########5 | 8507/10000 [01:04<00:10, 142.79it/s]
 85%|########5 | 8522/10000 [01:04<00:10, 142.69it/s]
 85%|########5 | 8537/10000 [01:04<00:10, 142.76it/s]
 86%|########5 | 8552/10000 [01:04<00:10, 142.81it/s]
 86%|########5 | 8567/10000 [01:04<00:10, 142.68it/s]
 86%|########5 | 8582/10000 [01:04<00:09, 142.73it/s]
 86%|########5 | 8597/10000 [01:04<00:09, 142.01it/s]
 86%|########6 | 8612/10000 [01:05<00:09, 142.27it/s]
 86%|########6 | 8627/10000 [01:05<00:09, 142.44it/s]
 86%|########6 | 8642/10000 [01:05<00:09, 142.56it/s]
 87%|########6 | 8657/10000 [01:05<00:09, 142.68it/s]
 87%|########6 | 8672/10000 [01:05<00:09, 141.48it/s]
 87%|########6 | 8687/10000 [01:05<00:09, 141.09it/s]
 87%|########7 | 8702/10000 [01:05<00:09, 140.65it/s]
 87%|########7 | 8717/10000 [01:05<00:09, 140.56it/s]
 87%|########7 | 8732/10000 [01:05<00:09, 140.47it/s]
 87%|########7 | 8747/10000 [01:05<00:08, 140.32it/s]
 88%|########7 | 8762/10000 [01:06<00:08, 140.17it/s]
 88%|########7 | 8777/10000 [01:06<00:08, 140.18it/s]
 88%|########7 | 8792/10000 [01:06<00:08, 140.01it/s]
 88%|########8 | 8807/10000 [01:06<00:08, 139.96it/s]
 88%|########8 | 8821/10000 [01:06<00:08, 139.91it/s]
 88%|########8 | 8836/10000 [01:06<00:08, 140.01it/s]
 89%|########8 | 8851/10000 [01:06<00:08, 139.93it/s]
 89%|########8 | 8865/10000 [01:06<00:08, 139.91it/s]
 89%|########8 | 8880/10000 [01:06<00:07, 140.71it/s]
 89%|########8 | 8895/10000 [01:07<00:07, 141.40it/s]
 89%|########9 | 8910/10000 [01:07<00:07, 141.83it/s]
 89%|########9 | 8925/10000 [01:07<00:07, 142.18it/s]
 89%|########9 | 8940/10000 [01:07<00:07, 142.41it/s]
 90%|########9 | 8955/10000 [01:07<00:07, 142.59it/s]
 90%|########9 | 8970/10000 [01:07<00:07, 142.71it/s]
 90%|########9 | 8985/10000 [01:07<00:07, 142.74it/s]
 90%|######### | 9000/10000 [01:07<00:07, 142.84it/s]
 90%|######### | 9015/10000 [01:07<00:06, 142.84it/s]
 90%|######### | 9030/10000 [01:07<00:06, 142.89it/s]
 90%|######### | 9045/10000 [01:08<00:06, 142.81it/s]
 91%|######### | 9060/10000 [01:08<00:06, 142.82it/s]
 91%|######### | 9075/10000 [01:08<00:06, 142.85it/s]
 91%|######### | 9090/10000 [01:08<00:06, 142.89it/s]
 91%|#########1| 9105/10000 [01:08<00:06, 142.36it/s]
 91%|#########1| 9120/10000 [01:08<00:06, 142.48it/s]
 91%|#########1| 9135/10000 [01:08<00:06, 142.53it/s]
 92%|#########1| 9150/10000 [01:08<00:05, 142.64it/s]
 92%|#########1| 9165/10000 [01:08<00:05, 142.68it/s]
 92%|#########1| 9180/10000 [01:09<00:05, 142.47it/s]
 92%|#########1| 9195/10000 [01:09<00:05, 142.58it/s]
 92%|#########2| 9210/10000 [01:09<00:05, 142.66it/s]
 92%|#########2| 9225/10000 [01:09<00:05, 142.76it/s]
 92%|#########2| 9240/10000 [01:09<00:05, 142.78it/s]
 93%|#########2| 9255/10000 [01:09<00:05, 142.77it/s]
 93%|#########2| 9270/10000 [01:09<00:05, 142.71it/s]
 93%|#########2| 9285/10000 [01:09<00:05, 142.80it/s]
 93%|#########3| 9300/10000 [01:09<00:04, 142.85it/s]
 93%|#########3| 9315/10000 [01:09<00:04, 142.68it/s]
 93%|#########3| 9330/10000 [01:10<00:04, 142.66it/s]
 93%|#########3| 9345/10000 [01:10<00:04, 142.64it/s]
 94%|#########3| 9360/10000 [01:10<00:04, 142.74it/s]
 94%|#########3| 9375/10000 [01:10<00:04, 142.81it/s]
 94%|#########3| 9390/10000 [01:10<00:04, 142.80it/s]
 94%|#########4| 9405/10000 [01:10<00:04, 142.88it/s]
 94%|#########4| 9420/10000 [01:10<00:04, 142.77it/s]
 94%|#########4| 9435/10000 [01:10<00:03, 142.77it/s]
 94%|#########4| 9450/10000 [01:10<00:03, 142.76it/s]
 95%|#########4| 9465/10000 [01:11<00:03, 142.84it/s]
 95%|#########4| 9480/10000 [01:11<00:03, 142.86it/s]
 95%|#########4| 9495/10000 [01:11<00:03, 142.91it/s]
 95%|#########5| 9510/10000 [01:11<00:03, 142.92it/s]
 95%|#########5| 9525/10000 [01:11<00:03, 142.92it/s]
 95%|#########5| 9540/10000 [01:11<00:03, 142.94it/s]
 96%|#########5| 9555/10000 [01:11<00:03, 142.96it/s]
 96%|#########5| 9570/10000 [01:11<00:03, 142.63it/s]
 96%|#########5| 9585/10000 [01:11<00:02, 142.76it/s]
 96%|#########6| 9600/10000 [01:11<00:02, 142.84it/s]
 96%|#########6| 9615/10000 [01:12<00:02, 142.84it/s]
 96%|#########6| 9630/10000 [01:12<00:02, 142.86it/s]
 96%|#########6| 9645/10000 [01:12<00:02, 142.91it/s]
 97%|#########6| 9660/10000 [01:12<00:02, 142.91it/s]
 97%|#########6| 9675/10000 [01:12<00:02, 142.41it/s]
 97%|#########6| 9690/10000 [01:12<00:02, 142.24it/s]
 97%|#########7| 9705/10000 [01:12<00:02, 142.30it/s]
 97%|#########7| 9720/10000 [01:12<00:01, 142.52it/s]
 97%|#########7| 9735/10000 [01:12<00:01, 142.59it/s]
 98%|#########7| 9750/10000 [01:13<00:01, 142.67it/s]
 98%|#########7| 9765/10000 [01:13<00:01, 142.29it/s]
 98%|#########7| 9780/10000 [01:13<00:01, 142.36it/s]
 98%|#########7| 9795/10000 [01:13<00:01, 142.29it/s]
 98%|#########8| 9810/10000 [01:13<00:01, 142.32it/s]
 98%|#########8| 9825/10000 [01:13<00:01, 141.46it/s]
 98%|#########8| 9840/10000 [01:13<00:01, 141.78it/s]
 99%|#########8| 9855/10000 [01:13<00:01, 141.99it/s]
 99%|#########8| 9870/10000 [01:13<00:00, 142.12it/s]
 99%|#########8| 9885/10000 [01:13<00:00, 142.21it/s]
 99%|#########9| 9900/10000 [01:14<00:00, 142.39it/s]
 99%|#########9| 9915/10000 [01:14<00:00, 142.53it/s]
 99%|#########9| 9930/10000 [01:14<00:00, 142.62it/s]
 99%|#########9| 9945/10000 [01:14<00:00, 142.71it/s]
100%|#########9| 9960/10000 [01:14<00:00, 141.98it/s]
100%|#########9| 9975/10000 [01:14<00:00, 142.26it/s]
100%|#########9| 9990/10000 [01:14<00:00, 142.28it/s]
100%|##########| 10000/10000 [01:14<00:00, 133.73it/s]
The inversion result from `emcee`:
============================
Summary for inversion result
============================
SUCCESS
----------------------------
sampler: <emcee.ensemble.EnsembleSampler object>
blob_names: ['log_likelihood', 'log_prior']

Post-sampling analysis

By default the raw sampler resulting object is attached to cofi’s inversion result.

Optionally, you can convert that into an arviz data structure to have access to a range of analysis functions. (See more details in arviz documentation).

import arviz as az

labels = ["m0", "m1", "m2","m3","m4"]

sampler = inv_result_3.sampler
az_idata = az.from_emcee(sampler, var_names=labels)
# az_idata = inv_result_3.to_arviz()      # alternatively

az_idata.get("posterior")

<xarray.Dataset>
Dimensions:  (chain: 32, draw: 10000)
Coordinates:
  * chain    (chain) int64 0 1 2 3 4 5 6 7 8 9 ... 22 23 24 25 26 27 28 29 30 31
  * draw     (draw) int64 0 1 2 3 4 5 6 7 ... 9993 9994 9995 9996 9997 9998 9999
Data variables:
    m0       (chain, draw) float64 0.0001012 0.0001012 0.0001012 ... 1.171 1.171
    m1       (chain, draw) float64 -3.205e-06 -3.205e-06 ... -2.775 -2.775
    m2       (chain, draw) float64 1.438e-05 1.438e-05 1.438e-05 ... 1.326 1.326
    m3       (chain, draw) float64 -0.0002025 -0.0002025 ... -0.2008 -0.2008
    m4       (chain, draw) float64 5.183e-05 5.183e-05 ... 0.006248 0.006248
Attributes:
    created_at:                 2022-12-14T01:59:30.389389
    arviz_version:              0.12.1
    inference_library:          emcee
    inference_library_version:  3.1.2

xarray.Dataset

• Dimensions:
  • chain: 32
  • draw: 10000
• Coordinates: (2)
  • chain
    (chain)
    int64
    0 1 2 3 4 5 6 ... 26 27 28 29 30 31

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
           18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31])

  • draw
    (draw)
    int64
    0 1 2 3 4 ... 9996 9997 9998 9999

    array([   0,    1,    2, ..., 9997, 9998, 9999])

• Data variables: (5)
  • m0
    (chain, draw)
    float64
    0.0001012 0.0001012 ... 1.171 1.171

    array([[1.01238831e-04, 1.01238831e-04, 1.01238831e-04, ...,
            1.45088602e+00, 1.38002082e+00, 1.38002082e+00],
           [8.31876319e-05, 8.31876319e-05, 1.52174366e-04, ...,
            1.51517930e+00, 1.51517930e+00, 1.48589695e+00],
           [9.48056292e-05, 8.53147679e-05, 9.98155615e-05, ...,
            1.24348303e+00, 1.26851908e+00, 1.19061769e+00],
           ...,
           [3.29125542e-05, 5.74061345e-05, 5.74061345e-05, ...,
            1.47912178e+00, 1.53812328e+00, 1.53812328e+00],
           [1.31452790e-04, 2.08187720e-04, 2.15119855e-04, ...,
            1.59154318e+00, 1.61105930e+00, 1.61105930e+00],
           [4.45953770e-05, 2.99428267e-05, 2.99428267e-05, ...,
            1.30830689e+00, 1.17131386e+00, 1.17131386e+00]])

  • m1
    (chain, draw)
    float64
    -3.205e-06 -3.205e-06 ... -2.775

    array([[-3.20533943e-06, -3.20533943e-06, -3.20533943e-06, ...,
            -3.17805074e+00, -3.15107190e+00, -3.15107190e+00],
           [ 2.53496339e-04,  2.53496339e-04,  1.35083974e-04, ...,
            -3.28620491e+00, -3.28620491e+00, -3.26423750e+00],
           [-7.22480561e-05, -3.22623621e-05, -7.95841215e-05, ...,
            -2.79067964e+00, -2.82542557e+00, -2.72717634e+00],
           ...,
           [ 1.77273179e-05,  1.03162338e-04,  1.03162338e-04, ...,
            -2.93089800e+00, -3.05657874e+00, -3.05657874e+00],
           [ 1.42244928e-04,  2.39293938e-04,  2.54614357e-04, ...,
            -3.14925965e+00, -3.09728461e+00, -3.09728461e+00],
           [-1.51778258e-04, -9.56591164e-05, -9.56591164e-05, ...,
            -2.90912641e+00, -2.77471539e+00, -2.77471539e+00]])

  • m2
    (chain, draw)
    float64
    1.438e-05 1.438e-05 ... 1.326 1.326

    array([[ 1.43830789e-05,  1.43830789e-05,  1.43830789e-05, ...,
             1.42300162e+00,  1.40963578e+00,  1.40963578e+00],
           [-4.11633117e-05, -4.11633117e-05, -1.71947711e-04, ...,
             1.49630121e+00,  1.49630121e+00,  1.48383130e+00],
           [-2.25178200e-04, -1.76861600e-04, -1.66081625e-04, ...,
             1.31568916e+00,  1.32834592e+00,  1.30910855e+00],
           ...,
           [ 6.13805194e-05,  6.17797361e-05,  6.17797361e-05, ...,
             1.32926739e+00,  1.37201285e+00,  1.37201285e+00],
           [ 4.04146196e-05,  1.01856932e-04,  1.17581424e-04, ...,
             1.39603300e+00,  1.35820797e+00,  1.35820797e+00],
           [ 8.02383215e-05,  2.23295493e-05,  2.23295493e-05, ...,
             1.36428273e+00,  1.32624759e+00,  1.32624759e+00]])

  • m3
    (chain, draw)
    float64
    -0.0002025 -0.0002025 ... -0.2008

    array([[-2.02518028e-04, -2.02518028e-04, -2.02518028e-04, ...,
            -2.08649725e-01, -2.05996910e-01, -2.05996910e-01],
           [-1.46604346e-04, -1.46604346e-04, -2.00270707e-04, ...,
            -2.20049099e-01, -2.20049099e-01, -2.18384612e-01],
           [ 3.26288663e-04,  2.49050359e-04,  2.49458615e-04, ...,
            -1.98987631e-01, -2.00647867e-01, -1.99338184e-01],
           ...,
           [-1.09500588e-04, -2.28779123e-04, -2.28779123e-04, ...,
            -1.97442853e-01, -2.01868012e-01, -2.01868012e-01],
           [ 9.57661385e-07,  1.33065318e-04,  1.26521779e-04, ...,
            -2.06140801e-01, -2.00676069e-01, -2.00676069e-01],
           [-4.03407495e-05, -1.59649331e-05, -1.59649331e-05, ...,
            -2.05173343e-01, -2.00758246e-01, -2.00758246e-01]])

  • m4
    (chain, draw)
    float64
    5.183e-05 5.183e-05 ... 0.006248

    array([[ 5.18311658e-05,  5.18311658e-05,  5.18311658e-05, ...,
             6.48046418e-03,  6.36407302e-03,  6.36407302e-03],
           [-1.35791009e-04, -1.35791009e-04, -1.48953676e-04, ...,
             6.90711666e-03,  6.90711666e-03,  6.83591036e-03],
           [ 9.12554764e-05,  7.22924336e-05,  3.74121394e-05, ...,
             6.24147235e-03,  6.30109307e-03,  6.28232774e-03],
           ...,
           [-1.30427524e-04, -3.40704556e-04, -3.40704556e-04, ...,
             6.09558694e-03,  6.23151777e-03,  6.23151777e-03],
           [-2.31186010e-05, -8.57069340e-05, -9.46208003e-05, ...,
             6.43743334e-03,  6.25336031e-03,  6.25336031e-03],
           [ 9.08284564e-05,  6.10826410e-05,  6.10826410e-05, ...,
             6.41030010e-03,  6.24815225e-03,  6.24815225e-03]])

• Attributes: (4)

  created_at :

  2022-12-14T01:59:30.389389

  arviz_version :

  0.12.1

  inference_library :

  emcee

  inference_library_version :

  3.1.2

# a standard `trace` plot
axes = az.plot_trace(az_idata, backend_kwargs={"constrained_layout":True});

# add legends
for i, axes_pair in enumerate(axes):
    ax1 = axes_pair[0]
    ax2 = axes_pair[1]
    #ax1.axvline(true_model[i], linestyle='dotted', color='red')
    ax1.set_xlabel("parameter value")
    ax1.set_ylabel("density value")
    ax2.set_xlabel("number of iterations")
    ax2.set_ylabel("parameter value")

#tau = sampler.get_autocorr_time()
#print(f"autocorrelation time: {tau}")

# a Corner plot

fig, axes = plt.subplots(nparams, nparams, figsize=(12,8))

if(True): # if we are plotting the model ensemble use this
    az.plot_pair(
        az_idata.sel(draw=slice(300,None)),
        marginals=True,
        #reference_values=dict(zip([f"m{i}" for i in range(4)], true_model.tolist())),
        ax=axes,
    );
else: # if we wish to plot a kernel density plot then use this option
    az.plot_pair(
        az_idata.sel(draw=slice(300,None)),
        marginals=True,
        #reference_values=dict(zip([f"m{i}" for i in range(4)], true_model.tolist())),
        kind="kde",
        kde_kwargs={
            "hdi_probs": [0.3, 0.6, 0.9],  # Plot 30%, 60% and 90% HDI contours
            "contourf_kwargs": {"cmap": "Blues"},
        },
        ax=axes,
    );

Now we plot the predicted curves for the posterior ensemble of solutions.

flat_samples = sampler.get_chain(discard=300, thin=30, flat=True)
inds = np.random.randint(len(flat_samples), size=100) # get a random selection from posterior ensemble

plot_data(title="Eustatic sea-level")
plt.xlim(0,maxtime)
plot_models(flat_samples[inds],color="lightgrey")
plot_model(ref_x,ref_y, "Reference model", color="darkorange")
#plt.xlim(15,20.)
#plt.ylim(-140,-100)

Expected values, credible intervals and model covariance matrix from the ensemble

print("\n Expected value and 95% credible intervals ")
for i in range(ndim):
    mcmc = np.percentile(flat_samples[:, i], [5, 50, 95])
    print(" {} {:7.3f} [{:7.3f}, {:7.3f}]".format(labels[i],mcmc[1],mcmc[0],mcmc[2]))

Expected value and 95% credible intervals
m0   1.446 [  1.015,   1.866]
m1  -3.117 [ -3.559,  -2.666]
m2   1.413 [  1.272,   1.554]
m3  -0.209 [ -0.225,  -0.193]
m4   0.007 [  0.006,   0.007]

CMpost = np.cov(flat_samples.T)
CM_std= np.std(flat_samples,axis=0)
print('Posterior model covariance matrix\n',CMpost)
print('\n Posterior estimate of model standard deviations in each parameter')
for i in range(ndim):
    print("    {} {:7.4f}".format(labels[i],CM_std[i]))

Posterior model covariance matrix
 [[ 6.62741795e-02 -6.23674611e-02  1.66426392e-02 -1.60006642e-03
   4.73533740e-05]
 [-6.23674611e-02  7.41224055e-02 -2.23459116e-02  2.32404338e-03
  -7.20781669e-05]
 [ 1.66426392e-02 -2.23459116e-02  7.40198914e-03 -8.23986122e-04
   2.66247032e-05]
 [-1.60006642e-03  2.32404338e-03 -8.23986122e-04  9.68093102e-05
  -3.23598370e-06]
 [ 4.73533740e-05 -7.20781669e-05  2.66247032e-05 -3.23598370e-06
   1.11096148e-07]]

 Posterior estimate of model standard deviations in each parameter
    m0  0.2574
    m1  0.2722
    m2  0.0860
    m3  0.0098
    m4  0.0003

---

Challenge - Change the prior model bounds

Replace the previous prior bounds to new values

The original uniform bounds had

\({\mathbf l}^T = (-10.,-10.,-10.,-10.)\), and \({\mathbf u}^T = (10.,10.,10.,10.)\).

Lets replace with

\({\mathbf l}^T = (-0.5,-10.,-10.,-10.)\), and \({\mathbf u}^T = (0.5,10.,10.,10.)\).

We have only changed the bounds of the first parameter. However since the true value of constant term was 6, these bounds are now inconsistent with the true model.

What does this do to the posterior distribution?

Start from the code template below:

m_lower_bound = <CHANGE ME>             # lower bound for uniform prior
m_upper_bound = <CHANGE ME>             # upper bound for uniform prior

def log_prior(model):    # uniform distribution
    for i in range(len(m_lower_bound)):
        if model[i] < m_lower_bound[i] or model[i] > m_upper_bound[i]: return -np.inf
    return 0.0 # model lies within bounds -> return log(1)

######## CoFI BaseProblem - update information
inv_problem.set_log_prior(log_prior)

######## CoFI Inversion - run it
inv_4 = Inversion(inv_problem, inv_options_3)
inv_result_4 = inv_4.run()

flat_samples = inv_result_4.sampler.get_chain(discard=300, thin=30, flat=True)
inds = np.random.randint(len(flat_samples), size=100) # get a random selection from posterior ensemble

print("Resulting samples with prior model lower bounds of <CHANGE ME>, upper bounds of <CHANGE ME>")
plot_data()
plot_models(flat_samples[inds])
plot_model(x, true_y, "True model", color="darkorange")

# Copy the template above, Replace <CHANGE ME> with your answer

#@title Solution

m_lower_bound = np.array([-1.0,-10,-10,-10])             # lower bound for uniform prior
m_upper_bound = np.array([1.0,10,10,10])                 # upper bound for uniform prior

def log_prior(model):    # uniform distribution
    for i in range(len(m_lower_bound)):
        if model[i] < m_lower_bound[i] or model[i] > m_upper_bound[i]: return -np.inf
    return 0.0 # model lies within bounds -> return log(1)

######## CoFI BaseProblem - update information
inv_problem.set_log_prior(log_prior)

######## CoFI Inversion - run it
inv_4 = Inversion(inv_problem, inv_options_3)
inv_result_4 = inv_4.run()

flat_samples = inv_result_4.sampler.get_chain(discard=300, thin=30, flat=True)
inds = np.random.randint(len(flat_samples), size=100) # get a random selection from posterior ensemble

print("Resulting samples with prior model lower bounds of [-1,-10,-10,-10], upper bounds of [2,10,10,10]")
plot_data()
plot_models(flat_samples[inds],color="lightgrey")
plot_model(ref_x, ref_y, "Reference model", color="darkorange")

  0%|          | 0/10000 [00:00<?, ?it/s]
  0%|          | 15/10000 [00:00<01:10, 142.07it/s]
  0%|          | 30/10000 [00:00<01:09, 142.77it/s]
  0%|          | 45/10000 [00:00<01:14, 134.18it/s]
  1%|          | 60/10000 [00:00<01:12, 137.65it/s]
  1%|          | 75/10000 [00:00<01:11, 139.72it/s]
  1%|          | 90/10000 [00:00<01:10, 140.84it/s]
  1%|1         | 105/10000 [00:00<01:09, 141.68it/s]
  1%|1         | 120/10000 [00:00<01:10, 140.25it/s]
  1%|1         | 135/10000 [00:00<01:09, 141.25it/s]
  2%|1         | 150/10000 [00:01<01:09, 141.90it/s]
  2%|1         | 165/10000 [00:01<01:09, 142.13it/s]
  2%|1         | 180/10000 [00:01<01:08, 142.51it/s]
  2%|1         | 195/10000 [00:01<01:08, 142.80it/s]
  2%|2         | 210/10000 [00:01<01:08, 142.72it/s]
  2%|2         | 225/10000 [00:01<01:08, 142.97it/s]
  2%|2         | 240/10000 [00:01<01:08, 143.14it/s]
  3%|2         | 255/10000 [00:01<01:08, 143.23it/s]
  3%|2         | 270/10000 [00:01<01:07, 143.27it/s]
  3%|2         | 285/10000 [00:02<01:07, 143.33it/s]
  3%|3         | 300/10000 [00:02<01:07, 143.39it/s]
  3%|3         | 315/10000 [00:02<01:07, 143.43it/s]
  3%|3         | 330/10000 [00:02<01:07, 143.43it/s]
  3%|3         | 345/10000 [00:02<01:07, 143.50it/s]
  4%|3         | 360/10000 [00:02<01:07, 143.53it/s]
  4%|3         | 375/10000 [00:02<01:07, 143.52it/s]
  4%|3         | 390/10000 [00:02<01:07, 142.79it/s]
  4%|4         | 405/10000 [00:02<01:07, 143.04it/s]
  4%|4         | 420/10000 [00:02<01:06, 143.12it/s]
  4%|4         | 435/10000 [00:03<01:06, 143.26it/s]
  4%|4         | 450/10000 [00:03<01:06, 143.34it/s]
  5%|4         | 465/10000 [00:03<01:06, 143.40it/s]
  5%|4         | 480/10000 [00:03<01:06, 143.43it/s]
  5%|4         | 495/10000 [00:03<01:18, 121.71it/s]
  5%|5         | 510/10000 [00:03<01:18, 120.93it/s]
  5%|5         | 524/10000 [00:03<01:15, 125.73it/s]
  5%|5         | 539/10000 [00:03<01:12, 130.02it/s]
  6%|5         | 554/10000 [00:03<01:10, 133.15it/s]
  6%|5         | 569/10000 [00:04<01:09, 135.47it/s]
  6%|5         | 583/10000 [00:04<01:09, 136.33it/s]
  6%|5         | 598/10000 [00:04<01:08, 137.81it/s]
  6%|6         | 613/10000 [00:04<01:07, 138.84it/s]
  6%|6         | 628/10000 [00:04<01:06, 140.10it/s]
  6%|6         | 643/10000 [00:04<01:06, 141.15it/s]
  7%|6         | 658/10000 [00:04<01:05, 141.91it/s]
  7%|6         | 673/10000 [00:04<01:05, 142.45it/s]
  7%|6         | 688/10000 [00:04<01:05, 142.81it/s]
  7%|7         | 703/10000 [00:05<01:05, 142.98it/s]
  7%|7         | 718/10000 [00:05<01:04, 143.18it/s]
  7%|7         | 733/10000 [00:05<01:04, 143.35it/s]
  7%|7         | 748/10000 [00:05<01:04, 143.43it/s]
  8%|7         | 763/10000 [00:05<01:06, 137.91it/s]
  8%|7         | 778/10000 [00:05<01:06, 139.58it/s]
  8%|7         | 793/10000 [00:05<01:05, 140.76it/s]
  8%|8         | 808/10000 [00:05<01:04, 141.57it/s]
  8%|8         | 823/10000 [00:05<01:04, 142.14it/s]
  8%|8         | 838/10000 [00:05<01:04, 142.54it/s]
  9%|8         | 853/10000 [00:06<01:04, 140.93it/s]
  9%|8         | 868/10000 [00:06<01:04, 141.75it/s]
  9%|8         | 883/10000 [00:06<01:04, 142.01it/s]
  9%|8         | 898/10000 [00:06<01:03, 142.50it/s]
  9%|9         | 913/10000 [00:06<01:03, 142.86it/s]
  9%|9         | 928/10000 [00:06<01:03, 142.71it/s]
  9%|9         | 943/10000 [00:06<01:03, 143.02it/s]
 10%|9         | 958/10000 [00:06<01:03, 143.20it/s]
 10%|9         | 973/10000 [00:06<01:03, 143.28it/s]
 10%|9         | 988/10000 [00:07<01:02, 143.39it/s]
 10%|#         | 1003/10000 [00:07<01:02, 143.47it/s]
 10%|#         | 1018/10000 [00:07<01:02, 143.53it/s]
 10%|#         | 1033/10000 [00:07<01:02, 143.55it/s]
 10%|#         | 1048/10000 [00:07<01:02, 143.58it/s]
 11%|#         | 1063/10000 [00:07<01:02, 143.61it/s]
 11%|#         | 1078/10000 [00:07<01:02, 143.06it/s]
 11%|#         | 1093/10000 [00:07<01:02, 142.41it/s]
 11%|#1        | 1108/10000 [00:07<01:02, 142.74it/s]
 11%|#1        | 1123/10000 [00:07<01:02, 143.03it/s]
 11%|#1        | 1138/10000 [00:08<01:01, 143.21it/s]
 12%|#1        | 1153/10000 [00:08<01:02, 140.50it/s]
 12%|#1        | 1168/10000 [00:08<01:02, 141.38it/s]
 12%|#1        | 1183/10000 [00:08<01:02, 142.08it/s]
 12%|#1        | 1198/10000 [00:08<01:01, 142.54it/s]
 12%|#2        | 1213/10000 [00:08<01:01, 142.86it/s]
 12%|#2        | 1228/10000 [00:08<01:02, 139.36it/s]
 12%|#2        | 1243/10000 [00:08<01:02, 140.48it/s]
 13%|#2        | 1258/10000 [00:08<01:01, 141.35it/s]
 13%|#2        | 1273/10000 [00:09<01:01, 142.03it/s]
 13%|#2        | 1288/10000 [00:09<01:01, 142.50it/s]
 13%|#3        | 1303/10000 [00:09<01:00, 142.61it/s]
 13%|#3        | 1318/10000 [00:09<01:00, 142.79it/s]
 13%|#3        | 1333/10000 [00:09<01:00, 143.02it/s]
 13%|#3        | 1348/10000 [00:09<01:00, 143.26it/s]
 14%|#3        | 1363/10000 [00:09<01:00, 143.40it/s]
 14%|#3        | 1378/10000 [00:09<01:00, 143.50it/s]
 14%|#3        | 1393/10000 [00:09<00:59, 143.57it/s]
 14%|#4        | 1408/10000 [00:09<00:59, 143.57it/s]
 14%|#4        | 1423/10000 [00:10<00:59, 143.66it/s]
 14%|#4        | 1438/10000 [00:10<00:59, 143.71it/s]
 15%|#4        | 1453/10000 [00:10<00:59, 143.73it/s]
 15%|#4        | 1468/10000 [00:10<00:59, 143.72it/s]
 15%|#4        | 1483/10000 [00:10<00:59, 143.76it/s]
 15%|#4        | 1498/10000 [00:10<00:59, 143.79it/s]
 15%|#5        | 1513/10000 [00:10<00:59, 143.77it/s]
 15%|#5        | 1528/10000 [00:10<00:58, 143.77it/s]
 15%|#5        | 1543/10000 [00:10<00:58, 143.76it/s]
 16%|#5        | 1558/10000 [00:11<00:58, 143.74it/s]
 16%|#5        | 1573/10000 [00:11<00:58, 143.73it/s]
 16%|#5        | 1588/10000 [00:11<00:58, 143.75it/s]
 16%|#6        | 1603/10000 [00:11<00:58, 143.72it/s]
 16%|#6        | 1618/10000 [00:11<00:58, 143.71it/s]
 16%|#6        | 1633/10000 [00:11<00:58, 143.72it/s]
 16%|#6        | 1648/10000 [00:11<00:58, 143.71it/s]
 17%|#6        | 1663/10000 [00:11<00:58, 143.72it/s]
 17%|#6        | 1678/10000 [00:11<00:57, 143.71it/s]
 17%|#6        | 1693/10000 [00:11<00:57, 143.72it/s]
 17%|#7        | 1708/10000 [00:12<00:57, 143.74it/s]
 17%|#7        | 1723/10000 [00:12<00:57, 143.70it/s]
 17%|#7        | 1738/10000 [00:12<00:57, 143.66it/s]
 18%|#7        | 1753/10000 [00:12<00:57, 143.67it/s]
 18%|#7        | 1768/10000 [00:12<00:57, 143.68it/s]
 18%|#7        | 1783/10000 [00:12<00:57, 143.70it/s]
 18%|#7        | 1798/10000 [00:12<00:57, 143.70it/s]
 18%|#8        | 1813/10000 [00:12<00:57, 143.01it/s]
 18%|#8        | 1828/10000 [00:12<00:57, 143.25it/s]
 18%|#8        | 1843/10000 [00:13<00:56, 143.41it/s]
 19%|#8        | 1858/10000 [00:13<00:56, 143.49it/s]
 19%|#8        | 1873/10000 [00:13<00:56, 143.57it/s]
 19%|#8        | 1888/10000 [00:13<00:56, 143.57it/s]
 19%|#9        | 1903/10000 [00:13<00:56, 143.60it/s]
 19%|#9        | 1918/10000 [00:13<00:56, 143.61it/s]
 19%|#9        | 1933/10000 [00:13<00:56, 143.66it/s]
 19%|#9        | 1948/10000 [00:13<00:56, 143.69it/s]
 20%|#9        | 1963/10000 [00:13<00:55, 143.69it/s]
 20%|#9        | 1978/10000 [00:13<00:55, 143.64it/s]
 20%|#9        | 1993/10000 [00:14<00:55, 143.66it/s]
 20%|##        | 2008/10000 [00:14<00:55, 143.03it/s]
 20%|##        | 2023/10000 [00:14<00:55, 143.26it/s]
 20%|##        | 2038/10000 [00:14<00:55, 143.41it/s]
 21%|##        | 2053/10000 [00:14<00:55, 143.54it/s]
 21%|##        | 2068/10000 [00:14<00:55, 143.62it/s]
 21%|##        | 2083/10000 [00:14<00:55, 143.70it/s]
 21%|##        | 2098/10000 [00:14<00:54, 143.72it/s]
 21%|##1       | 2113/10000 [00:14<00:54, 143.70it/s]
 21%|##1       | 2128/10000 [00:14<00:54, 143.71it/s]
 21%|##1       | 2143/10000 [00:15<00:58, 133.90it/s]
 22%|##1       | 2158/10000 [00:15<00:57, 136.66it/s]
 22%|##1       | 2173/10000 [00:15<00:56, 138.71it/s]
 22%|##1       | 2188/10000 [00:15<00:55, 140.18it/s]
 22%|##2       | 2203/10000 [00:15<00:55, 141.23it/s]
 22%|##2       | 2218/10000 [00:15<00:54, 141.96it/s]
 22%|##2       | 2233/10000 [00:15<00:54, 142.47it/s]
 22%|##2       | 2248/10000 [00:15<00:54, 142.84it/s]
 23%|##2       | 2263/10000 [00:15<00:54, 143.07it/s]
 23%|##2       | 2278/10000 [00:16<00:53, 143.27it/s]
 23%|##2       | 2293/10000 [00:16<00:53, 143.40it/s]
 23%|##3       | 2308/10000 [00:16<00:53, 143.45it/s]
 23%|##3       | 2323/10000 [00:16<00:53, 143.55it/s]
 23%|##3       | 2338/10000 [00:16<00:53, 143.59it/s]
 24%|##3       | 2353/10000 [00:16<00:53, 143.66it/s]
 24%|##3       | 2368/10000 [00:16<00:53, 143.72it/s]
 24%|##3       | 2383/10000 [00:16<00:52, 143.74it/s]
 24%|##3       | 2398/10000 [00:16<00:52, 143.75it/s]
 24%|##4       | 2413/10000 [00:16<00:52, 143.74it/s]
 24%|##4       | 2428/10000 [00:17<00:52, 143.13it/s]
 24%|##4       | 2443/10000 [00:17<00:52, 143.34it/s]
 25%|##4       | 2458/10000 [00:17<00:58, 129.50it/s]
 25%|##4       | 2473/10000 [00:17<00:56, 133.46it/s]
 25%|##4       | 2488/10000 [00:17<00:55, 136.39it/s]
 25%|##5       | 2503/10000 [00:17<00:54, 138.11it/s]
 25%|##5       | 2518/10000 [00:17<00:53, 139.10it/s]
 25%|##5       | 2533/10000 [00:17<00:53, 140.44it/s]
 25%|##5       | 2548/10000 [00:17<00:52, 141.36it/s]
 26%|##5       | 2563/10000 [00:18<00:52, 141.75it/s]
 26%|##5       | 2578/10000 [00:18<00:52, 142.30it/s]
 26%|##5       | 2593/10000 [00:18<00:51, 142.69it/s]
 26%|##6       | 2608/10000 [00:18<00:51, 142.95it/s]
 26%|##6       | 2623/10000 [00:18<00:51, 143.19it/s]
 26%|##6       | 2638/10000 [00:18<00:51, 143.33it/s]
 27%|##6       | 2653/10000 [00:18<00:54, 135.34it/s]
 27%|##6       | 2668/10000 [00:18<00:53, 136.77it/s]
 27%|##6       | 2682/10000 [00:18<00:55, 131.24it/s]
 27%|##6       | 2696/10000 [00:19<00:56, 130.15it/s]
 27%|##7       | 2711/10000 [00:19<00:54, 134.00it/s]
 27%|##7       | 2726/10000 [00:19<00:53, 136.55it/s]
 27%|##7       | 2741/10000 [00:19<00:52, 138.50it/s]
 28%|##7       | 2756/10000 [00:19<00:51, 140.03it/s]
 28%|##7       | 2771/10000 [00:19<00:51, 141.11it/s]
 28%|##7       | 2786/10000 [00:19<00:50, 141.87it/s]
 28%|##8       | 2801/10000 [00:19<00:50, 142.22it/s]
 28%|##8       | 2816/10000 [00:19<00:50, 142.64it/s]
 28%|##8       | 2831/10000 [00:20<00:50, 142.93it/s]
 28%|##8       | 2846/10000 [00:20<00:49, 143.17it/s]
 29%|##8       | 2861/10000 [00:20<00:49, 143.34it/s]
 29%|##8       | 2876/10000 [00:20<00:49, 143.42it/s]
 29%|##8       | 2891/10000 [00:20<00:49, 143.50it/s]
 29%|##9       | 2906/10000 [00:20<00:49, 143.58it/s]
 29%|##9       | 2921/10000 [00:20<00:55, 127.28it/s]
 29%|##9       | 2936/10000 [00:20<00:53, 131.71it/s]
 30%|##9       | 2951/10000 [00:20<00:52, 135.06it/s]
 30%|##9       | 2965/10000 [00:21<00:55, 127.48it/s]
 30%|##9       | 2980/10000 [00:21<00:53, 132.00it/s]
 30%|##9       | 2994/10000 [00:21<00:53, 131.99it/s]
 30%|###       | 3009/10000 [00:21<00:51, 135.40it/s]
 30%|###       | 3024/10000 [00:21<00:50, 137.13it/s]
 30%|###       | 3039/10000 [00:21<00:50, 139.08it/s]
 31%|###       | 3054/10000 [00:21<00:49, 140.40it/s]
 31%|###       | 3069/10000 [00:21<00:49, 141.38it/s]
 31%|###       | 3084/10000 [00:21<00:48, 142.08it/s]
 31%|###       | 3099/10000 [00:21<00:48, 142.56it/s]
 31%|###1      | 3114/10000 [00:22<00:48, 142.90it/s]
 31%|###1      | 3129/10000 [00:22<00:47, 143.19it/s]
 31%|###1      | 3144/10000 [00:22<00:47, 143.36it/s]
 32%|###1      | 3159/10000 [00:22<00:47, 143.38it/s]
 32%|###1      | 3174/10000 [00:22<00:47, 143.45it/s]
 32%|###1      | 3189/10000 [00:22<00:47, 143.53it/s]
 32%|###2      | 3204/10000 [00:22<00:47, 143.59it/s]
 32%|###2      | 3219/10000 [00:22<00:47, 142.91it/s]
 32%|###2      | 3234/10000 [00:22<00:47, 143.14it/s]
 32%|###2      | 3249/10000 [00:22<00:47, 143.27it/s]
 33%|###2      | 3264/10000 [00:23<00:46, 143.39it/s]
 33%|###2      | 3279/10000 [00:23<00:46, 143.49it/s]
 33%|###2      | 3294/10000 [00:23<00:46, 143.53it/s]
 33%|###3      | 3309/10000 [00:23<00:46, 143.46it/s]
 33%|###3      | 3324/10000 [00:23<00:46, 143.30it/s]
 33%|###3      | 3339/10000 [00:23<00:46, 143.22it/s]
 34%|###3      | 3354/10000 [00:23<00:46, 143.39it/s]
 34%|###3      | 3369/10000 [00:23<00:46, 143.49it/s]
 34%|###3      | 3384/10000 [00:23<00:46, 143.55it/s]
 34%|###3      | 3399/10000 [00:24<00:45, 143.64it/s]
 34%|###4      | 3414/10000 [00:24<00:46, 142.88it/s]
 34%|###4      | 3429/10000 [00:24<00:46, 141.04it/s]
 34%|###4      | 3444/10000 [00:24<00:46, 141.81it/s]
 35%|###4      | 3459/10000 [00:24<00:45, 142.40it/s]
 35%|###4      | 3474/10000 [00:24<00:45, 142.78it/s]
 35%|###4      | 3489/10000 [00:24<00:45, 142.99it/s]
 35%|###5      | 3504/10000 [00:24<00:45, 143.19it/s]
 35%|###5      | 3519/10000 [00:24<00:45, 143.31it/s]
 35%|###5      | 3534/10000 [00:24<00:45, 143.42it/s]
 35%|###5      | 3549/10000 [00:25<00:44, 143.52it/s]
 36%|###5      | 3564/10000 [00:25<00:44, 143.60it/s]
 36%|###5      | 3579/10000 [00:25<00:44, 143.62it/s]
 36%|###5      | 3594/10000 [00:25<00:44, 143.63it/s]
 36%|###6      | 3609/10000 [00:25<00:44, 143.64it/s]
 36%|###6      | 3624/10000 [00:25<00:44, 143.65it/s]
 36%|###6      | 3639/10000 [00:25<00:44, 143.66it/s]
 37%|###6      | 3654/10000 [00:25<00:44, 143.69it/s]
 37%|###6      | 3669/10000 [00:25<00:44, 143.35it/s]
 37%|###6      | 3684/10000 [00:26<00:44, 143.34it/s]
 37%|###6      | 3699/10000 [00:26<00:43, 143.43it/s]
 37%|###7      | 3714/10000 [00:26<00:43, 143.51it/s]
 37%|###7      | 3729/10000 [00:26<00:43, 143.53it/s]
 37%|###7      | 3744/10000 [00:26<00:43, 143.54it/s]
 38%|###7      | 3759/10000 [00:26<00:43, 143.54it/s]
 38%|###7      | 3774/10000 [00:26<00:43, 143.58it/s]
 38%|###7      | 3789/10000 [00:26<00:43, 143.57it/s]
 38%|###8      | 3804/10000 [00:26<00:43, 143.34it/s]
 38%|###8      | 3819/10000 [00:26<00:43, 143.40it/s]
 38%|###8      | 3834/10000 [00:27<00:42, 143.49it/s]
 38%|###8      | 3849/10000 [00:27<00:42, 143.51it/s]
 39%|###8      | 3864/10000 [00:27<00:42, 143.56it/s]
 39%|###8      | 3879/10000 [00:27<00:42, 143.54it/s]
 39%|###8      | 3894/10000 [00:27<00:42, 143.58it/s]
 39%|###9      | 3909/10000 [00:27<00:42, 143.60it/s]
 39%|###9      | 3924/10000 [00:27<00:42, 143.61it/s]
 39%|###9      | 3939/10000 [00:27<00:42, 142.94it/s]
 40%|###9      | 3954/10000 [00:27<00:42, 143.16it/s]
 40%|###9      | 3969/10000 [00:28<00:42, 143.27it/s]
 40%|###9      | 3984/10000 [00:28<00:41, 143.36it/s]
 40%|###9      | 3999/10000 [00:28<00:41, 143.47it/s]
 40%|####      | 4014/10000 [00:28<00:41, 143.52it/s]
 40%|####      | 4029/10000 [00:28<00:41, 143.57it/s]
 40%|####      | 4044/10000 [00:28<00:41, 143.21it/s]
 41%|####      | 4059/10000 [00:28<00:41, 143.38it/s]
 41%|####      | 4074/10000 [00:28<00:41, 143.34it/s]
 41%|####      | 4089/10000 [00:28<00:41, 143.46it/s]
 41%|####1     | 4104/10000 [00:28<00:41, 143.50it/s]
 41%|####1     | 4119/10000 [00:29<00:40, 143.58it/s]
 41%|####1     | 4134/10000 [00:29<00:40, 143.64it/s]
 41%|####1     | 4149/10000 [00:29<00:40, 143.34it/s]
 42%|####1     | 4164/10000 [00:29<00:40, 143.40it/s]
 42%|####1     | 4179/10000 [00:29<00:40, 143.47it/s]
 42%|####1     | 4194/10000 [00:29<00:40, 143.29it/s]
 42%|####2     | 4209/10000 [00:29<00:40, 143.36it/s]
 42%|####2     | 4224/10000 [00:29<00:40, 143.48it/s]
 42%|####2     | 4239/10000 [00:29<00:40, 143.56it/s]
 43%|####2     | 4254/10000 [00:30<00:40, 143.58it/s]
 43%|####2     | 4269/10000 [00:30<00:39, 143.52it/s]
 43%|####2     | 4284/10000 [00:30<00:39, 143.61it/s]
 43%|####2     | 4299/10000 [00:30<00:39, 143.66it/s]
 43%|####3     | 4314/10000 [00:30<00:39, 143.70it/s]
 43%|####3     | 4329/10000 [00:30<00:39, 143.69it/s]
 43%|####3     | 4344/10000 [00:30<00:39, 143.73it/s]
 44%|####3     | 4359/10000 [00:30<00:39, 143.71it/s]
 44%|####3     | 4374/10000 [00:30<00:39, 143.71it/s]
 44%|####3     | 4389/10000 [00:30<00:39, 143.70it/s]
 44%|####4     | 4404/10000 [00:31<00:38, 143.51it/s]
 44%|####4     | 4419/10000 [00:31<00:38, 143.57it/s]
 44%|####4     | 4434/10000 [00:31<00:38, 143.60it/s]
 44%|####4     | 4449/10000 [00:31<00:38, 143.62it/s]
 45%|####4     | 4464/10000 [00:31<00:38, 143.65it/s]
 45%|####4     | 4479/10000 [00:31<00:38, 143.67it/s]
 45%|####4     | 4494/10000 [00:31<00:38, 143.58it/s]
 45%|####5     | 4509/10000 [00:31<00:38, 143.65it/s]
 45%|####5     | 4524/10000 [00:31<00:38, 143.67it/s]
 45%|####5     | 4539/10000 [00:31<00:38, 143.70it/s]
 46%|####5     | 4554/10000 [00:32<00:37, 143.75it/s]
 46%|####5     | 4569/10000 [00:32<00:37, 143.76it/s]
 46%|####5     | 4584/10000 [00:32<00:37, 143.75it/s]
 46%|####5     | 4599/10000 [00:32<00:37, 143.66it/s]
 46%|####6     | 4614/10000 [00:32<00:37, 143.69it/s]
 46%|####6     | 4629/10000 [00:32<00:37, 143.64it/s]
 46%|####6     | 4644/10000 [00:32<00:37, 143.66it/s]
 47%|####6     | 4659/10000 [00:32<00:37, 143.05it/s]
 47%|####6     | 4674/10000 [00:32<00:37, 143.15it/s]
 47%|####6     | 4689/10000 [00:33<00:37, 143.23it/s]
 47%|####7     | 4704/10000 [00:33<00:36, 143.32it/s]
 47%|####7     | 4719/10000 [00:33<00:36, 143.44it/s]
 47%|####7     | 4734/10000 [00:33<00:36, 143.46it/s]
 47%|####7     | 4749/10000 [00:33<00:36, 143.53it/s]
 48%|####7     | 4764/10000 [00:33<00:36, 143.55it/s]
 48%|####7     | 4779/10000 [00:33<00:36, 143.57it/s]
 48%|####7     | 4794/10000 [00:33<00:36, 143.57it/s]
 48%|####8     | 4809/10000 [00:33<00:36, 143.62it/s]
 48%|####8     | 4824/10000 [00:33<00:36, 143.57it/s]
 48%|####8     | 4839/10000 [00:34<00:35, 143.59it/s]
 49%|####8     | 4854/10000 [00:34<00:36, 142.81it/s]
 49%|####8     | 4869/10000 [00:34<00:35, 143.09it/s]
 49%|####8     | 4884/10000 [00:34<00:35, 143.28it/s]
 49%|####8     | 4899/10000 [00:34<00:35, 143.35it/s]
 49%|####9     | 4914/10000 [00:34<00:35, 143.46it/s]
 49%|####9     | 4929/10000 [00:34<00:36, 138.69it/s]
 49%|####9     | 4944/10000 [00:34<00:36, 140.13it/s]
 50%|####9     | 4959/10000 [00:34<00:35, 141.18it/s]
 50%|####9     | 4974/10000 [00:35<00:35, 141.89it/s]
 50%|####9     | 4989/10000 [00:35<00:35, 142.40it/s]
 50%|#####     | 5004/10000 [00:35<00:34, 142.75it/s]
 50%|#####     | 5019/10000 [00:35<00:34, 142.99it/s]
 50%|#####     | 5034/10000 [00:35<00:34, 143.17it/s]
 50%|#####     | 5049/10000 [00:35<00:34, 143.29it/s]
 51%|#####     | 5064/10000 [00:35<00:34, 143.39it/s]
 51%|#####     | 5079/10000 [00:35<00:34, 143.47it/s]
 51%|#####     | 5094/10000 [00:35<00:34, 143.48it/s]
 51%|#####1    | 5109/10000 [00:35<00:34, 143.50it/s]
 51%|#####1    | 5124/10000 [00:36<00:33, 143.53it/s]
 51%|#####1    | 5139/10000 [00:36<00:33, 143.57it/s]
 52%|#####1    | 5154/10000 [00:36<00:33, 143.58it/s]
 52%|#####1    | 5169/10000 [00:36<00:33, 143.57it/s]
 52%|#####1    | 5184/10000 [00:36<00:33, 143.60it/s]
 52%|#####1    | 5199/10000 [00:36<00:33, 143.59it/s]
 52%|#####2    | 5214/10000 [00:36<00:33, 143.61it/s]
 52%|#####2    | 5229/10000 [00:36<00:33, 143.65it/s]
 52%|#####2    | 5244/10000 [00:36<00:33, 143.66it/s]
 53%|#####2    | 5259/10000 [00:37<00:33, 143.66it/s]
 53%|#####2    | 5274/10000 [00:37<00:32, 143.67it/s]
 53%|#####2    | 5289/10000 [00:37<00:32, 143.71it/s]
 53%|#####3    | 5304/10000 [00:37<00:32, 143.75it/s]
 53%|#####3    | 5319/10000 [00:37<00:32, 143.72it/s]
 53%|#####3    | 5334/10000 [00:37<00:32, 143.75it/s]
 53%|#####3    | 5349/10000 [00:37<00:32, 143.80it/s]
 54%|#####3    | 5364/10000 [00:37<00:32, 143.19it/s]
 54%|#####3    | 5379/10000 [00:37<00:32, 143.35it/s]
 54%|#####3    | 5394/10000 [00:37<00:32, 143.53it/s]
 54%|#####4    | 5409/10000 [00:38<00:31, 143.63it/s]
 54%|#####4    | 5424/10000 [00:38<00:31, 143.65it/s]
 54%|#####4    | 5439/10000 [00:38<00:31, 143.60it/s]
 55%|#####4    | 5454/10000 [00:38<00:31, 143.63it/s]
 55%|#####4    | 5469/10000 [00:38<00:31, 143.67it/s]
 55%|#####4    | 5484/10000 [00:38<00:31, 143.61it/s]
 55%|#####4    | 5499/10000 [00:38<00:31, 143.62it/s]
 55%|#####5    | 5514/10000 [00:38<00:31, 143.51it/s]
 55%|#####5    | 5529/10000 [00:38<00:31, 143.59it/s]
 55%|#####5    | 5544/10000 [00:39<00:31, 143.55it/s]
 56%|#####5    | 5559/10000 [00:39<00:30, 143.60it/s]
 56%|#####5    | 5574/10000 [00:39<00:30, 143.68it/s]
 56%|#####5    | 5589/10000 [00:39<00:30, 143.48it/s]
 56%|#####6    | 5604/10000 [00:39<00:30, 143.57it/s]
 56%|#####6    | 5619/10000 [00:39<00:30, 143.63it/s]
 56%|#####6    | 5634/10000 [00:39<00:30, 143.68it/s]
 56%|#####6    | 5649/10000 [00:39<00:30, 143.72it/s]
 57%|#####6    | 5664/10000 [00:39<00:30, 143.74it/s]
 57%|#####6    | 5679/10000 [00:39<00:30, 143.80it/s]
 57%|#####6    | 5694/10000 [00:40<00:29, 143.81it/s]
 57%|#####7    | 5709/10000 [00:40<00:29, 143.86it/s]
 57%|#####7    | 5724/10000 [00:40<00:29, 143.88it/s]
 57%|#####7    | 5739/10000 [00:40<00:29, 143.90it/s]
 58%|#####7    | 5754/10000 [00:40<00:29, 143.86it/s]
 58%|#####7    | 5769/10000 [00:40<00:29, 143.85it/s]
 58%|#####7    | 5784/10000 [00:40<00:29, 143.81it/s]
 58%|#####7    | 5799/10000 [00:40<00:29, 143.81it/s]
 58%|#####8    | 5814/10000 [00:40<00:29, 143.80it/s]
 58%|#####8    | 5829/10000 [00:40<00:29, 143.79it/s]
 58%|#####8    | 5844/10000 [00:41<00:28, 143.83it/s]
 59%|#####8    | 5859/10000 [00:41<00:28, 143.85it/s]
 59%|#####8    | 5874/10000 [00:41<00:28, 143.80it/s]
 59%|#####8    | 5889/10000 [00:41<00:28, 143.81it/s]
 59%|#####9    | 5904/10000 [00:41<00:28, 143.81it/s]
 59%|#####9    | 5919/10000 [00:41<00:28, 143.84it/s]
 59%|#####9    | 5934/10000 [00:41<00:28, 143.85it/s]
 59%|#####9    | 5949/10000 [00:41<00:28, 143.86it/s]
 60%|#####9    | 5964/10000 [00:41<00:28, 143.81it/s]
 60%|#####9    | 5979/10000 [00:42<00:27, 143.82it/s]
 60%|#####9    | 5994/10000 [00:42<00:27, 143.84it/s]
 60%|######    | 6009/10000 [00:42<00:27, 143.76it/s]
 60%|######    | 6024/10000 [00:42<00:27, 143.79it/s]
 60%|######    | 6039/10000 [00:42<00:27, 143.79it/s]
 61%|######    | 6054/10000 [00:42<00:27, 143.81it/s]
 61%|######    | 6069/10000 [00:42<00:27, 143.80it/s]
 61%|######    | 6084/10000 [00:42<00:27, 143.36it/s]
 61%|######    | 6099/10000 [00:42<00:27, 143.53it/s]
 61%|######1   | 6114/10000 [00:42<00:27, 143.66it/s]
 61%|######1   | 6129/10000 [00:43<00:26, 143.73it/s]
 61%|######1   | 6144/10000 [00:43<00:26, 143.78it/s]
 62%|######1   | 6159/10000 [00:43<00:26, 143.77it/s]
 62%|######1   | 6174/10000 [00:43<00:26, 143.80it/s]
 62%|######1   | 6189/10000 [00:43<00:26, 143.85it/s]
 62%|######2   | 6204/10000 [00:43<00:26, 143.83it/s]
 62%|######2   | 6219/10000 [00:43<00:27, 135.33it/s]
 62%|######2   | 6233/10000 [00:43<00:36, 104.14it/s]
 62%|######2   | 6248/10000 [00:44<00:33, 113.55it/s]
 63%|######2   | 6263/10000 [00:44<00:30, 120.75it/s]
 63%|######2   | 6278/10000 [00:44<00:29, 126.93it/s]
 63%|######2   | 6293/10000 [00:44<00:28, 131.64it/s]
 63%|######3   | 6308/10000 [00:44<00:27, 135.10it/s]
 63%|######3   | 6323/10000 [00:44<00:26, 137.62it/s]
 63%|######3   | 6338/10000 [00:44<00:26, 139.47it/s]
 64%|######3   | 6353/10000 [00:44<00:25, 140.70it/s]
 64%|######3   | 6368/10000 [00:44<00:25, 141.63it/s]
 64%|######3   | 6383/10000 [00:44<00:25, 142.25it/s]
 64%|######3   | 6398/10000 [00:45<00:25, 142.75it/s]
 64%|######4   | 6413/10000 [00:45<00:25, 143.07it/s]
 64%|######4   | 6428/10000 [00:45<00:24, 143.31it/s]
 64%|######4   | 6443/10000 [00:45<00:24, 143.38it/s]
 65%|######4   | 6458/10000 [00:45<00:24, 143.52it/s]
 65%|######4   | 6473/10000 [00:45<00:24, 143.64it/s]
 65%|######4   | 6488/10000 [00:45<00:24, 143.72it/s]
 65%|######5   | 6503/10000 [00:45<00:24, 143.77it/s]
 65%|######5   | 6518/10000 [00:45<00:24, 143.78it/s]
 65%|######5   | 6533/10000 [00:46<00:24, 142.89it/s]
 65%|######5   | 6548/10000 [00:46<00:24, 143.06it/s]
 66%|######5   | 6563/10000 [00:46<00:24, 143.15it/s]
 66%|######5   | 6578/10000 [00:46<00:23, 143.36it/s]
 66%|######5   | 6593/10000 [00:46<00:23, 143.46it/s]
 66%|######6   | 6608/10000 [00:46<00:23, 143.52it/s]
 66%|######6   | 6623/10000 [00:46<00:23, 143.59it/s]
 66%|######6   | 6638/10000 [00:46<00:23, 143.35it/s]
 67%|######6   | 6653/10000 [00:46<00:23, 143.31it/s]
 67%|######6   | 6668/10000 [00:46<00:23, 143.38it/s]
 67%|######6   | 6683/10000 [00:47<00:23, 143.50it/s]
 67%|######6   | 6698/10000 [00:47<00:23, 143.56it/s]
 67%|######7   | 6713/10000 [00:47<00:22, 143.61it/s]
 67%|######7   | 6728/10000 [00:47<00:22, 143.38it/s]
 67%|######7   | 6743/10000 [00:47<00:22, 143.50it/s]
 68%|######7   | 6758/10000 [00:47<00:22, 143.59it/s]
 68%|######7   | 6773/10000 [00:47<00:22, 143.62it/s]
 68%|######7   | 6788/10000 [00:47<00:22, 142.93it/s]
 68%|######8   | 6803/10000 [00:47<00:22, 142.55it/s]
 68%|######8   | 6818/10000 [00:48<00:22, 142.86it/s]
 68%|######8   | 6833/10000 [00:48<00:22, 143.14it/s]
 68%|######8   | 6848/10000 [00:48<00:21, 143.32it/s]
 69%|######8   | 6863/10000 [00:48<00:21, 143.48it/s]
 69%|######8   | 6878/10000 [00:48<00:21, 143.59it/s]
 69%|######8   | 6893/10000 [00:48<00:21, 143.69it/s]
 69%|######9   | 6908/10000 [00:48<00:21, 143.71it/s]
 69%|######9   | 6923/10000 [00:48<00:21, 143.75it/s]
 69%|######9   | 6938/10000 [00:48<00:21, 143.76it/s]
 70%|######9   | 6953/10000 [00:48<00:21, 143.81it/s]
 70%|######9   | 6968/10000 [00:49<00:21, 143.81it/s]
 70%|######9   | 6983/10000 [00:49<00:20, 143.87it/s]
 70%|######9   | 6998/10000 [00:49<00:20, 143.90it/s]
 70%|#######   | 7013/10000 [00:49<00:20, 143.56it/s]
 70%|#######   | 7028/10000 [00:49<00:20, 143.62it/s]
 70%|#######   | 7043/10000 [00:49<00:20, 143.65it/s]
 71%|#######   | 7058/10000 [00:49<00:20, 143.69it/s]
 71%|#######   | 7073/10000 [00:49<00:20, 143.73it/s]
 71%|#######   | 7088/10000 [00:49<00:20, 143.77it/s]
 71%|#######1  | 7103/10000 [00:49<00:20, 143.80it/s]
 71%|#######1  | 7118/10000 [00:50<00:20, 143.83it/s]
 71%|#######1  | 7133/10000 [00:50<00:19, 143.84it/s]
 71%|#######1  | 7148/10000 [00:50<00:19, 143.80it/s]
 72%|#######1  | 7163/10000 [00:50<00:19, 143.80it/s]
 72%|#######1  | 7178/10000 [00:50<00:19, 143.84it/s]
 72%|#######1  | 7193/10000 [00:50<00:19, 143.84it/s]
 72%|#######2  | 7208/10000 [00:50<00:19, 143.85it/s]
 72%|#######2  | 7223/10000 [00:50<00:19, 143.86it/s]
 72%|#######2  | 7238/10000 [00:50<00:19, 143.85it/s]
 73%|#######2  | 7253/10000 [00:51<00:19, 143.86it/s]
 73%|#######2  | 7268/10000 [00:51<00:18, 143.89it/s]
 73%|#######2  | 7283/10000 [00:51<00:18, 143.87it/s]
 73%|#######2  | 7298/10000 [00:51<00:18, 143.89it/s]
 73%|#######3  | 7313/10000 [00:51<00:18, 143.88it/s]
 73%|#######3  | 7328/10000 [00:51<00:18, 143.89it/s]
 73%|#######3  | 7343/10000 [00:51<00:18, 143.89it/s]
 74%|#######3  | 7358/10000 [00:51<00:18, 143.90it/s]
 74%|#######3  | 7373/10000 [00:51<00:18, 143.86it/s]
 74%|#######3  | 7388/10000 [00:51<00:18, 143.87it/s]
 74%|#######4  | 7403/10000 [00:52<00:18, 143.85it/s]
 74%|#######4  | 7418/10000 [00:52<00:17, 143.85it/s]
 74%|#######4  | 7433/10000 [00:52<00:17, 143.85it/s]
 74%|#######4  | 7448/10000 [00:52<00:17, 143.83it/s]
 75%|#######4  | 7463/10000 [00:52<00:17, 143.86it/s]
 75%|#######4  | 7478/10000 [00:52<00:17, 143.78it/s]
 75%|#######4  | 7493/10000 [00:52<00:17, 143.81it/s]
 75%|#######5  | 7508/10000 [00:52<00:17, 143.14it/s]
 75%|#######5  | 7523/10000 [00:52<00:17, 143.35it/s]
 75%|#######5  | 7538/10000 [00:53<00:17, 143.47it/s]
 76%|#######5  | 7553/10000 [00:53<00:17, 143.61it/s]
 76%|#######5  | 7568/10000 [00:53<00:16, 143.66it/s]
 76%|#######5  | 7583/10000 [00:53<00:16, 143.73it/s]
 76%|#######5  | 7598/10000 [00:53<00:16, 143.80it/s]
 76%|#######6  | 7613/10000 [00:53<00:16, 143.85it/s]
 76%|#######6  | 7628/10000 [00:53<00:16, 143.81it/s]
 76%|#######6  | 7643/10000 [00:53<00:16, 142.80it/s]
 77%|#######6  | 7658/10000 [00:53<00:16, 143.10it/s]
 77%|#######6  | 7673/10000 [00:53<00:16, 143.24it/s]
 77%|#######6  | 7688/10000 [00:54<00:16, 143.37it/s]
 77%|#######7  | 7703/10000 [00:54<00:16, 142.69it/s]
 77%|#######7  | 7718/10000 [00:54<00:15, 143.04it/s]
 77%|#######7  | 7733/10000 [00:54<00:15, 143.27it/s]
 77%|#######7  | 7748/10000 [00:54<00:15, 143.45it/s]
 78%|#######7  | 7763/10000 [00:54<00:15, 143.35it/s]
 78%|#######7  | 7778/10000 [00:54<00:15, 143.40it/s]
 78%|#######7  | 7793/10000 [00:54<00:15, 143.55it/s]
 78%|#######8  | 7808/10000 [00:54<00:15, 143.62it/s]
 78%|#######8  | 7823/10000 [00:55<00:15, 143.66it/s]
 78%|#######8  | 7838/10000 [00:55<00:15, 143.70it/s]
 79%|#######8  | 7853/10000 [00:55<00:14, 143.74it/s]
 79%|#######8  | 7868/10000 [00:55<00:14, 143.77it/s]
 79%|#######8  | 7883/10000 [00:55<00:14, 143.75it/s]
 79%|#######8  | 7898/10000 [00:55<00:14, 143.79it/s]
 79%|#######9  | 7913/10000 [00:55<00:14, 143.69it/s]
 79%|#######9  | 7928/10000 [00:55<00:14, 143.71it/s]
 79%|#######9  | 7943/10000 [00:55<00:14, 143.76it/s]
 80%|#######9  | 7958/10000 [00:55<00:14, 143.80it/s]
 80%|#######9  | 7973/10000 [00:56<00:14, 143.80it/s]
 80%|#######9  | 7988/10000 [00:56<00:13, 143.81it/s]
 80%|########  | 8003/10000 [00:56<00:13, 143.81it/s]
 80%|########  | 8018/10000 [00:56<00:13, 143.82it/s]
 80%|########  | 8033/10000 [00:56<00:13, 143.82it/s]
 80%|########  | 8048/10000 [00:56<00:13, 143.85it/s]
 81%|########  | 8063/10000 [00:56<00:13, 143.88it/s]
 81%|########  | 8078/10000 [00:56<00:13, 143.86it/s]
 81%|########  | 8093/10000 [00:56<00:13, 143.86it/s]
 81%|########1 | 8108/10000 [00:56<00:13, 143.84it/s]
 81%|########1 | 8123/10000 [00:57<00:13, 143.85it/s]
 81%|########1 | 8138/10000 [00:57<00:12, 143.86it/s]
 82%|########1 | 8153/10000 [00:57<00:12, 143.86it/s]
 82%|########1 | 8168/10000 [00:57<00:12, 143.74it/s]
 82%|########1 | 8183/10000 [00:57<00:12, 143.76it/s]
 82%|########1 | 8198/10000 [00:57<00:12, 143.81it/s]
 82%|########2 | 8213/10000 [00:57<00:12, 143.80it/s]
 82%|########2 | 8228/10000 [00:57<00:12, 143.08it/s]
 82%|########2 | 8243/10000 [00:57<00:12, 143.29it/s]
 83%|########2 | 8258/10000 [00:58<00:12, 143.41it/s]
 83%|########2 | 8273/10000 [00:58<00:12, 143.54it/s]
 83%|########2 | 8288/10000 [00:58<00:11, 143.62it/s]
 83%|########3 | 8303/10000 [00:58<00:11, 143.70it/s]
 83%|########3 | 8318/10000 [00:58<00:11, 143.73it/s]
 83%|########3 | 8333/10000 [00:58<00:11, 143.72it/s]
 83%|########3 | 8348/10000 [00:58<00:11, 143.75it/s]
 84%|########3 | 8363/10000 [00:58<00:11, 143.71it/s]
 84%|########3 | 8378/10000 [00:58<00:11, 143.77it/s]
 84%|########3 | 8393/10000 [00:58<00:11, 143.74it/s]
 84%|########4 | 8408/10000 [00:59<00:11, 143.78it/s]
 84%|########4 | 8423/10000 [00:59<00:10, 143.81it/s]
 84%|########4 | 8438/10000 [00:59<00:10, 143.84it/s]
 85%|########4 | 8453/10000 [00:59<00:10, 143.64it/s]
 85%|########4 | 8468/10000 [00:59<00:10, 143.73it/s]
 85%|########4 | 8483/10000 [00:59<00:10, 143.76it/s]
 85%|########4 | 8498/10000 [00:59<00:10, 143.82it/s]
 85%|########5 | 8513/10000 [00:59<00:10, 143.84it/s]
 85%|########5 | 8528/10000 [00:59<00:10, 143.84it/s]
 85%|########5 | 8543/10000 [01:00<00:10, 143.82it/s]
 86%|########5 | 8558/10000 [01:00<00:10, 143.81it/s]
 86%|########5 | 8573/10000 [01:00<00:09, 143.83it/s]
 86%|########5 | 8588/10000 [01:00<00:09, 143.82it/s]
 86%|########6 | 8603/10000 [01:00<00:09, 143.77it/s]
 86%|########6 | 8618/10000 [01:00<00:09, 143.78it/s]
 86%|########6 | 8633/10000 [01:00<00:09, 143.77it/s]
 86%|########6 | 8648/10000 [01:00<00:09, 143.80it/s]
 87%|########6 | 8663/10000 [01:00<00:09, 143.75it/s]
 87%|########6 | 8678/10000 [01:00<00:09, 143.77it/s]
 87%|########6 | 8693/10000 [01:01<00:09, 143.80it/s]
 87%|########7 | 8708/10000 [01:01<00:08, 143.76it/s]
 87%|########7 | 8723/10000 [01:01<00:08, 143.78it/s]
 87%|########7 | 8738/10000 [01:01<00:08, 143.81it/s]
 88%|########7 | 8753/10000 [01:01<00:08, 143.85it/s]
 88%|########7 | 8768/10000 [01:01<00:08, 143.89it/s]
 88%|########7 | 8783/10000 [01:01<00:08, 143.89it/s]
 88%|########7 | 8798/10000 [01:01<00:08, 143.89it/s]
 88%|########8 | 8813/10000 [01:01<00:08, 143.88it/s]
 88%|########8 | 8828/10000 [01:01<00:08, 143.85it/s]
 88%|########8 | 8843/10000 [01:02<00:08, 143.69it/s]
 89%|########8 | 8858/10000 [01:02<00:07, 143.45it/s]
 89%|########8 | 8873/10000 [01:02<00:07, 143.59it/s]
 89%|########8 | 8888/10000 [01:02<00:07, 143.66it/s]
 89%|########9 | 8903/10000 [01:02<00:07, 143.68it/s]
 89%|########9 | 8918/10000 [01:02<00:07, 143.69it/s]
 89%|########9 | 8933/10000 [01:02<00:07, 143.78it/s]
 89%|########9 | 8948/10000 [01:02<00:07, 143.19it/s]
 90%|########9 | 8963/10000 [01:02<00:07, 143.41it/s]
 90%|########9 | 8978/10000 [01:03<00:07, 143.53it/s]
 90%|########9 | 8993/10000 [01:03<00:07, 143.07it/s]
 90%|######### | 9008/10000 [01:03<00:06, 143.28it/s]
 90%|######### | 9023/10000 [01:03<00:06, 143.45it/s]
 90%|######### | 9038/10000 [01:03<00:06, 143.60it/s]
 91%|######### | 9053/10000 [01:03<00:06, 143.71it/s]
 91%|######### | 9068/10000 [01:03<00:06, 143.72it/s]
 91%|######### | 9083/10000 [01:03<00:06, 143.74it/s]
 91%|######### | 9098/10000 [01:03<00:06, 143.79it/s]
 91%|#########1| 9113/10000 [01:03<00:06, 143.73it/s]
 91%|#########1| 9128/10000 [01:04<00:06, 143.76it/s]
 91%|#########1| 9143/10000 [01:04<00:05, 143.07it/s]
 92%|#########1| 9158/10000 [01:04<00:05, 143.30it/s]
 92%|#########1| 9173/10000 [01:04<00:05, 143.48it/s]
 92%|#########1| 9188/10000 [01:04<00:05, 143.59it/s]
 92%|#########2| 9203/10000 [01:04<00:05, 143.69it/s]
 92%|#########2| 9218/10000 [01:04<00:05, 143.77it/s]
 92%|#########2| 9233/10000 [01:04<00:05, 143.78it/s]
 92%|#########2| 9248/10000 [01:04<00:05, 143.82it/s]
 93%|#########2| 9263/10000 [01:05<00:05, 143.83it/s]
 93%|#########2| 9278/10000 [01:05<00:05, 143.85it/s]
 93%|#########2| 9293/10000 [01:05<00:04, 143.89it/s]
 93%|#########3| 9308/10000 [01:05<00:04, 143.89it/s]
 93%|#########3| 9323/10000 [01:05<00:04, 143.83it/s]
 93%|#########3| 9338/10000 [01:05<00:04, 143.88it/s]
 94%|#########3| 9353/10000 [01:05<00:04, 143.88it/s]
 94%|#########3| 9368/10000 [01:05<00:04, 143.91it/s]
 94%|#########3| 9383/10000 [01:05<00:04, 143.92it/s]
 94%|#########3| 9398/10000 [01:05<00:04, 143.91it/s]
 94%|#########4| 9413/10000 [01:06<00:04, 143.93it/s]
 94%|#########4| 9428/10000 [01:06<00:03, 143.95it/s]
 94%|#########4| 9443/10000 [01:06<00:03, 144.01it/s]
 95%|#########4| 9458/10000 [01:06<00:03, 143.97it/s]
 95%|#########4| 9473/10000 [01:06<00:03, 143.93it/s]
 95%|#########4| 9488/10000 [01:06<00:03, 143.90it/s]
 95%|#########5| 9503/10000 [01:06<00:03, 143.90it/s]
 95%|#########5| 9518/10000 [01:06<00:03, 143.86it/s]
 95%|#########5| 9533/10000 [01:06<00:03, 143.88it/s]
 95%|#########5| 9548/10000 [01:07<00:03, 143.82it/s]
 96%|#########5| 9563/10000 [01:07<00:03, 143.85it/s]
 96%|#########5| 9578/10000 [01:07<00:02, 143.86it/s]
 96%|#########5| 9593/10000 [01:07<00:02, 143.82it/s]
 96%|#########6| 9608/10000 [01:07<00:02, 143.85it/s]
 96%|#########6| 9623/10000 [01:07<00:02, 143.83it/s]
 96%|#########6| 9638/10000 [01:07<00:02, 143.84it/s]
 97%|#########6| 9653/10000 [01:07<00:02, 143.25it/s]
 97%|#########6| 9668/10000 [01:07<00:02, 143.43it/s]
 97%|#########6| 9683/10000 [01:07<00:02, 143.57it/s]
 97%|#########6| 9698/10000 [01:08<00:02, 143.67it/s]
 97%|#########7| 9713/10000 [01:08<00:01, 143.78it/s]
 97%|#########7| 9728/10000 [01:08<00:01, 143.82it/s]
 97%|#########7| 9743/10000 [01:08<00:01, 143.83it/s]
 98%|#########7| 9758/10000 [01:08<00:01, 143.86it/s]
 98%|#########7| 9773/10000 [01:08<00:01, 143.81it/s]
 98%|#########7| 9788/10000 [01:08<00:01, 143.82it/s]
 98%|#########8| 9803/10000 [01:08<00:01, 143.77it/s]
 98%|#########8| 9818/10000 [01:08<00:01, 143.80it/s]
 98%|#########8| 9833/10000 [01:08<00:01, 143.83it/s]
 98%|#########8| 9848/10000 [01:09<00:01, 143.82it/s]
 99%|#########8| 9863/10000 [01:09<00:00, 143.82it/s]
 99%|#########8| 9878/10000 [01:09<00:00, 143.65it/s]
 99%|#########8| 9893/10000 [01:09<00:00, 143.69it/s]
 99%|#########9| 9908/10000 [01:09<00:00, 143.76it/s]
 99%|#########9| 9923/10000 [01:09<00:00, 143.79it/s]
 99%|#########9| 9938/10000 [01:09<00:00, 143.80it/s]
100%|#########9| 9953/10000 [01:09<00:00, 143.84it/s]
100%|#########9| 9968/10000 [01:09<00:00, 143.91it/s]
100%|#########9| 9983/10000 [01:10<00:00, 143.89it/s]
100%|#########9| 9998/10000 [01:10<00:00, 143.94it/s]
100%|##########| 10000/10000 [01:10<00:00, 142.55it/s]
Resulting samples with prior model lower bounds of [-1,-10,-10,-10], upper bounds of [2,10,10,10]

Is there much change to the posterior distribution?

---

Watermark

watermark_list = ["cofi", "numpy", "scipy", "matplotlib", "emcee", "arviz"]
for pkg in watermark_list:
    pkg_var = __import__(pkg)
    print(pkg, getattr(pkg_var, "__version__"))

cofi 0.1.2.dev22
numpy 1.21.6
scipy 1.9.1
matplotlib 3.5.3
emcee 3.1.2
arviz 0.12.1

sphinx_gallery_thumbnail_number = -1