The Tool Box
The Tool Box
Overview plot : The first displays the correlation of the dataset variables with your target variable. In my example SalePrice of houses, from Kaggle dataset.
Overview table : The second function displays the different values existing in each of the columns. For exemple third row first column, RL [1151 = 78,84%], where RL is the value, 1151 its occurence and 78,84% its percentage.
How to systematically remove collinear variables in Python?
Source : stats.stackexchange.com
This checks VIF values and then drops variables whose VIF is more than 5.
from statsmodels.stats.outliers_influence import variance_inflation_factor def calculate_vif_(X, thresh=5.0): variables = list(range(X.shape)) dropped = True while dropped: dropped = False vif = [variance_inflation_factor(X.iloc[:, variables].values, ix) for ix in range(X.iloc[:, variables].shape)] maxloc = vif.index(max(vif)) if max(vif) > thresh: print('dropping \'' + X.iloc[:, variables].columns[maxloc] + '\' at index: ' + str(maxloc)) del variables[maxloc] dropped = True print('Remaining variables:') print(X.columns[variables]) return X.iloc[:, variables]
Regression : linear regression, Support Vector Regression (SVR), and regression trees
Classification : logistic regression, Naïve Bayes, decision trees, and K Nearest Neighbors, Decision trees, kernel approximation
Jupyter notebook offers possibilities as wide as unsuspected. Here is a compliation of links to articles that deal with this subject :
Executing shell commands, splitting notebook cells, collapsing heading, Qgrid (dynamic table as Excel), Slide shows (fixed, or interactive), embedding contents (url, pdf, youtube video, etc), or interactive widgets :
Environment switching, plus a list of 9 useful extensions, but without examples:
Profile report of a dataframe, interactive plot with plotly, and useful magic commands:
Variable inspector, execute time, hide code input:
Notify, code folding, debug:
Much more to explore here:
Qgrid demo and more:
Unofficial Jupyter Notebook Extensions page:
GroupBy, Numpy, Pandas, Matplotlib, Seaborn, Bokeh, Dash, Scikit-Learn, Keras, Template, NLK, Spyder, Jupyter, Panel, Tableau, GRETL, statistics, formula, CAP, ROC, AUC, MSE, R2, Adjust r2, t-student, p-value, chi-squared, z-test, F1 Score, Clustering, association rule, Classification, Regression, Bayes, Github, dimensionality reduction, Monte Carlo, statsmodel, sampling, scaling, cross validation, distribution,