Fitting a single model with drfsc¶

Load, fit, predict, and score using drfsc and WDBC example dataset¶

In this notebook we use the Breast Cancer Wisconsin (Diagnostic) Data Set (WDBC) dataset (available from the UCI database.) to demonstrate how to use the drfsc package and some of its functionality.

This example dataset is quite small, so only a small number of partitions will be used. This can be expanded arbitrarily, based on the dataset.

Note: some packages used in this notebook are not part of the drfsc package, and must be installed separately.

In [1]:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from drfsc import drfsc, utils
from sklearn.model_selection import train_test_split

import numpy as np import matplotlib.pyplot as plt import pandas as pd from drfsc import drfsc, utils from sklearn.model_selection import train_test_split

Loading data¶

We start by first loading the dataset.

In [2]:

data = utils.load_wdbc("wdbc.data")
print(f"Shape: {data.shape}, Dimensions: {data.ndim}")

data = utils.load_wdbc("wdbc.data") print(f"Shape: {data.shape}, Dimensions: {data.ndim}")

Shape: (569, 32), Dimensions: 2

As can be seen from the code below, the first column is an ID column (not used), the second column is the label, and the remaining columns are the features. We split the data accordingly.

In [3]:

data = pd.DataFrame(data)
data.head()

data = pd.DataFrame(data) data.head()

Out[3]:

	0	1	2	3	4	5	6	7	8	9	...	22	23	24	25	26	27	28	29	30	31
0	842302.0	1.0	17.99	10.38	122.80	1001.0	0.11840	0.27760	0.3001	0.14710	...	25.38	17.33	184.60	2019.0	0.1622	0.6656	0.7119	0.2654	0.4601	0.11890
1	842517.0	1.0	20.57	17.77	132.90	1326.0	0.08474	0.07864	0.0869	0.07017	...	24.99	23.41	158.80	1956.0	0.1238	0.1866	0.2416	0.1860	0.2750	0.08902
2	84300903.0	1.0	19.69	21.25	130.00	1203.0	0.10960	0.15990	0.1974	0.12790	...	23.57	25.53	152.50	1709.0	0.1444	0.4245	0.4504	0.2430	0.3613	0.08758
3	84348301.0	1.0	11.42	20.38	77.58	386.1	0.14250	0.28390	0.2414	0.10520	...	14.91	26.50	98.87	567.7	0.2098	0.8663	0.6869	0.2575	0.6638	0.17300
4	84358402.0	1.0	20.29	14.34	135.10	1297.0	0.10030	0.13280	0.1980	0.10430	...	22.54	16.67	152.20	1575.0	0.1374	0.2050	0.4000	0.1625	0.2364	0.07678

5 rows × 32 columns

In [4]:

X = data.loc[:, 2:]
X.columns = [f"x_{i}" for i in range(1, X.shape[1] + 1)] # renaming columns
X.head()

X = data.loc[:, 2:] X.columns = [f"x_{i}" for i in range(1, X.shape[1] + 1)] # renaming columns X.head()

Out[4]:

	x_1	x_2	x_3	x_4	x_5	x_6	x_7	x_8	x_9	x_10	...	x_21	x_22	x_23	x_24	x_25	x_26	x_27	x_28	x_29	x_30
0	17.99	10.38	122.80	1001.0	0.11840	0.27760	0.3001	0.14710	0.2419	0.07871	...	25.38	17.33	184.60	2019.0	0.1622	0.6656	0.7119	0.2654	0.4601	0.11890
1	20.57	17.77	132.90	1326.0	0.08474	0.07864	0.0869	0.07017	0.1812	0.05667	...	24.99	23.41	158.80	1956.0	0.1238	0.1866	0.2416	0.1860	0.2750	0.08902
2	19.69	21.25	130.00	1203.0	0.10960	0.15990	0.1974	0.12790	0.2069	0.05999	...	23.57	25.53	152.50	1709.0	0.1444	0.4245	0.4504	0.2430	0.3613	0.08758
3	11.42	20.38	77.58	386.1	0.14250	0.28390	0.2414	0.10520	0.2597	0.09744	...	14.91	26.50	98.87	567.7	0.2098	0.8663	0.6869	0.2575	0.6638	0.17300
4	20.29	14.34	135.10	1297.0	0.10030	0.13280	0.1980	0.10430	0.1809	0.05883	...	22.54	16.67	152.20	1575.0	0.1374	0.2050	0.4000	0.1625	0.2364	0.07678

5 rows × 30 columns

We can also visualise the correlation between the features, and remove highly correlated features as a preprocessing step. See here for a comment regarding removing correlated features.

In [5]:

import seaborn as sns # seaborn not included as drfsc dependency

plt.figure(figsize=(20,20))
# Generate a mask to onlyshow the bottom triangle
mask = np.triu(np.ones_like(X.corr(), dtype=bool))
# generate heatmap
sns.heatmap(X.corr(), annot=True, mask=mask, vmin=-1, vmax=1, cmap='coolwarm')
plt.title('Correlation Coefficient Of Predictors')
plt.show()

import seaborn as sns # seaborn not included as drfsc dependency plt.figure(figsize=(20,20)) # Generate a mask to onlyshow the bottom triangle mask = np.triu(np.ones_like(X.corr(), dtype=bool)) # generate heatmap sns.heatmap(X.corr(), annot=True, mask=mask, vmin=-1, vmax=1, cmap='coolwarm') plt.title('Correlation Coefficient Of Predictors') plt.show()

In [6]:

'''
Calculation of how many times a feature was an in high correlation.
Count = 1 denotes only autocorrelation.
'''
df_CorrCount = utils.get_corr_df(X, level=0.8)
display(df_CorrCount)

''' Calculation of how many times a feature was an in high correlation. Count = 1 denotes only autocorrelation. ''' df_CorrCount = utils.get_corr_df(X, level=0.8) display(df_CorrCount)

	Count
x_1	7.0
x_2	2.0
x_3	7.0
x_4	8.0
x_5	2.0
x_6	6.0
x_7	5.0
x_8	10.0
x_9	1.0
x_10	1.0
x_11	3.0
x_12	1.0
x_13	3.0
x_14	5.0
x_15	1.0
x_16	3.0
x_17	2.0
x_18	1.0
x_19	1.0
x_20	2.0
x_21	7.0
x_22	2.0
x_23	8.0
x_24	8.0
x_25	2.0
x_26	5.0
x_27	5.0
x_28	7.0
x_29	1.0
x_30	2.0

From this information, we can (optionally) drop the highly correlated features (see below cell).

In [7]:

# col2drop = ['x_1','x_8','x_23','x_24','x_28']
# X = X.drop(columns=col2drop)

# col2drop = ['x_1','x_8','x_23','x_24','x_28'] # X = X.drop(columns=col2drop)

In [8]:

Y = data.loc[:, 1]
Y.head()

Y = data.loc[:, 1] Y.head()

Out[8]:

0    1.0
1    1.0
2    1.0
3    1.0
4    1.0
Name: 1, dtype: float64

The data then needs to be split into training/validation/testing partitions for use by DRFSC. This is done in the standard way using scikit-learn's train_test_split function.

In [9]:

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42, stratify=Y)
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.2, random_state=42, stratify=Y_train)

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42, stratify=Y) X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.2, random_state=42, stratify=Y_train)

To initialize our model, we call the DRFSC class from the drfsc module. For this tutorial, we will use 1 iteration (n_runs), and 2 vertical and 2 horizontal partitions (n_vbins and n_hbins respectively).

In this notebook we show the results for when output='single'. This means that we return a single model. The other option is output='ensemble', where an ensemble is created based on the number of horizontal partitions.

In [10]:

model = drfsc.DRFSC(n_vbins=2, n_hbins=2, n_runs=1, output='single', verbose=True)

model = drfsc.DRFSC(n_vbins=2, n_hbins=2, n_runs=1, output='single', verbose=True)

RFSC_base Initialised with with parameters: 
             n_models = 300, 
             n_iters = 150, 
             tuning = 50, 
             metric = roc_auc, 
             alpha = 0.99 
 ------------
DRFSC Initialised with parameters: 
             n_vbins = 2, 
             n_hbins = 2, 
             n_runs = 1, 
             redistribute = False, 
             sharing = all, 
             k = 0, 
             output = single, 
             metric = roc_auc, 
             max_processes is 8 
 ------------

To load the data into the DRFSC model, we call the load_data method from the DRFSC class. This function preprocesses the data. Here, we can specify the degree of polynomial expansion desired (here we use degree=2). Note: In the reminder of this tutorial in the explanations we will use notation that clearly references the class and the corresponding method e.g. DRFSC.load_data, while in coding example created object of the class i.e. model.load_data.

In [11]:

X_train, X_val, Y_train, Y_val, X_test, Y_test = model.load_data(X_train, X_val, Y_train, Y_val, X_test, Y_test, polynomial=2)

X_train, X_val, Y_train, Y_val, X_test, Y_test = model.load_data(X_train, X_val, Y_train, Y_val, X_test, Y_test, polynomial=2)

Information for Loaded Data: 
 -------------
'X_train' SHAPE: (364, 496), TYPE: ndarray
'X_val'   SHAPE: (91, 496), TYPE: ndarray
'Y_train' SHAPE: (364,), TYPE: ndarray
'Y_val'   SHAPE: (91,), TYPE: ndarray
-------------

Initial feature probabilities¶

RFSC optimization works by first assigning a probability to all features in each partition. We denote by $\mu_i$ the regressor inclusion probability (RIP) of feature $i$. This is the probability that a feature will be included in the randomly generated model. By default, a uniform probability distribution is used, i.e. the initial mu values will be set to 1/n_features for all features by default.

If we want to specify some initial $\mathbf{\mu}$ vector for the RFSC optimization, this can be done via the DRFSC.set_initial_mu() method. Below, we set the initial probabilities to be 0.1 for all features. If the DRFSC.set_initial_probabilities() is not called, the default initialization will be used.

In [12]:

model.set_initial_mu(0.1)

model.set_initial_mu(0.1)

If instead we only want to change the RIP of specific features, we can pass to DRFSC.set_initial_mu() a dictionary where key are feature names (as strings) and the values are the specified RIP, as shown below.

In [13]:

model.set_initial_mu({'x_2': 0.1, 'x_3': 0.1, 'x_9': 0.1})

model.set_initial_mu({'x_2': 0.1, 'x_3': 0.1, 'x_9': 0.1})

We can also specify a _type parameter in DRFSC.set_initial_mu. If we pass _type='index', we can specify the initial RIPs by passing a dictionary with column indicies as keys and RIPs as values. E.g.,

In [14]:

model.set_initial_mu({3: 0.1, 4: 0.1, 10: 0.1}, _type= 'index')

model.set_initial_mu({3: 0.1, 4: 0.1, 10: 0.1}, _type= 'index')

Updating the parameters for the RFSC optimisation sub-routine¶

Our DRFSC model solve several RFSC sub-problems in parallel. The parameters of the RFSC optimization process can be updated by calling the DRFSC.set_rfsc_params() method. Information for the parameters which can be updated are found in the [API documentation] for RFSC(https://markcc309.github.io/drfsc/rfsc_base_api/). The parameters are updated by passing a dictionary containing the parameters to be updated, and their new values.

An example is shown below, where we decrease 'n_models' to $150$ (default $300$), and increase the significance threshold for feature inclusion 'alpha' to $0.995$ (default $0.99$).

In [15]:

model.set_rfsc_params({'n_models': 150, 'alpha': 0.995})

model.set_rfsc_params({'n_models': 150, 'alpha': 0.995})

Updated RFSC model parameters: {'n_models': 150, 'n_iters': 150, 'tuning': 50, 'tol': 0.002, 'alpha': 0.995, 'rip_cutoff': 1, 'metric': 'roc_auc', 'verbose': True}

Note: By default, the 'metric' parameter will be inherited from the DRFSC model. It is not recommended to change this behaviour, though it can be done with the DRFSC.set_rfsc_params() method.

Model fitting¶

To fit our DRFSC model, we simply call the DRFSC.fit() method and pass our training and validation data and labels. This will fit the model to the data and return the best model found.

In [16]:

model.fit(X_train, X_val, Y_train, Y_val)

model.fit(X_train, X_val, Y_train, Y_val)

Number of Samples: 364. Horizontal Disitribution SHAPE: (182, 2)
Number of Features: 496. Vertical Distribution SHAPE: (249, 2)
New best model for hbin 0. roc_auc=0.97317 -- Model features [0, 453, 14, 466, 339, 248]
New best model for hbin 1. roc_auc=0.98813 -- Model features [0, 448, 73, 132, 80, 100, 487, 54]
New best model for hbin 0. roc_auc=0.98194 -- Model features [0, 457, 213, 41, 234, 114]
M: {0: {0, 132, 14, 408, 41, 54, 62, 448, 453, 73, 457, 80, 466, 339, 213, 353, 100, 487, 234, 491, 114, 248}, 1: {0, 132, 14, 408, 41, 54, 62, 448, 453, 73, 457, 80, 466, 339, 213, 353, 100, 487, 234, 491, 114, 248}}

Out[16]:

DRFSC(n_vbins=2, n_hbins=2, n_runs=1, redistribute_features=False, feature_sharing=all, k=0, output=single, metric=roc_auc, verbose=True, polynomial=1, preprocess=True, max_processes=8)

Once a single model has been fit, we can view the features and coefficients of the final model by calling the features_ and coef_ attributes. if the input data is a numpy array and has no feature names, we can instead use the feature_num attribute to access the indicies of the model features.

In [17]:

model.features_

model.features_

Out[17]:

['1',
 'x_21 x_28',
 'x_2 x_14',
 'x_4 x_18',
 'x_2 x_21',
 'x_3 x_13',
 'x_27 x_28',
 'x_1 x_24']

In [18]:

model.features_num

model.features_num

Out[18]:

[0, 448, 73, 132, 80, 100, 487, 54]

In [19]:

model.coef_

model.coef_

Out[19]:

array([ -9.32051611,  43.76254564,  83.18183096, -33.41996069,
        18.51812967,  14.24009388,   7.00639308,   7.32392097])

We can also access the model object itself by calling the model attribute.

In [20]:

model.model

model.model

Out[20]:

<statsmodels.discrete.discrete_model.BinaryResultsWrapper at 0x2210c9f2a90>

After a model has been fit, we can use it to predict either the probabilities or labels using the DRFSC.predict_proba() or DRFSC.predict() methods. These method takes our test set data (X_test) as a sole argument, and returns the predicted probabilities and labels respectively.

In [21]:

y_prob = model.predict_proba(X_test)
y_prob

y_prob = model.predict_proba(X_test) y_prob

Out[21]:

array([1.04928122e-03, 1.00000000e+00, 7.01575824e-02, 9.84575154e-01,
       9.98796634e-01, 1.39726382e-02, 9.99629687e-01, 2.87055984e-03,
       1.87580960e-03, 3.34766331e-03, 9.96317561e-01, 1.64488715e-03,
       1.00000000e+00, 1.61966563e-02, 1.74937204e-02, 1.49468948e-04,
       8.45460471e-01, 7.15460499e-03, 3.44811845e-01, 3.59498026e-04,
       8.75569465e-04, 4.59793726e-03, 1.00000000e+00, 9.99999877e-01,
       9.94028951e-01, 6.59835110e-04, 1.16571310e-03, 9.99998549e-01,
       9.99999493e-01, 1.00000000e+00, 1.00000000e+00, 3.13604217e-03,
       1.00000000e+00, 1.00000000e+00, 4.46253174e-03, 6.35357943e-02,
       1.89576968e-02, 2.26115920e-04, 9.99977385e-01, 6.71788291e-01,
       8.10717658e-04, 6.80355374e-03, 9.95009294e-01, 1.50301009e-03,
       9.24907206e-03, 9.99996956e-01, 2.54466482e-01, 9.99997843e-01,
       2.42193649e-03, 1.74494915e-03, 1.00000000e+00, 1.00000000e+00,
       7.26417745e-02, 6.90077847e-04, 2.18171491e-04, 9.99999849e-01,
       7.24933192e-03, 4.45730366e-04, 6.63528811e-03, 1.00000000e+00,
       4.28233140e-02, 3.79618814e-02, 9.99635227e-01, 6.60172095e-05,
       2.01244956e-04, 8.65920878e-03, 1.00000000e+00, 1.00000000e+00,
       3.48145301e-01, 9.87326686e-01, 5.06352881e-04, 1.00000000e+00,
       9.99991532e-01, 3.02008630e-04, 9.53316373e-04, 9.61370693e-02,
       3.63631830e-04, 9.99999926e-01, 7.80842412e-03, 9.96281278e-01,
       9.99956856e-01, 1.00000000e+00, 7.73018257e-04, 7.34375337e-03,
       1.88869805e-04, 7.93793626e-01, 1.00000000e+00, 9.99969689e-01,
       1.00000000e+00, 1.21348611e-03, 1.00000000e+00, 4.16067862e-04,
       1.84154040e-01, 1.30565363e-02, 8.25135654e-02, 5.60788811e-02,
       1.47981479e-01, 9.99993655e-01, 1.56219194e-03, 5.50416443e-04,
       5.79811996e-04, 8.35166663e-04, 8.73915591e-03, 2.06109130e-02,
       3.84260247e-03, 5.94979000e-03, 1.00000000e+00, 9.89138483e-01,
       1.00000000e+00, 6.18069011e-04, 5.48638308e-04, 6.65531051e-01,
       9.67850414e-01, 9.99999993e-01])

In [22]:

y_pred = model.predict(X_test)
y_pred

y_pred = model.predict(X_test) y_pred

Out[22]:

array([0., 1., 0., 1., 1., 0., 1., 0., 0., 0., 1., 0., 1., 0., 0., 0., 1.,
       0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 1., 1., 0., 1., 1.,
       0., 0., 0., 0., 1., 1., 0., 0., 1., 0., 0., 1., 0., 1., 0., 0., 1.,
       1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 1., 1.,
       0., 1., 0., 1., 1., 0., 0., 0., 0., 1., 0., 1., 1., 1., 0., 0., 0.,
       1., 1., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,
       0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 1.])

Model Evaluation¶

To find out how well our model performed, we can use the DRFSC.score() method. This method takes our test set data (X_test) and labels (Y_test) as arguments, and a metric as an optional argument (here we use the default metric, roc-auc -- see here for info on roc-auc. It then calculates the metric on the test set, and returns the performance.

In [23]:

model.score(X_test, Y_test)

model.score(X_test, Y_test)

Out[23]:

{'metric': 'roc_auc', 'score': 0.996031746031746}

Model Visualization¶

To visualize the results, we can use the DRFSC.feature_importance(), DRFSC.pos_neg_prediction(), and DRFSC.single_prediction() methods.

The DRFSC.feature_importance() method takes no arguments and displays the a histogram of the final model features and their coefficeints (in absolute value). This plot gives an indication of the features which are most important to the model.

The DRFSC.pos_neg_prediction() method takes as an argument the index of the sample we want to visualize. If we want to visualise a prediction on the test set, the test set should also be passed as an argument. This method displays the positive and negative predictions for the sample, computed by multiplying the data by the model coefficients and separating the positive and negative contributions.

• E.g. usage: DRFSC.pos_neg_prediction(0, X_test) for visualising the prediction on the first sample in the test set.

The DRFSC.single_prediction() method takes the same arguments as DRFSC.pos_neg_prediction(), and diplays the model coefficients weighted by the data for the sample. This is useful for visualising the model's prediction on a single sample.

• E.g. usage: DRFSC.single_prediction(0, X_test) for visualising the prediction on the first sample in the test set.

In [24]:

model.feature_importance()
plt.show()

model.feature_importance() plt.show()

In [25]:

model.pos_neg_prediction(0, X_test)
plt.show()

model.pos_neg_prediction(0, X_test) plt.show()

In [26]:

model.single_prediction(0, X_test)
plt.show()

model.single_prediction(0, X_test) plt.show()