NNS.reg is a very robust regression
technique capable of nonlinear regressions of continuous variables and
classification tasks in machine learning problems.
We have extended the NNS.reg
applications per the use of an ensemble method of classification in
NNS.boost. In short,
NNS.reg is the base learner instead of
trees.
One major advantage NNS.boost has over tree
based methods is the ability to seamlessly extrapolate beyond the
current range of observations.
Popular boosting algorithms take a series of weak learning decision
tree models, and aggregate their outputs. NNS is also a
decision tree of sorts, by partitioning each regressor with respect to
the dependent variable. We can directly control the number of “splits”
with the NNS.reg(..., order = , ...)
parameter.
We can see how NNS partitions each regressor by calling
the $rhs.partitions output. You will notice that each
partition is not an equal interval, nor of equal length, which
differentiates NNS from other bandwidth or tree-based
techniques.
Higher dependence between a regressor and the dependent variable will
allow for a larger number of partitions. This is determined internally
with the NNS.dep measure.
## V1 V2 V3 V4
## 1: 4.300000 2.000000 1.000000 0.100000
## 2: 4.385417 2.170833 1.050000 0.200000
## 3: 4.580521 2.319048 1.200000 0.300000
## 4: 4.700000 2.500000 1.300000 0.400000
## 5: 4.800000 2.600000 1.400000 0.500000
## 6: 4.900000 2.700000 1.500000 0.600000
## 7: 5.000000 2.800000 1.600000 1.000000
## 8: 5.100000 2.900000 1.700000 1.100000
## 9: 5.200000 3.000000 1.900000 1.200000
## 10: 5.300000 3.100000 3.416305 1.300000
## 11: 5.400000 3.200000 3.837990 1.400000
## 12: 5.519872 3.300000 4.000000 1.500000
## 13: 5.700000 3.400000 4.188889 1.600000
## 14: 5.800000 3.500000 4.400000 1.700000
## 15: 5.900000 3.600000 4.500000 1.800000
## 16: 6.000000 3.700000 4.674969 1.900000
## 17: 6.100000 3.800000 4.863889 2.000000
## 18: 6.200000 4.002637 5.000000 2.114583
## 19: 6.300000 4.400000 5.100000 2.300000
## 20: 6.400000 NA 5.200000 2.431040
## 21: 6.500000 NA 5.333333 2.500000
## 22: 6.600000 NA 5.500000 NA
## 23: 6.700000 NA 5.614583 NA
## 24: 6.800000 NA 5.848512 NA
## 25: 6.900000 NA 6.333750 NA
## 26: 7.050000 NA 6.900000 NA
## 27: 7.221250 NA NA NA
## 28: 7.691246 NA NA NA
## 29: 7.900000 NA NA NA
## V1 V2 V3 V4NNS.boost()Through resampling of the training set and letting each iterated set of data speak for themselves (while paying extra attention to the residuals throughout), we can test various regressor combinations in these dynamic decision trees…only keeping those combinations that add predictive value. From there we simply aggregate the predictions.
NNS.boost will automatically search for
an accuracy threshold from the training set, reporting
iterations remaining and level obtained in the console. A plot of the
frequency of the learning accuracy on the training set is also
provided.
Once a threshold is obtained,
NNS.boost will test various feature
combinations against different splits of the training set and report
back the frequency of each regressor used in the final estimate.
Let’s have a look and see how it works. We use 140 random
iris observations as our training set with the 10 holdout
observations as our test set. For brevity, we set
epochs = 10, learner.trials = 10, folds = 1.
NOTE: Base category of response variable should be 1, not 0
for classification problems when using
NNS.boost(..., type = "CLASS").
test.set = 141:150
a = NNS.boost(IVs.train = iris[-test.set, 1:4],
DV.train = iris[-test.set, 5],
IVs.test = iris[test.set, 1:4],
epochs = 10, learner.trials = 10,
status = FALSE, balance = TRUE,
type = "CLASS", folds = 1)
a$results
[1] 3 3 3 3 3 3 3 3 3 3
a$feature.weights
Petal.Width Petal.Length Sepal.Length
0.4285714 0.4285714 0.1428571
a$feature.frequency
Petal.Width Petal.Length Sepal.Length
3 3 1
mean( a$results == as.numeric(iris[test.set, 5]) )
[1] 1A perfect classification, using the features weighted per the output above.
NNS.stack()The NNS.stack() routine cross-validates
for a given objective function the n.best parameter in the
multivariate NNS.reg function as well as
the threshold parameter in the dimension reduction
NNS.reg version.
NNS.stack can be used for classification
via
NNS.stack(..., type = "CLASS", ...).
For brevity, we set folds = 1.
NOTE: Base category of response variable should be 1, not 0
for classification problems when using
NNS.stack(..., type = "CLASS").
b = NNS.stack(IVs.train = iris[-test.set, 1:4],
DV.train = iris[-test.set, 5],
IVs.test = iris[test.set, 1:4],
type = "CLASS", balance = TRUE,
ncores = 1, folds = 1)
b$OBJfn.reg
[1] 1
$NNS.reg.n.best
[1] 1
$probability.threshold
[1] 0.43875
$OBJfn.dim.red
[1] 0.9798658
$NNS.dim.red.threshold
[1] 0.93
$reg
[1] 3 3 3 3 3 3 3 3 3 3
$reg.pred.int
NULL
$dim.red
[1] 3 3 3 3 3 3 3 3 3 3
$dim.red.pred.int
NULL
$stack
[1] 3 3 3 3 3 3 3 3 3 3
$pred.int
NULLdepth = "max" will force all observations to be
their own partition, forcing a perfect fit of the multivariate
regression. In essence, this is the basis for a kNN nearest
neighbor type of classification.
n.best = 1 will use the single nearest neighbor.
When coupled with depth = "max", NNS will
emulate a kNN = 1 but as the dimensions increase the
results diverge demonstrating NNS is less sensitive to the
curse of dimensionality than kNN.
extreme will use the maximum or minimum
threshold obtained, and may result in errors if that
threshold cannot be eclipsed by subsequent iterations.
If the user is so motivated, detailed arguments further examples are provided within the following: