Jump to main content
Jump to site search

Issue 7, 2015
Previous Article Next Article

Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

Author affiliations

Abstract

Fourier transform infrared (IR) spectroscopy in combination with multivariate data analysis is a versatile tool that can be applied to disease diagnosis. However, a rigorous validation of the obtained models is necessary in order to obtain robust results. This work evaluates the advantages of the use of permutation testing for determining the statistical significance of the misclassification errors obtained from IR based diagnostic models through cross validation (CV). The model performance, estimated by CV, is compared to a distribution of CV-performance values obtained using randomly permuted class labels. The distribution of ‘random CV-values’ is considered as a null distribution and used to establish the significance of the model estimators obtained using real class labels. ATR-FTIR spectra of serum samples were classified using random forest (RF) classifiers according to two criteria, the tag number (a randomly assigned pseudo class membership) and the level of urea (real class). CV errors obtained were compared to the null distribution of CV errors from a permutation test and an independent validation set. The procedure was evaluated testing typical conditions leading to overoptimistic estimations provided by the CV like e.g. the size of subsamples used during CV, variable selection and the use of replicates. Results show that for the tag number (pseudo class), CV indicated classification errors between 23 and 33% depending on the subsample size employed. Those values were even lower when variable selection or replicates were used. However, permutation testing indicated that those CV errors were non-significant. In contrast, for sample classification according to their levels of urea, all cross validation errors were found to be significant. Although the proposed method is computationally intensive, it provides a simple way of calculating an empirical p-value of the CV-estimator, thus establishing the statistical significance and providing a feasibility indicator especially useful for studies where the number of samples is limited.

Graphical abstract: Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

Back to tab navigation

Publication details

The article was received on 03 Oct 2014, accepted on 31 Oct 2014 and first published on 31 Oct 2014


Article type: Paper
DOI: 10.1039/C4AN01783H
Author version
available:
Download author version (PDF)
Analyst, 2015,140, 2422-2427

  •   Request permissions

    Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

    D. Pérez-Guaita, J. Kuligowski, S. Garrigues, G. Quintás and B. R. Wood, Analyst, 2015, 140, 2422
    DOI: 10.1039/C4AN01783H

Search articles by author

Spotlight

Advertisements