Lost in the Forest: Encoding categorical variables and the absent levels problem

Simple item page
dc.contributor.authorSmith HL
dc.contributor.authorBiggs PJ
dc.contributor.authorFrench NP
dc.contributor.authorSmith ANH
dc.contributor.authorMarshall JC
dc.contributor.editorGama J
dc.date.accessioned2024-07-15T01:04:36Z
dc.date.available2024-07-15T01:04:36Z
dc.date.issued2024-04-10
dc.description.abstractLevels of a predictor variable that are absent when a classification tree is grown can not be subject to an explicit splitting rule. This is an issue if these absent levels are present in a new observation for prediction. To date, there remains no satisfactory solution for absent levels in random forest models. Unlike missing data, absent levels are fully observed and known. Ordinal encoding of predictors allows absent levels to be integrated and used for prediction. Using a case study on source attribution of Campylobacter species using whole genome sequencing (WGS) data as predictors, we examine how target-agnostic versus target-based encoding of predictor variables with absent levels affects the accuracy of random forest models. We show that a target-based encoding approach using class probabilities, with absent levels designated the highest rank, is systematically biased, and that this bias is resolved by encoding absent levels according to the a priori hypothesis of equal class probability. We present a novel method of ordinal encoding predictors via principal coordinates analysis (PCO) which capitalizes on the similarity between pairs of predictor levels. Absent levels are encoded according to their similarity to each of the other levels in the training data. We show that the PCO-encoding method performs at least as well as the target-based approach and is not biased.
dc.description.confidentialfalse
dc.identifier.citationSmith HL, Biggs PJ, French NP, Smith ANH, Marshall JC. (2024). Lost in the Forest: Encoding categorical variables and the absent levels problem. Data Mining and Knowledge Discovery.
dc.identifier.doi10.1007/s10618-024-01019-w
dc.identifier.eissn1573-756X
dc.identifier.elements-typejournal-article
dc.identifier.issn1384-5810
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/70175
dc.languageEnglish
dc.publisherSpringer Nature
dc.publisher.urihttps://link.springer.com/article/10.1007/s10618-024-01019-w
dc.relation.isPartOfData Mining and Knowledge Discovery
dc.rights(c) 2024 The Author/s
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectAbsent levels
dc.subjectCampylobacter
dc.subjectclassifcation
dc.subjectrandom forest
dc.subjectsource attribution
dc.subjectvariable encoding
dc.titleLost in the Forest: Encoding categorical variables and the absent levels problem
dc.typeJournal article
pubs.elements-id488364
pubs.organisational-groupCollege of Health
Files
Original bundle
Now showing 1 - 3 of 3
Thumbnail Image
Name:
Published version.pdf
Size:
1004.77 KB
Format:
Adobe Portable Document Format
Description:
488364 PDF.pdf
Download
Thumbnail Image
Name:
Evidence 1.pdf
Size:
20.2 KB
Format:
Adobe Portable Document Format
Description:
10618_2024_1019_MOESM1_ESM.pdf
Download
Thumbnail Image
Name:
Evidence 2.pdf
Size:
14.42 KB
Format:
Adobe Portable Document Format
Description:
10618_2024_1019_MOESM2_ESM.pdf
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
9.22 KB
Format:
Plain Text
Description:
Download
Collections
Journal Articles