{"id":5431,"date":"2020-05-24T00:59:02","date_gmt":"2020-05-23T23:59:02","guid":{"rendered":"http:\/\/mathscitech.org\/articles\/?p=5431"},"modified":"2024-10-15T10:00:03","modified_gmt":"2024-10-15T09:00:03","slug":"fuzzy-classifier","status":"publish","type":"post","link":"https:\/\/mathscitech.org\/articles\/fuzzy-classifier","title":{"rendered":"Fuzzy Classifiers and Quantile Statistics for continuous data monitoring with adaptive thresholds"},"content":{"rendered":"

Abstract<\/strong> This brief note explores the use of fuzzy classifiers, with membership functions chosen using a statistical heuristic (quantile statistics), to monitor time-series metrics. The time series can arise from environmental measurements, industrial process control data, or sensor system outputs. We demonstrate implementation using the R language<\/a> on an example dataset (ozone levels in New York City). Click here to skip straight to the coded solution<\/a>), or read on for the discussion.<\/p>\n

\"\"

Fuzzy classification into 5 classes using p10 and p90 levels to achieve an 80-20 rule in the outermost classes and graded class membership in the inner three classes. Comparison with crisp classifier using the same 80-20 rule is shown in the bottom panel of the figure.<\/p><\/div>\n

<\/p>\n

1. Case study: providing automatic classification of urban air quality<\/h3>\n

Time-series approach<\/strong>
\nConsider the problem of monitoring ozone levels in an urban environment and issuing a daily text message alert to residents indicating meaningful risk levels. What should these levels be, and how should the thresholds be set? If we imagine a scenario where clinical studies on human response do not yet exist on which to base the thresholds, an alternative approach is to provide a statistical classification of ozone levels relative to previous history, i.e. classed as VERY-LOW, LOW, VERY-HIGH, HIGH, and NORMAL. Let’s see how we might do this using an example dataset from a New York City study<\/p>\n

Looking at the NYC ozone study<\/strong>
\nThe NYC ozone dataset from the State Department of Conservation is built into R and measures mean ozone levels (alongside 5 other air quality variables) over 153 days in the summer of 1973 (May 1st to Sep 30th, 1300-1500h, Roosevelt Island station). Let’s load it up in R and inspect it.<\/p>\n

<\/p>\n

\r\n### Load airquality dataset\r\nlibrary(datasets)       \r\nairquality               \r\nhelp(airquality)        # show dataset details\r\n\r\nairquality$Ozone        # parts per billion\r\n\r\n### Data visualization\r\nplot(airquality$Ozone)\r\nlines(airquality$Ozone)\r\ntitle(\"Ozone level at Roosevelt Island, NYC\\n mean ppb, 1300h-1500h, May 1-Sep 30, 1973\") \r\nhist(airquality$Ozone)\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
1973 ozone study in NYC over 153 days in the summer.  Exploratory plot in R using airquality$Ozone in library(datasets)<\/p><\/div>\n
To summarize the observations, use boxplot() to show graphically its non-parametric statistics i.e. the 5-number summary at min (p0), p25, p50, p75, and max (p100) levels.  quantile() shows the numerical values.  Note that by convention the 5-number summary captures the middle 50% of the distribution (between p25 and p75).<\/p>\n
<\/p>\n
\r\nboxplot(airquality$Ozone)\r\nclean <- function(v) { v[!is.na(v)] }      # remove's NAs from a list  (ex: clean(d))\r\ndd <- clean(airquality$Ozone)\r\nquantile(dd)                               # 5-number summary at min, Q1, Q2, Q3, max levels\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Boxplot shows 5-number summary: min\/max are the whiskers, Quartile 1, Quartile 3 (25th and 75th percentiles) are the box bottom and top, and Quartile 2 (median) is the bar within the box.<\/p><\/div>\n
2. Choosing a statistical classifier<\/h3>\n
Classifiers are heuristics matching input data to categorical labels, with the \"goodness of fit\" depending on an (often subjective) separate estimate of the correct answer.<\/p>\n
In what follows, we use the non-parametric statistics heuristic with p-levels associated with quantiles as a way to capture a fixed % of the distribution.  For time-series data that are, or can be assumed to be, essentially stationary, the selection of thresholds can be done once and adjusted dynamically; for highly varying, seasonal, or other non-stationary time-series, the analysis becomes more complicated, with the central ideas below requiring adaptation.<\/p>\n
Let us take as a tenet the use of the 80-20 rule for flagging HIGH and LOW outliers, i.e. 1 in 5 observations should be abnormal, and the remaining 4 considered within usual levels of variation.  This implies setting the quantiles at p10 and p90 levels.  Notice, quantile() is overridden with the new probs vector containing the p10 and p90 levels, and the resulting distribution is plotted against these thresholding curves.<\/p>\n
<\/p>\n
\r\nqq <- quantile(dd, probs=c(0, 0.1, 0.5, 0.9, 1)) # 5-number summary at custom levels\r\nqq\r\nplot(airquality$Ozone)\r\ncurve(0*x+qq[[2]],add=TRUE)    # add horizontal lines at the quartile levels\r\ncurve(0*x+qq[[4]],add=TRUE)\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Using custom levels of p10 and p90 for the upper and lower classes leaves still a broad range of values in the middle class.<\/p><\/div>\n
3. The difference between crisp (conventional) and fuzzy classifiers.<\/h3>\n
If this is all we do, then it remains only to formalize the classification around the pre-selected p10 and p90 thresholds.  The is given below, and it should be no surprise that the results show 80% of the values classed as normal, with 10% in each of the two classes LOW and HIGH.  The size of the buckets was pre-determined by the tenet of an 80-20 classification outcome.  The only choice was what this meant in terms of the thresholds, and this was provided by the quantile() function at the respective p-levels.<\/p>\n
<\/p>\n
\r\ncc3 <- c(\"LOW\", \"NORMAL\", \"HIGH\")\r\n\r\n# Crisp classifier\r\nclassify <- function(x) {\r\n    if (x < qq[[2]]) cc3[1]\r\n    else if (x > qq[[4]]) cc3[3]\r\n    else cc3[2]\r\n}\r\nb <- sapply(dd, classify)\r\nb\r\ntable(b)\r\ntable(b)\/length(dd)\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Non-fuzzy classifier using the p10 and p90 threshold levels.<\/p><\/div>\n
But you should be protesting at the use of such a classifier.  Within the \"normal\" bucket, containing 80% of the distribution, there is a wide range of values, which leads to a problem that bedevils all attempts at crisp classification into a manageable number of levels: (1) values at the top and bottom ends (for example 85 and 18 ppb) are both classed together as normal despite the fact that they are closer to points on the other side of the class boundaries (for example 89 and 12 ppb) that are classed as HIGH and LOW respectively.<\/p>\n
Crisp classifiers get around the above limitations by using more complex mathematical models<\/strong>
\nThe discussion so far has taken the simplest possible approach to classification, and the critique of the simple approach above is deserved.  But of course, no one in statistical classification would use the threshold logic as described above without accommodating for distance between points.  One approach would be to find the separating hyperplane that minimizes the average distance between pairs on either sides of that plane.  Another approach, similar, is based on choosing centroids of clusters, again minimizing distance between pairs of points.  Both of these optimization algorithms get around the above critique and give good results, but they are much more complex mathematically (see the classic reference [Tibshirani\/2009] Elements of Statistical Learning, free PDF download<\/a> courtesy of the authors & publishers).<\/p>\n
4. The fuzzy classifier: making a naive classification approach work with low computational impact<\/h3>\n
The appeal with fuzzy classification is that it is able to preserve the simple approach and resolve discontinuity problem in a computationally efficient way using simple min-max in a point calculation, rather than requiring to solve a combinatorial optimization problem ranging over all pairs of points.<\/strong><\/p>\n
The key to fuzzy classification is a definition of set membership that includes degree of membership<\/em> as a mechanism to allow \"fuzzy boundaries\".  A single element now is a member of all classes but to different degrees.  The final classification has a strength, which matches more closely our intuition that classifications are inherently fuzzy, resisting crisp boundaries.  <\/p>\n
Let's look at the how this is implemented.<\/p>\n
<\/p>\n
\r\n# Defining linear membership functions L(x), M(x), H(x) (low, medium, high) based on p10 and p90 quantile levels\r\nx0   <- qq[[2]]\r\nx1   <- qq[[4]]\r\nLy0  <- 1.0\r\nLy1  <- 0.0\r\nHy0  <- 0.0\r\nHy1  <- 1.0\r\nxmin <- qq[[1]]\r\nxmed <- qq[[3]]\r\nxmax <- qq[5]\r\n\r\nL <- function(x) {                                       # Low\r\n         if (x < x0) Ly0\r\n    else if (x > x1) Ly1\r\n    else             Ly0 + (x-x0)*(Ly1-Ly0)\/(x1-x0)\r\n}\r\n\r\nH <- function(x) {                                       # High \r\n         if (x < x0) Hy0\r\n    else if (x > x1) Hy1\r\n    else             Hy0 + (x-x0)*(Hy1-Hy0)\/(x1-x0)\r\n}\r\n\r\nM <- function(x) {                                      # Medium\r\n         if (x < xmed) Hy0 + (x-xmin)*(Hy1-Hy0)\/(xmed-xmin)\r\n    else               Hy0 + (x-xmax)*(Ly1-Ly0)\/(xmax-xmed)\r\n}\r\n\r\n# show fuzzy membership functions\r\nx <- 1:max(dd)\r\nplot(c(1,max(dd)),c(0,1))\r\nlines(x,sapply(x,L))\r\nlines(x,sapply(x,H))\r\nlines(x,sapply(x,M))\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Fuzzy membership functions L(x) - Low, M(x) - Medium, H(x) - High, using the p10 and p90 thresholds.<\/p><\/div>\n
Let's talk through what is going on above.  The fuzzy classifier above uses membership functions L(x), M(x), H(x) (low, medium, high) to generate overlapping degrees of membership for each class.  The membership functions are defined over the same domain, allowing classifications that show graded membership.<\/p>\n
What happens next is the key bit: the classification of a given point x is made by applying a fuzzification operation across the candidate fuzzy sets.  In this case the fuzzification operator is the simple mathematical maximum, max().  Working an example: the value 70 is classified into the \"LOW\" bucket based on L(70) returning the highest (max) value from the three choices L(70), M(70), H(70).<\/p>\n
<\/p>\n
\r\n# example:\r\ncc3 <- c(\"LOW\", \"NORMAL\", \"HIGH\")\r\nfv <- c(L(70),M(70),H(70))     # fuzzy vector\r\nfv\r\nmax(fv)\r\nwhich.max(fv)\r\ncc3[which.max(fv)]\r\n<\/pre>\n
<\/code><\/p>\n
We use this notion to define the fuzzy classifier itself, and apply it to the dataset:<\/p>\n
<\/p>\n
\r\n# Fuzzy classifier into 3 classes\r\ncc3 <- c(\"LOW\", \"NORMAL\", \"HIGH\")\r\n\r\nfuzzy_classify3 <- function(x) {\r\n    fv <- c(L(x), M(x), H(x))\r\n    # print(fv)     #debug\r\n    # print(max(fv))  #debug\r\n    m3 <- which.max(fv)\r\n    cc3[m3]\r\n}\r\n\r\nbb <- sapply(dd, fuzzy_classify3)\r\nbb\r\ntable(bb)\r\ntable(bb)\/length(dd)\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Example showing fuzzy classification using max over membership functions.<\/p><\/div>\n
\"\"
Fuzzy classification of NYC ozone levels into 3 classes.<\/p><\/div>\n
Notice that the medium cluster now occurs only 35% of the time (or 44% of the 80% previously classed as NORMAL), which is a slight problem.  The fuzzy classifier into three clusters no longer preserves the 80-20 rule that led us to use p10 and p90 thresholds in the first place.  <\/p>\n
The fix is to use 5 clusters where the outermost two (VERY HIGH and VERY LOW) preserve the 80-20 rule based on the p90 and p10 clusters in line with the crisp classifier .  Fuzziness is limited to the middle cluster now distinguished into three: medium-high, medium-low (now labelled HIGH and LOW), and the reduced MIDDLE class holding the 35% of instances.  <\/p>\n
<\/p>\n
\r\ncc5 <- c(\"VERY LOW\", \"LOW\", \"MEDIUM\", \"HIGH\", \"VERY HIGH\")\r\n\r\nfuzzy_classify5 <- function(x) {\r\n    fv <- c(L(x), M(x), H(x))\r\n    # print(fv)     #debug\r\n    # print(max(fv))  #debug\r\n    m3 <- which.max(fv)\r\n    m5 <- if (m3==1 && max(fv)==1) 1 else if (m3==3 && max(fv)==1) 5 else (1+m3)\r\n    cc5[m5]\r\n}\r\n\r\nbb <- sapply(dd, fuzzy_classify5)\r\nbb\r\ntable(bb)\r\ntable(bb)\/length(dd)\r\n<\/pre>\n
<\/code><\/p>\n
\"\"
Fuzzy classification into 5 classes using p10 and p90 levels to achieve an 80-20 rule in the outermost classes and graded class membership in the inner three classes.  Comparison with crisp classifier using the same 80-20 rule is shown in the bottom panel of the figure.<\/p><\/div>\n
\"\"
Fuzzy classification into 5 classes.<\/p><\/div>\n
5. Evaluating the outcome<\/h3>\n
The fuzzy classifier described above provides a continuous transition between the classification strengths between classes that share a common boundary.  This can be seen by observing that each membership functions are piecewise linear and therefore piecewise continuous hence the max hull is also piecewise continuous.  <\/p>\n
The fact that the classification changes themselves are abrupt was not the concern (this is true in any categorical labeling, including in the separating hyperplane solution).  Rather it is the degree of membership separating two objects across class boundaries e.g. 25 (classed LOW) and 26 (classed MEDIUM).  Using the fuzzy classifier, this inter-cluster membership gradient is arbitrarily small based on the distance between the points, in the above example it is less than 2%.<\/p>\n
\"\"
The difference in membership between classes is less than 2% at the class boundary.<\/p><\/div>\n
This continuous degree of membership allows both categorical (rules-based) and proportional (level-based) responses to be used as part of an overall fuzzy systems control.<\/p>\n
6. Using fuzzy classifiers - adaptive thresholds, unsupervised vs. supervised learning, and a general framework<\/h3>\n
In the application context of the daily text alert, fuzzy classification would provide the autonomous categorization each alert\/alarm.  The 80-20 tenet would be reflected in the fact that approximately 20% of alerts will be in the VERY HIGH or VERY LOW risk category.<\/p>\n
Stepping back from the example context, the general framework for using a fuzzy classifier is as follows:<\/p>\n
A. TRAIN the classifier on available history.
\n  1. Where possible, CHOOSE simple thresholds.  In the above, we used the 80-20 rule and therefore p10\/p90 levels
\n  2. DEFINE a fuzzy classifier on 5 classes - the outermost two classes preserve the 20% outliers, the inner three classes hold the 80% and provide finer grained classifications of these cases.<\/p>\n
B. MONITOR for new inputs
\n  3. CLASSIFY - classify incoming observations as done above using max() fuzzifier.
\n  4. RESPOND - decide response with rules which could include alerts or mitigation actions or both<\/p>\n
C. UPDATE thresholds dynamically
\n  5. RECOMPUTE thresholds - add new observations to past history and reform the membership functions with each observation.<\/p>\n
This approach offers:
\n1. a methodology for continuous monitoring (watchdog), including the use of sensors to instrument a process
\n2. a way to dynamically adapt the sensitivities, i.e. for the watchdog \"to learn\"
\n3. lowers barrier to entry -- unsupervised monitoring can start using descriptive statistics adjusted for any application specific tenets.  If specific training data are available, thresholds can be refined using known subsets.
\n4. computationally efficient classifier<\/p>\n
One should, of course, compare a heuristic as above with the optimized classifier based on a more computationally intensive combinatorial optimization approach.<\/p>\n
7. Possibilities<\/h3>\n
For the application programmer, the appeal of fuzzy classifiers lies in the opportunity to define a fuzzy control system while remaining in the categorical abstraction layer  of e.g. if HIGH then CLOSE WINDOWS, etc.  This keeps the quantitative basis of the underlying analog sensor (in this case the ozone levels) encapsulated.  As a second example, a fuzzy temperature checking could we use a thermometer and knowledge of normal body temperatures to classify temperatures into VERY LOW, LOW, NORMAL, ELEVATED, HIGH FEVER levels.  A user does not need to understand how the heat from our bodies warms mercury which rises (analog thermometer) or how a temperature sensor is digitally sampled.  The layer driving the appropriate response is the fuzzy classification; the number itself is a second, supporting layer of information.<\/p>\n
Sensors becomes interesting for training set purposes.  If the sensors can sample data from an a-priori known phenomena or pre-classified events, then the statistics around these known samples can enhance the detection\/classification.  For analog sensors, this opens up myriad possibilities:<\/p>\n