TY - JOUR

T1 - Non-parametric entropy estimators based on simple linear regression

AU - Hino, Hideitsu

AU - Koshijima, Kensuke

AU - Murata, Noboru

N1 - Funding Information:
The authors would like to express their special thanks to the editor, the associate editor and three anonymous reviewers whose comments led to valuable improvements of the paper. H.H. was supported by JSPS KAKENHI Grant Number 25870811 and 26120504 , and N.M. was supported by JSPS KAKENHI Grant Number 25120009 .
Publisher Copyright:
© 2015 Elsevier B.V. All rights reserved.

PY - 2015/9

Y1 - 2015/9

N2 - Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

AB - Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

KW - Entropy estimation

KW - Non-parametric

KW - Simple linear regression

UR - http://www.scopus.com/inward/record.url?scp=84926286540&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84926286540&partnerID=8YFLogxK

U2 - 10.1016/j.csda.2015.03.011

DO - 10.1016/j.csda.2015.03.011

M3 - Article

AN - SCOPUS:84926286540

SN - 0167-9473

VL - 89

SP - 72

EP - 84

JO - Computational Statistics and Data Analysis

JF - Computational Statistics and Data Analysis

M1 - 6063

ER -