Modern datasets are characterized by numerous features related by complex dependency structures. To deal with these data, dimensionality reduction techniques are essential. Many of these techniques rely on the concept of intrinsic dimension (id), a measure of the complexity of the dataset. However, the estimation of this quantity is not trivial: often, the id depends rather dramatically on the scale of the distances among data points. At short distances, the id can be grossly overestimated due to the presence of noise, becoming smaller and approximately scale-independent only at large distances. An immediate approach to examining the scale dependence consists in decimating the dataset, which unavoidably induces non-negligible statistical errors at large scale. This article introduces a novel statistical method, Gride, that allows estimating the id as an explicit function of the scale without performing any decimation. Our approach is based on rigorous distributional results that enable the quantification of uncertainty of the estimates. Moreover, our method is simple and computationally efficient since it relies only on the distances among data points. Through simulation studies, we show that Gride is asymptotically unbiased, provides comparable estimates to other state-of-the-art methods, and is more robust to short-scale noise than other likelihood-based approaches.

Denti, F., Doimo, D., Laio, A., Mira, A., The generalized ratios intrinsic dimension estimator, <<SCIENTIFIC REPORTS>>, 2022; 12 (1): 20005-20020. [doi:10.1038/s41598-022-20991-1] [https://hdl.handle.net/10807/219504]

The generalized ratios intrinsic dimension estimator

Denti, Francesco
Primo
;
2022

Abstract

Modern datasets are characterized by numerous features related by complex dependency structures. To deal with these data, dimensionality reduction techniques are essential. Many of these techniques rely on the concept of intrinsic dimension (id), a measure of the complexity of the dataset. However, the estimation of this quantity is not trivial: often, the id depends rather dramatically on the scale of the distances among data points. At short distances, the id can be grossly overestimated due to the presence of noise, becoming smaller and approximately scale-independent only at large distances. An immediate approach to examining the scale dependence consists in decimating the dataset, which unavoidably induces non-negligible statistical errors at large scale. This article introduces a novel statistical method, Gride, that allows estimating the id as an explicit function of the scale without performing any decimation. Our approach is based on rigorous distributional results that enable the quantification of uncertainty of the estimates. Moreover, our method is simple and computationally efficient since it relies only on the distances among data points. Through simulation studies, we show that Gride is asymptotically unbiased, provides comparable estimates to other state-of-the-art methods, and is more robust to short-scale noise than other likelihood-based approaches.
2022
Inglese
Denti, F., Doimo, D., Laio, A., Mira, A., The generalized ratios intrinsic dimension estimator, <<SCIENTIFIC REPORTS>>, 2022; 12 (1): 20005-20020. [doi:10.1038/s41598-022-20991-1] [https://hdl.handle.net/10807/219504]
File in questo prodotto:
File Dimensione Formato  
s41598-022-20991-1.pdf

accesso aperto

Licenza: Creative commons
Dimensione 2.65 MB
Formato Adobe PDF
2.65 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10807/219504
Citazioni
  • ???jsp.display-item.citation.pmc??? 1
  • Scopus 9
  • ???jsp.display-item.citation.isi??? 6
social impact