-
Notifications
You must be signed in to change notification settings - Fork 6
/
README.Rmd
120 lines (91 loc) · 3.71 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
---
output: github_document
---
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "##",
fig.path = "man/figures/",
fig.width = 9,
fig.height = 3,
warning = FALSE,
dpi = 150
)
```
# proxyC: R package for large-scale similarity/distance computation
<!-- badges: start -->
[](https://CRAN.R-project.org/package=proxyC)
[](https://CRAN.R-project.org/package=proxyC)
[](https://CRAN.R-project.org/package=proxyC)
[](https://github.com/koheiw/proxyC/actions)
[](https://app.codecov.io/gh/koheiw/proxyC)
<!-- badges: end -->
**proxyC** computes proximity between rows or columns of large matrices efficiently in C++. It is optimized for large sparse matrices using the Armadillo and Intel TBB libraries. Among several built-in similarity/distance measures, computation of correlation, cosine similarity and Euclidean distance is particularly fast.
This code was originally written for [**quanteda**](https://github.com/quanteda/quanteda) to compute similarity/distance between documents or features in large corpora, but separated as a stand-alone package to make it available for broader data scientific purposes.
## Install
Since **proxyC** v0.4.0, it requires the Intel oneAPI Threading Building Blocks for parallel computing. Windows and Mac users can download a binary package from CRAN, but Linux users must install the library by executing the commands below:
```{bash, eval=FALSE}
# Fedora, CentOS, RHEL
sudo yum install tbb-devel
# Debian and Ubuntu
sudo apt install libtbb-dev
```
```{r eval=FALSE}
install.packages("proxyC")
```
## Performance
```{r}
require(Matrix)
require(microbenchmark)
require(ggplot2)
require(magrittr)
# Set number of threads
options("proxyC.threads" = 8)
# Make a matrix with 99% zeros
sm1k <- rsparsematrix(1000, 1000, 0.01) # 1,000 columns
sm10k <- rsparsematrix(1000, 10000, 0.01) # 10,000 columns
# Convert to dense format
dm1k <- as.matrix(sm1k)
dm10k <- as.matrix(sm10k)
```
## Cosine similarity between columns
With sparse matrices, **proxyC** is roughly 10 to 100 times faster than **proxy**.
```{r, cahce=TRUE}
bm1 <- microbenchmark(
"proxy 1k" = proxy::simil(dm1k, method = "cosine"),
"proxyC 1k" = proxyC::simil(sm1k, margin = 2, method = "cosine"),
"proxy 10k" = proxy::simil(dm10k, method = "cosine"),
"proxyC 10k" = proxyC::simil(sm10k, margin = 2, method = "cosine"),
times = 10
)
autoplot(bm1)
```
## Cosine similarity greater than 0.9
If `min_simil` is used, **proxyC** becomes even faster because small similarity scores are floored to zero.
```{r, cahce=TRUE}
bm2 <- microbenchmark(
"proxyC all" = proxyC::simil(sm1k, margin = 2, method = "cosine"),
"proxyC min_simil" = proxyC::simil(sm1k, margin = 2, method = "cosine", min_simil = 0.9),
times = 10
)
autoplot(bm2)
```
Flooring by `min_simil` makes the resulting object much smaller.
```{r, cahce=TRUE}
proxyC::simil(sm10k, margin = 2, method = "cosine") %>%
object.size() %>%
print(units = "MB")
proxyC::simil(sm10k, margin = 2, method = "cosine", min_simil = 0.9) %>%
object.size() %>%
print(units = "MB")
```
## Top-10 correlation
If `rank` is used, **proxyC** only returns top-n values.
```{r, cahce=TRUE}
bm3 <- microbenchmark(
"proxyC rank" = proxyC::simil(sm1k, margin = 2, method = "correlation", rank = 10),
"proxyC all" = proxyC::simil(sm1k, margin = 2, method = "correlation"),
times = 10
)
autoplot(bm3)
```