-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.Rmd
242 lines (165 loc) · 19.5 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
---
output:
github_document:
html_preview: FALSE
link-citations: TRUE
bibliography: 'citation_README.bib'
csl: 'apa-single-spaced.csl'
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
[](http://www.repostatus.org/#active) [](https://www.tidyverse.org/lifecycle/#maturing) [](https://travis-ci.org/gederajeg/collogetr) [](https://ci.appveyor.com/project/gederajeg/collogetr) [](https://codecov.io/github/gederajeg/collogetr?branch=master) [](http://dx.doi.org/10.26180/5b7b9c5e32779)
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
scaps <- function(concept) {
paste("<span style='font-variant:small-caps;'>", concept, "</span>", sep="")
}
```
# collogetr
## Overview
collogetr performs (i) collocates retrieval (currently from the sentence-based (Indonesian) [Leipzig Corpora](http://wortschatz.uni-leipzig.de/en/download)) and (ii) computation of collocation association-measures. The function `colloc_leipzig()` is used to retrieve window-span collocates for a set of word forms (viz. the *nodeword(s)* or *keyword(s)*).
Two other functions (namely, `assoc_prepare()` and `assoc_prepare_dca()`) serve to process the output of `colloc_leipzig()` into tabular/data frame formats, which then become the input data for computing the **association measure** between the collocates and the node (as in Stefanowitsch and Gries' [-@stefanowitsch_collostructions_2003] [collostructional/collocation analysis](http://www.linguistics.ucsb.edu/faculty/stgries/teaching/groningen/index.html)) [@hoffmann_collostructional_2013; @stefanowitsch_corpora_2009; see also, @gries_more_2015]. The function `assoc_prepare()` generates input data for computing [*Simple Collexeme/Collocational Analysis*](http://www.linguistics.ucsb.edu/faculty/stgries/research/2003_AS-STG_Collostructions_IJCL.pdf) (SCA), meanwhile `assoc_prepare_dca()` uses the output of `assoc_prepare()` to generate input data for computing [*Distinctive Collexeme/Collocates Analysis*](http://www.linguistics.ucsb.edu/faculty/stgries/research/2004_STG-AS_ExtendingCollostructions_IJCL.pdf) (DCA) [@hilpert_distinctive_2006; @gries_extending_2004]. Based on the output of `assoc_prepare()`, SCA can then be computed using the default `collex_fye()`, which is based on Fisher-Yates Exact test (other association measures available for computing SCA include (i) `collex_llr()` for Log-Likelihood Ratio, (ii) `collex_MI()` for Mutual Information score, (iii) `collex_TScore()` for T-Score, (iv) `collex_chisq()` for Chi-Square-based score, and (v) `collex_logOR()` for Log~10~ Odds Ratio). DCA is computed using `collex_fye_dca()`.
collogetr is built on top of the core packages in the [tidyverse](https://www.tidyverse.org).
## Installation
Install collogetr from GitHub with [devtools](https://github.com/hadley/devtools):
```{r eval = FALSE}
library(devtools)
install_github("gederajeg/collogetr")
```
## Usages
### Load collogetr
```{r load-collogetr, message = FALSE}
library(collogetr)
```
### Citation for collogetr
To cite collogetr in publication, type as follows:
```{r cite-collogetr, eval = TRUE}
citation("collogetr")
```
### Package data
The package has three data sets for demonstration. The important one is the `demo_corpus_leipzig` whose documentation can be accessed via `?demo_corpus_leipzig`. Another data is a list of Indonesian stopwords (i.e. `stopwords`) that can be filtered out when performing collocational measure. The last one is `leipzig_corpus_path` containing character vector of full path to my Leipzig Corpus files in my computer.
#### Accepted inputs
`colloc_leipzig()` accepts two types of corpus-input data:
1. A named-list object with character-vector elements of each Leipzig Corpus Files, represented by `demo_corpus_leipzig` and the format of which is shown below:
```{r leipzig_list_input}
lapply(demo_corpus_leipzig[1:2], sample, 2)
```
2. Full-paths to the Leipzig Corpus plain texts, as in the `leipzig_corpus_path`.
```{r leipzig_path_input}
leipzig_corpus_path[1:2]
```
In terms of the input strings for the `pattern` argument, `colloc_leipzig()` accepts three scenarios:
1. Plain string representing a whole word form, such as `"memberikan"` 'to give'
2. Regex of a whole word, such as `"^memberikan$"` 'to give'
3. Regex of a whole word with word boundary character (`\\b`), such as `"\\bmemberikan\\b"`.
All of these three forms will be used to match the exact word form of the search pattern after the corpus file is tokenised into individual words. That is, input patterns following scenario 1 or 3 will be turned into their exact search pattern represented in scenario 2 (i.e., with the beginning- and end-of-line anchors, hence `"^...$"`). So user can directly use the input pattern in scenario 2 for the `pattern` argument. If there are more than one word to be searched, put them into a character vector (e.g., `c("^memberi$", "^membawa$")`).
### Demo
#### Retrieving the collocates
The codes below show how one may retrieve the collocates for the Indonesian verb *mengatakan* 'to say sth.'. The function `colloc_leipzig()` will print out progress messages of the stages onto the console. It generates warning(s) when a search pattern or node word is not found in a corpus file or in all loaded corpus files.
```{r example, eval = TRUE, message = FALSE, warning = FALSE}
out <- colloc_leipzig(leipzig_corpus_list = demo_corpus_leipzig,
pattern = "mengatakan",
window = "r",
span = 1L,
save_interim = FALSE)
```
In the example above, the collocates are restricted to those occurring _one_ word (i.e. `span = 1L`) to the _right_ (`window = "r"`) of *mengatakan* 'to say'. The `"r"` character in `window` stands for *right*-side collocates (`"l"` for *left*-side collocates and `"b"` for *both* right- and left-side collocates). The `span` argument requires integer (i.e., a whole number) to indicate the range of words covered in the specified window. The `pattern` argument requires one or more exact word forms; if more than one, put into a character vector (e.g., `c("mengatakan", "menjanjikan")`).
The `save_interim` is `FALSE` means that no output is saved into the computer, but in the console (i.e., in the `out` object). If `save_interim = TRUE`, the function will save the outputs into the files in the computer. `colloc_leipzig()` has specified the default file names for the outputs via these arguments: (i) `freqlist_output_file`, (ii) `colloc_output_file`, (iii) `corpussize_output_file`, and (iv) `search_pattern_output_file`. It is recommended that the output filenames are stored as a character vector. See **Examples** "(2)" in the documentation of `colloc_leipzig()` for a call when `save_interim = TRUE`.
#### Exploring the output of `colloc_leipzig()`.
The output of `colloc_leipzig()` is a list of `r length(out)` elements:
1. `colloc_df`; a table/tibble of raw collocates data with columns for:
1. corpus names
2. sentence id in which the collocates and the node word(s) are found
3. the collocates (column `w`)
5. the span information (e.g., `"r1"` for one-word, right-side collocates)
6. the node word
7. the text/sentence match in which the collocates and the node are found
2. `freqlist_df`; a table/tibble of word-frequency list in the loaded corpus
3. `corpussize_df`; a table/tibble of total word-tokens in the loaded corpus
4. `pattern`; a character vector of the search pattern/node
```{r colloc_output}
str(out)
```
The `freqlist_df` and `corpussize_df` are important for performing the collocational strength measure for the search pattern with the collocates.
#### Preparing input data for *Simple Collexeme/Collocational Analysis* (SCA).
First we need to call `assoc_prepare()` for generating the data SCA. The demo illustrates it with in-console output of `colloc_leipzig()`. See the **Examples** "2.2" in the documentation for `assoc_prepare()` for handling saved outputs (`?assoc_prepare()`).
```{r assoc-prepare}
assoc_tb <- assoc_prepare(colloc_out = out,
window_span = "r1",
per_corpus = FALSE, # combine all data across corpus
stopword_list = collogetr::stopwords,
float_digits = 3L)
```
Inspect the output of `assoc_prepare()`:
```{r assoc-prepare-head}
head(assoc_tb)
```
The `assoc_prepare()` and `collex_fye()` functions are designed following the tidy principle so that the association/collocation measure is performed in a row-wise fashion, benefiting from the combination of [*nested* column](http://r4ds.had.co.nz/many-models.html#list-columns-1) [cf., @wickham_r_2017, p. 409] for the input-data (using `tidyr::nest()`) and `purrr`'s `map_*` function. `assoc_prepare()` includes calculating the expected co-occurrence frequencies between the collocates/collexemes and the node word/construction.
The column `data` in `assoc_tb` above consists of nested tibble/table as a list. Each contains required data for performing association measure for each of the collocates in column `w` [@gries_more_2015; @gries_50-something_2013; @stefanowitsch_corpora_2009; @stefanowitsch_collostructions_2003]. This nested column can be inspected as follows (for the first row, namely for the word *pihaknya* 'the party').
```{r inspect-nested-column}
# get the tibble in the `data` column for the first row
assoc_tb$data[[1]]
```
Column `a` indicates the co-occurrence frequency between the node word and the collocates column `w`, meanwhile `a_exp` indicates the *expected co-occurrence frequency* between them. The `n_w_in_corp` represents the total token/occurrence frequency of a given collocate. The `n_pattern` stores the total token/occurrence frequency of the node word in the corpus. Column `b`, `c`, and `d` are required for the association measure that is essentially based on 2-by-2 crosstabulation table. The `assoc` column indicates whether the value in `a` is higher than that in `a_exp`, thus indicating *attraction* or *positive association* between the node word and the collocate. The reverse is *repulsion* or *negative association* when the value in `a` is less/lower than that in `a_exp`.
#### *Simple Collexeme/Collocates Analysis (SCA)*
As in the *Collostructional Analysis* [@stefanowitsch_collostructions_2003], `collex_fye()` uses one-tailed *Fisher-Yates Exact* test whose *p*-~FisherExact~value is log-transformed to the base of 10 to indicate the collostruction strength between the collocates and the node word [@gries_converging_2005]. `collex_fye()` simultaneously performs two uni-directional measures of *Delta P* [@gries_50-something_2013; @gries_more_2015, p. 524]. One of these shows the extent to which the presence of the node-word cues the presence of the collocates/collexemes; the other one determines the extent to which the collocates/collexemes cues the presence of the node-word.
Here is the codes to perform the SCA using `collex_fye()`:
```{r perform-fye}
# perform FYE test for Collexeme Analysis
am_fye <- collex_fye(df = assoc_tb, collstr_digit = 3)
```
Now we can retrieve the top-10 most strongly attracted collocates to *mengatakan* 'to say sth.'. The association strength is shown in the `collstr` column, which stands for *collostruction strength*. The higher, the stronger the association.
```{r attracted-collocates}
# get the top-10 most strongly attracted collocates
dplyr::top_n(am_fye, 10, collstr)
```
Column `a` contains the co-occurrence frequency of the collocates (`w`) with the `node` as its R1 collocates in the demo corpus. `p_fye` shows the one-tailed *p*~FisherExact~-value.
#### *Distinctive Collexeme/Collocate Analysis* (DCA)
The idea of distinctive collexemes/collocates is to contrast *two* functionally/semantically similar constructions or words in terms of the collocates that are (significantly) more frequent for one of the two contrasted constructions/words [@hilpert_distinctive_2006; see @gries_extending_2004]. `colloc_leipzig()` can be used to retrieve collocates of *two* functionally/semantically similar words by specifying the `pattern` argument with two character vectors of words.
The following example use one of the Leipzig corpus files (not included in the package but can be downloaded from the Leipzig Corpora webpage for free), namely the `"ind_mixed_2012_1M-sentences"`. The aim is to contrast collocational preferences of two deadjectival transitive verbs based on the root *kuat* 'strong' framed within two causative morphological schemas: one with *per-*+ADJ and the other with ADJ+*-kan*. Theoretically, the *per-* schema indicates that the direct object of the verb is caused to have *more* of the characteristic indicated by the adjectival root, meanwhile the *-kan* schema indicates that the direct object is caused to have the characteristic indicated by the root (that is not previously had). The focus here is on the R1 collocates of the verbs (i.e. one word immediately to the right of the verbs in the sentences).
```{r dca-data-retrieval, message = FALSE, warning = FALSE, eval = FALSE}
my_leipzig_path <- collogetr::leipzig_corpus_path[1]
dca_coll <- collogetr::colloc_leipzig(leipzig_path = my_leipzig_path,
pattern = c("memperkuat", "menguatkan"),
window = "r",
span = 1,
save_interim = FALSE)
```
Then, we prepare the output into the format required for performing DCA with `collex_fye_dca()`.
```{r load-dca-coll, echo = FALSE}
dca_coll <- collogetr::dca_coll
```
```{r prepare-dca-input}
assoc_tb <- assoc_prepare(colloc_out = dca_coll,
window_span = "r1",
per_corpus = FALSE,
stopword_list = collogetr::stopwords,
float_digits = 3L)
# prepare the dca input table
dca_tb <- assoc_prepare_dca(assoc_tb)
```
Compute DCA for the two verbs and view the results snippet.
```{r dca-compute}
dca_res <- collex_fye_dca(dca_tb)
head(dca_res, 10)
```
The package also includes a function called `dca_top_collex()` to retrieve the top-n distinctive collocates for one of the two contrasted words. The `dist_for` argument can be specified by either the character vector of the name of the contrasted words, or the character IDs of the constructions/words (e.g., `..., dist_for = "a", ...` or `..., dist_for = "A", ...` for construction/word appearing in the second column from the output of `collex_fye_dca()`; `..., dist_for = "b", ...` or `..., dist_for = "B", ...` for construction/word appearing in the third column).
```{r top-collex-a}
# retrieve distinctive collocates for Construction A (i.e., memperkuat)
dist_for_a <- dca_top_collex(dca_res, dist_for = "memperkuat", top_n = 10)
head(dist_for_a)
```
The codes below retrieve the distinctive collocates for *menguatkan* 'to strengthen' or Construction B.
```{r top-collex-b}
# retrieve distinctive collocates for Construction B (i.e., menguatkan)
dist_for_b <- dca_top_collex(dca_res, dist_for = "menguatkan", top_n = 10)
head(dist_for_b)
```
### Session info
```{r sessinfo}
devtools::session_info()
```
### References