Welcome to the AXOLOTL-24 Shared Task on Explainable Semantic Change Modeling! "AXOLOTL" stands for "Ascertain and eXplain Overhauls of the Lexicon Over Time at LChange'24".
Below, we explain the essence of two AXOLOTL subtasks in a small and comprehensible English example.
In the first subtask, you are given a set of target words, each accompanied by two sets of usages (a text fragment containing the word): from the older time period (A) and the more modern time period (B). In addition, a dictionary’s sense inventory with sense IDs for tim period A is given.
For instance, for the English target word "cell" in time period A, a sense inventory and a set of usage examples for each sense are given:
cell_1: A subordinate monastic establishment- "In Ireland there are innumerable chapels and cells and more than 400 churches"
- "Edward the Confessor established a cell of monks on the site of his old wooden church."
cell_2: An individual cell for a monk or nun in a monastic community- "The cells of the Celtic monks are preserved and cleaned"
- "In a Convent there are numerous little cells."
Fom the time period B only a set of usages is provided:
- "The monks are staying in their cells."
- "Gregor Mendel spent a good amount of time outside his cell."
- "An American company has applied to experiment in Britain on Parkinson's disease sufferers by injecting their brains with cells from pigs."
- "In multicellular organisms, groups of cells form tissues and tissues come together to form organs."
Comparing the usages from the two time periods, we can notice that the third and the fourth usages in the time period B correspond to a newly gained biological sense, which is absent in time period A. This sense is not covered by the provided sense inventory, therefore a system should assign these usages a new sense ID (for example, cell_3), while re-using the existing sense IDs for the rest of the usages:
| Usages from time period B | sense_id |
|---|---|
| "The monks are staying in their cells" | cell_2 |
| "Gregor Mendel spent a good amount of time outside his cell" | cell_2 |
| "An American company has applied to experiment in Britain on Parkinson's disease sufferers by injecting their brains with cells from pigs" | cell_3 |
| "In multicellular organisms, groups of cells form tissues and tissues come together to form organs" | cell_3 |
The performance of the system is evaluated by comparing its predictions with the gold data using Adjusted Rand Index (ARI) for all the usages from time period B and macro F1 score for all B usages with senses already existing in A.
See our evaluation scripts for more details.
The second subtask involves the creation of dictionary-like definitions for the novel (gained) senses of every target word. The definitions can be based on existing ontologies or generated from scratch: this is completely up to the participants.
If a participant came up with a solution for the Subtask 1, novel senses are inferred trivially from the Subtask 1 predictions (all the senses of the usages in time period B not attested in the A sense inventory). However, it is also possible to solve Subtask 2 independently, without Subtask 1. In the example below, we continue with the English word "cell", assuming that the participant already has a hypothetical solution for Subtask 1.
Since the usages of the predicted novel sense cell_3 are all referring to a constituent unit of a living organism, we may want to define this sense as, e.g., "A unit of a living organism". Then the submission for this target word will look like:
| Target word | sense_id | Definition |
|---|---|---|
| cell | cell_3 |
"A unit of a living organism" |
The performance of the system is evaluated with BLEU and BertScore, comparing the generated definitions to the gold ones from manually annotated data. The generated definitions are mapped to the gold ones by maximizing their BertScore similarity.
See our evaluation scripts for more details.