nlp-group-project-xlnet-base-cased.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nThis notebook provides a baseline for each setting in [Subtask A of SemEval 2022 Task 2](https://sites.google.com/view/semeval2022task2-idiomaticity#h.qq7eefmehqf9). In addition this provides some helpful pre-processing scripts that you are free to use with your experiments. \n\nPlease start by stepping through this notebook so you have a clear idea as to what is expected of the task and what you need to submit. \n\nThese baselines are based on the results described in the paper “[AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models](https://arxiv.org/abs/2109.04413)”. \n\n## Zero-shot setting: Methodology \n\nNote that in the zero-shot setting you are NOT allowed to train the model using the one-shot data. \n\nIn the zero-shot setting, we choose to include the context (the sentences preceding and succeeding the one containing the idioms). We do not add the idiom as an additional feature (in the “second input sentence”). This is based on the results presented in the dataset paper. \n\nWe use Multilingual BERT for this setting.\n\n## One-shot setting: Methodology\n\nIn the one shot setting, we train the model on both the zero-shot and one-shot data. In this setting, we exclude the context (the sentences preceding and succeeding the one containing the idioms) and also add the idiom as an additional feature in the “second sentence”. Again, this is based on the results presented in the dataset paper. \n\nWe also use Multilingual BERT for this setting.\n","metadata":{"id":"h9b_p85gxaGc"}},{"cell_type":"markdown","source":"# Setup ","metadata":{"id":"do-TXGBemGgH"}},{"cell_type":"markdown","source":"Download the Task data and evaluation scripts","metadata":{"id":"5WsITUAnzvFl"}},{"cell_type":"code","source":"!git clone https://github.com/H-TayyarMadabushi/SemEval_2022_Task2-idiomaticity.git","metadata":{"id":"Qq3qhQdpl-1-","outputId":"1859662b-7e59-4b6c-9a9d-540d3bf3eca1","execution":{"iopub.status.busy":"2021-12-11T19:57:12.550626Z","iopub.execute_input":"2021-12-11T19:57:12.551002Z","iopub.status.idle":"2021-12-11T19:57:14.540271Z","shell.execute_reply.started":"2021-12-11T19:57:12.550918Z","shell.execute_reply":"2021-12-11T19:57:14.539232Z"},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"Cloning into 'SemEval_2022_Task2-idiomaticity'...\nremote: Enumerating objects: 92, done.\u001b[K\nremote: Counting objects: 100% (92/92), done.\u001b[K\nremote: Compressing objects: 100% (75/75), done.\u001b[K\nremote: Total 92 (delta 31), reused 67 (delta 15), pack-reused 0\u001b[K\nUnpacking objects: 100% (92/92), 1.72 MiB | 3.90 MiB/s, done.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"Download the “AStitchInLanguageModels” code which we make use of. ","metadata":{"id":"u-0POB9tzfNx"}},{"cell_type":"code","source":"!git clone https://github.com/H-TayyarMadabushi/AStitchInLanguageModels.git","metadata":{"id":"affNQCRktdx4","outputId":"3aa27ecd-47dc-40f5-e3ee-621a1e2bfc18","execution":{"iopub.status.busy":"2021-12-11T19:57:14.542603Z","iopub.execute_input":"2021-12-11T19:57:14.542875Z","iopub.status.idle":"2021-12-11T19:57:19.910926Z","shell.execute_reply.started":"2021-12-11T19:57:14.542836Z","shell.execute_reply":"2021-12-11T19:57:19.910090Z"},"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"Cloning into 'AStitchInLanguageModels'...\nremote: Enumerating objects: 1030, done.\u001b[K\nremote: Counting objects: 100% (1030/1030), done.\u001b[K\nremote: Compressing objects: 100% (772/772), done.\u001b[K\nremote: Total 1030 (delta 382), reused 803 (delta 202), pack-reused 0\u001b[K\nReceiving objects: 100% (1030/1030), 79.86 MiB | 26.31 MiB/s, done.\nResolving deltas: 100% (382/382), done.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"Download and install an editable version of huggingfaces transformers. ","metadata":{"id":"60w-An2vzikk"}},{"cell_type":"code","source":"!git clone https://github.com/huggingface/transformers.git\n%cd transformers/\n!pip install --editable .","metadata":{"id":"m8BhcLYcmVvd","outputId":"fd067df6-8986-4464-8177-53d1f6459fc4","execution":{"iopub.status.busy":"2021-12-11T19:57:19.914350Z","iopub.execute_input":"2021-12-11T19:57:19.914626Z","iopub.status.idle":"2021-12-11T19:57:53.635030Z","shell.execute_reply.started":"2021-12-11T19:57:19.914597Z","shell.execute_reply":"2021-12-11T19:57:53.634115Z"},"trusted":true},"execution_count":3,"outputs":[{"name":"stdout","text":"Cloning into 'transformers'...\nremote: Enumerating objects: 91488, done.\u001b[K\nremote: Total 91488 (delta 0), reused 0 (delta 0), pack-reused 91488\u001b[K\nReceiving objects: 100% (91488/91488), 75.65 MiB | 26.74 MiB/s, done.\nResolving deltas: 100% (66021/66021), done.\n/kaggle/working/transformers\nObtaining file:///kaggle/working/transformers\n  Installing build dependencies ... \u001b[?25ldone\n\u001b[?25h  Checking if build backend supports build_editable ... \u001b[?25ldone\n\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (2021.11.10)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (4.62.3)\nRequirement already satisfied: tokenizers<0.11,>=0.10.1 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (0.10.3)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (1.19.5)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (6.0)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (3.3.2)\nRequirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (0.0.46)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (4.8.2)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (21.0)\nRequirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (0.1.2)\nRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers==4.14.0.dev0) (2.25.1)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers==4.14.0.dev0) (3.10.0.2)\nRequirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->transformers==4.14.0.dev0) (3.0.6)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers==4.14.0.dev0) (3.6.0)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.14.0.dev0) (2021.10.8)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.14.0.dev0) (1.26.7)\nRequirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.14.0.dev0) (2.10)\nRequirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.14.0.dev0) (4.0.0)\nRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers==4.14.0.dev0) (1.16.0)\nRequirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers==4.14.0.dev0) (1.1.0)\nRequirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers==4.14.0.dev0) (8.0.3)\nInstalling collected packages: transformers\n  Attempting uninstall: transformers\n    Found existing installation: transformers 4.12.5\n    Uninstalling transformers-4.12.5:\n      Successfully uninstalled transformers-4.12.5\n  Running setup.py develop for transformers\n\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\nallennlp 2.8.0 requires transformers<4.13,>=4.1, but you have transformers 4.14.0.dev0 which is incompatible.\u001b[0m\nSuccessfully installed transformers-4.14.0.dev0\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n[Errno 2] No such file or directory: '/content/'\n/kaggle/working/transformers\n","output_type":"stream"}]},{"cell_type":"markdown","source":"Required for run_glue ... ","metadata":{"id":"huVMnwTSzmjJ"}},{"cell_type":"code","source":"%cd /kaggle/working ","metadata":{"execution":{"iopub.status.busy":"2021-12-11T20:01:59.631258Z","iopub.execute_input":"2021-12-11T20:01:59.632179Z","iopub.status.idle":"2021-12-11T20:01:59.639033Z","shell.execute_reply.started":"2021-12-11T20:01:59.632139Z","shell.execute_reply":"2021-12-11T20:01:59.638086Z"},"trusted":true},"execution_count":17,"outputs":[{"name":"stdout","text":"/kaggle/working\n","output_type":"stream"}]},{"cell_type":"code","source":"## run_glue needs this. \n!pip install datasets","metadata":{"id":"2tsWits5tw1t","outputId":"953a00f1-9432-4286-b06d-8241f08dab9d","execution":{"iopub.status.busy":"2021-12-11T20:02:03.746513Z","iopub.execute_input":"2021-12-11T20:02:03.747029Z","iopub.status.idle":"2021-12-11T20:02:11.619773Z","shell.execute_reply.started":"2021-12-11T20:02:03.746983Z","shell.execute_reply":"2021-12-11T20:02:11.618932Z"},"trusted":true},"execution_count":18,"outputs":[{"name":"stdout","text":"Requirement already satisfied: datasets in /opt/conda/lib/python3.7/site-packages (1.16.1)\nRequirement already satisfied: multiprocess in /opt/conda/lib/python3.7/site-packages (from datasets) (0.70.12.2)\nRequirement already satisfied: xxhash in /opt/conda/lib/python3.7/site-packages (from datasets) (2.0.2)\nRequirement already satisfied: pyarrow!=4.0.0,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (6.0.0)\nRequirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from datasets) (3.8.1)\nRequirement already satisfied: dill in /opt/conda/lib/python3.7/site-packages (from datasets) (0.3.4)\nRequirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from datasets) (21.0)\nRequirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (2.25.1)\nRequirement already satisfied: huggingface-hub<1.0.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (0.1.2)\nRequirement already satisfied: fsspec[http]>=2021.05.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (2021.11.1)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from datasets) (1.19.5)\nRequirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from datasets) (1.3.4)\nRequirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.7/site-packages (from datasets) (4.62.3)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from datasets) (4.8.2)\nRequirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (6.0)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (3.10.0.2)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (3.3.2)\nRequirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->datasets) (3.0.6)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (2021.10.8)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (1.26.7)\nRequirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (4.0.0)\nRequirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (2.10)\nRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (1.7.2)\nRequirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (1.2.0)\nRequirement already satisfied: asynctest==0.13.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (0.13.0)\nRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (21.2.0)\nRequirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (2.0.7)\nRequirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (5.2.0)\nRequirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (1.2.0)\nRequirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets) (4.0.1)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->datasets) (3.6.0)\nRequirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets) (2.8.0)\nRequirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets) (2021.3)\nRequirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.16.0)\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n","output_type":"stream"}]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Editable install requires runtime restart unless we do this. ","metadata":{"id":"-igYdTTgzp9e"}},{"cell_type":"code","source":"import site\nsite.main()\n","metadata":{"id":"uOuKplBmmbeB","execution":{"iopub.status.busy":"2021-12-11T20:02:13.693958Z","iopub.execute_input":"2021-12-11T20:02:13.694203Z","iopub.status.idle":"2021-12-11T20:02:13.729532Z","shell.execute_reply.started":"2021-12-11T20:02:13.694176Z","shell.execute_reply":"2021-12-11T20:02:13.728877Z"},"trusted":true},"execution_count":24,"outputs":[]},{"cell_type":"markdown","source":"# Imports and Helper functions","metadata":{"id":"TvC8kAGNnKk_"}},{"cell_type":"code","source":"import os\nimport csv\n\nfrom pathlib import Path","metadata":{"id":"aOw3MaG7nN77","execution":{"iopub.status.busy":"2021-12-11T20:02:14.288920Z","iopub.execute_input":"2021-12-11T20:02:14.289607Z","iopub.status.idle":"2021-12-11T20:02:14.293872Z","shell.execute_reply.started":"2021-12-11T20:02:14.289564Z","shell.execute_reply":"2021-12-11T20:02:14.293119Z"},"trusted":true},"execution_count":25,"outputs":[]},{"cell_type":"code","source":"def load_csv( path, delimiter=',' ) : \n  header = None\n  data   = list()\n  with open( path, encoding='utf-8') as csvfile:\n    reader = csv.reader( csvfile, delimiter=delimiter ) \n    for row in reader : \n      if header is None : \n        header = row\n        continue\n      data.append( row ) \n  return header, data\n","metadata":{"id":"MzDtW9eXnOhG","execution":{"iopub.status.busy":"2021-12-11T20:02:15.720432Z","iopub.execute_input":"2021-12-11T20:02:15.721403Z","iopub.status.idle":"2021-12-11T20:02:15.728045Z","shell.execute_reply.started":"2021-12-11T20:02:15.721346Z","shell.execute_reply":"2021-12-11T20:02:15.727311Z"},"trusted":true},"execution_count":26,"outputs":[]},{"cell_type":"code","source":"def write_csv( data, location ) : \n  with open( location, 'w', encoding='utf-8') as csvfile:\n    writer = csv.writer( csvfile ) \n    writer.writerows( data ) \n  print( \"Wrote {}\".format( location ) ) \n  return\n","metadata":{"id":"WwtDsdtAnSZu","execution":{"iopub.status.busy":"2021-12-11T20:02:16.003988Z","iopub.execute_input":"2021-12-11T20:02:16.004680Z","iopub.status.idle":"2021-12-11T20:02:16.010177Z","shell.execute_reply.started":"2021-12-11T20:02:16.004639Z","shell.execute_reply":"2021-12-11T20:02:16.009070Z"},"trusted":true},"execution_count":27,"outputs":[]},{"cell_type":"markdown","source":"The following function creates a submission file from the predictions output by run_glue (the text classification script from huggingface transformers - see below). \n\nNote that we set it up so we can load up results for only one setting. \n\nIt requires as input the submission format file, which is available with the data. You can call this after completing each setting to load up results for both settings (see below).\n","metadata":{"id":"p9Io3D3_z4wt"}},{"cell_type":"code","source":"def insert_to_submission_file( submission_format_file, input_file, prediction_format_file, setting ) :\n    submission_header, submission_content = load_csv( submission_format_file )\n    input_header     , input_data         = load_csv( input_file             )\n    prediction_header, prediction_data    = load_csv( prediction_format_file, '\\t' )\n\n    assert len( input_data ) == len( prediction_data )\n\n    ## submission_header ['ID', 'Language', 'Setting', 'Label']\n    ## input_header      ['label', 'sentence1' ]\n    ## prediction_header ['index', 'prediction']\n\n    prediction_data = list( reversed( prediction_data ) )\n\n    started_insert  = False\n    for elem in submission_content : \n        if elem[ submission_header.index( 'Setting' ) ] != setting :\n            if started_insert :\n                if len( prediction_data ) == 0 :\n                    break\n                else : \n                    raise Exception( \"Update should to contiguous ... something wrong.\" ) \n            continue\n        started_insert = True\n        elem[ submission_header.index( 'Label' ) ] = prediction_data.pop()[ prediction_header.index( 'prediction' ) ]\n\n    return [ submission_header ] + submission_content","metadata":{"id":"Re31vnLoQWww","execution":{"iopub.status.busy":"2021-12-11T20:02:18.719357Z","iopub.execute_input":"2021-12-11T20:02:18.719898Z","iopub.status.idle":"2021-12-11T20:02:18.726673Z","shell.execute_reply.started":"2021-12-11T20:02:18.719861Z","shell.execute_reply":"2021-12-11T20:02:18.725996Z"},"trusted":true},"execution_count":28,"outputs":[]},{"cell_type":"markdown","source":"# Pre-process: Create train and dev and evaluation data in required format\n\nIn the zero-shot setting, we choose to include the context (the sentences preceding and succeeding the one containing the idioms). We do not add the idiom as an additional feature (in the “second input sentence”). \n\nIn the one shot setting, we train the model on both the zero-shot and one-shot data. In this setting, we exclude the context (the sentences preceding and succeeding the one containing the idioms) and also add the idiom as an additional feature in the “second sentence”. \n","metadata":{"id":"44LyZ-OXmgQW"}},{"cell_type":"markdown","source":"## Functions for pre-processing","metadata":{"id":"8-3ymBcEmxaV"}},{"cell_type":"markdown","source":"### _get_train_data\n\nThis function generates training data in the format required by the huggingface’s example script. It will include and exclude the MWE and the context based on parameters. \n","metadata":{"id":"MthVK7EQm6m_"}},{"cell_type":"code","source":"def _get_train_data( data_location, file_name, include_context, include_idiom ) :\n    \n    file_name = os.path.join( data_location, file_name ) \n\n    header, data = load_csv( file_name )\n\n    out_header = [ 'label', 'sentence1' ]\n    if include_idiom :\n        out_header = [ 'label', 'sentence1', 'sentence2' ]\n        \n    # ['DataID', 'Language', 'MWE', 'Setting', 'Previous', 'Target', 'Next', 'Label']\n    out_data = list()\n    for elem in data :\n        label     = elem[ header.index( 'Label'  ) ]\n        sentence1 = elem[ header.index( 'Target' ) ]\n        if include_context :\n            sentence1 = ' '.join( [ elem[ header.index( 'Previous' ) ], elem[ header.index( 'Target' ) ], elem[ header.index( 'Next' ) ] ] )\n        this_row = None\n        if not include_idiom :\n            this_row = [ label, sentence1 ] \n        else :\n            sentence2 = elem[ header.index( 'MWE' ) ]\n            this_row = [ label, sentence1, sentence2 ]\n        out_data.append( this_row )\n        assert len( out_header ) == len( this_row )\n    return [ out_header ] + out_data","metadata":{"id":"cPGq-Y1Jmvv5","execution":{"iopub.status.busy":"2021-12-11T20:02:22.773057Z","iopub.execute_input":"2021-12-11T20:02:22.773913Z","iopub.status.idle":"2021-12-11T20:02:22.783168Z","shell.execute_reply.started":"2021-12-11T20:02:22.773867Z","shell.execute_reply":"2021-12-11T20:02:22.782310Z"},"trusted":true},"execution_count":29,"outputs":[]},{"cell_type":"markdown","source":"### _get_dev_eval_data\n\nThis function generates training dev and eval data in the format required by the huggingface’s example script. It will include and exclude the MWE and the context based on parameters. \n\nAdditionally, if there is no gold label provides (as in the case of eval) it will generate a file that can be used to generate predictions.\n","metadata":{"id":"cytociCB3WZM"}},{"cell_type":"code","source":"def _get_dev_eval_data( data_location, input_file_name, gold_file_name, include_context, include_idiom ) :\n\n    input_headers, input_data = load_csv( os.path.join( data_location, input_file_name ) )\n    gold_header  = gold_data = None\n    if not gold_file_name is None : \n        gold_header  , gold_data  = load_csv( os.path.join( data_location, gold_file_name  ) )\n        assert len( input_data ) == len( gold_data )\n\n    # ['ID', 'Language', 'MWE', 'Previous', 'Target', 'Next']\n    # ['ID', 'DataID', 'Language', 'Label']\n    \n    out_header = [ 'label', 'sentence1' ]\n    if include_idiom :\n        out_header = [ 'label', 'sentence1', 'sentence2' ]\n\n    out_data = list()\n    for index in range( len( input_data ) ) :\n        label = 1\n        if not gold_file_name is None : \n            this_input_id = input_data[ index ][ input_headers.index( 'ID' ) ]\n            this_gold_id  = gold_data [ index ][ gold_header  .index( 'ID' ) ]\n            assert this_input_id == this_gold_id\n            \n            label     = gold_data[ index ][ gold_header.index( 'Label'  ) ]\n            \n        elem      = input_data[ index ]\n        sentence1 = elem[ input_headers.index( 'Target' ) ]\n        if include_context :\n            sentence1 = ' '.join( [ elem[ input_headers.index( 'Previous' ) ], elem[ input_headers.index( 'Target' ) ], elem[ input_headers.index( 'Next' ) ] ] )\n        this_row = None\n        if not include_idiom :\n            this_row = [ label, sentence1 ] \n        else :\n            sentence2 = elem[ input_headers.index( 'MWE' ) ]\n            this_row = [ label, sentence1, sentence2 ]\n        assert len( out_header ) == len( this_row ) \n        out_data.append( this_row )\n        \n\n    return [ out_header ] + out_data\n","metadata":{"id":"qe4YQJ9Sm-B2","execution":{"iopub.status.busy":"2021-12-11T20:02:23.152312Z","iopub.execute_input":"2021-12-11T20:02:23.153011Z","iopub.status.idle":"2021-12-11T20:02:23.163436Z","shell.execute_reply.started":"2021-12-11T20:02:23.152973Z","shell.execute_reply":"2021-12-11T20:02:23.162698Z"},"trusted":true},"execution_count":30,"outputs":[]},{"cell_type":"markdown","source":"### create_data\n\nThis function generates the training, development and evaluation data. \n","metadata":{"id":"DjIbyTnn3fHP"}},{"cell_type":"code","source":"\"\"\"\nBased on the results presented in `AStitchInLanguageModels' we work with not including the idiom for the zero shot setting and including it in the one shot setting.\n\"\"\"\ndef create_data( input_location, output_location ) :\n\n    \n    ## Zero shot data\n    train_data = _get_train_data(\n        data_location   = input_location,\n        file_name       = 'train_zero_shot.csv',\n        include_context = False,\n        include_idiom   = True\n    )\n    write_csv( train_data, os.path.join( output_location, 'ZeroShot', 'train.csv' ) )\n    \n    dev_data = _get_dev_eval_data(\n        data_location    = input_location,\n        input_file_name  = 'dev.csv',\n        gold_file_name   = 'dev_gold.csv', \n        include_context  = False,\n        include_idiom    = True\n    )        \n    write_csv( dev_data, os.path.join( output_location, 'ZeroShot', 'dev.csv' ) )\n    \n    eval_data = _get_dev_eval_data(\n        data_location    = input_location,\n        input_file_name  = 'eval.csv',\n        gold_file_name   = None , ## Don't have gold evaluation file -- submit to CodaLab\n        include_context  = True,\n        include_idiom    = False\n    )\n    write_csv( eval_data, os.path.join( output_location, 'ZeroShot', 'eval.csv' ) )\n\n\n    ## OneShot Data (combine both for training)\n    train_zero_data = _get_train_data(\n        data_location   = input_location,\n        file_name       = 'train_zero_shot.csv',\n        include_context = False,\n        include_idiom   = True\n    )\n    train_one_data = _get_train_data(\n        data_location   = input_location,\n        file_name       = 'train_one_shot.csv',\n        include_context = False,\n        include_idiom   = True\n    )\n\n    assert train_zero_data[0] == train_one_data[0] ## Headers\n    train_data = train_one_data + train_zero_data[1:]\n    write_csv( train_data, os.path.join( output_location, 'OneShot', 'train.csv' ) )\n    \n    dev_data = _get_dev_eval_data(\n        data_location    = input_location,\n        input_file_name  = 'dev.csv',\n        gold_file_name   = 'dev_gold.csv', \n        include_context  = False,\n        include_idiom    = True\n    )        \n    write_csv( dev_data, os.path.join( output_location, 'OneShot', 'dev.csv' ) )\n    \n    eval_data = _get_dev_eval_data(\n        data_location    = input_location,\n        input_file_name  = 'eval.csv',\n        gold_file_name   = None,\n        include_context  = False,\n        include_idiom    = True\n    )\n    write_csv( eval_data, os.path.join( output_location, 'OneShot', 'eval.csv' ) )\n\n    return","metadata":{"id":"w1tr-zNvnBCV","execution":{"iopub.status.busy":"2021-12-11T20:17:10.868609Z","iopub.execute_input":"2021-12-11T20:17:10.869323Z","iopub.status.idle":"2021-12-11T20:17:10.883600Z","shell.execute_reply.started":"2021-12-11T20:17:10.869283Z","shell.execute_reply":"2021-12-11T20:17:10.882234Z"},"trusted":true},"execution_count":40,"outputs":[]},{"cell_type":"markdown","source":"## Setup and Create data","metadata":{"id":"mmQfvym8ndKH"}},{"cell_type":"code","source":"!pwd ","metadata":{"id":"mxCgaHlKnpMR","outputId":"f2368d63-bbea-4d91-f43f-dc57e001288e","execution":{"iopub.status.busy":"2021-12-11T20:17:13.752214Z","iopub.execute_input":"2021-12-11T20:17:13.752930Z","iopub.status.idle":"2021-12-11T20:17:14.465613Z","shell.execute_reply.started":"2021-12-11T20:17:13.752892Z","shell.execute_reply":"2021-12-11T20:17:14.464670Z"},"trusted":true},"execution_count":41,"outputs":[{"name":"stdout","text":"/kaggle/working\n","output_type":"stream"}]},{"cell_type":"code","source":"outpath = 'Data'\n    \nPath( os.path.join( outpath, 'ZeroShot' ) ).mkdir(parents=True, exist_ok=True)\nPath( os.path.join( outpath, 'OneShot' ) ).mkdir(parents=True, exist_ok=True)\n\ncreate_data( 'SemEval_2022_Task2-idiomaticity/SubTaskA/Data/', outpath )","metadata":{"id":"pkeKLg-Hngs4","outputId":"6914517a-85c7-4bf6-d29b-82b52c8f4a6b","execution":{"iopub.status.busy":"2021-12-11T20:17:15.355603Z","iopub.execute_input":"2021-12-11T20:17:15.358353Z","iopub.status.idle":"2021-12-11T20:17:16.948527Z","shell.execute_reply.started":"2021-12-11T20:17:15.358302Z","shell.execute_reply":"2021-12-11T20:17:16.947801Z"},"trusted":true},"execution_count":42,"outputs":[{"name":"stdout","text":"Wrote Data/ZeroShot/train.csv\nWrote Data/ZeroShot/dev.csv\nWrote Data/ZeroShot/eval.csv\nWrote Data/OneShot/train.csv\nWrote Data/OneShot/dev.csv\nWrote Data/OneShot/eval.csv\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# Zero Shot Setting","metadata":{"id":"uP-Ol7hfoC8a"}},{"cell_type":"markdown","source":"## Train Zero shot","metadata":{"id":"o-GiQvnkoL67"}},{"cell_type":"code","source":"import os\nos.environ[\"WANDB_DISABLED\"] = \"true\"\n!python3 /kaggle/working/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path 'xlnet-base-cased' \\\n    \t--do_train \\\n    \t--do_eval \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/ZeroShot/0/ \\\n    \t--seed 0 \\\n    \t--train_file      Data/ZeroShot/train.csv \\\n    \t--validation_file Data/ZeroShot/dev.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n        --overwrite_output_dir \\\n\t    --save_total_limit 1","metadata":{"id":"_k0BwA0uoKAu","execution":{"iopub.status.busy":"2021-12-11T20:17:53.966237Z","iopub.execute_input":"2021-12-11T20:17:53.966907Z","iopub.status.idle":"2021-12-11T20:31:22.541851Z","shell.execute_reply.started":"2021-12-11T20:17:53.966870Z","shell.execute_reply":"2021-12-11T20:31:22.540924Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":43,"outputs":[{"name":"stdout","text":"Using the `WAND_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).\nDownloading and preparing dataset csv/default to /root/.cache/huggingface/datasets/csv/default-12e71bc13f70c5e4/0.0.0/bf68a4c4aefa545d0712b2fcbb1b327f905bbe2f6425fbc5e8c25234acb9e14a...\n100%|███████████████████████████████████████████| 2/2 [00:00<00:00, 9198.04it/s]\n100%|███████████████████████████████████████████| 2/2 [00:00<00:00, 1489.98it/s]\nDataset csv downloaded and prepared to /root/.cache/huggingface/datasets/csv/default-12e71bc13f70c5e4/0.0.0/bf68a4c4aefa545d0712b2fcbb1b327f905bbe2f6425fbc5e8c25234acb9e14a. Subsequent calls will reuse this data.\n100%|████████████████████████████████████████████| 2/2 [00:00<00:00, 609.73it/s]\n[INFO|file_utils.py:1887] 2021-12-11 20:18:01,115 >> https://huggingface.co/xlnet-base-cased/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpi_96fzpi\nDownloading: 100%|██████████████████████████████| 760/760 [00:00<00:00, 513kB/s]\n[INFO|file_utils.py:1891] 2021-12-11 20:18:01,455 >> storing https://huggingface.co/xlnet-base-cased/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/06bdb0f5882dbb833618c81c3b4c996a0c79422fa2c95ffea3827f92fc2dba6b.da982e2e596ec73828dbae86525a1870e513bd63aae5a2dc773ccc840ac5c346\n[INFO|file_utils.py:1899] 2021-12-11 20:18:01,455 >> creating metadata file for /root/.cache/huggingface/transformers/06bdb0f5882dbb833618c81c3b4c996a0c79422fa2c95ffea3827f92fc2dba6b.da982e2e596ec73828dbae86525a1870e513bd63aae5a2dc773ccc840ac5c346\n[INFO|configuration_utils.py:604] 2021-12-11 20:18:01,455 >> loading configuration file https://huggingface.co/xlnet-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/06bdb0f5882dbb833618c81c3b4c996a0c79422fa2c95ffea3827f92fc2dba6b.da982e2e596ec73828dbae86525a1870e513bd63aae5a2dc773ccc840ac5c346\n[INFO|configuration_utils.py:641] 2021-12-11 20:18:01,459 >> Model config XLNetConfig {\n  \"_name_or_path\": \"xlnet-base-cased\",\n  \"architectures\": [\n    \"XLNetLMHeadModel\"\n  ],\n  \"attn_type\": \"bi\",\n  \"bi_data\": false,\n  \"bos_token_id\": 1,\n  \"clamp_len\": -1,\n  \"d_head\": 64,\n  \"d_inner\": 3072,\n  \"d_model\": 768,\n  \"dropout\": 0.1,\n  \"end_n_top\": 5,\n  \"eos_token_id\": 2,\n  \"ff_activation\": \"gelu\",\n  \"initializer_range\": 0.02,\n  \"layer_norm_eps\": 1e-12,\n  \"mem_len\": null,\n  \"model_type\": \"xlnet\",\n  \"n_head\": 12,\n  \"n_layer\": 12,\n  \"pad_token_id\": 5,\n  \"reuse_len\": null,\n  \"same_length\": false,\n  \"start_n_top\": 5,\n  \"summary_activation\": \"tanh\",\n  \"summary_last_dropout\": 0.1,\n  \"summary_type\": \"last\",\n  \"summary_use_proj\": true,\n  \"task_specific_params\": {\n    \"text-generation\": {\n      \"do_sample\": true,\n      \"max_length\": 250\n    }\n  },\n  \"transformers_version\": \"4.14.0.dev0\",\n  \"untie_r\": true,\n  \"use_mems_eval\": true,\n  \"use_mems_train\": false,\n  \"vocab_size\": 32000\n}\n\n[INFO|tokenization_auto.py:352] 2021-12-11 20:18:01,794 >> Could not locate the tokenizer configuration file, will try to use the model config instead.\n[INFO|configuration_utils.py:604] 2021-12-11 20:18:02,486 >> loading configuration file https://huggingface.co/xlnet-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/06bdb0f5882dbb833618c81c3b4c996a0c79422fa2c95ffea3827f92fc2dba6b.da982e2e596ec73828dbae86525a1870e513bd63aae5a2dc773ccc840ac5c346\n[INFO|configuration_utils.py:641] 2021-12-11 20:18:02,487 >> Model config XLNetConfig {\n  \"_name_or_path\": \"xlnet-base-cased\",\n  \"architectures\": [\n    \"XLNetLMHeadModel\"\n  ],\n  \"attn_type\": \"bi\",\n  \"bi_data\": false,\n  \"bos_token_id\": 1,\n  \"clamp_len\": -1,\n  \"d_head\": 64,\n  \"d_inner\": 3072,\n  \"d_model\": 768,\n  \"dropout\": 0.1,\n  \"end_n_top\": 5,\n  \"eos_token_id\": 2,\n  \"ff_activation\": \"gelu\",\n  \"initializer_range\": 0.02,\n  \"layer_norm_eps\": 1e-12,\n  \"mem_len\": null,\n  \"model_type\": \"xlnet\",\n  \"n_head\": 12,\n  \"n_layer\": 12,\n  \"pad_token_id\": 5,\n  \"reuse_len\": null,\n  \"same_length\": false,\n  \"start_n_top\": 5,\n  \"summary_activation\": \"tanh\",\n  \"summary_last_dropout\": 0.1,\n  \"summary_type\": \"last\",\n  \"summary_use_proj\": true,\n  \"task_specific_params\": {\n    \"text-generation\": {\n      \"do_sample\": true,\n      \"max_length\": 250\n    }\n  },\n  \"transformers_version\": \"4.14.0.dev0\",\n  \"untie_r\": true,\n  \"use_mems_eval\": true,\n  \"use_mems_train\": false,\n  \"vocab_size\": 32000\n}\n\n[INFO|file_utils.py:1887] 2021-12-11 20:18:03,168 >> https://huggingface.co/xlnet-base-cased/resolve/main/spiece.model not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpx86oizrx\nDownloading: 100%|███████████████████████████| 779k/779k [00:00<00:00, 2.49MB/s]\n[INFO|file_utils.py:1891] 2021-12-11 20:18:03,834 >> storing https://huggingface.co/xlnet-base-cased/resolve/main/spiece.model in cache at /root/.cache/huggingface/transformers/df73bc9f8d13bf2ea4dab95624895e45a550a0f0a825e41fc25440bf367ee3c8.d93497120e3a865e2970f26abdf7bf375896f97fde8b874b70909592a6c785c9\n[INFO|file_utils.py:1899] 2021-12-11 20:18:03,834 >> creating metadata file for /root/.cache/huggingface/transformers/df73bc9f8d13bf2ea4dab95624895e45a550a0f0a825e41fc25440bf367ee3c8.d93497120e3a865e2970f26abdf7bf375896f97fde8b874b70909592a6c785c9\n[INFO|file_utils.py:1887] 2021-12-11 20:18:04,174 >> https://huggingface.co/xlnet-base-cased/resolve/main/tokenizer.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp7efrjn65\nDownloading: 100%|█████████████████████████| 1.32M/1.32M [00:00<00:00, 3.53MB/s]\n[INFO|file_utils.py:1891] 2021-12-11 20:18:04,919 >> storing https://huggingface.co/xlnet-base-cased/resolve/main/tokenizer.json in cache at /root/.cache/huggingface/transformers/46f47734f3dcaef7e236b9a3e887f27814e18836a8db7e6a49148000058a1a54.2a683f915238b4f560dab0c724066cf0a7de9a851e96b0fb3a1e7f0881552f53\n[INFO|file_utils.py:1899] 2021-12-11 20:18:04,919 >> creating metadata file for /root/.cache/huggingface/transformers/46f47734f3dcaef7e236b9a3e887f27814e18836a8db7e6a49148000058a1a54.2a683f915238b4f560dab0c724066cf0a7de9a851e96b0fb3a1e7f0881552f53\n[INFO|tokenization_utils_base.py:1742] 2021-12-11 20:18:05,938 >> loading file https://huggingface.co/xlnet-base-cased/resolve/main/spiece.model from cache at /root/.cache/huggingface/transformers/df73bc9f8d13bf2ea4dab95624895e45a550a0f0a825e41fc25440bf367ee3c8.d93497120e3a865e2970f26abdf7bf375896f97fde8b874b70909592a6c785c9\n[INFO|tokenization_utils_base.py:1742] 2021-12-11 20:18:05,938 >> loading file https://huggingface.co/xlnet-base-cased/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/46f47734f3dcaef7e236b9a3e887f27814e18836a8db7e6a49148000058a1a54.2a683f915238b4f560dab0c724066cf0a7de9a851e96b0fb3a1e7f0881552f53\n[INFO|tokenization_utils_base.py:1742] 2021-12-11 20:18:05,938 >> loading file https://huggingface.co/xlnet-base-cased/resolve/main/added_tokens.json from cache at None\n[INFO|tokenization_utils_base.py:1742] 2021-12-11 20:18:05,938 >> loading file https://huggingface.co/xlnet-base-cased/resolve/main/special_tokens_map.json from cache at None\n[INFO|tokenization_utils_base.py:1742] 2021-12-11 20:18:05,938 >> loading file https://huggingface.co/xlnet-base-cased/resolve/main/tokenizer_config.json from cache at None\n[INFO|configuration_utils.py:604] 2021-12-11 20:18:06,615 >> loading configuration file https://huggingface.co/xlnet-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/06bdb0f5882dbb833618c81c3b4c996a0c79422fa2c95ffea3827f92fc2dba6b.da982e2e596ec73828dbae86525a1870e513bd63aae5a2dc773ccc840ac5c346\n[INFO|configuration_utils.py:641] 2021-12-11 20:18:06,616 >> Model config XLNetConfig {\n  \"_name_or_path\": \"xlnet-base-cased\",\n  \"architectures\": [\n    \"XLNetLMHeadModel\"\n  ],\n  \"attn_type\": \"bi\",\n  \"bi_data\": false,\n  \"bos_token_id\": 1,\n  \"clamp_len\": -1,\n  \"d_head\": 64,\n  \"d_inner\": 3072,\n  \"d_model\": 768,\n  \"dropout\": 0.1,\n  \"end_n_top\": 5,\n  \"eos_token_id\": 2,\n  \"ff_activation\": \"gelu\",\n  \"initializer_range\": 0.02,\n  \"layer_norm_eps\": 1e-12,\n  \"mem_len\": null,\n  \"model_type\": \"xlnet\",\n  \"n_head\": 12,\n  \"n_layer\": 12,\n  \"pad_token_id\": 5,\n  \"reuse_len\": null,\n  \"same_length\": false,\n  \"start_n_top\": 5,\n  \"summary_activation\": \"tanh\",\n  \"summary_last_dropout\": 0.1,\n  \"summary_type\": \"last\",\n  \"summary_use_proj\": true,\n  \"task_specific_params\": {\n    \"text-generation\": {\n      \"do_sample\": true,\n      \"max_length\": 250\n    }\n  },\n  \"transformers_version\": \"4.14.0.dev0\",\n  \"untie_r\": true,\n  \"use_mems_eval\": true,\n  \"use_mems_train\": false,\n  \"vocab_size\": 32000\n}\n\n[INFO|file_utils.py:1887] 2021-12-11 20:18:07,058 >> https://huggingface.co/xlnet-base-cased/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpje65utvk\nDownloading: 100%|███████████████████████████| 445M/445M [00:13<00:00, 35.9MB/s]\n[INFO|file_utils.py:1891] 2021-12-11 20:18:20,362 >> storing https://huggingface.co/xlnet-base-cased/resolve/main/pytorch_model.bin in cache at /root/.cache/huggingface/transformers/9461853998373b0b2f8ef8011a13b62a2c5f540b2c535ef3ea46ed8a062b16a9.3e214f11a50e9e03eb47535b58522fc3cc11ac67c120a9450f6276de151af987\n[INFO|file_utils.py:1899] 2021-12-11 20:18:20,362 >> creating metadata file for /root/.cache/huggingface/transformers/9461853998373b0b2f8ef8011a13b62a2c5f540b2c535ef3ea46ed8a062b16a9.3e214f11a50e9e03eb47535b58522fc3cc11ac67c120a9450f6276de151af987\n[INFO|modeling_utils.py:1352] 2021-12-11 20:18:20,362 >> loading weights file https://huggingface.co/xlnet-base-cased/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/9461853998373b0b2f8ef8011a13b62a2c5f540b2c535ef3ea46ed8a062b16a9.3e214f11a50e9e03eb47535b58522fc3cc11ac67c120a9450f6276de151af987\n[WARNING|modeling_utils.py:1611] 2021-12-11 20:18:21,933 >> Some weights of the model checkpoint at xlnet-base-cased were not used when initializing XLNetForSequenceClassification: ['lm_loss.bias', 'lm_loss.weight']\n- This IS expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n[WARNING|modeling_utils.py:1622] 2021-12-11 20:18:21,933 >> Some weights of XLNetForSequenceClassification were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: ['sequence_summary.summary.bias', 'sequence_summary.summary.weight', 'logits_proj.bias', 'logits_proj.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n100%|█████████████████████████████████████████████| 5/5 [00:01<00:00,  3.97ba/s]\n100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  6.48ba/s]\n[INFO|trainer.py:549] 2021-12-11 20:18:27,026 >> The following columns in the training set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:1204] 2021-12-11 20:18:27,033 >> ***** Running training *****\n[INFO|trainer.py:1205] 2021-12-11 20:18:27,033 >>   Num examples = 4491\n[INFO|trainer.py:1206] 2021-12-11 20:18:27,033 >>   Num Epochs = 9\n[INFO|trainer.py:1207] 2021-12-11 20:18:27,034 >>   Instantaneous batch size per device = 32\n[INFO|trainer.py:1208] 2021-12-11 20:18:27,034 >>   Total train batch size (w. parallel, distributed & accumulation) = 32\n[INFO|trainer.py:1209] 2021-12-11 20:18:27,034 >>   Gradient Accumulation steps = 1\n[INFO|trainer.py:1210] 2021-12-11 20:18:27,034 >>   Total optimization steps = 1269\n 11%|████▍                                   | 141/1269 [01:16<08:18,  2.26it/s][INFO|trainer.py:549] 2021-12-11 20:19:43,705 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:19:43,707 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:19:43,707 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:19:43,707 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.68it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 23.00it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.28it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.69it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.41it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.27it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 20.18it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 19.97it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 19.79it/s]\u001b[A\n 31%|█████████████▍                             | 29/93 [00:01<00:03, 19.70it/s]\u001b[A\n 33%|██████████████▎                            | 31/93 [00:01<00:03, 19.69it/s]\u001b[A\n 35%|███████████████▎                           | 33/93 [00:01<00:03, 19.66it/s]\u001b[A\n 38%|████████████████▏                          | 35/93 [00:01<00:02, 19.61it/s]\u001b[A\n 40%|█████████████████                          | 37/93 [00:01<00:02, 19.61it/s]\u001b[A\n 42%|██████████████████                         | 39/93 [00:01<00:02, 19.63it/s]\u001b[A\n 44%|██████████████████▉                        | 41/93 [00:02<00:02, 19.69it/s]\u001b[A\n 46%|███████████████████▉                       | 43/93 [00:02<00:02, 19.76it/s]\u001b[A\n 48%|████████████████████▊                      | 45/93 [00:02<00:02, 19.80it/s]\u001b[A\n 51%|█████████████████████▋                     | 47/93 [00:02<00:02, 19.69it/s]\u001b[A\n 53%|██████████████████████▋                    | 49/93 [00:02<00:02, 19.64it/s]\u001b[A\n 55%|███████████████████████▌                   | 51/93 [00:02<00:02, 19.65it/s]\u001b[A\n 57%|████████████████████████▌                  | 53/93 [00:02<00:02, 19.61it/s]\u001b[A\n 59%|█████████████████████████▍                 | 55/93 [00:02<00:01, 19.62it/s]\u001b[A\n 61%|██████████████████████████▎                | 57/93 [00:02<00:01, 19.67it/s]\u001b[A\n 63%|███████████████████████████▎               | 59/93 [00:02<00:01, 19.73it/s]\u001b[A\n 66%|████████████████████████████▏              | 61/93 [00:03<00:01, 19.80it/s]\u001b[A\n 68%|█████████████████████████████▏             | 63/93 [00:03<00:01, 19.86it/s]\u001b[A\n 70%|██████████████████████████████             | 65/93 [00:03<00:01, 19.83it/s]\u001b[A\n 72%|██████████████████████████████▉            | 67/93 [00:03<00:01, 19.70it/s]\u001b[A\n 74%|███████████████████████████████▉           | 69/93 [00:03<00:01, 19.65it/s]\u001b[A\n 76%|████████████████████████████████▊          | 71/93 [00:03<00:01, 19.66it/s]\u001b[A\n 78%|█████████████████████████████████▊         | 73/93 [00:03<00:01, 19.66it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.66it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.72it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:03<00:00, 19.78it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.76it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.74it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.70it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.55it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.57it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 0.9787214398384094, 'eval_accuracy': 0.7388362884521484, 'eval_f1': 0.7387673839740503, 'eval_runtime': 4.6969, 'eval_samples_per_second': 157.337, 'eval_steps_per_second': 19.8, 'epoch': 1.0}\n 11%|████▍                                   | 141/1269 [01:21<08:18,  2.26it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.61it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:19:48,406 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-141\n[INFO|configuration_utils.py:425] 2021-12-11 20:19:48,407 >> Configuration saved in models/ZeroShot/0/checkpoint-141/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:19:49,235 >> Model weights saved in models/ZeroShot/0/checkpoint-141/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:19:49,235 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-141/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:19:49,235 >> Special tokens file saved in models/ZeroShot/0/checkpoint-141/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:19:51,731 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-846] due to args.save_total_limit\n[INFO|trainer.py:2111] 2021-12-11 20:19:51,887 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-1269] due to args.save_total_limit\n 22%|████████▉                               | 282/1269 [02:41<07:21,  2.23it/s][INFO|trainer.py:549] 2021-12-11 20:21:08,683 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:21:08,685 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:21:08,685 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:21:08,685 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.88it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 23.05it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.31it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.66it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.37it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.23it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 20.12it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 20.01it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 19.95it/s]\u001b[A\n 32%|█████████████▊                             | 30/93 [00:01<00:03, 19.85it/s]\u001b[A\n 34%|██████████████▊                            | 32/93 [00:01<00:03, 19.87it/s]\u001b[A\n 37%|███████████████▋                           | 34/93 [00:01<00:02, 19.88it/s]\u001b[A\n 39%|████████████████▋                          | 36/93 [00:01<00:02, 19.91it/s]\u001b[A\n 42%|██████████████████                         | 39/93 [00:01<00:02, 19.94it/s]\u001b[A\n 44%|██████████████████▉                        | 41/93 [00:02<00:02, 19.94it/s]\u001b[A\n 46%|███████████████████▉                       | 43/93 [00:02<00:02, 19.92it/s]\u001b[A\n 48%|████████████████████▊                      | 45/93 [00:02<00:02, 19.88it/s]\u001b[A\n 51%|█████████████████████▋                     | 47/93 [00:02<00:02, 19.78it/s]\u001b[A\n 53%|██████████████████████▋                    | 49/93 [00:02<00:02, 19.72it/s]\u001b[A\n 55%|███████████████████████▌                   | 51/93 [00:02<00:02, 19.72it/s]\u001b[A\n 57%|████████████████████████▌                  | 53/93 [00:02<00:02, 19.78it/s]\u001b[A\n 59%|█████████████████████████▍                 | 55/93 [00:02<00:01, 19.81it/s]\u001b[A\n 61%|██████████████████████████▎                | 57/93 [00:02<00:01, 19.84it/s]\u001b[A\n 63%|███████████████████████████▎               | 59/93 [00:02<00:01, 19.87it/s]\u001b[A\n 66%|████████████████████████████▏              | 61/93 [00:03<00:01, 19.89it/s]\u001b[A\n 68%|█████████████████████████████▏             | 63/93 [00:03<00:01, 19.90it/s]\u001b[A\n 70%|██████████████████████████████             | 65/93 [00:03<00:01, 19.87it/s]\u001b[A\n 72%|██████████████████████████████▉            | 67/93 [00:03<00:01, 19.81it/s]\u001b[A\n 74%|███████████████████████████████▉           | 69/93 [00:03<00:01, 19.76it/s]\u001b[A\n 76%|████████████████████████████████▊          | 71/93 [00:03<00:01, 19.82it/s]\u001b[A\n 78%|█████████████████████████████████▊         | 73/93 [00:03<00:01, 19.85it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.88it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.89it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:03<00:00, 19.90it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.92it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.91it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.88it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.70it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.65it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 1.0771437883377075, 'eval_accuracy': 0.7104194760322571, 'eval_f1': 0.7099728579812207, 'eval_runtime': 4.6677, 'eval_samples_per_second': 158.321, 'eval_steps_per_second': 19.924, 'epoch': 2.0}\n 22%|████████▉                               | 282/1269 [02:46<07:21,  2.23it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.71it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:21:13,355 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-282\n[INFO|configuration_utils.py:425] 2021-12-11 20:21:13,356 >> Configuration saved in models/ZeroShot/0/checkpoint-282/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:21:14,160 >> Model weights saved in models/ZeroShot/0/checkpoint-282/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:21:14,161 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-282/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:21:14,161 >> Special tokens file saved in models/ZeroShot/0/checkpoint-282/special_tokens_map.json\n 33%|█████████████▎                          | 423/1269 [04:05<06:13,  2.26it/s][INFO|trainer.py:549] 2021-12-11 20:22:32,539 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:22:32,542 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:22:32,542 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:22:32,542 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.32it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.70it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.27it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.58it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.34it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.16it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 20.07it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 19.94it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 19.90it/s]\u001b[A\n 31%|█████████████▍                             | 29/93 [00:01<00:03, 19.82it/s]\u001b[A\n 33%|██████████████▎                            | 31/93 [00:01<00:03, 19.66it/s]\u001b[A\n 35%|███████████████▎                           | 33/93 [00:01<00:03, 18.94it/s]\u001b[A\n 38%|████████████████▏                          | 35/93 [00:01<00:03, 18.87it/s]\u001b[A\n 40%|█████████████████                          | 37/93 [00:01<00:02, 18.85it/s]\u001b[A\n 42%|██████████████████                         | 39/93 [00:01<00:02, 18.90it/s]\u001b[A\n 44%|██████████████████▉                        | 41/93 [00:02<00:02, 18.84it/s]\u001b[A\n 46%|███████████████████▉                       | 43/93 [00:02<00:02, 18.93it/s]\u001b[A\n 48%|████████████████████▊                      | 45/93 [00:02<00:02, 18.96it/s]\u001b[A\n 51%|█████████████████████▋                     | 47/93 [00:02<00:02, 19.04it/s]\u001b[A\n 53%|██████████████████████▋                    | 49/93 [00:02<00:02, 18.98it/s]\u001b[A\n 55%|███████████████████████▌                   | 51/93 [00:02<00:02, 18.96it/s]\u001b[A\n 57%|████████████████████████▌                  | 53/93 [00:02<00:02, 18.97it/s]\u001b[A\n 59%|█████████████████████████▍                 | 55/93 [00:02<00:01, 19.22it/s]\u001b[A\n 61%|██████████████████████████▎                | 57/93 [00:02<00:01, 19.02it/s]\u001b[A\n 63%|███████████████████████████▎               | 59/93 [00:03<00:01, 19.00it/s]\u001b[A\n 66%|████████████████████████████▏              | 61/93 [00:03<00:01, 18.93it/s]\u001b[A\n 68%|█████████████████████████████▏             | 63/93 [00:03<00:01, 18.85it/s]\u001b[A\n 70%|██████████████████████████████             | 65/93 [00:03<00:01, 18.93it/s]\u001b[A\n 72%|██████████████████████████████▉            | 67/93 [00:03<00:01, 19.03it/s]\u001b[A\n 74%|███████████████████████████████▉           | 69/93 [00:03<00:01, 19.21it/s]\u001b[A\n 76%|████████████████████████████████▊          | 71/93 [00:03<00:01, 19.42it/s]\u001b[A\n 78%|█████████████████████████████████▊         | 73/93 [00:03<00:01, 19.56it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.61it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.66it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:04<00:00, 19.70it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.71it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.72it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.72it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.67it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.64it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 1.3410654067993164, 'eval_accuracy': 0.7320703864097595, 'eval_f1': 0.7320659498417906, 'eval_runtime': 4.776, 'eval_samples_per_second': 154.733, 'eval_steps_per_second': 19.472, 'epoch': 3.0}\n 33%|█████████████▎                          | 423/1269 [04:10<06:13,  2.26it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.68it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:22:37,320 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-423\n[INFO|configuration_utils.py:425] 2021-12-11 20:22:37,321 >> Configuration saved in models/ZeroShot/0/checkpoint-423/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:22:38,144 >> Model weights saved in models/ZeroShot/0/checkpoint-423/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:22:38,145 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-423/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:22:38,145 >> Special tokens file saved in models/ZeroShot/0/checkpoint-423/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:22:40,074 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-282] due to args.save_total_limit\n{'loss': 0.1874, 'learning_rate': 1.2119779353821908e-05, 'epoch': 3.55}        \n 44%|█████████████████▊                      | 564/1269 [05:29<05:11,  2.27it/s][INFO|trainer.py:549] 2021-12-11 20:23:56,753 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:23:56,755 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:23:56,755 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:23:56,755 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.67it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.88it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.31it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.65it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.28it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.16it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 20.09it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 20.06it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 19.91it/s]\u001b[A\n 31%|█████████████▍                             | 29/93 [00:01<00:03, 19.83it/s]\u001b[A\n 33%|██████████████▎                            | 31/93 [00:01<00:03, 19.77it/s]\u001b[A\n 35%|███████████████▎                           | 33/93 [00:01<00:03, 19.74it/s]\u001b[A\n 38%|████████████████▏                          | 35/93 [00:01<00:02, 19.72it/s]\u001b[A\n 40%|█████████████████                          | 37/93 [00:01<00:02, 19.74it/s]\u001b[A\n 42%|██████████████████                         | 39/93 [00:01<00:02, 19.78it/s]\u001b[A\n 44%|██████████████████▉                        | 41/93 [00:02<00:02, 19.81it/s]\u001b[A\n 46%|███████████████████▉                       | 43/93 [00:02<00:02, 19.74it/s]\u001b[A\n 48%|████████████████████▊                      | 45/93 [00:02<00:02, 19.72it/s]\u001b[A\n 51%|█████████████████████▋                     | 47/93 [00:02<00:02, 19.66it/s]\u001b[A\n 53%|██████████████████████▋                    | 49/93 [00:02<00:02, 19.65it/s]\u001b[A\n 55%|███████████████████████▌                   | 51/93 [00:02<00:02, 19.64it/s]\u001b[A\n 57%|████████████████████████▌                  | 53/93 [00:02<00:02, 19.64it/s]\u001b[A\n 59%|█████████████████████████▍                 | 55/93 [00:02<00:01, 19.64it/s]\u001b[A\n 61%|██████████████████████████▎                | 57/93 [00:02<00:01, 19.71it/s]\u001b[A\n 63%|███████████████████████████▎               | 59/93 [00:02<00:01, 19.74it/s]\u001b[A\n 66%|████████████████████████████▏              | 61/93 [00:03<00:01, 19.73it/s]\u001b[A\n 68%|█████████████████████████████▏             | 63/93 [00:03<00:01, 19.69it/s]\u001b[A\n 70%|██████████████████████████████             | 65/93 [00:03<00:01, 19.66it/s]\u001b[A\n 72%|██████████████████████████████▉            | 67/93 [00:03<00:01, 19.63it/s]\u001b[A\n 74%|███████████████████████████████▉           | 69/93 [00:03<00:01, 19.64it/s]\u001b[A\n 76%|████████████████████████████████▊          | 71/93 [00:03<00:01, 19.64it/s]\u001b[A\n 78%|█████████████████████████████████▊         | 73/93 [00:03<00:01, 19.65it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.68it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.71it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:03<00:00, 19.72it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.67it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.70it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.56it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.54it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.55it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 1.7039062976837158, 'eval_accuracy': 0.7225980758666992, 'eval_f1': 0.7213223918869948, 'eval_runtime': 4.7, 'eval_samples_per_second': 157.233, 'eval_steps_per_second': 19.787, 'epoch': 4.0}\n 44%|█████████████████▊                      | 564/1269 [05:34<05:11,  2.27it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.53it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:24:01,457 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-564\n[INFO|configuration_utils.py:425] 2021-12-11 20:24:01,458 >> Configuration saved in models/ZeroShot/0/checkpoint-564/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:24:02,272 >> Model weights saved in models/ZeroShot/0/checkpoint-564/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:24:02,273 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-564/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:24:02,273 >> Special tokens file saved in models/ZeroShot/0/checkpoint-564/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:24:04,498 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-423] due to args.save_total_limit\n 56%|██████████████████████▏                 | 705/1269 [06:54<04:12,  2.24it/s][INFO|trainer.py:549] 2021-12-11 20:25:21,108 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:25:21,111 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:25:21,111 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:25:21,111 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.37it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.69it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.19it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.66it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.36it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.10it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 19.89it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 19.81it/s]\u001b[A\n 28%|████████████                               | 26/93 [00:01<00:03, 19.77it/s]\u001b[A\n 30%|████████████▉                              | 28/93 [00:01<00:03, 19.75it/s]\u001b[A\n 32%|█████████████▊                             | 30/93 [00:01<00:03, 19.78it/s]\u001b[A\n 34%|██████████████▊                            | 32/93 [00:01<00:03, 19.80it/s]\u001b[A\n 37%|███████████████▋                           | 34/93 [00:01<00:02, 19.83it/s]\u001b[A\n 39%|████████████████▋                          | 36/93 [00:01<00:02, 19.79it/s]\u001b[A\n 41%|█████████████████▌                         | 38/93 [00:01<00:02, 19.75it/s]\u001b[A\n 43%|██████████████████▍                        | 40/93 [00:01<00:02, 19.59it/s]\u001b[A\n 45%|███████████████████▍                       | 42/93 [00:02<00:02, 19.52it/s]\u001b[A\n 47%|████████████████████▎                      | 44/93 [00:02<00:02, 19.53it/s]\u001b[A\n 49%|█████████████████████▎                     | 46/93 [00:02<00:02, 19.53it/s]\u001b[A\n 52%|██████████████████████▏                    | 48/93 [00:02<00:02, 19.60it/s]\u001b[A\n 54%|███████████████████████                    | 50/93 [00:02<00:02, 19.69it/s]\u001b[A\n 56%|████████████████████████                   | 52/93 [00:02<00:02, 19.75it/s]\u001b[A\n 58%|████████████████████████▉                  | 54/93 [00:02<00:01, 19.78it/s]\u001b[A\n 60%|█████████████████████████▉                 | 56/93 [00:02<00:01, 19.81it/s]\u001b[A\n 62%|██████████████████████████▊                | 58/93 [00:02<00:01, 19.71it/s]\u001b[A\n 65%|███████████████████████████▋               | 60/93 [00:02<00:01, 19.63it/s]\u001b[A\n 67%|████████████████████████████▋              | 62/93 [00:03<00:01, 19.62it/s]\u001b[A\n 69%|█████████████████████████████▌             | 64/93 [00:03<00:01, 19.60it/s]\u001b[A\n 71%|██████████████████████████████▌            | 66/93 [00:03<00:01, 19.59it/s]\u001b[A\n 73%|███████████████████████████████▍           | 68/93 [00:03<00:01, 19.63it/s]\u001b[A\n 75%|████████████████████████████████▎          | 70/93 [00:03<00:01, 19.67it/s]\u001b[A\n 77%|█████████████████████████████████▎         | 72/93 [00:03<00:01, 19.70it/s]\u001b[A\n 80%|██████████████████████████████████▏        | 74/93 [00:03<00:00, 19.75it/s]\u001b[A\n 82%|███████████████████████████████████▏       | 76/93 [00:03<00:00, 19.76it/s]\u001b[A\n 84%|████████████████████████████████████       | 78/93 [00:03<00:00, 19.58it/s]\u001b[A\n 86%|████████████████████████████████████▉      | 80/93 [00:04<00:00, 19.47it/s]\u001b[A\n 88%|█████████████████████████████████████▉     | 82/93 [00:04<00:00, 19.45it/s]\u001b[A\n 90%|██████████████████████████████████████▊    | 84/93 [00:04<00:00, 19.41it/s]\u001b[A\n 92%|███████████████████████████████████████▊   | 86/93 [00:04<00:00, 19.40it/s]\u001b[A\n 95%|████████████████████████████████████████▋  | 88/93 [00:04<00:00, 19.50it/s]\u001b[A\n 97%|█████████████████████████████████████████▌ | 90/93 [00:04<00:00, 19.63it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 1.9909651279449463, 'eval_accuracy': 0.7401894330978394, 'eval_f1': 0.740177551199789, 'eval_runtime': 4.7097, 'eval_samples_per_second': 156.909, 'eval_steps_per_second': 19.746, 'epoch': 5.0}\n 56%|██████████████████████▏                 | 705/1269 [06:58<04:12,  2.24it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.73it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:25:25,823 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-705\n[INFO|configuration_utils.py:425] 2021-12-11 20:25:25,824 >> Configuration saved in models/ZeroShot/0/checkpoint-705/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:25:26,659 >> Model weights saved in models/ZeroShot/0/checkpoint-705/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:25:26,660 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-705/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:25:26,660 >> Special tokens file saved in models/ZeroShot/0/checkpoint-705/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:25:28,645 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-141] due to args.save_total_limit\n[INFO|trainer.py:2111] 2021-12-11 20:25:28,839 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-564] due to args.save_total_limit\n 67%|██████████████████████████▋             | 846/1269 [08:18<03:06,  2.27it/s][INFO|trainer.py:549] 2021-12-11 20:26:45,251 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:26:45,253 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:26:45,253 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:26:45,253 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 28.93it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.86it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.30it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.54it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.19it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 19.92it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 19.85it/s]\u001b[A\n 25%|██████████▋                                | 23/93 [00:01<00:03, 19.85it/s]\u001b[A\n 27%|███████████▌                               | 25/93 [00:01<00:03, 19.88it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 19.90it/s]\u001b[A\n 32%|█████████████▊                             | 30/93 [00:01<00:03, 19.94it/s]\u001b[A\n 34%|██████████████▊                            | 32/93 [00:01<00:03, 19.92it/s]\u001b[A\n 37%|███████████████▋                           | 34/93 [00:01<00:02, 19.90it/s]\u001b[A\n 39%|████████████████▋                          | 36/93 [00:01<00:02, 19.79it/s]\u001b[A\n 41%|█████████████████▌                         | 38/93 [00:01<00:02, 19.74it/s]\u001b[A\n 43%|██████████████████▍                        | 40/93 [00:01<00:02, 19.77it/s]\u001b[A\n 45%|███████████████████▍                       | 42/93 [00:02<00:02, 19.70it/s]\u001b[A\n 47%|████████████████████▎                      | 44/93 [00:02<00:02, 19.63it/s]\u001b[A\n 49%|█████████████████████▎                     | 46/93 [00:02<00:02, 19.68it/s]\u001b[A\n 52%|██████████████████████▏                    | 48/93 [00:02<00:02, 19.70it/s]\u001b[A\n 54%|███████████████████████                    | 50/93 [00:02<00:02, 19.71it/s]\u001b[A\n 56%|████████████████████████                   | 52/93 [00:02<00:02, 19.74it/s]\u001b[A\n 58%|████████████████████████▉                  | 54/93 [00:02<00:01, 19.74it/s]\u001b[A\n 60%|█████████████████████████▉                 | 56/93 [00:02<00:01, 19.61it/s]\u001b[A\n 62%|██████████████████████████▊                | 58/93 [00:02<00:01, 19.58it/s]\u001b[A\n 65%|███████████████████████████▋               | 60/93 [00:02<00:01, 19.57it/s]\u001b[A\n 67%|████████████████████████████▋              | 62/93 [00:03<00:01, 19.33it/s]\u001b[A\n 69%|█████████████████████████████▌             | 64/93 [00:03<00:01, 18.45it/s]\u001b[A\n 71%|██████████████████████████████▌            | 66/93 [00:03<00:01, 18.19it/s]\u001b[A\n 73%|███████████████████████████████▍           | 68/93 [00:03<00:01, 18.39it/s]\u001b[A\n 75%|████████████████████████████████▎          | 70/93 [00:03<00:01, 18.56it/s]\u001b[A\n 77%|█████████████████████████████████▎         | 72/93 [00:03<00:01, 18.73it/s]\u001b[A\n 80%|██████████████████████████████████▏        | 74/93 [00:03<00:01, 18.96it/s]\u001b[A\n 82%|███████████████████████████████████▏       | 76/93 [00:03<00:00, 19.18it/s]\u001b[A\n 84%|████████████████████████████████████       | 78/93 [00:03<00:00, 19.29it/s]\u001b[A\n 86%|████████████████████████████████████▉      | 80/93 [00:04<00:00, 19.38it/s]\u001b[A\n 88%|█████████████████████████████████████▉     | 82/93 [00:04<00:00, 19.40it/s]\u001b[A\n 90%|██████████████████████████████████████▊    | 84/93 [00:04<00:00, 19.52it/s]\u001b[A\n 92%|███████████████████████████████████████▊   | 86/93 [00:04<00:00, 19.64it/s]\u001b[A\n 95%|████████████████████████████████████████▋  | 88/93 [00:04<00:00, 19.70it/s]\u001b[A\n 97%|█████████████████████████████████████████▌ | 90/93 [00:04<00:00, 19.74it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 2.1359317302703857, 'eval_accuracy': 0.7401894330978394, 'eval_f1': 0.7400823600099653, 'eval_runtime': 4.7456, 'eval_samples_per_second': 155.722, 'eval_steps_per_second': 19.597, 'epoch': 6.0}\n 67%|██████████████████████████▋             | 846/1269 [08:22<03:06,  2.27it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.75it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:26:50,001 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-846\n[INFO|configuration_utils.py:425] 2021-12-11 20:26:50,002 >> Configuration saved in models/ZeroShot/0/checkpoint-846/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:26:50,824 >> Model weights saved in models/ZeroShot/0/checkpoint-846/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:26:50,825 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-846/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:26:50,825 >> Special tokens file saved in models/ZeroShot/0/checkpoint-846/special_tokens_map.json\n 78%|███████████████████████████████         | 987/1269 [09:41<02:04,  2.26it/s][INFO|trainer.py:549] 2021-12-11 20:28:08,956 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:28:08,958 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:28:08,959 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:28:08,959 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.12it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.70it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.18it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.62it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.35it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.19it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 20.00it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 19.81it/s]\u001b[A\n 28%|████████████                               | 26/93 [00:01<00:03, 19.78it/s]\u001b[A\n 30%|████████████▉                              | 28/93 [00:01<00:03, 19.77it/s]\u001b[A\n 32%|█████████████▊                             | 30/93 [00:01<00:03, 19.73it/s]\u001b[A\n 34%|██████████████▊                            | 32/93 [00:01<00:03, 19.73it/s]\u001b[A\n 37%|███████████████▋                           | 34/93 [00:01<00:02, 19.76it/s]\u001b[A\n 39%|████████████████▋                          | 36/93 [00:01<00:02, 19.80it/s]\u001b[A\n 41%|█████████████████▌                         | 38/93 [00:01<00:02, 19.83it/s]\u001b[A\n 43%|██████████████████▍                        | 40/93 [00:01<00:02, 19.84it/s]\u001b[A\n 45%|███████████████████▍                       | 42/93 [00:02<00:02, 19.67it/s]\u001b[A\n 47%|████████████████████▎                      | 44/93 [00:02<00:02, 19.59it/s]\u001b[A\n 49%|█████████████████████▎                     | 46/93 [00:02<00:02, 19.58it/s]\u001b[A\n 52%|██████████████████████▏                    | 48/93 [00:02<00:02, 19.55it/s]\u001b[A\n 54%|███████████████████████                    | 50/93 [00:02<00:02, 19.55it/s]\u001b[A\n 56%|████████████████████████                   | 52/93 [00:02<00:02, 19.62it/s]\u001b[A\n 58%|████████████████████████▉                  | 54/93 [00:02<00:01, 19.67it/s]\u001b[A\n 60%|█████████████████████████▉                 | 56/93 [00:02<00:01, 19.73it/s]\u001b[A\n 62%|██████████████████████████▊                | 58/93 [00:02<00:01, 19.77it/s]\u001b[A\n 65%|███████████████████████████▋               | 60/93 [00:02<00:01, 19.78it/s]\u001b[A\n 67%|████████████████████████████▋              | 62/93 [00:03<00:01, 19.67it/s]\u001b[A\n 69%|█████████████████████████████▌             | 64/93 [00:03<00:01, 19.63it/s]\u001b[A\n 71%|██████████████████████████████▌            | 66/93 [00:03<00:01, 19.65it/s]\u001b[A\n 73%|███████████████████████████████▍           | 68/93 [00:03<00:01, 19.63it/s]\u001b[A\n 75%|████████████████████████████████▎          | 70/93 [00:03<00:01, 19.65it/s]\u001b[A\n 77%|█████████████████████████████████▎         | 72/93 [00:03<00:01, 19.74it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.86it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.89it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:03<00:00, 19.87it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.55it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.48it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.51it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.55it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.59it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 2.1124963760375977, 'eval_accuracy': 0.748308539390564, 'eval_f1': 0.7482711889238884, 'eval_runtime': 4.7034, 'eval_samples_per_second': 157.119, 'eval_steps_per_second': 19.773, 'epoch': 7.0}\n 78%|███████████████████████████████         | 987/1269 [09:46<02:04,  2.26it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.64it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:28:13,664 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-987\n[INFO|configuration_utils.py:425] 2021-12-11 20:28:13,666 >> Configuration saved in models/ZeroShot/0/checkpoint-987/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:28:14,547 >> Model weights saved in models/ZeroShot/0/checkpoint-987/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:28:14,548 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-987/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:28:14,548 >> Special tokens file saved in models/ZeroShot/0/checkpoint-987/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:28:16,551 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-705] due to args.save_total_limit\n[INFO|trainer.py:2111] 2021-12-11 20:28:16,739 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-846] due to args.save_total_limit\n{'loss': 0.026, 'learning_rate': 4.239558707643815e-06, 'epoch': 7.09}          \n 89%|██████████████████████████████████▋    | 1128/1269 [11:06<01:02,  2.27it/s][INFO|trainer.py:549] 2021-12-11 20:29:33,089 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:29:33,091 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:29:33,091 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:29:33,091 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.70it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.92it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:04, 20.63it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:04, 19.65it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:04, 19.43it/s]\u001b[A\n 18%|███████▊                                   | 17/93 [00:00<00:03, 19.10it/s]\u001b[A\n 20%|████████▊                                  | 19/93 [00:00<00:03, 18.69it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 18.70it/s]\u001b[A\n 25%|██████████▋                                | 23/93 [00:01<00:03, 18.77it/s]\u001b[A\n 27%|███████████▌                               | 25/93 [00:01<00:03, 18.79it/s]\u001b[A\n 29%|████████████▍                              | 27/93 [00:01<00:03, 18.57it/s]\u001b[A\n 31%|█████████████▍                             | 29/93 [00:01<00:03, 18.50it/s]\u001b[A\n 33%|██████████████▎                            | 31/93 [00:01<00:03, 18.44it/s]\u001b[A\n 35%|███████████████▎                           | 33/93 [00:01<00:03, 18.27it/s]\u001b[A\n 38%|████████████████▏                          | 35/93 [00:01<00:03, 18.15it/s]\u001b[A\n 40%|█████████████████                          | 37/93 [00:01<00:03, 18.07it/s]\u001b[A\n 42%|██████████████████                         | 39/93 [00:02<00:02, 18.39it/s]\u001b[A\n 44%|██████████████████▉                        | 41/93 [00:02<00:02, 18.69it/s]\u001b[A\n 46%|███████████████████▉                       | 43/93 [00:02<00:02, 18.84it/s]\u001b[A\n 48%|████████████████████▊                      | 45/93 [00:02<00:02, 19.05it/s]\u001b[A\n 51%|█████████████████████▋                     | 47/93 [00:02<00:02, 19.25it/s]\u001b[A\n 53%|██████████████████████▋                    | 49/93 [00:02<00:02, 19.41it/s]\u001b[A\n 55%|███████████████████████▌                   | 51/93 [00:02<00:02, 19.54it/s]\u001b[A\n 57%|████████████████████████▌                  | 53/93 [00:02<00:02, 19.65it/s]\u001b[A\n 59%|█████████████████████████▍                 | 55/93 [00:02<00:01, 19.69it/s]\u001b[A\n 61%|██████████████████████████▎                | 57/93 [00:02<00:01, 19.59it/s]\u001b[A\n 63%|███████████████████████████▎               | 59/93 [00:03<00:01, 19.58it/s]\u001b[A\n 66%|████████████████████████████▏              | 61/93 [00:03<00:01, 19.59it/s]\u001b[A\n 68%|█████████████████████████████▏             | 63/93 [00:03<00:01, 19.59it/s]\u001b[A\n 70%|██████████████████████████████             | 65/93 [00:03<00:01, 19.60it/s]\u001b[A\n 72%|██████████████████████████████▉            | 67/93 [00:03<00:01, 19.65it/s]\u001b[A\n 74%|███████████████████████████████▉           | 69/93 [00:03<00:01, 19.71it/s]\u001b[A\n 76%|████████████████████████████████▊          | 71/93 [00:03<00:01, 19.78it/s]\u001b[A\n 78%|█████████████████████████████████▊         | 73/93 [00:03<00:01, 19.79it/s]\u001b[A\n 81%|██████████████████████████████████▋        | 75/93 [00:03<00:00, 19.73it/s]\u001b[A\n 83%|███████████████████████████████████▌       | 77/93 [00:03<00:00, 19.64it/s]\u001b[A\n 85%|████████████████████████████████████▌      | 79/93 [00:04<00:00, 19.63it/s]\u001b[A\n 87%|█████████████████████████████████████▍     | 81/93 [00:04<00:00, 19.65it/s]\u001b[A\n 89%|██████████████████████████████████████▍    | 83/93 [00:04<00:00, 19.64it/s]\u001b[A\n 91%|███████████████████████████████████████▎   | 85/93 [00:04<00:00, 19.65it/s]\u001b[A\n 94%|████████████████████████████████████████▏  | 87/93 [00:04<00:00, 19.71it/s]\u001b[A\n 96%|█████████████████████████████████████████▏ | 89/93 [00:04<00:00, 19.74it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 2.2225940227508545, 'eval_accuracy': 0.7523680925369263, 'eval_f1': 0.7523662511879322, 'eval_runtime': 4.8195, 'eval_samples_per_second': 153.337, 'eval_steps_per_second': 19.297, 'epoch': 8.0}\n 89%|██████████████████████████████████▋    | 1128/1269 [11:10<01:02,  2.27it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.69it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:29:37,913 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-1128\n[INFO|configuration_utils.py:425] 2021-12-11 20:29:37,914 >> Configuration saved in models/ZeroShot/0/checkpoint-1128/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:29:38,743 >> Model weights saved in models/ZeroShot/0/checkpoint-1128/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:29:38,744 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-1128/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:29:38,744 >> Special tokens file saved in models/ZeroShot/0/checkpoint-1128/special_tokens_map.json\n[INFO|trainer.py:2111] 2021-12-11 20:29:40,613 >> Deleting older checkpoint [models/ZeroShot/0/checkpoint-987] due to args.save_total_limit\n100%|███████████████████████████████████████| 1269/1269 [12:30<00:00,  2.26it/s][INFO|trainer.py:549] 2021-12-11 20:30:57,249 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:30:57,251 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:30:57,251 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:30:57,251 >>   Batch size = 8\n\n  0%|                                                    | 0/93 [00:00<?, ?it/s]\u001b[A\n  3%|█▍                                          | 3/93 [00:00<00:03, 29.43it/s]\u001b[A\n  6%|██▊                                         | 6/93 [00:00<00:03, 22.73it/s]\u001b[A\n 10%|████▎                                       | 9/93 [00:00<00:03, 21.39it/s]\u001b[A\n 13%|█████▌                                     | 12/93 [00:00<00:03, 20.78it/s]\u001b[A\n 16%|██████▉                                    | 15/93 [00:00<00:03, 20.44it/s]\u001b[A\n 19%|████████▎                                  | 18/93 [00:00<00:03, 20.05it/s]\u001b[A\n 23%|█████████▋                                 | 21/93 [00:01<00:03, 19.90it/s]\u001b[A\n 26%|███████████                                | 24/93 [00:01<00:03, 19.80it/s]\u001b[A\n 28%|████████████                               | 26/93 [00:01<00:03, 19.70it/s]\u001b[A\n 30%|████████████▉                              | 28/93 [00:01<00:03, 19.74it/s]\u001b[A\n 32%|█████████████▊                             | 30/93 [00:01<00:03, 19.70it/s]\u001b[A\n 34%|██████████████▊                            | 32/93 [00:01<00:03, 19.66it/s]\u001b[A\n 37%|███████████████▋                           | 34/93 [00:01<00:03, 19.60it/s]\u001b[A\n 39%|████████████████▋                          | 36/93 [00:01<00:02, 19.54it/s]\u001b[A\n 41%|█████████████████▌                         | 38/93 [00:01<00:02, 19.49it/s]\u001b[A\n 43%|██████████████████▍                        | 40/93 [00:01<00:02, 19.47it/s]\u001b[A\n 45%|███████████████████▍                       | 42/93 [00:02<00:02, 19.45it/s]\u001b[A\n 47%|████████████████████▎                      | 44/93 [00:02<00:02, 19.44it/s]\u001b[A\n 49%|█████████████████████▎                     | 46/93 [00:02<00:02, 19.49it/s]\u001b[A\n 52%|██████████████████████▏                    | 48/93 [00:02<00:02, 19.58it/s]\u001b[A\n 54%|███████████████████████                    | 50/93 [00:02<00:02, 19.63it/s]\u001b[A\n 56%|████████████████████████                   | 52/93 [00:02<00:02, 19.61it/s]\u001b[A\n 58%|████████████████████████▉                  | 54/93 [00:02<00:01, 19.57it/s]\u001b[A\n 60%|█████████████████████████▉                 | 56/93 [00:02<00:01, 19.49it/s]\u001b[A\n 62%|██████████████████████████▊                | 58/93 [00:02<00:01, 19.47it/s]\u001b[A\n 65%|███████████████████████████▋               | 60/93 [00:03<00:01, 19.48it/s]\u001b[A\n 67%|████████████████████████████▋              | 62/93 [00:03<00:01, 19.49it/s]\u001b[A\n 69%|█████████████████████████████▌             | 64/93 [00:03<00:01, 19.49it/s]\u001b[A\n 71%|██████████████████████████████▌            | 66/93 [00:03<00:01, 19.56it/s]\u001b[A\n 73%|███████████████████████████████▍           | 68/93 [00:03<00:01, 19.64it/s]\u001b[A\n 75%|████████████████████████████████▎          | 70/93 [00:03<00:01, 19.70it/s]\u001b[A\n 77%|█████████████████████████████████▎         | 72/93 [00:03<00:01, 19.72it/s]\u001b[A\n 80%|██████████████████████████████████▏        | 74/93 [00:03<00:00, 19.67it/s]\u001b[A\n 82%|███████████████████████████████████▏       | 76/93 [00:03<00:00, 19.59it/s]\u001b[A\n 84%|████████████████████████████████████       | 78/93 [00:03<00:00, 19.56it/s]\u001b[A\n 86%|████████████████████████████████████▉      | 80/93 [00:04<00:00, 19.59it/s]\u001b[A\n 88%|█████████████████████████████████████▉     | 82/93 [00:04<00:00, 19.27it/s]\u001b[A\n 90%|██████████████████████████████████████▊    | 84/93 [00:04<00:00, 19.01it/s]\u001b[A\n 92%|███████████████████████████████████████▊   | 86/93 [00:04<00:00, 18.91it/s]\u001b[A\n 95%|████████████████████████████████████████▋  | 88/93 [00:04<00:00, 18.94it/s]\u001b[A\n 97%|█████████████████████████████████████████▌ | 90/93 [00:04<00:00, 18.89it/s]\u001b[A\n                                                                                \u001b[A\n\u001b[A{'eval_loss': 2.3209517002105713, 'eval_accuracy': 0.7442489862442017, 'eval_f1': 0.7441571644456655, 'eval_runtime': 4.7589, 'eval_samples_per_second': 155.286, 'eval_steps_per_second': 19.542, 'epoch': 9.0}\n100%|███████████████████████████████████████| 1269/1269 [12:34<00:00,  2.26it/s]\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 18.79it/s]\u001b[A\n                                                                                \u001b[A[INFO|trainer.py:2033] 2021-12-11 20:31:02,013 >> Saving model checkpoint to models/ZeroShot/0/checkpoint-1269\n[INFO|configuration_utils.py:425] 2021-12-11 20:31:02,014 >> Configuration saved in models/ZeroShot/0/checkpoint-1269/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:31:02,831 >> Model weights saved in models/ZeroShot/0/checkpoint-1269/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:31:02,831 >> tokenizer config file saved in models/ZeroShot/0/checkpoint-1269/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:31:02,832 >> Special tokens file saved in models/ZeroShot/0/checkpoint-1269/special_tokens_map.json\n[INFO|trainer.py:1425] 2021-12-11 20:31:04,728 >> \n\nTraining completed. Do not forget to share your model on huggingface.co/models =)\n\n\n[INFO|trainer.py:1434] 2021-12-11 20:31:04,729 >> Loading best model from models/ZeroShot/0/checkpoint-1128 (score: 0.7523662511879322).\n{'train_runtime': 758.7711, 'train_samples_per_second': 53.269, 'train_steps_per_second': 1.672, 'train_loss': 0.08620423458992167, 'epoch': 9.0}\n100%|███████████████████████████████████████| 1269/1269 [12:38<00:00,  1.67it/s]\n[INFO|trainer.py:2033] 2021-12-11 20:31:05,813 >> Saving model checkpoint to models/ZeroShot/0/\n[INFO|configuration_utils.py:425] 2021-12-11 20:31:05,814 >> Configuration saved in models/ZeroShot/0/config.json\n[INFO|modeling_utils.py:1070] 2021-12-11 20:31:14,436 >> Model weights saved in models/ZeroShot/0/pytorch_model.bin\n[INFO|tokenization_utils_base.py:2043] 2021-12-11 20:31:14,437 >> tokenizer config file saved in models/ZeroShot/0/tokenizer_config.json\n[INFO|tokenization_utils_base.py:2049] 2021-12-11 20:31:14,438 >> Special tokens file saved in models/ZeroShot/0/special_tokens_map.json\n***** train metrics *****\n  epoch                    =        9.0\n  train_loss               =     0.0862\n  train_runtime            = 0:12:38.77\n  train_samples            =       4491\n  train_samples_per_second =     53.269\n  train_steps_per_second   =      1.672\n[INFO|trainer.py:549] 2021-12-11 20:31:16,265 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence2, sentence1.\n[INFO|trainer.py:2281] 2021-12-11 20:31:16,267 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:31:16,267 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:31:16,267 >>   Batch size = 8\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.90it/s]\n***** eval metrics *****\n  epoch                   =        9.0\n  eval_accuracy           =     0.7524\n  eval_f1                 =     0.7524\n  eval_loss               =     2.2226\n  eval_runtime            = 0:00:04.73\n  eval_samples            =        739\n  eval_samples_per_second =    156.215\n  eval_steps_per_second   =     19.659\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Evaluation On Dev Data","metadata":{"id":"5bN4iUHWP45b"}},{"cell_type":"code","source":"!python3 /kaggle/working/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path '/kaggle/working/models/ZeroShot/0' \\\n    \t--do_predict \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/ZeroShot/0/eval-dev/ \\\n    \t--seed 0 \\\n    \t--train_file      Data/ZeroShot/train.csv \\\n    \t--validation_file Data/ZeroShot/dev.csv \\\n      --test_file Data/ZeroShot/dev.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n\t    --save_total_limit 1","metadata":{"id":"houOZpcYO-Pw","outputId":"e8583e14-6ffc-4c18-c718-98400b394f30","execution":{"iopub.status.busy":"2021-12-11T20:32:30.314689Z","iopub.execute_input":"2021-12-11T20:32:30.314979Z","iopub.status.idle":"2021-12-11T20:32:50.996496Z","shell.execute_reply.started":"2021-12-11T20:32:30.314947Z","shell.execute_reply":"2021-12-11T20:32:50.995410Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":45,"outputs":[{"name":"stdout","text":"Using the `WAND_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).\n100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 571.51it/s]\n[INFO|configuration_utils.py:602] 2021-12-11 20:32:35,394 >> loading configuration file /kaggle/working/models/ZeroShot/0/config.json\n[INFO|configuration_utils.py:641] 2021-12-11 20:32:35,398 >> Model config XLNetConfig {\n  \"_name_or_path\": \"/kaggle/working/models/ZeroShot/0\",\n  \"architectures\": [\n    \"XLNetForSequenceClassification\"\n  ],\n  \"attn_type\": \"bi\",\n  \"bi_data\": false,\n  \"bos_token_id\": 1,\n  \"clamp_len\": -1,\n  \"d_head\": 64,\n  \"d_inner\": 3072,\n  \"d_model\": 768,\n  \"dropout\": 0.1,\n  \"end_n_top\": 5,\n  \"eos_token_id\": 2,\n  \"ff_activation\": \"gelu\",\n  \"initializer_range\": 0.02,\n  \"layer_norm_eps\": 1e-12,\n  \"mem_len\": null,\n  \"model_type\": \"xlnet\",\n  \"n_head\": 12,\n  \"n_layer\": 12,\n  \"pad_token_id\": 5,\n  \"problem_type\": \"single_label_classification\",\n  \"reuse_len\": null,\n  \"same_length\": false,\n  \"start_n_top\": 5,\n  \"summary_activation\": \"tanh\",\n  \"summary_last_dropout\": 0.1,\n  \"summary_type\": \"last\",\n  \"summary_use_proj\": true,\n  \"task_specific_params\": {\n    \"text-generation\": {\n      \"do_sample\": true,\n      \"max_length\": 250\n    }\n  },\n  \"torch_dtype\": \"float32\",\n  \"transformers_version\": \"4.14.0.dev0\",\n  \"untie_r\": true,\n  \"use_mems_eval\": true,\n  \"use_mems_train\": false,\n  \"vocab_size\": 32000\n}\n\n[INFO|tokenization_utils_base.py:1671] 2021-12-11 20:32:35,420 >> Didn't find file /kaggle/working/models/ZeroShot/0/added_tokens.json. We won't load it.\n[INFO|tokenization_utils_base.py:1740] 2021-12-11 20:32:35,424 >> loading file /kaggle/working/models/ZeroShot/0/spiece.model\n[INFO|tokenization_utils_base.py:1740] 2021-12-11 20:32:35,424 >> loading file /kaggle/working/models/ZeroShot/0/tokenizer.json\n[INFO|tokenization_utils_base.py:1740] 2021-12-11 20:32:35,424 >> loading file None\n[INFO|tokenization_utils_base.py:1740] 2021-12-11 20:32:35,425 >> loading file /kaggle/working/models/ZeroShot/0/special_tokens_map.json\n[INFO|tokenization_utils_base.py:1740] 2021-12-11 20:32:35,425 >> loading file /kaggle/working/models/ZeroShot/0/tokenizer_config.json\n[INFO|modeling_utils.py:1350] 2021-12-11 20:32:35,543 >> loading weights file /kaggle/working/models/ZeroShot/0/pytorch_model.bin\n[INFO|modeling_utils.py:1619] 2021-12-11 20:32:36,710 >> All model checkpoint weights were used when initializing XLNetForSequenceClassification.\n\n[INFO|modeling_utils.py:1628] 2021-12-11 20:32:36,710 >> All the weights of XLNetForSequenceClassification were initialized from the model checkpoint at /kaggle/working/models/ZeroShot/0.\nIf your task is similar to the task the model of the checkpoint was trained on, you can already use XLNetForSequenceClassification for predictions without further training.\n100%|█████████████████████████████████████████████| 5/5 [00:00<00:00,  5.38ba/s]\n100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  6.75ba/s]\n100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  5.42ba/s]\n[INFO|trainer.py:549] 2021-12-11 20:32:40,341 >> The following columns in the evaluation set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence1, sentence2.\n[INFO|trainer.py:2281] 2021-12-11 20:32:40,343 >> ***** Running Evaluation *****\n[INFO|trainer.py:2283] 2021-12-11 20:32:40,343 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:32:40,343 >>   Batch size = 8\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.90it/s]\n***** eval metrics *****\n  eval_accuracy           =     0.7524\n  eval_f1                 =     0.7524\n  eval_loss               =     2.2226\n  eval_runtime            = 0:00:05.08\n  eval_samples            =        739\n  eval_samples_per_second =    145.214\n  eval_steps_per_second   =     18.275\n/kaggle/working/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py:514: FutureWarning: remove_columns_ is deprecated and will be removed in the next major version of datasets. Use Dataset.remove_columns instead.\n  test_dataset.remove_columns_(\"label\")\n[INFO|trainer.py:549] 2021-12-11 20:32:45,437 >> The following columns in the test set  don't have a corresponding argument in `XLNetForSequenceClassification.forward` and have been ignored: sentence1, sentence2.\n[INFO|trainer.py:2281] 2021-12-11 20:32:45,439 >> ***** Running Prediction *****\n[INFO|trainer.py:2283] 2021-12-11 20:32:45,439 >>   Num examples = 739\n[INFO|trainer.py:2286] 2021-12-11 20:32:45,439 >>   Batch size = 8\n100%|███████████████████████████████████████████| 93/93 [00:04<00:00, 19.90it/s]\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### Use predictions to create the submission file (for dev data)","metadata":{"id":"EHqPYuTS3muJ"}},{"cell_type":"code","source":"params = {\n    'submission_format_file' : '/kaggle/working/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/dev_submission_format.csv' ,\n    'input_file'             : '/kaggle/working/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/dev.csv'                   ,\n    'prediction_format_file' : '/kaggle/working/models/ZeroShot/0/eval-dev/test_results_None.txt'                        ,\n    }\nparams[ 'setting' ] = 'zero_shot'","metadata":{"id":"qfWUuwg7Qm-t","execution":{"iopub.status.busy":"2021-12-11T20:34:18.512760Z","iopub.execute_input":"2021-12-11T20:34:18.513209Z","iopub.status.idle":"2021-12-11T20:34:18.518047Z","shell.execute_reply.started":"2021-12-11T20:34:18.513171Z","shell.execute_reply":"2021-12-11T20:34:18.516997Z"},"trusted":true},"execution_count":50,"outputs":[]},{"cell_type":"code","source":" updated_data = insert_to_submission_file( **params )","metadata":{"id":"cgFGlGnTROJZ","execution":{"iopub.status.busy":"2021-12-11T20:34:20.141241Z","iopub.execute_input":"2021-12-11T20:34:20.141833Z","iopub.status.idle":"2021-12-11T20:34:20.166120Z","shell.execute_reply.started":"2021-12-11T20:34:20.141779Z","shell.execute_reply":"2021-12-11T20:34:20.165330Z"},"trusted":true},"execution_count":51,"outputs":[]},{"cell_type":"code","source":"!mkdir -p outputs","metadata":{"id":"zXcRbv70RZfR","execution":{"iopub.status.busy":"2021-12-11T20:34:21.369648Z","iopub.execute_input":"2021-12-11T20:34:21.370471Z","iopub.status.idle":"2021-12-11T20:34:22.251218Z","shell.execute_reply.started":"2021-12-11T20:34:21.370419Z","shell.execute_reply":"2021-12-11T20:34:22.249915Z"},"trusted":true},"execution_count":52,"outputs":[]},{"cell_type":"code","source":"write_csv( updated_data, 'outputs/zero_shot_dev_formated.csv' ) ","metadata":{"id":"2ezIzyWTRePp","outputId":"7de89b60-8070-46de-b295-fa90cf847955","execution":{"iopub.status.busy":"2021-12-11T20:34:23.881155Z","iopub.execute_input":"2021-12-11T20:34:23.881698Z","iopub.status.idle":"2021-12-11T20:34:23.889351Z","shell.execute_reply.started":"2021-12-11T20:34:23.881657Z","shell.execute_reply":"2021-12-11T20:34:23.888429Z"},"trusted":true},"execution_count":53,"outputs":[{"name":"stdout","text":"Wrote outputs/zero_shot_dev_formated.csv\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### For the development data, we can run evaluation script.","metadata":{"id":"rZpFkOSQ3x_A"}},{"cell_type":"code","source":"import sys\nsys.path.append( '/kaggle/working/SemEval_2022_Task2-idiomaticity/SubTaskA/' ) \nfrom SubTask1Evaluator import evaluate_submission\n\n\nsubmission_file = 'outputs/zero_shot_dev_formated.csv'\ngold_file       = '/kaggle/working/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/dev_gold.csv'\n\nresults = evaluate_submission( submission_file, gold_file )\n# %reload_ext google.colab.data_table\nimport pandas as pd\ndf = pd.DataFrame(data=results[1:], columns=results[0])\ndf","metadata":{"id":"Dn6MyP9jRnFA","outputId":"a4e5e3e6-287b-4de6-c643-e569e9c2b42e","execution":{"iopub.status.busy":"2021-12-11T20:35:27.087682Z","iopub.execute_input":"2021-12-11T20:35:27.087991Z","iopub.status.idle":"2021-12-11T20:35:27.151022Z","shell.execute_reply.started":"2021-12-11T20:35:27.087940Z","shell.execute_reply":"2021-12-11T20:35:27.150242Z"},"trusted":true},"execution_count":58,"outputs":[{"execution_count":58,"output_type":"execute_result","data":{"text/plain":"    Settings Languages    F1 Score (Macro)\n0  zero_shot        EN            0.789322\n1  zero_shot        PT            0.613095\n2  zero_shot     EN,PT            0.752366\n3   one_shot        EN  (None, None, None)\n4   one_shot        PT  (None, None, None)\n5   one_shot     EN,PT  (None, None, None)","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Settings</th>\n      <th>Languages</th>\n      <th>F1 Score (Macro)</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>zero_shot</td>\n      <td>EN</td>\n      <td>0.789322</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>zero_shot</td>\n      <td>PT</td>\n      <td>0.613095</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>zero_shot</td>\n      <td>EN,PT</td>\n      <td>0.752366</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>one_shot</td>\n      <td>EN</td>\n      <td>(None, None, None)</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>one_shot</td>\n      <td>PT</td>\n      <td>(None, None, None)</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>one_shot</td>\n      <td>EN,PT</td>\n      <td>(None, None, None)</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"## Generate Eval Data output","metadata":{"id":"9UX9ys7tTKVi"}},{"cell_type":"code","source":"!python /content/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path '/content/models/ZeroShot/0' \\\n    \t--do_predict \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/ZeroShot/0/eval-eval/ \\\n    \t--seed 0 \\\n    \t--train_file      Data/ZeroShot/train.csv \\\n    \t--validation_file Data/ZeroShot/dev.csv \\\n      --test_file Data/ZeroShot/eval.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n\t    --save_total_limit 1","metadata":{"id":"ets4flitTRZZ","outputId":"1deda94d-73b9-4955-c0d4-4999e3b306ee"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Use predictions to create the submission file (for eval data)","metadata":{"id":"KSS6PFfb4AAl"}},{"cell_type":"code","source":"params = {\n    'submission_format_file' : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/eval_submission_format.csv' ,\n    'input_file'             : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/eval.csv'                   ,\n    'prediction_format_file' : '/content/models/ZeroShot/0/eval-eval/test_results_None.txt'                        ,\n    }\nparams[ 'setting' ] = 'zero_shot'","metadata":{"id":"WqxzrRBfTcnq"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" updated_data = insert_to_submission_file( **params )","metadata":{"id":"IHqanuVDTz-r"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"write_csv( updated_data, 'outputs/zero_shot_eval_formated.csv' ) ","metadata":{"id":"CKDBuxMLT5QU","outputId":"f5eefcfb-ace6-4225-f64a-92bba8dd3ec0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**NOTE**: You can submit this file, but it only has results for the zero-shot setting.","metadata":{"id":"qyWXLvHI4Cdt"}},{"cell_type":"markdown","source":"# One Shot Setting","metadata":{"id":"WY7Irn_YvIni"}},{"cell_type":"markdown","source":"## Train One shot","metadata":{"id":"oVYvUH3OvR9b"}},{"cell_type":"code","source":"!python /content/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path 'bert-base-multilingual-cased' \\\n    \t--do_train \\\n    \t--do_eval \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/OneShot/1/ \\\n    \t--seed 1 \\\n    \t--train_file      Data/OneShot/train.csv \\\n    \t--validation_file Data/OneShot/dev.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n\t    --save_total_limit 1","metadata":{"id":"YQO751yzvVJI","outputId":"355f4d09-4507-4040-8874-20526e0fe4e7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/gdrive')","metadata":{"id":"hW8stSsnIKWo","outputId":"a797d11f-5b2f-4854-883f-fa1d592dafe1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create save path\n!mkdir -p /content/gdrive/MyDrive/ColabData/SemEval2022Task2/TaskA/OneShot/1/\n## Copy saved model.\n!cp -r /content/models/OneShot/1/* /content/gdrive/MyDrive/ColabData/SemEval2022Task2/TaskA/OneShot/1/","metadata":{"id":"f0uO16BfcIur"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluation On Dev Data","metadata":{"id":"6dijAXZ7dD5V"}},{"cell_type":"code","source":"!python /content/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path '/content/models/OneShot/1' \\\n    \t--do_predict \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/OneShot/1/eval-dev/ \\\n    \t--seed 1 \\\n    \t--train_file      Data/OneShot/train.csv \\\n    \t--validation_file Data/OneShot/dev.csv \\\n      --test_file Data/OneShot/dev.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n\t    --save_total_limit 1","metadata":{"id":"2903I4hKdJuE","outputId":"76a28edb-007c-457e-b3a9-8530d8ef5d68"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Use predictions to create the submission file (for dev data)","metadata":{"id":"iKjobV-x4R_8"}},{"cell_type":"code","source":"params = {\n    'submission_format_file' : '/content/outputs/zero_shot_dev_formated.csv' ,\n    'input_file'             : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/dev.csv'                   ,\n    'prediction_format_file' : '/content/models/OneShot/1/eval-dev/test_results_None.txt'                        ,\n    }\nparams[ 'setting' ] = 'one_shot'","metadata":{"id":"KZIEpIstdg_j"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" updated_data = insert_to_submission_file( **params )\n write_csv( updated_data, 'outputs/both_dev_formated.csv' ) ","metadata":{"id":"xrS0hvEDdspS","outputId":"9d87a4b1-6d24-47ce-8277-b426ec562b43"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### For the development data, we can run evaluation script.","metadata":{"id":"cB2cBSmA4ZSR"}},{"cell_type":"code","source":"import sys\nsys.path.append( '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/' ) \nfrom SubTask1Evaluator import evaluate_submission\n\n\nsubmission_file = 'outputs/both_dev_formated.csv'\ngold_file       = '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/dev_gold.csv'\n\nresults = evaluate_submission( submission_file, gold_file )\n%reload_ext google.colab.data_table\nimport pandas as pd\ndf = pd.DataFrame(data=results[1:], columns=results[0])\ndf","metadata":{"id":"vTWEOw3Qd0Tz","outputId":"4fff9308-e5e0-4a30-f0d2-ef4634363cb0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generate Eval Data output","metadata":{"id":"nQhQptb1ePms"}},{"cell_type":"code","source":"!python /content/AStitchInLanguageModels/Dataset/Task2/Utils/run_glue_f1_macro.py \\\n    \t--model_name_or_path '/content/models/OneShot/1' \\\n    \t--do_predict \\\n    \t--max_seq_length 128 \\\n    \t--per_device_train_batch_size 32 \\\n    \t--learning_rate 2e-5 \\\n    \t--num_train_epochs 9 \\\n    \t--evaluation_strategy \"epoch\" \\\n    \t--output_dir models/OneShot/1/eval-eval/ \\\n    \t--seed 1 \\\n    \t--train_file      Data/OneShot/train.csv \\\n    \t--validation_file Data/OneShot/dev.csv \\\n      --test_file Data/OneShot/eval.csv \\\n\t    --evaluation_strategy \"epoch\" \\\n\t    --save_strategy \"epoch\"  \\\n\t    --load_best_model_at_end \\\n\t    --metric_for_best_model \"f1\" \\\n\t    --save_total_limit 1","metadata":{"id":"81G4ULAseSPB","outputId":"9c1dcaec-c058-4f3b-c963-9cf040e854e1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Use predictions to create the submission file (for eval data)","metadata":{"id":"UnOf3yts4gcb"}},{"cell_type":"markdown","source":"#### Create One Shot submission","metadata":{"id":"XaG_XF6JJNuV"}},{"cell_type":"code","source":"params = {\n    'submission_format_file' : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/eval_submission_format.csv' ,\n    'input_file'             : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/eval.csv'                   ,\n    'prediction_format_file' : '/content/models/OneShot/1/eval-eval/test_results_None.txt'                         ,\n    }\nparams[ 'setting' ] = 'one_shot'\n","metadata":{"id":"zk1EaW7IJa3A"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" updated_data = insert_to_submission_file( **params )\n write_csv( updated_data, 'outputs/one_shot_eval_formated.csv' ) ","metadata":{"id":"1IjSoShpJv6z","outputId":"71d6fc47-8164-46b4-ccc2-44e171cbe30e"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Combine Zero Shot and One Shot submission files.\n\nDo this by loading zero shot data as submission file format.","metadata":{"id":"cZlTBYSiJRP6"}},{"cell_type":"code","source":"params = {\n    'submission_format_file' : '/content/outputs/zero_shot_eval_formated.csv' ,\n    'input_file'             : '/content/SemEval_2022_Task2-idiomaticity/SubTaskA/Data/eval.csv'                   ,\n    'prediction_format_file' : '/content/models/OneShot/1/eval-eval/test_results_None.txt'                        ,\n    }\nparams[ 'setting' ] = 'one_shot'\n","metadata":{"id":"QwpGn7d3edI1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" updated_data = insert_to_submission_file( **params )\n write_csv( updated_data, 'outputs/task2_subtaska.csv' ) ","metadata":{"id":"440OwBxyfJJs","outputId":"99274dd4-42b0-41c5-d390-b511f5ee4c9c"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Download Submission File","metadata":{"id":"rLz_mxylfc1e"}},{"cell_type":"code","source":"from google.colab import files\nfiles.download('/content/outputs/task2_subtaska.csv') \n## Remeber to put this in a folder called \"submission\".","metadata":{"id":"R6vl3zemfeg6","outputId":"b9519c6b-2fd9-49b6-c582-75dfbb9dfd8b"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discussion","metadata":{"id":"EuNKaVgbf2a2"}},{"cell_type":"code","source":"df","metadata":{"id":"cd78IDfLf3Xy","outputId":"585f4392-9f86-4733-f06d-7b56382a0cea"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice the significant jump in F1 scores with the introduction of just one positive and one negative example. \n\nNote that your position on the leaderboard will be based on rows with index 2 and 5 (combined results for both languages). The rest of the results for information and ablation studies. \n\n\n\n","metadata":{"id":"IEYD3V3I5qEN"}}]}