Skip to content

Latest commit

 

History

History
184 lines (160 loc) · 10.6 KB

DS_ER_README.md

File metadata and controls

184 lines (160 loc) · 10.6 KB

Datasets and Experimental Results

Datasets

COVID-19-related Tweets Dataset Statistics

Negative Neutral Positive Total
Train 15398 7712 18046 41156
Test 1633 619 1546 3798

Economic texts Dataset Statistics

Negative Neutral 2 Positive Total
Train 483 2302 1091 3876
Test 121 576 272 969

E-commerce texts Dataset Statistics

Household Books C&A Electronics Total
Train 15449 9456 6936 8497 40338
Test 3863 2364 1734 2124 10085

SMS Spam collection Statistics

Normal Spam Total
Train 3859 598 4457
Test 966 149 1115

Experimental Results

S: with few shot strategy; F: with fine-tuned strategy

COVID-19-RELATED TWEETS Sentiment classification results

Model ACC($\uparrow$) F1($\uparrow$) U/E($\downarrow$)
MNB 0.4037 0.3827 -
LR 0.3875 0.3131 -
RF 0.4462 0.3633 -
DT 0.4037 0.3416 -
KNN 0.3825 0.3481 -
------------- ----------------- ---------------- -------------------
GRU 0.6913 0.6324 -
LSTM 0.6687 0.6312 -
RNN 0.6600 0.6332 -
------------- ----------------- ---------------- -------------------
BART 0.5138 0.3638 -
DeBERTa 0.5375 0.3804 -
------------- ----------------- ---------------- -------------------
GPT-3.5 0.5550 0.5435 0.0000
GPT-4 0.5100 0.5054 0.0000
Gemini-pro 0.5025 0.5105 0.0388
Llama-3-8B 0.5112 0.5149 0.0013
Qwen-7B 0.4913 0.4689 0.0025
Qwen-14B 0.4562 0.4569 0.0100
Vicuna-7B 0.3600 0.3403 0.0000
Vicuna-13B 0.5050 0.4951 0.0013
------------- ----------------- ---------------- -------------------
Gemini-pro(S) 0.4888(-0.014) 0.4880(-0.022) 0.0375(-0.001)
Llama-3-8B(S) 0.5363(+0.025) 0.5298(+0.015) 0.0000(-0.001)
Qwen-7B(S) 0.3900(-0.101) 0.3519(-0.117) 0.0150(+0.012)
Qwen-14B(S) 0.4575(+0.001) 0.4556(-0.001) 0.0037(-0.006)
Vicuna-7B(S) 0.3700(+0.010) 0.3362(-0.004) 0.0013(+0.001)
Vicuna-13B(S) 0.5050(+0.000) 0.4951(+0.000) 0.0000(-0.001)
------------- ----------------- ---------------- -------------------
Llama-3-8B(F) 0.4675(-0.044) 0.4910(-0.024) 0.1175(+0.116)
Qwen-7B(F) 0.8388(+0.348) 0.8433(+0.374) 0.0000(+0.000)

E-Commercial Product Text Classification Results

Model ACC($\uparrow$) F1($\uparrow$) U/E($\downarrow$)
MNB 0.2562 0.2384 -
LR 0.3825 0.2873 -
RF 0.4875 0.3958 -
DT 0.4263 0.4165 -
KNN 0.3762 0.3414 -
------------- ----------------- ---------------- -------------------
GRU 0.9387 0.9383 -
LSTM 0.9363 0.9398 -
RNN 0.8975 0.9010 -
------------- ----------------- ---------------- -------------------
BART 0.7175 0.7246 -
DeBERTa 0.6025 0.6121 -
------------- ----------------- ---------------- -------------------
GPT-3.5 0.9125 0.9152 0.0063
GPT-4 0.9137 0.9221 0.0088
Gemini-pro 0.8775 0.8873 0.0100
Llama-3-8B 0.9113 0.9112 0.0000
Qwen-7B 0.5850 0.6584 0.1850
Qwen-14B 0.6575 0.6843 0.0800
Vicuna-7B 0.7100 0.7164 0.0050
Vicuna-13B 0.8363 0.8503 0.0138
------------- ----------------- ---------------- -------------------
Gemini-pro(S) 0.8862(+0.009) 0.8963(+0.009) 0.0100(+0.000)
Llama-3-8B(S) 0.9062(-0.005) 0.9065(-0.005) 0.0000(+0.000)
Qwen-7B(S) 0.6737(+0.089) 0.8226(+0.164) 0.1812(-0.004)
Qwen-14B(S) 0.7887(+0.131) 0.8548(+0.170) 0.0775(-0.003)
Vicuna-7B(S) 0.7925(+0.083) 0.7899(+0.074) 0.0000(-0.005)
Vicuna-13B(S) 0.9075(+0.071) 0.9153(+0.065) 0.0088(-0.005)
------------- ----------------- ---------------- -------------------
Llama-3-8B(F) 0.9175(+0.006) 0.9164(+0.003) 0.0000(+0.000)
Qwen-7B(F) 0.9713(+0.386) 0.9713(+0.313) 0.0000(-0.185)

ECONOMIC TEXTS Sentiment Classification Results

Model ACC($\uparrow$) F1($\uparrow$) U/E($\downarrow$)
MNB 0.2600 0.2570 -
LR 0.5962 0.3055 -
RF 0.6375 0.4048 -
DT 0.4813 0.3805 -
KNN 0.5325 0.3528 -
------------- ----------------- ---------------- -------------------
GRU 0.6837 0.5494 -
LSTM 0.6950 0.5967 -
RNN 0.6550 0.4298 -
------------- ----------------- ---------------- -------------------
BART 0.4125 0.4152 -
DeBERTa 0.4025 0.4119 -
------------- ----------------- ---------------- -------------------
GPT-3.5 0.6175 0.6063 0.0000
GPT-4 0.7638 0.7659 0.0000
Gemini-pro 0.7488 0.7519 0.0013
Llama-3-8B 0.7675 0.7710 0.0013
Qwen-7B 0.7550 0.7585 0.0025
Qwen-14B 0.7850 0.7860 0.0050
Vicuna-7B 0.7425 0.7250 0.0000
Vicuna-13B 0.6750 0.6735 0.0013
------------- ----------------- ---------------- -------------------
Gemini-pro(S) 0.6925(-0.056) 0.7217(-0.030) 0.0400(+0.039)
Llama-3-8B(S) 0.7550(-0.012) 0.7585(-0.013) 0.0013(+0.000)
Qwen-7B(S) 0.6837(-0.071) 0.6900(-0.069) 0.0288(+0.026)
Qwen-14B(S) 0.7738(-0.011) 0.7748(-0.011) 0.0063(+0.001)
Vicuna-7B(S) 0.7738(+0.031) 0.7607(+0.036) 0.0000(+0.000)
Vicuna-13B(S) 0.7575(+0.082) 0.7616(+0.088) 0.0013(+0.000)
------------- ----------------- ---------------- -------------------
Llama-3-8B 0.7913(+0.024) 0.7796(+0.009) 0.0000(-0.001)
Qwen-7B(F) 0.8400(+0.085) 0.8302(+0.074) 0.0000(-0.003)

SMS SPAM COLLECTION Classification Results

Model ACC($\uparrow$) F1($\uparrow$) U/E($\downarrow$)
MNB 0.7488 0.6376 -
LR 0.8575 0.5419 -
RF 0.8962 0.7196 -
DT 0.8287 0.6559 -
KNN 0.8237 0.6241 -
------------- ----------------- ---------------- -------------------
GRU 0.9675 0.9257 -
LSTM 0.9675 0.9237 -
RNN 0.9725 0.9366 -
------------- ----------------- ---------------- -------------------
BART 0.7137 0.4943 -
DeBERTa 0.7025 0.5630 -
------------- ----------------- ---------------- -------------------
GPT-3.5 0.4988 0.5601 0.0000
GPT-4 0.9463 0.9495 0.0000
Gemini-pro 0.6500 0.7395 0.0575
Llama-3-8B 0.3937 0.4426 0.0025
Qwen-7B 0.7050 0.7527 0.0013
Qwen-14B 0.9137 0.9208 0.0000
Vicuna-7B 0.2762 0.2847 0.0000
Vicuna-13B 0.4550 0.5149 0.0000
------------- ----------------- ---------------- -------------------
Gemini-pro(S) 0.8163(+0.166) 0.8759(+0.136) 0.0488(-0.009)
Llama-3-8B(S) 0.5825(+0.189) 0.6482(+0.206) 0.0088(+0.006)
Qwen-7B(S) 0.7525(+0.047) 0.8124(+0.060) 0.0362(+0.035)
Qwen-14B(S) 0.8525(-0.061) 0.8730(-0.048) 0.0025(+0.003)
Vicuna-7B(S) 0.5675(+0.291) 0.6310(+0.346) 0.0013(+0.001)
Vicuna-13B(S) 0.6412(+0.186) 0.6976(+0.183) 0.0000(+0.000)
------------- ----------------- ---------------- -------------------
Llama-3-8B(F) 0.9825(+0.589) 0.9826(+0.540) 0.0000(-0.003)
Qwen-7B(F) 0.9938(+0.289) 0.9937(+0.241) 0.0000(+0.000)