From 04afc8266b45dcd402d6f3589a08025b313b0e9b Mon Sep 17 00:00:00 2001 From: prajdabre Date: Mon, 22 Apr 2024 11:53:43 +0900 Subject: [PATCH] Industry track info --- _data/navigation.yml | 4 +- _pages/calls/Industry Track.md | 237 +++++++++++++-------- _pages/calls/Main Conference Papers.md | 284 ------------------------- 3 files changed, 155 insertions(+), 370 deletions(-) diff --git a/_data/navigation.yml b/_data/navigation.yml index 40a2d46..e532cd9 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -139,8 +139,8 @@ calls: url: /calls/main_conference_papers/ # - title: "System Demonstrations" # url: /calls/demos/ - # - title: "Industry Track" - # url: /calls/industry_track/ + - title: "Industry Track" + url: /calls/industry_track/ # - title: "Style Files and Formatting" # url: /calls/style-and-formatting/ # - title: "Volunteers" diff --git a/_pages/calls/Industry Track.md b/_pages/calls/Industry Track.md index b0f4969..1ba8a72 100644 --- a/_pages/calls/Industry Track.md +++ b/_pages/calls/Industry Track.md @@ -10,115 +10,184 @@ toc_icon: "cog" --- ## Overview -EMNLP 2023 Industry Track in Singapore - -Conference: **Dec 6th to Dec 10th, 2023** - -**Paper submission deadline: July 23th, 2023** - -**BACKGROUND** Language technologies and their applications are an integral and critical part of our daily lives. The development of many of these technologies trace their roots to academic and industrial research laboratories where researchers invented a plethora of algorithms, benchmarked them against shared datasets and perfected the performance of these algorithms to provide plausible solutions to real-world applications. The impact of language technology has been significant in recent years, due to the remarkable development of large-scale models. The development of scaling law has increased the intelligence of these models, setting a new benchmark for artificial intelligence. The mutual influence of language technology in academia and industry has culminated in the flourishing of industrial applications. - -## TOPICS -The industry track welcomes submissions describing innovations and implementations of applications in all areas of speech and natural language processing, ranging from early prototypes to production-ready systems. With the rapid development observed recently in Generative Ai and large language models (LLM), this year we are particularly encouraging submissions of papers leveraging this technology to develop real-world applications. We are also welcoming papers describing publicly available open-source or open-access systems. The focus of this track is on papers that advance the understanding of, and demonstrate the effective handling of, practical issues related to the deployment of language processing technologies in real-world systems. There is no requirement that systems are for commercial use or not. This track provides an opportunity to highlight the key insights and new research challenges that arise from real world implementations. Topics include, but are not limited to, the following (in alphabetical order): - -- Best practices and lessons learned -- Continuous maintenance and improvement of deployed systems -- Case studies from design to deployment -- Design of application-relevant datasets -- Ethics and bias in deployed NLP systems -- Handling unexpected user behavior -- Implementation at speed, scale, and low-cost -- Negative results related to real-world applications -- Novel previously unsolved NLP problems -- Offline and online system evaluation methodologies -- Online learning for deployed systems -- System combination and hybridization -- In addition, opinion/vision papers related to real-world applications are also welcome. - - -- **Deployed**: Must describe a system that solves a non-trivial real-world problem. The focus may include describing the problem related to actual use cases, its significance (against opportunity size, value proposition, and ideal end state), design/formulation of methods, tradeoff design decision for solutions, deployment challenges, and lessons learned. -- **Emerging**: Must describe the development of a system that solves a non-trivial real-world problem (it need not be deployed or even close, but there needs to be evidence that this development is intended for real-world deployment). Papers that describe enabling infrastructure for large-scale deployment of natural language processing techniques also fall in this category. -- **Discovery**: Must include results obtained from NLP applications in real world scenarios that result in actionable insights. These discoveries should reveal promising directions in their application areas, leading to further system or societal enhancements. For example, an actionable discovery from an analysis of call center transcripts may reveal that certain language choices negatively impact customer experience, leading to better training of service representatives and improved customer experience. - -## IMPORTANT DATES - - - - - - - - - - - - - - - - - -
Anonymity period beginsJune 16, 2023
Paper submission deadlineJuly 23, 2023
Notification of acceptanceOctober 10, 2023
Camera-ready version of papers dueOctober 24, 2023
-Note: All deadlines are 11:59PM UTC-12:00 (“anywhere on Earth”). - -## EVALUATION AND DECISION CRITERIA -Submissions will be reviewed in a double-blind manner and assessed based on their novelty, technical quality, potential impact, and clarity. Submissions in the industry track should emphasize real-world implementations of natural language processing systems, the development of such systems, or provide insights based on real-world datasets with obvious industry impact. For papers that rely heavily on empirical evaluations, the experimental methods and results should be clear, well executed, and repeatable (though the data may be proprietary). - -## PAPER SUBMISSION -Industry Track Papers submission link +EMNLP 2024 Industry Track (Miami, Florida, USA) + +Main Conference: **12 - 14 November 2024** + +**Paper Submission Deadline: 18 July 2024** + +[Industry Track Papers submission link](https://openreview.net/group?id=EMNLP/2024/Industry_Track) + +## Goal + +Language technologies and their applications are an integral and critical part of our daily lives. The development of many of these technologies trace their roots to academic and industrial research laboratories where researchers invented a plethora of algorithms, benchmarked them against shared datasets and perfected the performance of these algorithms to provide plausible solutions to real-world applications. + +The EMNLP 2024 Industry Track aims to highlight this mutual influence of language technology in academia and industry, which has significantly contributed to the proliferation of industrial applications. The track provides the opportunity for researchers, engineers, practitioners and users to meet and discuss the latest language technologies methods as deployed in a real-world setting and aims to be the premier forum for knowledge sharing across the boundary between academia and industry. + +We acknowledge the unique difficulties encountered when adapting language technologies for building novel and robust real-world applications as the journey from theoretical research to practical deployment is fraught with new challenges. These range from the technical aspects of system deployment and optimizing for efficiency, to making informed design choices or methodological considerations of incorporating human feedback and oversight. + +To provide a forum to address these multifaceted issues, we are seeking submissions that not only delve into research but also demonstrate the application of systems in real-world scenarios, irrespective of whether they involve proprietary data. + +Contributions are invited across all spectrums of language technologies and systems, with a special emphasis on innovations and implementations that hold relevance to real-world applications. We encourage submissions from both non-profit and for-profit sectors, with the understanding that the end-users of these systems extend beyond the NLP community. Please note that if submissions involve proprietary data, there is no requirement to make this data available. + + +## Topics of Interest +The EMNLP 2024 Industry Track provides the opportunity to highlight the key insights and new research challenges that arise from the development and deployment of real-world applications using language technologies. + +Relevant areas include: + +System design, efficiency, maintainability and scalability of real-world applications, with topics in alphabetical order including, but not limited to: +Benchmarks and methods for improving the latency and efficiency of systems +Continuous maintenance and improvement of deployed systems +Efficient methods for training and inference +Enabling infrastructure for large-scale deployment +Human-in-the-Loop approaches to application development +Implementation at speed, scale or low-cost +System combination + +Novel applications and use cases, with topics in alphabetical order including, but not limited to: +Best practices, lessons learned or a vision on deploying real-world applications +Case studies, from design to deployment +Description of an application or system +Design of application-relevant datasets +Development of methods under system constraints (model or data size) +Novel NLP applications + +Methods for deployed systems, with topics in alphabetical order including, but not limited to: +Ethics, bias, fairness and harmlessness +Interpretability +Interactive systems +Offline and online system evaluation methodologies +Online learning +Robustness + +## Important Dates +- Paper submission deadline: July 18, 2024 +- Notification: October 1, 2024 +- Camera-ready version of papers due: October 15, 2024 +- Main conference: 12 - 14 November 2024 + +**Note: All deadlines are 11:59PM UTC-12:00 (“anywhere on Earth”).** + +Following the [ACL and ARR Policies for Review and Citation](https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Review_and_Citation), updated in early 2024, there is no anonymity period requirement, e.g. one may upload the paper to arXiv at any time. + + +## Evaluation Criteria +Submissions will be reviewed in a double-blind manner and assessed based on their novelty, technical quality, potential impact, and clarity. Submissions to the EMNLP 2024 Industry Track should emphasize real-world implementations of natural language processing systems, the development of such systems, or provide insights based on real-world datasets with obvious industry impact. For papers that rely heavily on empirical evaluations, the experimental methods and results should be clear, well executed, and repeatable (though the data may be proprietary). + + +## Paper Submission + +Authors are invited to submit original papers that are not previously published, accepted to be published, or under consideration for publication in any other forum. + -**Submission**: Authors are invited to submit original, full-length (6 page) industry track papers that are not previously published, accepted to be published, or under consideration for publication in any other forum. Manuscripts should be submitted electronically, in PDF format and formatted using EMNLP 2023 formatting requirements. Please do not modify these style files, nor should you use templates designed for other conferences. Submissions that do not conform to the required styles, including paper size, margin width, and font size restrictions, will be rejected without review. +**Submission** + +The papers should be submitted through the [EMNLP 2024 industry track online submission system](https://openreview.net/group?id=EMNLP/2024/Industry_Track). + +**Formatting** + +Manuscripts should be submitted electronically, in PDF format and formatted using the templates available here. Please do not modify these style files, nor should you use templates designed for other conferences. Submissions that do not conform to the required styles, including paper size, margin width, and font size restrictions, will be rejected without review. + +**Length** + +Industry Track papers cannot exceed 6 pages in length; however, references do not count toward the page limit, nor do the following optional sections: acknowledgments (only in the final version), ethical considerations, and appendices. + +**Appendices** + +Appendices need to appear in the main paper’s PDF, after the bibliography. Note that paper submissions need to remain fully self-contained, as these appendices are completely optional, and reviewers are not even asked to review or download them. Appendices may include pre-processing decisions, model parameters, feature templates, pseudocode, sample system inputs/outputs, information about user studies, additional error analysis and details needed for improving the clarity of the experiments described in the paper. + +**Supplementary Materials**: + +Authors may submit separate files as supplementary materials such as software or data. + +**Anonymization for Double-blind Review**: + +Industry Track submissions, their appendices, and supplementary materials must not include authors’ names and affiliations. Self-references that reveal the authors’ identities must be avoided. For example, instead of “We previously showed (Smith, 1991) …” or even “We previously showed (Anonymous, 1991) …”, please use “Smith previously showed (Smith, 1991) …”. Authors should also be careful not to reveal their affiliation indirectly, for example through screenshots or trade names. + +The submissions should avoid links to non-anonymized repositories: the code should be either submitted as supplementary material, or as a link to an anonymized repository (e.g., Anonymous GitHub or Anonym Share). Please avoid any links to storage services like Dropbox (which may track the reviewers downloading the resources). + +Papers that do not conform to these requirements will be rejected without review. + +**Citation and Comparison**: + +Authors are expected to cite all refereed publications relevant to their submission but may be excused for not knowing about all unpublished work (especially work that has been recently posted and/or is not widely cited). + +In cases where a preprint has been superseded by a refereed publication, the refereed publication should be cited in addition to or instead of the preprint version. + +Papers (whether refereed or not) appearing less than 3 months before the submission deadline are considered contemporaneous to a submission, and authors are therefore not obliged to make detailed comparisons that require additional experimentation and/or in-depth analysis. + +For more information, see the [ACL Policies for Review and Citation](https://www.aclweb.org/adminwiki/index.php?title=ACL_Policies_for_Review_and_Citation). + +**Writing Assistance**: + +The EMNLP 2024 Industry Track adheres to the ACL policy on using writing assistants available [here](https://2023.aclweb.org/blog/ACL-2023-policy/). + +**Camera Ready Version**: + +Accepted papers will be given one additional page of content (up to 7 pages; ethical considerations, acknowledgments and references do not count against this limit) so that reviewers’ comments can be considered. Previous presentations of the work (e.g. preprints on arXiv.org) should be indicated in a footnote that should be excluded from the review submission, but included in the final version of papers appearing in the EMNLP 2024 proceedings. + +The final version should remove anonymization in text, citation, and figures. For example, the final version may include the name of the authors’ institutions, trade names, and screenshots of identifiable products. + +In addition to the final version of the paper, authors of all papers accepted to the Industry Track will be given an option to submit a presentation video, slides, and/or a pdf of their poster to the ACL Anthology. + +**Presentation Requirement for Accepted Papers**: + +EMNLP 2024 Industry track papers will be presented orally or as posters as determined by the program committee. All accepted papers must be presented at the conference (either via online or onsite presence). At least one author of each accepted paper must register for EMNLP 2024. The EMNLP 2024 Industry Track will run in parallel with the Main Conference Track. + +**Presentation Mode**: -Industry Track papers cannot exceed 6 pages in length (excluding ethical considerations and references). The papers can have an optional appendix, which will appear in the main paper’s PDF, after the bibliography. For example, pre-processing decisions, model parameters, feature templates, pseudocode, sample system inputs/outputs, information about user studies, additional errors analysis and other details that are necessary for the replication of the work described in the paper can be put into appendices. Note however, the paper submissions need to remain fully self-contained, as these supplementary materials are completely optional, and reviewers are not even asked to review or download them. +Accepted papers will be presented orally or as posters as determined by the program committee. The decisions as to which papers will be presented orally and which as poster presentations will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between papers presented orally and as posters. -The papers should be submitted through the EMNLP 2023 industry track online submission system. +**Authorship**: -**Final version**: Accepted papers will be given one additional page of content (up to 7 pages; ethical considerations, acknowledgements and references do not count against this limit) so that reviewers’ comments can be taken into account. Previous presentations of the work (e.g. preprints on arXiv.org) should be indicated in a footnote that should be excluded from the review submission, but included in the final version of papers appearing in the EMNLP 2023 proceedings. +The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. Each author listed on a submission to EMNLP 2024 Industry Track will be notified of submissions and the final decision. No changes to the order or composition of authorship may be made to submissions to EMNLP 2024 Industry Track after the paper submission deadline. -Presentation requirement for accepted papers: Industry track papers will be presented orally or as posters to be determined by the program committee. All accepted papers must be presented at the conference (either via online or onsite presence). At least one author of each accepted paper must register for EMNLP 2023 by the early registration deadline. The 2023 Industry Track will run in parallel with the Research Track. -**Presentation Mode**: Accepted papers will be presented orally or as posters as determined by the program committee. The decisions as to which papers will be presented orally and which as poster presentations will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between papers presented orally and as posters. +## Multiple Submissions Policy -**Authorship**: The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. Each author listed on a submission to EMNLP 2023 Industry Track will be notified of submissions and the final decision. No changes to the order or composition of authorship may be made to submissions to EMNLP 2023 Industry Track after the paper submission deadline. +EMNLP 2024 Industry Track will not consider any paper that is under review in a journal or another conference at the time of submission, and submitted papers must not be submitted elsewhere during the EMNLP 2024 Industry Track review period. This policy covers all refereed and archival conferences and workshops (e.g., NeurIPS, ACL workshops), as well as ARR. In addition, we will not consider any paper that overlaps significantly in content or results with papers that will be (or have been) published elsewhere. Authors submitting more than one paper to the EMNLP 2024 Industry Track must ensure that their submissions do not overlap significantly (>25%) with each other in content or results. -## ANONYMITY PERIOD +Submissions of identical or closely related work to multiple EMNLP 2024 tracks (e.g. main conference or Industry Track) will be treated as duplicate submissions. Such submissions violating our multiple submission policy will be rejected without review. The authors should also include the papers that their paper overlaps or extends in the references section as follows: Anonymous Authors, “Title of the paper”, Under submission at EMNLP 2024 (TRACK NAME). -The anonymity period for the EMNLP 2023 Industry Track is from June 16, 2023 to September 26, 2023. You may not make a non-anonymized version of your paper available online to the general community (for example, via a preprint server) during the anonymity period. You may not update the non-anonymized version during the anonymity period, and we ask you not to advertise it on social media or take other actions that would further compromise double-blind reviewing during the anonymity period. +## Ethics Policy -## MULTIPLE SUBMISSION POLICY -EMNLP 2023 will not consider any paper that is under review in a journal or another conference at the time of submission, and submitted papers must not be submitted elsewhere during the EMNLP 2023 review period. This policy covers all refereed and archival conferences and workshops (e.g., NeurIPS, ACL workshops), as well as ARR. In addition, we will not consider any paper that overlaps significantly in content or results with papers that will be (or have been) published elsewhere. Authors submitting more than one paper to EMNLP 2023 must ensure that their submissions do not overlap significantly (>25%) with each other in content or results. +Authors are required to honor the ethical code set out in the [ACL Code of Ethics](https://www.aclweb.org/portal/content/acl-code-ethics). The consideration of the ethical impact of our research, use of data, and potential applications of our work has always been an important consideration, and as artificial intelligence is becoming more mainstream, these issues are increasingly pertinent. We ask that all authors read the code, and ensure that their work is conformant to this code. Where a paper may raise ethical issues, we ask that you include in the paper an explicit discussion of these issues, which will be taken into account in the review process. We reserve the right to reject papers on ethical grounds, where the authors are judged to have operated counter to the code of ethics, or have inadequately addressed legitimate ethical concerns with their work. +Authors will be allowed extra space after the 6th page for an optional broader impact statement or other discussion of ethics. The EMNLP 2024 Industry Track review form will include a section addressing these issues and papers flagged for ethical concerns by reviewers or ACs will be further reviewed by an ethics committee. Note that an ethical considerations section is not required, but papers working with sensitive data or on sensitive tasks that do not discuss these issues will not be accepted. Conversely, the mere inclusion of an ethical considerations section does not guarantee acceptance. In addition to acceptance or rejection, papers may receive a conditional acceptance recommendation. Camera-ready versions of papers designated as conditional accept will be re-reviewed by the ethics committee to determine whether the concerns have been adequately addressed. Please read the ethics FAQ for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics. -Submissions of identical or closely related work to multiple EMNLP 2023 tracks (e.g. to research track and industry track) will be treated as duplicate submissions. Such submissions violating our multiple submission policy will be rejected without review. The authors should also include the papers that their paper overlaps or extends in the references section as follows: *Anonymous Authors, “Title of the paper”, Under submission at EMNLP 2023 (TRACK NAME)* +## Frequently Asked Questions -## ETHICS POLICY -Authors are required to honor the ethical code set out in the ACL [Code of Ethics](https://www.aclweb.org/portal/content/acl-code-ethics). The consideration of the ethical impact of our research, use of data, and potential applications of our work has always been an important consideration, and as artificial intelligence is becoming more mainstream, these issues are increasingly pertinent. We ask that all authors read the code, and ensure that their work is conformant to this code. Where a paper may raise ethical issues, we ask that you include in the paper an explicit discussion of these issues, which will be taken into account in the review process. We reserve the right to reject papers on ethical grounds, where the authors are judged to have operated counter to the code of ethics, or have inadequately addressed legitimate ethical concerns with their work. +**Is the EMNLP 2024 Industry Track only for participants from industry?** -Authors will be allowed extra space after the 6th page for an optional broader impact statement or other discussion of ethics. The EMNLP review form will include a section addressing these issues and papers flagged for ethical concerns by reviewers or ACs will be further reviewed by an ethics committee. Note that an ethical considerations section is not required, but papers working with sensitive data or on sensitive tasks that do not discuss these issues will not be accepted. Conversely, the mere inclusion of an ethical considerations section does not guarantee acceptance. In addition to acceptance or rejection, papers may receive a conditional acceptance recommendation. Camera-ready versions of papers designated as conditional accept will be re-reviewed by the ethics committee to determine whether the concerns have been adequately addressed. Please read the [ethics FAQ](https://2022.emnlp.org/call-for-papers/ethics-faq) for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics. +No, the industry track welcomes participants from the entire ACL community. Researchers working on real-world applications that match the industry track call for papers are invited to submit papers. Everyone is welcome to attend industry track sessions. -## CONTACT INFORMATION -emnlp2023-industry-track@googlegroups.com +**Can academics and students also submit papers to industry track?** -**Industry Track Co-Chairs** +Yes. Consider submitting a paper to the industry track if your work matches the industry track call for papers. -- [Imed Zitouni](https://www.linkedin.com/in/imed-zitouni-0829) (Google) +**I work in industry. Can I still submit my paper to the main conference track?** -- [Mingxuan Wang](https://mingxuan.github.io/) (Bytedance) +Absolutely. The industry track offers a forum to submit papers describing aspects of real-world applications that may differ in focus from the main conference track reviewing criteria. -**General chair**: [Yuji Matsumoto](https://www.riken.jp/en/research/labs/aip/goalorient_tech/knowl_acqui/index.html), RIKEN Center for Advanced Intelligence Project +**What do you mean by real-world applications?** +Real-world applications those deployed or expected to be deployed for real-world use, i.e. outside controlled environments such as laboratories, classrooms or experimental crowd-sourced setups. -## FREQUENTLY ASKED QUESTIONS -**Is the industry track only for participants from industry?** No, the industry track welcomes participants from the entire ACL community. Researchers working on real-world applications that match the industry track call for papers are invited to submit papers. Everyone is welcome to attend industry track sessions. +**Will the papers in the industry track be published in the proceedings?** -**What do you mean by real-world applications?** We are looking for applications that are deployed (or expected to be deployed) for real-world use, i.e. outside controlled environments such as laboratories, classrooms or experimental crowd-sourced setups. +Yes, industry track papers will be published as a separate volume of the proceedings. For example, see the [EMNLP 2023 Industry Track proceedings](https://aclanthology.org/events/emnlp-2023/#2023emnlp-industry). -**Can students also submit papers to industry track?** Yes! If your work matches the industry track call for papers, consider submitting a paper to the industry track. +**How do I decide whether to submit to the main conference or the industry track?** -**I work in industry. Can I still submit my paper to the research track?** Absolutely! There are no changes to the main conference submissions. The industry track offers a forum to submit papers describing aspects of real-world applications that may differ in focus from the research track reviewing criteria. +Authors are advised to review all calls for papers and submit to the call that best matches your work. The list of topics and reviewing criteria may be helpful. For example, papers best suited for the industry track include those describing language technology systems, key lessons learned or challenges from their deployment or experiments on proprietary data. -**Will the papers in the industry track be published in the proceedings?** Yes, industry track papers will be published as a separate volume of the proceedings. For example, see the EMNLP 2022 proceedings. +If you still need help deciding, feel free to reach out to the track chairs. -**How do I decide whether to submit to the research track or the industry track?** Papers describing key lessons learned and challenges pertaining to real-world deployment of NLP and speech technologies are best suited for industry track. Authors are advised to review the call for papers for both tracks and submit to the track that best matches your work. The list of topics and reviewing criteria may be helpful. You can also reach out to the track chairs if you need help deciding. +## Industry Track Chairs +- [Franck Dernoncourt](http://francky.me/) (Adobe Research) +- [Daniel Preoțiuc-Pietro](https://www.preotiuc.ro/) (Bloomberg) +- [Anastasia Shimorina](https://openreview.net/profile?id=~Anastasia_Shimorina1) (Orange) +Contact: [emnlp2024-industry-track@googlegroups.com](mailto:emnlp2024-industry-track@googlegroups.com) diff --git a/_pages/calls/Main Conference Papers.md b/_pages/calls/Main Conference Papers.md index ffba6a4..7a3fe99 100644 --- a/_pages/calls/Main Conference Papers.md +++ b/_pages/calls/Main Conference Papers.md @@ -128,287 +128,3 @@ __Program Chairs:__ For questions related to paper submission, and the review process in general, email: For questions about commitment and post review related topics:: - - - - -