A comprehensive survey on applications of transformers for deep learning tasks

Basit Öğe Kaydı
dc.authoridBentahar, Jamal/0000-0002-3136-4849
dc.contributor.authorIslam, Saidul
dc.contributor.authorElmekki, Hanae
dc.contributor.authorElsebai, Ahmed
dc.contributor.authorBentahar, Jamal
dc.contributor.authorDrawel, Nagat
dc.contributor.authorRjoub, Gaith
dc.contributor.authorPedrycz, Witold
dc.date.accessioned2024-05-19T14:47:04Z
dc.date.available2024-05-19T14:47:04Z
dc.date.issued2024
dc.departmentİstinye Üniversitesien_US
dc.description.abstractTransformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer -based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi -modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology.en_US
dc.description.sponsorshipNatural Sciences and Engineering Research Council of Canada (NSERC); NSERC through the Horizon Granten_US
dc.description.sponsorshipJamal Bentahar and Witold Pedrycz would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for their financial support through Discovery Grant. Jamal Bentahar is also supported by NSERC through the Horizon Grant.en_US
dc.identifier.doi10.1016/j.eswa.2023.122666
dc.identifier.issn0957-4174
dc.identifier.issn1873-6793
dc.identifier.scopus2-s2.0-85183596144en_US
dc.identifier.scopusqualityQ1en_US
dc.identifier.urihttps://doi.org10.1016/j.eswa.2023.122666
dc.identifier.urihttps://hdl.handle.net/20.500.12713/5645
dc.identifier.volume241en_US
dc.identifier.wosWOS:001125930700001en_US
dc.identifier.wosqualityN/Aen_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScopusen_US
dc.language.isoenen_US
dc.publisherPergamon-Elsevier Science Ltden_US
dc.relation.ispartofExpert Systems With Applicationsen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.snmz20240519_kaen_US
dc.subjectTransformeren_US
dc.subjectSelf-Attentionen_US
dc.subjectDeep Learningen_US
dc.subjectNatural Language Processing (Nlp)en_US
dc.subjectComputer Vision (Cv)en_US
dc.subjectMulti-Modalityen_US
dc.titleA comprehensive survey on applications of transformers for deep learning tasksen_US
dc.typeReview Articleen_US

Dosyalar

Koleksiyon

WoS İndeksli Yayınlar Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu