Sparsity in transformers: A systematic literature review

Basit Öğe Kaydı
dc.contributor.authorFarina, M.
dc.contributor.authorAhmad, U.
dc.contributor.authorTaha, A.
dc.contributor.authorYounes, H.
dc.contributor.authorMesbah, Y.
dc.contributor.authorYu, X.
dc.contributor.authorPedrycz W.
dc.date.accessioned2024-05-19T14:33:41Z
dc.date.available2024-05-19T14:33:41Z
dc.date.issued2024
dc.departmentİstinye Üniversitesien_US
dc.description.abstractTransformers have become the state-of-the-art architectures for various tasks in Natural Language Processing (NLP) and Computer Vision (CV); however, their space and computational complexity present significant challenges for real-world applications. A promising approach to address these issues is the introduction of sparsity, which involves the deliberate removal of certain parameters or activations from the neural network. In this systematic literature review, we aimed to provide a comprehensive overview of current research on sparsity in transformers. We analyzed the different sparsity techniques applied to transformers, their impact on model performance, and their efficiency in terms of time and space complexity. Moreover, we identified the major gaps and challenges in the existing literature. Our study also highlighted the importance of investigating sparsity in transformers for computational efficiency, reduced resource requirements, scalability, environmental impact, and hardware-algorithm co-design. By synthesizing the current state of research on sparsity in transformer-based models, we also provided valuable insights into their efficiency, impact on model performance, and potential trade-offs, contributing to advancing the field further. © 2024 Elsevier B.V.en_US
dc.identifier.doi10.1016/j.neucom.2024.127468
dc.identifier.issn0925-2312
dc.identifier.scopus2-s2.0-85188146827en_US
dc.identifier.scopusqualityQ1en_US
dc.identifier.urihttps://doi.org/10.1016/j.neucom.2024.127468
dc.identifier.urihttps://hdl.handle.net/20.500.12713/4306
dc.identifier.volume582en_US
dc.indekslendigikaynakScopusen_US
dc.language.isoenen_US
dc.publisherElsevier B.V.en_US
dc.relation.ispartofNeurocomputingen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccessen_US
dc.snmz20240519_kaen_US
dc.subjectComputer Visionen_US
dc.subjectNatural Language Processingen_US
dc.subjectSparsityen_US
dc.subjectSystematic Literature Reviewen_US
dc.subjectTransformersen_US
dc.titleSparsity in transformers: A systematic literature reviewen_US
dc.typeShort Surveyen_US

Dosyalar

Koleksiyon

Scopus İndeksli Yayınlar Koleksiyonu
Bilgisayar Mühendisliği Bölümü Diğer Yayınlar Koleksiyonu
WoS İndeksli Yayınlar Koleksiyonu