FROM CITATION COUNTS TO COMPUTATIONAL SCIENCE: METHODS, DATA, AND APPLICATIONS IN SCIENTOMETRICS
DOI:
https://doi.org/10.59787/2413-5488-2026-55-3-93-107Keywords:
scientometrics, bibliometrics, citation analysis, research evaluation, data infrastructures, openalex, science mapping, altmetrics, machine learning, natural language processing, citation normalization, open science, knowledge graph, scholarly communication, computational scientometrics.Abstract
In the last 60 years, the science of scientometrics, the quantitative analysis of science as a communication system, has developed from an esoteric interest in citation indexing to a key tool for research policy and evaluation. The aim of this review is to trace and summarise that trajectory under the single argument that scientometric indicators are only as meaningful as the data infrastructures on which they are built, and the evaluative contexts in which they are applied. We present the literature in five methodological layers, linked by a common thread on research evaluation and policy, and spanning bibliographic data sources, citation indicators and normalisation, science mapping and visualisation, altmetrics and the widening of impact, as well as the recent computational turn based on machine learning and natural language processing. Along with this narrative review, we present a companion bibliometric analysis, mapping the field by topic within the OpenAlex database, which reveals an analytic corpus of 37,166 works, whose growth we illustrate, whose core journals we identify, and whose thematic structure we discuss as a characterization of the field from a methodological perspective. The review finds that the best lasting gains in the field have been in the quality of data coverage, data normalisation and in the responsible interpretation of indicators, and the current transition from counting to understanding indicators should be subject to the same standards.
References
Abramo, G., Cicero, T., & D’Angelo, C. A. (2026). Data source effects in research performance assessment of individuals and institutions: Comparing OpenAlex with established bibliographic databases. Scientometrics, 131(5), 3411–3441. https://doi.org/10.1007/s11192-026-05638-6
Ahmed, B., Wang, L., Al-Shamayleh, A. S., Afzal, M. T., Mustafa, G., Alrawagfeh, W., & Akhunzada, A. (2023). Machine learning approach for effective ranking of researcher assessment parameters. IEEE Access, 11, 133294–133312. https://doi.org/10.1109/ACCESS.2023.3336950
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1), 2158244019829575. https://doi.org/10.1177/2158244019829575
Baccini, A., De Nicolao, G., & Petrovich, E. (2019). Citation gaming induced by bibliometric evaluation: A country-level comparative analysis. PLOS ONE, 14(9), e0221212. https://doi.org/10.1371/journal.pone.0221212
Bornmann, L., & Haunschild, R. (2017). An empirical look at the Nature Index. Journal of the Association for Information Science and Technology, 68(3), 653–659. https://doi.org/10.1002/asi.23682
Cohan, A., Feldman, S., Beltagy, I., Downey, D., & Weld, D. S. (2020). SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 2270–2282). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.207
Culbert, J. H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M., Donner, P., & Mayr, P. (2025). Reference coverage analysis of OpenAlex compared to Web of Science and Scopus. Scientometrics, 130(4), 2475–2492. https://doi.org/10.1007/s11192-025-05293-3
Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133, 285–296. https://doi.org/10.1016/j.jbusres.2021.04.070
Ellegaard, O. (2026). The method of bibliometric analysis: A critical evaluation. Scientometrics, 131, 4175–4196. https://doi.org/10.1007/s11192-026-05670-6
Faghri, A., & Bergman, T. L. (2024). Highly ranked scholars and the influence of countries and regions in research fields, disciplines, and specialties. Quantitative Science Studies, 5(2), 464–483. https://doi.org/10.1162/qss_a_00291
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., Vespignani, A., Waltman, L., Wang, D., & Barabási, A.-L. (2018). Science of science. Science, 359(6379), eaao0185. https://doi.org/10.1126/science.aao0185
Gagliardi, A. R., Chen, R. H., Boury, H., Albert, M., Chow, J., DaCosta, R. S., Deyell, R., Finelli, A., Keating, A., Légaré, F., McGowan, J., McLeod, R., Moore, A., Parulekar, W., Saginur, R., Straus, S. E., & Protze, S. (2023). DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey. PLOS ONE, 18(5), e0270616. https://doi.org/10.1371/journal.pone.0270616
Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108–111. https://doi.org/10.1126/science.122.3159.108
Genc, H. N., & Kocak, N. (2024). Bibliometric analysis of studies on artificial intelligence in science education with VOSviewer. Journal of Education in Science, Environment and Health, 10(4), 183–195. https://doi.org/10.55549/jeseh.756
González, P., Fors, M., & Torres, A. (2025). Altmetrics in the evaluation of scholarly impact: A systematic and critical literature review. Frontiers in Research Metrics and Analytics, 10, 1693304. https://doi.org/10.3389/frma.2025.1693304
Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv. https://doi.org/10.48550/arXiv.2203.05794
Hassan, S. U., Saleem, A., Soroya, S. H., Safder, I., Iqbal, S., Jamil, S., Bukhari, F., Aljohani, N. R., & Nawaz, R. (2021). Sentiment analysis of tweets through Altmetrics: A machine learning approach. Journal of Information Science, 47(6), 712–726. https://doi.org/10.1177/0165551520930917
Haunschild, R., & Bornmann, L. (2024). The use of OpenAlex to produce meaningful bibliometric global overlay maps of science on the individual, institutional, and national levels. PLOS ONE, 19(12), e0308041. https://doi.org/10.1371/journal.pone.0308041
Hendricks, G., Tkaczyk, D., Lin, J., & Feeney, P. (2020). Crossref: The sustainable source of community-owned scholarly metadata. Quantitative Science Studies, 1(1), 414–427. https://doi.org/10.1162/qss_a_00022
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102
Hood, W. W., & Wilson, C. S. (2001). The literature of bibliometrics, scientometrics, and informetrics. Scientometrics, 52(2), 291–314. https://doi.org/10.1023/A:1017919924342
Hook, D. W., Porter, S. J., & Herzog, C. (2018). Dimensions: Building context for search and evaluation. Frontiers in Research Metrics and Analytics, 3, 23. https://doi.org/10.3389/frma.2018.00023
Ioannidis, J. P. A., Boyack, K. W., & Wouters, P. F. (2016). Citation metrics: A primer on how (not) to normalize. PLOS Biology, 14(9), e1002542. https://doi.org/10.1371/journal.pbio.1002542
Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D. (2018). Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6, 391–406. https://doi.org/10.1162/tacl_a_00028
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Klavans, R., & Boyack, K. W. (2017). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? Journal of the Association for Information Science and Technology, 68(4), 984–998. https://doi.org/10.1002/asi.23734
Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators—A state-of-the-art report. Scientometrics, 109(3), 2129–2150. https://doi.org/10.1007/s11192-016-2150-8
Lu, X., Li, L., & Shen, Z. (2026). Revisiting the field normalization approaches and practices. Scientometrics. https://doi.org/10.1007/s11192-026-05676-0
Lyu, D., Ruan, X., Xie, J., & Cheng, Y. (2021). The classification of citing motivations: A meta-synthesis. Scientometrics, 126(4), 3243–3264. https://doi.org/10.1007/s11192-021-03908-z
Martín-Martín, A., Costas, R., van Leeuwen, T., & Delgado López-Cózar, E. (2018). Evidence of open access of scientific publications in Google Scholar: A large-scale analysis. Journal of Informetrics, 12(3), 819–841. https://doi.org/10.1016/j.joi.2018.06.012
Molldrem, S., Moses, J., Smith, E., Tumilty, E., & Farroni, J. (2025). Metrics producing science: The evaluative way in the emergence of US translational science. Science & Technology Studies. https://doi.org/10.23987/sts.145702
Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. https://doi.org/10.1007/s11192-015-1765-5
Moral-Muñoz, J. A., Herrera-Viedma, E., Santisteban-Espejo, A., & Cobo, M. J. (2020). Software tools for conducting bibliometric analysis in science: An up-to-date review. El Profesional de la Información, 29(1), e290103. https://doi.org/10.3145/epi.2020.ene.03
Omer, A. A. A., Issa, I. I. M., Abuker, Y. Y. A., & Asad, S. (2025). Evaluating the use of VOSviewer in bibliometric and science-mapping studies: Trends, current state and future directions. International Journal of Current Research in Science, Engineering & Technology, 8(4), 483–497. https://doi.org/10.30967/IJCRSET/Altyeb-Ali-Abaker-Omer/208
Ortega, J. L. (2019). Exploratory analysis of Publons metrics and their relationship with bibliometric and altmetric impact. Aslib Journal of Information Management, 71(1), 124–136. https://doi.org/10.1108/AJIM-06-2018-0153
Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West, J., & Haustein, S. (2018). The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles. PeerJ, 6, e4375. https://doi.org/10.7717/peerj.4375
Radhakrishnan, S., Erbis, S., Isaacs, J. A., & Kamarthi, S. (2017). Novel keyword co-occurrence network-based methods to foster systematic reviews of scientific literature. PLOS ONE, 12(3), e0172778. https://doi.org/10.1371/journal.pone.0172778
Saeidnia, H. R., Hosseini, E., Abdoli, S., & Ausloos, M. (2025). Unleashing the power of AI: A systematic review of cutting-edge techniques in AI-enhanced scientometrics, webometrics and bibliometrics. Library Hi Tech, 43(4–5), 1292–1314. https://doi.org/10.1108/LHT-10-2023-0514
Safder, I., Ali, M., Aljohani, N. R., Nawaz, R., & Hassan, S. U. (2023). Neural machine translation for in-text citation classification. Journal of the Association for Information Science and Technology, 74(10), 1229–1240. https://doi.org/10.1002/asi.24817
Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037–2062. https://doi.org/10.1002/asi.23833
Taylor, M. (2026). Evaluating open access advantages for citations and altmetrics (2011–2021): A dynamic and evolving relationship. Quantitative Science Studies, 7, 643–670. https://doi.org/10.1162/QSS.a.470
Tshitoyan, V., Dagdelen, J., Weston, L., Dunn, A., Rong, Z., Kononova, O., Persson, K. A., Ceder, G., & Jain, A. (2019). Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763), 95–98. https://doi.org/10.1038/s41586-019-1335-8
Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful citations. In AAAI Workshop: Scholarly Big Data (Vol. 15, p. 13).
van Eck, N. J., & Waltman, L. (2017). Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics, 111(2), 1053–1070. https://doi.org/10.1007/s11192-017-2300-7
Visser, M., van Eck, N. J., & Waltman, L. (2021). Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. Quantitative Science Studies, 2(1), 20–41. https://doi.org/10.1162/qss_a_00112
Wahle, J. P., Ruas, T., Mohammad, S., & Gipp, B. (2022). D3: A massive dataset of scholarly metadata for analyzing the state of computer science research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 2642–2651). European Language Resources Association. https://doi.org/10.63317/3hvgsmuea6o7
Waltman, L., & van Eck, N. J. (2019). Field normalization of scientometric indicators. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 281–300). Springer International Publishing. https://doi.org/10.1007/978-3-030-02511-3_11
Wang, X., Gu, P., & Shehzadi, K. (2026). Global open science research in the digital & intelligent era: Advances, focuses and prospects. Scientometrics. 10.1007/s11192-026-05597-y
Weimer, V., Heck, T., van Leeuwen, T., & Rittberger, M. (2023). The quantification of open scholarship—A mapping review. Quantitative Science Studies, 4(3), 650–670. https://doi.org/10.1162/qss_a_00266
Zhang, L., Cao, Z., Shang, Y., Sivertsen, G., & Huang, Y. (2024). Missing institutions in OpenAlex: Possible reasons, implications, and solutions. Scientometrics, 129(10), 5869–5891. https://doi.org/10.1007/s11192-023-04923-y
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Scientific and analytical journal "Higher school of Kazakhstan"
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
