Environmentally-Friendly Natural Language Processing and Tips for Detecting the Use of AI-Generated Text

Few researchers have access to the resources needed to train the state-of-the-art language models (LMs) used in cutting-edge technologies. Processing “big data” over computational frameworks and expensive GPUs, there are substantial environmental implications: in 2019, one team [1] of researchers estimated that 626,000 pounds of carbon dioxide were produced from the costs associated to producing one model’s parameters (GPT-2’s)—the lifetime emissions of approximately five cars. Its developers, OpenAI, reported in 2018 that “since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time” [2]. After OpenAI released GPT-3 in 2021, a report estimated that by using “10,000 GPUs and 400 gigabits per second of network connectivity per server”, the months it took to process “45 Terabytes of text data from all over the internet” means that “GPT-3 could have easily cost 10 or 20 million dollars to train” [3]. 

Staring down this trend in 2018, OpenAI even suggested: “it’s worth preparing for the implications of systems far outside today’s capabilities” [2]. 

This talk will demonstrate software aspiring to address the profound need for more efficient systems. The presented NLP framework operates shallow neural networks that are optimized locally, using only a single pass over training data, and without the need for gradient descent—the de facto standard algorithm, underlying training processes for most modern applied AI technologies. Like GPT-2, the presented NLP framework allows for pre-training on vast quantities of text to produce large generative LMs that operate quickly on GPUs; and, unlike GPT-2–3, these pre-training processes can easily be parallelized across cheap CPU-based resources for low-emissions NLP. Our framework is in early stages of development [4] and is based on some of the conclusions drawn from unpublished discoveries [5] that extend from the same statistical theories developed by the presenter to detect when AI, or bots, generate text [6–10]. 

A review of tips and tricks used to identify bots will then be followed by a Q/A session, likely to consider ChatGPT’s implications. 

References 

[1] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645– 3650 Florence, Italy, July 28 – August 2, 2019. 

[2] OpenAi, “Ai and compute,” Blog Post: https://openai.com/blog/ai-and-compute/, 2018.

[3] A. Kurenkov, “Gpt-3 is no longer the only game in town,” Blog Post: https://lastweekin.ai/p/gpt-3-is-no-longer-the-only-game, 2021. 

[4] J. R. Williams, “It’s a Machine and Natural,” Github: https://github.com/jakerylandwilliams/IaMaN/, October 2022. 

[5] J. R. Williams and H. S. Heidenreich, “To know by the company words keep and what else lies in the vicinity,” 2022. [Online]. Available: https://arxiv.org/abs/2205.00148 

[6] E. M. Clark, J. R. Williams, C. A. Jones, R. A. Galbraith, C. M. Danforth, P. S. Dodds, “Sifting robotic from organic text: A natural language approach for detecting automation on Twitter.” 2016. Journal of Computational Science. 

[7] E. M. Clark, C. A. Jones, J. R. Williams, A. N. Kurti, M. C. Norotsky, C. M. Danforth, P. S. Dodds, “Vaporous Marketing: Uncovering Pervasive E-Cigarette Advertisements on Twitter.” 2016. PLoS.

[8] G. C. Santia, M. I. Mujib, and J. R. Williams. “Detecting Social Bots on Facebook in an Information Veracity Context.” 2019. Proc. of the 13th International AAAI Conference on Web and Social Media

[9] H. S. Heidenreich, M. I. Mujib, and J. R. Williams. “Investigating Coordinated ‘Social’ Targeting of High Profile Twitter Accounts.” 2020. [Online]. Available: https://arxiv.org/abs/2008.02874

[10] H. S. Heidenreich and J. R. Williams, “The Earth Is Flat and the Sun Is Not a Star: The Susceptibility of GPT-2 to Universal Adversarial Triggers.” 2021. Proceedings of the Fourth International AAAI/ACM Conference on Artificial Intelligence, Ethics and Society