Google AI Introduces 4Ms Approach to Reduce Carbon Emission of Machine Learning Models

The total quantity of greenhouse gas emissions generated by anything — a person, organization, event, or product – is known as the carbon footprint. The processes producing more carbon footprint use more resources, generating more greenhouse gases and causing greater climate change. Contributing to the small decrease of greenhouse gas emissions can reduce a great amount of overall carbon footprint. 

With the rising popularity of Machine learning (ML) applications, there are constant concerns about the rising carbon footprint because of increased computation costs. These concerns highlight the need for precise data to determine the true carbon footprint, which can assist in identifying solutions to reduce ML’s carbon emissions.

A recent Google study looks into operational carbon emissions from the training of natural language processing (NLP) models, such as the energy cost of operating ML hardware, including data center overheads. It looks into best practices that could lower the carbon footprint.

The team presents four essential methods that significantly reduce machine learning workloads’ carbon (and energy) footprint. These methods are currently in use at Google and are available to anyone utilizing Google Cloud services. Google uses renewable energy sources to meet 100% of its operating energy needs. Google has pledged to decarbonize all energy usage by 2030, to operate on 100 percent carbon-free energy 24 hours a day.

The 4Ms: Energy and Carbon Footprint Reduction Best Practices are as follows:

1. Model: The researchers state that selecting efficient ML model architectures is crucial as it has the potential of improving ML quality while cutting computation time in half.
2. Machine: Compared to general-purpose processors, using processors and systems specialized for ML training can enhance performance and energy efficiency by 2x–5x.
3. Mechanization: Mostly, on-premise data centers are older and smaller. So the expense of new energy-efficient cooling and power distribution systems cannot be amortized. Cloud-based data centers are brand-new, custom-designed warehouses with energy-efficiency features for 50,000 servers. They offer exceptionally efficient power utilization (PUE). Therefore, computing in the cloud rather than on-premise saves 1.4–2 times more energy and cuts pollution. 
4. Optimization of the map. Furthermore, the cloud allows clients to choose the area with the cleanest energy, resulting in a 5x–10x reduction in gross carbon footprint. 

Google’s total energy use rises every year, which is unsurprising given the increased use of its services. This has increased ML workloads significantly, as has the amount of computation per training run. The 4Ms — improved models, ML-specific hardware, and efficient data centers — substantially offset this load increase. Google’s data demonstrate that machine learning training and inference account for only 10%–15% of Google’s overall energy use during the last three years, with each year split 35 percent for inference and 25 percent for training.

To find improved machine learning models, Google employs neural architecture search (NAS). NAS is often conducted only once per problem domain/search space combination. The resulting model can then be reused for hundreds of applications. For example, the Evolved Transformer model discovered using NAS is open-sourced and freely available. The one-time cost of NAS is generally more than offset by emission reductions from continuing use, as the improved model discovered by NAS is often more efficient.

Other researchers conducted a study to look at training the Transformer model. For this, they used an Nvidia P100 GPU in a typical data center with an energy mix similar to the global average. The Primer model, which was just released, decreases the computation required to obtain the same precision by 4x. Using newer-generation ML hardware, such as TPUv4, improves performance by 14 times over the P100, for a total of 57 times. Efficient cloud data centers result in an 83-fold reduction in total energy consumption by saving 1.4 times more energy than the average data center. Furthermore, a data center powered by a low-carbon energy source can cut carbon emissions by another 9x, resulting in a total reduction of 747x in four years.

The Google team believes that the lifecycle costs of manufacturing computing equipment of all types and sizes are much more likely than the operational cost of ML training in the information technology sector. The manufacturing costs of emission estimates include the embedded carbon emitted from manufacturing all components involved, from chips to data center buildings.

In addition to using the 4Ms method, service providers and users can take simple measures to improve their carbon footprint performance:

• Customers should analyze and reduce their energy use and carbon footprint by having data center providers report data center efficiency and the cleanliness of the energy supply per location.
• Engineers should train models on the fastest processors in the greenest data centers, which are increasingly on the cloud.
• Researchers in machine learning should focus on designing more efficient models, such as utilizing sparsity or including retrieval to decrease the model. In addition, they should report their energy consumption and carbon impact. This will not only encourage competition beyond model quality but also assure correct accounting of their work.

Paper: https://www.techrxiv.org/articles/preprint/The_Carbon_Footprint_of_Machine_Learning_Training_Will_Plateau_Then_Shrink/19139645/1

Reference: https://ai.googleblog.com/2022/02/good-news-about-carbon-footprint-of.html

Suggested