\nMeta bets big on AI with custom chips \u2014 and a supercomputer<\/br>
\n2023-05-18 21:57:16<\/br><\/p>\n
At a virtual event this morning, Meta lifted the curtains on its efforts to develop in-house infrastructure for AI workloads, including generative AI like the type that underpins its recently launched ad design and creation tools<\/a>.<\/p>\n It was an attempt at a projection of strength from Meta, which historically has been slow to adopt AI-friendly hardware systems \u2014 hobbling its ability to keep pace with rivals such as Google and Microsoft.<\/p>\n \u201c<\/strong>Building our own [hardware] capabilities gives us control at every layer of the stack, from datacenter design to training frameworks,\u201d Alexis Bjorlin, VP of Infrastructure at Meta, told TechCrunch. \u201cThis level of vertical integration is needed to push the boundaries of AI research at scale.\u201d<\/span><\/p>\n Over the past decade or so, Meta has spent billions of dollars recruiting top data scientists and building new kinds of AI, including AI that now powers the discovery engines, moderation filters and ad recommenders found throughout its apps and services. But the company has struggled<\/a> to turn many of its more ambitious AI research innovations into products, particularly on the generative AI front.<\/p>\n Until 2022, Meta largely ran its AI workloads using a combination of CPUs \u2014 which tend to be less efficient for those sorts of tasks than GPUs \u2014 and a custom chip designed for accelerating AI algorithms. Meta pulled the plug on a large-scale rollout of the custom chip, which was planned for 2022, and instead placed orders for billions of dollars\u2019 worth of Nvidia GPUs that required major redesigns of several of its datacenters.<\/p>\n In an effort to turn things around, Meta made plans to start developing a more ambitious in-house chip, due out in 2025, capable of both training AI models and running them. And that was the main topic of today\u2019s presentation.<\/p>\n Meta calls the new chip the Meta Training and Inference Accelerator, or MTIA for short, and describes it as a part of a \u201cfamily\u201d of chips for accelerating AI training and inferencing workloads. (\u201cInferencing\u201d refers to running a trained model.) The MTIA is an ASIC, a kind of chip that combines different circuits on one board, allowing it to be programmed to carry out one or many tasks in parallel.<\/p>\n An AI chip Meta custom-designed for AI workloads.<\/p>\n<\/div>\n \u201cTo gain better levels of efficiency and performance across our important workloads, we needed a tailored solution that\u2019s co-designed with the model, software stack and the system hardware,\u201d Bjorlin continued. \u201cThis provides a better experience for our users across a variety of services.\u201d<\/b><\/p>\n Custom AI chips are increasingly the name of the game among the Big Tech players. Google created a processor, the TPU (short for \u201ctensor processing unit\u201d), to train large generative AI systems like PaLM-2<\/a> and Imagen<\/a>. Amazon offers proprietary chips to AWS customers both for training (Trainium<\/a>) and inferencing (Inferentia<\/a>). And Microsoft, reportedly<\/a>, is working with AMD to develop an in-house AI chip called Athena.<\/p>\n Meta says that it created the first generation of the MTIA \u2014 MTIA v1 \u2014 in 2020, built on a 7-nanometer process. It can scale beyond its internal 128MB of memory to up to 128GB, and in a Meta-designed benchmark test \u2014 which, of course, has to be taken with a grain of salt \u2014 Meta claims that the MTIA handled \u201clow-complexity\u201d and \u201cmedium-complexity\u201d AI models more efficiently than a GPU.<\/p>\n Work remains to be done in the memory and networking areas of the chip, Meta says, which present bottlenecks as the size of AI models grow, requiring workloads to be split up across several chips. (Not coincidentally, Meta recently acquired<\/a> an Oslo-based team building AI networking tech at British chip unicorn Graphcore.) And for now, the MTIA\u2019s focus is strictly on inference \u2014 not training \u2014 for \u201crecommendation workloads\u201d across Meta\u2019s app family.<\/p>\n But Meta stressed that the MTIA, which it continues to refine, \u201cgreatly\u201d increases the company\u2019s efficiency in terms of performance per Watt when running recommendation workloads \u2014 in turn allowing Meta to run \u201cmore enhanced\u201d and \u201ccutting-edge\u201d (ostensibly) AI workloads.<\/p>\n Perhaps one day, Meta will relegate the bulk of its AI workloads to banks of MTIAs. But for now, the social network\u2019s relying on the GPUs in its research-focused supercomputer, the Research SuperCluster (RSC).<\/p>\n First unveiled in January 2022, the RSC \u2014 assembled in partnership with Penguin Computing, Nvidia and Pure Storage \u2014 has completed its second-phase buildout. Meta says that it now contains a total of 2,000 Nvidia DGX A100 systems sporting 16,000 Nvidia A100 GPUs.<\/p>\n So why build an in-house supercomputer? Well, for one, there\u2019s peer pressure. Several years ago, Microsoft made a big to-do about its AI supercomputer built in partnership<\/a> with OpenAI, and more recently said that it would team up with Nvidia to build a new AI supercomputer<\/a> in the Azure cloud. Elsewhere, Google\u2019s been touting its own AI-focused supercomputer<\/a>, which has 26,000 Nvidia H100 GPUs \u2014 putting it ahead of Meta\u2019s.<\/p>\n Meta\u2019s supercomputer for AI research.<\/p>\n<\/div>\n But beyond keeping up with the Joneses, Meta says that the RSC confers the benefit of allowing its researchers to train models using real-world examples from Meta\u2019s production systems. That\u2019s unlike the company\u2019s previous AI infrastructure, which leveraged only open source and publicly available data sets.<\/p>\n \u201cThe RSC AI supercomputer is used for pushing the boundaries of AI research in several domains, including generative AI,\u201d a Meta spokesperson said. \u201cIt\u2019s really about AI research productivity. We wanted to provide AI researchers with a state-of-the-art infrastructure for them to be able to develop models and empower them with a training platform to advance AI.\u201d<\/p>\n At its peak, the RSC can reach nearly 5 exaflops of computing power, which the company claims makes it among the world\u2019s fastest. (Lest that impress, it\u2019s worth noting<\/a> some experts view the exaflops performance metric with a pinch of salt and that the RSC is far outgunned<\/a> by many of the world\u2019s fastest supercomputers.)<\/p>\n Meta says that it used the RSC to train LLaMA, a tortured acronym for \u201cLarge Language Model Meta AI\u201d \u2014 a large language model that the company shared as a \u201cgated release\u201d to researchers earlier in the year (and which subsequently<\/a> leaked in various internet communities). The largest LLaMA model was trained on 2,048 A100 GPUs, Meta says, which took 21 days.<\/p>\n \u201cBuilding our own supercomputing capabilities gives us control at every layer of the stack; from datacenter design to training frameworks,\u201d the spokesperson added. \u201cRSC will help Meta\u2019s AI researchers build new and better AI models that can learn from trillions of examples; work across hundreds of different languages; seamlessly analyze text, images, and video together; develop new augmented reality tools; and much more.\u201d<\/p>\n In addition to MTIA, Meta is developing another chip to handle particular types of computing workloads, the company revealed at today\u2019s event. Called the Meta Scalable Video Processor, or MSVP, Meta says that it\u2019s its first in-house-developed ASIC solution designed for the processing needs of video on demand and live streaming.<\/p>\n Meta began ideating<\/a> custom server-side video chips years ago, readers might recall, announcing an ASIC for video transcoding and inferencing work in 2019. This is the fruit of some of those efforts, as well as a renewed push for a competitive advantage in the area of live video specifically.<\/p>\n \u201cOn Facebook alone, people spend 50% of their time on the app watching video,\u201d Meta technical lead managers Harikrishna Reddy and Yunqing Chen wrote in a co-authored blog post published this morning. \u201cTo serve the wide variety of devices all over the world (mobile devices, laptops, TVs, etc.), videos uploaded to Facebook or Instagram, for example, are transcoded into multiple bitstreams, with different encoding formats, resolutions and quality \u2026 MSVP is programmable and scalable, and can be configured to efficiently support both the high-quality transcoding needed for VOD as well as the low latency and faster processing times that live streaming requires.\u201d<\/p>\n Meta\u2019s custom chip designed to accelerate video workloads, like streaming and transcoding.<\/p>\n<\/div>\n Meta says that its plan is to eventually offload the majority of its \u201cstable and mature\u201d video processing workloads to the MSVP and use software video encoding only for workloads that require specific customization and \u201csignificantly\u201d higher quality. Work continues on improving video quality with MSVP using preprocessing methods like smart denoising and image enhancement, Meta says, as well as post-processing methods such as artifact removal and super-resolution.<\/p>\n \u201cIn the future, MSVP will allow us to support even more of Meta\u2019s most important use cases and needs, including short-form videos \u2014 enabling efficient delivery of generative AI, AR\/VR and other metaverse content,\u201d Reddy and Chen said.<\/p>\n If there\u2019s a common thread in today\u2019s hardware announcements, it\u2019s that Meta\u2019s attempting desperately to pick up the pace where it concerns AI, specifically generative AI.<\/p>\n As much had been telegraphed prior. In February, CEO Mark Zuckerberg \u2014 which has reportedly made upping Meta\u2019s compute capacity for AI a top priority \u2014 announced<\/a> a new top-level generative AI team to, in his words, \u201cturbocharge\u201d the company\u2019s R&D. CTO Andrew Bosworth likewise said recently<\/a> that generative AI was the area where he and Zuckerberg were spending the most time. And chief scientist Yann LeCun has said<\/a> that Meta plans to deploy generative AI tools to create items in virtual reality,<\/p>\n \u201cWe\u2019re exploring chat experiences in WhatsApp and Messenger, visual creation tools for posts in Facebook and Instagram and ads, and over time video and multi-modal experiences as well,\u201d Zuckerberg said during Meta\u2019s Q1 earnings call in April. \u201cI expect that these tools will be valuable for everyone from regular people to creators to businesses. For example, I expect that a lot of interest in AI agents for business messaging and customer support will come once we nail that experience. Over time, this will extend to our work on the metaverse, too, where people will much more easily be able to create avatars, objects, worlds, and code to tie all of them together.\u201d<\/span><\/p>\n In part, Meta\u2019s feeling increasing pressure from investors concerned that the company\u2019s not moving fast enough to capture the (potentially large) market for generative AI. It has no answer \u2014 yet \u2014 to chatbots like Bard, Bing Chat or ChatGPT. Nor has it made much progress on image generation, another key segment that\u2019s seen explosive growth.<\/p>\n If the predictions are right, the total addressable market for generative AI software could<\/a> be $150 billion. Goldman Sachs predicts that it\u2019ll raise GDP by 7%.<\/p>\n![\"Meta](\"https:\/\/i0.wp.com\/techcrunch.com\/wp-content\/uploads\/2023\/05\/acceleratorMTIA.webp?resize=1024%2C576&ssl=1\")
<\/p>\nA supercomputer for AI<\/h2>\n
![\"Meta](\"https:\/\/i0.wp.com\/techcrunch.com\/wp-content\/uploads\/2023\/05\/RSC_16x9-3.webp?w=1024&ssl=1\")
<\/p>\nVideo transcoder<\/h2>\n
![\"Meta](\"https:\/\/i0.wp.com\/techcrunch.com\/wp-content\/uploads\/2023\/05\/MSVP_16x9.webp?w=1024&ssl=1\")
<\/p>\nAI focus<\/h2>\n