Notice: Function _load_textdomain_just_in_time was called incorrectly. Translation loading for the wp-plugin-hostgator domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/scienrds/scienceandnerds/wp-includes/functions.php on line 6114

Notice: Function _load_textdomain_just_in_time was called incorrectly. Translation loading for the ol-scrapes domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/scienrds/scienceandnerds/wp-includes/functions.php on line 6114

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893

Warning: Cannot modify header information - headers already sent by (output started at /home4/scienrds/scienceandnerds/wp-includes/functions.php:6114) in /home4/scienrds/scienceandnerds/wp-includes/rest-api/class-wp-rest-server.php on line 1893
{"id":39157,"date":"2024-02-14T21:58:35","date_gmt":"2024-02-14T21:58:35","guid":{"rendered":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/"},"modified":"2024-02-14T21:58:36","modified_gmt":"2024-02-14T21:58:36","slug":"how-quickly-do-large-language-models-learn-unexpected-skills","status":"publish","type":"post","link":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/","title":{"rendered":"How Quickly Do Large Language Models Learn Unexpected Skills?"},"content":{"rendered":"

Source:https:\/\/www.quantamagazine.org\/how-quickly-do-large-language-models-learn-unexpected-skills-20240213\/#comments<\/a><\/br>
\nHow Quickly Do Large Language Models Learn Unexpected Skills?<\/br>
\n2024-02-14 21:58:35<\/br><\/p>\n

\n

Two years ago, in a project called the Beyond the Imitation Game benchmark<\/a>, or BIG-bench, 450 researchers compiled a list of 204 tasks designed to test the capabilities of large language models, which power chatbots like ChatGPT. On most tasks, performance improved predictably and smoothly as the models scaled up \u2014 the larger the model, the better it got. But with other tasks, the jump in ability wasn\u2019t smooth. The performance remained near zero for a while, then performance jumped. Other studies found similar leaps in ability.<\/p>\n

The authors described this as \u201cbreakthrough\u201d behavior; other researchers have likened it to a phase transition in physics, like when liquid water freezes into ice. In a paper<\/a> published in August 2022, researchers noted that these behaviors are not only surprising but unpredictable, and that they should inform the evolving conversations around AI safety, potential and risk. They called the abilities \u201cemergent<\/a>,\u201d a word that describes collective behaviors that only appear once a system reaches a high level of complexity.<\/p>\n

But things may not be so simple. A new paper<\/a> by a trio of researchers at Stanford University posits that the sudden appearance of these abilities is just a consequence of the way researchers measure the LLM\u2019s performance. The abilities, they argue, are neither unpredictable nor sudden. \u201cThe transition is much more predictable than people give it credit for,\u201d said Sanmi Koyejo<\/a>, a computer scientist at Stanford and the paper\u2019s senior author. \u201cStrong claims of emergence have as much to do with the way we choose to measure as they do with what the models are doing.\u201d<\/p>\n

We\u2019re only now seeing and studying this behavior because of how large these models have become. Large language models train by analyzing enormous datasets of text<\/a> \u2014 words from online sources including books, web searches and Wikipedia \u2014 and finding links between words that often appear together. The size is measured in terms of parameters, roughly analogous to all the ways that words can be connected. The more parameters, the more connections an LLM can find. GPT-2 had 1.5 billion parameters, while GPT-3.5, the LLM that powers ChatGPT, uses 350 billion. GPT-4, which debuted in March 2023 and now underlies Microsoft Copilot, reportedly uses 1.75 trillion.<\/p>\n

That rapid growth has brought an astonishing surge in performance and efficacy, and no one is disputing that large enough LLMs can complete tasks that smaller models can\u2019t, including ones for which they weren\u2019t trained. The trio at Stanford who cast emergence as a \u201cmirage\u201d recognize that LLMs become more effective as they scale up; in fact, the added complexity<\/a> of larger models should make it possible to get better at more difficult and diverse problems. But they argue that whether this improvement looks smooth and predictable or jagged and sharp results from the choice of metric \u2014 or even a paucity of test examples \u2014 rather than the model\u2019s inner workings.<\/p>\n

Three-digit addition offers an example. In the 2022 BIG-bench study, researchers reported that with fewer parameters, both GPT-3 and another LLM named LAMDA failed to accurately complete addition problems. However, when GPT-3 trained using 13 billion parameters, its ability changed as if with the flip of a switch. Suddenly, it could add \u2014 and LAMDA could, too, at 68 billion parameters. This suggests that the ability to add emerges at a certain threshold.<\/p>\n

But the Stanford researchers point out that the LLMs were judged only on accuracy: Either they could do it perfectly, or they couldn\u2019t. So even if an LLM predicted most of the digits correctly, it failed. That didn\u2019t seem right. If you\u2019re calculating 100 plus 278, then 376 seems like a much more accurate answer than, say, \u22129.34.<\/p>\n

So instead, Koyejo and his collaborators tested the same task using a metric that awards partial credit. \u201cWe can ask: How well does it predict the first digit? Then the second? Then the third?\u201d he said.<\/p>\n

Koyejo credits the idea for the new work to his graduate student Rylan Schaeffer, who he said noticed that an LLM\u2019s performance seems to change with how its ability is measured. Together with Brando Miranda, another Stanford graduate student, they chose new metrics showing that as parameters increased, the LLMs predicted an increasingly correct sequence of digits in addition problems. This suggests that the ability to add isn\u2019t emergent \u2014 meaning that it undergoes a sudden, unpredictable jump \u2014 but gradual and predictable. They find that with a different measuring stick, emergence vanishes.<\/p>\n<\/div>\n

<\/br><\/br><\/br><\/p>\n

Uncategorized<\/br>
\n<\/br>
\nSource:https:\/\/www.quantamagazine.org\/how-quickly-do-large-language-models-learn-unexpected-skills-20240213\/#comments<\/a><\/br><\/br><\/p>\n","protected":false},"excerpt":{"rendered":"

Source:https:\/\/www.quantamagazine.org\/how-quickly-do-large-language-models-learn-unexpected-skills-20240213\/#comments How Quickly Do Large Language Models Learn Unexpected Skills? 2024-02-14 21:58:35 Two years ago, in a project called the Beyond the Imitation Game benchmark, or BIG-bench, 450 researchers compiled a list of 204 tasks designed to test the capabilities of large language models, which power chatbots like ChatGPT. On most tasks, performance improved predictably […]<\/p>\n","protected":false},"author":1,"featured_media":39158,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","om_disable_all_campaigns":false,"pagelayer_contact_templates":[],"_pagelayer_content":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-39157","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"yoast_head":"\nHow Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds\" \/>\n<meta property=\"og:description\" content=\"Source:https:\/\/www.quantamagazine.org\/how-quickly-do-large-language-models-learn-unexpected-skills-20240213\/#comments How Quickly Do Large Language Models Learn Unexpected Skills? 2024-02-14 21:58:35 Two years ago, in a project called the Beyond the Imitation Game benchmark, or BIG-bench, 450 researchers compiled a list of 204 tasks designed to test the capabilities of large language models, which power chatbots like ChatGPT. On most tasks, performance improved predictably […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/\" \/>\n<meta property=\"og:site_name\" content=\"Science and Nerds\" \/>\n<meta property=\"article:published_time\" content=\"2024-02-14T21:58:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-14T21:58:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1440\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/\",\"url\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/\",\"name\":\"How Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds\",\"isPartOf\":{\"@id\":\"https:\/\/scienceandnerds.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1\",\"datePublished\":\"2024-02-14T21:58:35+00:00\",\"dateModified\":\"2024-02-14T21:58:36+00:00\",\"author\":{\"@id\":\"https:\/\/scienceandnerds.com\/#\/schema\/person\/ea2991abeb2b9ab04b32790dff28360e\"},\"breadcrumb\":{\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1\",\"width\":2560,\"height\":1440},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/scienceandnerds.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How Quickly Do Large Language Models Learn Unexpected Skills?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/scienceandnerds.com\/#website\",\"url\":\"https:\/\/scienceandnerds.com\/\",\"name\":\"Science and Nerds\",\"description\":\"My WordPress Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/scienceandnerds.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/scienceandnerds.com\/#\/schema\/person\/ea2991abeb2b9ab04b32790dff28360e\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/scienceandnerds.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7e6e14fc6691445ef2b2c0a3a6c43882?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7e6e14fc6691445ef2b2c0a3a6c43882?s=96&d=mm&r=g\",\"caption\":\"admin\"},\"sameAs\":[\"https:\/\/scienceandnerds.com\"],\"url\":\"https:\/\/scienceandnerds.com\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/","og_locale":"en_US","og_type":"article","og_title":"How Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds","og_description":"Source:https:\/\/www.quantamagazine.org\/how-quickly-do-large-language-models-learn-unexpected-skills-20240213\/#comments How Quickly Do Large Language Models Learn Unexpected Skills? 2024-02-14 21:58:35 Two years ago, in a project called the Beyond the Imitation Game benchmark, or BIG-bench, 450 researchers compiled a list of 204 tasks designed to test the capabilities of large language models, which power chatbots like ChatGPT. On most tasks, performance improved predictably […]","og_url":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/","og_site_name":"Science and Nerds","article_published_time":"2024-02-14T21:58:35+00:00","article_modified_time":"2024-02-14T21:58:36+00:00","og_image":[{"width":2560,"height":1440,"url":"https:\/\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp","type":"image\/webp"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/","url":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/","name":"How Quickly Do Large Language Models Learn Unexpected Skills? - Science and Nerds","isPartOf":{"@id":"https:\/\/scienceandnerds.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage"},"image":{"@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1","datePublished":"2024-02-14T21:58:35+00:00","dateModified":"2024-02-14T21:58:36+00:00","author":{"@id":"https:\/\/scienceandnerds.com\/#\/schema\/person\/ea2991abeb2b9ab04b32790dff28360e"},"breadcrumb":{"@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#primaryimage","url":"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1","contentUrl":"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1","width":2560,"height":1440},{"@type":"BreadcrumbList","@id":"https:\/\/scienceandnerds.com\/2024\/02\/14\/how-quickly-do-large-language-models-learn-unexpected-skills\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/scienceandnerds.com\/"},{"@type":"ListItem","position":2,"name":"How Quickly Do Large Language Models Learn Unexpected Skills?"}]},{"@type":"WebSite","@id":"https:\/\/scienceandnerds.com\/#website","url":"https:\/\/scienceandnerds.com\/","name":"Science and Nerds","description":"My WordPress Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/scienceandnerds.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/scienceandnerds.com\/#\/schema\/person\/ea2991abeb2b9ab04b32790dff28360e","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scienceandnerds.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/7e6e14fc6691445ef2b2c0a3a6c43882?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7e6e14fc6691445ef2b2c0a3a6c43882?s=96&d=mm&r=g","caption":"admin"},"sameAs":["https:\/\/scienceandnerds.com"],"url":"https:\/\/scienceandnerds.com\/author\/admin\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/scienceandnerds.com\/wp-content\/uploads\/2024\/02\/how-quickly-do-large-language-models-learn-unexpected-skills_65cd378c50f72.webp?fit=2560%2C1440&ssl=1","_links":{"self":[{"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/posts\/39157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/comments?post=39157"}],"version-history":[{"count":1,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/posts\/39157\/revisions"}],"predecessor-version":[{"id":39159,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/posts\/39157\/revisions\/39159"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/media\/39158"}],"wp:attachment":[{"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/media?parent=39157"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/categories?post=39157"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scienceandnerds.com\/wp-json\/wp\/v2\/tags?post=39157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}