wp-plugin-hostgator domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/scienrds/scienceandnerds/wp-includes/functions.php on line 6114ol-scrapes domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/scienrds/scienceandnerds/wp-includes/functions.php on line 6114Source:https:\/\/www.quantamagazine.org\/google-deepmind-trains-artificial-brainstorming-in-chess-ai-20231115\/#comments<\/a><\/br> When Covid-19 sent people home in early 2020, the computer scientist Tom Zahavy<\/a> rediscovered chess. He had played as a kid and had recently read Garry Kasparov\u2019s Deep Thinking<\/em>, a memoir of the grandmaster\u2019s 1997 matches against IBM\u2019s chess-playing computer, Deep Blue. He watched chess videos on YouTube and The Queen\u2019s Gambit <\/em>on Netflix.<\/p>\n Despite his renewed interest, Zahavy wasn\u2019t looking for ways to improve his game. \u201cI\u2019m not a great player,\u201d he said. \u201cI\u2019m better at chess puzzles\u201d \u2014 arrangements of pieces, often contrived and unlikely to occur during a real game, that challenge a player to find creative ways to gain the advantage.<\/p>\n The puzzles can help players sharpen their skills, but more recently they\u2019ve helped reveal the hidden limitations of chess programs. One of the most notorious puzzles, devised by the mathematician Sir Roger Penrose in 2017, puts stronger black pieces (such as the queen and rooks) on the board, but in awkward positions. An experienced human player, playing white, could readily steer the game into a draw, but powerful computer chess programs would say black had a clear advantage. That difference, Zahavy said, suggested that even though computers could defeat the world\u2019s best human players, they couldn\u2019t yet recognize and work through every kind of tough problem. Since then, Penrose and others have devised sprawling collections of puzzles that computers struggle to solve.<\/p>\n Chess has long been a touchstone for testing new ideas in artificial intelligence<\/a>, and Penrose\u2019s puzzles piqued Zahavy\u2019s interest. \u201cI was trying to understand what makes these positions so hard for computers when at least some of them we can solve as humans,\u201d he said. \u201cI was completely fascinated.\u201d It soon evolved into a professional interest: As a research scientist at Google DeepMind, Zahavy explores creative problem-solving approaches. The goal is to devise AI systems with a spectrum of possible behaviors beyond performing a single task.<\/p>\n A traditional AI chess program, trained to win, may not make sense of a Penrose puzzle, but Zahavy suspected that a program made up of many diverse systems, working together as a group, could make headway. So he and his colleagues developed a way to weave together multiple (up to 10) decision-making AI systems, each optimized and trained for different strategies, starting with AlphaZero, DeepMind\u2019s powerful chess program. The new system, they reported in August<\/a>, played better than AlphaZero alone, and it showed more skill \u2014 and more creativity \u2014 in dealing with Penrose\u2019s puzzles. These abilities came, in a sense, from self-collaboration: If one approach hit a wall, the program simply turned to another.<\/p>\n That approach fundamentally makes sense, said Allison Liemhetcharat<\/a>, a computer scientist at DoorDash who has worked with multi-agent approaches to problem-solving in robotics. \u201cWith a population of agents, there\u2019s a higher probability that the puzzles are in the domain that at least one of the agents was trained in.\u201d<\/p>\n The work suggests that teams of diverse AI systems could efficiently tackle hard problems well beyond the game board. \u201cThis is a great example that looking for more than one way to solve a problem \u2014 like winning a chess game \u2014 provides a lot of benefits,\u201d said Antoine Cully<\/a>, an AI researcher at Imperial College London who was not involved with the DeepMind project. He compared it to an artificial version of human brainstorming sessions. \u201cThis thought process leads to creative and effective solutions that one would miss without doing this exercise.\u201d<\/p>\n Before joining DeepMind, Zahavy was interested in deep reinforcement learning, an area of artificial intelligence in which a system uses neural networks to learn some task through trial and error. It\u2019s the basis for the most powerful chess programs (and used in other AI applications like self-driving cars). The system starts with its environment. In chess, for example, the environment includes the game board and possible moves. If the task is to drive a car, the environment includes everything around the vehicle. The system then makes decisions, takes actions and evaluates how close it came to its goal. As it gets closer to the goal, it accumulates rewards, and as the system racks up rewards it improves its performance. The \u201cdeep\u201d part of this approach describes the neural networks used to analyze and assess behaviors.<\/p>\n Reinforcement learning is how AlphaZero learned to become a chess master. DeepMind reported<\/a> that during the program\u2019s first nine hours of training, in December 2017, it played 44 million games against itself. At first, its moves were randomly determined, but over time it learned to select moves more likely to lead toward checkmate. After just hours of training, AlphaZero developed the ability to defeat any human chess player.<\/p>\n But as successful as reinforcement learning can be, it doesn\u2019t always lead to strategies that reflected a general understanding of the game. Over the last half-decade or so, Zahavy and others noticed an uptick in the peculiar glitches that could happen on systems trained with trial and error. A system that plays video games, for example, might find a loophole and figure out how to cheat or skip a level, or it could just as easily get stuck in a repetitive loop. Penrose-style puzzles similarly suggested a kind of blind spot, or glitch, in AlphaZero \u2014 it couldn\u2019t figure out how to approach a problem it had never seen before.<\/p>\n But maybe not all glitches are just errors. Zahavy suspected that AlphaZero\u2019s blind spots might actually be something else in disguise \u2014 decisions and behaviors tied to the system\u2019s internal rewards. Deep reinforcement learning systems, he said, don\u2019t know how to fail \u2014 or even how to recognize failure. The ability to fail has long been linked to creative problem-solving. \u201cCreativity has a human quality,\u201d Kasparov wrote in Deep Thinking<\/em>. \u201cIt accepts the notion of failure.\u201d<\/p>\n AI systems typically don\u2019t. And if a system doesn\u2019t recognize that it\u2019s failed to complete its task, then it may not try something else. Instead, it will just keep trying to do what it\u2019s already done. That\u2019s likely what led to those dead ends in video games \u2014 or to getting stuck on some Penrose challenges, Zahavy said. The system was chasing \u201cweird kinds of intrinsic rewards,\u201d he said, that it had developed during its training. Things that looked like mistakes from the outside were likely the consequence of developing specific but ultimately unsuccessful strategies.<\/p>\n The system regarded these weird rewards as steps toward the greater goal, which it couldn\u2019t actually achieve, and it didn\u2019t know to try something new. \u201cI was trying to make sense of them,\u201d Zahavy said.<\/p>\n Part of the reason these glitches can prove so consequential \u2014 and so useful \u2014 comes from what researchers recognize as a problem with generalization. While reinforcement learning systems can develop an effective strategy for connecting a given situation to a specific action \u2014 which researchers call a \u201cpolicy\u201d \u2014 they can\u2019t apply it to different problems. \u201cWhat normally tends to happen with reinforcement learning, almost regardless of the method, is that you get the policy that solves the particular instance of the problem you\u2019ve been training on, but it doesn\u2019t generalize,\u201d said Julian Togelius<\/a>, a computer scientist at New York University and research director at modl.ai.<\/p>\n Zahavy saw the Penrose puzzles as requiring just this sort of generalization. Maybe AlphaZero couldn\u2019t solve most puzzles because it was so focused on winning entire games, start to finish. But that approach introduced blind spots exposed by the unlikely arrangements of pieces in Penrose puzzles. Maybe, he reasoned, the program could learn to beat the puzzle if it had enough creative room to brainstorm and access different training methods.<\/p>\n So he and his colleagues first collected a set of 53 Penrose puzzles and 15 additional challenge puzzles. On its own, AlphaZero solved less 4% of the Penrose puzzles and under 12% of the rest. Zahavy wasn\u2019t surprised: Many of these puzzles were designed by chess masters to intentionally confuse computers.<\/p>\n As a test, the researchers tried training AlphaZero to play against itself using the Penrose puzzle arrangement as the starting position, instead of the full board of typical games. Its performance improved dramatically: It solved 96% of the Penrose puzzles and 76% of the challenge set. In general, when AlphaZero trained on a specific puzzle, it could solve that puzzle, just as it could win when it trained on a full game. Perhaps, Zahavy thought, if a chess program could somehow have access to all those different versions of AlphaZero, trained on those different positions, then that diversity could spark the ability to approach new problems productively. Perhaps it could generalize, in other words, solving not only the Penrose puzzles, but any broader chess problem.<\/p>\n His group decided to find out. They built the new, diversified version of AlphaZero, which includes multiple AI systems that trained independently and on a variety of situations. The algorithm that governs the overall system acts as a kind of virtual matchmaker, Zahavy said: one designed to identify which agent has the best chance of succeeding when it\u2019s time to make a move. He and his colleagues also coded in a \u201cdiversity bonus\u201d \u2014 a reward for the system whenever it pulled strategies from a large selection of choices.<\/p>\n When the new system was set loose to play its own games, the team observed a lot of variety. The diversified AI player experimented with new, effective openings and novel \u2014 but sound \u2014 decisions about specific strategies, such as when and where to castle. In most matches, it defeated the original AlphaZero. The team also found that the diversified version could solve twice as many challenge puzzles as the original and could solve more than half of the total catalog of Penrose puzzles.<\/p>\n \u201cThe idea is that instead of finding one solution, or one single policy, that would beat any player, here [it uses] the idea of creative diversity,\u201d Cully said.<\/p>\n With access to more and different played games, Zahavy said, the diversified AlphaZero had more options for sticky situations when they arose. \u201cIf you can control the kind of games that it sees, you basically control how it will generalize,\u201d he said. Those weird intrinsic rewards (and their associated moves) could become strengths for diverse behaviors. Then the system could learn to assess and value the disparate approaches and see when they were most successful. \u201cWe found that this group of agents can actually come to an agreement on these positions.\u201d<\/p>\n And, crucially, the implications extend beyond chess.<\/p>\n Cully said a diversified approach can help any AI system, not just those based on reinforcement learning. He\u2019s long used diversity to train physical systems, including a six-legged robot<\/a> that was allowed to explore various kinds of movement, before he intentionally \u201cinjured\u201d it, allowing it to continue moving using some of the techniques it had developed before. \u201cWe were just trying to find solutions that were different from all previous solutions we have found so far.\u201d Recently, he\u2019s also been collaborating with researchers to use diversity to identify promising new drug candidates and develop effective stock-trading strategies.<\/p>\n \u201cThe goal is to generate a large collection of potentially thousands of different solutions, where every solution is very different from the next,\u201d Cully said. So \u2014 just as the diversified chess player learned to do \u2014 for every type of problem, the overall system could choose the best possible solution. Zahavy\u2019s AI system, he said, clearly shows how \u201csearching for diverse strategies helps to think outside the box and find solutions.\u201d<\/p>\n Zahavy suspects that in order for AI systems to think creatively, researchers simply have to get them to consider more options. That hypothesis suggests a curious connection between humans and machines: Maybe intelligence is just a matter of computational power. For an AI system, maybe creativity boils down to the ability to consider and select from a large enough buffet of options. As the system gains rewards for selecting a variety of optimal strategies, this kind of creative problem-solving gets reinforced and strengthened. Ultimately, in theory, it could emulate any kind of problem-solving strategy recognized as a creative one in humans. Creativity would become a computational problem.<\/p>\n Liemhetcharat noted that a diversified AI system is unlikely to completely resolve the broader generalization problem in machine learning. But it\u2019s a step in the right direction. \u201cIt\u2019s mitigating one of the shortcomings,\u201d she said.<\/p>\n More practically, Zahavy\u2019s results resonate with recent efforts that show how cooperation can lead to better performance on hard tasks among humans. Most of the hits on the Billboard 100 list were written by teams of songwriters, for example, not individuals. And there\u2019s still room for improvement. The diverse approach is currently computationally expensive, since it must consider so many more possibilities than a typical system. Zahavy is also not convinced that even the diversified AlphaZero captures the entire spectrum of possibilities.<\/p>\n \u201cI still [think] there is room to find different solutions,\u201d he said. \u201cIt\u2019s not clear to me that given all the data in the world, there is [only] one answer to every question.\u201d<\/p>\n Quanta\u00a0is conducting a series of surveys to better serve our audience. Take our <\/i>computer science reader survey<\/i><\/a> and you will be entered to win free<\/i>\u00a0Quanta\u00a0merchandise.<\/i><\/p>\n<\/div>\n <\/br><\/br><\/br><\/p>\n
\nAI System Beats Chess Puzzles With \u2018Artificial Brainstorming\u2019<\/br>
\n2023-11-16 21:58:23<\/br><\/p>\nChasing Failures<\/strong><\/h2>\n
A Better Game<\/strong><\/h2>\n
Real-Life Creativity<\/strong><\/h2>\n