With 1.75 trillion parameters, Wu Dao 2.0 is roughly ten times the size of Open AI’s GPT-3. What’s more is there were just three months in-between Wu Dao 1.0 and its 2.0 iteration.
We know that GPT-3 is capable of generating text indiscernible from human-crafted prose. But May, 2020 seems like a long time ago in the time of artificial intelligence. How can this be? Wu Dao is being accelerated by Beijing itself, so these 1.75 trillion parameters (essentially, the model’s self-selected coefficients) are a full ten times larger than the 175 billion GPT-3 was trained on and 150 billion parameters larger than Google’s Switch Transformers.
This suggests China’s Multi Modal AI evolution can scale to something perhaps different from OpenAI’s GPT-3. How their corporate partners will monetize it is anybody’s guess.
China wants the fastest super-computers and appears to want the most parameters in their AI models. Wudao, literally meaning an understanding of natural laws, is intended to create cognitive intelligence dually driven by data and knowledge, making machines think like humans and enabling machine cognitive abilities to pass the Turing test, Tang Jie, BAAI’s vice director of academics, said during the forum.
China is going after the holy grail of AI, an innovation that could usher in a general artificial intelligence, AGI. A deep learning system called FastMoE, developed by BAAI researchers, is disclosed to have laid the cornerstone for Wudao.
In order to train a model on this many parameters and do so quickly — Wu Dao 2.0 arrived just three months after version 1.0’s release in March — the BAAI researchers first developed an open-source learning system akin to Google’s Mixture of Experts, dubbed FastMoE.
Just a few years ago we used to think an AI would never be able to create art as we know it. Huazhibing, a virtual student developed based on Wudao 2.0, the first of its kind in the country, also debuted at the forum this week (early June, 2021) with a video revealing her showcasing music, poem and painting for the forum. AI is able to write and make art like a human, after all.
China somehow thinks the combination of a large-scale model and immense computing power offers a plausible road map to achieving artificial general intelligence, AGI. Others aren’t so sure. At the Last Futurist we think the harnessing of quantum computing with the latest AI could bring us a step closer to an AGI, likely decades away and with a 78% chance that it won’t take place in the 21st century.
FastMoE is operable on PyTorch, enabled the model to be trained both on clusters of supercomputers and conventional GPUs. This gave FastMoE more flexibility than Google’s system since FastMoE doesn’t require proprietary hardware like Google’s TPUs and can therefore run on off-the-shelf hardware — supercomputing clusters notwithstanding.
Unlike most deep learning models which perform a single task — write copy, generate deep fakes, recognize faces, win at Go — Wu Dao is multi-modal, similar in theory to Facebook’s anti-hatespeech AI or Google’s recently released MUM.
The work was led by BAAI Research Academic Vice President and Tsinghua University Professor Tang Jie, with contributions from a team of more than 100 AI scientists from Peking University, Tsinghua University, Renmin University of China, Chinese Academy of Sciences and other institutes. The question if of course what will Wu Dao lead to?
BAAI researchers demonstrated Wu Dao’s abilities to perform natural language processing, text generation, image recognition, and the typical image generation tasks. We know AI can learn and master video games and games like Chess and Go.
Sooner or later AI will take up roles in our society, in the automation of tasks and the organization of cities and solving issues like climate change.
Wu Dao also showed off its ability to power virtual idols (with a little help from Microsoft-spinoff XiaoIce) and predict the 3D structures of proteins like AlphaFold. How will agents like Wu Dao impact innovation, healthcare, biotech, education, retail and so forth? When will AI models begin to be able to learn spontaneously in their own way?
The model can not only write essays, poems and couplets in traditional Chinese, it can both generate alt text based off of a static image and generate nearly photorealistic images based on natural language descriptions. Ten years ago we weren’t sure AI could do these things anytime soon.
So what can it do tomorrow? Wu Dao could evolve faster than GPT-3 it appears. China is intent on being the leader in AI by 2030 and America will have to invest a lot of money and concentration to be able to keep up it appears.