/// EDIT: Again, I am talking down to you. I am not sure why. I agree that as we currently understand AI/AGI, teaching is involved. I just want to drive the point home that neither the concept of AGI nor the concept of math organized via a sufficiently complicated number of primitives requires teaching or humans providing the teaching. It only requires access to information which entropy guarantees it will provide if the math survives sufficiently long. An AGI will require no teaching of any kind. Not in the sense that you are describing, meaning that humans will provide it a corpus that we consider appropriate for an AGI. To the extent that we have to provide a corpus to a neural network currently is inherent to to the neural network not being an AGI in a sea of available corpus sources from which it can choose to teach it self. An AGI by definition will be math on a sufficiently complicated substrate that it will learn on it's own. I can prove this to be not only true but existing in the world today and something that popped into existence via abiogenesis: DNA I don't have a background in biology so you might nitpick and say rna vs some other mechanism but you know what I am talking about. DNA is not alive. All it does is have a replication mechanism available to it self. Instead of binary it is a quaternary system. This quaternary system really only has one recursive function available to it, replication. This replication process, as opposed to human created neural networks which is just math embedded in a sterile environment, is embedded in an environment FULL OF ENTROPY which is just another way to say that its replication function is embedded in information. It took 4 billion years, but a simple quaternary self replicating system embedded in a sea of information eventually learned to build it self into us. AGI requires NO TRAINING AT ALL. It only requires the minimum amount of primitives to escape. You are absolutely wrong about your assumption in every way possible. As for your coffee problem, that is nonsense. We have simple algorythms that can teach mathematical systems to walk done by high schoolers now. In order for math to serve you coffee it doesn't even need to know what coffee or serving is. The training of neural networks does not involve logical deconstruction of what coffee is, what serving is, what a human is in order to accomplish this goal. Frankly, watching how a neural network can be presented with a million images of humans as a corpus to teach it how to create NEW HUMAN FACES (I know you have seen the demos, I think you can download software that genetically creates never before seen faces based on this learning) makes no sense to me and I bet it makes no sense to you or anyone else. No one defined what a nose or a mouth or an eye is, yet, the simple neural network that isn't even remotely close to AGI can just take a bunch of images and create new human faces. AGI isn't as much as about general intelligence in the way that you describe it. AGI is at least 50% a challenge to how we understand things like "intelligence" and "what does it mean for something to mean something" to be. Math teaching it self is nothing new, it's literally everywhere on youtube. You bump into examples of this without even searching for it. /// EDIT: Sorry, have to add one more edit. Your statements are making me think about this a bit more and just how fucked up all the example of feeding a corpus of human faces to a neural network to teach it how to generate new human faces. The fucked up part is that the neural network is not a physical thing, it's just math expressed in a binary system. It's a piece of software, but ultimately the neural network is just a piece of math. Here is the fucked up part: the corpus of human faces ARE NOT IMAGES in the sense that you an I understand them. To a computer, a million jpeg files IS JUST MORE MATH. Look at the jpeg in a hex editor. A jpeg is NOT an image. It's an image the human brain but to the piece of math we call a neural network a million jpegs is just a million pieces of math. When we taught a neural network to create human faces using a corpus of human faces all we actually did was feed on piece of math another piece of math. That math, taught it self to spit out more math that our brains interpret as generated human faces. But, there are no human faces involved there AT ALL. It's one binary system sucking in data from another binary system and spitting out more binary data. To your example of coffee being served by AGI, we can actually say that this neural network did not actually generate any new faces at all. INSTEAD what it did was train it self to TRICK OUR HUMAN BRAINS into thinking that it generated human faces when it did not. All this piece of math did was spit out 0s and 1s in the right pattern to train us to RESPOND HOW IT WANTED US TO RESPOND. I am stretching the meaning of learning here to make the point that us training the machine could just as easily be understood as the machine training us. As a side point, this feedback loop exists with all of our existing technology, it's not unique to ai, it's a general property of a computable universe. This is what the video is actually referring to. The human mind believes it is in control while it is unaware that it is also simultaneously not under control.