Scientific approach to artificial intelligence


Yesterday I read this wonderful blog post by Yarin Gal on the science of deep learning:

I think a scientific approach is the way forward not just for deep learning but every other
branch of AI. Empirical success on benchmarks is not enough if we actually want to
understand the success of different approaches which is what’s necessary in order to
have a science of AI.

Might anybody know useful papers that touch on the subject?


Is this scientific ?



Those are good technical breakthroughs but they don’t demonstrate a rigorous
scientific approach. Here are some questions that are essential for a scientific

  1. Have you done a rigorous analysis of the types of errors made by your model?
  2. How does the model generalise to other datasets? (i.e. how does it handle dataset shift)
  3. How does your model represent its uncertainty?

People are beginning to think carefully about these questions which are motivating
the development of bayesian deep learning:

However, Yarin Gal’s PhD thesis in particular:
might be the best resource on this subject right now. Feel free to correct me if I’m wrong.


Indeed, I can’t find anything better.

I wonder though in the long run whether newbs like me can bruteforce more breakthroughs or whether computer scientists will be able to find a real scientific way to solve this “easily”.



That’s a very good question and something I wonder about myself. In my opinion
if you can understand a paper and identify questions that haven’t been fully
addressed you can contribute to the field. :slight_smile:


For general ai, I have to agree with you, but not for game ai, RL or not.

I haven’t even completed hard math courses of highschool(calculus,probability,complex and imaginary numbers) and I don’t plan to do so, since there is an easier road.

about myself, I’ve been playing games since the (s)nes and dos times, games like super mario world/megaman (x) and killer instinct where you have to know “2d physics” and button combinations, it is easy to visualise like a web developer, my general algorithm solving skills are below average in competitions, that’s why older people who haven’t played these games have a hard time solving these “cognitive” challenges, I’ve also been rom hacking/reverse engineering games in the past.

Where in web dev, do we use the scientific method?

I just do this intuitively and have a basic knowhow of algos and datastructures + python.


You might be right about game ai. I think that for a field that’s as massive as AI,
there’s a lot of room for many different approaches.


true, what other fields aren’t solved yet specific to CS?

With our new understandings of the past 5 years bioinformatics and other fields, will grow exponentially if it hasn’t yet.


perhaps you don’t know about MIRI?

We do foundational mathematical research to ensure smarter-than-human artificial intelligence has a positive impact.

I personally can’t read through any kind of such academic paper. ain’t got enough patience or something. yet, I do consider myself a skeptic-ish scientist-wannabe.

we don’t need to have a “science of Ai” to transcend it, I believe. we’ve got a lot of science about a lot of stuff around human and evolution. that doesn’t mean we can use any of it to build a human from scratch. or that we’d ever be able to. here’s more about this idea:

there are many ways to do science, though. I find it much more important to have double blinded and replicable tests than rigorous analysis, statistical datasets or models.

here’s a quite interesting read on this topic:

basically, it tells the tale of how computers helped to solve, empirically, a long unexpected “impossible” problem trying to do math in this “scientific” manner you seen to describe.

and if you’ve read through all up to here, I feel you might find some interest in at least one of those non-scientific reads:

can you guess which one I haven’t read yet? very rarely I’ll link to something I haven’t checked through at least 2 levels deep, like this one.


I think that, as we do not still have any clear and usable definition of intelligence itself that everyone can accept and use, then how can we aim to make “real science” on it?

Without definitions, it can not be science.


I missed a bit of this discussion so here are my thoughts/responses:

  1. I am aware of MIRI.
  2. I don’t think there is a ‘correct’ definition of intelligence in the
    same way that there isn’t
    a correct definition of temperature. Some definitions simply happen to be
    more convenient
    than others.

Regarding intelligence in particular, I think that the best definition I’ve
come across(in my opinion)
is that of Shane Legg and Marcus Hutter: I
would also read this
paper if you’re interested:

This is one of the reasons why DeepMind has developed a large number of
environments for its
agents to solve.



The first article is about measuring intelligence, but not defining it, while the second is a collection of definitions.

My personal favourite (and the one I work based on) is:


Hello Sergio,

I must clarify the following:

  1. The first article presents a collection of definitions and they settle
    upon the following definition:

“Intelligence measures an agent’s ability to achieve goals in a wide range
of environments.” S. Legg and M. Hutter

  1. The second paper builds upon the first because they explain not only how
    to give a mathematical structure
    to their informal definition but also how to approximate it. This is

I actually think that embodiment is necessary in this universe so Shane
Legg’s definition is incomplete in my opinion.



Hi Aidan, Legg’s definition is ok for me, but it defines a particular aspect of intelligence, following goals, not the root definition where an intelligence can be goal-less and still take decisions, nor the nature of “intelligence” on its own.

Alexander’s definition is about thermodynamics, applying the 2nd law to the “cone” of future outcomes that follow if you take one given option so you can score them and thake your decision as the weightened average (its future entropy, normalised, is the weight here) over your options.

So, for me, Alexander’s definition is wider and more precise than any other one I know about.

I actually worked out the way to convert any set of goals into a utility function to feed the AI I built on those Alexander’s principles, and it really worked beyond my spectations.

I have a blog about it and there is a post that actually talks about it and shows a very complex environment (Lorentz atractors included) that the AI manages quite well solving and creating new strategies from thin air:

As you see, the definition I use allows me to actually code a general AI capable of dealing with any environmet as far as you can simulate the system, without any kind of training (it trains itself using a small set of future paths built on the go from its actual position/state, instead of the examples from the past used to train a NN).

Just wondering if anyone here have ever heard about these “causal entropic forces”, I think the article was not given the merit it deserves and basically forgotten.


Interesting. I’ll look into it further.

Thanks for sharing this. :slight_smile:



Sure you will want to check this: My AI based on entropy vs atari section on openai (pack man), no previous training, no code specific for that game (you could change the game in the last minute), and solved at 1 fps on a standard laptop, 1 core.


For some thoughts on what very probably is not Intelligence check out , I see it is available now for borrowing on my local library digital section. Among other things it details how a bunch of glib pitchmen (pitchwomen?) chat-bots came within one vote or beating the human participants in the Turing test. It would seem that many anonymous, introverted and introspective human participants appeared to the “judges” in the competition to be robots. This at least suggests that the Turing test may really be a poor measure of Intelligence. To paraphrase the old saying about Art … Turing may not know much about Intelligence, but he knows what he likes.


I have released a paper where I define intelligence based on the future entropy associated with each action, very similar with the 2nd law of thermodynamics:

These ideas are being tested in Atari environments with very nice results (this far we beated state-of-the-art like DQN or A3C in 44 out of 55 games so far, beated human records in 21/55 and found a bug that prevents score above 999,999 in 7 of them):