Getting Started with Deep Learning

Why Deep Learning?

Several months ago one of my former students, Dr. Julia S. Slige, began posting a series of tutorials on using R for data mining.  Julia even has a book on the topic published by O’Reilly.  Her posts and subsequent research got me to thinking about Deep Learning, a key part of my ongoing research into adaptive/immersive learning using game engines.

As a result of these musings, I have begun the process of learning to use the latest Deep Learning tools so as to enter this next phase of research.  As I like to share what I am learning, writing/blogging, as Bruffee (1999) postulates, helps to maintain the conversation that is learning.  I will be recording my musings and what I have learned along the way.  Who knows, it might even result in another textbook (that is how all of my previous textbooks got their start).

One of my goals as I share what I have learned is to make it easier for those who follow the path that I am blazing.  This is still early days of AI, Deep Learning, Machine Learning and lots of similar buzz words.  It can be confusing and overwhelming to make sense of it all so that you can apply the concepts to what you are interested.  Hopefully, after reading these blogs, it will be a little less confusing.

What is Deep Learning

There is a great deal of hype and excitement around Artificial Intelligence, Machine Learning, Neural Networks, and Deep Learning.  Unfortunately, the terms are frequently being used interchangeable which only adds to the confusion.  Please understand that each description below is a broad generalization.  Each of these areas is immense with many methods and theories. So before we go any further, let’s agree on the general terminology:

Artificial Intelligence
Artificial Intelligence or AI as it is commonly referred to, is the catch-all term for anything related to AI.  You’ve seen the movies; robots and computers that are able to interact and learn from their environments to accomplish their mission, some bad (Terminator, HAL), some helpful (R2D2), some annoying (C3PO).
The goal AI is to create software that learns as it interacts with the environment. At this time, implementations of AI are very narrow and specialized for specific fields. AI includes the subdomains of Machine Learning, Neural Networks, and Deep Learning. While AI has been around for years, only with the recent availability of cloud-based computing are the resources available to begin to simulate AI.

Machine Learning
Machine Learning (ML) relies upon algorithms to parse data, learn from the data, then make a prediction based upon the data.  ML systems are ‘trained’ using a lot of data (and I mean a LOT of data) to make accurate predictions.  You are probably already using ML in your everyday life: Spam filtering!  If you use a cloud-based email system such as Gmail, then you are using ML.  Have you noticed that the first time you receive a message that is probably spam, it goes into your inbox, but after that, similar messages go straight to your spam folder.  Gmail learns from you, based on your telling it what is spam and what isn’t.

Neural Networks
Neural Networks (NN) builds on ML.  The concept is that just as the human brain uses neurons to pass information, neural networks can create pathways that show lead to the correct answer more quickly.  ML makes a lot of mistakes as it is learning.  Neural Networks learn more quickly because a weight or value is given to paths that result in the correct answer.  These paths are stored in hidden layers.
Neural Networks are very processor intensive, even more so than ML.  Only with the advent of parallel processing and GPUs have Neural Networks begun to accomplish what they are capable of doing.

Deep Learning

Deep Learning (DL) combines ML and NN to accomplish the next phase of AI. By using sets of algorithms that are then layered.  Deep Learning is different from Neural Networks in that it uses more layers in the building of learning paths. Additionally, Neural Networks are usually supervised as they learn.  Deep Learning can be supervised or unsupervised (or a combination) as it goes through it’s learning process.  At this time Deep Learning is providing exciting advances in optical recognition i.e. read a stop sign, process natural language, speech systems, translations, even adding closed captioning to YouTube videos.

Why Focus on Deep Learning?

Why will I be skipping over implementations of Machine Learning and Neural Networks?  It comes down to goals.  While I will be sharing certain aspects of ML & Neural, the aspects of AI that I am interested in are concentrated in the Deep Learning domain.
My research is focused on creating immersive educational environments that individualizes instruction based on student performance.  My wife, Rosemary, and I created one of the first fully online schools back in the 90’s.  Our continued research and interest in game design, gamification, and now Deep Learning, are all focused on building a better, pedagogically superior, learning experience for all learners, regardless of location, background, disabilities, language, or previous experiences.

Programming Languages and Tools for Deep Learning

AWS/Google/Microsoft Azure/IBM Bluemix
All major cloud companies are trying to stake their claim in the AI field.  Each has slightly different offerings and tools (and pricing for those tools).
I have received research funding in the form of free access to the cloud infrastructure (no real money actually received) for each of the four mentioned companies for cloud computing over the last 5 years.
I have settled on using Amazon AWS for my personal and research work.  While the services, cost, and resources do not vary that much from company to company, Amazon’s customer-centric nature clicked with how I like to do things, thus that is where the majority of my time and energy has been spent.  Also, AWS supports both Lua Torch and Python Theano, the two dominant tool sets that we will be (at least intially) exploring.

Lua
I have to admit, I was excited to see that there was a Lua tool set for Deep Learning.  I have been developing with Lua for over 15 years and used it intensely for the last 7 years.  If you haven’t seen my Lua tutorials on Youtube, they can be found here.
The dominant tool in Lua is Torch.  It is considered a very flexible tool that keeps the easy of use of Lua.

Python Theano
Python seems to have broader support (and many tools) than Lua in deep learning.This might be caused by Python is in wider general use than Lua, which naturally makes it more attractive.
The dominate tool is Theano.  While there are many others in development and gaining in popularity (most notably Cafe2), Theano remains the primary Python Deep Learning tool.

Other languages:
R and Julia are also being used for deep Learning but do not appear to have the depth of support Lua and Python enjoy at this time.
Deep Learning is still relatively young as far as tools.  There are a lot of things in development, but from my research, The lead tools in Deep Learning are Lua Torch and Python Theano, so that is where I will be concentrating my efforts at this stage.

Next Steps

Now that we have an idea of what Deep Learning is and a few simple examples of what it is capable of doing, I will be exploring the tools.  In my next blog post, we will take a closer look at Python Theano and Lua Torch.

While I lean toward Lua and Torch just because of long familiarity, I don’t want to make the mistake of passing on a better set of tools due to familiarity.  Since this is a new skill set that I am learning, if I need to improve my Python skills to maximize success, I am very willing to do so.

Next:  Lua Torch v. Python Theano

References:
Bruffee, K. A. (1999) Collaborative learning. John Hopkins University Press
Slige, J. S. & Robinson, D. (2017) Text mining with R: A tidy approach.  O’Reilly Media
Slige, J.s. (2016) Data science-ish (blog)

Tags: AI, artificial intelligence, Deep Learning, Lua, Machine Learning, Python, Theano, Torch