Hands-On Reinforcement Learning with R

RLwithR

Reinforcement learning (RL) is an integral part of machine learning (ML), and is used to train algorithms. With this book, you’ll learn how to implement reinforcement learning with R, exploring practical examples such as using tabular Q-learning to control robots.

You’ll begin by learning the basic RL concepts, covering the agent-environment interface, Markov Decision Processes (MDPs), and policy gradient methods. You’ll then use R’s libraries to develop a model based on Markov chains. You will also learn how to solve a multi-armed bandit problem using various R packages. By applying dynamic programming and Monte Carlo methods, you will also find the best policy to make predictions. As you progress, you’ll use Temporal Difference (TD) learning for vehicle routing problem applications. Gradually, you’ll apply the concepts you’ve learned to real-world problems, including fraud detection in finance, and TD learning for planning activities in the healthcare sector. You’ll explore deep reinforcement learning using Keras, which uses the power of neural networks to increase RL’s potential. Finally, you’ll discover the scope of RL and explore the challenges in building and deploying machine learning models.

By the end of this book, you’ll be well-versed with RL and have the skills you need to efficiently implement it with R.

  • Understand how to use MDP to manage complex scenarios
  • Solve classic reinforcement learning problems such as the multi-armed bandit model
  • Use dynamic programming for optimal policy searching
  • Adopt Monte Carlo methods for prediction
  • Apply TD learning to search for the best path
  • Use tabular Q-learning to control robots
  • Handle environments using the OpenAI library to simulate real-world applications
  • Develop deep Q-learning algorithms to improve model performance

Hands-On Reinforcement Learning with R

The biggest Christmas sale

My Books on Python, Matlab and R at prices never seen before

5dollar-sm-2019_fb-linkedin

A fixture in the Packt calendar, the $5 campaign is something developers across the industry look forward to every year and 2019 isn’t looking to be any different. Packt Publishing always believed in making tech learning as accessible as possible, and by offering every eBook & video for $5 across the site throughout December they give developers the chance to grow their skills, indulge their curiosity and more for less.

You can buy my ebooks and videos for $ 5. These are the titles:

Implement key reinforcement learning algorithms and techniques using different R packages such as the Markov chain, MDP toolbox, contextual, and OpenAI Gym

Discover powerful ways to effectively solve real-world machine learning problems using key libraries including scikit-learn, TensorFlow, and PyTorch

Demonstrate fundamentals of Deep Learning and neural network methodologies using Keras 2.x

A practical guide to mastering reinforcement learning algorithms using Keras

Uncover the power of artificial neural networks by implementing them through R code.

Extract patterns and knowledge from your data in easy way using MATLAB

Build effective regression models in R to extract valuable insights from real data

Click here to get a preview of the available titles

Linear Decision Boundary of Logistic Regression

Now, we will study the concept of a decision boundary for a binary classification problem. We use synthetic data to create a clear example of how the decision boundary of logistic regression looks in comparison to the training samples. We start by generating two features, X1 and X2, at random. Since there are two features, we can say that the data for this problem are two-dimensional. This makes it easy to visualize. The concepts we illustrate here generalize to cases of more than two features, such as the real-world datasets you’re likely to see in your work; however, the decision boundary is harder to visualize in higher-dimensional spaces.

Perform the following steps:

  1. Generate the features using the following code:
np.random.seed(seed=6)
X_1_pos = np.random.uniform(low=1, high=7, size=(20,1))
print(X_1_pos[0:3])
X_1_neg = np.random.uniform(low=3, high=10, size=(20,1))
print(X_1_neg[0:3])
X_2_pos = np.random.uniform(low=1, high=7, size=(20,1))
print(X_1_pos[0:3])
X_2_neg = np.random.uniform(low=3, high=10, size=(20,1))
print(X_1_neg[0:3])

You don’t need to worry too much about why we selected the values we did; the plotting we do later should make it clear. Notice, however, that we are also going to assign the true class at the same time. The result of this is that we have 20 samples each in the positive and negative classes, for a total of 40 samples, and that we have two features for each sample. We show the first three values of each feature for both positive and negative classes.

The output should be the following:

IMG0

Generating synthetic data for a binary classification problem

  1. Plot these data, coloring the positive samples in red and the negative samples in blue. The plotting code is as follows:
plt.scatter(X_1_pos, X_2_pos, color='red', marker='x')
plt.scatter(X_1_neg, X_2_neg, color='blue', marker='x')
plt.xlabel('$X_1$')
plt.ylabel('$X_2$')
plt.legend(['Positive class', 'Negative class'])

The result should look like this:

 IMG1

Generating synthetic data for a binary classification problem

In order to use our synthetic features with scikit-learn, we need to assemble them into a matrix. We use NumPy’s block function for this to create a 40 by 2 matrix. There will be 40 rows because there are 40 total samples, and 2 columns because there are 2 features. We will arrange things so that the features for the positive samples come in the first 20 rows, and those for the negative samples after that.

  1. Create a 40 by 2 matrix and then show the shape and the first 3 rows:
X = np.block([[X_1_pos, X_2_pos], [X_1_neg, X_2_neg]])
print(X.shape)
print(X[0:3])

The output should be:

 IMG2

Combining synthetic features in to a matrix

We also need a response variable to go with these features. We know how we defined them, but we need an array of y values to let scikit-learn know.

  1. Create a vertical stack (vstack) of 20 1s and then 20 0s to match our arrangement of the features and reshape to the way that scikit-learn expects. Here is the code:
y = np.vstack((np.ones((20,1)), np.zeros((20,1)))).reshape(40,)
print(y[0:5])
print(y[-5:])

You will obtain the following output:

 IMG3

Create the response variable for the synthetic data

At this point, we are ready to fit a logistic regression model to these data with scikit-learn. We will use all of the data as training data and examine how well a linear model is able to fit the data.

  1. First, import the model class using the following code:
from sklearn.linear_model import LogisticRegression
  1. Now instantiate, indicating the liblinear solver, and show the model object using the following code:
example_lr = LogisticRegression(solver='liblinear')
example_lr

The output should be as follows:

 IMG3BIS

Fit a logistic regression model to the synthetic data in scikit-learn

  1. Now train the model on the synthetic data:
example_lr.fit(X, y)

How do the predictions from our fitted model look?

We first need to obtain these predictions, by using the trained model’s .predict method on the same samples we used for model training. Then, in order to add these predictions to the plot, using the color scheme of red = positive class and blue = negative class, we will create two lists of indices to use with the arrays, according to whether the prediction is 1 or 0. See whether you can understand how we’ve used a list comprehension, including an if statement, to accomplish this.

  1. Use this code to get predictions and separate them into indices of positive and negative class predictions. Show the indices of positive class predictions as a check:
y_pred = example_lr.predict(X)
positive_indices = [counter for counter in range(len(y_pred)) if y_pred[counter]==1]
negative_indices = [counter for counter in range(len(y_pred)) if y_pred[counter]==0]
positive_indices

The output should be:

IMG4

Positive class prediction indices

  1. Here is the plotting code:
plt.scatter(X_1_pos, X_2_pos, color='red', marker='x')
plt.scatter(X_1_neg, X_2_neg, color='blue', marker='x')
plt.scatter(X[positive_indices,0], X[positive_indices,1], s=150, marker='o',
edgecolors='red', facecolors='none')
plt.scatter(X[negative_indices,0], X[negative_indices,1], s=150, marker='o',
edgecolors='blue', facecolors='none')
plt.xlabel('$X_1$')
plt.ylabel('$X_2$')
plt.legend(['Positive class', 'Negative class', 'Positive predictions', 'Negative predictions'])

The plot should appear as follows:

 IMG5

Predictions and true classes plotted together

From the plot, it’s apparent that the classifier struggles with data points that are close to where you may imagine the linear decision boundary to be; some of these may end up on the wrong side of that boundary. Use this code to get the coefficients from the fitted model and print them:

theta_1 = example_lr.coef_[0][0]
theta_2 = example_lr.coef_[0][1]
print(theta_1, theta_2)

The output should look like this:

IMG6

Coefficients from the fitted model

  1. Use this code to get the intercept:
theta_0 = example_lr.intercept_

Now use the coefficients and intercept to define the linear decision boundary. This captures the dividing line of the inequality, X2 ≥ -(1/2)X1 – (0/2):

X_1_decision_boundary = np.array([0, 10])
X_2_decision_boundary = -(theta_1/theta_2)*X_1_decision_boundary - (theta_0/theta_2)

To summarize the last few steps, after using the .coef_ and .intercept_ methods to retrieve the model coefficients 12 and the intercept 0, we then used these to create a line defined by two points, according to the equation we described for the decision boundary.

  1. Plot the decision boundary using the following code, with some adjustments to assign the correct labels for the legend, and to move the legend to a location (loc) outside a plot that is getting crowded:
pos_true = plt.scatter(X_1_pos, X_2_pos, color='red', marker='x', label='Positive class')
neg_true = plt.scatter(X_1_neg, X_2_neg, color='blue', marker='x', label='Negative class')
pos_pred = plt.scatter(X[positive_indices,0], X[positive_indices,1], s=150, marker='o',
edgecolors='red', facecolors='none', label='Positive predictions')
neg_pred = plt.scatter(X[negative_indices,0], X[negative_indices,1], s=150, marker='o',
edgecolors='blue', facecolors='none', label='Negative predictions')
dec = plt.plot(X_1_decision_boundary, X_2_decision_boundary, 'k-', label='Decision boundary')
plt.xlabel('$X_1$')
plt.ylabel('$X_2$')
plt.legend(loc=[0.25, 1.05])

You will obtain the following plot:

 IMG7

True classes, predicted classes, and the decision boundary of a logistic regression

In this post, we discuss the basics of logistic regression along with various other methods for examining the relationship between features and a response variable.  To know, how to install the required packages to set up a data science coding environment, read the book Data Science Projects with Python on Packt Publishing.

Why is Logistic Regression Considered a Linear Model?

A model is considered linear if the transformation of features that is used to calculate the prediction is a linear combination of the features. The possibilities for a linear combination are that each feature can be multiplied by a numerical constant, these terms can be added together, and an additional constant can be added. For example, in a simple model with two features, X1 and X2, a linear combination would take the form:

FOR1

Linear combination of X1 and X2

The constants i, can be any number, positive, negative, or zero, for i = 0, 1, and 2 (although if a coefficient is 0, this removes a feature from the linear combination). A familiar example of a linear transformation of one variable is a straight line with the equation y = mx + b. In this case, o = b and 1 = mo is called the intercept of a linear combination, which should make sense when thinking about the equation of a straight line in slope-intercept form like this.

However, while these transformations are not part of the basic formulation of a linear combination, they could be added to a linear model by engineering features, for example defining a new feature, X3 = X12.

Predictions of logistic regression, which take the form of probabilities, are made using the sigmoid function. This function is clearly non-linear and is given by the following:

 FOR2

Non-linear sigmoid function

Why, then, is logistic regression considered a linear model? It turns out that the answer to this question lies in a different formulation of the sigmoid equation, called the logit function. We can derive the logic function by solving the sigmoid function for X; in other words, finding the inverse of the sigmoid function. First, we set the sigmoid equal to p, the probability of observing the positive class, then solve for X as shown in the following:

 FOR3

Solving for X

Here, we’ve used some laws of exponents and logs to solve for X. You may also see the logit expressed as:

 FOR4

Logit function

The probability of failure, q, is expressed in terms of the probability of successp: q = 1 – p, because probabilities sum to 1. Even though in our case, credit default would probably be considered a failure in the sense of real-world outcomes, the positive outcome (response variable = 1 in a binary problem) is conventionally considered “success” in mathematical terminology. The logit function is also called the log odds, because it is the natural logarithm of the odds ratiop/q. Odds ratios may be familiar from the world of gambling, via phrases such as “the odds are 2 to 1 that team a defeats team b.”

In general, what we’ve called capital X in these manipulations can stand for a linear combination of all the features. For example, this would be X = o + 1X1 + 2X2 in our simple case of two features. Logistic regression is considered a linear model because the features included in X are, in fact, only subject to a linear combination when the response variable is considered to be the log odds. This is an alternative way of formulating the problem, as compared to the sigmoid equation.

In summary, the features X1X2,…, Xj look like this in the sigmoid equation version of logistic regression:

 FOR5

Sigmoid version of logistic regression

But they look like this in the log odds version, which is why logistic regression is called a linear model:

 FOR6

Log odds version of logistic regression

Because of this way of looking at logistic regression, ideally the features of a logistic regression model would be linear in the log odds of the response variable.

This post is taken from the book Data Science Projects with Python by Packt Publishing written by Stephen Klosterman. The book explains descriptive analyses for future operations using predictive models.

Packt Humble Bundle package

I am pleased to announce that my title, Python Machine Learning Cookbook, is part of the Packt Humble Bundle package, currently on sale on the Humble Bundle website until May 27, 2019.

PMLC

Humble Bundle is a distribution platform that sells games, ebooks, software, and other digital content. Since their inception in 2010, their mission has been to support charities (“Humble”) while providing awesome content to customers at great prices (“Bundle”). So far they helped to raise of $140m for a number of featured charities.

www.humblebundle.com

Keras 2.x Projects

9 projects demonstrating faster experimentation of neural network and deep learning applications using Keras.

Keras 2.x Projects explains how to leverage the power of Keras to build and train state-ofthe-
art deep learning models through a series of practical projects that look at a range of
real-world application areas.

To begin with, you will quickly set up a deep learning environment by installing the Keras
library. Through each of the projects, you will explore and learn the advanced concepts of
deep learning and will learn how to compute and run your deep learning models using the
advanced offerings of Keras. You will train fully-connected multilayer networks,
convolutional neural networks, recurrent neural networks, autoencoders and generative
adversarial networks using real-world training datasets. The projects you will undertake
are all based on real-world scenarios of all complexity levels, covering topics such as
language recognition, stock volatility, energy consumption prediction, faster object
classification for self-driving vehicles, and more.

Keras2.xProjects

By the end of this book, you will be well versed with deep learning and its implementation
with Keras. You will have all the knowledge you need to train your own deep learning
models to solve different kinds of problems.

Keras 2.x Projects

Keras Reinforcement Learning Projects

9 projects exploring popular reinforcement learning techniques to build self-learning agents

Reinforcement learning has evolved a lot in the last couple of years and proven to be a successful technique in building smart and intelligent AI networks. Keras Reinforcement Learning Projects installs human-level performance into your applications using algorithms and techniques of reinforcement learning, coupled with Keras, a faster experimental library. In the following the link at the book:

kerasrlsmall

The book begins with getting you up and running with the concepts of reinforcement learning using Keras. You’ll learn how to simulate a random walk using Markov chains and select the best portfolio using dynamic programming (DP) and Python. You’ll also explore projects such as forecasting stock prices using Monte Carlo methods, delivering vehicle routing application using Temporal Distance (TD) learning algorithms, and balancing a Rotating Mechanical System using Markov decision processes.

Once you’ve understood the basics, you’ll move on to Modeling of a Segway, running a robot control system using deep reinforcement learning, and building a handwritten digit recognition model in Python using an image dataset. Finally, you’ll excel in playing the board game Go with the help of Q-Learning and reinforcement learning algorithms.

By the end of this book, you’ll not only have developed hands-on training on concepts, algorithms, and techniques of reinforcement learning but also be all set to explore the world of AI.

Why is OOP in R so confusing?

Author: Omar Trejo Navarro

Apart from the fact that R is an interpreted language, which can pretty much confuse people used to compiled languages, most programmers would agree that manipulating objects in R can sometimes be a nightmare. This has to do with two significant features of R that make OOP manipulation highly efficient:

  • R has several object models unlike Java, Python and others, which have only one
  • R implements parametric polymorphism, while other OOP (Object-oriented programming) languages generally implement polymorphism from inside an object

Let’s look at these two in a bit more detail.

Various object models

Working with OOP in R is different from what you may see in other languages such as Python, Java, C++, etc. These languages have a single object model. However, R has various object models, meaning that it lets us implement object-oriented systems in various ways. Specifically, R has the following object models:

  • S3
  • S4
  • Reference Classes
  • R6
  • Base Types

OOP1

In this article, we will look at S3, S4, and R6, since these are the most-used object models in R, but more on that later. Let’s first understand what parametric polymorphism is.

Parametric polymorphism

R implements parametric polymorphism, which implies that methods in R belong to functions, not classes. Parametric polymorphism basically lets you define a generic method or function for types of objects you haven’t yet defined and may never do. This means that you can use the same name for several functions with different sets of arguments and from various classes.

R’s method calls look just like function calls and R must have a mechanism to know which names require simple function calls and which names require method calls. This mechanism is called generics. Generics allow you to register certain names to be treated as methods in R, and they act as dispatchers. When these registered generic functions are called, R looks into a chain of attributes in the object being passed in the call in order to look for functions that match the method call for that object’s type. If it finds a function, it will call it.

You may have noted that the plot() and summary() functions in R may return different results depending on the objects being passed to them (e.g. a data frame or a linear model instance). This is because they are generic functions that implement polymorphism. This way of working provides simple interfaces for users and makes their tasks much simpler. For instance, if you are exploring a new package and you get some kind of result at some point derived from the package, try calling plot(result), and you may be surprised to get some kind of plot that makes sense. This is not common in other OOP languages.

Now that we have a basic idea of what parametric polymorphism is and how it sets R apart from other OOP languages, let’s look at S3, S4, and R6.

R’s most common object models

The S3 object model of R owes its roots to the object model of the S language, the predecessor of R. S’s object model evolved over time, and its third version introduced class attributes, which paved the way for S3. S3 is the fundamental object model of R, and many of R’s own built-in classes are of the S3 type.

Being the least formal object model in R, S3 does lack in some key respects:

  • It has no formal class definitions, thereby leaving no scope for inheritance or encapsulation
  • Polymorphism can only be implemented through generics

Nevertheless, it makes up for what it lacks in by providing quite a lot of flexibility to the programmers. In the words of Hadley Wickham, “S3 has a certain elegance in its minimalism: you can’t take away any part of it and still have a useful object-oriented system“.

However, one of the biggest demerits of S3 is that it doesn’t provide the required safety; it is very easy to create a class in S3, but it may lead to very confusing and hard-to-debug code if not used with utmost care. For example, a programmer could misspell a function and S3 would raise no issues. This is why S4 came into the picture, developed with the goal of adding safety. It keeps the code safe but, on the flip side, introduces a lot of verbosity to the code. It implements most features of modern object-oriented programming languages:

  • Formal class definitions
  • Inheritance
  • Polymorphism (parametric)
  • Encapsulation

In reality, S3 and S4 are really just ways to implement polymorphism for static functions. The R6 package provides a type of class that is similar to R’s Reference Classes, but it is more efficient and doesn’t depend on S4 classes and the methods package, as Reference Classes do.

When Reference Classes were introduced, some users called the new class system R5, following the names of R’s existing class systems S3 and S4. Although Reference Classes are not actually called R5 nowadays, the name of this package and its classes follow the pattern. Despite being first released over 3 years ago, R6 isn’t widely known. However, it is widely used. For example, R6 is used within Shiny and to manage database connections in the dplyr package.

The decision of which object model to use eventually boils down to a trade-off between flexibility, formality and code cleanness.

R Programming by Example

R is a high-level statistical language and is widely used among statisticians and data miners to develop analytical applications. Given the obvious advantages that R brings to the table, it has become imperative for data scientists to learn the essentials of the language.

R Programming by Example is a hands-on guide that helps you develop a strong fundamental base in R by taking you through a series of illustrations and examples. Written by Omar Trejo Navarro, the book starts with the basic concepts and gradually progresses towards more advanced concepts to give you a holistic view of R. Omar is a well-respected data consultant with expertise in applied mathematics and economics.

If you are an aspiring data science professional or statistician and want to learn more about R programming in a practical and engaging manner, R Programming by Example is the book for you!

OOP2

Regression Analysis with R

Design and develop statistical nodes to identify unique relationships within data at scale

Regression analysis is a statistical process which enables prediction of relationships between variables. The predictions are based on the casual effect of one variable upon another. Regression techniques for modeling and analyzing are employed on large set of data in order to reveal hidden relationship among the variables.

Regression Analysis with R

This book will give you a rundown explaining what regression analysis is, explaining you the process from scratch. The first few chapters give an understanding of what the different types of learning are – supervised and unsupervised, how these learnings differ from each other. We then move to covering the supervised learning in details covering the various aspects of regression analysis. The outline of chapters are arranged in a way that gives a feel of all the steps covered in a data science process – loading the training dataset, handling missing values, EDA on the dataset, transformations and feature engineering, model building, assessing the model fitting and performance, and finally making predictions on unseen datasets. Each chapter starts with explaining the theoretical concepts and once the reader gets comfortable with the theory, we move to the practical examples to support the understanding. The practical examples are illustrated using R code including the different packages in R such as R Stats, Caret and so on. Each chapter is a mix of theory and practical examples.

By the end of this book you will know all the concepts and pain-points related to regression analysis, and you will be able to implement your learning in your projects.

Regression Analysis with R

Tipi di variabili in R

python

In questo post approfondiremo il concetto di variabili in ambiente R fornendo una spiegazione sulle differenti tipologie di variabili.Il linguaggio R prevede due tipi di variabili:

1)     variabili globali;

2)     variabili locali;

Come si può intuire, le variabili globali sono accessibili a livello globale all’interno del programma, le variabili locali invece assumono significato solo ed esclusivamente nel settore di appartenenza, risultando visibili solo all’interno del metodo in cui vengono inizializzate.

Per la maggior parte dei compilatori, un nome di variabile può contenere fino a trentuno caratteri, in modo da poter adottare per una variabile un nome sufficientemente descrittivo, in R tale limite non è indicato. La scelta del nome assume un’importanza fondamentale al fine di rendere leggibile il codice; questo perché un codice leggibile sarà facilmente mantenibile anche da persone diverse dal programmatore che l’ha creato.

Abbiamo parlato d’inizializzazione della variabile intesa quale operazione di creazione della variabile; vediamone allora un esempio banale:

> a <- 1

In tale istruzione è stato utilizzato l’operatore di assegnazione (<-), con il significato di assegnare appunto alla locazione di memoria individuata dal nome a il valore 1. Il tipo attribuito alla variabile è stabilito in fase d’inizializzazione; sarà allora che si deciderà se assegnare a essa una stringa di testo, un valore booleano (true/false), un numero decimale etc.

Per approfondire l’argomento:

 

Variabili ed espressioni in R

python

Le Variabili ed espressioni in R sono trattate in modo moderno ed efficiente, infatti mentre nella maggior parte dei linguaggi di programmazione è necessaria una dichiarazione delle variabili utilizzate all’interno del programma, dichiarazione effettuata nella parte iniziale prima della sezione esecutiva dello stesso, in R tutto questo non è richiesto. Poiché il linguaggio non richiede la dichiarazione delle variabili; il tipo e la relativa dimensione saranno decisi nel momento in cui le stesse saranno inizializzate.

Con il termine variabile ci si riferisce a un tipo di dato il cui valore è variabile nel corso dell’esecuzione del programma. È però possibile assegnarne un valore iniziale, si parlerà allora d’inizializzazione della variabile. La fase d’inizializzazione, assume un’importanza fondamentale perché rappresenta il momento in cui la variabile è creata, tale momento coincide con quello in cui a essa è associato un dato valore.

A differenza dei linguaggi cosiddetti compilativi tale procedura può essere inserita in qualunque punto dello script, anche se i significati possono assumere valori differenti.

Nel momento in cui l’interprete s’imbatte in una variabile, deposita il valore relativo in una locazione di memoria e ogni volta che nel programma comparirà una chiamata a tale variabile, si riferirà a questa locazione. È regola di buona programmazione utilizzare dei nomi che ci permetteranno di riferirci in maniera univoca alle specifiche locazioni di memoria in cui i relativi dati sono stati depositati.

Per approfondire l’argomento: