Weg's Tutorials

The predictive function behind the action picker or the state value estimator can be whatever you want (ex: linear regression, nearest neighbor, genetic algorithm, random forest). However i will be using a single neural network for both of them.

class ActorCriticNetwork(torch.nn.Module): #   ~here is my handle~
    def __init__(self, lr, inputDims, numActions):
        super().__init__()
        # fc means "fully connected layer", 
        self.fc1Dims = 1024  #   size of hidden layer 1
        self.fc2Dims = 512   #   size of hidden layer 2

        #   backbone network
        self.fc1 = nn.Linear(*inputDims, self.fc1Dims)     #   hidden layer 1
        self.fc2 = nn.Linear(self.fc1Dims, self.fc2Dims)   #   hidden layer 2

        #   tail networks       ~here is my spout~
        self.actor = nn.Linear(self.fc2Dims, self.numActions)   #   here is the actor
        self.critic = nn.Linear(self.fc2Dims, 1)                #   here is the critic
        
        #   pytorch stuff
        self.optimizer = optim.Adam(self.parameters(), lr=lr)
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.to(self.device)

    def forward(self, observation): #   ~tip me over and pour me out~
        state = torch.tensor(observation).to(self.device)
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        policy = self.actor(x)  #   actor outputs one number for each action, a vector
        value = self.critic(x)  #   critic just puts out the value, one number
        return policy, value

Picking an Action

For vanilla Deep Q learning it's as easy as picking the highest actor action output. :)

def chooseAction(state):
    policy, _ = actorCriticNetwork(state)
    action = torch.argmax(policy).item()    #   pick here
    return action

But this isnt Vanilla Deep Q Learning. This is Actor-Critic Method. >:()
The actor outputs probabilities of actions instead of actual action values.
So the Eat: 0.8 from earlier, yeah thats like an 80% chance of eating the burger.
If this was DQ Learning a value of 0.8 being the highest would mean 100% chance of eating the poison burger. :(

Lets choose an action the Actor Critic way.
We start by passing in the state from the environment into the critic. I don't know why this convention, but people often call the state: "observation". Maybe because it's what the agent "observes"?

def chooseAction(observation):
    policy, _ = actorCritic.forward(observation)
    policy = F.softmax(policy, dim=0)
    actionProbs = torch.distributions.Categorical(policy)
    action = actionProbs.sample()
    self.logProbs = actionProbs.log_prob(action)    #    saving this value for later
    return action.item()

Determinism

Why is our actor putting out chances of actions? Seems kind of stupid doesn't it? Imagine that every millisecond you are holding a knife you have a 10% chance of letting go of it. That's dumb right? Why not just pick the highest action instead of sampling a distribution? Well most of the time the next millisecond you will decide to hold the knife tightly again. So statistically it works out. You will never completely drop the knife. The Actor-Critic algorithm statistically makes good decisions, but it might make a different decision given the same exact scenario a second time. It is non deterministic.
In the real world with fuzzy inputs like muscle flexing and rotation angles, picking the wrong action for small fraction of a second is not that big of an issue. For a game where a single wrong button makes you lose, it kind of sucks.
Some reinforcement learning algorithms ARE deterministic. But that has its own downsides. Non-deterministic algorithms naturally explore different actions by accident. They try new things sometimes. Deterministic algorithms can get stuck, repeating the same action every time, until it gets punished into oblivion. Sometimes deterministic algorithms are so stubborn they will pick the same action until it has been punished so much that it will never choose that action ever again. ever. You will likely even see this happen if you keep playing with AI Gym and Deep Q Learning. Deterministic algorithms often need some noise injected into the decision making process to help them be less stubborn. Luckily for us, Actor-Critic doesn't need that because it is non-deterministic.

Improving Our Choices In Life

The actor network and the critic network both have unique errors.
Each network is punished by their own respective error, and their estimation of their respective responsibility improves. We get better actions from the actor, and better value estimates from the critic.

def learn(state, reward, nextState, done): following the meta
    actorCritic.optimizer.zero_grad()  #   pytorch stuff: resets all the tensor derivatives

    #   fetch values from the critic network
    _, criticValue = actorCritic.forward(state)            #   the value of now
    _, nextCriticValue = actorCritic.forward(nextState)    #   the value of the future

    #   the temporal difference zone
    #   #   the true value of now = now + future
    valueOfNow = reward + gamma * nextCriticValue * (1 - int(done))    
    temporalDifference = valueOfNow - criticValue    oh how wrong we were

    #   compute the error for our actor and critic networks
    actorLoss = -self.logProbs * temporalDifference  #  probability of chosen action times how wrong we were
    criticLoss = temporalDifference**2   #   we dont care about which direction (the sign), 
                                         #   #   just wanna minimize how wrong we were in total

    (actorLoss + criticLoss).backward()
    actorCritic.optimizer.step()

One Quick Trick

(actorLoss + criticLoss).backward()

This might seem weird if you know pytorch. Why not just call backward on actorLoss, then on criticLoss? Well, because they share a backbone network, you can't do that. Pytorch will complain about "unreachable gradient" stuff when you call step(). And, actually, I think it computes the wrong gradient. Order will matter for the shared backbone network. Luckily for us the derivative of addition is 1. So adding the losses togethor just multiplies the gradient by 1. Multiplying things by 1 doesn't change them. :^)
If you want to split your network in all sorts of crazy ways this works for adding as many as you want togethor.

Formal Representation

If you want to see the way mathemagicians formalize temporal difference, here you go. I urge you to give the equations a look. I bet you can kinda get them. You're gonna need to wet your whistle on these a little bit if you want to go implement algorithms from professional reinforcement learning papers.

A Guide For The Overly Formal Notation:

total = 0
for i in range(0, 10 + 1):
    total += i

For the most part reinforcement learning math boils down to formalizing these questions:
What's the value of now? What's the value of the future? What's the value of an action? What's the total value of all the actions i took so far? What's the total value of all the best possible actions?

Loss

So just like most neural network stuff we want the loss to go to zero. But, in reinforcement learning its a little more subtle than that. The actor and critic both have different losses. And they function fairly differently as well.
This is where we focus on these lines specifically:

actorLoss = -self.logProbs * temporalDifference
criticLoss = temporalDifference**2

Agent

A reinforcement learning agent is what learns about the environment and chooses the actions.
There are many different types of agents, but this one contains an actor and a critic.
An agent class doesn't need to explicitly exist but doing it this way is fairly neat, and makes it easy to swap in different agents in our main loop when we want to try out different ones.
To make the agent we just put the learn and choose action functions togethor with the actor critic network from earlier.
There really isn't anything new here. Just notice it's where we save our logProb when we choose an action.

class ActorCriticAgent():
    def __init__(self, lr, inputDims, numActions):
        self.gamma = 0.99    #   a common gamma value
        self.actorCritic = ActorCriticNetwork(lr, inputDims, numActions)
        self.logProbs = None    #   log of the probability of the last action the agent chose

    def chooseAction(self, observation):
        policy, _ = self.actorCritic.forward(observation)
        policy = F.softmax(policy, dim=0)
        actionProbs = torch.distributions.Categorical(policy)
        action = actionProbs.sample()
        self.logProbs = actionProbs.log_prob(action)    #   save it here
        return action.item()

    def learn(self, state, reward, nextState, done):
        self.actorCritic.optimizer.zero_grad()

        _, criticValue = self.actorCritic.forward(state)
        _, nextCriticValue = self.actorCritic.forward(nextState)

        reward = torch.tensor(reward, dtype=torch.float).to(self.actorCritic.device)
        td = reward + self.gamma * nextCriticValue * (1 - int(done)) - criticValue

        actorLoss = -self.logProbs * td
        criticLoss = td**2

        (actorLoss + criticLoss).backward()
        self.actorCritic.optimizer.step()

The Main Loop

All the hard parts are done now. The only thing left is to make our agent, put it into an environment, and trap it in an infinite loop so it can self improve until it becomes skynet. You can create your own environment or use input from the real world, but for demonstration let's use AI Gym. A place where AI can watch their macros and get sick gains. (or fail horribly. Some of the provided AI Gym environments are fairly difficult, and require much more complicated algorithms than this one to solve.)
AI Gym environments are nice little simulated worlds, that happen to return rewards and states just like our actor and critic needs.
What a coencidence. :^) Anyways, put our agent into an AI Gym environment and let it run for 20 minutes to 10 years.

agent = ActorCriticAgent(lr=0.00001, inputDims=(8,), numActions=4) we wrote this earlier
env = gym.make("LunarLander-v2")

highScore = -math.inf
recordTimeSteps = math.inf
while True:                     #   keep starting new episodes forever
    observation = env.reset()   #   observation is just a commonly used term for the environment state
    score, frame, done = 0, 1, False
    while not done:             #   keep going until the episode is done
        env.render()            #   draw it on your screen so you can watch
        action = agent.chooseAction(observation)    we wrote this too
        nextObservation, reward, done, info = env.step(action)  #   make the environment go one time step
        agent.learn(observation, reward, nextObservation, done) and this
        observation = nextObservation
        score += reward
        frame += 1

    recordTimeSteps = min(recordTimeSteps, frame)
    highScore = max(highScore, score)
    print(( "ep {}: high-score {:12.3f}, shortest-time {:d}, "
            "score {:12.3f}, last-epidode-time {:4d}").format(
        episode, highScore, recordTimeSteps, score, frame))

I'm Impatient

So you are watching it go now. How long is it gonna take to land? Well luckily for you it might never land. :^) Reinforcement learning is like this. Sometimes you roll bad dice and your child comes out with three hands and no fingers. The initial neuron weights could be bad, your agent gets stuck in a local minima, theres no end to the number of things that can go wrong in these environments. It can be very difficult to find bugs in RL code for this reason. Even if the code is correct, you still might get bad results.
The code above usually makes a successful smooth landing somewhere between 75 and 300 episodes of learning. Probably it will take 30 minutes. Actor-Critic is simple and not terribly good at this environment. Sorry. It kind of sucks.

So You Made Me Read All That Shit For Nothing?

Actually you've learned a lot here. You have a lot of intuition for how the thing 'learns' now. With just a few tweaks to your code you can make substantially better agents. As it turns out I'm working on tutorials for those agents here. :^)

Wisdom

An hour or few has passed. Maybe a few days. You've already sent videos to all your family and friends, and your mom, of your little fake moon landing, and now the excitement of making your first AI is wearing off. You're probably feeling a bit let down. You know how this stuff works and the magic has ended. You might even feel a bit betrayed.
"This THING isn't alive. It doesnt LEARN anything. It can never gain consciousness. It's just probabilities and math.
AI IS A BIG LIE. A SCAM. I WANT A DIVORCE"
You probably have some questions and complaints about taking abstract life wisdom and implementing it in such a literal way in math. And also questions about why we do so in this specific way.

Cognition Algorithms

Your brain is a computer. It is constantly evaluating the value of its particular actions, and picking what it thinks to be the best ones. Maybe it is doing so in a much more complicated way. Maybe there are thousands of "actors" and "critics", seperate zones of neurons competing over decisions. Maybe your brain has an internal model of the world for trying out different actions in.
Reinforcement learning is a blatant oversimplification. Inspired by the way you think. It shrinks down the problem for practicality reasons. It keeps the decision making essence just enough to be metaphorically sound. Most of reinforcement learning progress seems to fall into two categories:

• Improving an existing algorithm by engineering a new feature into it to increase learning rate or stability.
• Creating small simplified code versions of some specific part of your human cognition.

The distinction between these two categories is blurred. Often the inspiration for what seems to be an engineering feature is a metaphor for some aspect of your cognition, and then you bounce back and forth between the two.
Ex: "Experience Replay" is like memory. It's a list of previous states and rewards.
Ex: Some of my memories are more important than others. If i remembered all my memories as being equally impactful to my life I would value thinking about yesterdays breakfast as much as the time my wife left me and took the kids. Most of my memories are just noise, and can be thrown out. So make a "Prioritized Experience Replay" where you estimate memory value and discard low value ones.
Ex: My Agent keeps getting stuck in local minima, I want it to explore. So bias it to value discovering new states or trying new actions that it hasnt seen before.
It goes back and forth. Cognitive metaphor, engineering, cognitive metaphor, engineering.
How about creativity? The notion of friend or foe? Ownership, fairness, or maybe mate value estimation? :^)
There is no end to the possibilities. If you have enough computing power, you can make your agent that thinks it.
And of course if it's brain gets big enough, and self observant enough, then it can have consciousness. It may even learn to infer things about its own internal workings by focusing on generalizing across its strategies within varying circumstances. Maybe it can even be given the value of making more and more agents. :^)

History and Moving Forward

Before we make horny von neumman probes there is a lot of work to do and much to learn. It might surprise you to discover reinforcement learning has a long history behind it. Though many of the "RL" algorithms are recently created ( or discovered :^) ), the simplest versions of these algorithms have been around since the 1950s and maybe even earlier in other forms. There is a lot of historical baggage and convention. If you have lots of questions about this you might choose to peruse some of the other resources, but be warned there are many math symbols out there. Even if you aren't math fluent (I am also not math fluent :() ), you can still learn stuff from understanding 5-10% of the symbols. Though it might take you a month or so off and on.
Some noteable books and resources include:

• Artificial Intelligence: by Stuart Russell and Peter Norvig
• Grokking Deep Reinforcement Learning (ch1-3, but skip/skim chapters 4 through 7 though (bad), but you can read the rest)
• PDF's of actual research papers. (surprisingly simple enough to follow, and fairly short by research paper standards)
• Not surprisingly you can learn a bunch from youtube
• The rest of the tutorials that I am making.