Weg's Tutorials

Let's set up our network that will predict action outcomes, Q Values.

class Network(torch.nn.Module):
    def __init__(self, lr, inputShape, numActions):
        super().__init__()
        self.inputShape = inputShape
        self.numActions = numActions
        self.fc1Dims = 1024
        self.fc2Dims = 512

        #   layers
        self.fc1 = nn.Linear(*self.inputShape, self.fc1Dims)
        self.fc2 = nn.Linear(self.fc1Dims, self.fc2Dims)
        self.fc3 = nn.Linear(self.fc2Dims, numActions)  # output 1 outcome estimate per action

        #   pytorch stuff
        self.optimizer = optim.Adam(self.parameters(), lr=lr)
        self.loss = nn.MSELoss()
        # self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.device = torch.device("cpu")
        self.to(self.device)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x

Choose Wisely (Greedily)

Let's make the agent now.
As I said before, the agent will pick the action with the highest value estimation. Why? Well, if you wanted to make an agent that got progressively worse at things, you could pick the lowest. :^)

def chooseAction(self, observation):
    # your env state probably comes in as a numpy array of floats. So we have to put it in a tensor
    state = torch.tensor(observation).float()
    state = state.to(self.dqn.device) # and put it on the gpu/cpu
    state = state.unsqueeze(0)  # pytorch likes inputs in a specific shape (1, 8) instead of just (8)

    qValues = self.dqn(state) # pass it through the network to get your estimations
    action = torch.argmax(qValues) # pick the highest
    return action.item()  # return an int instead of a tensor containing the index of the best action

This greedy selection might feel wrong to you. Sometimes we need to pick the action that is worse in the short term but gives us better options in the long term, right? Your intuition is good. However, although the QValues start off as short sighted, if an action gives the agent better options in the long term, and the agent is training well, then the QValue of the long-term action will slowly increase throughout training until it beats out the greedy actions.
An example of this might be in pacman, where the agent has to pick between two directions. One direction might have lots of dots to eat, but lead to a dead end. The other direction might have no dots, but lead to a new section of the maze full of even more dots to eat that the first direction.
At first you can expect your agent to go with the easy reward. In fact... sometimes it might never figure out the long term best action. Anyways the point is the QValues are not straightforward to interpret. It can be really difficult to figure out if the agent is being nearsighted or thinking long term just from the QValues alone. This greedy action selection, ends up being a lot less greedy than you would worry.

Better Forecasting

Unlike in Actor-Critic, this network only has one error to minimize. That one error is the difference between the reward the environment gave us, and what the QValue guessed it would be.

def learn(self, state, reward): 
    self.network.optimizer.zero_grad()  #   pytorch stuff: resets all the tensor derivatives

    # put our states in tensors and such
    state = torch.tensor(state).float().detach()  # detach it so we dont backprop through it
    state = state.to(self.dqn.device) # and put it on the gpu/cpu
    state = state.unsqueeze(0)

    reward = torch.tensor(reward).float().detach()  # have to put the reward in tensor form too
    reward = reward.to(self.network.device) # and on the proper device

    qValues = self.network(state) # predict what reward each action will get
    valueOfBestAction = qValues.max() # assume it took the best action

    # did we accurately predict how much reward the best action would give us?
    loss = self.network.loss(valueOfBestAction, reward) 

    loss.backward() # calculate the influence environment state and action choice had on the reward
    self.network.optimizer.step()   # tweak the weights to reduce the error

The Whole Agent

The agent class holds the network and functions as a nice place to put utilities needed for decision making. You've already seen all this code. It's just in one place now. I'm not a huge fan of unnecessary classes but this Agent will be convenient later when we add stuff to it. Also it is a pretty standard way of doing it.

class Agent():
    def __init__(self, lr, inputShape, numActions=2):
        self.network = Network(lr, inputShape, numActions)

    def chooseAction(self, observation):  # you've seen this code
        state = torch.tensor(observation).float()
        state = state.to(self.dqn.device) 
        state = state.unsqueeze(0)
    
        qValues = self.dqn(state)
        action = torch.argmax(qValues)  # there are a few ways to get the max value and its index (google)
        return action.item()  # .item() grabs the number out of the tensor (also worth a google)

    def learn(self, state, reward): # youve seen this before too
        self.network.optimizer.zero_grad()  #   pytorch stuff: resets all the tensor derivatives
    
        state = torch.tensor(state).float().detach()  # detach it so we dont backprop through it
        state = state.to(self.dqn.device) # and put it on the gpu/cpu
        state = state.unsqueeze(0)
    
        qValues = self.network(state)
        valueOfBestAction = qValues.max()
        loss = self.network.loss(valueOfBestAction, reward) 
    
        loss.backward()
        self.network.optimizer.step()

The Main Loop

All the hard parts are done now. Put the agent, into the environment, and trap it in an infinite loop so it can learn the futility of life and give up.

agent = Agent(lr=0.001, inputShape=(8,), numActions=2)
env = gym.make('CartPole-v1') # this is how you pick the env from ai gym

highScore = -math.inf
while True:                     #   keep starting new episodes forever
    observation = env.reset()   #   observation is just a commonly used term for the environment state
    score, frame, done = 0, 1, False
    while not done:             #   keep going until the env reports the episode is done
        env.render()            #   draw it on your screen so you can watch
        action = agent.chooseAction(observation)
        nextObservation, reward, done, info = env.step(action)  #   make the environment go one time step
        agent.learn(observation, reward, nextObservation, done)  #   make your network more accuracte
        observation = nextObservation
        score += reward
        frame += 1

    highScore = max(highScore, score)
    print(( "ep {}: high-score {:12.3f}, "
            "score {:12.3f}, last-episode-time {:4d}").format(
        episode, highScore, score, frame))

Did It Work?

It didn't did it? Do you have any guesses why?
Go scour the code a bit, and verify each line makes sense to you. Maybe do some printing.
Don't come back until you atleast tried.

Hey. I said stop reading this. Go try.