Comment Successor Byline with airSlate SignNow

Add initial successor

all right hi there today we're looking at a neroli plausible model learned successor representations in partially observable environments by Esther virtus and Manish Shani at this paper is a paper on a topic that has been interesting me for a while and that's successor representations so we'll will dive into all of this the title is very lengthy and complicated but ultimately we're dealing with a setting of reinforcement learning so if you know something about reinforcement learning in reinforcement learning usually you have a agent which you know let's just say this is you and there is an environment which is a big black box that you don't know anything about this is environment and what the environment gives you is what's called an observation so an observation could be anything but in this case let's just assume it's a it's you get a little picture right of what's in front of you so in front of you might be a tree and in front of you might be a house and then you can perform an action a and this action in this case might be to enter the house right and then the environment in the next step it gives you back a new picture and says oh you're now inside the house so here is a door that leads you to this room and the door that leads to that room and there's a little table in front of you alright so there's this it's this just this cycle of action observation and with that you're trying to collect some reward over time now there are different ways of achieving this this this reward over time so basically the reward is going to be for example alright what you could get a reward for finding the kitchen or for going as into as many rooms as possible or you know anything like this so the the other objective is to learn a what's called a policy so which actions to take so action one action to action 3 given the opposite relations that maximizes your rewards there is mainly two ways to go about this there's the model free and the model based reinforcement learning approach let's split them so in the model free model free approach what you're trying to do is you're trying to simply learn a policy and we call this here PI of s and s is your state and the state you can think of it as the observation so in this policy we'll simply output an action and this this is the kind of the simple setup of model free reinforcement learning the important thing here is you're trying to learn this usually there's parameters theta of this policy PI this could be a neural network and theta R then the weights of the neural network so you're trying to learn the neural network such that if you give it a state it just outputs the action so you have this neural network with your state you input the state into layer layer layer layer layer and then it outputs one of maybe three actions go north go south go west and maybe go east right this could be for for actions you're just trying to train the neural network using back prop and the reward signal through what's called the reinforced trick or variants they're off this is model free reinforcement learning it's very it's very easy to implement let's say and it's very applicable and it will simply give you a mapping you know you have to know nothing about how the world works it'll simply tell you at the end if you're in this state do that action and the reward will be high in contrast there is the other world this is the model-based reinforcement learning so in model-based reinforcement learning what you have is a model of the world and the model of the world let's is best described for example if you play chess right so if you play chess and this is a let's glue a simplified chess board here four by four and you have a pawn right here right you have a pawn and you know if I do the action of moving the pawn forward I know the pawn will then be in this square right here alright in the next time step I know that because I have a model of the world and know how the world works and I can predict basically the results of my actions so if you have a model based reinforcement learning setup if you know how the world works you can do something like a search so given your here in the state right you know if I do action one I go to this state if I do action to I go to that state and if I do action three I go to this other state and from each of the states you can then say ah but again I have three actions and I can you know go into these three states go into these maybe here too and maybe here I can go into these actually let's do three as well right and then the question more becomes can you find a path through this thing such that at the end you are in the state that you want to end up right so for example here is outside and then here you can go to the tree to the house or to the field and in the house you can go to the bedroom the bathroom or the kitchen and you know all of this you have a model so you can actually kind of compute what would happen if I do something and then search for the best path whereas in the model free reinforcement learning approach what you simply do is you'd say here is a state then the state is for example I am in the house and now give me the action that would maximize my future reward and you're trying to learn this directly so it's a very different style of reinforcement learning basically one is a one is a pure machine learning approach and the other one is a search problem now you can of course mix and match the two like for example people and alphago have done they have a model-based reinforcement learning that also has kind of learning machine learning elements but in between we have the successor features so the successor representations they are if you will they are somewhere in between the two so they kind of trade off the advantages of model-free where you you only have to learn a function right from state to something with the advantages of model-based the fact that you actually have a bit of an idea of how the world works and can adjust quickly to let's say different reward structures or things like this so what do successor representations do successor representations basically learn how states are connected and this is a classic successor representation so the successor representation M here of policy PI the policy remember is what tells you which action you should take in a given state you you define it as a connection between state I and state J and M of Si as J means given that I am in Si so this could be the kitchen and your goal is a finding that the bedroom and if this is the kitchen given that I am in state si what's the probability that in the future at some point I will transition to si right given that I'm in the kitchen what's the probability that I'll end up in the bedroom at some point in the future and this is formally expressed this is the expectation over your policy and it's the it's the indicator function that the future State sorry this is the the future State T plus K they see K goes from 0 to infinity so for all of the future and s T is the one you're in now so for any future state this is equal to SJ now of course this makes no sense unless you kind of discount have a discount factor here so if you're in state if you're in the bedroom further in the future then this value would be lower so that this value is high if you will transition from si 2 SJ with high probability in the near future and this is a successor representation right it basically tells you if you want to go from state si to state SJ how likely is that in the near future right so if the if this number is high you know that that these two states are closely connected that you can expect to end up in state as J somewhere down the line if you're in Si now and one more representation if you consider the vector M PI of s I given all of the SJ some dot here so this is a vector you can actually compare two states as AI so if one is if you plug in here you plug in the kitchen and then also you plug in the I don't know the garage if they and he will get out two vectors right you get two vectors if those vectors are very similar then you know that if you're in the kitchen or in the garage it doesn't matter you're gonna end up you you have a similar future trajectories basically however if those two vectors are far apart you know that these two states are far apart with respect to your policy so this is pretty cool things you can do with successive representations and I hope this gives you kind of some insight so another neat trick is that if you have a value function so and the value function in this case there's a simplified assumption but you don't actually need it the simplified assumption is that the reward only depends on the state you're in basically it doesn't matter how you get to the state like the actions you perform if you're in a given state or if you're in a given room in the house you'll get some reward like for example if you find the bedroom then you win that's a reward that would only be characterized by the state if that's the case you can compute the value function of the reinforcement learning problem simply by integrating over the over the successor representations so for each state you simply go over all of the possible other states and you ask how likely am I to go to that state and what reward will I have in that state and that's your value function so really pretty simple you can actually learn the successor representations by TD learning by temporal difference learning which is a method that's not applied for us like throughout reinforcement learning especially in thin places like Q learning and yeah and also for learning value functions so pretty neat successor representations this paper then goes from successive representations of individual state to successor representations over continuous space so right now we all have these states state kitchen you go to the bedroom you go to somewhere right and these states were kind of discrete places so there was a house and you have different you know different rooms in the house and you can go between them now we're dealing more with continuous states so you can generalize these successor representations to continuous state by considering not the states themselves but features of the of the state and a feature in this here you have to kind of imagine as binary features and the features let me give like some really dumb examples but maybe it helps you like that one feature could be the smell does it smell in the room like just binary does it smell or doesn't it smell right and then um one feature could there be is there sunlight and then one feature could be ah is it warm right with and these are all binary features and so you could kind of you have to build the features such that if the features are the same then the states should be fairly you know close in whatever since so for example if it smells but there is no sunlight you're probably somewhere in the bathroom like where exactly in XY coordinates you are in the bathroom it doesn't really matter to this as long as like the features are high-end so if it smells and there is no sunlight you're probably somewhere in the bathroom and that makes all the states in the bathroom all the coordinates close together so this is how you have to imagine these features you can define your successor representations exactly the same over these features except that the representation is now not from state I to state J but from a state to a given feature so that means if I am in state s T at the current time what is the probability that in the near future this feature will be high right so if I am right now in the or close to the bathroom let's say the the probability that smell oh sorry this should be a highlight the probability that smell is high in the future is very high right so this this number would be higher it's exactly the same except for these continuous features now and you can do the same thing including defining the value function as a simple linear in multiplication with these features that is an assumption under the assumption that the reward is a linear function of the features of the states which is the analogous assumption to saying that the reward only depends on the state in the linear case or somewhat of an analogous function not entirely alright so you can also learn this by temporal difference learning exactly the same so this is pretty cool these these are these successor representations and you can actually you know if you learn them you have kind of a model of how the world works not as much a model as the model based reinforce learning where you know exactly how it works right here you know exactly how the world works have this model in model free you don't know how the world works at all you simply know oh if I'm in this state and do this action that I'll turn out really well but in the successor representation framework you have you have an idea of what states there are we'll do the discrete case right now so this could be kitchen this could be outdoor this could be bedroom and so you you you have an idea what states there are and so on and how they connect to each other like you say ah from the kitchen I can easily go to the bedroom but I can not as well go to maybe the bathroom from outdoor I can easily go to the kitchen but I can't go to the bedroom and so on so you have kind of an idea of how all of these states connect to each other and that is the success representation you can already see how that helps learning agent a lot if you introduce the successor if you have these success representation and what this this paper deals with in essence is it says okay these successor representations are cool but it has only so far been done in a case where you have full observability and the full observability is the case where you kind of know what state you're in right you kind of know that sorry you are in the kitchen you are outdoors you are in the bedroom that is not known but what if you don't and I mean most problems you don't what if you just have a picture like here right you just see a tree in the house right you don't you kind of have to infer that you are outdoor right and if you're here you just get this picture of a couple of doors on the table and you have to infer that you are now in the living room alright so in essence there is an additional layer of complexity not only do you go from state from state to state to state but you don't actually observe the states what you observe is from each state you observe what are called observations right so you only observe these and you have to infer what they kind of have to guess what the underlying states are in order to know what you should do to get to the next state right you only ever observe the observations so this here is the actual thing this is the kitchen and this here could be a picture of the kitchen right there's a counter there's a stove yeah and so you get kind of what I what I mean in their example they have they simplify this to kind of a toy date to set up where you have this environment and this is this is one beautiful picture I don't why oh well just do you have one this setup and this is this box basically this box and it has this wall right and then you have an agent that is able to walk around in here like with whatever policy the policy determines how it works right but then what you observe is not the actual position but what you observe is for example for this position you observes a random point here so they basically add noise to each observe to each state and if you're in this state you will observe one of these points in this circle right so your your trajectory might look to you as you observe it much more much might go for example from here to here to here to here right and you kind of have to guess what the underlying state is and you see this here this this blue thing is what the agent actually does but the gray thing is what it observes and the observations are sometimes even outside of this above this boundary and this this orange thing is now the infer thing and that's what we actually want is to go from the observed to this inferred and we want to that the inferred is as close as possible to this true latent state right so the way they do it is they introduced this distributional distributed coding for the for the features of basic for the expectation of the features and basically what they say is they say we will build a framework where we can where we represent the features as as expectations over some distribution and the expectation will call mu and mu is simply the the kind of mean of this of this feature under this distribution this is very general so let's look at what and how to plug this in so what they now have to do is they have to learn these two things right they have to first of all if I draw this picture again these are the underlying states and they kind of transition into each other so this is state 1 state 2 state 3 and with action one action two we transition from state to state but also there are these observations observation 1 observation 2 observation 3 so the agent needs to learn two different things first of all it needs to learn given an observation what state am I probably in right this is the first thing it needs to learn and then the second thing it needs to learn is given this state and this action what's the next state that I will go to right and this is a and this of course these things down here they're not observed so these things down here you can only do in distribution so I'm going to represent this with a P here you can only kind of do this in distribution and the way they handle it is they always maintain the expected value of these things and that's they do this in this wake-sleep algorithm all right so this is Miri recording this part because I have done a terrible job at the first time so we want to understand this wake-sleep algorithm to compute the things that we don't know let me draw this actually again so the way this algorithm does it is actually pretty cool has two phases a sleep phase and a wake face and it alternates between the two constantly it's kind of like expectation maximization ultimately what you want to learn are two different sets of parameters W and T now you whenever you learn T you use W the one that you've already learned and whenever you learn W you use the T that you've already learned so it's kind of a bootstrapping each other up the two functions you learn here are this F W and the T here so T is just a matrix and F of W is a function the function has weights the weights W so see in the sleep phase you update W and in the wake phase you update team now why is this called wake and sleep it's because in the wake phase you're actually so called awake and you use real observations so in the wake face and I find it easier to start actually at the wake phase in the wake phase you collect observations so you let your agent go around its environment and collect a bunch of survey shoes you don't know what the states are what you do is simply you collect these observations now it's not that important what the policy is here so you basically follow us on policy and you collect these observations right and then what you what you say is okay I have the function f of w and remember since we're in the wake phase we're learning T so we assume we already have the W in in essence in practice we start out with a random one and write and then kind of alternate between the two phases until both get really good but so we already have a W and we use it to update team how do we do this this is we need to understand what this function f of w does f of w takes this mu and the current observation and produces a new mu so what is a mu this this mu here this mu here as we saw above here the MU is the expectation over the features and in essence the MU is a guess the MU is your best guess of what the features of the state are or in the in the discrete case you could also say a guess of what the state is right so you don't know the state right but you what you want to maintain is a distribution over state so you want to kind of maintain this distribution but you can't you know calculate you can't properly efficiently calculate with an entire distribution unless you assume it's some sort of Gaussian or so but what you can do is you can simply take its mean mu right and and that's your best guess for what the state is the state could be anywhere anywhere here right according to this distribution but you simply come up with mu which is your best guess so the funk the function f of w it takes in the best guess of where you were up until the last step and it also takes as an argument your current observation and it gives you the output of F is mu T right it's the best guess of where you are now yeah it's pretty so pretty straightforward if you think about it so for every for every observation you want to have kind of a guess of where your what your state is and that's mu right so what F does is it takes whatever observations you had these observations gave rise to a mu that guess where you are you take this mu and you take this observation and from that you derive the next guess of where you are you just say I guessed I was at in the kitchen before now I moved I observed that I move through some sort of door and there is some sort of table so given that I get I thought I was in the kitchen and that I observe this thing now I'm probably in the living room that's what f WS so you input the observations that you had and you input your current observation right to get the guess of where you're next and these are real observations right and then you simply update T what does T do T relates your current and your next guess and that's important we already said that F takes your your kind of last guess and gives you the next guess T does kind of the same thing but he does it without having without relying on an additional observation T simply says well if I am here all right if my guess is that I am in the kitchen then what's the probability that in the next step I'll be in the living room without observing any thing right it's t is simply relating states to each other or relating guesses of states to each other right it's simply it's simply saying well under the current policy that I am what is the kind of distribution of going from one room to the next room all right so you learn in the wake face you learn the T the T simply represents how you move from state to state so it's exactly basically this function here except that it's not from state to state but it relates your guess about your guess your mu of the state 1 to the MU of the state 2 right and then in the sleep phase so in the sleep phase you you now assume that you have a good estimate of how the states relates to each other and what you can then do is you can actually sample trajectories and this is why it's called sleeping it's kind of like dreaming so given that you have a Model T of how states transition to each other or your your guesses about States or precisely you can now sample state trajectories so you can dream up how you would move in an environment right and the assumption here is that you know the process that that if you have a state that gives you an observation for example in their experiments is always the state it is X Y coordinates and that's corrupted by Gaussian noise there is also ways to learn this transition is what's called the this is what's called the observation process right but you assume you know it so you can sample trajectories of states and corresponding observations all right now this is not the real world but this is using this T down here you kind of know how or you kind of have them some sort of model can learn a model of how you move about the world so you sample these trajectories and from these trajectories you can now learn the F of W function so you see since you know what the state is right you can compute these features exactly and then you can learn this F of W function that gives you that takes in the guess of the last state and the current observation and gives you the next the guess of the next state and that you can then use temporal difference learning this is always here also with the T here we have temporal difference kind of a temp temporal difference learning to learn the parameters W so it's very kind of convoluted but ultimately it's a simple process in the wake phase you go into the world and actually collect real observations right and you have a method of deriving from these observations deriving the guesses about the states and so what you can do is you can learn a transition between the states right if you have a good guess of what the state or given each observation you can learn how to transition from one state to the next state except you don't do it in actual States you do it in guesses about states then once you have a model of how you move from one state to the next state you can go and dream up such state trajectories right you can dream state trajectories and therefore also you can dream how you would observe them and even that you can learn than a better function that relates your guess your guess about a state given the observation to the actual features of the state since for this particular thing you know what the state is so this this is this two-step process notice the cool thing we've never actually had to learn this mu explicitly we never had to learn how to go from observations to your guesses about states because we can compute this recursively right so you simply start out with mu 0 which is a guess about the initial state and then you go to mu 1 and mu 2 and you never actually have to learn that function all right all right so that's how they that's how they kind of learn these these success representations and the experiments of this are fairly cool here is another diagram of how that looks like you have a state this gives you an observation and from that you derive a guess of what this state is right so you can now look at what the agent learned the agent actually learns dynamics of this room it means if you're here you probably go somewhere right there is no clear direction but if you're close to the wall you're next states are probably going to be inwards of this wall right and yeah I've already shown you this picture so they have a last cool experiment here where what they do is they specify a reward and the reward is down here and from each state you want to know which way do I have to go to right get the reward now if they give the agent the value of the latent state and the latent state here or just your X Y coordinates if they give this to the agent and they let it run they let it learn the structure of the world it will correctly conclude aha look these are the kind of here as high-value states lower lower lower lower lower value states right up until over here are the most low value states because you travel the longest to go to the reward if you just give it the observation the noisy observation it will actually assign high value to states here because of course it it doesn't infer the latent state it simply takes observation as face valises well I was here and I reached here pretty quickly so it must be a good state but in fact it wasn't here it was here and the added noise would just corrupt the observation so you see it learns kind of a wrong model of the world whereas if you use this DDC you see sorry about that if you use this DDC you see you're much closer to the true state of the world like to the to the one on the left here where you so on the left here you actually kind of cheat right you give it the actual state but on here you give it the observation but tell it it's actually a noisy observation and you use what this paper proposes and again it will learn to assign a low value to these states because it needs to go all the way around even though it has seen supposedly seen the agent go from here to here directly but it it kind of understands that it's just a noisy observation alright so this was this from this paper it's a very very cool approach I think to reinforcement learning and there's some more experiments where you can see that this BTC actually helps and I'm excited about successive representations and how to incorporate them in reinforcement learning because it seems a perfect kind of middle ground between model-based and model free RL alright with that thanks for listening and bye bye