Sunday, May 14, 2017

GANs, arms races, and self-consciousness

in evolution, there are always arms races between competing organisms. each organism has to form a model of the other one, in order to somehow outsmart it. that forces you to keep coming up with novel, creative solutions, because you have to do something that the other guy currently doesn't have a model of. this is similar to the idea of generative adversarial networks.

i wonder if what happened that made humans special, is that that process got condensed to happen inside a single organism. at some point, like ~70,000 years ago maybe, social interactions became the dominant factor in our fitness. so we developed the ability to model other people's minds incredibly well (people have argued this is why we have big brains).

but the unforeseen consequence was that you effectively have multiple minds running inside a single brain, and they're evaluating each other. maybe this sometimes feels like self-consciousness, although i think the feeling we usually call "self-consciousness" is just a subset of this.

that's like a high-bandwidth (since it's inside a single brain) version of the GANs or arms-race phenomenon that's happened throughout evolution. these models of other people's minds are constantly providing error signals (probably all the time, including during sleep) that are updating your "own" mind. that could be what drives us to have such rich minds.

Thursday, February 02, 2017

even if i concentrate on trying to be aware of my own bias, it's still hard to avoid a gut reaction of disagreeing with the conservative posts.

what if we think of this as a psychology experiment that reveals our own bias to us?

for example, no matter how many times you see your own (retinal) blind spot, you don't start to think that there's actually a hole in the world; you accept that it's just an artifact of your perception.

likewise, imagine if you're presented with conservative news stories, and you feel yourself disagreeing with them, but you simultaneously recognize that that disagreement is just a quirky psychological artifact... (or vice versa if you're a conservative)

Sunday, November 06, 2016

AI and media

so we have this thing now where liberals can choose to only read liberal media, and conservatives select conservative media. steve was pointing out that as more content is created by AI, it won't just be liberal vs conservative, but your own micro-customized perspective fed back in your face.

i guess it seems almost inevitable, if the internet is designed to get you "the information you want", then that's whatever matches your views. you could change the objective to giving "the information you *should* have", but then somebody else has to decide what that is. maybe this is the trap all intelligent life falls into, and why we don't see them everywhere in the galaxy?

one glimmer of hope would be if we can build AIs that don't just maximize a value function, but are actually alive in a deeper sense, that they're on the edge of life and death, with a feeling of mystery and wonder.

Thursday, October 06, 2016

not thought through email 2

you know how after you learn an association, if you present one of the two items alone, a representation of the other one appears after a short delay? e.g.,

i was thinking about this in the context of the remerge model:

it could be that the dynamics of hippocampus are particularly good for stitching together the topology of space using little chunks of sequence. and more recently in evolution this is also used for stitching together arbitrary concepts.

when two or more concepts are linked together for the first time, this is initially stored in recurrent MTL dynamics, such that presenting either item alone triggers a fast sequence including the other item. (dharsh's idea, if i understand right.)

but maybe, after more learning (or sleep/replay), some PFC neurons, which are repeatedly being exposed to rapid sequential activation of one item after the other, become specifically sensitive to the new compound. as in tea-jelly.

now the tea-jelly cells in PFC can become building blocks for new sequences in MTL. maybe one reason the PFC is huge in humans is that we have to know about crazy amounts of stuff, like global warming and tensorflow. these are things you can't get from a stacked convnet on the visual stream; only from binding multiple things?

this could also fit with the idea that episodic learning and statistical learning are part of the same mechanism. when you learn a new concept, it's originally from a single or small number of examples. it would be interesting if the final geometry of the abstract concept, even after it's been well consolidated, still somewhat depends on what first examples you learned it from.

not thought through email 1


would some of the trajectories have to be very long? like they give the example of college acceptance's value being conditional on whether you graduate...

how do you decide what to store in a trace? would a trajectory hop from homework to college application to graduation?

episodic trajectories might be one way of looking at "conflicting beliefs". you could easily have two episodes, which start from different starting points, and constitute logically incompatible beliefs. like, "everyone should earn a fair wage, therefore we should set a high minimum wage". and, "higher minimum age forces employers to reduce full-time employees".

how separable is the state generalization problem from the episodic idea? (although there's the good point about advantages of calculating similarity at decision time.)

they don't talk much (or maybe i missed it) about content-based lookup for starting trajectories. is it basically "for free" that you start trajectories near where you are now (or near where you're simulating yourself to be)? is it like rats where the trajectories are constantly going forward from your current position, maybe in theta rhythm?

i was thinking about a smooth continuum between "episodic" and "statistical/model-based". what if we picture the episodic trajectories as being stored in a RNN. when you experience a new trajectory, you could update the weights of the RNN such that whenever the RNN comes near the starting state of that trajectory, it will probably play out the exact sequence. but you could also update the weights in a different way (roughly: with a smaller update) such that the network is influenced by the statistics of the sequence, but doesn't deterministically play out that particular sequence.

this view is also kind of nice because episodes aren't separate from your statistical learning. some episodes from early in life might even form a big part of your statistical beliefs. especially as the episodes are replayed more and more, over time, from hippocampus out into cortex.

is the sampling of trajectories influenced by current goals?
like, the current goal representations in PFC could be continually exerting pressure on the sampling dynamics to produce outcomes that look like the goal representations?

Tuesday, September 27, 2016

for the previous post, one issue is how the sensory system can maintain both the current inputs and the forward modeled consequences. maybe one mechanism is in different phases of oscillations.

Monday, September 26, 2016

mixed selectivity, brain-wide decisions, planning and imagination

laurence hunt has been pointing out how decisions aren't computed in the brain: by one region or circuit doing a particular computation and passing the finished result on to another region or circuit, that then does a different computation.

instead, millisecond by millisecond, decision-related signals appear almost simultaneously all over cortex:

another example is how downstream circuits can be biased by an upstream calculation before that calculation is finished:

these things make sense intuitively because neurons have a lot of connections to each other. laurence also points outs that one of the canonical computations in the brain is lateral inhibition. he outlines a big picture of parallel competition across the brain - with different areas emphasizing competition in different features spaces:

and this matches up with "mixed selectivity". for example, if you look at the cells that project to midbrain dopamine neurons, you might think these cells would contain cleanly separated representations of the different signals you need to calculate RPE, like reward, expectation etc. but instead, many of the neurons are a bit correlated with multiple of these signals:
it's a signal smoothie.

stefano fusi and others observe that keeping this high dimensionality is important in order to respond to task demands:
i like how this is analogous to life. keeping lots of potential around is probably what life is. i wonder how this connects to mate lengyl's idea that moment-to-moment variability in neural signals encodes uncertainty in the beliefs they represent.

i also like this because it fits exactly with the idea of continuously metabolizing the world. compared to "decision making", i think this is a fundamentally different way of looking at what the brain is doing. for one thing, you generate rather than select actions. i suppose this is good in the real world where the action dimensionality is very high and candidate actions are rarely proposed to you. for another thing, it means that in a "stable state", you can keep emitting actions from your current abstract goal representation. maybe it's unfolded to different things depending on the current inputs and internal state...

but now we get to the really interesting part. let's start with laurence's model of lots of brain areas processing feature competitions in parallel, with simultaneous influences on one another. what i've been thinking is that this links together planning and imagination in a concrete way.

the key to this, in my current (probably infantile) thinking, is that PFC networks may be very good at holding on to their representations (both because individual neurons have longer time constants, and because of network-level properties). if you think of PFC as being the top of a sensory hierarchy, you could look at it as encoding a very abstract state of the world. or if you think of it as being the top of a motor hierarchy, you could view it as encoding a very abstract action plan. these two things are the same thing.

now, let's say that in the feature competitions, you include some bits of forward model -- i.e., how your potential actions will change the world -- and some modelling of the dynamics of the world itself (let's lump these both under the name "forward model" for now). as the overall state of the brain evolves forwards (under its lateral inhibition dynamics), your current abstract action plan will push the modelled future state of the world toward whatever the predicted consequences are. these predicted consequences are processed through the sensory/state/value hierarchy and compared against current goals. if they match current goals, fine, you continue unfolding your current action plan. if they don't match, then this is where the hyper-stability of PFC comes in. PFC, as the top of the sensory hierarchy, should be pushed toward believing the predicted consequences. but because it's pathologically stable, it doesn't get its own state pushed around. instead, it pushes on the action side of the hierarchy to make the abstract action representations a little bit more like something that produces consequences that match its beliefs.

importantly, when the current goals are not working (maybe signalled by something like low tonic dopamine), the PFC can let go of its hyper-stability and reorganize around new goals/abstract action plans. e.g.:
spiritually speaking, this can feel difficult. the more of the long-term goal-patterns are released, the more it's letting go of the self, which is what we're afraid of.

(here's a very ignoreable side note. it wouldn't have to be PFC alone that "insists" on goals. this insistence could be distributed over the whole brain too. the PFC idea is just a clean way to visualize the story.)

this process is an energy minimization in activation space (to satisfy as many of the constraints imposed by weights as possible). but it's nice that you never have to compute the energy or its gradient. i have no idea how learning would work in this kind of system but i'm sure people must have worked on it. something hebbian and simple?

so what i've just described is a way of doing planning, right? but you never need any part of the brain doing something like tree search over the whole abstract state space. instead, little bits of forward model nested in various areas can unfold in different kinds of feature spaces, to different lengths of time, but these computations are being influenced by constraints from ongoing computations in other areas.

this could explain pavlovian pruning:
because if you start simulating bad consequences in one part of the feature space, this suppresses the search. although maybe this kind of explanation is overkill.

i suppose this fits with replay because the dynamics of the brain have learned the statistics of the dynamics of the real world. it makes sense that little bits of the brain are perpetually playing out the little snippets of dynamics that they know about. obviously it's much more complicated with learning and stuff. this is just some stray thoughts.

where does dopamine come in? some more speculative thoughts. if you get a reward prediction error, this means the overall action plan/expectations didn't account for everything, so maybe you need to update your action plan. (or, i could see this going the other way, that a positive RPE means you should stabilize your abstract action plan, and engage more action along it.) this loosely matches with seamans/yang or cohen/braver/brown ideas that dopamine modulates PFC representational stability. rui costa's work seems consistent with this being at some level more abstract than just simple actions, e.g.:

what about "episodic RL"?
sam and nathaniel's model remembers discrete sequences, but maybe the whole system is on a continuum from non-parametric to parametric. when you experience a sequence/trajectory, this gets written into the weights of your RNNs. if it's written in really strongly, it becomes a path that future activity can almost deterministically follow (if it comes near the starting state). if it's written in weakly, it just influences the future dynamics a bit.

what about striatum? forward models ("what will happen if i do this?") and dynamics models are probably relatively easy to learn: you just match what you observe. i suppose inverse models are harder to learn ("what action should i do?"). but the striatum and habits maybe cache little pieces of inverse model. (i haven't thought through how this fully fits in yet.) it is a nice idea that the continuum between "model-based" and "model-free" has to do with which parts of forward-model-energy-minimization you replace with bits of inverse model.

there are some ML systems that need to be fed a "goal state", like this:
these kind of systems should mesh well with the metaphors i've been thinking about here.

this is getting so schizophrenic, let me close by reflecting about how the building blocks of nervous systems are things like central pattern generators. it's oscillations modulating oscillations.

Friday, September 16, 2016


once again i'm pretty sure i'm not thinking anything original here, it seems to be a mashup of other people's ideas. but it's me starting to internalize it maybe.

so, let's say in the brain you have an abstract representation of your action-plan, which predicts through forward models the consequences, and which also continuously contrasts these predicted consequences against goals. the system attempts to minimize the energy of its activations continuously, like a big multiple constraint satisfaction problem, with all comparisons/prediction errors happening in parallel.

how does exploration work in this system? as i understand karl's model, he has a term for epistemic value. or, in like dan russo's information directed sampling, you try to optimize your knowledge about which actions lead to reward.

how could i fit this into my intuitive framework?

generally, i think exploration is the drive to metabolize the outside world. why is this a property of living systems? because life is fundamentally dynamic. the energy of that metabolism is what keeps moving the system forward.

so what is exploration in my continuous action-generation system? why isn't it just making the actions that make predictions look most like goals? i guess one possible answer is simply that it's minimizing prediction errors everywhere. on the surface, it seems like this has a problem - that you should avoid exploring because it would generate surprise.

but i had an idea about this - maybe exploration is only relative to something that has *already* caused a prediction error. for example, let's say there's a door in your environment, which you've never seen. before you see it, you obviously can't choose to explore behind it. when you see it, *this* is what generates the prediction errors. especially because you know some stuff about doors: and your generative models fill in lots of uncertainty about what's behind the door. your exploration is then to minimize this uncertainty.

and finally, we avoid the darkroom problem because of the dopamine stuff, and hyper-stability in PFC etc.