- Compositionality = parts + syntax
- First part: 2024 paper
- System: RNN
- Inputs:
- Sensory signals,
- Text input passed through LLM
- Outputs:
- Motor outputs: fixation, decision
- Training:
- Tell it what to do with text in various ways.
- Finding:
- Could zero-shot new tasks specified through text.
- Explanation:
- RNN learns a compositional representation
- Question: Is that an inductive bias of the RNN?
- Observation: Dictionary learning/sparse coding.
- Question: Is that induced by the text?
- Finding: Could train the network using a binary vector indicating the task. But networks couldn’t generalise.
- Why not?
- Finding: Could train the network using a binary vector indicating the task. But networks couldn’t generalise.
- Modules are gated by the text.
- A related experiment with 3 binary variables
- Forcing system to search on the 3D hypercube of possibilities dramatically improved learning time.
- Question: How would the system learn to do this?
- Forcing system to search on the 3D hypercube of possibilities dramatically improved learning time.
- RNN learns a compositional representation
- Second part: Paper with Tiago
- System: mice performing escape behaviour
- Inputs:
- arena with wall in the middle,
- shelter point,
- Occasional loud noise
- Outputs: Mouse trajectories
- Model: Motor primitives
- Figure out which primitives to combine to build the trajectory.
- Questions:
- Similarity to dictionary learning
Chat afterwards:
- PPCs:
- To read: Deneve, Latham, Pouget (2001):
- How to map PPC reps to population means.
- Alex: “Peter derived the equation, the most beautiful to come out of our work.”
- vs. Sampling:
- Limited evidence of sampling beyond V1
- Sampling: more samples will give more information about distribution
- So distribution will be come better defined
- But will e.g. remain bimodal, if so.
- But in experiments: given more observations, people’s decisions become more certain.
- Suggests: People summarising distributions with means, etc.
- So distribution will be come better defined
- To read: Deneve, Latham, Pouget (2001):
- Olfaction:
- Reminder of the Mathis paper: is it inference that all, what’s the machinery for, if a glomerular readout suffices?
- Why does taste go through the thalamus, but smell does not?
- Is the granule cell system/solution just an older, evolutionary approach, before thalamus was available?
- Dimensionality of olfactory vs. visual space
- What’s the equivalent of activating single cones in the visual system?
- Learning by inference
- Like dictionary learning?
- Learn primitives to reconstruct environment
- Sequence the primitives for a specific task.
- Nested EM?
- Inner loop learns primitives
- Outer loop sequences primitives
- Like dictionary learning?
To read
- Zero-shot learning in RNNs trained through natural language
- Stepwise learning in mice doing the IBL task: Infinite hidden Markov models can dissect the complexities of learning
- The benefits of world and inverse models: Rumelhart and Jordan 1992
- Reading out PPCs: Deneve, Latham and Pouget
- No need for inference in the olfactory bulb.