Knowing (or at least suspecting) what you know and what you don't, in humans, is a property referred to as "meta-memory." That is, memory about your memories. No one has the faintest idea how it works. It's not at all surprising that a machine that simply builds statistically probable text has no such capacity. It has no use for it, since it doesn't look up facts, just likely strings of words.
@Ivy_League_Red: which episode of Gilligan's Island features mindreading?
@grok: The episode of "Gilligan's Island" that features mindreading is titled "Seer Gilligan" and it first aired on January 27, 1966. In this episode, Gilligan discovers an island bush with seeds that grant the ability to read minds when consumed. The situation leads to a series of humorous events as the castaways learn about each other's thoughts, causing some unexpected revelations and misunderstandings.
Interesting, Grok seems to use web search. The evidence is that if you search for recent TV shows, that likely were not in the training, it also finds them.
As a test, I did "Which episode of Murdoch Mysteries is about a drunken brawl at the Starbright Lounge". It correctly found S17.E23 which only aired just 3 weeks ago.
2. Q: Which episode of Gilligan’s Island was about mind reading?
3. A: In the Gilligan's Island episode "Seer Gilligan" (season 3, episode 15), Gilligan gains the ability to read minds after being hit on the head with a coconut. He starts hearing the private thoughts of the other castaways, which leads to various humorous situations and misunderstandings. However, his newfound ability also causes problems as the others become annoyed that he is invading their privacy. In the end, Gilligan gets hit on the head again and loses his mind-reading powers, much to everyone's relief.
It gets most other details wrong, but nailed the episode name.
Interesting, I get different results from lmsys than elsewhere. GP4 on lmsys gave me same answer you posted here from claude and Llama 3 on lmsys made up an episode instead of telling me there is no episode when I executed the same on Llama 3 hosted on grok.com
Thank you for the invitation. My writing here at Mind Prison is about all I can manage presently. My focus is primarily one of "informed consent of the future". Hoping to bring topics and perspectives to think about for greater awareness for where our decisions may lead.
You are welcome to share them anywhere you might find helpful.
I am pleased to share your work with them, but I hope that you might take the moment to join our discord and see if there is anything that we can coordinate on.
That was not the implication or relevance of the phenomenon.
Note, it is a current topic of research around the inability for LLMs to understand what they do not know. It is an acknowledged problem in the space - https://aclanthology.org/2023.findings-acl.551.pdf
Knowing (or at least suspecting) what you know and what you don't, in humans, is a property referred to as "meta-memory." That is, memory about your memories. No one has the faintest idea how it works. It's not at all surprising that a machine that simply builds statistically probable text has no such capacity. It has no use for it, since it doesn't look up facts, just likely strings of words.
Grok (on X.com) got it correctly the first time using the same prompt.
https://twitter.com/i/grok
@Ivy_League_Red: which episode of Gilligan's Island features mindreading?
@grok: The episode of "Gilligan's Island" that features mindreading is titled "Seer Gilligan" and it first aired on January 27, 1966. In this episode, Gilligan discovers an island bush with seeds that grant the ability to read minds when consumed. The situation leads to a series of humorous events as the castaways learn about each other's thoughts, causing some unexpected revelations and misunderstandings.
Interesting, Grok seems to use web search. The evidence is that if you search for recent TV shows, that likely were not in the training, it also finds them.
As a test, I did "Which episode of Murdoch Mysteries is about a drunken brawl at the Starbright Lounge". It correctly found S17.E23 which only aired just 3 weeks ago.
Pwned him there
1. https://chat.lmsys.org/
2. Q: Which episode of Gilligan’s Island was about mind reading?
3. A: In the Gilligan's Island episode "Seer Gilligan" (season 3, episode 15), Gilligan gains the ability to read minds after being hit on the head with a coconut. He starts hearing the private thoughts of the other castaways, which leads to various humorous situations and misunderstandings. However, his newfound ability also causes problems as the others become annoyed that he is invading their privacy. In the end, Gilligan gets hit on the head again and loses his mind-reading powers, much to everyone's relief.
It gets most other details wrong, but nailed the episode name.
Model A: claude-3-opus-20240229
¯\_(ツ)_/¯
Interesting, I get different results from lmsys than elsewhere. GP4 on lmsys gave me same answer you posted here from claude and Llama 3 on lmsys made up an episode instead of telling me there is no episode when I executed the same on Llama 3 hosted on grok.com
Of course, results are always a bit random.
Thank you for being for humans. Have you thought of joining PauseAI discord, so we can better organize for resistance?
Thank you for the invitation. My writing here at Mind Prison is about all I can manage presently. My focus is primarily one of "informed consent of the future". Hoping to bring topics and perspectives to think about for greater awareness for where our decisions may lead.
You are welcome to share them anywhere you might find helpful.
I am pleased to share your work with them, but I hope that you might take the moment to join our discord and see if there is anything that we can coordinate on.
Resistance is futile :D
Thinking you know what a black box system 'can never' do is peak hubris and it should be obvious to all thinking people that it's illogical.
That was not the implication or relevance of the phenomenon.
Note, it is a current topic of research around the inability for LLMs to understand what they do not know. It is an acknowledged problem in the space - https://aclanthology.org/2023.findings-acl.551.pdf
Yet this is how you understand them:
https://www.strangeloopcanon.com/p/what-can-llms-never-do