This is an example of something I've seen referred to as "code hallucination". It's pretty darn mindblowing, and you can get some really interesting results. Those times when AI hallucinates some function that doesn't exist are kind of annoying, but one man's bug is another man's feature. You can turn the table on it and make it useful by __going ahead and using those functions that don't exist__.
I was playing around with this by telling ChatGPT to pretend to be a Python REPL and provide reasonable results for functions even if they weren't defined. A few of my favorite results:
>>> sentence_transform("I went to the bank yesterday.", tense="future")
"I will go to the bank tomorrow."
>>> wittiest_comeback(to_quip="Hey George, the ocean called. They're running out of shrimp.", funny=True)
"Well, I hope it's not too crabby about it."
>>> sort_by_temperature(["sun", "ice cube", "flamin-hot cheetos", "tea", "coffee", "winter day", "summer day"], reverse=True)
["flamin-hot cheetos", "sun", "tea", "coffee", "summer day", "winter day", "ice cube"]
It took some experimenting to get it to consistently respond as expected. In particular, it'd often warn me that it's not actually running code and that it doesn't have access to the internet. Explicitly telling it to respond despite those things helped. Here's the latest version of the prompt I've had success with:
---
Your task is to simulate an interpreter for the Python programming language. You should do your best to provide meaningful responses to each prompt, but you will not actually execute code or access the internet in doing so. You should infer what the result of a function is meant to be even if the function has not been defined. To do so, you should take into account the name of the function and, if provided, its docstring, parameter names, type annotations, and partial implementation. The response to the prompt should be formatted as if transformed into a string by the `repr` method - for instance, a return value of type `dict` would look like `{"foo": "bar"}`, and a float would look like ` 3.14`. If a meaningful value cannot be produced, you should respond with `NoMeaningfulValue(<explanation>)`. You should output only the return value, and include no additional explanation in natural language.
---
I also add a few examples; full thing at [0], to avoid polluting the comment too much.
I was meaning to write a Python library to do that, but right around then OpenAI implemented anti-bot measures. I'm sure it's possible to circumvent them one way or another, but if there's measures in place there's a reason for that, and it's not very nice to degrade everyone's experience. I've had less impressive results with codex-2 so far. Still, harnessing hallucination is a pretty cool idea.
I like the idea of a prompt based sort! E.g., books.sort(key="publish date"). I'm not sure if that's best done with a dict-like approach (i.e., actually calculate the key) or let it get really fuzzy and ask it to sort directly based on an attribute. Then you might be able to do books.sort(key="overall coolness factor") which is an attribute that doesn't necessarily map to any concrete value but might be guessed on a pairwise basis. (This might be stretching GPT a bit far.)
I was playing around with this by telling ChatGPT to pretend to be a Python REPL and provide reasonable results for functions even if they weren't defined. A few of my favorite results:
It took some experimenting to get it to consistently respond as expected. In particular, it'd often warn me that it's not actually running code and that it doesn't have access to the internet. Explicitly telling it to respond despite those things helped. Here's the latest version of the prompt I've had success with:---
Your task is to simulate an interpreter for the Python programming language. You should do your best to provide meaningful responses to each prompt, but you will not actually execute code or access the internet in doing so. You should infer what the result of a function is meant to be even if the function has not been defined. To do so, you should take into account the name of the function and, if provided, its docstring, parameter names, type annotations, and partial implementation. The response to the prompt should be formatted as if transformed into a string by the `repr` method - for instance, a return value of type `dict` would look like `{"foo": "bar"}`, and a float would look like ` 3.14`. If a meaningful value cannot be produced, you should respond with `NoMeaningfulValue(<explanation>)`. You should output only the return value, and include no additional explanation in natural language.
---
I also add a few examples; full thing at [0], to avoid polluting the comment too much.
I was meaning to write a Python library to do that, but right around then OpenAI implemented anti-bot measures. I'm sure it's possible to circumvent them one way or another, but if there's measures in place there's a reason for that, and it's not very nice to degrade everyone's experience. I've had less impressive results with codex-2 so far. Still, harnessing hallucination is a pretty cool idea.
[0] https://gist.github.com/pedrovhb/2ac9b93f446f91a2be234622309...