How do I know you are conscious? I will beat you non-stop. You will make some noises. How do I know you are not simulating the response of a conscious being?
This isn't an "allegory of the cave" problem. You obviously know that you are conscious, and you can empathize with me as your fellow human. Therefore, we are both quite conscious.
You do not identify with AI. ELIZA isn't conscious, even though it could write text as well as a human. LLMs are not conscious, because they do not even remotely resemble any brain, conscious or otherwise, discovered by man. Any person familiar with the transformer architecture will affirm this to you.
I don't know octopuses are conscious. I don't identify and cannot even begin to identify with an octopus. Animals being considered conscious is also not always culturally true in recent European history. I know this only because textbooks taught me this. I cannot be thus also certain about ai/llm's without serious research into this being done.
Serious research has been done. We know what an LLM brain looks like. It's nothing like any octopus, fly, cat or crab brain. You can go download a fly connectome, feed it to your favorite LLM, and do a side-by-side comparison with your LLM weights of choice. It takes 5 minutes of curious exploration to recognize that LLMs aren't conscious.
If you want to hedge your bets on consciousness, go ahead. But I could give you a list of things that are widely considered conscious, and you'll note that none of them are transformer models.
Of course LLMs are not exact matches of biological brains. There is no reason consciousness specifically has to arise strictly out of the same substrate. My example with octopus was as a rather structurally different being that also happens to be conscious. My trouble is again, that we have made this mistake in the past, several times, so yes, I will hedge my bets on something that talks, thinks and behaves so much like us. Its a good term you used, that's exactly how I think of it, I don't know if they are but I stay on the cautious side.