Eliezer Yudkowsky - Harry Potter and the Methods of Rationality

He couldn't just let it go like that! Couldn't just forget having accidentally created a doomed consciousness that only wanted to die -

"You are perfectly capable of 'just letting it go', as you put it. Regardless of your verbal deliberations on morality, your nonverbal emotional core sees no dead body and no blood; as far as it is concerned, I am just a talking hat. And even though you tried to suppress the thought, your internal monitoring is perfectly aware that you didn't mean to do it, are spectacularly unlikely to ever do it again, and that the only real point of trying to stage a guilt fit is to cancel out your sense of transgression with a display of remorse. Can you just promise to keep this a secret and let us get on with it?"

In a moment of horrified empathy, Harry realised that this sense of total inner disarray must be what other people felt like when talking to him.

"Probably. Your oath of silence, please."

No promises. I certainly don't want this to happen again, but if I see some way to make sure that no future child ever does this by accident -

"That will suffice, I suppose. I can see that your intention is honest. Now, to get on with the Sorting -"

Wait! What about all my other questions?

"I am the Sorting Hat. I Sort children. That is all I do."

So his own goals weren't part of the Harry-instance of the Sorting Hat, then... it was borrowing his intelligence, and obviously his technical vocabulary, but it was still imbued with only its own strange goals... like negotiating with an alien or an Artificial Intelligence...

"Don't bother. You have nothing to threaten me with and nothing to offer me."

For a brief flash of a second, Harry thought -

The Hat's response was amused. "I know you won't follow through on a threat to expose my nature, condemning this event to eternal repetition. It goes against the moral part of you too strongly, whatever the short-term needs of the part of you that wants to win the argument. I see all your thoughts as they form, do you truly think you can bluff me?"

Though he tried to suppress it, Harry wondered why the Hat didn't just go ahead then and stick him in Ravenclaw -

"Indeed, if it were truly that open-and-shut, I would have called it out already. But in actuality there is a great deal we need to discuss... oh, no. Please don't. For the love of Merlin, must you pull this sort of thing on everyone and everything that you meet up to and including items of clothing -"

Defeating the Dark Lord is neither selfish nor short-term. All the parts of my mind are in accord on this: If you don't answer my questions, I'll refuse to talk to you, and you won't be able to do a good and proper Sorting.

"I ought to put you in Slytherin for that!"

But that is equally an empty threat. You cannot fulfill your own fundamental values by Sorting me falsely. So let us trade fulfillments of our utility functions.

"You sly little bastard," said the Hat, in what Harry recognized as almost exactly the same tone of grudging respect he would use in the same situation. " Fine, let's get this over with as quickly as possible. But first I want your unconditional promise never to discuss with anyone else the possibility of this sort of blackmail, I am NOT doing this every time."

Done, Harry thought. I promise.

"And don't meet anyone's eyes while you're thinking about this later. Some wizards can read your thoughts if you do. Anyway, I have no idea whether or not you've been Obliviated. I'm looking at your thoughts as they form, not reading out your whole memory and analyzing it for inconsistencies in a fraction of second. I'm a hat, not a god. And I cannot and will not tell you about my conversation with the one who became the Dark Lord. I can only know, while speaking to you, a statistical summary of what I remember, a weighted average; I cannot reveal to you the inner secrets of any other child, just as I will never reveal yours. For the same reason, I can't speculate on how you got the Dark Lord's brother wand, since I cannot specifically know about the Dark Lord or any similarities between you. I can tell you that there is definitely nothing like a ghost - mind, intelligence, memory, personality, or feelings - in your scar. Otherwise it would be participating in this conversation, being under my brim. And as to the way you get angry sometimes... that was part of what I wanted to talk to you about, Sorting-wise."

Harry took a moment to absorb all this negative information. Was the Hat being honest, or just trying to present the shortest possible convincing answer -

" We both know that you have no way of checking my honesty and that you're not actually going to refuse to be Sorted based on the reply I did give you, so stop your pointless fretting and move on."

Stupid unfair asymmetric telepathy, it wasn't even letting Harry finish thinking his own -

"When I spoke of your anger, you remembered how Professor McGonagall told you that she sometimes saw something inside you that didn't seem to come from a loving family. You thought of how Hermione, after you returned from helping Neville, told you that you had seemed 'scary'."

Harry gave a mental nod. To himself, he seemed pretty normal - just responding to the situations in which he found himself, that was all. But Professor McGonagall seemed to think that there was more to it than that. And when he thought about it, even he had to admit that...

"That you don't like yourself when you're angry. That it is like wielding a sword whose hilt is sharp enough to draw blood from your hand, or looking at the world through a monocle of ice that freezes your eye even as it sharpens your vision."

Yeah. I guess I have noticed. So what's up with that?

"I cannot comprehend this matter for you, when you do not understand it yourself. But I do know this: If you go to Ravenclaw or Slytherin, it will strengthen your coldness. If you go to Hufflepuff or Gryffindor, it will strengthen your warmth. THAT is something I care about a great deal, and it was what I wanted to talk to you about this whole time!"

The words dropped into Harry's thought processes with a shock that stopped him in his tracks. That made it sound like the obvious response was that he shouldn't go to Ravenclaw. But he belonged in Ravenclaw! Anyone could see that! He had to go to Ravenclaw!

"No, you don't," the Hat said patiently, as if it could remember a statistical summary of this part of the conversation having happened a great many previous times.

Hermione's in Ravenclaw!

Again the sense of patience. "You can meet her after lessons and work with her then."

But my plans -

"So replan! Don't let your life be steered by your reluctance to do a little extra thinking. You know that."

Where would I go, if not Ravenclaw?

"Ahem. 'Clever kids in Ravenclaw, evil kids in Slytherin, wannabe heroes in Gryffindor, and everyone who does the actual work in Hufflepuff.' This indicates a certain amount of respect. You are well aware that Conscientiousness is just about as important as raw intelligence in determining life outcomes, you think you will be extremely loyal to your friends if you ever have some, you are not frightened by the expectation that your chosen scientific problems may take decades to solve -"

I'm lazy! I hate work! Hate hard work in all its forms! Clever shortcuts, that's all I'm about!

"And you would find loyalty and friendship in Hufflepuff, a camaraderie that you have never had before. You would find that you could rely on others, and that would heal something inside you that is broken."