I was watching this video and I don’t agree that the compound game, as described by Yudkowsky, a coin flip followed by 100%:$1M vs. 95%:$5M, is equivalent to the 50%:$1M vs 45%:$5M game. The way Yudkowsky describes it, you have a further choice after the coin flip to change your preference, but a strictly equivalent game would not give the choice after the coin flip. The player can rationally change their preference from B to A after the coin flip because they can factor in their expectation of how they will feel for having lost an opportunity for guaranteed money after the coin flip. If a player can feel regret, for instance, their utility function may greatly magnify the expected feelings of regret compared to the expected feelings of happiness/security/whatever at having more money.
Moreover, the setup of this game (and similar “paradoxical” games), doesn’t account for the player’s trust of the game-maker. The player’s prior beliefs about how trustworthy the game maker is in his statement of the probabilities and in the construction of the game’s implements (coins, die, etc.) would also factor in to any games like this. For instance, the game maker may enjoy creating so-called “Newcomb-like problems” for rationalists to deal with, so he changes the game based on which preference you express for the pre-coin-flip game: if you prefer A, then he uses a trick coin that always ends the game, but if you prefer B, then he’ll use a trick coin that always continues the game.
So, I don’t disagree that a rational agent who is strictly about the money, always prefers more money, and who trusts the game maker should have the same preference between the compound game and the simple game, but I don’t think humans are inconsistent for having distinct preferences between them.