What a Ranking Actually Encodes

To produce an ordered list, a system has to take many dimensions of performance and collapse them into one number. That requires deciding how much each dimension is worth.

Those weights are a judgement. They are usually not published, and they are where nearly all disagreement between ranking systems comes from — not from different data.

• Predictive or descriptive? Is it ranking who has been best so far, or who will be best next? These produce very different lists, and the label rarely says which.
• How much regression? A system that trusts small samples will have volatile rankings and dramatic risers.
• Peak or accumulation? Rewarding the best stretch versus rewarding total production changes the order completely.
• How is availability handled? A player who missed six games can be ranked on what he did or on what he would have done.

Ask those four and most rankings become legible.

When two good systems disagree about a player, that is worth more attention than when they agree. It usually means the player is unusual along a dimension the systems weight differently — high peak and low availability, or strong process with poor results.

Consensus is comfortable and mostly already priced into how everyone behaves. The disagreements are where the thinking is still available to be done.