Are There Biases In Top Prospects Lists?

Two of the most common criticisms of baseball top prospects lists are that they are biased both towards high draft picks and recent signings who haven't played much professionally. These complaints seem to apply to all of the major prospecting sites: BA, BP, John Sickels, etc. (And even our lists.) Let's see if the biases exist, and if so, why.

To address the first question I took BA's 2011 Top 10 Prospects for each team (30 x 10 = 300 prospects) and divided them up by where (draft round and international signs) and when (by signing year) each player was acquired. I used the BA rankings for two reasons, they were readily available and BA lists overlap with BP and others a great deal, over 80% most years.

Here are how the 300 players were acquired, broken down by draft round and international signings:

How Acquired               #         %
1st round                 75      25.0
1st - supplemental        33      11.0
2nd round                 35      11.7
3rd round                 23       7.7
4th round                 11       3.7
5th round                 10       3.3
6th-10th round            20       6.7
11th - 20th round         12       4.0
21st - 50th round         11       3.7
Non-drafted free agent     1      0.03
International sign        69        23
TOTALS                   300       100

Nearly half (47.7%) of the 300 top prospects were picked in the first two rounds of the draft, and nearly two-thirds (62.4%) in the first five rounds. If you remove international signings who aren't eligible for the draft, those percentages rise to 61.9% of top prospects coming from the first two rounds and 81.0% from the first five rounds. There is no set definition of 'high draft picks', but I think we can agree that players chosen in rounds one through five certainly qualify. When over 62% of all prospects and 81% of draft-eligible prospects are acquired via 'high draft picks', it seems fair to say there is a significant bias towards high draft picks in top prospects rankings.

Next up is 'recent signings', here are the 300 prospects by the year they were signed:

Year          #         %
2010         62      20.7
2009         69        23
2008         54        18
2007         51        17
2006         37      12.3
2005         15         5
2004         12         4
TOTALS      300       100

As with 'high draft picks', there is no set definition of 'recent signings'. Again, I think we can agree that players signed in the past two years should qualify, which includes 43.7% of the top prospects. If we extend 'recent signings' to the past three years it increases to 61.7%. While the percentages aren't as high as with 'high draft picks', there is certainly a tendency towards 'recent signings' when nearly two-thirds of the prospects were acquired in the past 3 years. One other thing to remember, this data includes not only drafted players but also international signs, many of whom were only 16 or 17 at the time and are still teenagers even two or three years later, or about the age they would be drafted out of high school in the U.S.

As an aside, since I have the data available, here are the international signings broken down by country of origin and year signed:

Country               #        %
Dominican Republic   37     53.6
Venezuela            19     27.5
Cuba                  3      4.3
Curacao               3      4.3
Mexico                3      4.3
Korea                 1      1.4
Columbia              1      1.4
Panama                1      1.4
Australia             1      1.4
TOTALS               69      100


Year          #         %
2010          4       5.8
2009          6       8.7
2008         10      14.5
2007         17      24.6
2006         17      24.6
2005          7      10.1
2004          8      11.6
TOTALS       69       100

One last note from the BA lists that applies to both 'high draft picks' and 'recent signings', 24 of the 300 top prospects, or 8%, had yet to play a single professional game. The ultimate in potential over performance in prospect ranking.

I originally asked: if there is a bias, why? The data does show biases towards high draft picks and recently acquired players, but much of that can be explained by real world considerations in putting together top prospects lists. Looking at high draft picks, it just makes sense that if a team drafts Player A in the first round and pays them a huge signing bonus, their scouts believe that they are a better prospect than Player B who they drafted in the 15th round and paid a small bonus. The folks making the prospect lists don't have nearly the information available about each player that the team does, so it makes sense to rely on the team's evaluation to some degree.

Turning to the rankings for recent picks who have played little or no pro ball, there are several reasons for this to occur. First, there is some overlap with the first bias, top picks, since top picks tend to sign late and not play much that season. Second, the nature of creating prospect lists is to identify players earlier than the competition. If for example BA waited until a player reached AA before listing them as a prospect, they would lose their readership to list makers who identify prospects years earlier. And third, players signed in 2004 or 2005  generally are no longer prospects. Either they have reached the majors or flamed out after five or six minor league seasons (high school picks who miss a year or two with injuries and young international signs are the exception). If we wait several seasons to see a consistent record of professional performance, top prospects lists would be very short.

Bottom line: The biases do exist, but the complaints about them aren't as valid when you take into consideration the real-world factors affecting top prospects rankings. Agree, disagree, anything I've overlooked?