HDInsight: - CoC (Calculation of Cores)

  • Article

If you ever notice “99% of XX HDINSIGHT CORES” in your HDInsight dash board on wonder what it means? Or during cluster creation if you get error/warning “The cluster can have at most 0 data nodes” and wonder what
it means?

In this short article I’ll just explain what above means and show core calculation.

So let’s take an example if you have 740 HDInsight Core on your subscription and 740 HDInsight Core being used by 8 clusters (167 nodes). Each worker node consumes 4 cores and each head node consumes 8 cores.

Here is the calculation:-

Clusters

Data Nodes x Cores

Head Nodes x Cores

Total Cores used

First cluster

16 x 4 = 64

1 x 8 = 8

72

Second cluster

16 x 4 = 64

1 x 8 = 8

72

Third cluster

16 x 4 = 64

1 x 8 = 8

72

Forth cluster

16 x 4 =64

1 x 8 = 8

72

Fifth cluster

32 x 4= 128

1 x 8 = 8

136

Six cluster

32 x 4= 128

1 x 8 = 8

136

Seven cluster

32 x 4= 128

1 x 8 = 8

136

Eight cluster

7 x 4 =28

1 x 8 = 8

36

TOTAL

668 cores

64 cores

732

 

So if you try to create cluster at least with 1 node it requires 12 cores which is not available to subscription. That’s why you get error “The cluster can have at most 0 data nodes” and message “99% of 740 HDInsight cores” in the dashboard.

In case it’s require to increase the core, one of the options is to raise ticket from the portal.

 

The other option is reuse existing cluster in case you are not using it.

 

HTH.