[CI] End-of-year CI budget report: cost exceeded projection #5176
Description
@dmlc/xgboost-committer
In March 2019, AWS graciously granted us 12k USD worth of AWS credits to maintain our CI server (https://xgboost-ci.net). In addition, sponsors including NVIDIA are committed to monthly donation through Open Source Collective. Since then, we revamped our CI (#4234) and added test coverage for more platforms and targets (CUDA, multi-GPU, dask, Spark).
Now here is the end of year report. I had estimated the cloud cost to be 1000 USD/month, but thanks to active contributions over the year, the average cost has been greater, at 1600 USD/month. As a result, the 12k AWS credit ran out 3 months earlier than what I expected. See the following table:
| Month | Cost covered by AWS credit | Cost covered by donations | Total spend |
|---|---|---|---|
| Mar-19 | $ 709.58 | $ - | $ 709.58 |
| Apr-19 | $ 1,269.62 | $ - | $ 1,269.62 |
| May-19 | $ 1,985.30 | $ - | $ 1,985.30 |
| Jun-19 | $ 1,608.34 | $ - | $ 1,608.34 |
| Jul-19 | $ 1,522.61 | $ - | $ 1,522.61 |
| Aug-19 | $ 290.43 | $ 1,383.86 | $ 1,674.29 |
| Sep-19 | $ 1,038.17 | $ - | $ 1,038.17 |
| Oct-19 | $ 2,204.07 | $ - | $ 2,204.07 |
| Nov-19 | $ 1,304.33 | $ - | $ 1,304.33 |
| Dec-19 | $ 453.44 | $ 2,214.57 (*) | $ 2,668.01 |
| TOTAL | $ 12,385.89 | $ 3,598.43 | $15,984.32 |
| Today's balance (Dec 31) | |
|---|---|
| AWS Credit | $ - |
| Open Source Collective | $ 1,291.79 |
(*) There isn't actually enough balance in the donation account to cover this amount. For now, I will personally cover the difference.
I am reaching out to AWS for another round of donation. Absent additional funding, we will need to take drastic cost-saving measures. Let us find ways to keep the server running. I personally donated USD 3042 to keep the server going for another month, so that we can push out 1.0 release.
Cost by EC2 instance type:
| Month | P2 | G4 | C5 | Misc | Total cost ($) |
|---|---|---|---|---|---|
| Mar-19 | $651.80 | $0.00 | $2.71 | $55.07 | $709.58 |
| Apr-19 | $1,133.97 | $0.00 | $59.19 | $76.47 | $1,269.63 |
| May-19 | $1,159.73 | $0.00 | $745.00 | $80.57 | $1,985.30 |
| Jun-19 | $942.04 | $0.00 | $579.10 | $87.19 | $1,608.34 |
| Jul-19 | $863.41 | $0.00 | $545.44 | $113.76 | $1,522.61 |
| Aug-19 | $934.75 | $0.00 | $616.85 | $122.70 | $1,674.30 |
| Sep-19 | $535.05 | $0.00 | $372.18 | $130.94 | $1,038.16 |
| Oct-19 | $821.44 | $459.08 | $778.56 | $144.99 | $2,204.07 |
| Nov-19 | $321.95 | $385.00 | $446.01 | $151.38 | $1,304.34 |
| Dec-19 | $553.92 | $536.96 | $1,369.21 (**) | $207.92 | $2,668.02 |
| TOTAL | $7,918.07 | $1,381.04 | $5,514.24 | $1,170.99 | $15,984.34 |
(**) I accidentally assigned C5.9xlarge type to Windows workers, which blew up the cost by 700 USD. The issue has been fixed, by downgrading them to C5.4xlarge.
Cost by AWS service:
| Month | EC2 | ECR | S3 | Total |
|---|---|---|---|---|
| Mar-19 | $709.58 | $0.00 | $0.00 | $709.58 |
| Apr-19 | $1,265.43 | $4.11 | $0.09 | $1,269.63 |
| May-19 | $1,972.74 | $11.82 | $0.73 | $1,985.30 |
| Jun-19 | $1,581.12 | $23.92 | $3.30 | $1,608.34 |
| Jul-19 | $1,469.42 | $38.35 | $14.84 | $1,522.61 |
| Aug-19 | $1,611.53 | $54.02 | $8.75 | $1,674.30 |
| Sep-19 | $966.67 | $62.80 | $8.69 | $1,038.16 |
| Oct-19 | $2,127.13 | $69.87 | $7.07 | $2,204.07 |
| Nov-19 | $1,219.30 | $81.50 | $3.54 | $1,304.34 |
| Dec-19 | $2,550.89 | $99.94 | $17.18 | $2,668.02 |
| Total | $15,473.82 | $446.33 | $64.19 | $15,984.34 |