Upgrade to latest spark versions 3.3.0, 3.2.2 and 3.1.3 #415
Conversation
Add CI build for spark 3.3.0 Upgrade spark 3.2.0 to 3.2.2 Upgrade spark 3.1.2 to 3.1.3
The previous setup would cause test to end up using a stopped spark context causing them to fail. This seems to be due to some change in spark 3.3.0, by making the whole suite using a single SparkContext this issue goes away.
|
Tagging @mengxr and @WeichenXu123 for visibility since you most recently reviewed/merged PRs. But since this was several months back, please let me know if you are no longer maintaining this project and if there anyone else is? |
|
Tagging @felixcheung and @jkbradley who are in the graphframes organization. Do you know if there anyone who is currently interested reviewing PRs for this repository? |
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #415 +/- ##
==========================================
+ Coverage 91.26% 91.37% +0.11%
==========================================
Files 18 18
Lines 870 870
Branches 53 53
==========================================
+ Hits 794 795 +1
+ Misses 76 75 -1 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
LGTM! Thanks @eejbyfeldt!
@WeichenXu123 can you also take a quick look?
|
Thanks, @dbczumar and @WeichenXu123 for having a look! I also created a "follow up" here PR: #417 adding support for Scala 2.13 for the spark releases that provides pre-built version with it. I would be very grateful if you had a look at that as well. |
This add CI builds for spark 3.3.0 and updates to 3.2.2 and 3.1.3.
The new build for 3.3.0 required some small changes in the testsuite for it to run without failures.