-
Notifications
You must be signed in to change notification settings - Fork 252
Python: Replace SQLContext with SparkSession #424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Tagging @WeichenXu123, since you reviewed the most recent MR. |
@ttc0419 Thank you! I will take a look soon. |
The trait |
Spark 2.4.x does not support SparkSession.getActiveSession(). According to https://spark.apache.org/versioning-policy.html, spark 2.4.x is EOL and the last version was released 5 years ago. There's little meaning to support it.
I'm not familiar with Scala, probably need someone else and another MR to do that. |
We can do it in follow-up PR. |
@WeichenXu123 Anything else need to be done before merging this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## master #424 +/- ##
==========================================
- Coverage 91.37% 91.19% -0.19%
==========================================
Files 18 18
Lines 870 829 -41
Branches 53 49 -4
==========================================
- Hits 795 756 -39
+ Misses 75 73 -2
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
@WeichenXu123 Do we need another reviewer to merge this? |
I can merge it. |
According to the official document,
SQLContext
is deprecated since spark 2.0. UseSparkSession
instead.Notes:
This MR also removes CI for spark 2.4.8 since it does not support
SparkSession.getActiveSession()
. And according to versioning policy, spark 2.4.x is EOL and the last version was released in 5 years ago. There's little meaning to support it.Closes: #423