-
Notifications
You must be signed in to change notification settings - Fork 726
[GH-2085] use graphframes 0.9.0 #2098
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
7012ad7
to
ad55a66
Compare
ad55a66
to
8d50979
Compare
8d50979
to
81f3a09
Compare
We should wait until 0.9.0 is release and we can point to the non-snapshot version before merging this. If we disagree with the change in behavior, this PR gives us a chance to discuss before 0.9.0 is released. |
You don't just want to set the config to maintain the old behavior? It seems odd that the output of the dbscan function is based on a config a user could set for a transitive dependency they don't even know is used for dbscan |
I really dislike the way the spark configs are global configurations and so modifying a spark config changes the state of the whole cluster. I'm thinking of adding a setter to graphframes 0.9.0 so I can set this value on the connectedComponents instance without changing spark configs. Then the output schema of DBSCAN is not based on the user's setting of the graphframe spark config (one way or the other). |
Yeah it's definitely not the cleanest thing, and I was thinking the same thing a setting on the connected components object would be much cleaner |
da542d8
to
1119346
Compare
8fd455a
to
5bfc78a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be great to get this in and a 1.8.0 release with Spark 4 support!
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject
. Closes Bump graphframes version to 0.9.0 #2085What changes were proposed in this PR?
How was this patch tested?
unit tests
Did this PR include necessary documentation updates?