-
Notifications
You must be signed in to change notification settings - Fork 252
Rjurney/motif tutorial code min #520
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s.txt and split out requirements-dev.txt. Version bumps.
…ney/build-upgrades
…ney/build-upgrades
…who just pasted or tried to run the code without a new SparkSession.
…m SparkSession.sparkContext. Use click.echo instead of print
…ting a SparkContext. print-->click.echo
…mes stackexchange' command.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Do-over of #518 since I mistakenly merged it to
rjurney/build-upgrades
rather thanmaster
.This is a sub-PR of the monster PR #473. It is the code that corresponds to #511. It needs to get merged after #512 and before the docs PR #511.
These changes do the following:
graphframes.tutorials.download
. [Thought: thisClick
usage can be the basis for future CLI commands for agraphframes
command? Just an idea.]graphframes.tutorials.stackexchange
graphframes.tutorials.stackexchange
. No longer requires case sensitivity forid
/Id
fields.graphframes.tutorials.motif
- this is here to provide for future unit testability of tutorials and as a Github browsable reference that matches the Motif Finding tutorial in 3 of 3: Documentation cleanup and update. Added a motif finding tutorial. #511.In addition:
Nodes.parquet
andEdges.parquet
this PR creates can be wired into the unit tests for a more realistic setting in a near future PR by me. We could put thepython/graphframes/tutorials/data/
folder underpython/data
orpython/graphframes/data
to accommodate this. We need a real dataset for our unit tests, I don't have confidence in changes to algorithms like connected components or PageRank without real data and known outcomes.Why are the changes needed?
These changes are needed to make the docs in #511 work. Otherwise that PR's new Motif Finding Tutorial won't work. Merge me first :)