Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1 #430

eejbyfeldt · 2023-04-24T11:23:52Z

Adds support and make Spark 3.4.1 the default sbt build.

Building with spark 3.4.1 seems to require no actual code changes.

codecov-commenter · 2023-04-24T11:33:00Z

Codecov Report

Merging #430 (82bb440) into master (3d76998) will increase coverage by 0.23%.
Report is 1 commits behind head on master.
The diff coverage is n/a.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##           master     #430      +/-   ##
==========================================
+ Coverage   91.07%   91.30%   +0.23%     
==========================================
  Files          18       18              
  Lines         829      828       -1     
  Branches       49       55       +6     
==========================================
+ Hits          755      756       +1     
+ Misses         74       72       -2

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

📢 Have feedback on the report? Share it here.

EnricoMi

I think Spark 3.0 and 3.1 can be removed as they are EOL.

Latest Spark 3.2 version is 3.2.4, could be updated while we are here.

.github/workflows/scala-ci.yml

EnricoMi

LGTM!

galewis2 · 2023-05-04T13:28:05Z

Can this be merged with new packages built?

dinedal · 2023-09-06T21:48:48Z

@EnricoMi @eejbyfeldt could you please merge?

eejbyfeldt · 2023-09-07T05:37:20Z

Neither of us have commit rights to this repo. Merging would require someone like @WeichenXu123 that have merged PRs in the past.

dinedal · 2023-09-07T05:42:52Z

Also tagging @mengxr

rjurney · 2023-09-07T15:10:27Z

@WeichenXu123 can you merge this PR? I love this software, what do we have to do to get a release cut? It has been two years! I need my big data motif searches :)

rjurney · 2023-09-07T15:11:02Z

Or @mengxr can you cut a release?

rjurney · 2023-09-07T15:12:28Z

Does this need fixing?

https://github.com/graphframes/graphframes/actions/runs/6105938737/job/16570195522#step:5:1

WeichenXu123

LGTM

eejbyfeldt · 2023-09-07T15:20:42Z

Does this need fixing?

https://github.com/graphframes/graphframes/actions/runs/6105938737/job/16570195522#step:5:1

To me this looks unrelated to the changes. Retriggered CI to see it it passes the second time.

WeichenXu123 · 2023-09-07T15:21:09Z

https://github.com/graphframes/graphframes/actions/runs/6105938737/job/16570195522#step:5:1

Does all tests pass but just sbt raise error when exiting ?

eejbyfeldt · 2023-09-07T15:24:58Z

https://github.com/graphframes/graphframes/actions/runs/6105938737/job/16570195522#step:5:1

Does all tests pass but just sbt raise error when exiting ?

Looks there was a seg fault in the jvm:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe873dcc2ea, pid=2928, tid=2934
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x7cc2ea]  G1ParCopyClosure<(G1Barrier)0, (G1Mark)1>::do_oop(oopDesc**)+0x3a
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/graphframes/graphframes/core.2928)
#
# An error report file with more information is saved as:
# /home/runner/work/graphframes/graphframes/hs_err_pid2928.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Exception in thread "Thread-6" java.io.EOFException

rjurney · 2023-09-07T15:27:36Z

SHIP IT 😬

EnricoMi · 2023-09-08T06:24:32Z

Can we also get a Scala 2.13 release? #426

EnricoMi

Minor bump of Spark 3.3.2 to 3.3.3.

.github/workflows/python-ci.yml

.github/workflows/scala-ci.yml

dev/release.py

Co-authored-by: Enrico Minack <github@enrico.minack.dev>

dinedal · 2023-09-11T17:02:56Z

@WeichenXu123 @mengxr - thanks for your efforts here, it looks like we're close to merging this?

rjurney · 2023-09-11T18:58:32Z

Yes, thanks everyone! Let's keep this project alive.

rjurney · 2023-09-12T18:03:11Z

So now a release has to get cut?

WeichenXu123 · 2023-09-13T09:19:42Z

@rjurney spark 3.5 will be released soon, shall we wait and add spark 3.5 support and then release a new graphframe version ?

WeichenXu123 · 2023-09-25T14:03:19Z

@rjurney

PySpark 3.5 is delayed for some reason.

I released graphframe with spark 3.4.1 (version: graphframes/graphframes:0.8.3-spark3.4-s_2.12 and graphframes/graphframes:0.8.3-spark3.4-s_2.13)

but they are pending @mengxr 's approval:

WeichenXu123 · 2023-09-27T03:35:15Z

@rjurney

Graphframe 0.8.3 is released: https://github.com/graphframes/graphframes/releases/tag/v0.8.3

We can use it by command like:
bin/pyspark --packages graphframes:graphframes:0.8.3-spark3.4-s_2.12
or
bin/pyspark --packages graphframes:graphframes:0.8.3-spark3.4-s_2.13
depending your spark scala version.

Doc is here: https://graphframes.github.io/graphframes/

EnricoMi · 2023-09-27T07:52:33Z

@WeichenXu123 Spark 3.5.0 has been released, graphframes 0.8.3 can now be released for Spark 3.5.0 as well.

rjurney · 2023-10-09T01:06:09Z

Can you update the QuickStart doc with the new version? It still says 0.6

WeichenXu123 · 2023-10-14T04:08:43Z

@rjurney

Doc page is updated! See https://graphframes.github.io/graphframes/docs/_site/quick-start.html

rjurney · 2023-10-16T21:15:11Z

Thanks, I meant to get a PR in and didn't quite push it. I appreciate it. Thanks, Russell Jurney @rjurney <http://twitter.com/rjurney> ***@***.*** LI <http://linkedin.com/in/russelljurney> FB <http://facebook.com/jurney> datasyndrome.com Book a time on Calendly <https://calendly.com/rjurney_personal/30min>

…

On Fri, Oct 13, 2023 at 9:08 PM WeichenXu ***@***.***> wrote: @rjurney <https://github.com/rjurney> Doc page is updated! See https://graphframes.github.io/graphframes/docs/_site/quick-start.html — Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAKJJPWH4ZO3FKL5RNZTNLX7IF5LANCNFSM6AAAAAAXJN3ZXE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Spark 3.4.0

aa13932

EnricoMi reviewed Apr 24, 2023

View reviewed changes

.github/workflows/scala-ci.yml Show resolved Hide resolved

.github/workflows/scala-ci.yml Outdated Show resolved Hide resolved

.github/workflows/scala-ci.yml Outdated Show resolved Hide resolved

EnricoMi mentioned this pull request Apr 24, 2023

Release with Scala 2.13 #426

Closed

Emil Ejbyfeldt added 3 commits April 25, 2023 14:00

Drop EOL spark versions

2b48d1c

Upgrade 3.2 to 3.2.4

186e228

Align scala version with what is used in spark

daaeb87

eejbyfeldt force-pushed the spark-3.4.0 branch from 0ce2442 to daaeb87 Compare April 25, 2023 13:26

Emil Ejbyfeldt added 2 commits April 25, 2023 15:39

Used 2.12.12 in scala ci for sbt-scoverage compatibility

757b348

rebuild

5e984d6

eejbyfeldt requested a review from EnricoMi April 25, 2023 13:53

eejbyfeldt changed the title ~~Spark 3.4.0~~ Spark 3.4.0 and drop EOL 3.0 and 3.1 Apr 25, 2023

EnricoMi approved these changes Apr 25, 2023

View reviewed changes

eejbyfeldt changed the title ~~Spark 3.4.0 and drop EOL 3.0 and 3.1~~ Spark 3.4.0 and drop EOL Spark versions 3.0 and 3.1 Apr 25, 2023

Use Spark 3.4.1

a947f7c

eejbyfeldt changed the title ~~Spark 3.4.0 and drop EOL Spark versions 3.0 and 3.1~~ Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1 Sep 7, 2023

WeichenXu123 approved these changes Sep 7, 2023

View reviewed changes

Trigger CI

b223928

EnricoMi reviewed Sep 8, 2023

View reviewed changes

.github/workflows/python-ci.yml Outdated Show resolved Hide resolved

.github/workflows/scala-ci.yml Outdated Show resolved Hide resolved

.github/workflows/scala-ci.yml Outdated Show resolved Hide resolved

dev/release.py Outdated Show resolved Hide resolved

Apply suggestions from code review

82bb440

Co-authored-by: Enrico Minack <github@enrico.minack.dev>

WeichenXu123 merged commit 7e2cd1c into graphframes:master Sep 12, 2023

eejbyfeldt deleted the spark-3.4.0 branch September 12, 2023 09:23

bjornjorgensen mentioned this pull request Feb 16, 2025

1 of 3: Build a graphframes Python package during the build process #512

Merged

Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1 #430

Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1 #430

Uh oh!

Conversation

eejbyfeldt commented Apr 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Apr 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

EnricoMi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EnricoMi left a comment

Choose a reason for hiding this comment

Uh oh!

galewis2 commented May 4, 2023

Uh oh!

dinedal commented Sep 6, 2023

Uh oh!

eejbyfeldt commented Sep 7, 2023

Uh oh!

dinedal commented Sep 7, 2023

Uh oh!

rjurney commented Sep 7, 2023

Uh oh!

rjurney commented Sep 7, 2023

Uh oh!

rjurney commented Sep 7, 2023

Uh oh!

WeichenXu123 left a comment

Choose a reason for hiding this comment

Uh oh!

eejbyfeldt commented Sep 7, 2023

Uh oh!

WeichenXu123 commented Sep 7, 2023

Uh oh!

eejbyfeldt commented Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjurney commented Sep 7, 2023

Uh oh!

EnricoMi commented Sep 8, 2023

Uh oh!

EnricoMi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dinedal commented Sep 11, 2023

Uh oh!

rjurney commented Sep 11, 2023

Uh oh!

rjurney commented Sep 12, 2023

Uh oh!

WeichenXu123 commented Sep 13, 2023

Uh oh!

WeichenXu123 commented Sep 25, 2023

Uh oh!

WeichenXu123 commented Sep 27, 2023

Uh oh!

EnricoMi commented Sep 27, 2023

Uh oh!

rjurney commented Oct 9, 2023

Uh oh!

WeichenXu123 commented Oct 14, 2023

Uh oh!

rjurney commented Oct 16, 2023 via email

Uh oh!

Uh oh!

eejbyfeldt commented Apr 24, 2023 •

edited

Loading

codecov-commenter commented Apr 24, 2023 •

edited

Loading

eejbyfeldt commented Sep 7, 2023 •

edited

Loading