Mass plotting notebook #1052

srivarra · 2023-08-30T22:30:33Z

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Adds a notebook that demonstrates how to generate plots for a cohort.
Adds functionality to plot a continuous variable.

For cluster mask plots, adds the ability to erode the segmentation mask boundaries to improve the readability of cells.

How did you implement your changes

Added a notebook which allows the user to plot cluster masks and continuous variables for a cohort.

Remaining issues

May have accidentally ran a formatter in the plot_utils.py file.

This branch needs to be first merged into #1030. Then that will be merged into main.

Sample Plots

review-notebook-app · 2023-08-30T22:30:37Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…to masks/notebook

srivarra · 2023-08-30T23:29:23Z

Not sure why the other checks (testing, building) are not being displayed. They are running here: https://github.com/angelolab/ark-analysis/actions/runs/6030848048?pr=1030

camisowers

Code looks good but a couple general comments.

There seems to be some visualizations missing from the two LDA and the fiber segmentation notebooks.
Also the multicolored borders on the example plot you posted can be removed by setting interpolation=None. (#967) I see it's included for the non-continuous plotting, but is it working when saving the images?

srivarra · 2023-08-31T18:04:37Z

@camisowers

There seems to be some visualizations missing from the two LDA and the fiber segmentation notebooks.

Yeah I see they got cut off.

Also the multicolored borders on the example plot you posted can be removed by setting interpolation=None. (plot_neighborhood_cluster_result has multiple colors per cell #967) I see it's included for the non-continuous plotting, but is it working when saving the images?

Ah, it has to be the string "none" not the value None /:

…xie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells

* Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for

* pin to 0.19.3 * requirement less than 0.19.3

…o helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite

* single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix

…n GHA, the CI workflow should work

ngreenwald

Looks good!

ngreenwald · 2023-09-14T17:52:16Z

src/ark/segmentation/fiber_segmentation.py

@@ -246,7 +247,7 @@ def calculate_fiber_alignment(fiber_object_table, k=4, axis_thresh=2):
            # find index for smallest distances, excluding itself
            indy = fiber_dist_mat[indx, :].argsort()[1:1+k]
            neighbor_angles = filtered_lengths.orientation[indy]
-            fiber_scores.append(1 / (np.sqrt(np.sum((neighbor_angles - angle) ** 2)) / k))


How come this was changed?

In the function color_fibers_by_stat we take the inverse of the alignment score again in order to plot it. Cami suggested that we just remove the inversion entirely.

src/ark/utils/data_utils.py

ngreenwald · 2023-09-14T18:11:02Z

templates/3_Pixie_Cluster_Cells.ipynb

@@ -854,7 +854,7 @@
    "        xr_channel_names=None,\n",


Line #11. plot_utils.plot_pixel_cell_cluster(
You should check with Candace if she wants this to default to erode=True or False. I think the cell masks look better eroded.

Reply via ReviewNB

Got Candace's approval for cell mask erosion in the Pixie Pipeline notebooks.

review-notebook-app · 2023-09-14T18:11:04Z

View / edit / reply to this conversation on ReviewNB

ngreenwald commented on 2023-09-14T18:11:03Z
----------------------------------------------------------------

We should clear this output

templates/generic_cell_clustering.ipynb

review-notebook-app · 2023-09-14T18:11:06Z

View / edit / reply to this conversation on ReviewNB

ngreenwald commented on 2023-09-14T18:11:06Z
----------------------------------------------------------------

Does the example dataset cell table have multiple different clustering columns? Or just one? If it has multiple, we should use the one that has broader cell type definitions, so there are fewer different clusters

srivarra commented on 2023-09-14T18:45:07Z
----------------------------------------------------------------

I believe there is only one metacluster column.

review-notebook-app · 2023-09-14T18:11:07Z

View / edit / reply to this conversation on ReviewNB

ngreenwald commented on 2023-09-14T18:11:07Z
----------------------------------------------------------------

Looks great

srivarra · 2023-09-14T18:45:08Z

I believe there is only one metacluster column.

View entire conversation on ReviewNB

…leaned up some notebook outputs

…actoring step.

* plot_cluster Next Release v0.6.4 (#1012) * 0.6.4 * updated environment.yml * Update README.md * updated windows setup docs * added repo verison in git clone * file fix Deepcell upload loop cleaning (#1023) * loop check for successful deepcell upload * update test zip paths * helper functions pls * zip input, upload, and extract per batch * cleaning * docstring indent * fov -> fovs file name * remove overwrite warning * new dir for each test call * fix bad logic warning testing * remove arg from docstring * add helper function test * batch_num start at 1 * add previously processed warning * set timeouts and scrap retry * correct timeout errors and update tests * timeout 5 mins * add to unprocessed list after loop closes * continue loop after extraction * break instead of continue * skipped processing print fixes * adjust print statements * 3 seconds before second zip call update * updated readthedocs python version * updated plot_utils docstring * docstrings v2 * use GS [1,1] if cbar_visible is False, otherwise use GS [1, 2] for plot + colorbar * added cbar_visible T/F for tests * pycodestyle * added numpy list to cmap / norm function * Remove cells that don't have any pixel clusters expressed prior to Pixie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells * Include `img_sub_folder` as parameter in Mantis calls (#1050) * Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for * Update HuggingFace version (#1053) * v0.1.10 (#1057) * Pin to scikit-image to v0.19.3 (#1060) * pin to 0.19.3 * requirement less than 0.19.3 * Make sure overwrite functionality for pixel clustering propagates into helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite * Single mantis directory (#1061) * single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix * Mass plotting notebook (#1052) * added plotting notebook * updated readmed * git merge issues * git merge fixes v2 - docs/landing.md * type * Remove cells that don't have any pixel clusters expressed prior to Pixie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells * Include `img_sub_folder` as parameter in Mantis calls (#1050) * Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for * fixed clipped plots in the notebooks. * Update HuggingFace version (#1053) * v0.1.10 (#1057) * Added continuous variable segmentation plots * Pin to scikit-image to v0.19.3 (#1060) * pin to 0.19.3 * requirement less than 0.19.3 * Make sure overwrite functionality for pixel clustering propagates into helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite * Single mantis directory (#1061) * single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix * strange merge issue * removed branch filters on CI, so if you branch off of a branch and run GHA, the CI workflow should work * changed ci.yml, filter push on main branch only * python 3.9 doesn't support | for Union type overloading * added fovs parameter for continuous variable plots * updated notebook to include plots * updated cell clustering notebooks to add erosion (default to True), cleaned up some notebook outputs * fixed style imports * re-added test_generate_summary_stats. Was accidently deleted in a refactoring step. * newline at eof fiber_segmentation_test.py * seaborn-paper -> seaborn-v0_8-paper * added cell to view all plotting styles i mask gen nb * pycodestyle --------- Co-authored-by: alex-l-kong <31424707+alex-l-kong@users.noreply.github.com> Co-authored-by: camisowers <38049893+camisowers@users.noreply.github.com> * default ero * added erode=True for generate_cluster_mask --------- Co-authored-by: alex-l-kong <31424707+alex-l-kong@users.noreply.github.com> Co-authored-by: camisowers <38049893+camisowers@users.noreply.github.com>

added plotting notebook

72b8c34

srivarra self-assigned this Aug 30, 2023

Sricharan Reddy Varra added 3 commits August 30, 2023 15:31

updated readmed

4437558

git merge issues

c698ebc

git merge fixes v2 - docs/landing.md

93b4553

srivarra changed the base branch from main to masks/generation August 30, 2023 22:44

Sricharan Reddy Varra added 3 commits August 30, 2023 15:54

Merge branch 'masks/generation' into masks/notebook

81ff4fe

type

cfe16f6

Merge branch 'masks/notebook' of github.com:angelolab/ark-analysis in…

5a56e44

…to masks/notebook

srivarra requested a review from alex-l-kong August 30, 2023 23:27

srivarra assigned camisowers and unassigned camisowers Aug 30, 2023

srivarra requested a review from camisowers August 30, 2023 23:27

srivarra marked this pull request as ready for review August 30, 2023 23:27

srivarra removed request for alex-l-kong and camisowers August 30, 2023 23:27

srivarra marked this pull request as draft August 30, 2023 23:27

srivarra added the enhancement New feature or request label Aug 30, 2023

srivarra requested review from alex-l-kong and camisowers August 30, 2023 23:29

srivarra marked this pull request as ready for review August 30, 2023 23:29

camisowers requested changes Aug 31, 2023

View reviewed changes

alex-l-kong and others added 3 commits August 31, 2023 18:07

Remove cells that don't have any pixel clusters expressed prior to Pi…

b4baca2

…xie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells

fixed clipped plots in the notebooks.

59fe8a9

srivarra requested a review from camisowers August 31, 2023 21:20

Update HuggingFace version (#1053)

e5d0fd9

camisowers and others added 10 commits September 12, 2023 17:44

Pin to scikit-image to v0.19.3 (#1060)

4159adf

* pin to 0.19.3 * requirement less than 0.19.3

Single mantis directory (#1061)

ec690c5

* single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix

strange merge issue

caf0ce5

removed branch filters on CI, so if you branch off of a branch and ru…

a23a9ad

…n GHA, the CI workflow should work

Merge branch 'masks/generation' into masks/notebook

152181d

changed ci.yml, filter push on main branch only

681ebf8

python 3.9 doesn't support | for Union type overloading

299610e

added fovs parameter for continuous variable plots

f584f9b

updated notebook to include plots

3565f4f

srivarra requested a review from ngreenwald September 14, 2023 17:37

ngreenwald requested changes Sep 14, 2023

View reviewed changes

ngreenwald reviewed Sep 14, 2023

View reviewed changes

templates/generic_cell_clustering.ipynb Show resolved Hide resolved

Sricharan Reddy Varra added 7 commits September 15, 2023 14:48

updated cell clustering notebooks to add erosion (default to True), c…

0c7ac88

…leaned up some notebook outputs

fixed style imports

a66606d

re-added test_generate_summary_stats. Was accidently deleted in a ref…

16cb4e9

…actoring step.

newline at eof fiber_segmentation_test.py

42c4131

seaborn-paper -> seaborn-v0_8-paper

af8ee0a

added cell to view all plotting styles i mask gen nb

3203113

pycodestyle

2eb6471

srivarra requested a review from ngreenwald September 15, 2023 23:56

ngreenwald approved these changes Sep 17, 2023

View reviewed changes

ngreenwald merged commit b3ebaa0 into masks/generation Sep 17, 2023

ngreenwald deleted the masks/notebook branch September 17, 2023 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Mass plotting notebook #1052

Mass plotting notebook #1052

Uh oh!

srivarra commented Aug 30, 2023 •

edited

Loading

Uh oh!

review-notebook-app bot commented Aug 30, 2023

Uh oh!

srivarra commented Aug 30, 2023

Uh oh!

camisowers left a comment •

edited

Loading

Uh oh!

srivarra commented Aug 31, 2023 •

edited

Loading

Uh oh!

ngreenwald left a comment

Uh oh!

ngreenwald Sep 14, 2023

Uh oh!

srivarra Sep 14, 2023 •

edited

Loading

Uh oh!

Uh oh!

ngreenwald Sep 14, 2023 •

edited

Loading

Uh oh!

srivarra Sep 15, 2023

Uh oh!

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading

Uh oh!

Uh oh!

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading

Uh oh!

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading

Uh oh!

srivarra commented Sep 14, 2023

Uh oh!

Uh oh!

Mass plotting notebook #1052

Mass plotting notebook #1052

Uh oh!

Conversation

srivarra commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app bot commented Aug 30, 2023

Uh oh!

srivarra commented Aug 30, 2023

Uh oh!

camisowers left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srivarra commented Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngreenwald left a comment

Choose a reason for hiding this comment

Uh oh!

ngreenwald Sep 14, 2023

Choose a reason for hiding this comment

Uh oh!

srivarra Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngreenwald Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srivarra Sep 15, 2023

Choose a reason for hiding this comment

Uh oh!

review-notebook-app bot commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

review-notebook-app bot commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app bot commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srivarra commented Sep 14, 2023

Uh oh!

Uh oh!

srivarra commented Aug 30, 2023 •

edited

Loading

camisowers left a comment •

edited

Loading

srivarra commented Aug 31, 2023 •

edited

Loading

srivarra Sep 14, 2023 •

edited

Loading

ngreenwald Sep 14, 2023 •

edited

Loading

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading

review-notebook-app bot commented Sep 14, 2023 •

edited

Loading