Skip to content

Faster Hierarchical Agglomerative clustering, output clustering metrics to Data Sets #939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 24 commits into from
Jan 31, 2022
Merged

Faster Hierarchical Agglomerative clustering, output clustering metrics to Data Sets #939

merged 24 commits into from
Jan 31, 2022

Conversation

drroe
Copy link
Contributor

@drroe drroe commented Jan 31, 2022

Version 6.2.3.

This PR contains the following improvements.

  • Improve the speed of hierarchical agglomerative clustering by an order of magnitude.
  • Create data sets for Davies-Bouldin index, pseudo-F, and SSR/SST ratio clustering metrics.
  • Check for and fix a potential integer overflow when setting the random number generator seed from the wall clock.

drroe added 22 commits January 24, 2022 12:54
@drroe
Start experimental command to compare clusters
e0e94f8
@drroe
Report some stats
9a0d0c0
@drroe
Use time instead of wall time to seed RNG. Protect against int overflow
dbbfc7b 
of seed
@drroe
Use wallclock time again, finer grained so less likelyhood of rapid runs
b1b3b0e 
in succession using the same seed I hope
@drroe
Add missing letter
2e6fd7c
@drroe
Put DBI in separate file
993234a
@drroe
Put pseudo F calc in separate file
60c0098
@drroe
Take DBI/pseudo-F calc out of list and store those metrics in Control,
03d50ea 
have them calculated at the end of the run instead of just when info
happens. This way they can eventually be put into datasets.
@drroe
Allow cpptraj to be set via env var
a148105
@drroe
Add data sets for DBI and pseudo-F
fbc13e9
@drroe
Store SSR/SST value in set. Update manual.
effc938
@drroe
Fix TIMER compile
813a3e9
@drroe
Start exploring a different way of speeding up finding cluster to
6aad510 
cluster min distance by keeping track of closest cluster to each
cluster and updating as needed.
@drroe
FindMin needs to ensure iOut < jOut
2dd9db6
@drroe
Disable some debug info
f2be4d3
@drroe
Bench hieragglo
44073b6
@drroe
Remove old code and openmp code. Finding min is no longer the bottleneck
ed8c66c
@drroe
Check openmp
f411dea
@drroe
Remove debug info
abb8c01
@drroe
Merge branch 'master' into cluster.work
68f24a8
@drroe
Update copyright year. Add Franz and Klaus to contributor list.
33399a4
@drroe
Version 6.2.3. Revision bump for HA cluster speed improvement and DBI…
53a7926 
…/pSF/SSRSST DataSet output.
@drroe drroe self-assigned this Jan 31, 2022
@drroe
Copy link
Contributor Author

drroe commented Jan 31, 2022

Pytraj failure is because I added 3 new data sets during clustering, causing the offset used by pytraj to access cluster data to be invalid:

https://jenkins.jasonswails.com/blue/organizations/jenkins/amber-github%2Fpytraj/detail/pytraj/414/pipeline#step-22-log-663

@hainm is there any reason why we're using data[-2] and not data[0] or something? The former (two from the last I think?) means that every time a new data set is added to cluster the index becomes invalid. Can we make them [0]? I would test myself but I'm having issues getting pytraj compiled on my platform (unrelated glibc issues).

@hainm hainm mentioned this pull request Jan 31, 2022
Merged
@hainm
Copy link
Contributor

hainm commented Jan 31, 2022

Can we make them [0]? I would test myself but I'm having issues getting pytraj compiled on my platform (unrelated glibc issues).

For the record: Amber-MD/pytraj#1598 (comment)

@drroe drroe merged commit caf123a into Amber-MD:master Jan 31, 2022
@drroe drroe deleted the cluster.work branch January 31, 2022 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@drroe drroe

Labels
bugfix enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@drroe @hainm