Skip to content

Conversation

michelvocks
Copy link
Member

@michelvocks michelvocks commented Feb 2, 2019

Fixes #107 and #46

Related docs PR: gaia-pipeline/docs#11

TODO:

  • Add tags to worker UI
  • Fix tab panel order in UI
  • Add option to prevent primary instance to accept work
  • Write more tests
  • Write documentation

@michelvocks michelvocks added the Needs Work The PR still requires some work from the submitter. label Feb 2, 2019
@michelvocks michelvocks self-assigned this Feb 2, 2019
@codecov-io
Copy link

codecov-io commented Feb 2, 2019

Codecov Report

Merging #166 into master will decrease coverage by 4.65%.
The diff coverage is 53.7%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #166      +/-   ##
==========================================
- Coverage   66.98%   62.33%   -4.66%     
==========================================
  Files          36       47      +11     
  Lines        2732     3828    +1096     
==========================================
+ Hits         1830     2386     +556     
- Misses        665     1052     +387     
- Partials      237      390     +153
Impacted Files Coverage Δ
helper/rolehelper/role.go 100% <ø> (ø)
workers/server/server.go 0% <0%> (ø)
handlers/user.go 39.68% <0%> (ø) ⬆️
handlers/pipeline_run.go 0% <0%> (ø) ⬆️
handlers/vault.go 57.37% <0%> (-6.26%) ⬇️
handlers/auth.go 95% <100%> (+0.08%) ⬆️
workers/pipeline/build_python.go 82.53% <100%> (+1.5%) ⬆️
workers/pipeline/update_pipeline.go 52.5% <100%> (ø) ⬆️
helper/stringhelper/stringhelper.go 100% <100%> (ø)
handlers/handler.go 89.23% <100%> (+3.26%) ⬆️
... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c81a498...f51fc3c. Read the comment docs.

@michelvocks michelvocks force-pushed the worker_implementation branch from 4de33e0 to 1beed41 Compare February 8, 2019 10:25
@michelvocks michelvocks force-pushed the worker_implementation branch from 72d7b08 to 311620f Compare March 2, 2019 22:15
@Skarlso
Copy link
Member

Skarlso commented Apr 25, 2019

Duuddee! Nice, I'm going to have to review this at one point. :D

@Skarlso
Copy link
Member

Skarlso commented Apr 25, 2019

BTW, this should have been a bit separated into multiple PRs if possible. :D Like, the worker view, then the pipeline stuff and so on and so forth. :)

@michelvocks
Copy link
Member Author

michelvocks commented Apr 25, 2019

@Skarlso I thought about splitting this PR up in multiple PRs beforehand but this was almost impossible. The way this has been implemented now is a bit different from what we agreed on in the RFC since some things didn't work well or included some traps we didn't think of. 😭

Let me check if I can split this up when it's finished. 😄

@Skarlso
Copy link
Member

Skarlso commented Apr 25, 2019

@michelvocks It's fine, it will just be harder to review. :D

Ah, would you mind updating the RFC in that case? :P

@michelvocks
Copy link
Member Author

michelvocks commented Jun 14, 2019

@Skarlso Tests are looking fine now. The test coverage will be decreased a bit but mainly because codecov sucks 😞 I touched a few files where we don't have any tests and codecov simply counts them now as new code 🙄

The plan is to finish the documentation part within the next week. Would be great if you could have another look until then. 😄

@Skarlso
Copy link
Member

Skarlso commented Jun 15, 2019

@Skarlso sure thing dude. :) It's kind of hard to review 7700 lines of code change though... :D

@Skarlso
Copy link
Member

Skarlso commented Jun 16, 2019

Noice!
Screenshot 2019-06-16 at 15 06 46

@Skarlso
Copy link
Member

Skarlso commented Jun 16, 2019

@michelvocks Is there a was to delete the default tags? I mean what if my worker isn't supporting the Python environment? :)

@Skarlso
Copy link
Member

Skarlso commented Jun 16, 2019

@michelvocks Also, the pipelines need a way to display their affiliated tags. I can't view what environment my pipeline needs once I set a tag. I thought that maybe we'd need the ability to edit tags, but I reconsidered. In case an environment need changes for a pipeline we'd just delete it and re-create it. But we still need an ability to see them tags. :)

@michelvocks
Copy link
Member Author

@Skarlso sure thing dude. :) It's kind of hard to review 7700 lines of code change though... :D

I don't expect you to review all the lines of code. 😄
It already helps a lot if you just have a general look and may provide feedback like above. Super helpful 🤗

@michelvocks Is there a was to delete the default tags? I mean what if my worker isn't supporting the Python environment? :)

I've added the possibility to provide "negative tags". So if you provide a tag like "-golang" it will remove automatically the "golang" tag (and the "-golang" tag). Not the best solution in terms of user experience but should work for this use-case. 😄

@michelvocks Also, the pipelines need a way to display their affiliated tags. I can't view what environment my pipeline needs once I set a tag. I thought that maybe we'd need the ability to edit tags, but I reconsidered. In case an environment need changes for a pipeline we'd just delete it and re-create it. But we still need an ability to see them tags. :)

Whops, yeah. I added the tags to the overview page.

@Skarlso
Copy link
Member

Skarlso commented Jun 17, 2019

@michelvocks awesome, I'm okay with the negative tags. :D so do you tag a pipeline automatically when it's created with the appropriate environment tag? Like a Go pipeline gets go a Python pipeline gets Python etc?

@michelvocks
Copy link
Member Author

@michelvocks awesome, I'm okay with the negative tags. :D so do you tag a pipeline automatically when it's created with the appropriate environment tag? Like a Go pipeline gets go a Python pipeline gets Python etc?

Yep: https://github.com/gaia-pipeline/gaia/pull/166/files#diff-f88a32f252fb4994c4b7249afcc79b6cR57

@michelvocks
Copy link
Member Author

@Skarlso Docs PR: gaia-pipeline/docs#11

Still WIP. Need to update the RFC.

@michelvocks
Copy link
Member Author

@Skarlso Docs PR should be fine now. In my opinion, this is now ready for merge.

Let me know what you think. 🤗

// Evaluate and summarize all worker tags
var tags []string
if gaia.Cfg.WorkerTags != "" {
trimmedTags := strings.ReplaceAll(gaia.Cfg.WorkerTags, " ", "")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michelvocks hm, this means that we aren't allowing for tags with spaces in them. That is okay, I guess, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Tags are single words which can be combined with dashes (for example). From my experience, that is the expected behavior.

@Skarlso
Copy link
Member

Skarlso commented Jun 22, 2019

@michelvocks Well. Functionality wise this looks okay. I tested it and it worked. :D But man, the code needs refactoring. :DDD Create a bunch of follow up tickets so we can track the most common ones? Like the functions are too long and such? Otherwise, go ahead. :)

@michelvocks
Copy link
Member Author

michelvocks commented Jun 23, 2019

@michelvocks Well. Functionality wise this looks okay. I tested it and it worked. :D But man, the code needs refactoring. :DDD Create a bunch of follow up tickets so we can track the most common ones? Like the functions are too long and such? Otherwise, go ahead. :)

Is it only the length of some functions? In my opinion, it is quite common and normal to have a bit longer functions. It makes it often more readable because you can simply follow one block of code instead of switching between several functions. I also know other big projects (Kubernetes, Vault, Terraform etc.) which have bigger functions and they also discussed this and decided it can be beneficial.

@michelvocks
Copy link
Member Author

ping @Skarlso

@Skarlso
Copy link
Member

Skarlso commented Jun 25, 2019

I disagree with that. That is why cyclomatic complexity is a thing. And that is why lingers are saying that your functions is too long. It's harder to track down a bug in a big ass function than in a small function where it's basically clear at one quick look what it does. If I have scroll down several pages and still be in the same function I'll loose track of what the beginning of the function was even trying to do in the first place. 😂 Also if a function is huge it's more likely that it does not adhere to SRP. The single responsibility principle. It is likely however that its trying to do everything / too much of its own.

I approve but we have to refractor these into easily tracakble understandable fragments I think.
😊

Also, sorry for the late response, I'm on vacation in Austria. 😁

@michelvocks michelvocks merged commit 376c8ca into gaia-pipeline:master Jun 25, 2019
@michelvocks michelvocks deleted the worker_implementation branch June 25, 2019 08:37
@codecov-commenter
Copy link

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 53.70066% with 563 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@c81a498). Learn more about missing BASE report.
Report is 68 commits behind head on master.

Files with missing lines Patch % Lines
workers/agent/agent.go 48.68% 127 Missing and 48 partials ⚠️
workers/server/worker.go 48.05% 82 Missing and 38 partials ⚠️
handlers/workers.go 48.12% 45 Missing and 24 partials ⚠️
store/memdb/memdb.go 53.67% 42 Missing and 21 partials ⚠️
workers/server/server.go 0.00% 40 Missing ⚠️
workers/scheduler/scheduler.go 58.00% 17 Missing and 4 partials ⚠️
workers/pipeline/ticker.go 50.00% 10 Missing and 3 partials ⚠️
store/worker.go 75.60% 5 Missing and 5 partials ⚠️
store/pipeline.go 57.89% 7 Missing and 1 partial ⚠️
workers/agent/api/api.go 60.00% 4 Missing and 4 partials ⚠️
... and 14 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #166   +/-   ##
=========================================
  Coverage          ?   62.33%           
=========================================
  Files             ?       47           
  Lines             ?     3828           
  Branches          ?        0           
=========================================
  Hits              ?     2386           
  Misses            ?     1052           
  Partials          ?      390           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RFC: Distributer Pipeline execution via Workers
6 participants