Skip to content

Why is workload_requires needed? #182

@solo-driven

Description

@solo-driven

Question

ALL EXAMPLES ARE RUN LOCALLY

in the example for htcondor (https://github.com/riga/law/blob/master/examples/sequential_htcondor_at_cern/analysis/tasks.py) workload_requires is being used and results in the following graph:
image
Scheduled 45 tasks of which:

  • 45 ran successfully:
    • 32 CreateChars(...)
    • 1 CreateFullAlphabet(...)
    • 12 CreatePartialAlphabet(...)

But when I just comment it I get the following more clearer graph:
image

Scheduled 39 tasks of which:

  • 39 ran successfully:
    • 26 CreateChars(...)
    • 1 CreateFullAlphabet(...)
    • 12 CreatePartialAlphabet(...)

Does this change anything? Other than number of tasks decreases when no workflow_requirements is not provided from 45 to 39

In addition it is also possible by changing reruires and run to:

def requires(self):
    # require CreateChars for each index referred to by the branch_data of _this_ instance

    return CreateChars.req(self, branches=self.branch_data, branch=-1)


def run(self):
    # gather characters and save them
    alphabet = ""
    for inp in self.input()['collection'].targets.values():
        alphabet += inp.load()["char"]
  ....

to obtain the following graph:
image

And finally the result which I was expecting to see:
image

can be done by changing the CreateFullAlphabet:

def requires(self):
    return CreatePartialAlphabet.req(self)


def run(self):
    # loop over all targets holding partial alphabet fractions and concat them
    inputs = self.input()["collection"].targets
    parts = [
        inp.load().strip()
        for inp in inputs.values()
    ]
    alphabet = "-".join(parts)

I would really appreciate if you could help me with that, struggled a lot with this trying to find the reason for workload_requires. Thank you for reading

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions