Skip to content

Conversation

casperdcl
Copy link
Member

@casperdcl casperdcl commented Nov 24, 2020

based on tqdm.keras

Testing:

pip install "git+https://github.com/tqdm/tqdm@dask#egg=tqdm"

from tqdm.dask import TqdmCallback
import dask.array as da

with TqdmCallback(desc="compute"):
    x = da.random.random((10000, 10000), chunks=(1000, 1000))
    y = x + x.T
    z = y[::2, 5000:].mean(axis=1)
    z.compute()
from tqdm.dask import TqdmCallback
import dask.array as da

cb = TqdmCallback(desc="global")
cb.register()

x = da.random.random((10000, 10000), chunks=(1000, 1000))
y = x + x.T
z = y[::2, 5000:].mean(axis=1)
z.compute()

@casperdcl casperdcl added p3-enhancement 🔥 Much new such feature question/docs ‽ Documentation clarification candidate submodule ⊂ Periphery/subclasses to-merge ↰ Imminent submodule-notebook 📓 Much web such IDE synchronisation ⇶ Multi-thread/processing c1-quick 🕐 Complexity low labels Nov 24, 2020
@casperdcl casperdcl added this to the Non-breaking milestone Nov 24, 2020
@casperdcl casperdcl self-assigned this Nov 24, 2020
@casperdcl casperdcl mentioned this pull request Nov 24, 2020
Base automatically changed from devel to master November 27, 2020 14:36
@codecov
Copy link

codecov bot commented Feb 22, 2021

Codecov Report

Merging #1079 (7c42d80) into devel (4ff7753) will decrease coverage by 0.38%.
The diff coverage is 70.83%.

@@            Coverage Diff             @@
##            devel    #1079      +/-   ##
==========================================
- Coverage   84.81%   84.43%   -0.39%     
==========================================
  Files          24       25       +1     
  Lines        1627     1651      +24     
  Branches      270      273       +3     
==========================================
+ Hits         1380     1394      +14     
- Misses        209      218       +9     
- Partials       38       39       +1     

@casperdcl casperdcl changed the base branch from master to devel February 22, 2021 23:00
@casperdcl
Copy link
Member Author

@deschman from #278 (comment): do you mean the example code in the OP above doesn't work for you?

@deschman
Copy link

deschman commented Feb 23, 2021

I mean to say that I have been testing the devel branch with the code below.

import time

import dask
from tqdm.dask import TqdmCallback as ProgressBar

schedule = []

for i in range(5):
    schedule.append(dask.delayed(time.sleep)(i))

with ProgressBar():
    dask.compute(schedule)

This fails because Callback from dask 2021.2.0 does not have a pbar attribute.

Edit: I took another look at your test code above and that seems to fail for the same reason.

@casperdcl
Copy link
Member Author

interesting... I ran you code (ubuntu, pip install -U "dask==2021.2.0" "git+https://github.com/tqdm/tqdm@dask#egg=tqdm") and can't see any error:

100%|██████████████████| 5/5 [00:04<00:00,  1.25it/s]

@deschman
Copy link

deschman commented Feb 23, 2021

I stepped through this more slowly and found the problem. Below is a traceback that is handled without any message. During handling, TqdmCallback._finish is called, where self.pbar is not defined due to the handled error. Perhaps an update to requirements or better handling of this error would be helpful?

  File "c:\users\deschman\spyder-env\lib\site-packages\dask\local.py", line 436, in get_async
    start_state(dsk, state)

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\dask.py", line 27, in _start_state
    self.pbar = self.tqdm_class(total=sum(

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\notebook.py", line 240, in __init__
    self.container = self.status_printer(self.fp, total, self.desc, self.ncols)

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\notebook.py", line 117, in status_printer
    raise ImportError(

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

@casperdcl
Copy link
Member Author

ah right. That's just Python's default traceback - always important to look for the first error :)

casperdcl and others added 3 commits March 5, 2021 12:55
@casperdcl casperdcl merged commit 2a82405 into devel Mar 5, 2021
@casperdcl casperdcl deleted the dask branch March 5, 2021 13:48
@casperdcl casperdcl mentioned this pull request Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c1-quick 🕐 Complexity low p3-enhancement 🔥 Much new such feature question/docs ‽ Documentation clarification candidate submodule ⊂ Periphery/subclasses submodule-notebook 📓 Much web such IDE synchronisation ⇶ Multi-thread/processing to-merge ↰ Imminent
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dask support
2 participants