Skip to content

1177-MCV-BE Fix lazycache long lock issue #4949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 16, 2025

Conversation

moz-bozden
Copy link
Contributor

Internal issue
Internal main issue

The Redis based lazyCache implementation had a very long time (3 hours fixed), which prevented more quick updates.
As part of re-visiting the Redis caching in general to optimize query performance by using caching, we first fix this issue.

This adds a new lockDurationMs parameter to lazyCache, which defaults to 3 minutes. Also added some plausible lock duration to the calling functions, to be fine-tuned after fine examination of query performances.

@moz-bozden moz-bozden self-assigned this Jun 24, 2025
@moz-bozden moz-bozden requested a review from a team as a code owner June 24, 2025 00:02
@moz-bozden moz-bozden added the Bug label Jun 24, 2025
@moz-bozden moz-bozden requested review from data-sync-user and removed request for a team June 24, 2025 00:02
Copy link

github-actions bot commented Jun 24, 2025

Size Change: +10 B (0%)

Total Size: 6.87 MB

ℹ️ View Unchanged
Filename Size Change
web/dist/1-1.****************.png 9.92 kB 0 B
web/dist/1-2.****************.png 7.71 kB 0 B
web/dist/1-3.****************.png 14.2 kB 0 B
web/dist/1-4.****************.png 24.5 kB 0 B
web/dist/1-5.****************.png 15.8 kB 0 B
web/dist/1-6.****************.png 12.4 kB 0 B
web/dist/1-red-copy.****************.svg 9.1 kB 0 B
web/dist/171.****************.css 2.39 kB 0 B
web/dist/171.****************.js 5.92 kB 0 B
web/dist/178.****************.js 3.43 kB 0 B
web/dist/178.****************.css 4.34 kB 0 B
web/dist/185.****************.css 2.96 kB 0 B
web/dist/185.****************.js 6.61 kB 0 B
web/dist/2-1.****************.png 7.14 kB 0 B
web/dist/2-2.****************.png 14.2 kB 0 B
web/dist/2-3.****************.png 10.4 kB 0 B
web/dist/2-4.****************.png 9.92 kB 0 B
web/dist/2-5.****************.png 8.49 kB 0 B
web/dist/227.****************.js 629 B 0 B
web/dist/227.****************.css 391 B 0 B
web/dist/275.****************.js 15.8 kB 0 B
web/dist/275.****************.css 9.7 kB 0 B
web/dist/290.****************.js 6.69 kB 0 B
web/dist/303.****************.css 1.34 kB 0 B
web/dist/303.****************.js 1.68 kB 0 B
web/dist/353.****************.js 2.04 kB 0 B
web/dist/353.****************.css 1.29 kB 0 B
web/dist/357.****************.css 4.98 kB 0 B
web/dist/357.****************.js 16.8 kB 0 B
web/dist/397.****************.js 912 B 0 B
web/dist/397.****************.css 1.27 kB 0 B
web/dist/41.****************.css 3.37 kB 0 B
web/dist/41.****************.js 8.65 kB 0 B
web/dist/425.****************.css 3.12 kB 0 B
web/dist/425.****************.js 5.61 kB 0 B
web/dist/457.****************.js 5.52 kB 0 B
web/dist/457.****************.css 3.28 kB 0 B
web/dist/483.****************.js 3.17 kB 0 B
web/dist/483.****************.css 817 B 0 B
web/dist/513.****************.js 8.43 kB 0 B
web/dist/513.****************.css 5.46 kB 0 B
web/dist/616.****************.css 765 B 0 B
web/dist/659.****************.js 8.69 kB 0 B
web/dist/659.****************.css 4.25 kB 0 B
web/dist/752.****************.js 6.46 kB 0 B
web/dist/752.****************.css 2.78 kB 0 B
web/dist/778.****************.js 495 B 0 B
web/dist/778.****************.css 736 B 0 B
web/dist/846.****************.css 2.23 kB 0 B
web/dist/846.****************.js 3.43 kB 0 B
web/dist/909.****************.js 5.41 kB 0 B
web/dist/alert.****************.svg 745 B 0 B
web/dist/award.****************.svg 334 B 0 B
web/dist/background-wave-error.****************.svg 402 B 0 B
web/dist/background-wave-success.****************.svg 550 B 0 B
web/dist/background-wave.****************.svg 284 B 0 B
web/dist/bbc.****************.svg 5.4 kB 0 B
web/dist/challenge-mtn-post.****************.svg 3.3 kB 0 B
web/dist/challenge-mtn-pre.****************.svg 3.36 kB 0 B
web/dist/checkmark-green.****************.svg 219 B 0 B
web/dist/checkmark.****************.svg 313 B 0 B
web/dist/chevron-left.****************.svg 292 B 0 B
web/dist/chevron-right.****************.svg 294 B 0 B
web/dist/chrome-color.****************.svg 1.96 kB 0 B
web/dist/close-black-light.****************.svg 307 B 0 B
web/dist/close-black.****************.svg 820 B 0 B
web/dist/close.****************.svg 145 B 0 B
web/dist/common-voice-mars-neutral.****************.png 126 kB 0 B
web/dist/contact.****************.svg 971 B 0 B
web/dist/contribute.****************.png 600 kB 0 B
web/dist/cv-logo-black.****************.svg 2.33 kB 0 B
web/dist/cv-logo-white.****************.svg 2.85 kB 0 B
web/dist/dashboard.****************.png 955 kB 0 B
web/dist/datasets-intro-background-triangle.****************.svg 446 B 0 B
web/dist/datasets-intro-background.****************.svg 944 B 0 B
web/dist/deepspeech.****************.png 10.7 kB 0 B
web/dist/deepspeech@2x.****************.png 31.6 kB 0 B
web/dist/deepspeech@3x.****************.png 63.2 kB 0 B
web/dist/discord.****************.svg 18.4 kB 0 B
web/dist/discourse.****************.png 16.4 kB 0 B
web/dist/discourse.****************.svg 473 B 0 B
web/dist/discourse@2x.****************.png 44.8 kB 0 B
web/dist/discourse@3x.****************.png 74 kB 0 B
web/dist/donate-banner-desktop-coral.****************.svg 600 B 0 B
web/dist/donate-banner-desktop-pink.****************.svg 600 B 0 B
web/dist/donate-bg1.****************.png 57.8 kB 0 B
web/dist/donate-bg2.****************.png 43.4 kB 0 B
web/dist/down-arrow.****************.svg 399 B 0 B
web/dist/dropdown-arrow.****************.svg 165 B 0 B
web/dist/email-bg-light.****************.png 12.7 kB 0 B
web/dist/email-bg-md-light.****************.svg 919 B 0 B
web/dist/email-bg-md.****************.svg 1.11 kB 0 B
web/dist/email-bg-partnerships.****************.png 287 kB 0 B
web/dist/email-bg.****************.png 4.27 kB 0 B
web/dist/exclamation.****************.svg 534 B 0 B
web/dist/fast-company.****************.svg 19.8 kB 0 B
web/dist/feedback.****************.png 26.4 kB 0 B
web/dist/feedback@2x.****************.png 82.8 kB 0 B
web/dist/feedback@3x.****************.png 269 kB 0 B
web/dist/ff-color.****************.svg 4.21 kB 0 B
web/dist/ff-logo.****************.svg 91.6 kB 0 B
web/dist/financial-times.****************.svg 42.3 kB 0 B
web/dist/github.****************.svg 851 B 0 B
web/dist/globe_black.****************.svg 930 B 0 B
web/dist/grid.****************.svg 387 B 0 B
web/dist/guidelines-waves-footer-small.****************.png 11.8 kB 0 B
web/dist/guidelines-waves-footer.****************.png 154 kB 0 B
web/dist/guidelines-waves.****************.png 225 kB 0 B
web/dist/happy-mars@2x.****************.png 23 kB 0 B
web/dist/hex-done.****************.svg 336 B 0 B
web/dist/hex.****************.svg 300 B 0 B
web/dist/home.****************.svg 506 B 0 B
web/dist/ibm.****************.svg 498 B 0 B
web/dist/idea-light-bulb.****************.svg 285 B 0 B
web/dist/keyboard.****************.svg 354 B 0 B
web/dist/languages/test.json 20 B 0 B
web/dist/lenovo.****************.svg 912 B 0 B
web/dist/librispeech.****************.png 161 B 0 B
web/dist/listen-bg.****************.svg 1.91 kB 0 B
web/dist/mail.****************.svg 998 B 0 B
web/dist/main-9a8b795a.****************.js 41.9 kB +10 B (+0.02%)
web/dist/main-9a8b795a.****************.css 10.5 kB 0 B
web/dist/main-ef7d455c.****************.js 9.66 kB 0 B
web/dist/mars-avatar.****************.svg 1.35 kB 0 B
web/dist/mars-email-success.****************.svg 7.66 kB 0 B
web/dist/mars-request.****************.svg 6.94 kB 0 B
web/dist/mars-sad.****************.svg 6.14 kB 0 B
web/dist/mars-solo.****************.svg 1.92 kB 0 B
web/dist/mars.****************.svg 2.63 kB 0 B
web/dist/mesh.****************.svg 492 B 0 B
web/dist/mic.****************.svg 435 B 0 B
web/dist/MIT-technology-review.****************.svg 14.4 kB 0 B
web/dist/mozilla-common-voice_foundation-academia.****************.png 10.2 kB 0 B
web/dist/mozilla-common-voice_foundation-community.****************.png 10.9 kB 0 B
web/dist/mozilla-common-voice_foundation-corporates.****************.png 11.9 kB 0 B
web/dist/mozilla-common-voice_foundation-foundations.****************.png 12.3 kB 0 B
web/dist/mozilla-common-voice_foundation-governments.****************.png 7.51 kB 0 B
web/dist/mozilla-common-voice_foundation-mars_small.****************.png 24 kB 0 B
web/dist/mozilla-common-voice_foundation-mars.****************.png 49.3 kB 0 B
web/dist/mozilla-common-voice_foundation-small-business.****************.png 10.6 kB 0 B
web/dist/mozilla-logo.****************.svg 880 B 0 B
web/dist/mozilla.****************.svg 468 B 0 B
web/dist/nvidia-logo.****************.svg 23.8 kB 0 B
web/dist/play.****************.svg 2.1 kB 0 B
web/dist/plus.****************.svg 333 B 0 B
web/dist/question-mark.****************.svg 755 B 0 B
web/dist/red-robot.****************.svg 1.1 MB 0 B
web/dist/releases/cv-corpus-1.json 2.91 kB 0 B
web/dist/releases/cv-corpus-10.0-2022-07-04.json 14.5 kB 0 B
web/dist/releases/cv-corpus-10.0-delta-2022-07-04.json 7.49 kB 0 B
web/dist/releases/cv-corpus-11.0-2022-09-21.json 15 kB 0 B
web/dist/releases/cv-corpus-2.json 4.32 kB 0 B
web/dist/releases/cv-corpus-3.json 4.42 kB 0 B
web/dist/releases/cv-corpus-4-2019-12-10.json 5.48 kB 0 B
web/dist/releases/cv-corpus-5-2020-06-22.json 9.77 kB 0 B
web/dist/releases/cv-corpus-5-singleword.json 2.53 kB 0 B
web/dist/releases/cv-corpus-5.1-2020-06-22.json 9.78 kB 0 B
web/dist/releases/cv-corpus-5.1-singleword.json 2.51 kB 0 B
web/dist/releases/cv-corpus-6.0-2020-12-11.json 10.5 kB 0 B
web/dist/releases/cv-corpus-6.0-singleword.json 3.42 kB 0 B
web/dist/releases/cv-corpus-6.1-2020-12-11.json 10.7 kB 0 B
web/dist/releases/cv-corpus-6.1-singleword.json 3.51 kB 0 B
web/dist/releases/cv-corpus-7.0-2021-07-21.json 13.1 kB 0 B
web/dist/releases/cv-corpus-7.0-singleword.json 3.63 kB 0 B
web/dist/releases/cv-corpus-8.0-2022-01-19.json 13.2 kB 0 B
web/dist/releases/cv-corpus-9.0-2022-04-27.json 14.4 kB 0 B
web/dist/review-waves.****************.png 20.4 kB 0 B
web/dist/robot.****************.png 52.1 kB 0 B
web/dist/runtime.****************.js 2.61 kB 0 B
web/dist/safari-color.****************.svg 8.6 kB 0 B
web/dist/sap.****************.svg 842 B 0 B
web/dist/search.****************.svg 432 B 0 B
web/dist/segment-dots.****************.svg 474 B 0 B
web/dist/sodedif.****************.png 1.96 kB 0 B
web/dist/sound-waves-1.****************.png 54.7 kB 0 B
web/dist/sound-waves-2.****************.png 20.9 kB 0 B
web/dist/speak-bg.****************.svg 1.92 kB 0 B
web/dist/star.****************.svg 562 B 0 B
web/dist/stars-disabled.****************.svg 2.06 kB 0 B
web/dist/stars.****************.svg 2.1 kB 0 B
web/dist/success.****************.svg 1.44 kB 0 B
web/dist/support.****************.svg 1.9 kB 0 B
web/dist/table.****************.png 217 kB 0 B
web/dist/tail.****************.svg 357 B 0 B
web/dist/tatoeba.****************.png 21 kB 0 B
web/dist/ted.****************.png 178 B 0 B
web/dist/upload.****************.svg 546 B 0 B
web/dist/vendors-0bc0478e.****************.js 14.8 kB 0 B
web/dist/vendors-27545368.****************.js 4.43 kB 0 B
web/dist/vendors-2a46b080.****************.js 17.9 kB 0 B
web/dist/vendors-2b4841d6.****************.js 16.8 kB 0 B
web/dist/vendors-5178c671.****************.js 3.27 kB 0 B
web/dist/vendors-6ff3e2ec.****************.js 15.9 kB 0 B
web/dist/vendors-866ab763.****************.js 12.8 kB 0 B
web/dist/vendors-94223cbb.****************.js 13 kB 0 B
web/dist/vendors-96700d3a.****************.js 15.8 kB 0 B
web/dist/vendors-a06af057.****************.js 8.48 kB 0 B
web/dist/vendors-acdd0895.****************.js 140 kB 0 B
web/dist/vendors-b403787e.****************.js 19.8 kB 0 B
web/dist/vendors-c092ac97.****************.js 5.1 kB 0 B
web/dist/vendors-cb98e9ec.****************.js 53.7 kB 0 B
web/dist/vendors-cdd60c62.****************.js 7.6 kB 0 B
web/dist/vendors-d17272b1.****************.js 5.32 kB 0 B
web/dist/vendors-d2eb5610.****************.js 13.9 kB 0 B
web/dist/vendors-e4915c4c.****************.js 16.1 kB 0 B
web/dist/vendors-e5bca7e4.****************.js 16.7 kB 0 B
web/dist/vendors-efdee510.****************.js 11.2 kB 0 B
web/dist/vendors-f501a089.****************.js 1.15 kB 0 B
web/dist/venture-beat.****************.svg 61.7 kB 0 B
web/dist/voxforge.****************.png 10.5 kB 0 B
web/dist/wave-1.****************.svg 446 B 0 B
web/dist/wave-2.****************.svg 809 B 0 B
web/dist/wave-3.****************.svg 468 B 0 B
web/dist/wave-blue.****************.svg 4.39 kB 0 B
web/dist/wave-eq.****************.svg 1.03 kB 0 B
web/dist/wave-fading.****************.svg 346 B 0 B
web/dist/wave-grey.****************.svg 1.83 kB 0 B
web/dist/wave-top.****************.png 5.3 kB 0 B
web/dist/wave.****************.png 196 kB 0 B
web/dist/wave.****************.svg 1.04 kB 0 B
web/dist/waves-md.****************.svg 1.65 kB 0 B
web/dist/waves-small.****************.png 35.1 kB 0 B
web/dist/waves.****************.svg 1.78 kB 0 B
web/dist/waves.****************.png 33.4 kB 0 B
web/dist/waves@2x.****************.png 114 kB 0 B
web/dist/waves@3x.****************.png 228 kB 0 B
web/dist/waving-mars-clipped.****************.svg 92.3 kB 0 B
web/dist/waving-mars.****************.svg 92.3 kB 0 B

compressed-size-action

@moz-bozden moz-bozden requested review from moz-dfeller and removed request for data-sync-user June 24, 2025 00:12
@@ -144,37 +141,42 @@ export default class Model {
const languages = await this.db.getAllLanguages()
return languages
},
DAY
TimeUnits.DAY,
5 * TimeUnits.SECOND,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not touch the 3 minute default in many cases, especially as 5 seconds locking time is too small for some endpoints that need more than 10s to resolve. This way you would just spawn new requests for the same data before the initial request has even finished, leading to wasted calculations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More than 3 min is OK for known problem areas, but less than 3 min is probably not worth the potential wasted resource consumption

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only dropped below 3 min when it was obvious (some of them this one I tested on local), but you are right, there might be a temporary congestion resulting in delays. I'll up them to 3 min.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this one :) This 1 minute already seemed too low for me... Up it to 3 min?
What do you think?

  async findClipsWithFewVotes(
    client_id: string,
    locale_id: number,
    count: number,
    exemptFromSSRL?: boolean
  ): Promise<DBClip[]> {
    // get cached clips for given language
    const cachedClips: DBClip[] = await lazyCache(
      `new-clips-per-language-${locale_id}`,
      async () => {
        return await this.getClipsToBeValidated(locale_id, 10000)
      },
      TimeUnits.MINUTE,
      3 * TimeUnits.MINUTE
    )()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 min is the default so you can just omit the parameter here, i.e.

  async findClipsWithFewVotes(
    client_id: string,
    locale_id: number,
    count: number,
    exemptFromSSRL?: boolean
  ): Promise<DBClip[]> {
    // get cached clips for given language
    const cachedClips: DBClip[] = await lazyCache(
      `new-clips-per-language-${locale_id}`,
      async () => {
        return await this.getClipsToBeValidated(locale_id, 10000)
      },
      TimeUnits.MINUTE
    )()

Copy link
Contributor Author

@moz-bozden moz-bozden Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to be explicit on these, in case somebody changes the default.
What I mean is:

  • CacheTime is 1 minute
  • LockTime is 3 minutes

for this particular query. Isn't 1 minute already very small?
This is per-user hit per-locale...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, keeping it explicit is fine. Cache time is low to not have the same clips presented during community events with a couple hundred people participating at the same time.

@moz-bozden moz-bozden merged commit f0c0782 into main Jul 16, 2025
2 checks passed
@moz-bozden moz-bozden deleted the 1177-mcv-be-fix-lazycache-long-lock-issue branch July 16, 2025 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants