Skip to content

1173-MCV Fix: Use Redis to keep recorded "sentence_id"s per user #4944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

moz-bozden
Copy link
Contributor

@moz-bozden moz-bozden commented Jun 19, 2025

This fixes a race condition we debugged.

Explanation

Many new and/or low-resource languages must have seen this: You remember that you already recorded a sentence, you record it again, but when you submit it a pop-up comes out saying it could not be saved with two options: Retry - Cancel.
Even if you retry, you cannot pass it, the only way is to press cancel.
We found out the code is correct, but there is a race condition: The database (DB) updates are slower than selects. Here is what was happening:

  • The front-end (FE) requests 25 sentences for being recorded. The back-end (BE) provides them.
  • The user submits 25 sentences 5 by 5. The first 4 batches go without a problem. After the last 5 (or less remaining) are sent, FE requests a new batch (of 25).
  • While the database is updating the sentence tables incrementing the recorded counts (slow), it also selects new set - but with not-yet-updated values. This is a race condition. So the user gets a random selection of sentences, where some of them might be from the last set of the previous batch. If the text-corpus is small, or the contributing user count is slow, or small number of users record much (which is very much the case for every new/low-resource language) this happens more.
  • The user records that sentence again and it gets rejected - because by rule (to prevent bias) a user can only record a sentence once.

In this fix, we keep the sentence_id's which got recorded - for each user in memory for an hour and filter out from the results, which effectively overcomes the race condition caused by slow DB UPDATE process.

Related issues to be closed

TBA

@moz-bozden moz-bozden self-assigned this Jun 19, 2025
@moz-bozden moz-bozden requested a review from a team as a code owner June 19, 2025 15:15
@moz-bozden moz-bozden added the Bug label Jun 19, 2025
@moz-bozden moz-bozden requested review from moz-dfeller and removed request for a team June 19, 2025 15:15
Copy link

github-actions bot commented Jun 19, 2025

Size Change: +14 B (0%)

Total Size: 6.87 MB

ℹ️ View Unchanged
Filename Size Change
web/dist/1-1.****************.png 9.92 kB 0 B
web/dist/1-2.****************.png 7.71 kB 0 B
web/dist/1-3.****************.png 14.2 kB 0 B
web/dist/1-4.****************.png 24.5 kB 0 B
web/dist/1-5.****************.png 15.8 kB 0 B
web/dist/1-6.****************.png 12.4 kB 0 B
web/dist/1-red-copy.****************.svg 9.1 kB 0 B
web/dist/171.****************.css 2.39 kB 0 B
web/dist/171.****************.js 5.92 kB 0 B
web/dist/178.****************.js 3.43 kB 0 B
web/dist/178.****************.css 4.34 kB 0 B
web/dist/185.****************.css 2.96 kB 0 B
web/dist/185.****************.js 6.61 kB 0 B
web/dist/2-1.****************.png 7.14 kB 0 B
web/dist/2-2.****************.png 14.2 kB 0 B
web/dist/2-3.****************.png 10.4 kB 0 B
web/dist/2-4.****************.png 9.92 kB 0 B
web/dist/2-5.****************.png 8.49 kB 0 B
web/dist/227.****************.js 629 B 0 B
web/dist/227.****************.css 391 B 0 B
web/dist/275.****************.css 9.7 kB 0 B
web/dist/275.****************.js 15.8 kB 0 B
web/dist/290.****************.js 6.69 kB 0 B
web/dist/303.****************.css 1.34 kB 0 B
web/dist/303.****************.js 1.68 kB 0 B
web/dist/353.****************.js 2.04 kB 0 B
web/dist/353.****************.css 1.29 kB 0 B
web/dist/357.****************.css 4.98 kB 0 B
web/dist/357.****************.js 16.8 kB 0 B
web/dist/397.****************.js 912 B 0 B
web/dist/397.****************.css 1.27 kB 0 B
web/dist/41.****************.css 3.37 kB 0 B
web/dist/41.****************.js 8.65 kB 0 B
web/dist/425.****************.css 3.12 kB 0 B
web/dist/425.****************.js 5.61 kB 0 B
web/dist/457.****************.js 5.52 kB 0 B
web/dist/457.****************.css 3.28 kB 0 B
web/dist/483.****************.js 3.17 kB 0 B
web/dist/483.****************.css 817 B 0 B
web/dist/513.****************.js 8.43 kB 0 B
web/dist/513.****************.css 5.47 kB +9 B (+0.16%)
web/dist/616.****************.css 765 B 0 B
web/dist/659.****************.js 8.69 kB 0 B
web/dist/659.****************.css 4.25 kB 0 B
web/dist/752.****************.js 6.46 kB 0 B
web/dist/752.****************.css 2.78 kB 0 B
web/dist/778.****************.js 495 B 0 B
web/dist/778.****************.css 736 B 0 B
web/dist/846.****************.css 2.23 kB 0 B
web/dist/846.****************.js 3.43 kB 0 B
web/dist/909.****************.js 5.41 kB 0 B
web/dist/alert.****************.svg 745 B 0 B
web/dist/award.****************.svg 334 B 0 B
web/dist/background-wave-error.****************.svg 402 B 0 B
web/dist/background-wave-success.****************.svg 550 B 0 B
web/dist/background-wave.****************.svg 284 B 0 B
web/dist/bbc.****************.svg 5.4 kB 0 B
web/dist/challenge-mtn-post.****************.svg 3.3 kB 0 B
web/dist/challenge-mtn-pre.****************.svg 3.36 kB 0 B
web/dist/checkmark-green.****************.svg 219 B 0 B
web/dist/checkmark.****************.svg 313 B 0 B
web/dist/chevron-left.****************.svg 292 B 0 B
web/dist/chevron-right.****************.svg 294 B 0 B
web/dist/chrome-color.****************.svg 1.96 kB 0 B
web/dist/close-black-light.****************.svg 307 B 0 B
web/dist/close-black.****************.svg 820 B 0 B
web/dist/close.****************.svg 145 B 0 B
web/dist/common-voice-mars-neutral.****************.png 126 kB 0 B
web/dist/contact.****************.svg 971 B 0 B
web/dist/contribute.****************.png 600 kB 0 B
web/dist/cv-logo-black.****************.svg 2.33 kB 0 B
web/dist/cv-logo-white.****************.svg 2.85 kB 0 B
web/dist/dashboard.****************.png 955 kB 0 B
web/dist/datasets-intro-background-triangle.****************.svg 446 B 0 B
web/dist/datasets-intro-background.****************.svg 944 B 0 B
web/dist/deepspeech.****************.png 10.7 kB 0 B
web/dist/deepspeech@2x.****************.png 31.6 kB 0 B
web/dist/deepspeech@3x.****************.png 63.2 kB 0 B
web/dist/discord.****************.svg 18.4 kB 0 B
web/dist/discourse.****************.png 16.4 kB 0 B
web/dist/discourse.****************.svg 473 B 0 B
web/dist/discourse@2x.****************.png 44.8 kB 0 B
web/dist/discourse@3x.****************.png 74 kB 0 B
web/dist/donate-banner-desktop-coral.****************.svg 600 B 0 B
web/dist/donate-banner-desktop-pink.****************.svg 600 B 0 B
web/dist/donate-bg1.****************.png 57.8 kB 0 B
web/dist/donate-bg2.****************.png 43.4 kB 0 B
web/dist/down-arrow.****************.svg 399 B 0 B
web/dist/dropdown-arrow.****************.svg 165 B 0 B
web/dist/email-bg-light.****************.png 12.7 kB 0 B
web/dist/email-bg-md-light.****************.svg 919 B 0 B
web/dist/email-bg-md.****************.svg 1.11 kB 0 B
web/dist/email-bg-partnerships.****************.png 287 kB 0 B
web/dist/email-bg.****************.png 4.27 kB 0 B
web/dist/exclamation.****************.svg 534 B 0 B
web/dist/fast-company.****************.svg 19.8 kB 0 B
web/dist/feedback.****************.png 26.4 kB 0 B
web/dist/feedback@2x.****************.png 82.8 kB 0 B
web/dist/feedback@3x.****************.png 269 kB 0 B
web/dist/ff-color.****************.svg 4.21 kB 0 B
web/dist/ff-logo.****************.svg 91.6 kB 0 B
web/dist/financial-times.****************.svg 42.3 kB 0 B
web/dist/github.****************.svg 851 B 0 B
web/dist/globe_black.****************.svg 930 B 0 B
web/dist/grid.****************.svg 387 B 0 B
web/dist/guidelines-waves-footer-small.****************.png 11.8 kB 0 B
web/dist/guidelines-waves-footer.****************.png 154 kB 0 B
web/dist/guidelines-waves.****************.png 225 kB 0 B
web/dist/happy-mars@2x.****************.png 23 kB 0 B
web/dist/hex-done.****************.svg 336 B 0 B
web/dist/hex.****************.svg 300 B 0 B
web/dist/home.****************.svg 506 B 0 B
web/dist/ibm.****************.svg 498 B 0 B
web/dist/idea-light-bulb.****************.svg 285 B 0 B
web/dist/keyboard.****************.svg 354 B 0 B
web/dist/languages/test.json 20 B 0 B
web/dist/lenovo.****************.svg 912 B 0 B
web/dist/librispeech.****************.png 161 B 0 B
web/dist/listen-bg.****************.svg 1.91 kB 0 B
web/dist/mail.****************.svg 998 B 0 B
web/dist/main-9a8b795a.****************.js 41.9 kB 0 B
web/dist/main-9a8b795a.****************.css 10.5 kB +5 B (+0.05%)
web/dist/main-ef7d455c.****************.js 9.65 kB 0 B
web/dist/mars-avatar.****************.svg 1.35 kB 0 B
web/dist/mars-email-success.****************.svg 7.66 kB 0 B
web/dist/mars-request.****************.svg 6.94 kB 0 B
web/dist/mars-sad.****************.svg 6.14 kB 0 B
web/dist/mars-solo.****************.svg 1.92 kB 0 B
web/dist/mars.****************.svg 2.63 kB 0 B
web/dist/mesh.****************.svg 492 B 0 B
web/dist/mic.****************.svg 435 B 0 B
web/dist/MIT-technology-review.****************.svg 14.4 kB 0 B
web/dist/mozilla-common-voice_foundation-academia.****************.png 10.2 kB 0 B
web/dist/mozilla-common-voice_foundation-community.****************.png 10.9 kB 0 B
web/dist/mozilla-common-voice_foundation-corporates.****************.png 11.9 kB 0 B
web/dist/mozilla-common-voice_foundation-foundations.****************.png 12.3 kB 0 B
web/dist/mozilla-common-voice_foundation-governments.****************.png 7.51 kB 0 B
web/dist/mozilla-common-voice_foundation-mars_small.****************.png 24 kB 0 B
web/dist/mozilla-common-voice_foundation-mars.****************.png 49.3 kB 0 B
web/dist/mozilla-common-voice_foundation-small-business.****************.png 10.6 kB 0 B
web/dist/mozilla-logo.****************.svg 880 B 0 B
web/dist/mozilla.****************.svg 468 B 0 B
web/dist/nvidia-logo.****************.svg 23.8 kB 0 B
web/dist/play.****************.svg 2.1 kB 0 B
web/dist/plus.****************.svg 333 B 0 B
web/dist/question-mark.****************.svg 755 B 0 B
web/dist/red-robot.****************.svg 1.1 MB 0 B
web/dist/releases/cv-corpus-1.json 2.91 kB 0 B
web/dist/releases/cv-corpus-10.0-2022-07-04.json 14.5 kB 0 B
web/dist/releases/cv-corpus-10.0-delta-2022-07-04.json 7.49 kB 0 B
web/dist/releases/cv-corpus-11.0-2022-09-21.json 15 kB 0 B
web/dist/releases/cv-corpus-2.json 4.32 kB 0 B
web/dist/releases/cv-corpus-3.json 4.42 kB 0 B
web/dist/releases/cv-corpus-4-2019-12-10.json 5.48 kB 0 B
web/dist/releases/cv-corpus-5-2020-06-22.json 9.77 kB 0 B
web/dist/releases/cv-corpus-5-singleword.json 2.53 kB 0 B
web/dist/releases/cv-corpus-5.1-2020-06-22.json 9.78 kB 0 B
web/dist/releases/cv-corpus-5.1-singleword.json 2.51 kB 0 B
web/dist/releases/cv-corpus-6.0-2020-12-11.json 10.5 kB 0 B
web/dist/releases/cv-corpus-6.0-singleword.json 3.42 kB 0 B
web/dist/releases/cv-corpus-6.1-2020-12-11.json 10.7 kB 0 B
web/dist/releases/cv-corpus-6.1-singleword.json 3.51 kB 0 B
web/dist/releases/cv-corpus-7.0-2021-07-21.json 13.1 kB 0 B
web/dist/releases/cv-corpus-7.0-singleword.json 3.63 kB 0 B
web/dist/releases/cv-corpus-8.0-2022-01-19.json 13.2 kB 0 B
web/dist/releases/cv-corpus-9.0-2022-04-27.json 14.4 kB 0 B
web/dist/review-waves.****************.png 20.4 kB 0 B
web/dist/robot.****************.png 52.1 kB 0 B
web/dist/runtime.****************.js 2.61 kB 0 B
web/dist/safari-color.****************.svg 8.6 kB 0 B
web/dist/sap.****************.svg 842 B 0 B
web/dist/search.****************.svg 432 B 0 B
web/dist/segment-dots.****************.svg 474 B 0 B
web/dist/sodedif.****************.png 1.96 kB 0 B
web/dist/sound-waves-1.****************.png 54.7 kB 0 B
web/dist/sound-waves-2.****************.png 20.9 kB 0 B
web/dist/speak-bg.****************.svg 1.92 kB 0 B
web/dist/star.****************.svg 562 B 0 B
web/dist/stars-disabled.****************.svg 2.06 kB 0 B
web/dist/stars.****************.svg 2.1 kB 0 B
web/dist/success.****************.svg 1.44 kB 0 B
web/dist/support.****************.svg 1.9 kB 0 B
web/dist/table.****************.png 217 kB 0 B
web/dist/tail.****************.svg 357 B 0 B
web/dist/tatoeba.****************.png 21 kB 0 B
web/dist/ted.****************.png 178 B 0 B
web/dist/upload.****************.svg 546 B 0 B
web/dist/vendors-0bc0478e.****************.js 14.8 kB 0 B
web/dist/vendors-27545368.****************.js 4.43 kB 0 B
web/dist/vendors-2a46b080.****************.js 17.9 kB 0 B
web/dist/vendors-2b4841d6.****************.js 16.8 kB 0 B
web/dist/vendors-5178c671.****************.js 3.27 kB 0 B
web/dist/vendors-6ff3e2ec.****************.js 15.9 kB 0 B
web/dist/vendors-866ab763.****************.js 12.8 kB 0 B
web/dist/vendors-94223cbb.****************.js 13 kB 0 B
web/dist/vendors-96700d3a.****************.js 15.8 kB 0 B
web/dist/vendors-a06af057.****************.js 8.48 kB 0 B
web/dist/vendors-acdd0895.****************.js 140 kB 0 B
web/dist/vendors-b403787e.****************.js 19.8 kB 0 B
web/dist/vendors-c092ac97.****************.js 5.1 kB 0 B
web/dist/vendors-cb98e9ec.****************.js 53.7 kB 0 B
web/dist/vendors-cdd60c62.****************.js 7.6 kB 0 B
web/dist/vendors-d17272b1.****************.js 5.32 kB 0 B
web/dist/vendors-d2eb5610.****************.js 13.9 kB 0 B
web/dist/vendors-e4915c4c.****************.js 16.1 kB 0 B
web/dist/vendors-e5bca7e4.****************.js 16.7 kB 0 B
web/dist/vendors-efdee510.****************.js 11.2 kB 0 B
web/dist/vendors-f501a089.****************.js 1.15 kB 0 B
web/dist/venture-beat.****************.svg 61.7 kB 0 B
web/dist/voxforge.****************.png 10.5 kB 0 B
web/dist/wave-1.****************.svg 446 B 0 B
web/dist/wave-2.****************.svg 809 B 0 B
web/dist/wave-3.****************.svg 468 B 0 B
web/dist/wave-blue.****************.svg 4.39 kB 0 B
web/dist/wave-eq.****************.svg 1.03 kB 0 B
web/dist/wave-fading.****************.svg 346 B 0 B
web/dist/wave-grey.****************.svg 1.83 kB 0 B
web/dist/wave-top.****************.png 5.3 kB 0 B
web/dist/wave.****************.png 196 kB 0 B
web/dist/wave.****************.svg 1.04 kB 0 B
web/dist/waves-md.****************.svg 1.65 kB 0 B
web/dist/waves-small.****************.png 35.1 kB 0 B
web/dist/waves.****************.svg 1.78 kB 0 B
web/dist/waves.****************.png 33.4 kB 0 B
web/dist/waves@2x.****************.png 114 kB 0 B
web/dist/waves@3x.****************.png 228 kB 0 B
web/dist/waving-mars-clipped.****************.svg 92.3 kB 0 B
web/dist/waving-mars.****************.svg 92.3 kB 0 B

compressed-size-action

@moz-bozden moz-bozden requested a review from a team June 19, 2025 15:28
@moz-bozden moz-bozden requested a review from moz-dfeller June 19, 2025 19:12
@moz-bozden moz-bozden merged commit b914859 into main Jun 19, 2025
2 checks passed
@moz-bozden moz-bozden deleted the 1173-mcv-fix-use-be-redis-to-keep-recorded-sentence-ids-per-user branch June 19, 2025 20:14
Copy link

sentry-io bot commented Jun 26, 2025

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants