Skip to content

Issue with batch operation on room loading process #2068

@windmemory

Description

@windmemory

Recently we've encountered problems related to wechaty/puppet-service#31

From the tests we've done on the issue above, we know that large batch operations in a short period of time will trigger it, which will cause wechaty-puppet-hostie to restart.

Based on our test, comparing with Room.findAll(), Contact.findAll() seldom triggers the problem. Then I dig into the code and found that:

// src/user/contact.ts L224-L245
      const BATCH_SIZE = 16
      let   batchIndex = 0

      const invalidDict: { [id: string]: true } = {}

      while (batchIndex * BATCH_SIZE < contactList.length) {
        const batchContactList = contactList.slice(
          BATCH_SIZE * batchIndex,
          BATCH_SIZE * (batchIndex + 1),
        )
        await Promise.all(
          batchContactList.map(
            c => c.ready()
              .catch(e => {
                log.error('Contact', 'findAll() contact.ready() exception: %s', e.message)
                invalidDict[c.id] = true
              }),
          ),
        )

        batchIndex++
      }

The code in the method Contact.findAll() is doing a pagination call, which will reduce the count of requests made simultaneously, thus avoid the wechaty-puppet-hostie issue.

But the code in Room.findAll() does not have the pagination mechanism. So I think we should add pagination in the method.

We've tested the pagination in our scoped npm package @windmemory/wechaty, and seems all working as expected.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions