Thirdparty TTK Test-Collection is not repeatable #2717
Description
Summary:
Thirdparty TTK Test-Collection is not repeatable, the following result will occur if helm test command is run a second time:
--------------------FINAL REPORT--------------------
Test Suite:Thirdparty Tests
Environment:Development
┌───────────────────────────────────────────────────┐
│ SUMMARY │
├───────────────────┬───────────────────────────────┤
│ Total assertions │ 21 │
├───────────────────┼───────────────────────────────┤
│ Passed assertions │ 16 │
├───────────────────┼───────────────────────────────┤
│ Failed assertions │ 5 │
├───────────────────┼───────────────────────────────┤
│ Total requests │ 20 │
├───────────────────┼───────────────────────────────┤
│ Total test cases │ 8 │
├───────────────────┼───────────────────────────────┤
│ Passed percentage │ 76.19% │
├───────────────────┼───────────────────────────────┤
│ Started time │ Fri, 11 Mar 2022 09:53:29 GMT │
├───────────────────┼───────────────────────────────┤
│ Completed time │ Fri, 11 Mar 2022 09:53:31 GMT │
├───────────────────┼───────────────────────────────┤
│ Runtime duration │ 2809 ms │
└───────────────────┴───────────────────────────────┘
Workarounds:
1. The following manual steps can be followed to clear the necessary persistent data to re-run the test successfully again:
1.1 Account Linking - Happy Path OTP
1.1.1 PISP Cache
del c51ec534-ee48-4575-b6a9-ead2955b8069
1.1.2 Consent & Auth DB
delete from Consent where id='46876aac-5db8-4353-bb3c-a6a905843ce7';1.2. Third Party Transaction Request - Happy Path
1.2.1 PISP Cache
del b5d6206c-4f06-497d-af15-ed866ea6958f
del c2470148-1be2-4c0b-aece-aa8dcb92a6cc
1.2.2 Consent & Auth DB
delete from Consent where id='2acf1dfa-ce45-486e-b19e-ae4ad9804a63';1.3. All keys added to PISP Cache from running tests for completenes
- "c51ec534-ee48-4575-b6a9-ead2955b8069"
- "callback_329294234"
- "b51ec534-ee48-4575-b6a9-ead2955b8069"
- "b5d6206c-4f06-497d-af15-ed866ea6958f"
- "d51ec534-ee48-4575-b6a9-ead2955b8069"
- "c2470148-1be2-4c0b-aece-aa8dcb92a6cc"
- "key-parties-MSISDN-329294234-undefined"
2. Alternatively, re-run helm test command by excluding the Thirdparty tests as follows
❯ helm test moja3 --filter !name=ml-ttk-test-val-tp,!name=ml-ttk-test-setup-tpSeverity:
Low
Priority:
Low
Expected Behavior
Thirdparty TTK Test-Collection should be repeatable.
Steps to Reproduce
- Run Thirdparty Provisioning collection
- Run Thirdparty Test collection
- Run Thirdparty Test collection again and compare results
Specifications
- Component (if known): Thirdparty Test Collection
- Version: v13.1.0
- Platform: Testing-Toolkit
- Subsystem: n/a
- Type of testing: n/a
- Bug found/raised by: @mdebarros
Notes:
- Severity when opened: low
- Priority when opened: low