-
Notifications
You must be signed in to change notification settings - Fork 97
Fix message API request bodies and session target acceptance #6669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix message API request bodies and session target acceptance #6669
Conversation
📝 Walkthrough📝 WalkthroughWalkthroughThe changes in this pull request involve updates to the Changes
Assessment against linked issues
Possibly related PRs
Suggested labels
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
0463510
to
9485831
Compare
@esterlus pls verify |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (4)
scripts/local-cluster-hoprd.cfg.yaml (1)
14-15
: Document the new network_options configurationThe purpose and impact of the
ignore_timeframe
parameter should be documented. Consider adding comments in the YAML file or updating the configuration documentation to explain:
- What timeframes are being ignored
- When this option should be used
- The implications of different values
- Whether this is a boolean or numeric parameter
db/sql/src/db.rs (2)
Line range hint
28-28
: Consider making the peer cleanup constant privateThe constant
HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS
is marked as internal in its name but is publicly accessible. Consider making it private if it's only used within this module.-pub const HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS: u64 = 5 * 60; // 5 minutes - +const HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS: u64 = 5 * 60; // 5 minutes
Line range hint
246-350
: Consider adding connection pool stress testsWhile the current test coverage is good for peer retention logic, consider adding tests for:
- Connection pool exhaustion scenarios
- Connection timeout handling
- Concurrent database operations under high load
hoprd/rest-api/src/messages.rs (1)
539-539
: Consider adding API migration guideWith multiple breaking changes in the message endpoints, it would be helpful to provide a migration guide for API consumers.
Consider:
- Document the changes in the API changelog
- Provide example requests for both old and new formats
- Add a deprecation timeline for the old endpoints
- Include migration scripts or code snippets for common client implementations
Also applies to: 590-590, 642-642
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
⛔ Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
📒 Files selected for processing (5)
db/sql/src/db.rs
(1 hunks)hoprd/rest-api/Cargo.toml
(1 hunks)hoprd/rest-api/src/messages.rs
(3 hunks)hoprd/rest-api/src/session.rs
(3 hunks)scripts/local-cluster-hoprd.cfg.yaml
(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- hoprd/rest-api/Cargo.toml
🔇 Additional comments (7)
scripts/local-cluster-hoprd.cfg.yaml (1)
14-15
: Verify security implications of timeframe ignoring
The ability to ignore timeframes could have security implications for message processing. Please confirm:
- What security measures are in place when timeframes are ignored
- Whether this could enable replay attacks
- If this should be restricted in production environments
Let's check for any security-related code connected to this parameter:
db/sql/src/db.rs (2)
180-180
: LGTM: Improved logging with structured format
The change to structured logging enhances observability by explicitly including the number of affected rows.
Line range hint 73-77
: Verify connection pool settings and add monitoring
The peers database has a high connection limit (300) and long acquire timeout (60s). While this might be necessary for your use case, it could also mask underlying connection management issues.
Consider:
- Adding connection pool metrics monitoring
- Setting up alerts for connection pool saturation
- Implementing connection leak detection
Let's check for potential connection leaks:
✅ Verification successful
Based on the search results, I can now provide a final response regarding the connection pool settings:
Connection pool settings are well-configured with appropriate limits and timeouts
The database connection pools are configured with reasonable settings that align with common practices:
- Index DB: 30 max connections, suitable for index operations
- Peers DB: 300 max connections with 60s acquire timeout, justified by:
- Includes proper idle timeout (10 mins) to prevent connection leaks
- Has max lifetime setting (30 mins) to ensure connection recycling
- Higher limit is appropriate for peer management which typically requires more concurrent connections
- Tickets DB: 50 max connections, appropriate for ticket operations
- All pools have min_connections=0 which allows efficient resource usage
The high connection limit and timeout for peers database is justified by its use case and includes proper connection lifecycle management to prevent leaks.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for database connection usage patterns
# Look for places where connections might not be properly released
# Check for direct connection usage without proper cleanup
rg -A 3 "peers_db\."
# Look for transaction patterns
ast-grep --pattern 'transaction($$$).await'
Length of output: 440
Script:
#!/bin/bash
# Let's search for database connection and transaction patterns more comprehensively
# Look for database connection pool configurations
rg -A 5 "PoolOptions::new"
# Search for transaction usage patterns
rg -A 3 "\.begin\(\)|\.begin_with_config\(\)"
# Look for connection acquire timeout settings
rg "acquire_timeout|max_connections"
# Check for any explicit connection management
rg -A 3 "\.acquire\(\)|\.close\(\)"
Length of output: 100210
hoprd/rest-api/src/messages.rs (2)
539-539
:
API Breaking Change: pop
endpoint now accepts JSON body instead of query parameters
This change aligns with the PR objectives but introduces a breaking change in the API contract. Existing clients using query parameters will need to be updated.
Let's verify the API documentation and potential impact:
590-590
:
API Breaking Change: pop-all
endpoint now accepts JSON body instead of query parameters
This change fixes the regression reported in issue #6653 where the tag parameter was not being respected. However, it's a breaking change that requires client updates.
Let's verify the implementation across the codebase:
✅ Verification successful
Based on the evidence gathered from the codebase:
API Change Confirmed: pop-all
endpoint now accepts JSON body
The OpenAPI documentation in the code confirms this is an intentional change to a POST endpoint that accepts a JSON body. The change appears to be properly documented with clear request/response schemas and descriptions.
- The endpoint is documented to accept a
TagQueryRequest
body with clear description of its behavior - No existing client implementations were found in the codebase that would be affected
- The change fixes a reported issue (#6653) with tag parameter handling
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check for any integration tests that might need updating
rg -A 5 "messages/pop-all\?tag="
# Look for client implementations that might be affected
ast-grep --pattern 'fetch($_, "messages/pop-all")'
Length of output: 415
Script:
#!/bin/bash
# Let's try different approaches to find relevant code
# Search for API route definition
rg -l "pop-all" --type rust
# Look for test files that might be using this endpoint
fd -e rs test
Length of output: 39421
Script:
#!/bin/bash
# Let's check the specific file and surrounding context for the API change
rg -A 10 -B 10 "pop-all" hoprd/rest-api/src/messages.rs
# Also check for OpenAPI/Swagger documentation that might need updating
fd -e yaml swagger
fd -e yaml openapi
Length of output: 1129
Script:
#!/bin/bash
# Let's check the git history for this specific change
rg -l "messages/pop-all" hoprd/rest-api/src/messages.rs | xargs git log -p -- | rg -B 5 -A 5 "pop-all"
# Also check for any client implementations
rg -A 5 "messages/pop-all" --type typescript --type javascript
Length of output: 97
hoprd/rest-api/src/session.rs (2)
74-82
: LGTM! Clean refactoring of from_str
implementation.
The simplified implementation improves readability while maintaining the same functionality. The error handling for base64 decoding is properly maintained.
115-117
: Breaking change: target
field is now required.
The change improves API reliability by ensuring the target is always specified. This aligns with the PR objectives.
Let's verify that all clients are prepared for this breaking change:
✅ Verification successful
Let me gather more information about the client usage by checking for test files and any client implementations.
Let me check the test files specifically for websocket client usage patterns.
Based on the test file content, I can now generate the final response as we have sufficient information about client usage.
Breaking change confirmed: target
field is now required and clients are prepared.
The test file shows that clients are already providing the target
parameter in their requests:
- Test
test_websocket_send_receive_messages
uses the new API withtarget
parameter - The older deprecated tests use a different endpoint
/api/v3/messages/websocket
- All current tests using the new endpoint already include the target parameter in their requests
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Search for client code that might be affected by the breaking change
# Look for SessionWebsocketClientQueryRequest usage
rg -A 5 "SessionWebsocketClientQueryRequest"
# Look for websocket endpoint calls that might need updating
rg -A 5 "/api/v3/session/websocket"
Length of output: 3310
Script:
#!/bin/bash
# Look for test files that might show how the websocket endpoint is used
fd -e py -e rs test
-x rg -A 5 "session/websocket" {}
# Look for any client implementations or examples
rg -A 5 "websocket\?.*target"
Length of output: 40396
Script:
#!/bin/bash
# Look for websocket test files and their contents
cat tests/test_websocket_api.py
Length of output: 8993
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$ curl -s -X POST -H "Content-Type: application/json" -H "x-auth-token: ^^LOCAL-testing-123^^" -d '{"tag":2001}' http://127.0.0.1:19091/api/v3/messages/pop-all | jq
{
"messages": [
{
"tag": 2000,
"body": "test0",
"receivedAt": 1732277432917
},
{
"tag": 2000,
"body": "test0",
"receivedAt": 1732277435466
},
{
"tag": 2000,
"body": "test0",
"receivedAt": 1732277436187
},
{
"tag": 2001,
"body": "test0",
"receivedAt": 1732277488641
}
]
}
pop-all still popping all messages regardless of tag
Oh wait, no my bad |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pop-all / peek-all work as expected
websocket session also works now
great 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (1)
tests/test_websocket_api.py (1)
Line range hint
1-256
: Enhance test coverage for PR changesThe current test suite needs additional coverage for the changes mentioned in PR objectives:
Add tests to verify JSON payload acceptance for
/messages/
endpoints:
- Test
pop
,pop_all
, andpeek
with both JSON payloads and query parameters- Verify correct error responses for invalid payload formats
Add specific tests for the message tag regression (issue #6653):
- Test
pop_all
with tagged messages- Verify only messages with matching tags are returned
- Test edge cases with multiple tags
Consider moving the tag-related test logic from the deprecated section to new tests, as it provides better coverage.
Example test structure:
@pytest.mark.parametrize("endpoint", ["pop", "pop_all", "peek"]) async def test_message_endpoint_payload(endpoint): # Test JSON payload acceptance # Test query parameter rejection pass async def test_pop_all_with_tags(): # Test tag filtering # Test multiple tags # Test tag regression fix pass
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
📒 Files selected for processing (1)
tests/test_websocket_api.py
(1 hunks)
🔇 Additional comments (2)
tests/test_websocket_api.py (2)
Line range hint 190-256
: Preserve test coverage while deprecating old tests
Before removing the deprecated tests, ensure their valuable test coverage is migrated:
- The tag handling in
test_websocket_send_receive_messages_DEPRECATED
is more thorough than current tests - The multiple connection handling in
ws_connections
fixture provides better concurrent testing - The message format validation is more complete in the deprecated version
Consider extracting these test patterns into new tests before removing the deprecated code.
Here's a script to analyze test coverage differences:
24-25
: 🛠️ Refactor suggestion
Add tests to verify target field requirement
While adding the target parameter aligns with making it mandatory, consider:
- The hardcoded target "127.0.0.1:4677" might not be suitable for all test environments. Consider using a dynamic port from PORT_BASE.
- Add test cases to verify:
- Connection rejection when target is missing
- Connection rejection with invalid target format
- Connection success with various valid target formats
Here's a script to verify if we're missing any target-related test cases:
ec1b951
to
becefb6
Compare
@ausias-armesto also relevant for #6640 as it slightly changes the websocket session interface to always require the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 3
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
⛔ Files ignored due to path filters (1)
Cargo.lock
is excluded by!**/*.lock
📒 Files selected for processing (6)
db/sql/src/db.rs
(1 hunks)hoprd/rest-api/Cargo.toml
(1 hunks)hoprd/rest-api/src/messages.rs
(3 hunks)hoprd/rest-api/src/session.rs
(3 hunks)scripts/local-cluster-hoprd.cfg.yaml
(1 hunks)tests/test_websocket_api.py
(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- db/sql/src/db.rs
- hoprd/rest-api/Cargo.toml
- scripts/local-cluster-hoprd.cfg.yaml
🔇 Additional comments (4)
hoprd/rest-api/src/messages.rs (3)
642-642
: Update API documentation for breaking changes
The change to JSON body is consistent with other endpoints. However, ensure that the API documentation and changelog clearly communicate these breaking changes to clients.
Consider adding a migration guide in the README or API documentation to help users transition from query parameters to JSON body.
539-539
:
API Breaking Change: Consider versioning the endpoint
The change from query parameters to JSON body maintains consistency with other endpoints but introduces a breaking change. Consider versioning the endpoint (e.g., /v2/messages/pop
) to ensure backward compatibility.
590-590
: LGTM: Addresses tag parameter handling issue
The change to JSON body handling fixes the regression in tag parameter processing (issue #6653). The implementation correctly validates and processes the tag parameter.
✅ Verification successful
Change in pop_all
endpoint is consistent with other message endpoints
The verification shows that the codebase has a consistent pattern for tag parameter handling:
- GET endpoints (
size
,delete_messages
) use Query parameters - POST endpoints (
pop
,pop_all
,peek
) use JSON body - All endpoints use the same
TagQueryRequest
struct and validation logic for tags
The change from Query
to Json
in pop_all
aligns with this pattern, making the API more consistent.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Verify consistent tag parameter handling across message endpoints
rg -A 3 'if let Some\(tag\) = tag.*RESERVED_TAG_UPPER_LIMIT'
Length of output: 62
Script:
#!/bin/bash
# Search for tag validation patterns in messages.rs
rg -A 5 'tag: Option<u32>' hoprd/rest-api/src/messages.rs
# Search for tag parameter handling in message endpoints
ast-grep --pattern 'struct TagQueryRequest {
$$$
}'
# Look for tag validation logic
rg -A 5 'tag.*RESERVED_TAG' hoprd/rest-api/src/messages.rs
Length of output: 1526
Script:
#!/bin/bash
# Search for all TagQueryRequest usages in messages.rs
rg -B 3 -A 3 'TagQueryRequest' hoprd/rest-api/src/messages.rs
# Look for the endpoint handler signatures
ast-grep --pattern 'async fn $_($$$) -> $$ {
$$$
}'
Length of output: 3178
hoprd/rest-api/src/session.rs (1)
74-82
: LGTM! Clean refactoring of the from_str
implementation.
The refactored code is more idiomatic and easier to follow, with improved error handling through a single return path.
Fixes to the REST API behavior:
/messages/
endpointstarget
for session creation in websocket mandatoryDetails
Logging improvements:
db/sql/src/db.rs
: Updated the debug log statement to use structured logging for the number of rows affected.API request handling:
hoprd/rest-api/src/messages.rs
: Changed the request parameter type fromQuery
toJson
body for thepop
,pop_all
, andpeek
functions to handle JSON payloads. [1] [2] [3]Session target specification:
hoprd/rest-api/src/session.rs
: Simplified thefrom_str
implementation forSessionTargetSpec
to return the result directly.hoprd/rest-api/src/session.rs
: ChangedSessionWebsocketClientQueryRequest
to maketarget
a required field and updated the related implementation to handle this change. [1] [2]Configuration settings:
scripts/local-cluster-hoprd.cfg.yaml
: Addednetwork_options
with anignore_timeframe
setting to the configuration file.Notes
Fixes #6653