Skip to content

Conversation

Teebor-Choka
Copy link
Contributor

@Teebor-Choka Teebor-Choka commented Nov 22, 2024

Fixes to the REST API behavior:

  • Fix the incorrect body/query acceptance for some /messages/ endpoints
  • Make target for session creation in websocket mandatory

Details

Logging improvements:

  • db/sql/src/db.rs: Updated the debug log statement to use structured logging for the number of rows affected.

API request handling:

Session target specification:

Configuration settings:

Notes

Fixes #6653

@Teebor-Choka Teebor-Choka added this to the 2.2.0-rc.1 milestone Nov 22, 2024
@Teebor-Choka Teebor-Choka requested a review from a team November 22, 2024 11:48
@Teebor-Choka Teebor-Choka self-assigned this Nov 22, 2024
Copy link
Contributor

coderabbitai bot commented Nov 22, 2024

📝 Walkthrough
📝 Walkthrough

Walkthrough

The changes in this pull request involve updates to the HoprDb struct and its methods in the database module, enhancements to the API request handling by switching from query parameters to JSON payloads, and modifications to session handling. Additionally, a new configuration option has been introduced in the YAML file. The updates focus on improving logging, error handling, and the clarity of request processing, while also incrementing the version number of the hoprd-api package.

Changes

File Change Summary
db/sql/src/db.rs Updated HoprDb struct methods for SQLite connection settings, logging improvements, and error handling.
hoprd/rest-api/Cargo.toml Version number updated from "3.7.0" to "3.7.1".
hoprd/rest-api/src/messages.rs Changed request body type from Query to Json for pop, pop_all, and peek functions.
hoprd/rest-api/src/session.rs Refactored from_str method in SessionTargetSpec and made target field mandatory in SessionWebsocketClientQueryRequest.
scripts/local-cluster-hoprd.cfg.yaml Added new configuration option network_options.ignore_timeframe set to 0.

Assessment against linked issues

Objective Addressed Explanation
Ensure pop-all messages adhere to provided tag parameter (#6653)

Possibly related PRs

Suggested labels

crate:chain-indexer, crate:core-network

Suggested reviewers

  • NumberFour8

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@Teebor-Choka Teebor-Choka force-pushed the kauki/rest-api/fix-bugs-related-to-messages-and-sessions branch from 0463510 to 9485831 Compare November 22, 2024 11:52
@Teebor-Choka Teebor-Choka marked this pull request as ready for review November 22, 2024 11:56
@Teebor-Choka
Copy link
Contributor Author

@esterlus pls verify

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (4)
scripts/local-cluster-hoprd.cfg.yaml (1)

14-15: Document the new network_options configuration

The purpose and impact of the ignore_timeframe parameter should be documented. Consider adding comments in the YAML file or updating the configuration documentation to explain:

  • What timeframes are being ignored
  • When this option should be used
  • The implications of different values
  • Whether this is a boolean or numeric parameter
db/sql/src/db.rs (2)

Line range hint 28-28: Consider making the peer cleanup constant private

The constant HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS is marked as internal in its name but is publicly accessible. Consider making it private if it's only used within this module.

-pub const HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS: u64 = 5 * 60; // 5 minutes
-
+const HOPR_INTERNAL_DB_PEERS_PERSISTENCE_AFTER_RESTART_IN_SECONDS: u64 = 5 * 60; // 5 minutes

Line range hint 246-350: Consider adding connection pool stress tests

While the current test coverage is good for peer retention logic, consider adding tests for:

  1. Connection pool exhaustion scenarios
  2. Connection timeout handling
  3. Concurrent database operations under high load
hoprd/rest-api/src/messages.rs (1)

539-539: Consider adding API migration guide

With multiple breaking changes in the message endpoints, it would be helpful to provide a migration guide for API consumers.

Consider:

  1. Document the changes in the API changelog
  2. Provide example requests for both old and new formats
  3. Add a deprecation timeline for the old endpoints
  4. Include migration scripts or code snippets for common client implementations

Also applies to: 590-590, 642-642

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between a7ebb12 and 9485831.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • db/sql/src/db.rs (1 hunks)
  • hoprd/rest-api/Cargo.toml (1 hunks)
  • hoprd/rest-api/src/messages.rs (3 hunks)
  • hoprd/rest-api/src/session.rs (3 hunks)
  • scripts/local-cluster-hoprd.cfg.yaml (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • hoprd/rest-api/Cargo.toml
🔇 Additional comments (7)
scripts/local-cluster-hoprd.cfg.yaml (1)

14-15: Verify security implications of timeframe ignoring

The ability to ignore timeframes could have security implications for message processing. Please confirm:

  • What security measures are in place when timeframes are ignored
  • Whether this could enable replay attacks
  • If this should be restricted in production environments

Let's check for any security-related code connected to this parameter:

db/sql/src/db.rs (2)

180-180: LGTM: Improved logging with structured format

The change to structured logging enhances observability by explicitly including the number of affected rows.


Line range hint 73-77: Verify connection pool settings and add monitoring

The peers database has a high connection limit (300) and long acquire timeout (60s). While this might be necessary for your use case, it could also mask underlying connection management issues.

Consider:

  1. Adding connection pool metrics monitoring
  2. Setting up alerts for connection pool saturation
  3. Implementing connection leak detection

Let's check for potential connection leaks:

✅ Verification successful

Based on the search results, I can now provide a final response regarding the connection pool settings:

Connection pool settings are well-configured with appropriate limits and timeouts

The database connection pools are configured with reasonable settings that align with common practices:

  • Index DB: 30 max connections, suitable for index operations
  • Peers DB: 300 max connections with 60s acquire timeout, justified by:
    • Includes proper idle timeout (10 mins) to prevent connection leaks
    • Has max lifetime setting (30 mins) to ensure connection recycling
    • Higher limit is appropriate for peer management which typically requires more concurrent connections
  • Tickets DB: 50 max connections, appropriate for ticket operations
  • All pools have min_connections=0 which allows efficient resource usage

The high connection limit and timeout for peers database is justified by its use case and includes proper connection lifecycle management to prevent leaks.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for database connection usage patterns
# Look for places where connections might not be properly released

# Check for direct connection usage without proper cleanup
rg -A 3 "peers_db\." 

# Look for transaction patterns
ast-grep --pattern 'transaction($$$).await'

Length of output: 440


Script:

#!/bin/bash
# Let's search for database connection and transaction patterns more comprehensively

# Look for database connection pool configurations
rg -A 5 "PoolOptions::new"

# Search for transaction usage patterns
rg -A 3 "\.begin\(\)|\.begin_with_config\(\)"

# Look for connection acquire timeout settings
rg "acquire_timeout|max_connections"

# Check for any explicit connection management
rg -A 3 "\.acquire\(\)|\.close\(\)"

Length of output: 100210

hoprd/rest-api/src/messages.rs (2)

539-539: ⚠️ Potential issue

API Breaking Change: pop endpoint now accepts JSON body instead of query parameters

This change aligns with the PR objectives but introduces a breaking change in the API contract. Existing clients using query parameters will need to be updated.

Let's verify the API documentation and potential impact:


590-590: ⚠️ Potential issue

API Breaking Change: pop-all endpoint now accepts JSON body instead of query parameters

This change fixes the regression reported in issue #6653 where the tag parameter was not being respected. However, it's a breaking change that requires client updates.

Let's verify the implementation across the codebase:

✅ Verification successful

Based on the evidence gathered from the codebase:

API Change Confirmed: pop-all endpoint now accepts JSON body

The OpenAPI documentation in the code confirms this is an intentional change to a POST endpoint that accepts a JSON body. The change appears to be properly documented with clear request/response schemas and descriptions.

  • The endpoint is documented to accept a TagQueryRequest body with clear description of its behavior
  • No existing client implementations were found in the codebase that would be affected
  • The change fixes a reported issue (#6653) with tag parameter handling
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check for any integration tests that might need updating
rg -A 5 "messages/pop-all\?tag=" 

# Look for client implementations that might be affected
ast-grep --pattern 'fetch($_, "messages/pop-all")'

Length of output: 415


Script:

#!/bin/bash
# Let's try different approaches to find relevant code

# Search for API route definition
rg -l "pop-all" --type rust

# Look for test files that might be using this endpoint
fd -e rs test

Length of output: 39421


Script:

#!/bin/bash
# Let's check the specific file and surrounding context for the API change
rg -A 10 -B 10 "pop-all" hoprd/rest-api/src/messages.rs

# Also check for OpenAPI/Swagger documentation that might need updating
fd -e yaml swagger
fd -e yaml openapi

Length of output: 1129


Script:

#!/bin/bash
# Let's check the git history for this specific change
rg -l "messages/pop-all" hoprd/rest-api/src/messages.rs | xargs git log -p -- | rg -B 5 -A 5 "pop-all"

# Also check for any client implementations
rg -A 5 "messages/pop-all" --type typescript --type javascript

Length of output: 97

hoprd/rest-api/src/session.rs (2)

74-82: LGTM! Clean refactoring of from_str implementation.

The simplified implementation improves readability while maintaining the same functionality. The error handling for base64 decoding is properly maintained.


115-117: Breaking change: target field is now required.

The change improves API reliability by ensuring the target is always specified. This aligns with the PR objectives.

Let's verify that all clients are prepared for this breaking change:

✅ Verification successful

Let me gather more information about the client usage by checking for test files and any client implementations.


Let me check the test files specifically for websocket client usage patterns.


Based on the test file content, I can now generate the final response as we have sufficient information about client usage.

Breaking change confirmed: target field is now required and clients are prepared.

The test file shows that clients are already providing the target parameter in their requests:

  • Test test_websocket_send_receive_messages uses the new API with target parameter
  • The older deprecated tests use a different endpoint /api/v3/messages/websocket
  • All current tests using the new endpoint already include the target parameter in their requests
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Search for client code that might be affected by the breaking change

# Look for SessionWebsocketClientQueryRequest usage
rg -A 5 "SessionWebsocketClientQueryRequest" 

# Look for websocket endpoint calls that might need updating
rg -A 5 "/api/v3/session/websocket"

Length of output: 3310


Script:

#!/bin/bash
# Look for test files that might show how the websocket endpoint is used
fd -e py -e rs test 
  -x rg -A 5 "session/websocket" {}

# Look for any client implementations or examples
rg -A 5 "websocket\?.*target" 

Length of output: 40396


Script:

#!/bin/bash
# Look for websocket test files and their contents
cat tests/test_websocket_api.py

Length of output: 8993

Copy link
Member

@esterlus esterlus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ curl -s -X POST -H "Content-Type: application/json" -H "x-auth-token: ^^LOCAL-testing-123^^" -d '{"tag":2001}' http://127.0.0.1:19091/api/v3/messages/pop-all | jq
{
  "messages": [
    {
      "tag": 2000,
      "body": "test0",
      "receivedAt": 1732277432917
    },
    {
      "tag": 2000,
      "body": "test0",
      "receivedAt": 1732277435466
    },
    {
      "tag": 2000,
      "body": "test0",
      "receivedAt": 1732277436187
    },
    {
      "tag": 2001,
      "body": "test0",
      "receivedAt": 1732277488641
    }
  ]
}

pop-all still popping all messages regardless of tag

@esterlus
Copy link
Member

Oh wait, no my bad

Copy link
Member

@esterlus esterlus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pop-all / peek-all work as expected
websocket session also works now

great 👍

@github-actions github-actions bot added dependencies Pull requests that update a dependency file toolchain Developer and product happiness crate:hoprd-api testing Related to tests labels Nov 22, 2024
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
tests/test_websocket_api.py (1)

Line range hint 1-256: Enhance test coverage for PR changes

The current test suite needs additional coverage for the changes mentioned in PR objectives:

  1. Add tests to verify JSON payload acceptance for /messages/ endpoints:

    • Test pop, pop_all, and peek with both JSON payloads and query parameters
    • Verify correct error responses for invalid payload formats
  2. Add specific tests for the message tag regression (issue #6653):

    • Test pop_all with tagged messages
    • Verify only messages with matching tags are returned
    • Test edge cases with multiple tags
  3. Consider moving the tag-related test logic from the deprecated section to new tests, as it provides better coverage.

Example test structure:

@pytest.mark.parametrize("endpoint", ["pop", "pop_all", "peek"])
async def test_message_endpoint_payload(endpoint):
    # Test JSON payload acceptance
    # Test query parameter rejection
    pass

async def test_pop_all_with_tags():
    # Test tag filtering
    # Test multiple tags
    # Test tag regression fix
    pass
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 9485831 and ec1b951.

📒 Files selected for processing (1)
  • tests/test_websocket_api.py (1 hunks)
🔇 Additional comments (2)
tests/test_websocket_api.py (2)

Line range hint 190-256: Preserve test coverage while deprecating old tests

Before removing the deprecated tests, ensure their valuable test coverage is migrated:

  1. The tag handling in test_websocket_send_receive_messages_DEPRECATED is more thorough than current tests
  2. The multiple connection handling in ws_connections fixture provides better concurrent testing
  3. The message format validation is more complete in the deprecated version

Consider extracting these test patterns into new tests before removing the deprecated code.

Here's a script to analyze test coverage differences:


24-25: 🛠️ Refactor suggestion

Add tests to verify target field requirement

While adding the target parameter aligns with making it mandatory, consider:

  1. The hardcoded target "127.0.0.1:4677" might not be suitable for all test environments. Consider using a dynamic port from PORT_BASE.
  2. Add test cases to verify:
    • Connection rejection when target is missing
    • Connection rejection with invalid target format
    • Connection success with various valid target formats

Here's a script to verify if we're missing any target-related test cases:

@Teebor-Choka Teebor-Choka force-pushed the kauki/rest-api/fix-bugs-related-to-messages-and-sessions branch from ec1b951 to becefb6 Compare November 22, 2024 15:48
@Teebor-Choka
Copy link
Contributor Author

@ausias-armesto also relevant for #6640 as it slightly changes the websocket session interface to always require the target as mandatory.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between ec1b951 and becefb6.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (6)
  • db/sql/src/db.rs (1 hunks)
  • hoprd/rest-api/Cargo.toml (1 hunks)
  • hoprd/rest-api/src/messages.rs (3 hunks)
  • hoprd/rest-api/src/session.rs (3 hunks)
  • scripts/local-cluster-hoprd.cfg.yaml (1 hunks)
  • tests/test_websocket_api.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
  • db/sql/src/db.rs
  • hoprd/rest-api/Cargo.toml
  • scripts/local-cluster-hoprd.cfg.yaml
🔇 Additional comments (4)
hoprd/rest-api/src/messages.rs (3)

642-642: Update API documentation for breaking changes

The change to JSON body is consistent with other endpoints. However, ensure that the API documentation and changelog clearly communicate these breaking changes to clients.

Consider adding a migration guide in the README or API documentation to help users transition from query parameters to JSON body.


539-539: ⚠️ Potential issue

API Breaking Change: Consider versioning the endpoint

The change from query parameters to JSON body maintains consistency with other endpoints but introduces a breaking change. Consider versioning the endpoint (e.g., /v2/messages/pop) to ensure backward compatibility.


590-590: LGTM: Addresses tag parameter handling issue

The change to JSON body handling fixes the regression in tag parameter processing (issue #6653). The implementation correctly validates and processes the tag parameter.

✅ Verification successful

Change in pop_all endpoint is consistent with other message endpoints

The verification shows that the codebase has a consistent pattern for tag parameter handling:

  • GET endpoints (size, delete_messages) use Query parameters
  • POST endpoints (pop, pop_all, peek) use JSON body
  • All endpoints use the same TagQueryRequest struct and validation logic for tags

The change from Query to Json in pop_all aligns with this pattern, making the API more consistent.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Verify consistent tag parameter handling across message endpoints
rg -A 3 'if let Some\(tag\) = tag.*RESERVED_TAG_UPPER_LIMIT'

Length of output: 62


Script:

#!/bin/bash
# Search for tag validation patterns in messages.rs
rg -A 5 'tag: Option<u32>' hoprd/rest-api/src/messages.rs

# Search for tag parameter handling in message endpoints
ast-grep --pattern 'struct TagQueryRequest {
    $$$
}'

# Look for tag validation logic
rg -A 5 'tag.*RESERVED_TAG' hoprd/rest-api/src/messages.rs

Length of output: 1526


Script:

#!/bin/bash
# Search for all TagQueryRequest usages in messages.rs
rg -B 3 -A 3 'TagQueryRequest' hoprd/rest-api/src/messages.rs

# Look for the endpoint handler signatures
ast-grep --pattern 'async fn $_($$$) -> $$ {
    $$$
}'

Length of output: 3178

hoprd/rest-api/src/session.rs (1)

74-82: LGTM! Clean refactoring of the from_str implementation.

The refactored code is more idiomatic and easier to follow, with improved error handling through a single return path.

@Teebor-Choka Teebor-Choka merged commit c03e83f into master Nov 22, 2024
28 checks passed
@Teebor-Choka Teebor-Choka deleted the kauki/rest-api/fix-bugs-related-to-messages-and-sessions branch November 22, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:api crate:hoprd-api dependencies Pull requests that update a dependency file testing Related to tests toolchain Developer and product happiness
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Regression: pop-all messages does no longer adhere to provided tag parameter
2 participants