This is a draft for the initial implementation of the Surescripts client. This combines the initial testing scripts and validation work from the ENG-194 branch into a proper Surescripts client, with full unit testing and robust SFTP error handling.

Issues:

• https://linear.app/metriport/issue/ENG-306
• https://linear.app/metriport/issue/ENG-313 (CDK related changes)

Dependencies

• Upstream: None
• Downstream: None

Description

This creates a SurescriptsSftpClient class, which provides an abstraction layer for producing messages in the Surescripts pipe-delimited format for SFTP bulk transfers.

Testing

• Local: You should not test Surescripts locally. Surescripts has white-listed the specific IP addresses associated with outbound traffic from our staging VPC for their SFTP sandbox, and production IP addresses for their production SFTP endpoint. Furthermore, the credentials for authenticating with Surescripts should be loaded from AWS Secrets Manager only.

• Staging:
  There are commands and tests located within the sftp directory, as well as the surescripts extension of the base SftpClient implementation. These must be executed from a compute resource (Lambda, EC2) within the staging VPC. The simplest way to test the client against the Surescripts SFTP sandbox is:

1. connect by VPN to staging
2. use the SSH key stored in 1PW under "Surescripts Staging Instance" to ssh into the machine
3. run the test command within the metriport directory (a semi-initialized git clone of this repository)
4. check the corresponding metriport-surescripts-replica-staging bucket to ensure that generated files are present and meet manual validation
5. Run the automated scripts in the external/sftp directory (npx jest -- external/sftp)

• Production:
  [ TODO ]

Release Plan

This client implementation currently only interacts with the metriport-surescripts-replica-[staging|production] buckets, and provides an initial implementation for the associated Lambda functions from the SurescriptsNestedStack in the CDK.

• Upstream dependencies are met/released

Summary by CodeRabbit

• New Features

  • Introduced a robust Surescripts integration with support for patient and batch requests, verification, response handling, and request history verification via SFTP and AWS SQS.
  • Added a new SFTP client with connection management, gzip compression, and replica storage support on S3 or local filesystem.
  • Developed CLI tools for Surescripts operations including sending requests, receiving responses and verifications, batch requests, SFTP actions, and request history verification.
  • Implemented comprehensive file generation and parsing for Surescripts-compliant formats with strict schema validation and error handling.
  • Enhanced configuration with new environment variables and static methods for Surescripts-specific settings.

• Bug Fixes

  • Improved error handling and validation across Surescripts API interactions, SFTP client operations, and file parsing.

• Documentation

  • Added example environment variable configurations for Surescripts integration.

• Tests

  • Added extensive unit and integration tests covering Surescripts file parsing, ID generation, SFTP client functionality, and CLI commands.

• Chores

  • Refactored infrastructure stacks and Lambda setup to unify concurrency, add new Lambdas and queues, and improve environment variable injection and secret access management.