Skip to content

Conversation

thomasyopes
Copy link
Contributor

@thomasyopes thomasyopes commented May 5, 2025

Issues:

Dependencies

Description

  • Migrates resource diff job to resource-type level computation instead of per individual resource
  • Moves the resource diff job start/get logic to a shared logic
  • Moves fetching of computed bundles pre-signed-urls (EHR and Resource Diff) to shared logic
  • Moves refreshing the EHR bundles to shared logic
  • Moves resource diff job routing to shared routes
  • Split shared file into domain specific elements + added utility files for bundles / jobs
  • Re-typed shared bundle code to use Medplum types, not custom types
  • Moves resource diff utility to fhir-deduplication folder and makes it non-ehr-specific

Testing

  • Local
    • resource type workflow runs (including job updates)
    • canvas level API interactions work on shared commands / jobs / routes (both internal and CX facing)
    • diff utility produces the correct two output bundles.
  • Staging
    • resource type workflow runs (including job updates)
    • canvas level API interactions work on shared commands / jobs / routes (both internal and CX facing)
    • diff utility produces the correct two output bundles.
  • Sandbox
    • N/A
  • Production
    • resource type workflow runs (including job updates)
    • canvas level API interactions work on shared commands / jobs / routes (both internal and CX facing)
    • diff utility produces the correct two output bundles.

Release Plan

  • Before shipping, throttle the start resource diff job lambda to
  • Merge this

Summary by CodeRabbit

  • New Features

    • Added middleware to validate and process EHR source IDs in routes.
    • Introduced new internal API endpoints for managing EHR patient resource diff jobs and bundles.
    • Added utility modules for appointment date ranges, EHR client creation, JWT token handling, external ID parsing, and bundle operations.
    • Introduced classes for cloud and local processing of resource diff bundle computations and EHR bundle refreshes.
    • Added functions to compute resource differences and deduplicate FHIR resources with strict validation.
    • Added new bundle types and resource type validations for improved EHR data handling.
    • Added retry logic to API calls and SQS message sending for robustness.
  • Refactor

    • Unified resource diff bundle creation and refresh logic, removing direction-based handling in favor of explicit bundle types and job IDs.
    • Modularized shared utilities for JWT tokens, bundle operations, and mappings.
    • Replaced direct bundle fetching with asynchronous job-based processing and parallel resource handling.
    • Simplified EHR bundle refresh and resource diff start flows with improved job management and error handling.
    • Updated resource type handling to use standardized supported resource types and schemas.
    • Enhanced API request handling with centralized response validation and retry logic.
    • Consolidated resource diff bundle computation methods into a single streamlined process.
    • Updated import paths and code structure for better modularity and maintainability.
    • Removed obsolete resource diff direction enums and related code.
  • Bug Fixes

    • Improved validation for supported resource types and job status updates.
    • Strengthened error handling and reporting in job processing and bundle operations.
    • Fixed potential runtime errors by safely accessing resource properties and validating inputs.
  • Chores

    • Removed obsolete endpoints, Lambda handlers, and infrastructure related to prior resource diff bundle initiation.
    • Updated import paths and internal wiring to align with new utility modules and improved code structure.
    • Cleaned up unused types, constants, and functions related to removed resource diff direction handling.
    • Adjusted environment variables and permissions in infrastructure stacks to reflect removed components.
  • Documentation

    • Enhanced API route parameter validation and documentation for better clarity and consistency.

Thomas Yopes added 2 commits May 1, 2025 11:13
Ref: ENG-82

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Copy link

coderabbitai bot commented May 5, 2025

Walkthrough

This change set refactors and modularizes the EHR bundle/resource diff job infrastructure, removing legacy Canvas-specific resource diff logic and generalizing the workflow to support multiple EHRs. It introduces new utility modules for bundle and job management, updates API routes and handlers, and replaces direct Lambda invocations with SQS-driven job orchestration. Several files are deleted or replaced, type definitions are updated, and error handling and retry logic are improved throughout the codebase.

Changes

File(s) / Path(s) Change Summary
packages/api/src/command/mapping/cx.ts, .../patient.ts, .../consolidated-recreate.ts, .../cx-mapping.ts, .../jwt-token.ts, .../athenahealth/command/process-patients-from-appointments.ts, .../athenahealth/command/sync-patient.ts, .../athenahealth/shared.ts, .../canvas/command/process-patients-from-appointments.ts, .../canvas/command/sync-patient.ts, .../canvas/shared.ts, .../elation/command/process-patients-from-appointments.ts, .../elation/command/sync-patient.ts, .../elation/shared.ts, .../healthie/command/process-patients-from-appointments.ts, .../healthie/command/sync-patient.ts, .../healthie/shared.ts, .../jwt-token.ts, .../shared.ts, .../routes/ehr/shared.ts, .../routes/internal/jwt-token/athenahealth.ts, .../canvas.ts, .../elation.ts, .../healthie.ts Updated import paths to new utility modules; no logic changes.
packages/api/src/external/ehr/canvas/command/bundle/fetch-ehr-bundle.ts, .../fetch-resource-diff-bundle-pre-signed-urls.ts, .../job/create-resource-diff-bundles/get-job-payload.ts, .../job/create-resource-diff-bundles/start-job.ts, .../shared.ts, .../resource-diff.ts, .../jwt-token.ts, .../lambdas/src/ehr-start-resource-diff-bundles.ts Deleted legacy Canvas-specific and shared resource diff/bundle/job logic.
packages/api/src/external/ehr/canvas/command/bundle/refresh-ehr-bundles.ts Replaced legacy refresh logic with a new function refreshEhrBundles supporting generalized parameters and single resource type refresh.
packages/api/src/external/ehr/shared/utils/appointment.ts, .../client.ts, .../external-id.ts, .../fhir.ts, .../job.ts, .../jwt-token.ts, .../mappings.ts, .../patient.ts, .../bundle/functions.ts, .../bundle/types.ts Introduced new utility modules for appointment, client, external ID, FHIR, job, JWT token, mappings, patient, and bundle management.
packages/api/src/external/ehr/shared/command/bundle/fetch-bundle-presignd-urls.ts, .../refresh-ehr-bundles.ts Added new generalized functions for fetching bundle pre-signed URLs and refreshing bundles across EHRs.
packages/api/src/external/ehr/shared/job/bundle/create-resource-diff-bundles/get-job-payload.ts, .../start-job.ts, .../start-jobs-across-ehrs.ts Added new generalized job payload retrieval and job start logic for resource diff bundles.
packages/api/src/routes/ehr/canvas/patient.ts, .../routes/internal/ehr/canvas/patient.ts Refactored/removed Canvas-specific resource diff endpoints and logic; migrated to shared, generic job/bundle handling.
packages/api/src/routes/internal/ehr/index.ts, .../patient.ts, .../middleware.ts Added new internal EHR patient resource/job router and middleware for EHR ID validation.
packages/api/src/routes/internal/medical/patient-job.ts Added route for completing patient jobs with no entries; minor doc updates.
packages/core/src/external/ehr/api/api-shared.ts Added validateAndLogResponse utility for consistent API response validation/logging.
packages/core/src/external/ehr/api/bundle/refresh-ehr-bundle.ts, .../job/complete-job.ts, .../job/create-resource-diff-bundles/set-entry-status.ts, .../job/set-entry-status.ts Added/updated functions for refreshing EHR bundles, completing jobs, and setting job entry status with improved error handling and retries.
packages/core/src/external/ehr/api/job/initialize-job.ts, .../update-job-total.ts, .../sync-patient.ts, .../link-patient.ts Updated to use retry logic and centralized response validation.
packages/core/src/external/ehr/bundle/bundle-shared.ts Added/updated type guards and resource type validation for bundle operations.
packages/core/src/external/ehr/bundle/command/create-or-replace-bundle.ts, .../fetch-bundle.ts Improved error handling, resource type validation, and retry logic for S3 operations.
packages/core/src/external/ehr/bundle/create-resource-diff-bundles/steps/compute/ehr-compute-resource-diff-bundles-local.ts, .../ehr-compute-resource-diff-bundles.ts, .../utils.ts Refactored diff computation logic to batch processing, stricter validation, and improved deduplication.
packages/core/src/external/ehr/bundle/create-resource-diff-bundles/steps/compute/ehr-compute-resource-diff-bundles-cloud.ts, .../factory.ts, .../ehr-compute-resource-diff-bundles.ts Added SQS-based cloud handler and updated interfaces for diff bundle computation.
packages/core/src/external/ehr/bundle/create-resource-diff-bundles/steps/start/ehr-start-resource-diff-bundles-local.ts Refactored to remove legacy resource validation, streamline resource type iteration, and manage job totals/completion.
packages/core/src/external/ehr/bundle/job/create-resource-diff-bundles/create-resource-diff-bundle-shared.ts Added shared types/utilities for diff bundle job requests and SQS group IDs.
packages/core/src/external/ehr/bundle/job/create-resource-diff-bundles/steps/refresh/ehr-refresh-ehr-bundles-cloud.ts, .../ehr-refresh-ehr-bundles-factory.ts, .../ehr-refresh-ehr-bundles-local.ts, .../ehr-refresh-ehr-bundles.ts Added/updated cloud/local handlers and interfaces for refreshing EHR bundles.
packages/core/src/external/ehr/bundle/refresh-ehr-bundles/ehr-refresh-ehr-bundles-local.ts Enhanced job lifecycle/error handling and resource type iteration for EHR bundle refresh.
packages/core/src/external/ehr/canvas/index.ts Refactored to generalize resource type/bundle handling, update method signatures, and remove direction-based logic.
packages/core/src/external/ehr/elation/link-patient/elation-link-patient-cloud.ts, .../healthie/link-patient/healthie-link-patient-cloud.ts, .../sync-patient/ehr-sync-patient-cloud.ts Wrapped SQS message sending in retry logic.
packages/core/src/fhir-deduplication/compute-resources-xor.ts Added function for computing XOR of FHIR resource sets with validation and deduplication.
packages/shared/src/interface/external/ehr/fhir-resource.ts Unified resource types with external FHIR types, exported supported resource types, and updated bundle creation signatures.
packages/infra/lib/ehr-nested-stack.ts, .../api-stack.ts, .../api-stack/api-service.ts Removed all references to legacy start resource diff bundles queue/lambda, updated permissions, and environment variables.
packages/lambdas/src/ehr-compute-resource-diff-bundles.ts, .../ehr-refresh-ehr-bundles.ts Updated Lambda handlers to new message formats, resource type handling, and error reporting.

Sequence Diagram(s)

Generalized Resource Diff Bundle Job Flow

sequenceDiagram
    participant API as API Route
    participant JobStarter as startCreateResourceDiffBundlesJob
    participant JobDB as PatientJob Store
    participant BundleRefresher as refreshEhrBundles
    participant SQS as SQS Queue
    participant Lambda as Lambda Worker
    participant S3 as S3 Storage

    API->>JobStarter: POST /internal/ehr/:ehrId/patient/:id/resource/diff (start job)
    JobStarter->>JobDB: Create patient job (type: resource-diff)
    JobStarter->>BundleRefresher: For each supported resource type, refreshEhrBundles(...)
    BundleRefresher->>SQS: Send refresh job message
    SQS->>Lambda: Trigger Lambda for refresh
    Lambda->>S3: Create/replace empty diff bundles (Metriport-only, EHR-only)
    Lambda->>SQS: Send compute diff job message
    SQS->>Lambda: Trigger Lambda for compute
    Lambda->>S3: Fetch resources, compute diff, update bundles
    Lambda->>JobDB: Update job entry status (successful/failed)
Loading

Fetching Resource Diff Job Payload

sequenceDiagram
    participant API as API Route
    participant JobDB as PatientJob Store
    participant BundleFetcher as fetchBundlePreSignedUrls

    API->>JobDB: Fetch patient job by jobId
    alt job.status == "completed"
        API->>BundleFetcher: fetchBundlePreSignedUrls(...)
        BundleFetcher->>API: Return pre-signed URLs
        API->>API: Compose and return job payload with URLs
    else
        API->>API: Return job payload (no URLs)
    end
Loading

Refreshing EHR Bundles

sequenceDiagram
    participant API as API Route
    participant BundleRefresher as refreshEhrBundles
    participant SQS as SQS Queue
    participant Lambda as Lambda Worker
    participant S3 as S3 Storage

    API->>BundleRefresher: POST /internal/ehr/:ehrId/patient/:id/resource/refresh
    BundleRefresher->>SQS: Send refresh job message
    SQS->>Lambda: Trigger Lambda for refresh
    Lambda->>S3: Refresh EHR bundle for resource type
    Lambda->>JobDB: Update job entry status
Loading

JWT Token Management (Generalized)

sequenceDiagram
    participant API as API Route
    participant JWTUtils as checkJwtToken/saveJwtToken
    participant TokenStore as JWT Token Store

    API->>JWTUtils: checkJwtToken({token, source})
    JWTUtils->>TokenStore: getJwtToken({token, source})
    TokenStore-->>JWTUtils: Token info
    JWTUtils-->>API: {active, expired?}

    API->>JWTUtils: saveJwtToken({token, source, exp, data})
    JWTUtils->>TokenStore: findOrCreateJwtToken(...)
    TokenStore-->>JWTUtils: (persisted)
    JWTUtils-->>API: (done)
Loading

SQS-Based Resource Diff Computation

sequenceDiagram
    participant JobStarter as startCreateResourceDiffBundlesJob
    participant SQS as SQS Queue
    participant Lambda as Lambda Worker
    participant S3 as S3 Storage

    JobStarter->>SQS: Send compute diff job message (per resource type)
    SQS->>Lambda: Trigger Lambda for compute
    Lambda->>S3: Fetch resources, compute diff, update bundles
    Lambda->>JobDB: Update job entry status
Loading

These diagrams represent the new modular, EHR-agnostic resource diff and bundle management workflows, reflecting the new control flow and infrastructure introduced by this change set.

Note

⚡️ AI Code Reviews for VS Code, Cursor, Windsurf

CodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback.
Learn more here.


Note

⚡️ Faster reviews with caching

CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure Review - Disable Cache at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the Data Retention setting under your Organization Settings.
Enjoy the performance boost—your workflow just got faster.


🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Thomas Yopes added 11 commits May 5, 2025 10:40
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved into /shared/utils/jwt-token.ts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JWT stuff moved into /shared/utils/jwt-token.ts
Appointments stuff moved into /shared/utils/appointment.ts
External ID stuff moved into /shared/utils/external-id.ts
Client stuff moved into /shared/utils/client.ts

Thomas Yopes added 5 commits May 5, 2025 13:24
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
… 199-elation-contribution

Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
@@ -56,7 +55,6 @@ export async function recreateConsolidated({
createResourceDiffBundles({
cxId: patient.cxId,
patientId: patient.id,
direction: ResourceDiffDirection.METRIPORT_ONLY,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resource diff job now computes both directions at once.

Thomas Yopes added 2 commits May 6, 2025 11:57
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
@@ -45,7 +48,3 @@ export async function createCanvasClient(
getClient: CanvasApi.create,
});
}

export function getCreateCanvasResourceDiffBundlesJobType(direction: ResourceDiffDirection) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to jobs utils file

Thomas Yopes added 4 commits May 6, 2025 12:52
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Ref: ENG-199

Ref: #1040
Signed-off-by: Thomas Yopes <thomasyopes@Thomass-MBP.attlocal.net>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken from old shared file.

@coderabbitai coderabbitai bot mentioned this pull request Aug 5, 2025
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants