feat: Add distributed tracing for booking flow correlation

Part of #22969

Follow up:-

• Add more logs and replace all logger with distributedTracer

Summary

This PR implements distributed tracing for Cal.com's booking flow to improve observability and debugging capabilities beyond the current per-request ID system. The implementation adds trace context propagation across multiple booking operations including calendar events, payment processing, webhook execution, and scheduled reminders.

Key Changes:

• New tracing library (packages/lib/tracing.ts) - Centralized trace context management with hierarchical spans
• Enhanced booking creation - Trace context initiation in handleNewBooking.ts
• Cross-component correlation - Trace propagation through EventManager, webhook scheduling, payment processing
• Comprehensive documentation - Debugging examples and Axiom queries demonstrating concrete benefits

Benefits over current request ID system:

• Cross-Request Correlation: Links operations spanning multiple API requests (booking → webhook → cron execution)
• Async Operation Tracking: Correlates scheduled operations back to original booking context
• Performance Monitoring: Enables end-to-end timing analysis across entire booking lifecycle
• Enhanced Debugging: Complete booking flow visibility in Axiom with correlation queries

Review & Testing Checklist for Human

• End-to-end booking flow testing - Create bookings with different configurations and verify they complete successfully without errors
• Trace correlation verification - Check Axiom logs to confirm trace IDs properly correlate operations across booking → calendar → webhook → payment flows
• Async operation tracking - Verify webhook scheduling and execution maintains trace context correlation back to original booking
• Performance impact assessment - Monitor booking flow performance to ensure tracing overhead is minimal
• Edge case testing - Test scenarios where trace context might be missing (direct API calls, cron jobs) to ensure graceful fallback

Recommended test plan:

1. Create standard booking and verify trace appears in Axiom with correlated operations
2. Test payment flow and confirm payment success correlates to original booking trace
3. Verify webhook execution traces back to originating booking
4. Test seated bookings with multiple attendees for reminder correlation
5. Run Axiom queries from examples to confirm they return expected results

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    %% Main booking flow
    handleNewBooking["packages/features/bookings/lib/handleNewBooking.ts"]:::major-edit
    tracing["packages/lib/tracing.ts"]:::major-edit
    EventManager["packages/lib/EventManager.ts"]:::major-edit
    
    %% Webhook flow
    scheduleTrigger["packages/features/webhooks/lib/scheduleTrigger.ts"]:::major-edit
    handleWebhookScheduledTriggers["packages/features/webhooks/lib/handleWebhookScheduledTriggers.ts"]:::major-edit
    handleWebhookTrigger["packages/features/bookings/lib/handleWebhookTrigger.ts"]:::minor-edit
    
    %% Payment flow
    handlePaymentSuccess["packages/lib/payment/handlePaymentSuccess.ts"]:::minor-edit
    
    %% Seat booking flow
    handleSeats["packages/features/bookings/lib/handleSeats/handleSeats.ts"]:::major-edit
    scheduleNoShowTriggers["packages/features/bookings/lib/handleNewBooking/scheduleNoShowTriggers.ts"]:::minor-edit
    
    %% Reminder flow
    scheduleMandatoryReminder["packages/features/ee/workflows/lib/reminders/scheduleMandatoryReminder.ts"]:::minor-edit
    reminderScheduler["packages/features/ee/workflows/lib/reminders/reminderScheduler.ts"]:::minor-edit
    
    %% Documentation
    debuggingExamples["examples/distributed-tracing-debugging.md"]:::major-edit
    axiomQueries["examples/axiom-tracing-queries.md"]:::major-edit
    
    %% Flow connections
    handleNewBooking --> tracing
    handleNewBooking --> EventManager
    handleNewBooking --> scheduleTrigger
    handleNewBooking --> handleSeats
    
    scheduleTrigger --> handleWebhookScheduledTriggers
    handleWebhookScheduledTriggers --> handleWebhookTrigger
    
    handleNewBooking --> handlePaymentSuccess
    
    handleSeats --> scheduleMandatoryReminder
    handleNewBooking --> scheduleNoShowTriggers
    scheduleMandatoryReminder --> reminderScheduler
    
    %% Legend
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit
        L3["Context/No Edit"]:::context
    end
    
    %% Styling
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Notes

Implementation approach:

• Trace context is optional throughout - existing functionality continues to work without tracing
• Uses existing logger infrastructure with enhanced prefixes for correlation
• Backward compatible with current request ID logging
• Follows existing patterns for middleware and error handling

Axiom integration:

• Leverages existing Axiom setup with enhanced log prefixes
• Provides ready-to-use correlation queries for common debugging scenarios
• Enables proactive monitoring with alerting capabilities

Link to Devin run: https://app.devin.ai/sessions/d494fd7f104f4339a390e492f94bd47a
Requested by: @Udit-takkar

⚠️ Important: This implementation could not be tested end-to-end locally due to booking system complexity. Human testing of the complete booking flow is essential before merging.