Download • Documentation • Community

./daptin
.
. // logs truncated for brevity
.
INFO[2024-10-16 11:08:58] Listening websocket server at ... /live
INFO[2024-10-16 11:08:58] Our admin is [artpar@gmail.com]
INFO[2024-10-16 11:08:58] [ProcessId=86403] Listening at port: :6336
INFO[2024-10-16 11:08:58] Get certificate for [Parths-MacBook-Pro.local]: true
INFO[2024-10-16 11:08:58] Listening at: [:6336]
INFO[2024-10-16 11:08:58] TLS server listening on port :6443
INFO[2024-10-16 11:09:03] Member says: Message<members: Joining from 192.168.0.125:5336>

Server is up, sqlite database is used since we did not specify mysql or postgres.

call the signup "action" api to create a new user_account

curl 'http://localhost:6333/action/user_account/signup' -X POST \
--data-raw '{"attributes":{"email":"artpar@gmail.com","password":"artpar@gmail.com","name":"artpar@gmail.com","passwordConfirm":"artpar@gmail.com"}}'

On a fresh instance all actions are allowed to be executed by guests, so you shouldn't see this

[
    {
        "Attributes": {
            "message": "http error (403) forbidden and 0 more errors, forbidden",
            "title": "failed",
            "type": "error"
        },
        "ResponseType": "client.notify"
    }
]

You should see this

[
  {
    "ResponseType": "client.notify",
    "Attributes": {
      "__type": "client.notify",
      "message": "Sign-up successful. Redirecting to sign in",
      "title": "Success",
      "type": "success"
    }
  },
  {
    "ResponseType": "client.redirect",
    "Attributes": {
      "__type": "client.redirect",
      "delay": 2000,
      "location": "/auth/signin",
      "window": "self"
    }
  }
]

curl 'http://localhost:6336/action/user_account/signin' \
--data-raw '{"attributes":{"email":"artpar@gmail.com","password":"artpar@gmail.com"}}'

[
    {
        "Attributes": {
            "key": "token",
            "value": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImFydHBhckBnbWFpbC5jb20iLCJleHAiOjE3MjkzMjExMjIsImlhdCI6MTcyOTA2MTkyMiwiaXNzIjoiZGFwdGluLTAxOTIyOCIsImp0aSI6IjAxOTI5NDFmLTI2MGUtN2I0Ni1hMWFlLWYxMGZhZTcwMDE3OSIsIm5hbWUiOiJhcnRwYXJAZ21haWwuY29tIiwibmJmIjoxNzI5MDYxOTIyLCJzdWIiOiIwMTkyMmUxYS1kNWVhLTcxYzktYmQzZS02MTZkMjM3ODBmOTMifQ.H-GLmXCT-o7RxXrjo5Of0K8Nw5mpOOw6jgoXnd5KUxo"
        },
        "ResponseType": "client.store.set"
    },
    {
        "Attributes": {
            "key": "token",
            "value": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImFydHBhckBnbWFpbC5jb20iLCJleHAiOjE3MjkzMjExMjIsImlhdCI6MTcyOTA2MTkyMiwiaXNzIjoiZGFwdGluLTAxOTIyOCIsImp0aSI6IjAxOTI5NDFmLTI2MGUtN2I0Ni1hMWFlLWYxMGZhZTcwMDE3OSIsIm5hbWUiOiJhcnRwYXJAZ21haWwuY29tIiwibmJmIjoxNzI5MDYxOTIyLCJzdWIiOiIwMTkyMmUxYS1kNWVhLTcxYzktYmQzZS02MTZkMjM3ODBmOTMifQ.H-GLmXCT-o7RxXrjo5Of0K8Nw5mpOOw6jgoXnd5KUxo; SameSite=Strict"
        },
        "ResponseType": "client.cookie.set"
    },
    {
        "Attributes": {
            "message": "Logged in",
            "title": "Success",
            "type": "success"
        },
        "ResponseType": "client.notify"
    },
    {
        "Attributes": {
            "delay": 2000,
            "location": "/",
            "window": "self"
        },
        "ResponseType": "client.redirect"
    }
]

We will use

export TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImFydHBhckBnbWFpbC5jb20iLCJleHAiOjE3MjkzMjExMjIsImlhdCI6MTcyOTA2MTkyMiwiaXNzIjoiZGFwdGluLTAxOTIyOCIsImp0aSI6IjAxOTI5NDFmLTI2MGUtN2I0Ni1hMWFlLWYxMGZhZTcwMDE3OSIsIm5hbWUiOiJhcnRwYXJAZ21haWwuY29tIiwibmJmIjoxNzI5MDYxOTIyLCJzdWIiOiIwMTkyMmUxYS1kNWVhLTcxYzktYmQzZS02MTZkMjM3ODBmOTMifQ.H-GLmXCT-o7RxXrjo5Of0K8Nw5mpOOw6jgoXnd5KUxo 

for the rest of the api calls. This is a JWT token with following data

{
  "email": "artpar@gmail.com",                    // user email
  "exp": 1729321122,                              // token expiry
  "iat": 1729061922,                              // token issued at time
  "iss": "daptin-019228",                         // token issuer (your daptin instance)
  "jti": "0192941f-260e-7b46-a1ae-f10fae700179",  // unique identifier for this token
  "name": "artpar@gmail.com",                     // user name
  "nbf": 1729061922,                              // token valid not before timestamp
  "sub": "01922e1a-d5ea-71c9-bd3e-616d23780f93"   // user reference id
}

So you have an account and a token to authenticate as that account. But do you need it? No. Call to fetch all user accounts works without any authorization

curl http://localhost:6333/api/user_account

{
  "links": {
    "current_page": 1,
    "from": 0,
    "last_page": 1,
    "per_page": 10,
    "to": 10,
    "total": 1
  },
  "data": [
    {
      "type": "user_account",
      "id": "01929429-3d8f-7e53-8f15-a663e05fb01b",
      "attributes": {
        "__type": "user_account",
        "confirmed": 0,
        "created_at": "2024-10-16T07:09:43.86360642Z",
        "email": "artpar1@gmail.com",
        "name": "artpar1@gmail.com",
        "password": "",
        "permission": 2097151,
        "reference_id": "01929429-3d8f-7e53-8f15-a663e05fb01b",
        "updated_at": "2024-10-16T07:09:43.863622045Z",
        "user_account_id": "01929429-3d8f-7e53-8f15-a663e05fb01b"
      },
      "relationships": { /// ...}
    }
  ]
}

And so does all the data in all other tables (eg site, cloud_store, document, usergroup). And you can call update and delete APIs as well (not demonstrated here, but you can try, delete the sqlite database file after you are done playing to reset it all)

As the first user, it is an option for you to leave it open or enable the multi-tier permission and becoming the Administrator

curl 'http://localhost:6336/action/world/become_an_administrator' --compressed -X POST \
-H "Authorization:  Bearer $TOKEN" --data-raw '{}'

At this point, all other apis are locked-down and only accessible by administrator, that is you. You want to open up few or many of actions to guests or users.

... Will be updated soon

Overall Progress: 71% Complete (37/52 features documented)

• Session 1: Foundation (7 features) - Configuration, Statistics, Meta, Health, JS Models, Aggregation ✅
• Session 2: Real-time & Communication (12 features) - WebSocket, YJS, SMTP, CalDAV, FTP, Feeds ✅
• Session 3: Advanced Data & Analytics (8 features) - Aggregation, GraphQL, Import/Export, Relationships ✅
• Session 4: Infrastructure & Configuration (10 features) - Config API, Rate Limiting, GZIP, Caching, CORS, TLS ✅
• Session 5: Workflow & Automation (Planned)
• Session 6: Client Integration & Developer Experience (Planned)
• Session 7: Final Documentation & Polish (Planned)

Documentation Artifacts:

• /openapi.yaml - Self-updating API documentation
• SELF_DOCUMENTATION_MASTER_PLAN.md - Complete roadmap
• SESSION_HANDOFF.md - Progress tracking
• NEXT_SESSION_PROMPT.md - Next session guide

• Database backed persistence, 3NF normalized tables
• JSON API/GraphQL for CRUD apis
• User and group management and access control
• Social login with OAuth: tested with google, github, linkedin
• Actions for abstracting out business flows
• Extensive state tracking APIs
• Enable Data Auditing from a single toggle
• Synchronous Data Exchange with 3rd party APIs
• Multilingual tables support, supports Accept-Language header
• Cloud storage sync like gdrive, dropbox, b2, s3 and more
• Asset column to hold file and blob data, backed by storage
• Multiple websites under separate sub-domain/sub-paths
• Connect with external APIs by using extension points
• Data View Streams
• Flexible data import (auto create new tables and automated schema generation)
  • XLSX
  • JSON
  • CSV

https://github.com/daptin/daptin-js-client

Starter kit: https://github.com/daptin/vue_typescript_starter_kit

Define Schema

Find

Get By Id

Create

Delete

Delete relations

List relations

• Versioning of the data
• Authentication and authorization
• JSON API endpoint
• GraphQL endpoint
• Actions and integrations with external services

• Native binary
• Heroku
• Docker image
• Kubernetes YAML

• CRUD APIs
  • Read, search, filter
  • Create
  • Update
  • Delete
  • Relations
  • Execute
• Action APIs
  • Using actions
  • Actions list
• User APIs
  • User registration/signup
  • User login/signin
• State tracking APIs
  • State machines

• Guests
• Adding users
• User groups
• Data access permission
• Social login

• Cloud storage

• Create a subsite

OpenAPI V3 spec is auto generated for each endpoint exposed. This can be use to generate further documentation and clients.

curl http://localhost/apispec.yaml

• Normalised Db Design from JSON schema upload
• Json Api, with CRUD and Relationships
• OAuth Authentication, inbuilt jwt token generator (setups up secret itself)
• Authorization based on a slightly modified linux FS permission model
• Objects and action chains
• State tracking using state machine
• Data connectors -> Incoming/Outgoing data
• Plugin system -> Grow the system according to your needs
• Native support for different data types (geo location/time/colors/measurements)
• Configurable intelligent Validation for data in the APIs
• Pages/Sub-sites -> Create a sub-site for a target audience
• Define events all around the system
• Data conversion/exchange/transformations
• Store connectors for storing big files/subsites - rclone
• Market place to allow plugins/extensions to be installed
• Online entity designer
• Excel to entity identification
• CSV to entity identification

Based on comprehensive testing of a fresh Daptin instance, here are the key findings:

Daptin excels at self-discoverability through:

• Comprehensive OpenAPI Documentation at /openapi.yaml with detailed endpoint descriptions, parameters, and examples
• Meta-Endpoints for runtime discovery:
  • /api/world - Lists all 56 available entities
  • /api/action - Shows available actions per entity
  • /action/world/download_system_schema - Exports complete system configuration
• JSON:API Compliance with consistent CRUD patterns
• Clear Authentication Flow with public signup/signin endpoints

Daptin provides good self-management capabilities:

Strengths:

• ✅ Dynamic entity creation via API
• ✅ Programmatic server restart (/action/world/restart_daptin)
• ✅ Multi-admin support via usergroups
• ✅ Schema export/import functionality
• ✅ Multiple data format exports (JSON, CSV, XML, PDF)

Limitations:

• ❌ Some actions restricted even for admins (generate_random_data, get_action_schema)
• ❌ Schema changes require server restart
• ❌ No built-in admin UI

# 1. Create user (8+ character password required)
curl -X POST http://localhost:6336/action/user_account/signup \
  -H "Content-Type: application/json" \
  -d '{"attributes": {"email": "admin@test.com", "password": "testpass123"}}'

# 2. Get JWT token
TOKEN=$(curl -X POST http://localhost:6336/action/user_account/signin \
  -H "Content-Type: application/json" \
  -d '{"attributes": {"email": "admin@test.com", "password": "testpass123"}}' \
  | jq -r '.[0].Attributes.value')

# 3. Become admin (ONE-TIME ONLY!)
curl -X POST http://localhost:6336/action/world/become_an_administrator \
  -H "Authorization: Bearer $TOKEN"

• Empty API responses? Check Authorization header
• Password errors? Use 8+ characters
• Schema not updated? Restart server after changes
• 403 errors? Verify token is valid and included

1. Unique Security Model: Before admin setup, ALL users have full access (permission: 2097151)
2. Multi-Admin Support: Add users to "administrators" usergroup for admin access
3. Token Management: JWT tokens valid for 3 days, always include Authorization: Bearer $TOKEN
4. Column Types: Extensive type system with validations (see /server/resource/column_types.go)

WebSocket authentication works via query parameter, not headers:

# WORKING WebSocket Connection
curl --include \
  --no-buffer \
  --header "Connection: Upgrade" \
  --header "Upgrade: websocket" \
  --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
  --header "Sec-WebSocket-Version: 13" \
  "ws://localhost:6336/live?token=$TOKEN"

WebSocket Features Discovered:

• Pub/Sub Messaging: Subscribe to database events and custom topics
• Permission-Aware: Events filtered based on user permissions
• Distributed: Uses Olric for cluster-wide messaging
• Auto Topics: One system topic per database table

WebSocket Message Methods:

// Subscribe to table events
{"method": "subscribe", "attributes": {"topicName": "user_account,document"}}

// Create custom topic
{"method": "create-topicName", "attributes": {"name": "chat-room-1"}}

// Publish message
{"method": "new-message", "attributes": {"topicName": "chat-room-1", "message": "Hello!"}}

// List all topics
{"method": "list-topicName", "attributes": {}}

Real-time document collaboration with conflict resolution:

YJS Endpoints Pattern:

• /live/{typename}/{referenceId}/{columnName}/yjs - WebSocket collaboration
• /yjs/{documentName} - Direct YJS access
• Any file.* column type gets automatic YJS endpoints

Working Example from dadadash:

const yjsProvider = new WebsocketProvider(
  `ws://localhost:6336/live/document/${referenceId}/content/yjs?token=${token}`,
  'document-room',
  ydoc,
  {
    awareness: {
      user: { name: 'User Name', color: '#ff0000' }
    }
  }
);

Built on go-guerrilla with enterprise features:

Email Infrastructure:

• Entities: mail_server, mail_account, mail_box, mail
• Security: TLS/SSL, DKIM signing, SPF verification
• Actions: mail.send, aws.mail.send
• IMAP: Full email retrieval support

Complete Email Setup:

# 1. Enable SMTP
curl -X PUT http://localhost:6336/_config/backend/smtp.enable \
  -H "Authorization: Bearer $TOKEN" -d '"true"'

# 2. Create mail server
curl -X POST http://localhost:6336/api/mail_server \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/vnd.api+json" \
  -d '{
    "data": {
      "type": "mail_server",
      "attributes": {
        "hostname": "smtp.yourdomain.com",
        "is_enabled": true,
        "listen_interface": "0.0.0.0:465",
        "always_on_tls": true
      }
    }
  }'

# 3. Send email
curl -X POST http://localhost:6336/action/world/mail.send \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"attributes": {"from": "noreply@yourdomain.com", "to": ["user@example.com"], "subject": "Test", "body": "Hello!"}}'

• CalDAV/CardDAV: /caldav/* and /carddav/* endpoints
• FTP Server: Multi-site support with cloud storage backends
• RSS/Atom Feeds: Auto-generation from any entity

Documented Features:

1. WebSocket server architecture
2. Pub/Sub messaging patterns
3. Permission-aware event filtering
4. Custom topic management
5. YJS collaborative editing
6. YJS document persistence
7. Multi-editor support (Quill, CodeMirror)
8. User presence/awareness
9. SMTP server implementation
10. Email entity schemas (4 tables)
11. DKIM/SPF security
12. IMAP email retrieval
13. CalDAV calendar sync
14. CardDAV contact sync
15. FTP file transfer
16. RSS feed generation
17. Atom feed support
18. JSON feed format
19. Feed configuration patterns

WebSocket Authentication:

• Use query parameter ?token=JWT_TOKEN not headers
• Token validation happens during WebSocket upgrade
• Same JWT tokens from signin work perfectly

YJS Integration:

• Requires file-type columns (file.document, file.spreadsheet, etc.)
• Documents stored as ZIP with YJS state + plain text
• Supports real-time presence and conflict resolution

Server Configuration:

• Most features toggle via /_config API
• Some changes need server restart (actions, world schema)
• Configuration stored in database, persists across restarts

Testing Approach:

• Always verify configuration changes took effect
• Check multiple related endpoints for full feature validation
• Use real-world examples (dadadash) for integration patterns

Powerful data aggregation with SQL-like capabilities via REST:

Endpoint Pattern: /aggregate/{entityName}

Working Examples:

# Group by with count
curl -X GET "http://localhost:6336/aggregate/world?group=is_hidden&column=is_hidden,count" \
  -H "Authorization: Bearer $TOKEN"

# Response:
{
  "data": [
    {"type": "aggregate_world", "attributes": {"is_hidden": 0, "count": 60}},
    {"type": "aggregate_world", "attributes": {"is_hidden": 1, "count": 1}}
  ]
}

# Sum with filter
curl -X GET "http://localhost:6336/aggregate/world?filter=eq(is_top_level,1)&column=count" \
  -H "Authorization: Bearer $TOKEN"

Aggregation Features:

• Functions: count, sum(col), avg(col), min(col), max(col), first(col), last(col)
• Filters: eq(), not(), lt(), lte(), gt(), gte(), in(), notin(), is(), not()
• Advanced: group by multiple columns, having clauses, joins, time sampling
• Methods: Both GET (query params) and POST (JSON body) supported

Auto-generated GraphQL schema from database:

Enable GraphQL (Requires Restart):

# Method 1: Configuration API
curl -X POST http://localhost:6336/_config/backend/graphql.enable \
  -H "Authorization: Bearer $TOKEN" -d 'true'

# Method 2: System Action (if available)
curl -X POST http://localhost:6336/action/world/__enable_graphql \
  -H "Authorization: Bearer $TOKEN" -d '{"attributes":{}}'

# Restart server
curl -X POST http://localhost:6336/action/world/restart_daptin \
  -H "Authorization: Bearer $TOKEN" -d '{"attributes":{}}'

GraphQL Features:

• Auto Schema: Generated from all tables and relationships
• Operations: Queries, mutations, subscriptions
• Relationships: Automatic traversal
• Actions: Execute via mutations
• Security: Disabled by default, admin-only enable

Enterprise-grade data migration with streaming:

Export Action:

# Export to CSV
curl -X POST "http://localhost:6336/api/{entity}/action/__data_export" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "attributes": {
      "format": "csv",        # json, csv, xlsx, pdf, html
      "table_name": "books",
      "include_headers": true,
      "columns": ["title", "created_at"],
      "page_size": 1000      # For streaming large datasets
    }
  }'

Import Action:

# Import from CSV
curl -X POST "http://localhost:6336/api/{entity}/action/__data_import" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "attributes": {
      "table_name": "books",
      "batch_size": 500,
      "truncate_before_insert": false,
      "dump_file": [{
        "name": "books.csv",
        "file": "data:text/csv;base64,..."
      }]
    }
  }'

Import/Export Features:

• Formats: JSON, CSV, XLSX, PDF (export), HTML (export)
• Streaming: Memory-efficient for large datasets
• Batch Processing: Configurable batch sizes
• Schema Creation: CSV/XLSX can create new tables
• Base64 Response: Browser-friendly downloads

JSON:API compliant relationship handling:

Include Related Data:

# Get world with related user_account
curl -X GET "http://localhost:6336/api/world/{id}?include=user_account_id" \
  -H "Authorization: Bearer $TOKEN"

Relationship Types:

• belongs_to: Many-to-one (foreign key on subject)
• has_one: One-to-one relationship
• has_many: One-to-many relationship
• many_to_many: Via join tables (auto-created)

Features:

• Automatic foreign key tracking
• Cascade operations support
• Lazy/eager loading via include parameter
• Join table auto-management

Documented Features:

1. Aggregation endpoint patterns
2. Aggregate function syntax (7 functions)
3. Filter function syntax (10+ operators)
4. Group by and having clauses
5. GraphQL enable process
6. GraphQL auto-schema generation
7. Import/Export action system
8. Streaming architecture patterns

Authentication Requirements:

• Aggregation endpoints require valid JWT tokens
• Admin privileges needed for some features
• Token in Authorization header: Bearer $TOKEN

Configuration Patterns:

• GraphQL disabled by default (security)
• Enable via /_config/backend/ namespace
• Some changes require restart (GraphQL, world schema)

Data Operations:

• Import/Export via actions, not REST endpoints
• Base64 encoding for file transfers
• Streaming support for large datasets

API Consistency:

• JSON:API spec for relationships
• Consistent error responses
• Pagination on all list endpoints

Database-backed configuration with runtime updates:

Configuration API Pattern: /_config/{configType}/{key}

18 Backend Configuration Parameters:

# Set configuration value
curl -X POST http://localhost:6336/_config/backend/graphql.enable \
  -H "Authorization: Bearer $TOKEN" -d 'false'

# Get configuration value (if set)
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:6336/_config/backend/graphql.enable

Documented Parameters:

1. graphql.enable - Enable/disable GraphQL endpoint
2. gzip.enable - Enable/disable GZIP compression
3. limit.rate - API rate limiting per second
4. yjs.enabled - Enable YJS collaborative editing
5. caldav.enable - Enable CalDAV calendar sync
6. ftp.enable - Enable FTP server
7. ftp.listen_interface - FTP server interface
8. imap.enabled - Enable IMAP email server
9. imap.listen_interface - IMAP server interface
10. jwt.secret - JWT signing secret
11. jwt.token.issuer - JWT issuer name
12. language.default - Default language
13. hostname - Server hostname
14. encryption.secret - Data encryption secret
15. totp.secret - TOTP 2FA secret
16. password.reset.email.from - Password reset sender
17. yjs.storage.path - YJS document storage path
18. caldav.enable - CalDAV server enable

Configuration Features:

• Stored in _config table in database
• Environment-aware (debug/test/release)
• Versioning with previous value tracking
• Admin-only access required
• Changes persist across restarts

Rate Limiting:

• Per-route rate limiting
• IP + path based limiting
• Default 500 requests/second
• Returns 429 on limit exceeded
• Configurable via limit.rate

GZIP Compression:

• Automatic compression for responses
• Enabled via gzip.enable config
• Content-Encoding: gzip header
• Works with all API endpoints

Caching Architecture:

• Olric Distributed Cache for cluster-wide caching
• File Cache for static assets
• Cache namespaces: assets-cache
• Size limits: 2MB max file size
• Compression threshold: 5KB
• Expiry times:
  • Default: 24 hours
  • Images: 7 days
  • Videos: 14 days
  • Text files: 1 day

CORS Configuration:

• Fully configurable CORS headers
• Credentials support enabled
• Wildcard methods allowed
• Per-origin configuration
• Preflight handling

Certificate Management:

• Self-signed certificate generation
• RSA 2048-bit keys
• 365-day validity
• Automatic TLS on port 6443
• Certificate storage encrypted

Security Headers:

• CORS headers on all responses
• Authentication via JWT Bearer tokens
• Admin-only configuration access

Subsite Features:

• Multiple sites on single instance
• Host-based routing
• Path-based routing
• Static file serving
• Cloud storage integration

Site Configuration:

• Entity: site table
• Admin permission required
• Dynamic site loading
• Template engine support

Documented Features:

1. Configuration API pattern
2. 18 configuration parameters
3. Runtime configuration updates
4. Database-backed config storage
5. Rate limiting implementation
6. Per-route rate configuration
7. GZIP compression support
8. Olric distributed cache
9. File cache with size limits
10. CORS configuration

Configuration Best Practices:

• Use /_config/backend/ for server settings
• Changes take effect immediately (except GraphQL)
• Store secrets encrypted
• Environment-specific values

Performance Optimization:

• Rate limiting prevents abuse
• GZIP reduces bandwidth
• Caching improves response times
• Distributed cache for scaling

Security Hardening:

• CORS properly configured
• TLS auto-enabled on 6443
• JWT tokens for all admin operations
• Configuration requires admin role

For detailed documentation and examples, see todo.md in this repository and the comprehensive OpenAPI documentation at /openapi.yaml.