# ekoDB > Developer documentation for ekoDB — the agentic backend platform and AI-native real-time database. API reference, client libraries (Rust, Python, TypeScript, Go, Kotlin), architecture, and guides. - [ekoDB](/index.md) ## search - [Search the documentation](/search.md) ## ai-agents Run AI agents next to your data. Deploy an agent in the cloud, give it tools, goals, and scheduled tasks, and let it build and operate your backend. - [AI Agents](/ai-agents.md): Run AI agents next to your data. Deploy an agent in the cloud, give it tools, goals, and scheduled tasks, and let it build and operate your backend. ## api-urls Learn how ekoDB database URLs are structured and how to connect to your database. - [API URLs](/api-urls.md): Learn how ekoDB database URLs are structured and how to connect to your database. ## authentication Learn how to get your API keys and authenticate with ekoDB. - [Authentication](/authentication.md): Learn how to get your API keys and authenticate with ekoDB. ## basic-operations Basic CRUD operations for managing individual records using the REST API. - [Basic Operations](/basic-operations.md): Basic CRUD operations for managing individual records using the REST API. ## batch-operations Efficiently handle large-scale data operations using batch processing. - [Batch Operations](/batch-operations.md): Efficiently handle large-scale data operations using batch processing. ## build-your-first-agent Go from signup to a working AI agent against your own data in about 10 minutes, using only the dashboard. - [Build Your First AI Agent in 10 Minutes](/build-your-first-agent.md): Go from signup to a working AI agent against your own data in about 10 minutes, using only the dashboard. ## chat-rag Build conversational AI applications with retrieval-augmented generation and context-aware responses. - [Chat & RAG (Retrieval-Augmented Generation)](/chat-rag.md): Build conversational AI applications with retrieval-augmented generation and context-aware responses. ## choose-your-path Find the right starting point for your use case. - [Choose Your Path](/choose-your-path.md): Find the right starting point for your use case. ## client-libraries Official ekoDB client libraries for multiple programming languages. - [Client Libraries](/client-libraries.md): Official ekoDB client libraries for multiple programming languages. ### advanced-operations Learn advanced features like search, chat, and real-time operations with ekoDB client libraries. - [Advanced Operations](/client-libraries/advanced-operations.md): Learn advanced features like search, chat, and real-time operations with ekoDB client libraries. ### basic-operations Learn CRUD operations with ekoDB client libraries. - [Basic Operations](/client-libraries/basic-operations.md): Learn CRUD operations with ekoDB client libraries. ### installation Install ekoDB client libraries for your programming language. - [Installation](/client-libraries/installation.md): Install ekoDB client libraries for your programming language. ### querying-with-filters Build complex queries with the QueryBuilder in ekoDB client libraries - [Querying with Filters](/client-libraries/querying-with-filters.md): Build complex queries with the QueryBuilder in ekoDB client libraries ## collections-and-schemas Manage collections, define schemas, and configure data validation rules. - [Collections & Schemas](/collections-and-schemas.md): Manage collections, define schemas, and configure data validation rules. ## configuration Comprehensive guide to configuring your ekoDB instance via the REST API. - [Configuration](/configuration.md): Comprehensive guide to configuring your ekoDB instance via the REST API. ## data-types Understanding ekoDB's comprehensive type system with 16 supported data types. - [Data Types](/data-types.md): Understanding ekoDB's comprehensive type system with 16 supported data types. ## getting-started Learn how to get started with ekoDB and deploy your first database. - [Getting Started](/getting-started.md): Learn how to get started with ekoDB and deploy your first database. ## indexes Create and manage query and search indexes for optimal performance. - [Indexes](/indexes.md): Create and manage query and search indexes for optimal performance. ## introduction ekoDB is an AI-native database you deploy in one click — run a database on its own, or add AI agents that connect to it to build and operate your backend. - [Introduction](/introduction.md): ekoDB is an AI-native database you deploy in one click — run a database on its own, or add AI agents that connect to it to build and operate your backend. ## kv-store Fast key-value storage with linking capabilities for caching and metadata. - [Key-Value Store](/kv-store.md): Fast key-value storage with linking capabilities for caching and metadata. ## patterns-cache-warming Intelligent cache warming based on query access patterns for faster cold starts. - [Query Patterns](/patterns-cache-warming.md): Intelligent cache warming based on query access patterns for faster cold starts. ## query-expressions Complete reference for building filters, queries, and expressions in ekoDB - [Query Expressions](/query-expressions.md): Complete reference for building filters, queries, and expressions in ekoDB ## quick-start Get up and running with ekoDB in 5 minutes - from deployment to your first query. - [Quick Start Guide](/quick-start.md): Get up and running with ekoDB in 5 minutes - from deployment to your first query. ## real-time Subscribe to live data changes via WebSocket or Server-Sent Events (SSE). - [Real-Time Subscriptions](/real-time.md): Subscribe to live data changes via WebSocket or Server-Sent Events (SSE). ## reference ### architecture-patterns Reference architectures for different application types using ekoDB. - [Architecture Patterns](/reference/architecture-patterns.md): Reference architectures for different application types using ekoDB. ### error-codes Comprehensive guide to ekoDB HTTP status codes, error responses, and troubleshooting. - [Error Codes Reference](/reference/error-codes.md): Comprehensive guide to ekoDB HTTP status codes, error responses, and troubleshooting. ### functions-architecture Deep dive into ekoDB's composable Functions system for server-side business logic - [Functions Architecture](/reference/functions-architecture.md): Deep dive into ekoDB's composable Functions system for server-side business logic ### performance YCSB and embedded database performance benchmarks for ekoDB. - [Performance Benchmarks](/reference/performance.md): YCSB and embedded database performance benchmarks for ekoDB. ### transaction-architecture Comprehensive guide to ekoDB's transactions architecture, isolation levels, and implementation details. - [Transactions Architecture](/reference/transaction-architecture.md): Comprehensive guide to ekoDB's transactions architecture, isolation levels, and implementation details. ## ripples Real-time data propagation and replication across ekoDB instances using the Ripple system. - [Ripples - Data Propagation](/ripples.md): Real-time data propagation and replication across ekoDB instances using the Ripple system. ## scheduled-functions Automate recurring tasks with cron-based scheduled function execution. - [Scheduled Functions](/scheduled-functions.md): Automate recurring tasks with cron-based scheduled function execution. ## security Learn about ekoDB's enterprise-grade security features and best practices. - [Security](/security.md): Learn about ekoDB's enterprise-grade security features and best practices. ## support Get help with ekoDB and share your feedback with our team. - [Support & Feedback](/support.md): Get help with ekoDB and share your feedback with our team. ## system-administration Monitor health, manage WAL, configure replication, and analyze system performance. - [System Administration](/system-administration.md): Monitor health, manage WAL, configure replication, and analyze system performance. ## transactions Atomic, durable, isolated transactions with enforced configurable MVCC isolation and savepoint support. - [Transactions](/transactions.md): Atomic, durable, isolated transactions with enforced configurable MVCC isolation and savepoint support. ## using-ekodb Overview of ekoDB's core concepts and documentation structure. - [Using ekoDB](/using-ekodb.md): Overview of ekoDB's core concepts and documentation structure. ## vector-search AI-powered semantic search with vector embeddings for similarity matching and recommendations. - [Vector Search & Embeddings](/vector-search.md): AI-powered semantic search with vector embeddings for similarity matching and recommendations. ## white-paper Technical overview and architecture of ekoDB database system. - [ekoDB White Paper](/white-paper.md): Technical overview and architecture of ekoDB database system. ## why-ekodb Learn what makes ekoDB different and why it's the right choice for your next application. - [Why ekoDB?](/why-ekodb.md): Learn what makes ekoDB different and why it's the right choice for your next application. ## Optional - [ekoDB Website](https://ekodb.io): Marketing site, feature comparison, and benchmarks - [ekoDB Dashboard](https://app.ekodb.io): Web application and database management console --- # Full Documentation Content [Skip to main content](#__docusaurus_skipToContent_fallback) [![ekoDB Logo](/img/logo.png)![ekoDB Logo](/img/logo.png)](/index.md) [**ekoDB Docs**](/index.md)[Quick Start](/quick-start.md)[Overview](/introduction.md)[Using ekoDB](/using-ekodb.md)[AI Agents](/ai-agents.md)[Architecture](/white-paper.md) [ekoDB App](https://app.ekodb.io) Search # Search the documentation Type your search here Powered by[](https://www.algolia.com/) Getting Started * [Quick Start](/quick-start.md) * [Why ekoDB?](/why-ekodb.md) * [Deploy Database](/getting-started.md) * [Authentication](/authentication.md) Client Libraries * [Overview](/client-libraries/.md) * [Installation](/client-libraries/installation.md) * [Basic Operations](/client-libraries/basic-operations.md) Architecture * [White Paper](/white-paper.md) * [Performance](/reference/performance.md) * [Transactions](/reference/transaction-architecture.md) * [Functions](/reference/functions-architecture.md) * [Error Codes](/reference/error-codes.md) Resources * [ekoDB App](https://app.ekodb.io) * [Client Libraries](https://github.com/ekoDB/ekodb-client) * [Examples](https://github.com/ekoDB/ekodb-client/tree/main/examples) * [ekoDB Website](https://ekodb.io) * [Support](/support.md) Copyright © 2026 ekoDB Inc. All rights reserved. --- # AI Agents An AI Agent is a managed deployment that runs next to your data and connects to an ekoDB database. It can answer questions about your collections, build backends from a template or from scratch, run multi-step workflows, and keep your database tuned, all without you wiring up a separate service. Part of the platform, not bolted on The agent is part of ekoDB: you deploy it from the same dashboard, and it uses your database's built-in RAG and vector search — there is no second vector database, no separate RAG pipeline, and no glue code to maintain. An agent always connects to a database; a database runs perfectly well on its own. ## How it runs[​](#how-it-runs "Direct link to How it runs") ekoDB runs in the cloud, and so does the agent. From the dashboard you deploy a **database** and, separately, an **AI Agent** — each is its own managed deployment. The agent connects to a database you've deployed (you pick which one), and both run as managed services. You never host or operate them yourself. Once an agent is deployed, you can work with it three ways: * **The dashboard**, in your browser, with a full chat and management UI. * **The desktop app**, a native client for macOS, Linux, and Windows. * **The terminal client**, an interactive agent in your shell. The desktop app and the terminal client are thin clients. They authenticate, discover your deployments, and connect to your cloud agent. They never run the backend or hold your API keys. See [Get Started](https://ekodb.io/get-started) for the client install commands. ## What an agent can do[​](#what-an-agent-can-do "Direct link to What an agent can do") * **Chat with your data.** Ask questions in plain English. The agent answers from your own collections using ekoDB's native RAG and vector search. * **Build your backend.** The agent assembles collections, server-side functions, and scheduled jobs. It can start from ready-made templates for common backends (APIs, data models, auth, automation), or build the whole thing from scratch when no template fits what you need. * **Run real workflows.** Give an agent a system prompt, scoped tool permissions, goals, and scheduled tasks that run on their own. * **Use built-in tools.** File operations, web search, HTTP fetch, persistent memory, and reusable templates all come built in. * **Tune and debug itself.** The agent can explain a slow query, spot a missing index and create it, surface which data is hot, and read server logs to find problems. ## Agents for you, or for your users[​](#agents-for-you-or-for-your-users "Direct link to Agents for you, or for your users") An agent can serve two audiences: * **Internal.** Run it for yourself and your team, from the dashboard or a client. * **Customer-facing.** Expose an agent through a public customer endpoint so your own users can chat with it. You control CORS allowlists, anonymous access, rate limits, and token lifetime. A customer token can never escalate past the agent's permission ceiling, so it is safe to put in front of the public. For customer-facing agents, prefer stored functions over raw read/write tools so behavior stays auditable and versioned. ## Models[​](#models "Direct link to Models") Bring your own model key. The agent supports the tool-calling providers: OpenAI, Anthropic (Claude), and Google (Gemini). Your key stays server-side and is never exposed to clients. ## Concepts[​](#concepts "Direct link to Concepts") * **Agent.** A scoped assistant with its own system prompt, allowed collections, tool permissions, and model. You can run many agents on one deployment. * **Goal.** A higher-level objective the agent works toward across multiple steps. * **Task.** A unit of work, including scheduled tasks that run on a cron-like cadence without you in the loop. * **Tools.** The capabilities an agent can invoke: querying data, calling stored functions, file and web operations, and more. Permissions are per-agent, so you can mark write-capable tools as confirm-only or deny. Irreversible operations — deleting, moving, or copying files and running shell commands — always ask for a per-call confirmation in the interactive client, even when the tool is otherwise allowed, so a single broad "allow always" can never be reused to silently delete or overwrite something later. * **Memory.** Persistent context the agent carries across sessions. * **Templates.** Complete backend recipes (collections plus functions plus schedules) the agent applies in one step. ## Next steps[​](#next-steps "Direct link to Next steps") * [Build your first AI agent in 10 minutes](/build-your-first-agent.md) walks you from signup to a working agent against your own data. * [Chat & RAG](/chat-rag.md) covers the underlying chat and retrieval primitives the agent builds on. * [Functions](/scheduled-functions.md) covers the server-side functions an agent composes into backends. --- # API URLs Each database is namespaced by url: ``` {DATABASE_SUBDOMAIN}.{DATABASE_ENVIRONMENT}.{DATABASE_COMPUTE_PROVIDER}.ekodb.net ``` ## URL Structure[​](#url-structure "Direct link to URL Structure") **Example URLs:** ``` myapp.production.google.ekodb.net testdb.staging.google.ekodb.net dev-db.development.google.ekodb.net ``` ### Components[​](#components "Direct link to Components") * **`{DATABASE_SUBDOMAIN}`** - Your unique database identifier * **`{DATABASE_ENVIRONMENT}`** - Environment type (production, staging, development) * **`{DATABASE_COMPUTE_PROVIDER}`** - Cloud provider (google, azure, aws, etc.) * **`ekodb.net`** - ekoDB domain ## Supported Protocols[​](#supported-protocols "Direct link to Supported Protocols") Providers will vary based on availability of the region and other factors. ### ✅ Supported (Secure Only)[​](#-supported-secure-only "Direct link to ✅ Supported (Secure Only)") * **HTTPS** - `https://your-db.production.google.ekodb.net` * **WSS** - `wss://your-db.production.google.ekodb.net` ### ❌ Not Supported[​](#-not-supported "Direct link to ❌ Not Supported") * HTTP, FTP, TCP, UDP, and other insecure protocols ## Authentication[​](#authentication "Direct link to Authentication") Every connection requires a valid JWT token provided by ekoDB. **Exception:** The public endpoints `/api/health`, `/api/auth/register`, and `/api/auth/token` do not require authentication. All other APIs are protected by authentication. ### Getting Your API URL[​](#getting-your-api-url "Direct link to Getting Your API URL") 1. **Deploy a database** via [app.ekodb.io](https://app.ekodb.io) 2. **Find your URL** in the database dashboard 3. **Generate API key** for authentication 4. **Connect** using your preferred client library or REST API ## Example Connection[​](#example-connection "Direct link to Example Connection") ``` # Base URL BASE_URL="https://myapp.production.google.ekodb.net" # Health check (no auth required) curl $BASE_URL/api/health # Authenticated request curl -H "Authorization: Bearer YOUR_JWT_TOKEN" \ $BASE_URL/api/collections ``` ## Regional Availability[​](#regional-availability "Direct link to Regional Availability") Database URLs are assigned based on: * **Region selection** during deployment * **Provider availability** in that region * **Resource optimization** for performance For questions about URL structure or regional availability, contact . --- # Authentication ## Getting Your API Key[​](#getting-your-api-key "Direct link to Getting Your API Key") After deploying your ekoDB instance, you'll need an API key to connect your application. ### Step 1: Access Your Deployment[​](#step-1-access-your-deployment "Direct link to Step 1: Access Your Deployment") 1. Go to [app.ekodb.io](https://app.ekodb.io) 2. Navigate to your deployment from the dashboard 3. Click on the **"Keys"** tab Direct link format: `https://app.ekodb.io/deployments/{YOUR_DEPLOYMENT_ID}?active_tab=keys` ### Step 2: Use Your Admin API Key[​](#step-2-use-your-admin-api-key "Direct link to Step 2: Use Your Admin API Key") By default, ekoDB automatically generates an **admin API key** when you create a deployment. This key has full access to all collections and operations. Admin Keys Are Sensitive Admin API keys should **only be used for development and testing**. For production applications, create API keys with limited permissions (see below). ### Step 3: Copy Your API Key[​](#step-3-copy-your-api-key "Direct link to Step 3: Copy Your API Key") Copy the API key from the dashboard and store it securely. You'll use this to initialize your client library. Initialize Your Client Once you have your API key, see the [Client Libraries Installation Guide](/client-libraries/installation.md#configuration) for complete code examples on how to configure your client in TypeScript, Rust, Python, Go, or Kotlin. Client Libraries Handle Authentication When using ekoDB client libraries (Rust, Python, TypeScript, Go, Kotlin), you don't need to manually manage JWT tokens. The client automatically handles token generation and renewal for you. Simply provide your API key when initializing the client. ## Creating API Keys with Limited Permissions[​](#creating-api-keys-with-limited-permissions "Direct link to Creating API Keys with Limited Permissions") For production applications, create API keys with specific collection permissions. ### Register API Key[​](#register-api-key "Direct link to Register API Key") Register a new API key using the REST API. You must have an admin token to create new keys: * curl * HTTP ``` curl -X POST https://{EKODB_API_URL}/api/auth/register \ -H "Content-Type: application/json" \ -d '{ "api_key": "{ADMIN_KEY}", "label": "{LABEL}", "description": "{DESCRIPTION}", "collection_permissions": { "{COLLECTION_NAME}": { "read": true, "write": true } }, "is_admin": false }' ``` ``` POST https://{EKODB_API_URL}/api/auth/register Content-Type: application/json { "api_key": "{ADMIN_KEY}", "label": "{LABEL}", "description": "{DESCRIPTION}", "collection_permissions": { "{COLLECTION_NAME}": { "read": true, "write": true } }, "is_admin": false } ``` **Response:** ``` { "label": "{LABEL}", "description": "{DESCRIPTION}", "collection_permissions": { "{COLLECTION_NAME}": { "read": true, "write": true } }, "is_admin": false, "key": "{YOUR_NEW_API_KEY}" } ``` ### Using Your New API Key[​](#using-your-new-api-key "Direct link to Using Your New API Key") Once you have a scoped API key, use it to initialize your client: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://your-subdomain.production.google.ekodb.net") .api_key("{YOUR_NEW_API_KEY}") .build()?; ``` ``` from ekodb_client import Client client = Client.new( "https://your-subdomain.production.google.ekodb.net", "{YOUR_NEW_API_KEY}" ) ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "{YOUR_NEW_API_KEY}" }); await client.init(); ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "{YOUR_NEW_API_KEY}" }); await client.init(); ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://your-subdomain.production.google.ekodb.net") .apiKey("{YOUR_NEW_API_KEY}") .build() ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://your-subdomain.production.google.ekodb.net", "{YOUR_NEW_API_KEY}", ) ``` ## Generate JWT Token[​](#generate-jwt-token "Direct link to Generate JWT Token") Generate a JWT token using your API key: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go * 🌐 HTTP ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://your-subdomain.production.google.ekodb.net") .api_key("YOUR_API_KEY") .build()?; // Client automatically handles JWT token generation // No manual token generation needed when using the client library ``` ``` from ekodb_client import Client client = Client.new( "https://your-subdomain.production.google.ekodb.net", "YOUR_API_KEY" ) # Client automatically handles JWT token generation # No manual token generation needed when using the client library ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "YOUR_API_KEY" }); await client.init(); // Client automatically handles JWT token generation // No manual token generation needed when using the client library ``` ``` const response = await fetch( 'https://{EKODB_API_URL}/api/auth/token', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ api_key: '{YOUR_API_KEY}' }) }); const data = await response.json(); const token = data.token; ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://your-subdomain.production.google.ekodb.net") .apiKey("YOUR_API_KEY") .build() // Client automatically handles JWT token generation // No manual token generation needed when using the client library ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://your-subdomain.production.google.ekodb.net", "YOUR_API_KEY", ) // Client automatically handles JWT token generation // No manual token generation needed when using the client library ``` ``` POST https://{EKODB_API_URL}/api/auth/token Content-Type: application/json { "api_key": "{YOUR_API_KEY}" } # Response { "token": "{YOUR_JWT_TOKEN}" } ``` Client Libraries Handle Token Management When using ekoDB client libraries (Rust, Python, TypeScript, Go, Kotlin), you don't need to manually generate JWT tokens. The client automatically handles token generation, caching, and renewal. Only use direct API calls for token generation if you're not using a client library. ## Use JWT Token[​](#use-jwt-token "Direct link to Use JWT Token") Use the JWT token in the Authorization header: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go * 🌐 HTTP ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://your-subdomain.production.google.ekodb.net") .api_key("YOUR_API_KEY") .build()?; // Client automatically handles authentication let collections = client.list_collections().await?; println!("Collections: {:?}", collections); ``` ``` from ekodb_client import Client client = Client.new( "https://your-subdomain.production.google.ekodb.net", "YOUR_API_KEY" ) # Client automatically handles authentication collections = client.list_collections() print("Collections:", collections) ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "YOUR_API_KEY" }); await client.init(); // Client automatically handles authentication const collections = await client.listCollections(); console.log("Collections:", collections); ``` ``` const response = await fetch( 'https://{EKODB_API_URL}/api/collections', { headers: { 'Authorization': 'Bearer {YOUR_JWT_TOKEN}' } }); const collections = await response.json(); // collections now contains the list of available collections ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://your-subdomain.production.google.ekodb.net") .apiKey("YOUR_API_KEY") .build() // Client automatically handles authentication val collections = client.listCollections() println("Collections: $collections") ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://your-subdomain.production.google.ekodb.net", "YOUR_API_KEY", ) // Client automatically handles authentication collections, err := client.ListCollections() if err != nil { panic(err) } fmt.Println("Collections:", collections) ``` ``` GET https://{EKODB_API_URL}/api/collections Authorization: Bearer {YOUR_JWT_TOKEN} ``` Client Libraries Handle Authentication Automatically When using ekoDB client libraries (Rust, Python, TypeScript, Go, Kotlin), you don't need to manually manage JWT tokens or set Authorization headers. Simply initialize the client with your API key, and all subsequent requests are automatically authenticated. ## API Key Management[​](#api-key-management "Direct link to API Key Management") Manage API keys for your ekoDB instance. ### List All API Keys[​](#list-all-api-keys "Direct link to List All API Keys") Get a list of all API keys (admin only). ``` GET https://{EKODB_API_URL}/api/auth/keys Authorization: Bearer {ADMIN_TOKEN} # Response [ { "key": "eko_prod_abc123...", "label": "Production API", "description": "Main production key", "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true, "select_fields": null, "exclude_fields": null, "writable_fields": null }, "posts": { "read": true, "write": false, "select_fields": null, "exclude_fields": null, "writable_fields": null } } }, { "key": "eko_admin_def456...", "label": "Admin Key", "description": "Full admin access", "is_admin": true, "collection_permissions": {} } ] ``` ### Revoke API Key[​](#revoke-api-key "Direct link to Revoke API Key") Permanently revoke an API key (admin only). ``` DELETE https://{EKODB_API_URL}/api/auth/keys/{api_key} Authorization: Bearer {ADMIN_TOKEN} # Response { "status": "success", "message": "Key revoked successfully" } ``` The path parameter is the API key value itself (the `eko_...` string), not a separate key id — ekoDB identifies keys by their value. Revocation is Permanent Once revoked, an API key cannot be restored. Applications using the revoked key will immediately lose access. ### Get User Permissions[​](#get-user-permissions "Direct link to Get User Permissions") Get the permissions for the currently authenticated user. ``` GET https://{EKODB_API_URL}/api/auth/permissions Authorization: Bearer {YOUR_API_TOKEN} # Response { "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true }, "posts": { "read": true, "write": false }, "comments": { "read": true, "write": true } } } ``` Each entry in `collection_permissions` is an object with `read`/`write` booleans (and optional `select_fields`/`exclude_fields`/`writable_fields` for field-level scoping), not an array of strings. ### Set Collection Permissions (own token)[​](#set-collection-permissions-own-token "Direct link to Set Collection Permissions (own token)") Grant or change the permissions for a single collection on the **calling token**, and receive a refreshed JWT that reflects the change (admin only). The body specifies one collection plus its `read`/`write` flags; optional `select_fields`/`exclude_fields`/`writable_fields` apply field-level scoping. ``` POST https://{EKODB_API_URL}/api/auth/permissions Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "collection": "users", "read": true, "write": true } # Response { "token": "" } ``` ### Update Another Key's Collection Permissions[​](#update-another-keys-collection-permissions "Direct link to Update Another Key's Collection Permissions") Set the permissions for a single collection on a **specific API key** (admin only). The key is identified by its value in the `api_key` field. ``` PUT https://{EKODB_API_URL}/api/auth/permissions Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "api_key": "eko_live_abc123...", "collection": "posts", "read": true, "write": true } # Response — the updated API key object { "key": "eko_live_abc123...", "label": "Web App Key", "is_admin": false, "collection_permissions": { "posts": { "read": true, "write": true } } } ``` Each call sets one collection at a time. To change several collections, send one request per collection. ## Permission Levels[​](#permission-levels "Direct link to Permission Levels") ### Collection Permissions[​](#collection-permissions "Direct link to Collection Permissions") | Permission | Description | Operations Allowed | | ---------- | ---------------------------------- | ----------------------------------- | | `read` | View records in the collection | GET, query, search | | `write` | Create, update, and delete records | POST (insert), PUT (update), DELETE | note There are only two collection permissions: `read` and `write`. Deleting records is governed by `write` (there is no separate `delete` permission). Granting `write` allows insert, update, and delete. ### Admin Permissions[​](#admin-permissions "Direct link to Admin Permissions") Admin keys have full access to: * All collections (read and write) * Collection creation and deletion * Schema management * Index management * API key management * System administration Security Best Practice Use admin keys only for initial setup and administrative tasks. For application access, create API keys with minimal required permissions. ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete authentication workflow: ``` #!/bin/bash # 1. Create a new API key with specific permissions response=$(curl -s -X POST https://{EKODB_API_URL}/api/auth/register \ -H "Content-Type: application/json" \ -d '{ "api_key": "{ADMIN_KEY}", "label": "Web App Key", "description": "API key for web application", "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true }, "posts": { "read": true, "write": true }, "comments": { "read": true, "write": true } } }') new_api_key=$(echo $response | jq -r '.key') echo "New API Key: $new_api_key" # 2. Generate JWT token using the new key token_response=$(curl -s -X POST https://{EKODB_API_URL}/api/auth/token \ -H "Content-Type: application/json" \ -d "{\"api_key\": \"$new_api_key\"}") jwt_token=$(echo $token_response | jq -r '.token') echo "JWT Token: $jwt_token" # 3. Use the token to access allowed collections curl -X GET https://{EKODB_API_URL}/api/find/users \ -H "Authorization: Bearer $jwt_token" # 4. Verify permissions curl -X GET https://{EKODB_API_URL}/api/auth/permissions \ -H "Authorization: Bearer $jwt_token" # 5. Update one collection's permissions on that key (requires admin key) curl -X PUT https://{EKODB_API_URL}/api/auth/permissions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d "{ \"api_key\": \"$new_api_key\", \"collection\": \"posts\", \"read\": true, \"write\": false }" # 6. List all API keys (admin only) curl -X GET https://{EKODB_API_URL}/api/auth/keys \ -H "Authorization: Bearer {ADMIN_TOKEN}" # 7. Revoke a key when no longer needed (path param is the key value) curl -X DELETE https://{EKODB_API_URL}/api/auth/keys/$new_api_key \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### API Key Security[​](#api-key-security "Direct link to API Key Security") **Do:** * ✅ Store API keys securely (environment variables, secrets manager) * ✅ Use different keys for development, staging, and production * ✅ Create keys with minimal required permissions * ✅ Rotate keys regularly * ✅ Revoke keys immediately if compromised **Don't:** * ❌ Hardcode API keys in source code * ❌ Commit keys to version control * ❌ Share admin keys with applications * ❌ Use the same key across multiple environments * ❌ Expose keys in client-side code ### Permission Strategy[​](#permission-strategy "Direct link to Permission Strategy") **Principle of Least Privilege:** ``` # Good: Minimal permissions { "users": { "read": true, "write": false }, # Read-only for users "analytics": { "read": false, "write": false } # No access to analytics } # Avoid: Excessive permissions { "users": { "read": true, "write": true }, # More than needed if you only read "is_admin": true # Full access when not required } ``` ### Token Lifecycle[​](#token-lifecycle "Direct link to Token Lifecycle") **JWT Token Expiration:** JWT tokens expire after a period (default: 1 hour). When using direct HTTP API, implement token refresh: ``` # Check if token is expired if token_expired; then # Regenerate token new_token=$(curl -X POST .../api/auth/token \ -d '{"api_key": "YOUR_KEY"}' | jq -r '.token') fi ``` Client Libraries Handle This When using ekoDB client libraries, token refresh is automatic. You don't need to manually check expiration. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Common Authentication Errors[​](#common-authentication-errors "Direct link to Common Authentication Errors") **401 Unauthorized:** ``` { "error": "Invalid or expired token" } ``` **Solution:** Regenerate JWT token using your API key. **403 Forbidden:** ``` { "error": "No write permission for collection users" } ``` **Solution:** Update API key permissions to include required access. **Invalid API Key:** ``` { "error": "API key not found or revoked" } ``` **Solution:** Verify the API key is correct and hasn't been revoked. ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - Using authenticated requests * [Collections & Schemas](/collections-and-schemas.md) - Managing collections Great! You can now use the JWT token to authenticate your requests to the ekoDB API. Next, you can learn how to use the ekoDB API to manage your data! If you have any issues, please email to create a ticket. --- # Basic Operations Manage individual records using ekoDB's REST API. All operations support JSON payloads and return structured responses. ## Authentication[​](#authentication "Direct link to Authentication") All API requests require authentication via JWT tokens. Generate a token from your API key: ``` curl -X POST https://{EKODB_API_URL}/api/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key": "YOUR_API_KEY"}' ``` **Response:** ``` { "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." } ``` Use this token in all subsequent requests: ``` -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` Token Expiration Tokens expire after 1 hour by default. Client libraries handle token refresh automatically. ## Configuration[​](#configuration "Direct link to Configuration") Update ekoDB instance configuration via the REST API. ``` PUT https://{EKODB_API_URL}/api/config Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "use_typed_values": true } # Response { "status": "success", "use_typed_values": true } ``` **Common Configuration Options:** | Option | Type | Default | Description | | -------------------- | ------- | -------- | -------------------------------------- | | `use_typed_values` | boolean | `true` | Enable typed field values for records | | `storage_mode` | string | `"fast"` | Record persistence strategy (below) | | `durable_operations` | boolean | `true` | Enable fsync on every write | | `group_commits` | boolean | `true` | Batch concurrent writes for throughput | Full Configuration Reference See [Configuration](/configuration.md) for the complete list of all available options including LLM providers, chat settings, resource allocation, write performance tuning, and more. ### Storage Mode[​](#storage-mode "Direct link to Storage Mode") Control how ekoDB persists records to disk. Choose based on your workload characteristics. ``` PUT https://{EKODB_API_URL}/api/config Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "storage_mode": "balanced" } ``` **Available Modes:** | Mode | Write Speed | Read Speed | Disk Usage | Best For | | ---------- | ----------- | ---------- | ---------------- | -------------------------------------------- | | `fast` | Fastest | Fastest | WAL only | Low-latency reads, WAL durability (default) | | `balanced` | Fast | Fastest | Periodic batches | General web apps, APIs with disk persistence | **Mode Details:** * **`fast`**: In-memory + WAL only, no individual record files. Maximum throughput with data recoverable from WAL on restart. * **`balanced`**: Fast critical path (same as fast mode) + periodic checkpoint batching to disk. Near-fast-mode performance with eventual disk persistence. Choosing a Storage Mode * Use **`fast`** (default) for most workloads - provides low-latency reads with WAL-based durability * Use **`balanced`** when you need periodic disk checkpoints for additional persistence * Use **`fast`** for write-heavy workloads like logging, events, or time-series data ## Insert Record[​](#insert-record "Direct link to Insert Record") Create a new record in a collection. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::{Client, Record}; let mut record = Record::new(); record.insert("name", "John Doe"); record.insert("email", "john@example.com"); record.insert("age", 30); let result = client.insert("users", record, None).await?; println!("Record ID: {}", result.get_string("id").unwrap_or_default()); ``` ``` from ekodb_client import Client client = Client.new( os.getenv('EKODB_URL'), os.getenv('EKODB_API_KEY') ) result = await client.insert('users', { 'name': 'John Doe', 'email': 'john@example.com', 'age': 30 }) print(result['id']) # record_id_123 ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY, }); await client.init(); const result = await client.insert("users", { name: "John Doe", email: "john@example.com", age: 30, }); console.log(result.id); // record_id_123 ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY, }); await client.init(); const result = await client.insert("users", { name: "John Doe", email: "john@example.com", age: 30, }); console.log(result.id); // record_id_123 ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl(System.getenv("EKODB_URL")) .apiKey(System.getenv("EKODB_API_KEY")) .build() val result = client.insert("users", mapOf( "name" to "John Doe", "email" to "john@example.com", "age" to 30 )) println("Record ID: ${result.id}") ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( os.Getenv("EKODB_URL"), os.Getenv("EKODB_API_KEY"), ) result, err := client.Insert("users", map[string]interface{}{ "name": "John Doe", "email": "john@example.com", "age": 30, }) fmt.Println("Record ID:", result.ID) ``` ``` curl -X POST https://{EKODB_API_URL}/api/insert/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "John Doe", "email": "john@example.com", "age": 30 }' # Response { "id": "record_id_123" } ``` Get Full Record After Insert The insert operation returns only the record ID. To retrieve the full record with all fields, use a GET request with the returned ID. **Disable Real-Time Data Propagation (Ripples):** By default, all write operations (insert, update, delete) automatically propagate to configured peer nodes via the Ripple system. Use `bypass_ripple=true` to skip propagation for specific operations: **When to bypass:** * Background jobs that process locally * Temporary data that doesn't need replication * Development/testing operations * Operations during maintenance windows ``` POST https://{EKODB_API_URL}/api/insert/{collection}?bypass_ripple=true Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "name": "Local Processing User", "email": "local@example.com" } ``` ## Get Record by ID[​](#get-record-by-id "Direct link to Get Record by ID") Retrieve a specific record by its ID. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let record = client.find_by_id("users", "record_id_123").await?; println!("{:?}", record); ``` ``` record = await client.find_by_id('users', 'record_id_123') print(record) # { # 'id': 'record_id_123', # 'name': 'John Doe', # 'email': 'john@example.com', # 'age': 30, # 'created_at': '2026-01-15T10:30:00Z' # } ``` ``` const record = await client.findById("users", "record_id_123"); console.log(record); // { // id: 'record_id_123', // name: 'John Doe', // email: 'john@example.com', // age: 30, // created_at: '2026-01-15T10:30:00Z' // } ``` ``` const record = await client.findById("users", "record_id_123"); console.log(record); // { // id: 'record_id_123', // name: 'John Doe', // email: 'john@example.com', // age: 30, // created_at: '2026-01-15T10:30:00Z' // } ``` ``` val record = client.findById("users", "record_id_123") println(record) ``` ``` record, err := client.FindByID("users", "record_id_123") fmt.Println(record) ``` ``` curl https://{EKODB_API_URL}/api/find/users/record_id_123 \ -H "Authorization: Bearer {TOKEN}" # Response { "id": "record_id_123", "name": "John Doe", "email": "john@example.com", "age": 30, "created_at": "2026-01-15T10:30:00Z" } ``` ## Query Records[​](#query-records "Direct link to Query Records") Query records with filters, sorting, and pagination. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::QueryBuilder; let query = QueryBuilder::new() .gt("age", 25) .sort_desc("created_at") .limit(10) .skip(0) .build(); let results = client.find("users", query, None).await?; println!("{:?}", results); ``` ``` from ekodb_client import QueryBuilder query = QueryBuilder().gt("age", 25).sort_desc("created_at").limit(10).build() results = await client.find("users", query) print(results) ``` ``` import { QueryBuilder } from "@ekodb/ekodb-client"; const query = new QueryBuilder() .gt("age", 25) .sortDesc("created_at") .limit(10) .build(); const results = await client.find("users", query); console.log(results); ``` ``` const { QueryBuilder } = require("@ekodb/ekodb-client"); const query = new QueryBuilder() .gt("age", 25) .sortDesc("created_at") .limit(10) .build(); const results = await client.find("users", query); console.log(results); ``` ``` val query = QueryBuilder() .gt("age", 25) .sortDesc("created_at") .limit(10) .build() val results = client.find("users", query) println(results) ``` ``` query := ekodb.NewQueryBuilder(). Gt("age", 25). SortDescending("created_at"). Limit(10). Build() results, err := client.Find("users", query) fmt.Println(results) ``` ``` curl -X POST https://{EKODB_API_URL}/api/find/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Condition", "content": { "field": "age", "operator": "Gt", "value": 25 } }, "sort": [ { "field": "created_at", "ascending": false } ], "limit": 10, "skip": 0 }' # Response [ { "id": "record_1", "name": "John Doe", "age": 30 }, { "id": "record_2", "name": "Jane Smith", "age": 28 } ] ``` See [Query Expressions](/query-expressions.md) for detailed filter syntax. **Query Operators:** | Short Form | Long Form | Description | | ------------ | -------------------- | -------------------------------- | | `Eq` | `Equals` | Exact match | | `Ne` | `NotEquals` | Not equal to | | `Gt` | `GreaterThan` | Greater than | | `Lt` | `LessThan` | Less than | | `Gte` | `GreaterThanOrEqual` | Greater than or equal | | `Lte` | `LessThanOrEqual` | Less than or equal | | `Contains` | - | String contains (case-sensitive) | | `StartsWith` | - | String starts with | | `EndsWith` | - | String ends with | Logical Operators For combining filters with AND, OR, and NOT operations, see the [Query Expressions](/query-expressions.md#logical-operators) reference for the correct JSON structure. **Combine Filters with AND:** ``` { "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gt", "value": 25 } }, { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } } ] } } } ``` **Combine Filters with OR:** ``` { "filter": { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "admin" } }, { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "moderator" } } ] } } } ``` **Complex Nested Filters:** Find active users who are either admins OR (have age > 25 AND verified): ``` { "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } }, { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "admin" } }, { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gt", "value": 25 } }, { "type": "Condition", "content": { "field": "verified", "operator": "Eq", "value": true } } ] } } ] } } ] } } } ``` Advanced Queries For complex queries with nested filters and advanced operations, see the [Query Expressions](/query-expressions.md) reference. Using a client library? See [Client Libraries - Querying with Filters](/client-libraries/querying-with-filters.md). ## Update Record[​](#update-record "Direct link to Update Record") Update an existing record by ID. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let mut updates = Record::new(); updates.insert("email", "newemail@example.com"); updates.insert("age", 31); let result = client.update("users", "record_id_123", updates).await?; println!("Record ID: {}", result.get_string("id").unwrap_or_default()); ``` ``` result = await client.update('users', 'record_id_123', { 'email': 'newemail@example.com', 'age': 31 }) print(result['id']) # record_id_123 ``` ``` const result = await client.update("users", "record_id_123", { email: "newemail@example.com", age: 31, }); console.log(result.id); // record_id_123 ``` ``` const result = await client.update("users", "record_id_123", { email: "newemail@example.com", age: 31, }); console.log(result.id); // record_id_123 ``` ``` val result = client.update("users", "record_id_123", mapOf( "email" to "newemail@example.com", "age" to 31 )) println("Record ID: ${result.id}") ``` ``` result, err := client.Update("users", "record_id_123", map[string]interface{}{ "email": "newemail@example.com", "age": 31, }) fmt.Println("Record ID:", result.ID) ``` ``` curl -X PUT https://{EKODB_API_URL}/api/update/users/record_id_123 \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "email": "newemail@example.com", "age": 31 }' # Response { "id": "record_id_123", "message": "Record updated successfully" } ``` Partial Updates Only the fields provided in the request body are updated. Existing fields remain unchanged. ## Search Records[​](#search-records "Direct link to Search Records") Full-text search (via the `query` string) and vector similarity search (via a `vector` array) across records. ``` POST https://{EKODB_API_URL}/api/search/{collection} Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "query": "software engineer", "limit": 10 } # Response { "results": [ { "record": { "id": "record_1", "name": "John Doe", "title": "Senior Software Engineer" }, "score": 0.95 } ], "total": 1 } ``` **Search with Vector Similarity:** Semantic search runs when you pass a `vector` array (the query embedding), not a text `query`. The `vector_threshold` field sets the minimum similarity score. ``` POST https://{EKODB_API_URL}/api/search/{collection} Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "vector": [0.12, -0.04, 0.88, 0.31, ...], "vector_field": "embedding", "limit": 5, "vector_threshold": 0.8 } ``` **GET Alternative:** ``` GET https://{EKODB_API_URL}/api/search/{collection}?query=software%20engineer&limit=10 Authorization: Bearer {YOUR_API_TOKEN} ``` ## Delete Record[​](#delete-record "Direct link to Delete Record") Delete a record by ID. Deleted records move to trash and can be restored until they are permanently removed. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.delete("users", "record_id_123").await?; // Confirms deletion; returns no body (fast path — no record load) ``` ``` await client.delete('users', 'record_id_123') # Confirms deletion; returns None (fast path) ``` ``` await client.delete("users", "record_id_123"); // Resolves on success; returns void (fast path) ``` ``` await client.delete("users", "record_id_123"); // Resolves on success; returns void (fast path) ``` ``` client.delete("users", "record_id_123") // Throws on failure; returns Unit on success (fast path) ``` ``` err := client.Delete("users", "record_id_123") // Returns only an error; nil on success (fast path) ``` ``` # Fast path (default): minimal ack, no record body loaded curl -X DELETE https://{EKODB_API_URL}/api/delete/users/record_id_123 \ -H "Authorization: Bearer {TOKEN}" # Response { "id": "record_id_123", "deleted": true } # Opt in to the full deleted record (loads the body — useful for undo/audit) curl -X DELETE "https://{EKODB_API_URL}/api/delete/users/record_id_123?return_record=true" \ -H "Authorization: Bearer {TOKEN}" # Response { "id": "record_id_123", "name": "John Doe", "email": "john@example.com", "age": 30 } ``` Fast by default By default, delete returns a minimal `{id, deleted:true}` ack and does **not** read the record from disk — keeping deletes as fast as inserts. Pass `?return_record=true` (or set `delete_returns_record: true` in config) to load and return the full deleted record, useful for undo operations or audit logs. **Disable Real-Time Sync:** ``` DELETE https://{EKODB_API_URL}/api/delete/{collection}/{record_id}?bypass_ripple=true Authorization: Bearer {YOUR_API_TOKEN} ``` ## Collection-Level Operations[​](#collection-level-operations "Direct link to Collection-Level Operations") Full Collection Management For complete schema definitions, field types, and advanced collection configuration, see [Collections & Schemas](/collections-and-schemas.md). ### Create Collection[​](#create-collection "Direct link to Create Collection") Create a new collection with an optional schema. ``` POST https://{EKODB_API_URL}/api/collections/{collection} Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "fields": { "name": { "type": "string", "required": true }, "email": { "type": "string", "required": true }, "age": { "type": "number" } } } # Response { "status": "success", "message": "Collection initialized successfully" } ``` Working Examples **Want to use the REST API directly?** See examples in [JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/collection_management.js), [Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/collection_management.py), [Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/collection_management.go), or [Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/collection_management.rs) **Prefer a client library?** See examples in [Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_collection_management.rs), [Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_collection_management.py), [TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_collection_management.ts), [Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_collection_management.go), or [Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientCollectionManagement.kt) ### List Collections[​](#list-collections "Direct link to List Collections") Get a list of all collections in the database. ``` GET https://{EKODB_API_URL}/api/collections Authorization: Bearer {YOUR_API_TOKEN} # Response { "collections": [ { "name": "users", "record_count": 1234, "created_at": "2026-01-10T08:00:00Z" }, { "name": "products", "record_count": 567, "created_at": "2026-01-12T10:30:00Z" } ] } ``` ### Get Collection Info[​](#get-collection-info "Direct link to Get Collection Info") Retrieve information about a specific collection. ``` GET https://{EKODB_API_URL}/api/collections/{collection} Authorization: Bearer {YOUR_API_TOKEN} # Response { "name": "users", "record_count": 1234, "schema": { ... }, "created_at": "2026-01-10T08:00:00Z" } ``` ### Update Schema[​](#update-schema "Direct link to Update Schema") Update validation constraints for an existing collection schema. ``` PUT https://{EKODB_API_URL}/api/schemas/{collection} Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "constraints": { "email": { "required": true, "unique": true } } } # Response { "status": "success", "message": "Schema constraints updated successfully" } ``` Schema Updates Schema updates only affect **new records**. Existing records are not automatically migrated or validated against the new schema. ### Delete Collection[​](#delete-collection "Direct link to Delete Collection") ``` DELETE https://{EKODB_API_URL}/api/collections/{collection} Authorization: Bearer {ADMIN_TOKEN} ``` ### Restore from Trash[​](#restore-from-trash "Direct link to Restore from Trash") ``` # Restore single record POST https://{EKODB_API_URL}/api/trash/{collection}/{record_id} Authorization: Bearer {ADMIN_TOKEN} # Restore all records in collection POST https://{EKODB_API_URL}/api/trash/{collection} Authorization: Bearer {ADMIN_TOKEN} ``` Full Details For complete Delete and Restore documentation including response formats and trash retention policies, see [Collections & Schemas - Delete Collection](/collections-and-schemas.md#delete-collection). ## Using Operations Within Transactions[​](#using-operations-within-transactions "Direct link to Using Operations Within Transactions") All write operations (insert, update, delete) can be performed within a transaction by adding the `transaction_id` query parameter. Operations are tracked and can be committed or rolled back together. **Insert Within Transaction:** ``` POST https://{EKODB_API_URL}/api/insert/users?transaction_id=tx-001 Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "name": "John Doe", "email": "john@example.com" } ``` **Update Within Transaction:** ``` PUT https://{EKODB_API_URL}/api/update/users/user_123?transaction_id=tx-001 Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "status": "active" } ``` **Delete Within Transaction:** ``` DELETE https://{EKODB_API_URL}/api/delete/users/user_123?transaction_id=tx-001 Authorization: Bearer {YOUR_API_TOKEN} ``` **Complete Transaction Example:** ``` # 1. Begin transaction tx_id=$(curl -s -X POST https://{EKODB_API_URL}/api/transactions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"isolation_level": "Serializable"}' | jq -r '.transaction_id') # 2. Insert record in transaction curl -X POST https://{EKODB_API_URL}/api/insert/users?transaction_id=$tx_id \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"name": "Jane Doe", "email": "jane@example.com"}' # 3. Update record in transaction curl -X PUT https://{EKODB_API_URL}/api/update/users/user_456?transaction_id=$tx_id \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"status": "verified"}' # 4. Commit transaction (or rollback if needed) curl -X POST https://{EKODB_API_URL}/api/transactions/$tx_id/commit \ -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` Transaction Support For full transaction management including savepoints and rollback capabilities, see the [Transactions](/transactions.md) documentation. ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete workflow for creating and managing a user: ``` # 1. Insert user curl -X POST https://{EKODB_API_URL}/api/insert/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "name": "John Doe", "email": "john@example.com", "age": 30, "status": "pending" }' # Response: {"id": "user_123", ...} # 2. Get user by ID curl -X GET https://{EKODB_API_URL}/api/find/users/user_123 \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # 3. Update user status curl -X PUT https://{EKODB_API_URL}/api/update/users/user_123 \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "status": "active", "activated_at": "2026-01-15T11:00:00Z" }' # 4. Query active users curl -X POST https://{EKODB_API_URL}/api/find/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } }, "limit": 10 }' # 5. Search users curl -X POST https://{EKODB_API_URL}/api/search/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "query": "john", "limit": 5 }' # 6. Delete user curl -X DELETE https://{EKODB_API_URL}/api/delete/users/user_123 \ -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Use Appropriate HTTP Methods[​](#use-appropriate-http-methods "Direct link to Use Appropriate HTTP Methods") ``` # Create - POST POST /api/insert/{collection} # Read - GET GET /api/find/{collection}/{id} # Update - PUT PUT /api/update/{collection}/{id} # Delete - DELETE DELETE /api/delete/{collection}/{id} ``` ### Project Fields Over GET[​](#project-fields-over-get "Direct link to Project Fields Over GET") Limit the fields returned by passing a comma-joined `select_fields` (or `exclude_fields`) query string: ``` GET /api/find/users?select_fields=id,name,email GET /api/find/users?exclude_fields=secret ``` The primary key (`id`) is always returned. ### Handle Errors Gracefully[​](#handle-errors-gracefully "Direct link to Handle Errors Gracefully") ``` # Check status code response=$(curl -s -w "\n%{http_code}" -X GET .../api/find/users/user_123) status_code=$(echo "$response" | tail -n1) body=$(echo "$response" | sed '$d') if [ "$status_code" != "200" ]; then echo "Error: $body" fi ``` ### Use Filters for Efficient Queries[​](#use-filters-for-efficient-queries "Direct link to Use Filters for Efficient Queries") ``` # Good: Filter on indexed fields POST /api/find/users {"filter": {"type": "Condition", "content": {"field": "status", "operator": "Eq", "value": "active"}}} # Good: Limit results POST /api/find/users {"filter": {...}, "limit": 100} # Avoid: Fetching all records without filter POST /api/find/users {"limit": 10000} # Use pagination instead ``` ### Pagination for Large Result Sets[​](#pagination-for-large-result-sets "Direct link to Pagination for Large Result Sets") ``` # Page 1 POST /api/find/users {"limit": 100, "skip": 0} # Page 2 POST /api/find/users {"limit": 100, "skip": 100} # Page 3 POST /api/find/users {"limit": 100, "skip": 200} ``` ## Query Examples[​](#query-examples "Direct link to Query Examples") ### Simple Equality Query[​](#simple-equality-query "Direct link to Simple Equality Query") ``` curl -X POST https://my-first-db.development.google.ekodb.net/api/find/users \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } } }' ``` ### Range Query (Age Between 18 and 65)[​](#range-query-age-between-18-and-65 "Direct link to Range Query (Age Between 18 and 65)") ``` curl -X POST https://my-first-db.development.google.ekodb.net/api/find/users \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gte", "value": 18 } }, { "type": "Condition", "content": { "field": "age", "operator": "Lte", "value": 65 } } ] } }, "sort": [ { "field": "age", "ascending": true } ], "limit": 50 }' ``` ### String Contains Query[​](#string-contains-query "Direct link to String Contains Query") ``` curl -X POST https://my-first-db.development.google.ekodb.net/api/find/users \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Condition", "content": { "field": "email", "operator": "Contains", "value": "@gmail.com" } } }' ``` ### IN Array Query[​](#in-array-query "Direct link to IN Array Query") ``` curl -X POST https://my-first-db.development.google.ekodb.net/api/find/users \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Condition", "content": { "field": "role", "operator": "In", "value": ["admin", "moderator", "editor"] } } }' ``` Query Expression Reference For a complete reference of all operators and query patterns, see [Query Expressions](/query-expressions.md). *** ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Query Expressions](/query-expressions.md) - Complete query operators reference * [Batch Operations](/batch-operations.md) - Bulk operations for multiple records * [Transactions](/transactions.md) - Atomic, durable transactions * [Collections & Schemas](/collections-and-schemas.md) - Create and manage collections * [KV Store](/kv-store.md) - Simple key-value operations * [Client Libraries](/client-libraries/.md) - Language-specific client libraries * [Authentication](/authentication.md) - API key and JWT authentication ## Example Code[​](#example-code "Direct link to Example Code") ### Direct HTTP/REST API Examples[​](#direct-httprest-api-examples "Direct link to Direct HTTP/REST API Examples") Raw HTTP examples demonstrating the REST API directly: * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/simple_crud.js)** - `simple_crud.js` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/simple_crud.py)** - `simple_crud.py` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/simple_crud.go)** - `simple_crud.go` * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/simple_crud.rs)** - `simple_crud.rs` ### Client Library Examples[​](#client-library-examples "Direct link to Client Library Examples") Production-ready examples using official client libraries: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_simple_crud.rs)** - `client_simple_crud.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_simple_crud.py)** - `client_simple_crud.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_simple_crud.ts)** - `client_simple_crud.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_simple_crud.go)** - `client_simple_crud.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientSimpleCrud.kt)** - `ClientSimpleCrud.kt` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_simple_crud.js)** - `client_simple_crud.js` --- # Batch Operations Perform multiple create, update, or delete operations in a single request. Batch operations are optimized for high throughput and handle partial failures gracefully. Performance The server caps the number of records per batch request (default 10,000; configurable). Hosted plans may apply a lower quota. Batch operations are processed in chunks for optimal performance. First, generate an access token using your API key: ``` curl -X POST https://{EKODB_API_URL}/api/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key": "YOUR_API_KEY"}' ``` **Response:** ``` { "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." } ``` ## Batch Insert[​](#batch-insert "Direct link to Batch Insert") Insert multiple records in a single request. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let records = vec![ json!({ "name": "User 1", "email": "user1@example.com" }), json!({ "name": "User 2", "email": "user2@example.com" }) ]; let result = client.batch_insert("users", records).await?; println!("{:?}", result.successful); // ['id1', 'id2'] println!("{:?}", result.failed); // [] ``` ``` records = [ {'name': 'User 1', 'email': 'user1@example.com'}, {'name': 'User 2', 'email': 'user2@example.com'} ] result = await client.batch_insert('users', records) print(result['successful']) # ['id1', 'id2'] print(result['failed']) # [] ``` ``` const records = [ { name: "User 1", email: "user1@example.com" }, { name: "User 2", email: "user2@example.com" }, ]; const result = await client.batchInsert("users", records); console.log(result.successful); // ['id1', 'id2'] console.log(result.failed); // [] ``` ``` const records = [ { name: "User 1", email: "user1@example.com" }, { name: "User 2", email: "user2@example.com" }, ]; const result = await client.batchInsert("users", records); console.log(result.successful); // ['id1', 'id2'] console.log(result.failed); // [] ``` ``` val records = listOf( mapOf("name" to "User 1", "email" to "user1@example.com"), mapOf("name" to "User 2", "email" to "user2@example.com") ) val result = client.batchInsert("users", records) println(result.successful) // ['id1', 'id2'] println(result.failed) // [] ``` ``` records := []map[string]interface{}{ {"name": "User 1", "email": "user1@example.com"}, {"name": "User 2", "email": "user2@example.com"}, } result, err := client.BatchInsert("users", records) if err != nil { log.Fatal(err) } fmt.Println(result.Successful) // ['id1', 'id2'] fmt.Println(result.Failed) // [] ``` ``` curl -X POST https://{EKODB_API_URL}/api/batch/insert/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "inserts": [ { "data": { "name": "User 1", "email": "user1@example.com" } }, { "data": { "name": "User 2", "email": "user2@example.com" } } ] }' # Response { "successful": ["id1", "id2"], "failed": [] } ``` **Disable Real-Time Sync (Ripples):** Use `bypass_ripple=true` to skip real-time data propagation for bulk imports. This significantly improves throughput by avoiding network overhead. **When to bypass ripples:** * Initial data loading (millions of records) * Database migrations or imports * Maintenance operations * Temporary disconnection scenarios * Bulk data seeding in development **Performance impact:** Bypassing ripples can improve batch throughput by 2-5x for large operations. ``` POST https://{EKODB_API_URL}/api/batch/insert/{collection}?bypass_ripple=true Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "inserts": [ {"data": {"name": "Bulk User 1"}}, {"data": {"name": "Bulk User 2"}} ] } ``` Or set it in the request body: ``` { "bypass_ripple": true, "inserts": [ { "data": { "name": "User 1" } }, { "data": { "name": "User 2" } } ] } ``` After Bulk Import After completing a large import with `bypass_ripple=true`, you can synchronize replicas manually using WAL exports or by temporarily reconfiguring ripples. See [Ripples - Data Propagation](/ripples.md) for synchronization strategies. **Response on Partial Failure:** ``` { "successful": ["id1", "id2"], "failed": [ { "id": null, "error": "Chunk 3 failed: validation error" } ] } ``` ## Batch Update[​](#batch-update "Direct link to Batch Update") Update multiple records by ID. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let updates = vec![ json!({ "id": "record_id_1", "data": { "status": "active" } }), json!({ "id": "record_id_2", "data": { "status": "active" } }) ]; let result = client.batch_update("users", updates).await?; println!("{:?}", result.successful); // ['record_id_1', 'record_id_2'] println!("{:?}", result.failed); // [] ``` ``` updates = [ {'id': 'record_id_1', 'data': {'status': 'active'}}, {'id': 'record_id_2', 'data': {'status': 'active'}} ] result = await client.batch_update('users', updates) print(result['successful']) # ['record_id_1', 'record_id_2'] ``` ``` const updates = [ { id: "record_id_1", data: { status: "active" } }, { id: "record_id_2", data: { status: "active" } }, ]; const result = await client.batchUpdate("users", updates); console.log(result.successful); // ['record_id_1', 'record_id_2'] console.log(result.failed); // [] ``` ``` const updates = [ { id: "record_id_1", data: { status: "active" } }, { id: "record_id_2", data: { status: "active" } }, ]; const result = await client.batchUpdate("users", updates); console.log(result.successful); // ['record_id_1', 'record_id_2'] console.log(result.failed); // [] ``` ``` val updates = listOf( mapOf("id" to "record_id_1", "data" to mapOf("status" to "active")), mapOf("id" to "record_id_2", "data" to mapOf("status" to "active")) ) val result = client.batchUpdate("users", updates) println(result.successful) // ['record_id_1', 'record_id_2'] println(result.failed) // [] ``` ``` updates := []map[string]interface{}{ {"id": "record_id_1", "data": map[string]interface{}{"status": "active"}}, {"id": "record_id_2", "data": map[string]interface{}{"status": "active"}}, } result, err := client.BatchUpdate("users", updates) if err != nil { log.Fatal(err) } fmt.Println(result.Successful) // ['record_id_1', 'record_id_2'] fmt.Println(result.Failed) // [] ``` ``` curl -X PUT https://{EKODB_API_URL}/api/batch/update/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "updates": [ { "id": "record_id_1", "data": { "status": "active" } }, { "id": "record_id_2", "data": { "status": "active" } } ] }' # Response { "successful": ["record_id_1", "record_id_2"], "failed": [] } ``` **Disable Real-Time Sync:** ``` PUT https://{EKODB_API_URL}/api/batch/update/{collection}?bypass_ripple=true ``` Or in request body: ``` { "bypass_ripple": true, "updates": [...] } ``` **Response on Partial Failure:** ``` { "successful": ["record_id_1"], "failed": [ { "id": "record_id_2", "error": "Record not found or update failed" } ] } ``` ## Batch Delete[​](#batch-delete "Direct link to Batch Delete") Delete multiple records by ID. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let ids = vec!["record_id_1", "record_id_2", "record_id_3"]; let result = client.batch_delete("users", ids).await?; println!("{:?}", result.successful); // ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` ids = ['record_id_1', 'record_id_2', 'record_id_3'] result = await client.batch_delete('users', ids) print(result['successful']) # ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` const ids = ["record_id_1", "record_id_2", "record_id_3"]; const result = await client.batchDelete("users", ids); console.log(result.successful); // ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` const ids = ["record_id_1", "record_id_2", "record_id_3"]; const result = await client.batchDelete("users", ids); console.log(result.successful); // ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` val ids = listOf("record_id_1", "record_id_2", "record_id_3") val result = client.batchDelete("users", ids) println(result.successful) // ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` ids := []string{"record_id_1", "record_id_2", "record_id_3"} result, err := client.BatchDelete("users", ids) if err != nil { log.Fatal(err) } fmt.Println(result.Successful) // ['record_id_1', 'record_id_2', 'record_id_3'] ``` ``` curl -X DELETE https://{EKODB_API_URL}/api/batch/delete/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "deletes": [ { "id": "record_id_1" }, { "id": "record_id_2" }, { "id": "record_id_3" } ] }' # Response { "successful": ["record_id_1", "record_id_2", "record_id_3"], "failed": [] } ``` **Disable Real-Time Sync:** ``` DELETE https://{EKODB_API_URL}/api/batch/delete/{collection}?bypass_ripple=true ``` Or in request body: ``` { "bypass_ripple": true, "deletes": [...] } ``` **Response on Partial Failure:** ``` { "successful": ["record_id_1", "record_id_2"], "failed": [ { "id": "record_id_3", "error": "Record not found or delete failed" } ] } ``` ## Using Batch Operations Within Transactions[​](#using-batch-operations-within-transactions "Direct link to Using Batch Operations Within Transactions") All batch operations can be performed within a transaction by adding the `transaction_id` query parameter. This ensures atomicity across all batch operations. **Batch Insert in Transaction:** ``` POST https://{EKODB_API_URL}/api/batch/insert/users?transaction_id=tx-001 Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "inserts": [ {"data": {"name": "User 1", "email": "user1@example.com"}}, {"data": {"name": "User 2", "email": "user2@example.com"}} ] } ``` **Batch Update in Transaction:** ``` PUT https://{EKODB_API_URL}/api/batch/update/users?transaction_id=tx-001 Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "updates": [ {"id": "user_1", "data": {"status": "active"}}, {"id": "user_2", "data": {"status": "active"}} ] } ``` **Batch Delete in Transaction:** ``` DELETE https://{EKODB_API_URL}/api/batch/delete/users?transaction_id=tx-001 Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "deletes": [ {"id": "user_1"}, {"id": "user_2"} ] } ``` **Complete Transaction Example:** ``` # Begin transaction tx_id=$(curl -s -X POST https://{EKODB_API_URL}/api/transactions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"isolation_level": "Serializable"}' | jq -r '.transaction_id') # Batch insert in transaction curl -X POST https://{EKODB_API_URL}/api/batch/insert/users?transaction_id=$tx_id \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "inserts": [ {"data": {"name": "User 1", "email": "user1@example.com"}}, {"data": {"name": "User 2", "email": "user2@example.com"}} ] }' # Batch update in transaction curl -X PUT https://{EKODB_API_URL}/api/batch/update/users?transaction_id=$tx_id \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "updates": [ {"id": "user_1", "data": {"verified": true}}, {"id": "user_2", "data": {"verified": true}} ] }' # Commit or rollback based on results if [ $? -eq 0 ]; then curl -X POST https://{EKODB_API_URL}/api/transactions/$tx_id/commit \ -H "Authorization: Bearer {YOUR_API_TOKEN}" else curl -X POST https://{EKODB_API_URL}/api/transactions/$tx_id/rollback \ -H "Authorization: Bearer {YOUR_API_TOKEN}" fi ``` Atomic Batch Operations Using transactions with batch operations ensures that either all operations succeed or all are rolled back together. See [Transactions](/transactions.md) for full transaction management. ## Response Status Codes[​](#response-status-codes "Direct link to Response Status Codes") Batch operations return different status codes based on the result: * **201 Created** - All inserts successful * **200 OK** - All updates/deletes successful * **207 Multi-Status** - Partial success (some succeeded, some failed) * **400 Bad Request** - All operations failed * **503 Service Unavailable** - Server is processing too many concurrent batch operations ## Error Handling[​](#error-handling "Direct link to Error Handling") Batch operations continue processing even if individual records fail: ``` # Example: Batch insert with validation error curl -X POST https://{EKODB_API_URL}/api/batch/insert/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "inserts": [ {"data": {"name": "Valid User", "email": "valid@example.com"}}, {"data": {"name": ""}}, {"data": {"name": "Another Valid", "email": "valid2@example.com"}} ] }' # Response (207 Multi-Status): { "successful": ["id1", "id3"], "failed": [ { "id": null, "error": "Validation error: name cannot be empty" } ] } ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Choose Appropriate Batch Size[​](#choose-appropriate-batch-size "Direct link to Choose Appropriate Batch Size") ``` # Small batch (< 100 records) - Fast response POST /api/batch/insert/users {"inserts": [...]} # 50 records # Medium batch (100-1,000 records) - Balanced POST /api/batch/insert/users {"inserts": [...]} # 500 records # Large batch (1,000-10,000 records) - Maximum throughput POST /api/batch/insert/users {"inserts": [...]} # 5,000 records ``` ### Handle Partial Failures[​](#handle-partial-failures "Direct link to Handle Partial Failures") Always check both `successful` and `failed` arrays: ``` # Check response response=$(curl -X POST .../api/batch/insert/users ...) # Parse response successful_count=$(echo $response | jq '.successful | length') failed_count=$(echo $response | jq '.failed | length') if [ $failed_count -gt 0 ]; then echo "Warning: $failed_count operations failed" # Log failures or retry fi ``` ### Monitor Performance[​](#monitor-performance "Direct link to Monitor Performance") ``` # Start with smaller batches to test performance curl -X POST .../api/batch/insert/users -d '{"inserts": [...]}' # 100 records # Monitor response time and adjust batch size # - Fast response (< 1s): Can increase batch size # - Slow response (> 5s): Reduce batch size # - 503 errors: Server overloaded, wait and retry with smaller batches # Use appropriate timeouts for large batches curl --max-time 60 -X POST .../api/batch/insert/users ... # Handle rate limiting gracefully if [ $http_code -eq 503 ]; then echo "Server busy, waiting 5 seconds..." sleep 5 # Retry with smaller batch fi ``` ### Use Transactions for Atomicity[​](#use-transactions-for-atomicity "Direct link to Use Transactions for Atomicity") For operations that must succeed or fail together: ``` # Begin transaction tx_id=$(curl -X POST .../api/transactions | jq -r '.transaction_id') # Perform batch operations curl -X POST .../api/batch/insert/users?transaction_id=$tx_id ... # Commit or rollback based on results if [ $success ]; then curl -X POST .../api/transactions/$tx_id/commit else curl -X POST .../api/transactions/$tx_id/rollback fi ``` ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete workflow for batch inserting users with error handling: ``` #!/bin/bash # Batch insert 1000 users response=$(curl -s -X POST https://{EKODB_API_URL}/api/batch/insert/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d @- << 'EOF' { "inserts": [ {"data": {"name": "User 1", "email": "user1@example.com"}}, {"data": {"name": "User 2", "email": "user2@example.com"}}, {"data": {"name": "User 3", "email": "user3@example.com"}} ] } EOF ) # Parse response successful=$(echo $response | jq -r '.successful[]') failed=$(echo $response | jq -r '.failed[]') # Log results echo "Successfully inserted: $(echo $response | jq '.successful | length') records" echo "Failed: $(echo $response | jq '.failed | length') records" # Handle failures if [ "$(echo $response | jq '.failed | length')" -gt 0 ]; then echo "Failed records:" echo $response | jq '.failed' # Optionally retry failed records # ... retry logic here ... fi ``` ## Limits[​](#limits "Direct link to Limits") The server enforces two batch limits, applied uniformly to every batch request (they are not keyed by deployment tier): * **Max batch size** — the maximum number of records in a single batch request. A request that exceeds it is rejected before any record is processed. Default: 10,000 records (configurable on the server). * **Max concurrent batch operations** — the number of batch operations the server will process simultaneously. Once that ceiling is reached, further batch requests receive `503 Service Unavailable`. Hosted plans may set lower batch sizes and concurrency ceilings as part of their quota; the following are illustrative plan guidance, not server-enforced per-tier limits: | Plan | Suggested Batch Size | Suggested Concurrency | | ---------- | -------------------- | --------------------- | | Free | 500 | 5 | | Starter | 2,000 | 10 | | Pro | 10,000 | 50 | | Enterprise | 20,000 | 100 | Concurrency If you receive a `503 Service Unavailable` response, the server is processing too many concurrent batch operations. Wait and retry after a few seconds. ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - Single record CRUD operations * [Query Expressions](/query-expressions.md) - Filter syntax for batch queries * [Transactions](/transactions.md) - Atomic, durable transactions * [Authentication](/authentication.md) - API key and JWT authentication ## Example Code[​](#example-code "Direct link to Example Code") ### Direct HTTP/REST API Examples[​](#direct-httprest-api-examples "Direct link to Direct HTTP/REST API Examples") Raw HTTP examples demonstrating the REST API directly: * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/batch_operations.js)**: `batch_operations.js` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/batch_operations.py)**: `batch_operations.py` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/batch_operations.go)**: `batch_operations.go` * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/batch_operations.rs)**: `batch_operations.rs` ### Client Library Examples[​](#client-library-examples "Direct link to Client Library Examples") Production-ready examples using official client libraries: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_batch_operations.rs)**: `client_batch_operations.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_batch_operations.py)**: `client_batch_operations.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_batch_operations.ts)**: `client_batch_operations.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_batch_operations.go)**: `client_batch_operations.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientBatchOperations.kt)**: `ClientBatchOperations.kt` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_batch_operations.js)**: `client_batch_operations.js` --- # Build Your First AI Agent in 10 Minutes This tutorial takes you from a fresh account to an AI agent that answers questions about your own data and runs a scheduled task. Everything here happens in the dashboard, so there is nothing to install. What you'll have at the end A free ekoDB database with sample data, an AI agent connected to it, and a scheduled task the agent runs on its own. ## Before you start[​](#before-you-start "Direct link to Before you start") You need an approved ekoDB account. Sign up at [app.ekodb.io](https://app.ekodb.io). New accounts are reviewed by the team, and you will get an email when yours is approved. While you wait, read the [AI Agents overview](/ai-agents.md) so the pieces make sense when you start clicking. ## Step 1: Deploy a database (about 2 minutes)[​](#step-1-deploy-a-database-about-2-minutes "Direct link to Step 1: Deploy a database (about 2 minutes)") 1. Sign in at [app.ekodb.io](https://app.ekodb.io). 2. Click **Create Deployment**. 3. Choose the **Database** module type. 4. Pick a zone close to you, then the **Free** machine tier. 5. Choose a subdomain and the development environment, and create the deployment. Wait for the status to reach running. This is your live backend. ## Step 2: Put some data in it (about 2 minutes)[​](#step-2-put-some-data-in-it-about-2-minutes "Direct link to Step 2: Put some data in it (about 2 minutes)") Open your deployment and go to the **Collections** tab. Create a collection called `items` and add a few records using the **Add Record** form. For example, three records shaped like this: ``` { "name": "Wireless mouse", "category": "accessories", "quantity": 3, "restock_threshold": 5 } { "name": "Mechanical keyboard", "category": "accessories", "quantity": 12, "restock_threshold": 10 } { "name": "USB-C cable", "category": "cables", "quantity": 1, "restock_threshold": 5 } ``` The schema is inferred from the first record, so the fields you enter on that one (`name`, `category`, `quantity`, `restock_threshold`) become the columns for the rest. You now have real data to ask questions about. ## Step 3: Chat with your data (about 1 minute)[​](#step-3-chat-with-your-data-about-1-minute "Direct link to Step 3: Chat with your data (about 1 minute)") Switch to the **Chat** tab on the database deployment. This is ekoDB's built-in chat, backed by RAG over your collections. Try: > List every item where `quantity` is below `restock_threshold`. The answer comes straight from your `items` collection. No pipeline, no second database. ## Step 4: Deploy an AI agent (about 2 minutes)[​](#step-4-deploy-an-ai-agent-about-2-minutes "Direct link to Step 4: Deploy an AI agent (about 2 minutes)") The built-in chat is great for questions. An agent goes further: it has tools, goals, and scheduled tasks. Deploy one: 1. Click **Create Deployment** again. 2. Choose the **AI Agent** module type. 3. Pick the **Free** tier, and point it at the database you created in Step 1. 4. Create the deployment and wait for it to reach running. ## Step 5: Talk to your agent (about 1 minute)[​](#step-5-talk-to-your-agent-about-1-minute "Direct link to Step 5: Talk to your agent (about 1 minute)") Open the agent deployment and go to its **Chat** tab. This is the agent, with full tool access to your database. Ask it to do something a plain chat can't: > Create a `restock_alerts` collection, then add an item to it for every product in `items` where `quantity` is below `restock_threshold`. The agent plans the steps, creates the collection, queries `items`, and writes the alerts, showing each tool call as it goes. ## Step 6: Give it a scheduled task (about 2 minutes)[​](#step-6-give-it-a-scheduled-task-about-2-minutes "Direct link to Step 6: Give it a scheduled task (about 2 minutes)") In the agent deployment, open the **Tasks** (or **Goals**) panel and create a scheduled task, for example: > Every morning, check `items` for anything below its restock threshold and refresh `restock_alerts`. Save it and trigger it once manually to confirm it runs. From now on the agent does this on its own, no one in the loop. ## What you built[​](#what-you-built "Direct link to What you built") In about ten minutes you deployed a database, loaded data, talked to it in plain English, deployed an agent with tools, and gave that agent a recurring job. From here: * Define a focused agent with a custom system prompt and scoped permissions on the **Agents** panel. * Expose an agent to your own customers through its public endpoint (see [AI Agents](/ai-agents.md#agents-for-you-or-for-your-users)). * Install the [desktop app or terminal client](https://ekodb.io/install) to work with the same agent outside the browser. --- # Chat & RAG (Retrieval-Augmented Generation) Build intelligent conversational applications that combine LLMs with your data for context-aware, accurate responses. Integrated AI ekoDB provides built-in chat session management and RAG capabilities - no separate infrastructure needed. ## Quick Start[​](#quick-start "Direct link to Quick Start") * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::{Client, CreateChatSessionRequest, ChatMessageRequest, CollectionConfig}; let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("your-api-key") .build()?; // 1. Create a chat session let session = client.create_chat_session(CreateChatSessionRequest { collections: vec![CollectionConfig { collection_name: "knowledge_base".to_string(), fields: vec![], search_options: None, }], llm_provider: "openai".to_string(), llm_model: Some("gpt-4".to_string()), system_prompt: Some("You are a helpful assistant.".to_string()), ..Default::default() }).await?; // 2. Send a message let response = client.chat_message( &session.chat_id, ChatMessageRequest::new("How do I optimize database queries?") ).await?; println!("AI: {:?}", response.responses); ``` ``` from ekodb_client import Client client = Client.new( "https://my-first-db.development.google.ekodb.net", "your-api-key" ) # 1. Create a chat session session = await client.create_chat_session( collections=[{ 'collection_name': 'knowledge_base', 'fields': ['content', 'title'] }], llm_provider='openai', llm_model='gpt-4', system_prompt='You are a helpful assistant.' ) # 2. Send a message - automatically retrieves relevant context response = await client.chat_message( session['chat_id'], 'How do I optimize database queries?' ) print(response['responses']) # AI response print(response['context_snippets']) # Retrieved documents ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY, }); await client.init(); // 1. Create a chat session const session = await client.createChatSession({ collections: [ { collection_name: "knowledge_base", fields: ["content", "title"], }, ], llm_provider: "openai", llm_model: "gpt-4", system_prompt: "You are a helpful assistant that answers questions based on the provided context.", }); // 2. Send a message - automatically retrieves relevant context const response = await client.chatMessage(session.chat_id, { message: "How do I optimize database queries?", }); console.log(response.responses); // AI response console.log(response.context_snippets); // Retrieved documents used for context ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY, }); await client.init(); // 1. Create a chat session const session = await client.createChatSession({ collections: [ { collection_name: "knowledge_base", fields: ["content", "title"], }, ], llm_provider: "openai", llm_model: "gpt-4", system_prompt: "You are a helpful assistant.", }); // 2. Send a message const response = await client.chatMessage(session.chat_id, { message: "How do I optimize database queries?", }); console.log(response.responses); console.log(response.context_snippets); ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://my-first-db.development.google.ekodb.net") .apiKey("your-api-key") .build() // 1. Create a chat session val session = client.createChatSession( collections = listOf( CollectionConfig( collectionName = "knowledge_base", fields = listOf("content", "title") ) ), llmProvider = "openai", llmModel = "gpt-4", systemPrompt = "You are a helpful assistant." ) // 2. Send a message val response = client.chatMessage(session.chatId, buildJsonObject { put("message", "How do I optimize database queries?") }) println(response["responses"]) ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://my-first-db.development.google.ekodb.net", "your-api-key", ) // 1. Create a chat session llmModel := "gpt-4" systemPrompt := "You are a helpful assistant." session, err := client.CreateChatSession(ekodb.CreateChatSessionRequest{ Collections: []ekodb.CollectionConfig{{ CollectionName: "knowledge_base", Fields: []interface{}{"content", "title"}, }}, LLMProvider: "openai", LLMModel: &llmModel, SystemPrompt: &systemPrompt, }) // 2. Send a message response, err := client.ChatMessage(session.ChatID, ekodb.ChatMessageRequest{ Message: "How do I optimize database queries?", }) fmt.Println(response.Responses) ``` ``` # 1. Create a chat session SESSION=$(curl -X POST https://{EKODB_API_URL}/api/chat \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "collections": [{ "collection_name": "knowledge_base", "fields": [{"field_name": "content"}, {"field_name": "title"}] }], "llm_provider": "openai", "llm_model": "gpt-4", "system_prompt": "You are a helpful assistant." }' | jq -r '.id') # 2. Send a message curl -X POST https://{EKODB_API_URL}/api/chat/$SESSION/messages \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "message": "How do I optimize database queries?" }' ``` ## Core Concepts[​](#core-concepts "Direct link to Core Concepts") ### Centralized Architecture[​](#centralized-architecture "Direct link to Centralized Architecture") All chat sessions and messages are stored in two database-wide collections: * **`chat_configurations_{database}`** - Session metadata and configuration * **`chat_messages_{database}`** - All messages from all sessions **Benefits:** * ✅ Scalable to millions of sessions * ✅ No per-session collection management * ✅ Easy cross-session querying * ✅ Simplified data model ### Chat Sessions[​](#chat-sessions "Direct link to Chat Sessions") A chat session represents a conversation thread: ``` { id: 'session_uuid', llm_provider: 'openai', // or 'anthropic', 'perplexity' llm_model: 'gpt-4', collections: [...], // Data sources to search system_prompt: '...', max_context_messages: 10, created_at: '2025-01-22T...', updated_at: '2025-01-22T...', parent_id: null, // For branching conversations branch_point_idx: null, title: null // User-set session title } ``` ### Message Flow[​](#message-flow "Direct link to Message Flow") ``` User Message ↓ Search Collections (Semantic + Text) ↓ Retrieve Relevant Context ↓ Build Prompt (System + Context + History + User Message) ↓ LLM Generation ↓ Store User Message + AI Response ↓ Return Response with Context ``` ## Creating Chat Sessions[​](#creating-chat-sessions "Direct link to Creating Chat Sessions") ### Basic Chat Session[​](#basic-chat-session "Direct link to Basic Chat Session") ``` const session = await client.createChatSession({ llm_provider: "openai", llm_model: "gpt-4", system_prompt: "You are a helpful assistant.", }); ``` ### RAG Chat Session[​](#rag-chat-session "Direct link to RAG Chat Session") ``` const session = await client.createChatSession({ collections: [ { collection_name: "documentation", fields: [ { field_name: "content", search_options: { weight: 1.0, // Search relevance weight language: "english", }, }, { field_name: "title", search_options: { weight: 0.5, }, }, ], }, { collection_name: "faqs", fields: ["question", "answer"], }, ], llm_provider: "openai", llm_model: "gpt-4", system_prompt: "Answer questions based on the provided documentation and FAQs.", max_context_messages: 10, // Include last 10 messages in context }); ``` ### Multi-Provider Support[​](#multi-provider-support "Direct link to Multi-Provider Support") ``` // OpenAI const openaiChat = await client.createChatSession({ llm_provider: "openai", llm_model: "gpt-4-turbo", }); // Anthropic const claudeChat = await client.createChatSession({ llm_provider: "anthropic", llm_model: "claude-3-opus-20240229", }); // Perplexity const perplexityChat = await client.createChatSession({ llm_provider: "perplexity", llm_model: "pplx-70b-online", }); ``` ## Sending Messages[​](#sending-messages "Direct link to Sending Messages") ### Simple Message[​](#simple-message "Direct link to Simple Message") ``` const response = await client.sendChatMessage(sessionId, { message: "What is ekoDB?", }); console.log(response.responses); // AI response (array of strings) ``` ### With Context[​](#with-context "Direct link to With Context") When collections are configured, ekoDB automatically: 1. Searches collections for relevant context 2. Ranks results by relevance 3. Includes context in the LLM prompt 4. Returns both response and context used ``` const response = await client.sendChatMessage(sessionId, { message: 'How do vector searches work?', }); // Response includes: { chat_id: 'session_uuid', message_id: 'msg_uuid', responses: ['Vector searches work by...'], // AI response (array of strings) context_snippets: [ // Retrieved documents { collection: 'documentation', record: { title: 'Vector Search Guide', content: '...' }, score: 0.95, matched_fields: ['content'] } ], execution_time_ms: 142, token_usage: { prompt_tokens: 512, completion_tokens: 88, total_tokens: 600 } } ``` ## Message Management[​](#message-management "Direct link to Message Management") ### List Messages[​](#list-messages "Direct link to List Messages") ``` const messages = await client.getChatMessages(sessionId, { limit: 50, skip: 0, sort: "asc", // chronological order }); ``` ### Update Message[​](#update-message "Direct link to Update Message") ``` await client.updateChatMessage(sessionId, messageId, { content: "Updated message content", }); ``` ### Delete Message[​](#delete-message "Direct link to Delete Message") ``` await client.deleteChatMessage(sessionId, messageId); ``` ### Mark as Forgotten[​](#mark-as-forgotten "Direct link to Mark as Forgotten") Exclude specific messages from context window: ``` await client.toggleMessageForgotten(sessionId, messageId, true); ``` ### Regenerate Response[​](#regenerate-response "Direct link to Regenerate Response") Generate a new AI response for the same user message: ``` const newResponse = await client.regenerateResponse(sessionId, messageId); ``` ## Advanced Features[​](#advanced-features "Direct link to Advanced Features") ### Branching Conversations[​](#branching-conversations "Direct link to Branching Conversations") Create alternative conversation paths from any point: ``` // Branch from message 5 in parent session const branchSession = await client.createChatSession({ parent_id: parentSessionId, branch_point_idx: 5, // Branch from 5th message llm_provider: "openai", llm_model: "gpt-4", }); // New session starts with messages 0-5 from parent // Can explore different conversation paths ``` ### Merging Sessions[​](#merging-sessions "Direct link to Merging Sessions") Combine multiple conversation threads: ``` const mergedSession = await client.mergeChatSessions({ session_ids: [sessionId1, sessionId2], strategy: "chronological", // or 'interleaved' llm_provider: "openai", llm_model: "gpt-4", }); ``` ### History Compaction[​](#history-compaction "Direct link to History Compaction") Fold a session's older messages into a single summary message to reclaim context-window budget. The most-recent messages are kept verbatim; everything older is summarized and the originals are marked `forgotten` so they stop being replayed on subsequent turns. ``` POST /api/chat/{chat_id}/compact Content-Type: application/json { "keep_recent": 50 // Optional. Defaults to the session's // max_context_messages (or 50). 0 compacts all. } ``` Response: ``` { folded: 120, // Older messages folded into the summary kept_recent: 50, // Recent messages kept verbatim summary_chars: 842, // Length of the inserted summary (0 if none) summary_message_id: 'msg_uuid', // ID of the synthetic summary message already_compact: false // true when nothing needed folding } ``` Every client library exposes this directly — for example, in TypeScript: ``` const result = await client.compactChat(chatId, 50); // keepRecent optional console.log(`Folded ${result.folded}, kept ${result.kept_recent} recent`); ``` The same method ships in every client: `compactChat` (TypeScript, Kotlin), `compact_chat` (Rust, Python), and `CompactChat` (Go), each returning the response shape above. ## Real-World Examples[​](#real-world-examples "Direct link to Real-World Examples") ### Customer Support Bot[​](#customer-support-bot "Direct link to Customer Support Bot") ``` // 1. Create knowledge base await client.batchInsert("support_articles", articles); await client.createIndex("support_articles", ["title", "content"]); // 2. Create support chat session const supportSession = await client.createChatSession({ collections: [ { collection_name: "support_articles", fields: ["title", "content", "category"], }, ], llm_provider: "openai", llm_model: "gpt-4", system_prompt: `You are a customer support agent. Answer questions based on our support documentation. Be helpful, concise, and professional.`, }); // 3. Handle customer query const response = await client.sendChatMessage(supportSession.id, { message: "How do I reset my password?", }); // Response includes relevant support articles as context ``` ### Document Q\&A[​](#document-qa "Direct link to Document Q\&A") ``` // RAG over internal documents const docSession = await client.createChatSession({ collections: [ { collection_name: "company_docs", fields: [ { field_name: "content", search_options: { weight: 1.0 }, }, ], }, ], llm_provider: "anthropic", llm_model: "claude-3-sonnet-20240229", system_prompt: "Answer questions about company policies and procedures based on the provided documents.", }); const answer = await client.sendChatMessage(docSession.id, { message: "What is our vacation policy?", }); ``` ### Code Assistant[​](#code-assistant "Direct link to Code Assistant") ``` // Code documentation chatbot const codeSession = await client.createChatSession({ collections: [ { collection_name: "code_docs", fields: ["description", "code", "examples"], }, { collection_name: "api_reference", fields: ["method", "parameters", "returns"], }, ], llm_provider: "openai", llm_model: "gpt-4", system_prompt: `You are a code assistant. Help developers by: - Providing accurate code examples - Explaining concepts clearly - Referencing official documentation`, max_context_messages: 15, }); ``` ## Hybrid Search Integration[​](#hybrid-search-integration "Direct link to Hybrid Search Integration") Combine text search with vector similarity: ``` // Store embeddings with documents await client.insert("knowledge_base", { title: "Vector Search Guide", content: "Vector search enables...", embedding: vectorEmbedding, // From OpenAI, Cohere, etc. }); // Chat session uses hybrid search automatically const session = await client.createChatSession({ collections: [ { collection_name: "knowledge_base", fields: [ { field_name: "content", search_options: { weight: 0.6 } }, { field_name: "embedding", search_options: { weight: 0.4, type: "vector" }, }, ], }, ], llm_provider: "openai", llm_model: "gpt-4", }); // Searches use both text relevance and semantic similarity ``` ## Performance Optimization[​](#performance-optimization "Direct link to Performance Optimization") ### 1. Limit Context Messages[​](#1-limit-context-messages "Direct link to 1. Limit Context Messages") ``` const session = await client.createChatSession({ max_context_messages: 5, // Only include last 5 messages // ... other config }); ``` ### 2. Use Targeted Collections[​](#2-use-targeted-collections "Direct link to 2. Use Targeted Collections") ``` // Only search relevant collections const session = await client.createChatSession({ collections: [ { collection_name: "recent_docs", // Smaller, focused collection fields: ["content"], }, ], // ... other config }); ``` ### 3. Index Your Data[​](#3-index-your-data "Direct link to 3. Index Your Data") ``` // Create indexes for faster search await client.createIndex("knowledge_base", ["title", "content"]); ``` ### 4. Use Efficient Models[​](#4-use-efficient-models "Direct link to 4. Use Efficient Models") ``` // Balance cost/performance const session = await client.createChatSession({ llm_provider: "openai", llm_model: "gpt-3.5-turbo", // Faster, cheaper for simple queries }); ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") 1. **System Prompts**: Be specific about behavior and constraints 2. **Context Limits**: Balance context quality vs token costs 3. **Collection Design**: Structure data for efficient retrieval 4. **Error Handling**: Handle LLM failures gracefully 5. **Rate Limiting**: Respect provider rate limits 6. **Cost Monitoring**: Track token usage and costs 7. **Caching**: Cache common responses when appropriate 8. **Testing**: Test with real user queries ## API Reference[​](#api-reference "Direct link to API Reference") ### createChatSession()[​](#createchatsession "Direct link to createChatSession()") ``` client.createChatSession(options: { collections?: CollectionConfig[], llm_provider: 'openai' | 'anthropic' | 'perplexity', llm_model: string, system_prompt?: string, max_context_messages?: number, parent_id?: string, branch_point_idx?: number, }): Promise ``` ### sendChatMessage()[​](#sendchatmessage "Direct link to sendChatMessage()") ``` client.sendChatMessage( sessionId: string, options: { message: string, } ): Promise<{ content: string, context: ContextDocument[], message_id: string, created_at: string, }> ``` ### getChatMessages()[​](#getchatmessages "Direct link to getChatMessages()") ``` client.getChatMessages( sessionId: string, options?: { limit?: number, skip?: number, sort?: 'asc' | 'desc', } ): Promise ``` ### getChatModels()[​](#getchatmodels "Direct link to getChatModels()") Get all available chat models organized by provider: ``` client.getChatModels(): Promise> ``` **Response Example:** ``` { "openai": ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo", "gpt-4o"], "anthropic": ["claude-3-opus-20240229", "claude-3-sonnet-20240229"], "perplexity": ["llama-3.1-sonar-small-128k-online"] } ``` ### getChatModel()[​](#getchatmodel "Direct link to getChatModel()") Get available models for a specific provider: ``` client.getChatModel(provider: string): Promise ``` **Example:** ``` const openaiModels = await client.getChatModel("openai"); // ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo", "gpt-4o", "gpt-4o-mini"] ``` **REST API:** ``` # List all models by provider GET /api/chat_models # Get models for a specific provider GET /api/chat_models/openai ``` ### Provider status[​](#provider-status "Direct link to Provider status") `GET /api/chat_models` returns a `providers` map beside the four lists, so a key the provider rejected is never mistaken for a key that was never set — both used to be an empty list: ``` { "openai": [], "anthropic": ["claude-sonnet-4-5"], "perplexity": [], "gemini": [], "providers": { "anthropic": { "status": "ok", "verified": true, "model_count": 1 }, "gemini": { "status": "not_configured", "verified": false }, "openai": { "status": "auth_failed", "verified": true, "http_status": 401, "message": "Failed to fetch OpenAI models: 401 Unauthorized: ..." }, "perplexity": { "status": "not_configured", "verified": false } } } ``` | `status` | Meaning | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ok` | The key is usable: the provider listed its models with it, or — for Perplexity, which has no models endpoint — a key is present (`verified` tells the two apart) | | `not_configured` | No key — for OpenAI-compatible endpoints, no key and no custom `openai_api_url` either (a self-hosted endpoint needs no key) | | `auth_failed` | The provider rejected the key (401) | | `permission_denied` | The key is accepted but may not use this resource or region (403) | | `billing` | The account cannot pay (402, or a quota / spend-limit code) | | `rate_limited` | The provider is rate limiting ekoDB (429) | | `unavailable` | The provider answered 5xx or an unusable body | | `unreachable` | Nothing answered: DNS, connect, TLS, or a timeout | | `request_error` | The provider refused the request itself (any other 4xx) | `verified` is `true` only when the status is the provider's own answer about the key; a 5xx, a refused connection, or a missing key says nothing about it. Perplexity publishes no models endpoint, so a configured Perplexity key is `ok` with `verified: false` until the first chat call. `GET /api/chat_models/{provider}` returns the bare list on success; on failure it returns a classified body instead of a bare 404: ``` { "error": "Failed to fetch OpenAI models: 401 Unauthorized: ...", "error_kind": "provider_auth_failed", "provider": "openai", "provider_status": 401 } ``` with `502` for a rejected key, a permission problem, a provider that is down or unreachable; `402` for billing; `429` for a provider rate limit; and `404` (with `error_kind` set) for a provider that is not configured or not known. A key changed through [`PUT /api/config`](/configuration.md#llm-providers) is probed the moment it is saved, and the outcome comes back in the same shape as `provider_checks`. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Rejected API Key[​](#rejected-api-key "Direct link to Rejected API Key") **Problem**: Chat fails, or a provider's model list is empty although a key is configured **Diagnosis**: `GET /api/chat_models` — the provider's row in `providers` says why: `auth_failed` is a rejected key, `not_configured` a missing one, `unreachable` a network problem between ekoDB and the provider. A chat error carries the same classification as `error_kind` (see [Error Codes](/reference/error-codes.md#provider-failures-llm-features)). **Solutions**: * `auth_failed`: the key is wrong, revoked or expired — set a new one; the save reports the probe's outcome * `permission_denied`: the key works but may not use this model or region * `billing`: add credits or a payment method at the provider * `not_configured`: set a key for that provider ### No Context Retrieved[​](#no-context-retrieved "Direct link to No Context Retrieved") **Problem**: AI responses don't use your data **Solutions**: * Verify collections are configured correctly * Check collection has data * Ensure search fields exist * Try different search weights ### Token Limit Errors[​](#token-limit-errors "Direct link to Token Limit Errors") **Problem**: Context too large for LLM **Solutions**: * Reduce `max_context_messages` * Limit collection search results * Use shorter documents * Switch to model with larger context window ### Slow Responses[​](#slow-responses "Direct link to Slow Responses") **Problem**: Chat responses are slow **Solutions**: * Create indexes on search fields * Reduce number of collections searched * Use faster LLM model * Limit context size ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Vector Search](/vector-search.md) - Semantic search with embeddings * [Indexes](/indexes.md) - Optimize search performance * [Client Libraries](/client-libraries/advanced-operations.md) - Full API examples * [System Administration](/system-administration.md) - Monitor and manage **Chat Models API Examples:** * Rust - `client_chat_models.rs` * Python - `client_chat_models.py` * TypeScript - `client_chat_models.ts` * Go - `client_chat_models.go` * Kotlin - `ClientChatModels.kt` ## Summary[​](#summary "Direct link to Summary") Chat & RAG in ekoDB enables: ✅ **Conversational AI** - Natural language interactions ✅ **Context-aware responses** - Answers based on your data ✅ **Multi-provider support** - OpenAI, Anthropic, Perplexity ✅ **Branching conversations** - Explore alternative paths ✅ **Hybrid search** - Text + vector semantic matching ✅ **Integrated** - No separate infrastructure needed ✅ **Production-ready** - Scalable and reliable --- # Choose Your Path Not sure where to begin? This guide helps you find the right documentation for your needs. *** ## 🚀 Most Popular: Quick Start[​](#-most-popular-quick-start "Direct link to 🚀 Most Popular: Quick Start") **Just want to get running?** → **[5-Minute Quick Start](/quick-start.md)** - Deploy, insert, and query your first record **Perfect for:** First-time users, proof of concepts, testing ekoDB *** ## 🎯 What's Your Main Goal?[​](#-whats-your-main-goal "Direct link to 🎯 What's Your Main Goal?") ### 📱 Build an Application (Web, Mobile, Backend)[​](#-build-an-application-web-mobile-backend "Direct link to 📱 Build an Application (Web, Mobile, Backend)") **Use our client libraries for the best developer experience** → **[Client Libraries Guide](/client-libraries/.md)** **What you get:** * Type-safe SDKs (TypeScript, Python, Rust, Go, Kotlin) * Auto-complete in your IDE * Built-in authentication & retry logic * WebSocket support for real-time features **Start here:** [Installation](/client-libraries/installation.md) → [Basic Operations](/client-libraries/basic-operations.md) *** ### 🔌 Direct API Integration[​](#-direct-api-integration "Direct link to 🔌 Direct API Integration") **Use HTTP REST API directly without SDKs** → **[REST API Documentation](/basic-operations.md)** **Perfect for:** * Serverless functions (Lambda, Cloudflare Workers) * Languages without official client libraries * Custom integrations * Microservices **Start here:** [Basic Operations](/basic-operations.md) → [Authentication](/authentication.md) *** ### 🤖 AI/ML Application (built into your database)[​](#-aiml-application-built-into-your-database "Direct link to 🤖 AI/ML Application (built into your database)") **Build with vector search, embeddings, and RAG — no separate AI service** → **[Chat & RAG](/chat-rag.md)** + **[Vector Search](/vector-search.md)** **What you get:** * High-performance vector search * Semantic similarity & hybrid search * Built-in chat session management with RAG over your own collections * Native LLM integration (OpenAI, Claude, Gemini) **Start here:** [Chat & RAG](/chat-rag.md) → [Advanced Operations](/client-libraries/advanced-operations.md) *** ### 🦾 Autonomous AI Agents[​](#-autonomous-ai-agents "Direct link to 🦾 Autonomous AI Agents") **Deploy an agent that uses tools, pursues goals, and runs scheduled tasks** → **[Build Your First AI Agent](/build-your-first-agent.md)** **What you get:** * A separate **AI Agent** deployment that connects to your database * Tools, goals, scheduled tasks, persistent memory, and ready-made templates * Customer-facing agent endpoints with scoped permissions * Built on the same chat/RAG primitives above **Start here:** [AI Agents](/ai-agents.md) → [Build Your First AI Agent](/build-your-first-agent.md) **Note:** an agent always connects to a database — deploy a database first (or alongside it). *** ### ⚡ High Performance / Scale[​](#-high-performance--scale "Direct link to ⚡ High Performance / Scale") **Optimize for speed and throughput** → **[Batch Operations](/batch-operations.md)** + **[White Paper](/white-paper.md)** **What you get:** * Bulk insert/update/delete operations * ACID transactions with savepoints * Performance tuning guidance **Start here:** [Batch Operations](/batch-operations.md) → [Transactions](/transactions.md) *** ## 📚 Need Something Specific?[​](#-need-something-specific "Direct link to 📚 Need Something Specific?") ### Transactions & ACID Guarantees[​](#transactions--acid-guarantees "Direct link to Transactions & ACID Guarantees") → **[Transactions Guide](/transactions.md)** Multi-document transactions, isolation levels, savepoints, automatic rollback ### Stored Procedures / Reusable Logic[​](#stored-procedures--reusable-logic "Direct link to Stored Procedures / Reusable Logic") → **[Functions](/client-libraries/advanced-operations.md#functions)** JSON-based function composition, parameterized queries, version control ### Search (Full-Text / Vector / Hybrid)[​](#search-full-text--vector--hybrid "Direct link to Search (Full-Text / Vector / Hybrid)") → **[Advanced Operations - Search](/client-libraries/advanced-operations.md#search-operations)** Text search, vector search, fuzzy matching, hybrid search ### Real-Time / WebSocket[​](#real-time--websocket "Direct link to Real-Time / WebSocket") → **[Advanced Operations - Real-Time](/client-libraries/advanced-operations.md#real-time-operations)** Live queries, WebSocket subscriptions, TTL-based expiration ### Multi-Region Replication[​](#multi-region-replication "Direct link to Multi-Region Replication") → **[White Paper - Ripple Section](/white-paper.md#91-ripple-system)** Cross-instance replication, data propagation, automatic failover ### Key-Value Store[​](#key-value-store "Direct link to Key-Value Store") → **[KV Store Guide](/kv-store.md)** Fast in-memory caching, TTL expiration, document linking *** ## 🆘 Still Not Sure?[​](#-still-not-sure "Direct link to 🆘 Still Not Sure?") **Quick Decision Tree:** 1. **First time with ekoDB?** * → [Quick Start](/quick-start.md) (Start here!) 2. **Building an app?** * Want type safety? → [Client Libraries](/client-libraries/.md) * Prefer HTTP calls? → [REST API](/basic-operations.md) 3. **Building with AI?** * Chat/RAG/vector in your app? → [Chat & RAG](/chat-rag.md) * Want an autonomous agent (tools, goals, schedules)? → [Build Your First AI Agent](/build-your-first-agent.md) 4. **Need extreme performance?** * → [Batch Operations](/batch-operations.md) + [Functions](/client-libraries/advanced-operations.md#functions) 5. **Want to understand architecture?** * → [White Paper](/white-paper.md) *** ## 📞 Need Help?[​](#-need-help "Direct link to 📞 Need Help?") * **Email**: * **Support Portal**: [app.ekodb.io/support](https://app.ekodb.io/support) *** **Ready to start?** Pick your path above and let's build! 🚀 --- # Client Libraries ekoDB provides **6 official client libraries** for multiple programming languages: Rust, Python, TypeScript, Go, Kotlin, and JavaScript. We **strongly recommend** using these libraries instead of making direct API calls, as they provide: * ✅ **Type Safety** - Strong typing and compile-time checks * ✅ **Automatic Authentication** - Handles token management automatically * ✅ **Retry Logic** - Automatic retries with exponential backoff * ✅ **Rate Limit Handling** - Respects rate limits and retry-after headers * ✅ **Better Developer Experience** - Intuitive APIs and comprehensive documentation * ✅ **Connection Pooling** - Efficient HTTP connection management * ✅ **Error Handling** - Structured error types and clear error messages ## Choose Your Language[​](#choose-your-language "Direct link to Choose Your Language") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go [![Crates.io](https://img.shields.io/crates/v/ekodb_client.svg)](https://crates.io/crates/ekodb_client) ``` cargo add ekodb_client ``` **Resources:** * 📦 [crates.io/crates/ekodb\_client](https://crates.io/crates/ekodb_client) * 📖 [docs.rs/ekodb\_client](https://docs.rs/ekodb_client) * 💡 Client Library Examples [![PyPI version](https://img.shields.io/pypi/v/ekodb-client.svg)](https://pypi.org/project/ekodb-client/) [![Python versions](https://img.shields.io/pypi/pyversions/ekodb-client.svg)](https://pypi.org/project/ekodb-client/) ``` pip install ekodb-client ``` **Resources:** * 📦 [pypi.org/project/ekodb-client](https://pypi.org/project/ekodb-client/) * 💡 Client Library Examples [![npm version](https://img.shields.io/npm/v/@ekodb/ekodb-client.svg)](https://www.npmjs.com/package/@ekodb/ekodb-client) [![npm downloads](https://img.shields.io/npm/dm/@ekodb/ekodb-client.svg)](https://www.npmjs.com/package/@ekodb/ekodb-client) ``` npm install @ekodb/ekodb-client ``` **Resources:** * 📦 [npmjs.com/package/@ekodb/ekodb-client](https://www.npmjs.com/package/@ekodb/ekodb-client) * 💡 Client Library Examples [![npm version](https://img.shields.io/npm/v/@ekodb/ekodb-client.svg)](https://www.npmjs.com/package/@ekodb/ekodb-client) ``` npm install @ekodb/ekodb-client ``` **Resources:** * 📦 [npmjs.com/package/@ekodb/ekodb-client](https://www.npmjs.com/package/@ekodb/ekodb-client) * 💡 Client Library Examples [![Maven Central](https://img.shields.io/maven-central/v/io.ekodb/ekodb-client-kt.svg)](https://central.sonatype.com/artifact/io.ekodb/ekodb-client-kt) ``` dependencies { implementation("io.ekodb:ekodb-client-kt:0.23.0") } ``` **Resources:** * 📦 [Maven Central](https://central.sonatype.com/artifact/io.ekodb/ekodb-client-kt) * 💡 Client Library Examples [![Go Reference](https://pkg.go.dev/badge/github.com/ekoDB/ekodb-client-go.svg)](https://pkg.go.dev/github.com/ekoDB/ekodb-client-go) [![Go Report Card](https://goreportcard.com/badge/github.com/ekoDB/ekodb-client-go)](https://goreportcard.com/report/github.com/ekoDB/ekodb-client-go) ``` go get github.com/ekoDB/ekodb-client-go ``` **Resources:** * 📦 [pkg.go.dev/github.com/ekoDB/ekodb-client-go](https://pkg.go.dev/github.com/ekoDB/ekodb-client-go) * 💡 Client Library Examples ## Quick Start[​](#quick-start "Direct link to Quick Start") Here's a quick example using TypeScript: ``` import { EkoDBClient } from "@ekodb/ekodb-client"; // Create client const client = new EkoDBClient({ baseURL: "https://{YOUR_SUBDOMAIN}.{ENVIRONMENT}.{PROVIDER}.ekodb.net", apiKey: "your-api-key" }); await client.init(); // Insert a record const result = await client.insert("users", { name: "John Doe", email: "john@example.com" }); console.log("Inserted:", result); ``` Complete Examples See the guides below for complete examples in all supported languages (Rust, Python, TypeScript, Go, Kotlin, JavaScript). ## Next Steps[​](#next-steps "Direct link to Next Steps") * [Installation Guide](/client-libraries/installation.md) - Detailed installation instructions for each language * [Basic Operations](/client-libraries/basic-operations.md) - CRUD operations with client libraries * [GitHub Examples](https://github.com/ekoDB/ekodb-client/tree/main/examples) - 129 (client library & direct HTTP examples) * [API Reference](/basic-operations.md) - Direct HTTP API documentation ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: --- # Advanced Operations This guide covers advanced features available in ekoDB client libraries, including search operations, chat functionality, and real-time data handling. Complete Examples Available For complete, runnable examples, visit the ekoDB Examples Repository. It contains **265+ examples** across Rust, Go, Python, TypeScript, Kotlin, and JavaScript. ## Options Structs[​](#options-structs "Direct link to Options Structs") New in v0.8.0 Cleaner method signatures using builder pattern for operation options (Rust, TypeScript, Kotlin). ### Overview[​](#overview "Direct link to Overview") Instead of long parameter lists, use options structs for cleaner, more maintainable code: **Before (v0.7.x):** ``` await client.insert("users", record, "1h", true, "tx_123", false); // What do these parameters mean? 🤔 ``` **After (v0.8.0):** ``` await client.insert("users", record, { ttl: "1h", bypassRipple: true, transactionId: "tx_123", bypassCache: false }); // Much clearer! ✨ ``` ### Insert with Options[​](#insert-with-options "Direct link to Insert with Options") * 🦀 Rust * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🐍 Python * 🔷 Go ``` use ekodb_client::{Client, Record, InsertOptions}; let mut record = Record::new(); record.insert("name", "Alice"); record.insert("email", "alice@example.com"); // Use builder pattern for options let options = InsertOptions::new() .ttl("1h") .bypass_ripple(true) .transaction_id("tx_123") .bypass_cache(false); let result = client.insert("users", record, Some(options)).await?; ``` ``` import { EkoDBClient, InsertOptions } from "@ekodb/ekodb-client"; const record = { name: "Alice", email: "alice@example.com" }; // Use options object const options: InsertOptions = { ttl: "1h", bypassRipple: true, transactionId: "tx_123", bypassCache: false }; const result = await client.insert("users", record, options); ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const record = { name: "Alice", email: "alice@example.com" }; // Use options object const options = { ttl: "1h", bypassRipple: true, transactionId: "tx_123", bypassCache: false }; const result = await client.insert("users", record, options); ``` ``` import io.ekodb.client.EkoDBClient import io.ekodb.client.types.Record import io.ekodb.client.InsertOptions val record = Record.new() .insert("name", "Alice") .insert("email", "alice@example.com") // Use options object val options = InsertOptions( ttl = "1h", bypassRipple = true, transactionId = "tx_123", bypassCache = false ) val result = client.insert("users", record, options) ``` ``` # Python uses optional keyword arguments (Pythonic approach) record = { "name": "Alice", "email": "alice@example.com" } result = await client.insert( "users", record, ttl="1h", bypass_ripple=True, transaction_id="tx_123", bypass_cache=False ) ``` ``` // Go uses variadic options (idiomatic approach) record := map[string]interface{}{ "name": "Alice", "email": "alice@example.com", } result, err := client.Insert("users", record, InsertOptions{ TTL: stringPtr("1h"), BypassRipple: boolPtr(true), TransactionID: stringPtr("tx_123"), BypassCache: boolPtr(false), }) ``` ### Update with Options[​](#update-with-options "Direct link to Update with Options") * 🦀 Rust * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🐍 Python * 🔷 Go ``` use ekodb_client::UpdateOptions; let options = UpdateOptions::new() .bypass_ripple(true) .transaction_id("tx_456"); let updated = client.update("users", "user-123", updates, Some(options)).await?; ``` ``` const options: UpdateOptions = { bypassRipple: true, transactionId: "tx_456" }; const updated = await client.update("users", "user-123", updates, options); ``` ``` const options = { bypassRipple: true, transactionId: "tx_456" }; const updated = await client.update("users", "user-123", updates, options); ``` ``` val options = UpdateOptions( bypassRipple = true, transactionId = "tx_456" ) val updated = client.update("users", "user-123", updates, options) ``` ``` # Keyword arguments updated = await client.update( "users", "user-123", updates, bypass_ripple=True, transaction_id="tx_456" ) ``` ``` updated, err := client.Update("users", "user-123", updates, UpdateOptions{ BypassRipple: boolPtr(true), TransactionID: stringPtr("tx_456"), }) ``` ### Available Options Structs[​](#available-options-structs "Direct link to Available Options Structs") | Struct | Available Fields | Languages | | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | --------- | | **InsertOptions** | `ttl`, `bypass_ripple`, `transaction_id`, `bypass_cache` | All | | **UpdateOptions** | `bypass_ripple`, `transaction_id`, `bypass_cache` | All | | **UpsertOptions** | `bypass_ripple`, `transaction_id` | All | | **DeleteOptions** | `bypass_ripple`, `transaction_id` | All | | **FindOptions** (Go) / `Query` fields (Rust/TS/Python/Kotlin) | `bypass_cache`, `bypass_ripple`, `limit`, `skip`, `sort`, `filter`, `select_fields`, `exclude_fields` | All | Language-Specific Patterns * **Rust/TypeScript/Kotlin**: Dedicated options structs with builder pattern * **Python**: Optional keyword arguments (Pythonic) * **Go**: Variadic options with pointers (idiomatic) All approaches provide the same functionality with language-appropriate ergonomics. ## Search Operations[​](#search-operations "Direct link to Search Operations") ekoDB provides powerful search capabilities including full-text search, fuzzy search, and vector search. ### Full-Text Search[​](#full-text-search "Direct link to Full-Text Search") Search across all fields in your documents: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::{Client, SearchQuery}; let search = SearchQuery::builder() .query("database") .min_score(0.1) .limit(10) .build(); let results = client.search("articles", search).await?; for result in results.results { println!("Score: {:.4} - {:?}", result.score, result.record); } ``` ``` from ekodb_client import Client, SearchQuery search = SearchQuery( query="database", min_score=0.1, limit=10 ) results = await client.search("articles", search) for result in results.results: print(f"Score: {result.score:.4f} - {result.record}") ``` ``` import { EkoDBClient, SearchQuery } from "@ekodb/ekodb-client"; const search: SearchQuery = { query: "database", minScore: 0.1, limit: 10 }; const results = await client.search("articles", search); for (const result of results.results) { console.log(`Score: ${result.score.toFixed(4)} - ${JSON.stringify(result.record)}`); } ``` ``` const results = await client.search("articles", { query: "database", minScore: 0.1, limit: 10 }); for (const result of results.results) { console.log(`Score: ${result.score.toFixed(4)} - ${JSON.stringify(result.record)}`); } ``` ``` import io.ekodb.client.EkoDBClient val results = client.search("articles") { query = "database" minScore = 0.1 limit = 10 } results.results.forEach { result -> println("Score: ${"%.4f".format(result.score)} - ${result.record}") } ``` ``` import ekodb "github.com/ekoDB/ekodb-client-go" searchQuery := ekodb.SearchQuery{ Query: "database", MinScore: 0.1, Limit: 10, } results, err := client.Search("articles", searchQuery) if err != nil { log.Fatal(err) } for _, result := range results.Results { fmt.Printf("Score: %.4f - %+v\n", result.Score, result.Record) } ``` ### Field-Weighted Search[​](#field-weighted-search "Direct link to Field-Weighted Search") Search with custom field weights to prioritize certain fields: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use std::collections::HashMap; let mut weights = HashMap::new(); weights.insert("title".to_string(), 2.0); weights.insert("description".to_string(), 1.0); let search = SearchQuery::builder() .query("rust database") .fields(vec!["title".to_string(), "description".to_string()]) .weights(weights) .limit(5) .build(); let results = client.search("articles", search).await?; ``` ``` search = SearchQuery( query="rust database", fields=["title", "description"], weights={"title": 2.0, "description": 1.0}, limit=5 ) results = await client.search("articles", search) ``` ``` const search: SearchQuery = { query: "rust database", fields: ["title", "description"], weights: { title: 2.0, description: 1.0 }, limit: 5 }; const results = await client.search("articles", search); ``` ``` const results = await client.search("articles", { query: "rust database", fields: ["title", "description"], weights: { title: 2.0, description: 1.0 }, limit: 5 }); ``` ``` val results = client.search("articles") { query = "rust database" fields = listOf("title", "description") weights = mapOf("title" to 2.0, "description" to 1.0) limit = 5 } ``` ``` searchQuery := ekodb.SearchQuery{ Query: "rust database", Fields: []string{"title", "description"}, Weights: map[string]float64{"title": 2.0, "description": 1.0}, Limit: 5, } results, err := client.Search("articles", searchQuery) ``` ### Fuzzy Search[​](#fuzzy-search "Direct link to Fuzzy Search") Enable typo tolerance with fuzzy matching: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let search = SearchQuery::builder() .query("databse") // Typo: "databse" instead of "database" .fuzzy(true) .fuzziness(2) // Allow up to 2 character differences .limit(10) .build(); let results = client.search("articles", search).await?; ``` ``` search = SearchQuery( query="databse", # Typo fuzzy=True, fuzziness=2, limit=10 ) results = await client.search("articles", search) ``` ``` const search: SearchQuery = { query: "databse", // Typo fuzzy: true, fuzziness: 2, limit: 10 }; const results = await client.search("articles", search); ``` ``` const results = await client.search("articles", { query: "databse", // Typo fuzzy: true, fuzziness: 2, limit: 10 }); ``` ``` val results = client.search("articles") { query = "databse" // Typo fuzzy = true fuzziness = 2 limit = 10 } ``` ``` searchQuery := ekodb.SearchQuery{ Query: "databse", // Typo Fuzzy: true, Fuzziness: 2, Limit: 10, } results, err := client.Search("articles", searchQuery) ``` ### Vector Search[​](#vector-search "Direct link to Vector Search") Perform semantic similarity search using embeddings: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // First, create a collection with vector index let schema = Schema::builder() .add_field("content", FieldType::String) .add_field("embedding", FieldType::Vector(384)) // 384-dimensional vector .build(); client.create_collection("documents", schema).await?; // Insert document with embedding let mut doc = Record::new(); doc.insert("content", "ekoDB is a high-performance database"); doc.insert("embedding", vec![0.1, 0.2, 0.3, /* ... 384 dimensions */]); client.insert("documents", doc, None).await?; // Search by vector similarity let query_vector = vec![0.1, 0.2, 0.3, /* ... */]; let search = SearchQuery::builder() .vector(query_vector) .limit(10) .build(); let results = client.search("documents", search).await?; ``` ``` # Create collection with vector index schema = Schema( fields={ "content": FieldType.String(), "embedding": FieldType.Vector(384) } ) client.create_collection("documents", schema) # Insert with embedding doc = { "content": "ekoDB is a high-performance database", "embedding": [0.1, 0.2, 0.3, ...] # 384 dimensions } await client.insert("documents", doc) # Vector search query_vector = [0.1, 0.2, 0.3, ...] search = SearchQuery(vector=query_vector, limit=10) results = await client.search("documents", search) ``` ``` // Create collection with vector index const schema = { fields: { content: { field_type: "String" }, embedding: { field_type: "Vector", index: { type: "vector", algorithm: "flat", metric: "cosine" } } } }; await client.createCollection("documents", schema); // Insert with embedding await client.insert("documents", { content: "ekoDB is a high-performance database", embedding: [0.1, 0.2, 0.3, /* ... 384 dimensions */] }); // Vector search const queryVector = [0.1, 0.2, 0.3, /* ... */]; const results = await client.search("documents", { vector: queryVector, limit: 10 }); ``` ``` // Insert with embedding await client.insert("documents", { content: "ekoDB is a high-performance database", embedding: [0.1, 0.2, 0.3, /* ... 384 dimensions */] }); // Vector search const queryVector = [0.1, 0.2, 0.3, /* ... */]; const results = await client.search("documents", { vector: queryVector, limit: 10 }); ``` ``` // Insert with embedding val doc = Record.new() .insert("content", "ekoDB is a high-performance database") .insert("embedding", listOf(0.1, 0.2, 0.3, /* ... 384 dimensions */)) client.insert("documents", doc) // Vector search val queryVector = listOf(0.1, 0.2, 0.3, /* ... */) val results = client.search("documents") { vector = queryVector limit = 10 } ``` ``` // Create collection with vector index schema := ekodb.NewSchemaBuilder(). AddField("content", ekodb.NewFieldTypeSchemaBuilder("String").Build()). AddField("embedding", ekodb.NewFieldTypeSchemaBuilder("Vector"). Dimensions(384). Build()). Build() client.CreateCollection("documents", schema) // Insert with embedding doc := ekodb.Record{ "content": "ekoDB is a high-performance database", "embedding": []float64{0.1, 0.2, 0.3, /* ... 384 dimensions */}, } client.Insert("documents", doc) // Vector search queryVector := []float64{0.1, 0.2, 0.3, /* ... */} searchQuery := ekodb.SearchQuery{ Vector: queryVector, Limit: 10, } results, err := client.Search("documents", searchQuery) ``` ### Hybrid Search[​](#hybrid-search "Direct link to Hybrid Search") Combine text and vector search for best results: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let search = SearchQuery::builder() .query("database performance") // Text query .vector(query_vector) // Vector query .limit(10) .build(); let results = client.search("documents", search).await?; ``` ``` search = SearchQuery( query="database performance", # Text query vector=query_vector, # Vector query limit=10 ) results = await client.search("documents", search) ``` ``` const search: SearchQuery = { query: "database performance", // Text query vector: queryVector, // Vector query limit: 10 }; const results = await client.search("documents", search); ``` ``` const results = await client.search("documents", { query: "database performance", // Text query vector: queryVector, // Vector query limit: 10 }); ``` ``` val results = client.search("documents") { query = "database performance" // Text query vector = queryVector // Vector query limit = 10 } ``` ``` searchQuery := ekodb.SearchQuery{ Query: "database performance", // Text query Vector: queryVector, // Vector query Limit: 10, } results, err := client.Search("documents", searchQuery) ``` ## Chat Operations[​](#chat-operations "Direct link to Chat Operations") Build AI-powered chat applications with built-in context management and session handling. ### Basic Chat[​](#basic-chat "Direct link to Basic Chat") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::{Client, CreateChatSessionRequest, ChatMessageRequest, CollectionConfig}; // Create a chat session let session = client.create_chat_session(CreateChatSessionRequest { collections: vec![CollectionConfig { collection_name: "products".to_string(), fields: vec![], search_options: None, }], llm_provider: "openai".to_string(), llm_model: Some("gpt-4.1".to_string()), system_prompt: Some("You are a helpful assistant.".to_string()), ..Default::default() }).await?; // Send a message let response = client.chat_message( &session.chat_id, ChatMessageRequest::new("What products do you have?") ).await?; println!("AI: {:?}", response.responses); ``` ``` # Create a chat session session = await client.create_chat_session( collections=[("products", [])], llm_provider="openai", llm_model="gpt-4.1", system_prompt="You are a helpful assistant." ) # Send a message response = await client.chat_message( session["chat_id"], "What products do you have?" ) print(f"AI: {response['responses']}") ``` ``` // Create a chat session const session = await client.createChatSession({ collections: [{ collection_name: "products", fields: [] }], llm_provider: "openai", llm_model: "gpt-4.1", system_prompt: "You are a helpful assistant.", }); // Send a message const response = await client.chatMessage(session.chat_id, { message: "What products do you have?", }); console.log(`AI: ${response.responses}`); ``` ``` // Create a chat session const session = await client.createChatSession({ collections: [{ collection_name: "products", fields: [] }], llm_provider: "openai", llm_model: "gpt-4.1", system_prompt: "You are a helpful assistant.", }); // Send a message const response = await client.chatMessage(session.chat_id, { message: "What products do you have?", }); console.log(`AI: ${response.responses}`); ``` ``` // Create a chat session val session = client.createChatSession(buildJsonObject { putJsonArray("collections") { add(buildJsonObject { put("collection_name", "products") putJsonArray("fields") {} }) } put("llm_provider", "openai") put("llm_model", "gpt-4.1") put("system_prompt", "You are a helpful assistant.") }) // Send a message val response = client.chatMessage(session["chat_id"].toString(), buildJsonObject { put("message", "What products do you have?") }) println("AI: ${response["responses"]}") ``` ``` import ekodb "github.com/ekoDB/ekodb-client-go" // Create a chat session llmModel := "gpt-4.1" systemPrompt := "You are a helpful assistant." session, err := client.CreateChatSession(ekodb.CreateChatSessionRequest{ Collections: []ekodb.CollectionConfig{{ CollectionName: "products", Fields: []interface{}{}, }}, LLMProvider: "openai", LLMModel: &llmModel, SystemPrompt: &systemPrompt, }) // Send a message response, err := client.ChatMessage(session.ChatID, ekodb.ChatMessageRequest{ Message: "What products do you have?", }) fmt.Printf("AI: %v\n", response.Responses) ``` ### Chat Sessions[​](#chat-sessions "Direct link to Chat Sessions") Manage conversation history with sessions: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // Send multiple messages in the same session let response1 = client.chat_message( &session.chat_id, ChatMessageRequest::new("What's the price of ekoDB Pro?") ).await?; let response2 = client.chat_message( &session.chat_id, ChatMessageRequest::new("What features does it include?") ).await?; // Get session message history let messages = client.get_chat_session_messages( &session.chat_id, None ).await?; println!("Total messages: {}", messages.messages.len()); ``` ``` # Send multiple messages in the same session response1 = await client.chat_message( session["chat_id"], "What's the price of ekoDB Pro?" ) response2 = await client.chat_message( session["chat_id"], "What features does it include?" ) # Get session message history messages = await client.get_chat_session_messages(session["chat_id"]) print(f"Total messages: {messages['total']}") ``` ``` // Send multiple messages in the same session const response1 = await client.chatMessage(session.chat_id, { message: "What's the price of ekoDB Pro?", }); const response2 = await client.chatMessage(session.chat_id, { message: "What features does it include?", }); // Get session message history const messages = await client.getChatSessionMessages(session.chat_id, { limit: 10, sort: "asc", }); console.log(`Total messages: ${messages.messages.length}`); ``` ``` // Send multiple messages in the same session const response1 = await client.chatMessage(session.chat_id, { message: "What's the price of ekoDB Pro?", }); const response2 = await client.chatMessage(session.chat_id, { message: "What features does it include?", }); // Get session message history const messages = await client.getChatSessionMessages(session.chat_id, { limit: 10, sort: "asc", }); console.log(`Total messages: ${messages.messages.length}`); ``` ``` // Send multiple messages in the same session val response1 = client.chatMessage(session["chat_id"].toString(), buildJsonObject { put("message", "What's the price of ekoDB Pro?") }) val response2 = client.chatMessage(session["chat_id"].toString(), buildJsonObject { put("message", "What features does it include?") }) // Get session message history val messages = client.getChatSessionMessages(session["chat_id"].toString()) println("Total messages: ${messages["total"]}") ``` ``` // Send multiple messages in the same session response1, err := client.ChatMessage(session.ChatID, ekodb.ChatMessageRequest{ Message: "What's the price of ekoDB Pro?", }) response2, err := client.ChatMessage(session.ChatID, ekodb.ChatMessageRequest{ Message: "What features does it include?", }) // Get session message history messages, err := client.GetChatSessionMessages(session.ChatID, nil) fmt.Printf("Total messages: %d\n", messages.Total) ``` Schema-Aware Chat Queries ekoDB's chat system features **intelligent query understanding** that works with your natural data structure: * **Field Name Matching**: Queries like "What is the price?" automatically find records with a `price` field, even if it contains numeric data (not text-searchable) * **Multi-Turn Context**: Follow-up questions use conversation history to enhance search relevance * **No Denormalization Required**: Works with structured data as you'd naturally model it **Example:** ``` // Your data structure { "product": "ekoDB", "description": "High-performance database", "price": 99 // Numeric field } ``` When a user asks "What is the price?", ekoDB: 1. Checks the collection schema for fields matching "price" 2. Finds records with that field name 3. Provides the full record to the LLM 4. LLM responds: "The price is $99" **See complete Chat Session examples in all languages:** * **Rust**: `client_chat_sessions.rs` * **Python**: `client_chat_sessions.py` * **TypeScript**: `client_chat_sessions.ts` * **Go**: `client_chat_sessions.go` * **Kotlin**: `ClientChatSessions.kt` ## Real-Time Operations[​](#real-time-operations "Direct link to Real-Time Operations") ### WebSocket Queries[​](#websocket-queries "Direct link to WebSocket Queries") Subscribe to real-time data changes: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::WebSocketClient; // Connect to WebSocket let ws_url = "wss://your-subdomain.production.google.ekodb.net"; let mut ws_client = client.websocket(ws_url).await?; // Subscribe to collection changes let results = ws_client.find_all("users").await?; // Process real-time updates for record in results { println!("New/Updated record: {:?}", record); } ws_client.close().await?; ``` ``` # Connect to WebSocket ws_url = "wss://your-subdomain.production.google.ekodb.net" ws_client = client.websocket(ws_url) # Subscribe to collection changes results = await ws_client.find_all("users") # Process real-time updates for record in results: print(f"New/Updated record: {record}") ws_client.close() ``` ``` // Connect to WebSocket const wsUrl = "wss://your-subdomain.production.google.ekodb.net"; const wsClient = await client.websocket(wsUrl); // Subscribe to collection changes const results = await wsClient.findAll("users"); // Process real-time updates for (const record of results) { console.log("New/Updated record:", record); } await wsClient.close(); ``` ``` // Connect to WebSocket const wsUrl = "wss://your-subdomain.production.google.ekodb.net"; const wsClient = await client.websocket(wsUrl); // Subscribe to collection changes const results = await wsClient.findAll("users"); // Process real-time updates for (const record of results) { console.log("New/Updated record:", record); } await wsClient.close(); ``` ``` // Connect to WebSocket val wsUrl = "wss://your-subdomain.production.google.ekodb.net" val wsClient = client.websocket(wsUrl) // Subscribe to collection changes val results = wsClient.findAll("users") // Process real-time updates results.forEach { record -> println("New/Updated record: $record") } wsClient.close() ``` ``` // Connect to WebSocket wsURL := "wss://your-subdomain.production.google.ekodb.net" wsClient, err := client.WebSocket(wsURL) // Subscribe to collection changes results, err := wsClient.FindAll("users") // Process real-time updates for _, record := range results { fmt.Printf("New/Updated record: %+v\n", record) } wsClient.Close() ``` ### WebSocket CRUD Operations[​](#websocket-crud-operations "Direct link to WebSocket CRUD Operations") All 14 server-supported CRUD operations are available over WebSocket. The persistent WS connection eliminates HTTP overhead per request — zero TLS handshake, reuses the authenticated connection. All methods support `messageId` for concurrent request correlation. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 🐹 Go * 🟣 Kotlin ``` // Connect via convenience method (derives WS URL, attaches schema cache) let ws = client.connect_ws().await?; // Insert let record = ws.insert("users", json!({"name": "Alice"}), None).await?; // Query with filter let results = ws.query("users", Some(json!({"field": "status", "operator": "Eq", "value": "active"})), None, Some(10), None, ).await?; // Find by ID let user = ws.find_by_id("users", "record-id").await?; // Update ws.update("users", "record-id", json!({"name": "Updated"}), None).await?; // Delete ws.delete("users", "record-id", None).await?; // Batch operations ws.batch_insert("logs", vec![json!({"msg": "a"}), json!({"msg": "b"})], None).await?; // Search let hits = ws.text_search("docs", "rust async", None, Some(10)).await?; // Collection management let collections = ws.list_collections().await?; ws.create_collection("new_coll", None).await?; // Atomic field actions ws.update_with_action("counters", "views", "increment", "count", Some(json!(1))).await?; ``` ``` ws = await client.websocket("wss://my-first-db.development.google.ekodb.net") result = await ws.ws_insert("users", {"name": "Alice"}) results = await ws.ws_query("users", filter={"field": "status", "operator": "Eq", "value": "active"}) user = await ws.ws_find_by_id("users", "record-id") await ws.ws_update("users", "record-id", {"name": "Updated"}) await ws.ws_delete("users", "record-id") await ws.ws_batch_insert("logs", [{"msg": "a"}, {"msg": "b"}]) hits = await ws.ws_text_search("docs", "python async", limit=10) collections = await ws.ws_list_collections() ``` ``` const ws = new WebSocketClient("wss://my-first-db.development.google.ekodb.net", token); const record = await ws.insert("users", { name: "Alice" }); const results = await ws.query("users", { filter: { field: "status", operator: "Eq", value: "active" } }); const user = await ws.findById("users", "record-id"); await ws.update("users", "record-id", { name: "Updated" }); await ws.delete("users", "record-id"); await ws.batchInsert("logs", [{ msg: "a" }, { msg: "b" }]); const hits = await ws.textSearch("docs", "typescript async", undefined, 10); const collections = await ws.listCollections(); ``` ``` ws, _ := client.ConnectWS() result, _ := ws.Insert("users", map[string]interface{}{"name": "Alice"}) results, _ := ws.Query("users", ekodb.QueryOptions{ Filter: map[string]interface{}{"field": "status", "operator": "Eq", "value": "active"}, Limit: 10, }) user, _ := ws.FindByID("users", "record-id") ws.Update("users", "record-id", map[string]interface{}{"name": "Updated"}) ws.Delete("users", "record-id") ws.BatchInsert("logs", []map[string]interface{}{{"msg": "a"}, {"msg": "b"}}) hits, _ := ws.TextSearch("docs", "go async", nil, 10) collections, _ := ws.ListCollections() ``` ``` val ws = client.websocket("wss://my-first-db.development.google.ekodb.net") ws.connect() val record = ws.insert("users", buildJsonObject { put("name", "Alice") }) val results = ws.query("users", filter = buildJsonObject { put("field", "status"); put("operator", "Eq"); put("value", "active") }) val user = ws.findById("users", "record-id") ws.update("users", "record-id", buildJsonObject { put("name", "Updated") }) ws.delete("users", "record-id") ws.batchInsert("users", listOf(buildJsonObject { put("name", "Bob") })) val collections = ws.listCollections() ``` ### Schema Cache[​](#schema-cache "Direct link to Schema Cache") The schema cache stores each collection's `primary_key_alias` and version in memory. This ensures `extractRecordId()` works correctly regardless of how users configure their ID field names. The cache is LRU with configurable TTL, and auto-invalidates via WebSocket `SchemaChanged` events. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 🐹 Go * 🟣 Kotlin ``` // Enable at client creation let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("key") .schema_cache(true) .schema_cache_ttl(300) // seconds .schema_cache_max(100) // max collections .build()?; // Extract IDs correctly with any primary_key_alias let id = client.extract_id("users", &record); // Auto-invalidates when connected via WS let ws = client.connect_ws().await?; ``` ``` # Schema cache is enabled server-side via the Rust client internals. # In Python, use extractRecordId for safe ID extraction: from ekodb_client import extract_record_id # Tries "id", then "_id", then custom candidates record_id = extract_record_id(record) # With custom alias record_id = extract_record_id(record, extra_candidates=["user_id"]) ``` ``` import { SchemaCache, extractRecordId } from "@ekodb/ekodb-client"; // Create and attach cache to WS client const cache = new SchemaCache({ enabled: true, ttlSeconds: 300, maxEntries: 100, }); ws.setSchemaCache(cache); // Extract IDs — cache-aware via WS client const id = ws.extractId("users", record); // Or standalone (tries "id", "_id") const id2 = extractRecordId(record); ``` ``` // Enable schema cache on client client.EnableSchemaCache(5*time.Minute, 100) // Extract IDs using cached primary_key_alias id := client.ExtractRecordID("users", record) // ConnectWS auto-attaches the cache ws, _ := client.ConnectWS() ``` ``` // Create and attach schema cache val cache = SchemaCache(enabled = true, maxEntries = 100, ttlMs = 300_000) ws.schemaCache = cache // Extract IDs using cached alias val id = ws.extractId("users", record) // Or standalone val id2 = extractRecordId(record, listOf("user_id")) ``` ### SSE Subscriptions[​](#sse-subscriptions "Direct link to SSE Subscriptions") Subscribe to collection mutations via Server-Sent Events. Works behind reverse proxies that block WebSocket upgrades. Also delivers `schema_changed` events for automatic schema cache invalidation. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 🐹 Go * 🟣 Kotlin ``` let rx = client.subscribe_sse("orders", None, None).await?; while let Some(event) = rx.recv().await { println!("{}: {} on {}", event.event, event.record_ids.join(", "), event.collection); } // With filter — only receive mutations where status = "active" let rx = client.subscribe_sse( "orders", Some("status"), Some("active"), ).await?; ``` ``` # SSE subscriptions are available via the REST endpoint import httpx async with httpx.AsyncClient() as http: async with http.stream( "GET", f"{base_url}/api/subscribe/orders", headers={"Authorization": f"Bearer {token}"}, ) as response: async for line in response.aiter_lines(): if line.startswith("data: "): data = json.loads(line[6:]) print(f"Mutation: {data}") ``` ``` // Using the EventSource API or fetch with streaming const response = await fetch( `${baseUrl}/api/subscribe/orders?filter_field=status&filter_value=active`, { headers: { Authorization: `Bearer ${token}` } }, ); const reader = response.body!.getReader(); const decoder = new TextDecoder(); while (true) { const { done, value } = await reader.read(); if (done) break; const text = decoder.decode(value); // Parse SSE events: "event: mutation\ndata: {...}\n\n" for (const line of text.split("\n")) { if (line.startsWith("data: ")) { const event = JSON.parse(line.slice(6)); console.log("Mutation:", event); } } } ``` ``` // Using the streaming HTTP client req, _ := http.NewRequest("GET", baseURL+"/api/subscribe/orders?filter_field=status&filter_value=active", nil) req.Header.Set("Authorization", "Bearer "+token) req.Header.Set("Accept", "text/event-stream") resp, _ := client.StreamClient.Do(req) defer resp.Body.Close() scanner := bufio.NewScanner(resp.Body) for scanner.Scan() { line := scanner.Text() if strings.HasPrefix(line, "data: ") { data := line[6:] fmt.Println("Mutation:", data) } } ``` ``` // Using Ktor streaming val client = HttpClient { install(ContentNegotiation) { json() } } client.prepareGet("$baseUrl/api/subscribe/orders") { header("Authorization", "Bearer $token") header("Accept", "text/event-stream") }.execute { response -> val channel = response.bodyAsChannel() while (!channel.isClosedForRead) { val line = channel.readUTF8Line() ?: break if (line.startsWith("data: ")) { val data = line.substring(6) println("Mutation: $data") } } } ``` > SSE also delivers `schema_changed` events, automatically invalidating the client's schema cache when a collection's configuration changes. *** ## Joins[​](#joins "Direct link to Joins") ekoDB supports cross-collection joins to combine data from multiple collections in a single query. ### Single Collection Join[​](#single-collection-join "Direct link to Single Collection Join") Join users with their department data: * 🦀 Rust * 📘 TypeScript * 🐍 Python * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::{Client, QueryBuilder, JoinBuilder}; // Join users with departments let join = JoinBuilder::single( "departments", // Target collection "department_id", // Local field (in users) "id", // Foreign field (in departments) "department" // Output field name ); let query = QueryBuilder::new() .join(join) .limit(10) .build(); let users = client.find("users", query, None).await?; for user in users { println!("User: {:?}, Department: {:?}", user["name"], user["department"]); } ``` ``` import { EkoDBClient, QueryBuilder, JoinBuilder } from "@ekodb/ekodb-client"; // Join users with departments const join = JoinBuilder.single( "departments", // Target collection "department_id", // Local field (in users) "id", // Foreign field (in departments) "department" // Output field name ); const query = new QueryBuilder() .join(join) .limit(10) .build(); const users = await client.find("users", query); users.forEach(user => { console.log(`User: ${user.name}, Department: ${user.department[0]?.name}`); }); ``` ``` from ekodb_client import Client, QueryBuilder, JoinBuilder # Join users with departments join = JoinBuilder.single( "departments", # Target collection "department_id", # Local field (in users) "id", # Foreign field (in departments) "department" # Output field name ) query = QueryBuilder() \ .join(join) \ .limit(10) \ .build() users = await client.find("users", query) for user in users: print(f"User: {user['name']}, Department: {user['department'][0]['name']}") ``` ``` const { EkoDBClient, QueryBuilder, JoinBuilder } = require("@ekodb/ekodb-client"); // Join users with departments const join = JoinBuilder.single( "departments", // Target collection "department_id", // Local field (in users) "id", // Foreign field (in departments) "department" // Output field name ); const query = new QueryBuilder() .join(join) .limit(10) .build(); const users = await client.find("users", query); users.forEach(user => { console.log(`User: ${user.name}, Department: ${user.department[0]?.name}`); }); ``` ``` import io.ekodb.client.EkoDBClient import io.ekodb.client.QueryBuilder import io.ekodb.client.JoinBuilder // Join users with departments val join = JoinBuilder.single( "departments", // Target collection "department_id", // Local field (in users) "id", // Foreign field (in departments) "department" // Output field name ) val query = QueryBuilder() .join(join) .limit(10) .build() val users = client.find("users", query) users.forEach { user -> println("User: ${user["name"]}, Department: ${user["department"]}") } ``` ``` import ekodb "github.com/ekoDB/ekodb-client-go" // Join users with departments join := ekodb.JoinBuilder.Single( "departments", // Target collection "department_id", // Local field (in users) "id", // Foreign field (in departments) "department", // Output field name ) query := ekodb.NewQueryBuilder(). Join(join). Limit(10). Build() users, _ := client.Find("users", query) for _, user := range users { fmt.Printf("User: %v, Department: %v\n", user["name"], user["department"]) } ``` Complete Join Examples **Join examples** - Single and multi-collection joins with filtering: * **Rust**: `client_joins.rs` * **Python**: `client_joins.py` * **TypeScript**: `client_joins.ts` * **JavaScript**: `client_joins.js` * **Go**: `client_joins.go` * **Kotlin**: `ClientJoins.kt` *** ### TTL (Time-To-Live)[​](#ttl-time-to-live "Direct link to TTL (Time-To-Live)") Set automatic expiration for documents: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::options::InsertOptions; // Insert with 1 hour TTL let mut session = Record::new(); session.insert("user_id", "user-123"); session.insert("token", "abc123"); let options = InsertOptions::new().ttl("1h"); // Expires in 1 hour let result = client.insert("sessions", session, Some(options)).await?; ``` ``` # Insert with 1 hour TTL session = { "user_id": "user-123", "token": "abc123" } result = await client.insert( "sessions", session, ttl="1h" # Expires in 1 hour ) ``` ``` // Insert with 1 hour TTL const session = { userId: "user-123", token: "abc123" }; const result = await client.insert("sessions", session, { ttl: "1h" // Expires in 1 hour }); ``` ``` // Insert with 1 hour TTL const session = { userId: "user-123", token: "abc123" }; const result = await client.insert("sessions", session, { ttl: "1h" // Expires in 1 hour }); ``` ``` // Insert with 1 hour TTL val session = mapOf( "user_id" to "user-123", "token" to "abc123" ) val result = client.insert( "sessions", session, InsertOptions(ttl = "1h") // Expires in 1 hour ) ``` ``` // Insert with 1 hour TTL session := ekodb.Record{ "user_id": "user-123", "token": "abc123", } result, err := client.Insert("sessions", session, ekodb.InsertOptions{ TTL: "1h", // Expires in 1 hour }) ``` Complete TTL Examples **Document TTL examples** - Insert with expiration, verify expiration works: * **Rust**: `client_document_ttl.rs` * **Python**: `client_document_ttl.py` * **TypeScript**: `client_document_ttl.ts` * **JavaScript**: `client_document_ttl.js` * **Go**: `client_document_ttl.go` * **Kotlin**: `ClientDocumentTtl.kt` **WebSocket TTL examples** - TTL with real-time connections: * **Rust**: `client_websocket_ttl.rs` * **Python**: `client_websocket_ttl.py` * **TypeScript**: `client_websocket_ttl.ts` * **JavaScript**: `client_websocket_ttl.js` * **Go**: `client_websocket_ttl.go` * **Kotlin**: `ClientWebsocketTtl.kt` *** ## Transactions[​](#transactions "Direct link to Transactions") ekoDB supports ACID transactions with multiple isolation levels. Transactions ensure data consistency when performing multiple operations that must succeed or fail together. ### Isolation Levels[​](#isolation-levels "Direct link to Isolation Levels") | Level | Description | | ------------------ | ---------------------------------------------------- | | `READ_UNCOMMITTED` | Can read uncommitted changes from other transactions | | `READ_COMMITTED` | Only reads committed data (default) | | `REPEATABLE_READ` | Same data on re-read within transaction | | `SERIALIZABLE` | Highest isolation, transactions appear sequential | ### Basic Transaction[​](#basic-transaction "Direct link to Basic Transaction") * 🔷 Go * 🦀 Rust * 🐍 Python * 📘 TypeScript ``` // Start a transaction txID, err := client.BeginTransaction("SERIALIZABLE") if err != nil { log.Fatal(err) } // Perform operations within transaction context // (Pass txID in headers for transactional operations) // Check transaction status status, err := client.GetTransactionStatus(txID) fmt.Printf("Status: %v\n", status) // Commit when all operations succeed err = client.CommitTransaction(txID) if err != nil { // Rollback on failure client.RollbackTransaction(txID) log.Fatal(err) } ``` ``` // Start a transaction let tx_id = client.begin_transaction("SERIALIZABLE").await?; // Perform operations within transaction context // Commit when done client.commit_transaction(&tx_id).await?; // Or rollback on failure // client.rollback_transaction(&tx_id).await?; ``` ``` # Start a transaction tx_id = client.begin_transaction("SERIALIZABLE") # Perform operations within transaction context try: # ... your operations ... client.commit_transaction(tx_id) except Exception as e: client.rollback_transaction(tx_id) raise e ``` ``` // Start a transaction const txId = await client.beginTransaction("SERIALIZABLE"); try { // Perform operations within transaction context // Commit when done await client.commitTransaction(txId); } catch (error) { // Rollback on failure await client.rollbackTransaction(txId); throw error; } ``` Transaction Best Practices * Use the lowest isolation level that meets your consistency requirements * Keep transactions short to minimize lock contention * Always handle rollback in error cases * For detailed transaction patterns, see [Transactions](/transactions.md) *** ## RAG Helpers[​](#rag-helpers "Direct link to RAG Helpers") The Go client includes convenience methods for RAG (Retrieval-Augmented Generation) workflows: ### Generate Embeddings[​](#generate-embeddings "Direct link to Generate Embeddings") Generate embedding vectors from text using ekoDB's native Functions: ``` // Generate embedding for text embedding, err := client.Embed("Hello world", "text-embedding-3-small") if err != nil { log.Fatal(err) } fmt.Printf("Generated %d dimensions\n", len(embedding)) ``` ### Text Search[​](#text-search "Direct link to Text Search") Perform full-text search with stemming and fuzzy matching: ``` // Search for documents by text results, err := client.TextSearch("documents", "database performance", 10) if err != nil { log.Fatal(err) } for _, doc := range results { fmt.Printf("Found: %v\n", doc["title"]) } ``` ### Hybrid Search[​](#hybrid-search-1 "Direct link to Hybrid Search") Combine semantic similarity (vector) with keyword matching (text): ``` // Generate embedding for query embedding, _ := client.Embed("How to optimize queries?", "text-embedding-3-small") // Perform hybrid search results, err := client.HybridSearch("documents", "optimize queries", embedding, 5) if err != nil { log.Fatal(err) } ``` ### Find All Records[​](#find-all-records "Direct link to Find All Records") Simple method to retrieve all records from a collection: ``` // Get all messages (up to limit) allMessages, err := client.FindAll("messages", 1000) if err != nil { log.Fatal(err) } fmt.Printf("Found %d messages\n", len(allMessages)) ``` *** ## Functions[​](#functions "Direct link to Functions") Server-Side Feature Functions are ekoDB's stored procedures system that runs on the server. They can be called from any client library or via REST API to execute complex business logic, queries, CRUD operations, AI workflows, and batch processing. Deep Dive For comprehensive architecture details, operation types, and advanced patterns, see [Functions Architecture](/reference/functions-architecture.md). **Functions** let you create, store, and execute complete business logic as composable operations. Define your data logic once in ekoDB, then call it like puzzle pieces from any client. ### What You Can Do[​](#what-you-can-do "Direct link to What You Can Do") * ✅ **Complete business logic** - Queries, CRUD, AI operations in one place * ✅ **Parameterize everything** - Dynamic values via `{{param_name}}` * ✅ **Version control** - Track function versions * ✅ **Compose like puzzles** - Chain operations together * ✅ **Call from anywhere** - REST API or any client library ### Function Capabilities[​](#function-capabilities "Direct link to Function Capabilities") * **Query Operations**: Find, filter, search, vector search, hybrid search * **CRUD Operations**: Insert, update, delete (single and batch) * **Transformations**: Group, project, count * **AI Operations**: Chat completions, embeddings generation * **Conditional Logic**: If/then/else, foreach loops * **External Integrations**: HTTP requests to any REST API ### Basic Example[​](#basic-example "Direct link to Basic Example") Create a function to query active users: ``` POST /api/functions Content-Type: application/json { "label": "get_active_users", "name": "Get Active Users", "description": "Returns active users with a limit", "parameters": { "limit": { "default": 10, "required": false } }, "functions": [ { "type": "Query", "collection": "users", "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } }, "limit": "{{limit}}" } ] } ``` `POST /api/functions` is **create-only**. If a function with the same `label` already exists, it returns **`409 Conflict`** rather than overwriting it — use `PUT /api/functions/{id_or_label}` (see [Managing Functions](#managing-functions)) to update an existing function. This keeps every saved label pointing at exactly one definition, so a call always resolves to the function you expect. ### Call a Function[​](#call-a-function "Direct link to Call a Function") Execute via REST API: ``` POST /api/functions/get_active_users Content-Type: application/json { "limit": 20 } ``` ### Managing Functions[​](#managing-functions "Direct link to Managing Functions") **REST API:** ``` # List all functions GET /api/functions # Get a specific function by ID or label GET /api/functions/get_active_users # Update a function by ID or label PUT /api/functions/{id_or_label} # Delete a function by ID or label DELETE /api/functions/get_active_users ``` **Client Library Methods:** * 🦀 Rust * 📘 TypeScript * 📦 JavaScript * 🐍 Python * 🔷 Go * 🟣 Kotlin ``` use ekodb_client::{Client, Function, ParameterDefinition, UserFunction}; // Create a user function using the builder pattern let user_func = UserFunction::new("get_active_users", "Get Active Users") .with_version("1.0.0") .with_parameter(ParameterDefinition { name: "collection".to_string(), required: true, description: Some("Collection to query".to_string()), default: None, }) .with_function(Function::FindAll { collection: "{{collection}}".to_string(), }) .with_tag("users") .with_tag("query"); let func_id = client.save_user_function(user_func).await?; // Get a user function by label let func = client.get_user_function("get_active_users").await?; // List all user functions (optionally filter by tags) let all_funcs = client.list_user_functions(None).await?; let tagged_funcs = client.list_user_functions(Some(vec!["users".to_string()])).await?; // Update a user function client.update_user_function("get_active_users", updated_func).await?; // Delete a user function client.delete_user_function("get_active_users").await?; ``` ``` // Create a user function const funcId = await client.saveUserFunction({ label: "get_active_users", name: "Get Active Users", version: "1.0.0", parameters: { collection: { required: true, description: "Collection to query" } }, functions: [{ type: "FindAll", collection: "{{collection}}" }], tags: ["users", "query"] }); // Get a user function by label const func = await client.getUserFunction("get_active_users"); // List all user functions (optionally filter by tags) const allFuncs = await client.listUserFunctions(); const taggedFuncs = await client.listUserFunctions(["users"]); // Update a user function await client.updateUserFunction("get_active_users", updatedFunc); // Delete a user function await client.deleteUserFunction("get_active_users"); ``` ``` // Create a user function const funcId = await client.saveUserFunction({ label: "get_active_users", name: "Get Active Users", version: "1.0.0", parameters: { collection: { required: true, description: "Collection to query" } }, functions: [{ type: "FindAll", collection: "{{collection}}" }], tags: ["users", "query"] }); // Get a user function by label const func = await client.getUserFunction("get_active_users"); // List all user functions (optionally filter by tags) const allFuncs = await client.listUserFunctions(); const taggedFuncs = await client.listUserFunctions(["users"]); // Update a user function await client.updateUserFunction("get_active_users", updatedFunc); // Delete a user function await client.deleteUserFunction("get_active_users"); ``` ``` # Create a user function func_id = await client.save_user_function({ "label": "get_active_users", "name": "Get Active Users", "version": "1.0.0", "parameters": { "collection": {"required": True, "description": "Collection to query"} }, "functions": [{"type": "FindAll", "collection": "{{collection}}"}], "tags": ["users", "query"] }) # Get a user function by label func = await client.get_user_function("get_active_users") # List all user functions (optionally filter by tags) all_funcs = await client.list_user_functions() tagged_funcs = await client.list_user_functions(tags=["users"]) # Update a user function await client.update_user_function("get_active_users", updated_func) # Delete a user function await client.delete_user_function("get_active_users") ``` ``` // Create a user function funcID, err := client.SaveUserFunction(ekodb.UserFunction{ Label: "get_active_users", Name: "Get Active Users", Version: &version, Parameters: map[string]ekodb.ParameterDefinition{ "collection": {Required: true, Description: "Collection to query"}, }, Functions: []ekodb.FunctionStageConfig{ ekodb.StageFindAll("{{collection}}"), }, Tags: []string{"users", "query"}, }) // Get a user function by label fn, err := client.GetUserFunction("get_active_users") // List all user functions (optionally filter by tags) allFuncs, err := client.ListUserFunctions(nil) taggedFuncs, err := client.ListUserFunctions([]string{"users"}) // Update a user function err = client.UpdateUserFunction("get_active_users", updatedFunc) // Delete a user function err = client.DeleteUserFunction("get_active_users") ``` ``` // Create a user function val funcId = client.saveUserFunction(buildJsonObject { put("label", "get_active_users") put("name", "Get Active Users") put("version", "1.0.0") putJsonObject("parameters") { putJsonObject("collection") { put("required", true) put("description", "Collection to query") } } putJsonArray("functions") { add(buildJsonObject { put("type", "FindAll") put("collection", "{{collection}}") }) } putJsonArray("tags") { add(JsonPrimitive("users")) add(JsonPrimitive("query")) } }) // Get a user function by label val func = client.getUserFunction("get_active_users") // List all user functions (optionally filter by tags) val allFuncs = client.listUserFunctions() val taggedFuncs = client.listUserFunctions(listOf("users")) // Update a user function client.updateUserFunction("get_active_users", updatedFunc) // Delete a user function client.deleteUserFunction("get_active_users") ``` ### Parameters[​](#parameters "Direct link to Parameters") Make functions dynamic with parameters: ``` { "parameters": { "status": { "default": "active", "required": false, "description": "Filter by status" }, "min_amount": { "required": true, "description": "Minimum amount" } } } ``` Reference parameters in your function definitions using `{{param_name}}`: ``` { "type": "Query", "collection": "orders", "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "{{status}}" } } } ``` ### Available Operations[​](#available-operations "Direct link to Available Operations") Functions support these operation types: #### Query Operations[​](#query-operations "Direct link to Query Operations") * `FindAll` - Retrieve all records * `Query` - Advanced filtering, sorting, pagination * `VectorSearch` - Semantic similarity search * `HybridSearch` - Combine text + vector search * `TextSearch` - Full-text search * `FindById` - Get specific record by ID * `FindOne` - Find one by key/value #### CRUD Operations[​](#crud-operations "Direct link to CRUD Operations") * `Insert` - Insert single record * `BatchInsert` - Insert multiple records * `Update` - Update with filter * `UpdateById` - Update specific record * `Delete` - Delete with filter * `DeleteById` - Delete specific record * `BatchDelete` - Delete multiple records #### Transformations[​](#transformations "Direct link to Transformations") * `Group` - Group and aggregate * `Project` - Select/exclude fields * `Count` - Count records #### AI Operations[​](#ai-operations "Direct link to AI Operations") * `Chat` - AI chat completions * `Embed` - Generate embeddings #### Logic & Control[​](#logic--control "Direct link to Logic & Control") * `If` - Conditional execution * `ForEach` - Loop over records * `CallFunction` - Call another function #### External Integrations[​](#external-integrations "Direct link to External Integrations") * `HttpRequest` - Call external APIs (Stripe, SendGrid, etc.) ### Example: AI-Powered Search[​](#example-ai-powered-search "Direct link to Example: AI-Powered Search") Combine embeddings with search: ``` { "label": "smart_search", "name": "Smart Product Search", "parameters": { "query": { "required": true } }, "functions": [ { "type": "Embed", "input_field": "query", "output_field": "query_embedding" }, { "type": "HybridSearch", "collection": "products", "query_text": "{{query}}", "query_vector": "{{query_embedding}}", "limit": 10 } ] } ``` ### Example: Batch Processing[​](#example-batch-processing "Direct link to Example: Batch Processing") Process multiple records with AI: ``` { "label": "enrich_articles", "name": "AI Content Enrichment", "functions": [ { "type": "Query", "collection": "articles", "filter": { "type": "Condition", "content": { "field": "embedding", "operator": "Eq", "value": null } }, "limit": 100 }, { "type": "ForEach", "functions": [ { "type": "Embed", "input_field": "content", "output_field": "embedding" }, { "type": "UpdateById", "collection": "articles", "record_id": "{{id}}", "updates": { "embedding": "{{embedding}}" } } ] } ] } ``` ### Best Practices[​](#best-practices "Direct link to Best Practices") * **Keep it simple** - Start with single operations, build up * **Use parameters** - Make functions reusable with dynamic values * **Filter early** - Reduce data before expensive operations * **Add descriptions** - Document what each function does * **Tag for organization** - Use tags like `analytics`, `users`, `ai` * **Version your functions** - Track changes with version field * **Test thoroughly** - Validate with edge cases ### Storage[​](#storage "Direct link to Storage") Functions are stored in a dedicated collection: `functions_{db_name}` (configurable) ### Complete Documentation[​](#complete-documentation "Direct link to Complete Documentation") For complete details including: * All operation types and parameters * Advanced parameter resolution * Conditional logic patterns * External API integration examples * Error handling **See complete Function examples in all languages:** * **Rust**: `client_function_composition.rs` * **Python**: `client_function_composition.py` * **TypeScript**: `client_function_composition.ts` * **JavaScript**: `client_function_composition.js` * **Go**: `client_functions.go` * **Kotlin**: `ClientFunctionComposition.kt` **See User Functions CRUD examples:** * **Rust**: `client_user_functions.rs` * **Python**: `client_user_functions.py` * **TypeScript**: `client_user_functions.ts` * **Go**: `client_user_functions.go` * **Kotlin**: `ClientUserFunctions.kt` *** ## WebSocket Chat Streaming[​](#websocket-chat-streaming "Direct link to WebSocket Chat Streaming") Stream real-time LLM responses via WebSocket. The server sends `ChatStreamEvent` messages as the model generates text, calls tools, or completes. ### Event Types[​](#event-types "Direct link to Event Types") | Event | Description | | ---------- | ----------------------------------------------------------------------------------------- | | `chunk` | A text token from the LLM (stream these as they arrive) | | `end` | Stream completed — includes `messageId`, `executionTimeMs`, `tokenUsage`, `contextWindow` | | `toolCall` | The LLM wants to execute a client-side tool | | `error` | An error occurred during streaming | ### Streaming Example[​](#streaming-example "Direct link to Streaming Example") * 🔷 Go * 📘 TypeScript * 🦀 Rust * 🐍 Python * 🟣 Kotlin ``` ws, _ := client.WebSocket("ws://localhost:8080") defer ws.Close() eventCh, _ := ws.ChatSend(chatID, "What is the capital of France?") for event := range eventCh { switch event.Type { case "chunk": fmt.Print(event.Content) case "end": fmt.Printf("\nDone in %dms\n", event.ExecutionTimeMs) if event.ContextWindow > 0 { fmt.Printf("Context window: %d tokens\n", event.ContextWindow) } case "toolCall": fmt.Printf("[Tool] %s\n", event.ToolName) ws.SendToolResult(chatID, event.CallID, true, map[string]string{"result": "done"}, "") case "error": fmt.Printf("Error: %s\n", event.Error) } } ``` ``` const ws = new WebSocketClient("ws://localhost:8080/api/ws", token); const stream = await ws.chatSend(chatId, "What is the capital of France?"); stream.on("event", (event) => { switch (event.type) { case "chunk": process.stdout.write(event.content); break; case "end": console.log(`\nDone in ${event.executionTimeMs}ms`); if (event.contextWindow) { console.log(`Context window: ${event.contextWindow} tokens`); } break; case "toolCall": console.log(`[Tool] ${event.toolName}`); ws.sendToolResult(chatId, event.callId, true, { result: "done" }); break; case "error": console.error(event.error); break; } }); ``` ``` let ws = client.websocket("ws://localhost:8080").await?; let mut stream = ws.chat_send_with_tools( &chat_id, "What is the capital of France?", None, None, None, None, ).await?; while let Some(event) = stream.recv().await { match event { ChatStreamEvent::Chunk(text) => print!("{}", text), ChatStreamEvent::End { execution_time_ms, context_window, .. } => { println!("\nDone in {}ms", execution_time_ms); if let Some(cw) = context_window { println!("Context window: {} tokens", cw); } } ChatStreamEvent::ToolCall { tool_name, call_id, arguments, .. } => { println!("[Tool] {}", tool_name); let result = serde_json::json!({"result": "done"}); ws.send_tool_result(&chat_id, &call_id, true, Some(result), None).await?; } ChatStreamEvent::Error(err) => eprintln!("Error: {}", err), } } ``` ``` ws = await client.websocket("ws://localhost:8080") stream = await ws.chat_send(chat_id, "What is the capital of France?") async for event in stream: if event.type == "chunk": print(event.content, end="") elif event.type == "end": print(f"\nDone in {event.execution_time_ms}ms") if event.context_window: print(f"Context window: {event.context_window} tokens") elif event.type == "tool_call": print(f"[Tool] {event.tool_name}") await ws.send_tool_result(chat_id, event.call_id, True, {"result": "done"}) elif event.type == "error": print(f"Error: {event.error}") ``` ``` val ws = client.webSocket("ws://localhost:8080") val events = ws.chatSend(chatId, "What is the capital of France?") events.collect { event -> when (event) { is ChatStreamEvent.Chunk -> print(event.content) is ChatStreamEvent.End -> { println("\nDone in ${event.executionTimeMs}ms") event.contextWindow?.let { println("Context window: $it tokens") } } is ChatStreamEvent.ToolCall -> { println("[Tool] ${event.toolName}") val result = mapOf("result" to "done") ws.sendToolResult(chatId, event.callId, true, result) } is ChatStreamEvent.Error -> println("Error: ${event.error}") } } ``` Slow tools: keep the turn alive The server bounds how long it waits for a client tool result at `client_tool_timeout_secs` (default **60s**, tunable at runtime via `/api/config`). A tool that runs longer — a pending human confirmation or a long-running operation — would otherwise time the turn out mid-response. To extend the window, send a **keepalive** to the REST tool-result endpoint (`POST /api/chat/{chat_id}/tool-result` with `{ "call_id": "...", "keepalive": true }`) while the tool is still working, then submit the real result when it completes. A keepalive resets the deadline without delivering a result, so send it periodically (well under the timeout) while the tool is pending. SDK helpers: `submitChatToolKeepalive` (TypeScript/Kotlin), `submit_chat_tool_keepalive` (Rust/Python), `SubmitChatToolKeepalive` (Go). *** ## Goals, Tasks & Agents[​](#goals-tasks--agents "Direct link to Goals, Tasks & Agents") Manage AI planning workflows with goals (multi-step plans), tasks (scheduled/triggered jobs), and agents (named AI profiles). ### Goals[​](#goals "Direct link to Goals") Goals are multi-step plans the AI can create, execute, and track. Each goal has steps that move through a lifecycle: `active` → `pending_review` → approved/rejected. * 🔷 Go * 📘 TypeScript * 🦀 Rust * 🐍 Python * 🟣 Kotlin ``` // Create a goal goal, _ := client.GoalCreate(map[string]interface{}{ "title": "Migrate user data", "description": "Move users from legacy to new schema", "status": "active", }) // List and search goals goals, _ := client.GoalList() results, _ := client.GoalSearch("migrate") // Complete (moves to pending_review) client.GoalComplete(goalID, map[string]interface{}{"summary": "All records migrated"}) // Approve or reject client.GoalApprove(goalID) client.GoalReject(goalID, map[string]interface{}{"reason": "Missing validation"}) // Step lifecycle client.GoalStepStart(goalID, 0) client.GoalStepComplete(goalID, 0, map[string]interface{}{"result": "done"}) client.GoalStepFail(goalID, 1, map[string]interface{}{"error": "timeout"}) ``` ``` // Create a goal const goal = await client.goalCreate({ title: "Migrate user data", description: "Move users from legacy to new schema", status: "active", }); // List and search const goals = await client.goalList(); const results = await client.goalSearch("migrate"); // Lifecycle await client.goalComplete(goalId, { summary: "All records migrated" }); await client.goalApprove(goalId); await client.goalReject(goalId, { reason: "Missing validation" }); // Steps await client.goalStepStart(goalId, 0); await client.goalStepComplete(goalId, 0, { result: "done" }); ``` ``` // Create a goal let goal = client.goal_create(serde_json::json!({ "title": "Migrate user data", "description": "Move users from legacy to new schema", "status": "active", })).await?; // List and search let goals = client.goal_list().await?; let results = client.goal_search("migrate").await?; // Lifecycle client.goal_complete(&goal_id, serde_json::json!({"summary": "Done"})).await?; client.goal_approve(&goal_id).await?; // Steps client.goal_step_start(&goal_id, 0).await?; client.goal_step_complete(&goal_id, 0, serde_json::json!({"result": "done"})).await?; ``` ``` # Create a goal goal = await client.goal_create({ "title": "Migrate user data", "description": "Move users from legacy to new schema", "status": "active", }) # List and search goals = await client.goal_list() results = await client.goal_search("migrate") # Lifecycle await client.goal_complete(goal_id, {"summary": "Done"}) await client.goal_approve(goal_id) # Steps await client.goal_step_start(goal_id, 0) await client.goal_step_complete(goal_id, 0, {"result": "done"}) ``` ``` // Create a goal val goal = client.goalCreate(buildJsonObject { put("title", "Migrate user data") put("description", "Move users from legacy to new schema") put("status", "active") }) // List and search val goals = client.goalList() val results = client.goalSearch("migrate") // Lifecycle client.goalComplete(goalId, buildJsonObject { put("summary", "Done") }) client.goalApprove(goalId) // Steps client.goalStepStart(goalId, 0) client.goalStepComplete(goalId, 0, buildJsonObject { put("result", "done") }) ``` ### Tasks[​](#tasks "Direct link to Tasks") Tasks represent scheduled or triggered jobs with lifecycle management. | Method | Description | | ---------------------------------------- | -------------------------------------- | | `taskCreate` | Create a new task | | `taskList` / `taskGet` | List or get task details | | `taskDue(now)` | Get tasks due at a given time | | `taskStart` / `taskPause` / `taskResume` | Lifecycle transitions | | `taskSucceed` / `taskFail` | Terminal states with result/error data | | `taskDelete` | Remove a task | ### Agents[​](#agents "Direct link to Agents") Agents are named AI profiles with specific models and configurations. | Method | Description | | ----------------------------- | ------------------------------------ | | `agentCreate` | Create a new agent profile | | `agentList` / `agentGet` | List or get agent details | | `agentGetByName` | Look up by name | | `agentUpdate` / `agentDelete` | Modify or remove | | `agentsByDeployment` | List agents on a specific deployment | *** ## Schedule Management[​](#schedule-management "Direct link to Schedule Management") Manage cron-based scheduled execution of functions and tasks. * 🔷 Go * 📘 TypeScript * 🦀 Rust * 🐍 Python * 🟣 Kotlin ``` // Create a schedule sched, _ := client.CreateSchedule(map[string]interface{}{ "name": "nightly-backup", "cron": "0 2 * * *", "task_type": "backup", }) schedID := sched["id"].(string) // List, get, update schedules, _ := client.ListSchedules() client.UpdateSchedule(schedID, map[string]interface{}{"cron": "0 3 * * *"}) // Pause and resume client.PauseSchedule(schedID) client.ResumeSchedule(schedID) // Delete client.DeleteSchedule(schedID) ``` ``` const sched = await client.createSchedule({ name: "nightly-backup", cron: "0 2 * * *", taskType: "backup", }); const schedId = sched.id; const schedules = await client.listSchedules(); await client.pauseSchedule(schedId); await client.resumeSchedule(schedId); await client.deleteSchedule(schedId); ``` ``` let sched = client.create_schedule(serde_json::json!({ "name": "nightly-backup", "cron": "0 2 * * *", "task_type": "backup", })).await?; let sched_id = sched["id"].as_str().unwrap(); let schedules = client.list_schedules().await?; client.pause_schedule(&sched_id).await?; client.resume_schedule(&sched_id).await?; client.delete_schedule(&sched_id).await?; ``` ``` sched = await client.create_schedule({ "name": "nightly-backup", "cron": "0 2 * * *", "task_type": "backup", }) sched_id = sched["id"] schedules = await client.list_schedules() await client.pause_schedule(sched_id) await client.resume_schedule(sched_id) await client.delete_schedule(sched_id) ``` ``` val sched = client.createSchedule(buildJsonObject { put("name", "nightly-backup") put("cron", "0 2 * * *") put("taskType", "backup") }) val schedId = sched.getString("id") val schedules = client.listSchedules() client.pauseSchedule(schedId) client.resumeSchedule(schedId) client.deleteSchedule(schedId) ``` *** ## Next Steps[​](#next-steps "Direct link to Next Steps") * [GitHub Examples](https://github.com/ekoDB/ekodb-client/tree/main/examples) - 129 (client library & direct HTTP examples) * [API Reference](/basic-operations.md) - Direct HTTP API documentation ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: --- # Basic Operations This guide covers the essential CRUD (Create, Read, Update, Delete) operations using ekoDB client libraries. Complete Examples Available For complete, runnable examples, visit the ekoDB Examples Repository. It contains **93 examples** (56 client library + 37 direct API examples). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, make sure you have: 1. ✅ [Installed the client library](/client-libraries/installation.md) 2. ✅ [Retrieved your API key](/authentication.md) ## Client Initialization[​](#client-initialization "Direct link to Client Initialization") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://your-subdomain.production.google.ekodb.net") .api_key("your-api-key") .build()?; ``` ``` from ekodb_client import Client client = Client.new( "https://your-subdomain.production.google.ekodb.net", "your-api-key" ) ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "your-api-key" }); await client.init(); ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: "https://your-subdomain.production.google.ekodb.net", apiKey: "your-api-key" }); await client.init(); ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.Builder() .baseUrl("https://your-subdomain.production.google.ekodb.net") .apiKey("your-api-key") .build() ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://your-subdomain.production.google.ekodb.net", "your-api-key", ) ``` ## Insert Records[​](#insert-records "Direct link to Insert Records") Create new records in a collection: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Record; let mut user = Record::new(); user.insert("name", "Alice Smith"); user.insert("email", "alice@example.com"); user.insert("age", 28); let result = client.insert("users", user, None).await?; println!("Created user with ID: {:?}", result.get("id")); ``` ``` user = { "name": "Alice Smith", "email": "alice@example.com", "age": 28 } result = await client.insert("users", user) print(f"Created user with ID: {result['id']}") ``` ``` const user = { name: "Alice Smith", email: "alice@example.com", age: 28 }; const result = await client.insert("users", user); console.log(`Created user with ID: ${result.id}`); ``` ``` const user = { name: "Alice Smith", email: "alice@example.com", age: 28 }; const result = await client.insert("users", user); console.log(`Created user with ID: ${result.id}`); ``` ``` import io.ekodb.client.types.Record val user = Record.new() .insert("name", "Alice Smith") .insert("email", "alice@example.com") .insert("age", 28) val result = client.insert("users", user) println("Created user with ID: ${result["id"]}") ``` ``` user := map[string]interface{}{ "name": "Alice Smith", "email": "alice@example.com", "age": 28, } result, err := client.Insert("users", user) if err != nil { log.Fatal(err) } fmt.Printf("Created user with ID: %s\n", result["id"]) ``` ## Find Records[​](#find-records "Direct link to Find Records") ### Find by ID[​](#find-by-id "Direct link to Find by ID") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let user = client.find_by_id("users", "user-id-123").await?; println!("Found user: {:?}", user); ``` ``` user = await client.find_by_id("users", "user-id-123") print(f"Found user: {user}") ``` ``` const user = await client.findById("users", "user-id-123"); console.log("Found user:", user); ``` ``` const user = await client.findById("users", "user-id-123"); console.log("Found user:", user); ``` ``` val user = client.findById("users", "user-id-123") println("Found user: $user") ``` ``` user, err := client.FindByID("users", "user-id-123") if err != nil { log.Fatal(err) } fmt.Printf("Found user: %+v\n", user) ``` ### Find All Records[​](#find-all-records "Direct link to Find All Records") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let users = client.find_all("users", 100).await?; println!("Found {} users", users.len()); ``` ``` users = await client.find("users", limit=100) print(f"Found {len(users)} users") ``` ``` const users = await client.find("users", { limit: 100 }); console.log(`Found ${users.length} users`); ``` ``` const users = await client.find("users", { limit: 100 }); console.log(`Found ${users.length} users`); ``` ``` val users = client.find("users", limit = 100) println("Found ${users.size} users") ``` ``` users, err := client.Find("users", map[string]interface{}{"limit": 100}) if err != nil { log.Fatal(err) } fmt.Printf("Found %d users\n", len(users)) ``` ## Update Records[​](#update-records "Direct link to Update Records") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let mut updates = Record::new(); updates.insert("age", 29); updates.insert("email", "alice.smith@example.com"); let updated = client.update("users", "user-id-123", updates).await?; println!("Updated user: {:?}", updated); ``` ``` updates = { "age": 29, "email": "alice.smith@example.com" } updated = await client.update("users", "user-id-123", updates) print(f"Updated user: {updated}") ``` ``` const updates = { age: 29, email: "alice.smith@example.com" }; const updated = await client.update("users", "user-id-123", updates); console.log("Updated user:", updated); ``` ``` const updates = { age: 29, email: "alice.smith@example.com" }; const updated = await client.update("users", "user-id-123", updates); console.log("Updated user:", updated); ``` ``` val updates = Record.new() .insert("age", 29) .insert("email", "alice.smith@example.com") val updated = client.update("users", "user-id-123", updates) println("Updated user: $updated") ``` ``` updates := map[string]interface{}{ "age": 29, "email": "alice.smith@example.com", } updated, err := client.Update("users", "user-id-123", updates) if err != nil { log.Fatal(err) } fmt.Printf("Updated user: %+v\n", updated) ``` ## Delete Records[​](#delete-records "Direct link to Delete Records") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` client.delete("users", "user-id-123").await?; println!("User deleted successfully"); ``` ``` await client.delete("users", "user-id-123") print("User deleted successfully") ``` ``` await client.delete("users", "user-id-123"); console.log("User deleted successfully"); ``` ``` await client.delete("users", "user-id-123"); console.log("User deleted successfully"); ``` ``` client.delete("users", "user-id-123") println("User deleted successfully") ``` ``` err := client.Delete("users", "user-id-123") if err != nil { log.Fatal(err) } fmt.Println("User deleted successfully") ``` ## Convenience Methods[​](#convenience-methods "Direct link to Convenience Methods") New in v0.8.0 These ergonomic helper methods simplify common patterns and reduce boilerplate code. ### Upsert (Insert or Update)[​](#upsert-insert-or-update "Direct link to Upsert (Insert or Update)") Atomic insert-or-update operation. Attempts to update the record first; if it doesn't exist, it inserts it instead. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Record; let mut user = Record::new(); user.insert("name", "Alice Smith"); user.insert("email", "alice@example.com"); user.insert("age", 29); // Will update if exists, insert if not let result = client.upsert("users", "user-123", user, None).await?; println!("Upserted user: {:?}", result); ``` ``` user = { "name": "Alice Smith", "email": "alice@example.com", "age": 29 } # Will update if exists, insert if not result = await client.upsert("users", "user-123", user) print(f"Upserted user: {result}") ``` ``` const user = { name: "Alice Smith", email: "alice@example.com", age: 29 }; // Will update if exists, insert if not const result = await client.upsert("users", "user-123", user); console.log("Upserted user:", result); ``` ``` const user = { name: "Alice Smith", email: "alice@example.com", age: 29 }; // Will update if exists, insert if not const result = await client.upsert("users", "user-123", user); console.log("Upserted user:", result); ``` ``` import io.ekodb.client.types.Record val user = Record.new() .insert("name", "Alice Smith") .insert("email", "alice@example.com") .insert("age", 29) // Will update if exists, insert if not val result = client.upsert("users", "user-123", user) println("Upserted user: $result") ``` ``` user := map[string]interface{}{ "name": "Alice Smith", "email": "alice@example.com", "age": 29, } // Will update if exists, insert if not result, err := client.Upsert("users", "user-123", user) if err != nil { log.Fatal(err) } fmt.Printf("Upserted user: %+v\n", result) ``` ### Find One Record by Field[​](#find-one-record-by-field "Direct link to Find One Record by Field") Find a single record matching a specific field value. Returns `null`/`nil`/`None` if no match found. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // Find user by email let user = client.find_one("users", "email", "alice@example.com").await?; match user { Some(record) => println!("Found user: {:?}", record), None => println!("User not found"), } ``` ``` # Find user by email user = await client.find_one("users", "email", "alice@example.com") if user: print(f"Found user: {user}") else: print("User not found") ``` ``` // Find user by email const user = await client.findOne("users", "email", "alice@example.com"); if (user) { console.log("Found user:", user); } else { console.log("User not found"); } ``` ``` // Find user by email const user = await client.findOne("users", "email", "alice@example.com"); if (user) { console.log("Found user:", user); } else { console.log("User not found"); } ``` ``` // Find user by email val user = client.findOne("users", "email", "alice@example.com") if (user != null) { println("Found user: $user") } else { println("User not found") } ``` ``` // Find user by email user, err := client.FindOne("users", "email", "alice@example.com") if err != nil { log.Fatal(err) } if user != nil { fmt.Printf("Found user: %+v\n", user) } else { fmt.Println("User not found") } ``` ### Check Record Existence[​](#check-record-existence "Direct link to Check Record Existence") Efficiently check if a record exists without fetching the full document. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` if client.exists("users", "user-123").await? { println!("User exists"); } else { println!("User not found"); } ``` ``` if await client.exists("users", "user-123"): print("User exists") else: print("User not found") ``` ``` if (await client.exists("users", "user-123")) { console.log("User exists"); } else { console.log("User not found"); } ``` ``` if (await client.exists("users", "user-123")) { console.log("User exists"); } else { console.log("User not found"); } ``` ``` if (client.exists("users", "user-123")) { println("User exists") } else { println("User not found") } ``` ``` exists, err := client.Exists("users", "user-123") if err != nil { log.Fatal(err) } if exists { fmt.Println("User exists") } else { fmt.Println("User not found") } ``` ### Paginate Results[​](#paginate-results "Direct link to Paginate Results") Simplified pagination using page numbers (1-indexed) instead of skip/limit calculations. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // Get page 2 with 10 records per page let page = 2; let page_size = 10; let users = client.paginate("users", page, page_size).await?; println!("Found {} users on page {}", users.len(), page); ``` ``` # Get page 2 with 10 records per page page = 2 page_size = 10 users = await client.paginate("users", page, page_size) print(f"Found {len(users)} users on page {page}") ``` ``` // Get page 2 with 10 records per page const page = 2; const pageSize = 10; const users = await client.paginate("users", page, pageSize); console.log(`Found ${users.length} users on page ${page}`); ``` ``` // Get page 2 with 10 records per page const page = 2; const pageSize = 10; const users = await client.paginate("users", page, pageSize); console.log(`Found ${users.length} users on page ${page}`); ``` ``` // Get page 2 with 10 records per page val page = 2 val pageSize = 10 val users = client.paginate("users", page, pageSize) println("Found ${users.size} users on page $page") ``` ``` // Get page 2 with 10 records per page page := 2 pageSize := 10 users, err := client.Paginate("users", page, pageSize) if err != nil { log.Fatal(err) } fmt.Printf("Found %d users on page %d\n", len(users), page) ``` Why Use Convenience Methods? * **Upsert**: Eliminates need for exists-check-then-insert patterns * **FindOne**: Cleaner than building a query with `limit(1)` * **Exists**: More efficient than fetching the full record * **Paginate**: No need to calculate skip/offset manually ## Collection Management[​](#collection-management "Direct link to Collection Management") ### List Collections[​](#list-collections "Direct link to List Collections") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let collections = client.list_collections().await?; println!("Collections: {:?}", collections); ``` ``` collections = await client.list_collections() print(f"Collections: {collections}") ``` ``` const collections = await client.listCollections(); console.log("Collections:", collections); ``` ``` const collections = await client.listCollections(); console.log("Collections:", collections); ``` ``` val collections = client.listCollections() println("Collections: $collections") ``` ``` collections, err := client.ListCollections() if err != nil { log.Fatal(err) } fmt.Printf("Collections: %v\n", collections) ``` ### Delete Collection[​](#delete-collection "Direct link to Delete Collection") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` client.delete_collection("old_data").await?; println!("Collection deleted"); ``` ``` await client.delete_collection("old_data") print("Collection deleted") ``` ``` await client.deleteCollection("old_data"); console.log("Collection deleted"); ``` ``` await client.deleteCollection("old_data"); console.log("Collection deleted"); ``` ``` client.deleteCollection("old_data") println("Collection deleted") ``` ``` err := client.DeleteCollection("old_data") if err != nil { log.Fatal(err) } fmt.Println("Collection deleted") ``` ### Check Collection Exists[​](#check-collection-exists "Direct link to Check Collection Exists") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let exists = client.collection_exists("users").await?; if exists { println!("Collection exists"); } ``` ``` exists = await client.collection_exists("users") if exists: print("Collection exists") ``` ``` const exists = await client.collectionExists("users"); if (exists) { console.log("Collection exists"); } ``` ``` const exists = await client.collectionExists("users"); if (exists) { console.log("Collection exists"); } ``` ``` val exists = client.collectionExists("users") if (exists) { println("Collection exists") } ``` ``` exists, err := client.CollectionExists("users") if err != nil { log.Fatal(err) } if exists { fmt.Println("Collection exists") } ``` ### Count Documents[​](#count-documents "Direct link to Count Documents") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let count = client.count_documents("users").await?; println!("Total documents: {}", count); ``` ``` count = await client.count_documents("users") print(f"Total documents: {count}") ``` ``` const count = await client.countDocuments("users"); console.log(`Total documents: ${count}`); ``` ``` const count = await client.countDocuments("users"); console.log(`Total documents: ${count}`); ``` ``` val count = client.count("users") println("Total documents: $count") ``` ``` count, err := client.CountDocuments("users") if err != nil { log.Fatal(err) } fmt.Printf("Total documents: %d\n", count) ``` ### Restore from Trash[​](#restore-from-trash "Direct link to Restore from Trash") Restore deleted records from trash. Records remain in trash for 30 days before permanent deletion. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // Restore single record let restored = client.restore_deleted("users", "record_id_123").await?; if restored { println!("Record restored successfully"); } // Restore all records in collection let count = client.restore_collection("users").await?; println!("Restored {} records", count); ``` ``` # Restore single record restored = await client.restore_deleted("users", "record_id_123") if restored: print("Record restored successfully") ``` ``` // Restore single record const restored = await client.restoreRecord("users", "record_id_123"); console.log("Record restored:", restored); // Restore all records in collection const result = await client.restoreCollection("users"); console.log(`Restored ${result.recordsRestored} records`); ``` ``` // Restore single record const restored = await client.restoreRecord("users", "record_id_123"); console.log("Record restored:", restored); // Restore all records in collection const result = await client.restoreCollection("users"); console.log(`Restored ${result.recordsRestored} records`); ``` ``` // Use REST API directly for restore operations val response = httpClient.post("$baseUrl/api/trash/users/record_id_123") { header("Authorization", "Bearer $token") } ``` ``` // Restore single record err := client.RestoreRecord("users", "record_id_123") if err != nil { log.Fatal(err) } // Restore all records in collection count, err := client.RestoreCollection("users") fmt.Printf("Restored %d records\n", count) ``` Trash Retention All client libraries support restore operations. Deleted records are kept in trash for 30 days before permanent deletion. ## Complete Examples[​](#complete-examples "Direct link to Complete Examples") For complete, runnable examples with error handling and advanced features, visit: ### 📚 Example Repository[​](#-example-repository "Direct link to 📚 Example Repository") **ekoDB Client Examples** The repository contains **93 examples total** (56 client library + 37 direct API) organized by language: * **Rust Examples** - 15 client examples + 10 direct API examples * **Python Examples** - 16 client examples + 10 direct API examples * **TypeScript Examples** - 16 client examples (Node.js & browsers) * **Go Examples** - 15 client examples + 10 direct API examples * **Kotlin Examples** - 15 client examples with coroutines * **JavaScript Examples** - 12 client examples + 10 direct API examples for Node.js ### Featured Examples[​](#featured-examples "Direct link to Featured Examples") **Core Operations:** * **Simple CRUD** - Basic create, read, update, delete operations * **Query Builder** - Complex queries with filters, sorting, pagination * **Batch Operations** - Efficient bulk inserts, updates, deletes * **Schema Management** - Define and enforce data schemas **New in v0.8.0:** * **Convenience Methods** - Upsert, findOne, exists, paginate examples (all languages) * **Bypass Ripple** - Control ripple propagation in multi-node deployments (all languages) **Advanced Features:** * **Search** - Full-text search with scoring * **WebSocket** - Real-time queries and subscriptions * **TTL** - Automatic document expiration * **Key-Value** - Simple key-value store operations * **Chat & RAG** - Conversational AI with retrieval-augmented generation * **Functions** - Server-side function execution ### Running Examples[​](#running-examples "Direct link to Running Examples") Visit the ekoDB Examples Repository for complete, runnable examples in all supported languages. Each example can be copied and adapted to your own project. * Rust Examples * Python Examples * TypeScript/JavaScript Examples * Kotlin Examples * Go Examples ## Next Steps[​](#next-steps "Direct link to Next Steps") * More Examples - Query builder, search, batch operations, and more * [API Reference](/basic-operations.md) - Direct HTTP API documentation * [Authentication](/authentication.md) - Learn about API keys and permissions ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: --- # Installation ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before installing a client library, make sure you have: 1. ✅ **Deployed an ekoDB instance** - See [Getting Started](/getting-started.md) 2. ✅ **Retrieved your API key** - See [Authentication](/authentication.md) 3. ✅ **Your database URL** - Format: `https://{SUBDOMAIN}.{ENVIRONMENT}.{PROVIDER}.ekodb.net` ## Installation by Language[​](#installation-by-language "Direct link to Installation by Language") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ### Rust Installation[​](#rust-installation "Direct link to Rust Installation") **Requirements:** * Rust 1.70 or later * Tokio runtime for async operations **Install via Cargo:** ``` cargo add ekodb_client ``` The `ekodb_client` package already includes all required dependencies (`serde`, `serde_json`, `tokio`, etc.). **Optional: Add tokio for macros in your code** If you want to use `#[tokio::main]` or other tokio macros in your own code: ``` cargo add tokio --features macros ``` **Verify Installation:** ``` use ekodb_client::Client; fn main() { println!("ekoDB Rust client installed successfully!"); } ``` ### Python Installation[​](#python-installation "Direct link to Python Installation") **Requirements:** * Python 3.9 or later * pip package manager **Install via pip:** ``` pip install ekodb-client ``` Package Name Flexibility You can also use `pip install ekodb_client` - PyPI normalizes package names, so both work identically. **Install in a virtual environment (recommended):** ``` # Create virtual environment python -m venv venv # Activate (Linux/macOS) source venv/bin/activate # Activate (Windows) venv\Scripts\activate # Install ekodb-client pip install ekodb-client ``` **Verify Installation:** ``` import ekodb_client print("ekoDB Python client installed successfully!") print(f"Version: {ekodb_client.__version__}") ``` **Supported Platforms:** * Linux (x86\_64, aarch64) * macOS (Intel, Apple Silicon) * Windows (via source distribution) * FreeBSD (x86\_64, aarch64) ### TypeScript/JavaScript Installation[​](#typescriptjavascript-installation "Direct link to TypeScript/JavaScript Installation") **Requirements:** * Node.js 16 or later * npm, yarn, or pnpm **Install via npm:** ``` npm install @ekodb/ekodb-client ``` **Install via yarn:** ``` yarn add @ekodb/ekodb-client ``` **Install via pnpm:** ``` pnpm add @ekodb/ekodb-client ``` **Verify Installation:** ``` import { EkoDBClient } from "@ekodb/ekodb-client"; console.log("ekoDB TypeScript client installed successfully!"); ``` **TypeScript Configuration:** The package includes TypeScript definitions. Make sure your `tsconfig.json` has: ``` { "compilerOptions": { "target": "ES2020", "module": "commonjs", "lib": ["ES2020"], "esModuleInterop": true } } ``` ### JavaScript Installation[​](#javascript-installation "Direct link to JavaScript Installation") **Requirements:** * Node.js 16 or later * npm, yarn, or pnpm **Install via npm:** ``` npm install @ekodb/ekodb-client ``` **Install via yarn:** ``` yarn add @ekodb/ekodb-client ``` **Install via pnpm:** ``` pnpm add @ekodb/ekodb-client ``` **Verify Installation:** ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); console.log("ekoDB JavaScript client installed successfully!"); ``` **ES Modules:** ``` import { EkoDBClient } from "@ekodb/ekodb-client"; console.log("ekoDB JavaScript client installed successfully!"); ``` ### Kotlin Installation[​](#kotlin-installation "Direct link to Kotlin Installation") **Requirements:** * JDK 17 or later * Gradle or Maven **Install via Gradle (Kotlin DSL):** ``` dependencies { implementation("io.ekodb:ekodb-client-kt:0.23.0") } ``` The `ekodb-client-kt` package already includes all required dependencies (`kotlinx-coroutines-core`, `kotlinx-serialization-json`, `ktor-client-*`, etc.). **Install via Gradle (Groovy):** ``` dependencies { implementation 'io.ekodb:ekodb-client-kt:0.23.0' } ``` **Install via Maven:** ``` io.ekodb ekodb-client-kt 0.23.0 ``` **Verify Installation:** ``` import io.ekodb.client.EkoDBClient fun main() { println("ekoDB Kotlin client installed successfully!") } ``` ### Go Installation[​](#go-installation "Direct link to Go Installation") **Requirements:** * Go 1.21 or later **Install via go get:** ``` go get github.com/ekoDB/ekodb-client-go ``` **Initialize your module (if needed):** ``` go mod init your-project-name go get github.com/ekoDB/ekodb-client-go ``` **Verify Installation:** ``` package main import ( "fmt" "github.com/ekoDB/ekodb-client-go" ) func main() { fmt.Println("ekoDB Go client installed successfully!") } ``` **Update to latest version:** ``` go get -u github.com/ekoDB/ekodb-client-go ``` ## Configuration[​](#configuration "Direct link to Configuration") After installation, configure your client with your database URL and API key: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Client; use std::env; #[tokio::main] async fn main() -> Result<(), Box> { let client = Client::builder() .base_url(env::var("EKODB_URL")?) .api_key(env::var("EKODB_API_KEY")?) .timeout(std::time::Duration::from_secs(30)) .max_retries(3) .build()?; // Client is ready to use Ok(()) } ``` ``` import os from ekodb_client import Client client = Client.new( os.getenv("EKODB_URL"), os.getenv("EKODB_API_KEY"), should_retry=True, max_retries=3, timeout_secs=30 ) # Client is ready to use ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: process.env.EKODB_URL!, apiKey: process.env.EKODB_API_KEY!, shouldRetry: true, maxRetries: 3, timeout: 30000 }); await client.init(); // Client is ready to use ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY, shouldRetry: true, maxRetries: 3, timeout: 30000 }); await client.init(); // Client is ready to use ``` ``` import io.ekodb.client.EkoDBClient suspend fun main() { val client = EkoDBClient.builder() .baseUrl(System.getenv("EKODB_URL")) .apiKey(System.getenv("EKODB_API_KEY")) .timeout(30000) .maxRetries(3) .build() // Client is ready to use } ``` ``` import ( "os" "github.com/ekoDB/ekodb-client-go" ) client := ekodb.NewClient( os.Getenv("EKODB_URL"), os.Getenv("EKODB_API_KEY"), ) // Client is ready to use ``` ## Environment Variables[​](#environment-variables "Direct link to Environment Variables") We recommend using environment variables for configuration: ``` # .env file EKODB_URL=https://your-subdomain.production.google.ekodb.net EKODB_API_KEY=your-api-key-here ``` ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ### Compilation errors[​](#compilation-errors "Direct link to Compilation errors") Make sure you have the latest version: ``` cargo update ekodb_client ``` ### Feature flag issues[​](#feature-flag-issues "Direct link to Feature flag issues") If you get TLS-related errors, check your feature flags: ``` # Use rustls (default) ekodb_client = "0.23.0" # Or use native-tls ekodb_client = { version = "0.23.0", default-features = false, features = ["native-tls"] } ``` ### "No module named 'ekodb\_client'"[​](#no-module-named-ekodb_client "Direct link to \"No module named 'ekodb_client'\"") Make sure the package is installed: ``` # Both of these work (PyPI normalizes names) pip install ekodb-client pip install ekodb_client ``` Then import with underscore: ``` import ekodb_client # ✅ Correct from ekodb_client import Client # ✅ Correct ``` **Common mistake:** ``` import ekodb-client # ❌ Wrong - hyphens are invalid in Python imports ``` ### Platform-specific issues[​](#platform-specific-issues "Direct link to Platform-specific issues") If you encounter build errors, you may need to install from source: ``` pip install ekodb-client --no-binary ekodb-client ``` ### "Cannot find module '@ekodb/ekodb-client'"[​](#cannot-find-module-ekodbekodb-client "Direct link to \"Cannot find module '@ekodb/ekodb-client'\"") Make sure the package is installed: ``` npm list @ekodb/ekodb-client ``` If not installed, run: ``` npm install @ekodb/ekodb-client ``` ### TypeScript errors[​](#typescript-errors "Direct link to TypeScript errors") Ensure your `tsconfig.json` has proper settings: ``` { "compilerOptions": { "esModuleInterop": true, "moduleResolution": "node" } } ``` ### "Cannot find module '@ekodb/ekodb-client'"[​](#cannot-find-module-ekodbekodb-client-1 "Direct link to \"Cannot find module '@ekodb/ekodb-client'\"") Make sure the package is installed: ``` npm list @ekodb/ekodb-client ``` If not installed, run: ``` npm install @ekodb/ekodb-client ``` ### ES Module vs CommonJS[​](#es-module-vs-commonjs "Direct link to ES Module vs CommonJS") The package supports both. For ES modules: ``` import { EkoDBClient } from "@ekodb/ekodb-client"; ``` For CommonJS: ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); ``` ### Node.js version[​](#nodejs-version "Direct link to Node.js version") Ensure you're using Node.js 16 or later: ``` node --version ``` ### "Could not find io.ekodb:ekodb-client-kt"[​](#could-not-find-ioekodb "Direct link to could-not-find-ioekodb") Make sure Maven Central is in your repositories: ``` repositories { mavenCentral() } ``` If the package was just published, wait a few minutes for Maven Central to sync. ### Gradle sync issues[​](#gradle-sync-issues "Direct link to Gradle sync issues") Try refreshing Gradle dependencies: ``` ./gradlew clean build --refresh-dependencies ``` ### Compilation errors with coroutines[​](#compilation-errors-with-coroutines "Direct link to Compilation errors with coroutines") The `ekodb-client-kt` package already includes `kotlinx-coroutines-core` (1.9.0). If you're still getting coroutine-related errors, try: ``` ./gradlew clean build --refresh-dependencies ``` ### JDK version mismatch[​](#jdk-version-mismatch "Direct link to JDK version mismatch") Ensure you're using JDK 17 or later: ``` kotlin { jvmToolchain(17) } ``` ### "package github.com/ekoDB/ekodb-client-go is not in GOROOT"[​](#package-githubcomekodbekodb-client-go-is-not-in-goroot "Direct link to \"package github.com/ekoDB/ekodb-client-go is not in GOROOT\"") Run go mod tidy: ``` go mod tidy ``` ### Module not found[​](#module-not-found "Direct link to Module not found") Make sure you're using Go modules: ``` go mod init your-project go get github.com/ekoDB/ekodb-client-go ``` ### Version conflicts[​](#version-conflicts "Direct link to Version conflicts") If you have version conflicts, update to the latest: ``` go get -u github.com/ekoDB/ekodb-client-go ``` ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: --- # Querying with Filters All ekoDB client libraries provide a **QueryBuilder** for constructing complex queries with filters, sorting, and pagination using a fluent API. ## Simple Query[​](#simple-query "Direct link to Simple Query") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::{Client, QueryBuilder}; #[tokio::main] async fn main() -> Result<(), Box> { let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("YOUR_API_KEY") .build()?; // Find active users let query = QueryBuilder::new() .eq("status", "active") .build(); let result = client.find("users", query, None).await?; println!("{:?}", result); Ok(()) } ``` ``` from ekodb_client import Client, QueryBuilder client = Client.new( "https://my-first-db.development.google.ekodb.net", "YOUR_API_KEY" ) # Find active users query = QueryBuilder() \ .eq("status", "active") \ .build() result = await client.find("users", query) print(result) ``` ``` import { EkoDBClient, QueryBuilder } from '@ekodb/ekodb-client'; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "YOUR_API_KEY" }); await client.init(); // Find active users const query = new QueryBuilder() .eq("status", "active") .build(); const result = await client.find("users", query); console.log(result); ``` ``` const { EkoDBClient, QueryBuilder } = require('@ekodb/ekodb-client'); const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "YOUR_API_KEY" }); await client.init(); // Find active users const query = new QueryBuilder() .eq("status", "active") .build(); const result = await client.find("users", query); console.log(result); ``` ``` import io.ekodb.client.EkoDBClient import io.ekodb.client.QueryBuilder suspend fun main() { val client = EkoDBClient.builder() .baseUrl("https://my-first-db.development.google.ekodb.net") .apiKey("YOUR_API_KEY") .build() // Find active users val query = QueryBuilder() .eq("status", "active") .build() val result = client.find("users", query) println(result) } ``` ``` package main import ( "fmt" "github.com/ekoDB/ekodb-client-go" ) func main() { client := ekodb.NewClient( "https://my-first-db.development.google.ekodb.net", "YOUR_API_KEY", ) // Find active users query := ekodb.NewQueryBuilder(). Eq("status", "active"). Build() result, err := client.Find("users", query) if err != nil { panic(err) } fmt.Println(result) } ``` *** ## Complex Query[​](#complex-query "Direct link to Complex Query") Find active users aged 18-65 with gmail addresses, sorted by creation date: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let query = QueryBuilder::new() .eq("status", "active") .gte("age", 18) .lte("age", 65) .contains("email", "@gmail.com") .sort_desc("created_at") .limit(20) .build(); let result = client.find("users", query, None).await?; ``` ``` query = QueryBuilder() \ .eq("status", "active") \ .gte("age", 18) \ .lte("age", 65) \ .contains("email", "@gmail.com") \ .sort_desc("created_at") \ .limit(20) \ .build() result = await client.find("users", query) ``` ``` const query = new QueryBuilder() .eq("status", "active") .gte("age", 18) .lte("age", 65) .contains("email", "@gmail.com") .sortDesc("created_at") .limit(20) .build(); const result = await client.find("users", query); ``` ``` const query = new QueryBuilder() .eq("status", "active") .gte("age", 18) .lte("age", 65) .contains("email", "@gmail.com") .sortDesc("created_at") .limit(20) .build(); const result = await client.find("users", query); ``` ``` val query = QueryBuilder() .eq("status", "active") .gte("age", 18) .lte("age", 65) .contains("email", "@gmail.com") .sortDesc("created_at") .limit(20) .build() val result = client.find("users", query) ``` ``` query := ekodb.NewQueryBuilder(). Eq("status", "active"). Gte("age", 18). Lte("age", 65). Contains("email", "@gmail.com"). SortDescending("created_at"). Limit(20). Build() result, err := client.Find("users", query) ``` *** ## Logical Operators[​](#logical-operators "Direct link to Logical Operators") ### OR Query[​](#or-query "Direct link to OR Query") Find users who are either admins or moderators: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use serde_json::json; let query = QueryBuilder::new() .or(vec![ json!({ "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "admin" } }), json!({ "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "moderator" } }), ]) .build(); ``` ``` query = QueryBuilder() \ .or_([ { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "admin" } }, { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "moderator" } } ]) \ .build() ``` ``` const query = new QueryBuilder() .or([ { type: "Condition", content: { field: "role", operator: "Eq", value: "admin" } }, { type: "Condition", content: { field: "role", operator: "Eq", value: "moderator" } } ]) .build(); ``` ``` const query = new QueryBuilder() .or([ { type: "Condition", content: { field: "role", operator: "Eq", value: "admin" } }, { type: "Condition", content: { field: "role", operator: "Eq", value: "moderator" } } ]) .build(); ``` ``` val query = QueryBuilder() .or(listOf( mapOf( "type" to "Condition", "content" to mapOf( "field" to "role", "operator" to "Eq", "value" to "admin" ) ), mapOf( "type" to "Condition", "content" to mapOf( "field" to "role", "operator" to "Eq", "value" to "moderator" ) ) )) .build() ``` ``` query := ekodb.NewQueryBuilder(). Or([]map[string]interface{}{ { "type": "Condition", "content": map[string]interface{}{ "field": "role", "operator": "Eq", "value": "admin", }, }, { "type": "Condition", "content": map[string]interface{}{ "field": "role", "operator": "Eq", "value": "moderator", }, }, }). Build() ``` *** ## String Operators[​](#string-operators "Direct link to String Operators") ### StartsWith[​](#startswith "Direct link to StartsWith") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let query = QueryBuilder::new() .starts_with("name", "Dr. ") .build(); ``` ``` query = QueryBuilder() \ .starts_with("name", "Dr. ") \ .build() ``` ``` const query = new QueryBuilder() .startsWith("name", "Dr. ") .build(); ``` ``` const query = new QueryBuilder() .startsWith("name", "Dr. ") .build(); ``` ``` val query = QueryBuilder() .startsWith("name", "Dr. ") .build() ``` ``` query := ekodb.NewQueryBuilder(). StartsWith("name", "Dr. "). Build() ``` ### EndsWith[​](#endswith "Direct link to EndsWith") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let query = QueryBuilder::new() .ends_with("email", ".edu") .build(); ``` ``` query = QueryBuilder() \ .ends_with("email", ".edu") \ .build() ``` ``` const query = new QueryBuilder() .endsWith("email", ".edu") .build(); ``` ``` const query = new QueryBuilder() .endsWith("email", ".edu") .build(); ``` ``` val query = QueryBuilder() .endsWith("email", ".edu") .build() ``` ``` query := ekodb.NewQueryBuilder(). EndsWith("email", ".edu"). Build() ``` ### Contains[​](#contains "Direct link to Contains") * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` let query = QueryBuilder::new() .contains("description", "urgent") .build(); ``` ``` query = QueryBuilder() \ .contains("description", "urgent") \ .build() ``` ``` const query = new QueryBuilder() .contains("description", "urgent") .build(); ``` ``` const query = new QueryBuilder() .contains("description", "urgent") .build(); ``` ``` val query = QueryBuilder() .contains("description", "urgent") .build() ``` ``` query := ekodb.NewQueryBuilder(). Contains("description", "urgent"). Build() ``` *** ## Pagination[​](#pagination "Direct link to Pagination") Page through results with `skip` and `limit`: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` // Page 2 with 20 items per page let query = QueryBuilder::new() .eq("status", "active") .page(2, 20) // skip: 40, limit: 20 .sort_desc("created_at") .build(); ``` ``` # Page 2 with 20 items per page query = QueryBuilder() \ .eq("status", "active") \ .page(2, 20) \ # skip: 40, limit: 20 .sort_desc("created_at") \ .build() ``` ``` // Page 2 with 20 items per page const query = new QueryBuilder() .eq("status", "active") .page(2, 20) // skip: 40, limit: 20 .sortDesc("created_at") .build(); ``` ``` // Page 2 with 20 items per page const query = new QueryBuilder() .eq("status", "active") .page(2, 20) // skip: 40, limit: 20 .sortDesc("created_at") .build(); ``` ``` // Page 2 with 20 items per page val query = QueryBuilder() .eq("status", "active") .page(2, 20) // skip: 40, limit: 20 .sortDesc("created_at") .build() ``` ``` // Page 2 with 20 items per page query := ekodb.NewQueryBuilder(). Eq("status", "active"). Page(2, 20). // skip: 20, limit: 20 (Go's Page is 1-indexed; page 1 = skip 0) SortDescending("created_at"). Build() ``` *** ## Available Operators[​](#available-operators "Direct link to Available Operators") ### Comparison Operators[​](#comparison-operators "Direct link to Comparison Operators") | Method | Operator | Description | | -------------------- | -------- | --------------------- | | `.eq(field, value)` | `Eq` | Equals | | `.ne(field, value)` | `Ne` | Not equals | | `.gt(field, value)` | `Gt` | Greater than | | `.gte(field, value)` | `Gte` | Greater than or equal | | `.lt(field, value)` | `Lt` | Less than | | `.lte(field, value)` | `Lte` | Less than or equal | | `.in(field, array)` | `In` | Value in array | | `.nin(field, array)` | `NotIn` | Value not in array | ### String Operators[​](#string-operators-1 "Direct link to String Operators") | Method | Operator | Description | | ----------------------------- | ------------ | --------------- | | `.contains(field, substring)` | `Contains` | Substring match | | `.startsWith(field, prefix)` | `StartsWith` | Prefix match | | `.endsWith(field, suffix)` | `EndsWith` | Suffix match | ### Logical Operators[​](#logical-operators-1 "Direct link to Logical Operators") | Method | Description | | ------------------ | --------------------------------- | | `.and(conditions)` | All conditions must match | | `.or(conditions)` | At least one condition must match | | `.not(condition)` | Negate a condition | ### Sorting & Pagination[​](#sorting--pagination "Direct link to Sorting & Pagination") | Method | Description | | ------------------- | --------------------------------- | | `.sortAsc(field)` | Sort ascending | | `.sortDesc(field)` | Sort descending | | `.limit(n)` | Max results to return | | `.skip(n)` | Results to skip | | `.page(page, size)` | Convenience method for pagination | *** ## Query Expression Reference[​](#query-expression-reference "Direct link to Query Expression Reference") For the complete reference including all operators, JSON structure, and advanced patterns, see: **[Query Expressions Reference](/query-expressions.md)** Complete QueryBuilder Examples **QueryBuilder examples** - Complex queries with filtering, sorting, and pagination: * **Rust**: `client_query_builder.rs` * **Python**: `client_query_builder.py` * **TypeScript**: `client_query_builder.ts` * **JavaScript**: `client_query_builder.js` * **Go**: `client_query_builder.go` * **Kotlin**: `ClientQueryBuilder.kt` *** ## Next Steps[​](#next-steps "Direct link to Next Steps") * [Query Expressions](/query-expressions.md) - Complete operator reference * [Advanced Operations](/client-libraries/advanced-operations.md) - Joins, transactions, and batch operations * Examples Repository - Client library & direct HTTP examples ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: --- # Collections & Schemas Collections are containers for your data, similar to tables in traditional databases. Each collection can have an optional schema that enforces data structure, types, and validation rules. Collection vs Schema * **Collection** - The data container itself * **Schema** - The validation rules and structure definition for that collection ## Create Collection[​](#create-collection "Direct link to Create Collection") Create a new collection with optional schema validation. Admin Permission Required Creating collections requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use std::collections::HashMap; client.create_collection( "users", Some(Schema { fields: [ ("name".to_string(), FieldSchema { field_type: "String", required: true, ..Default::default() }), ("email".to_string(), FieldSchema { field_type: "String", required: true, ..Default::default() }), ("age".to_string(), FieldSchema { field_type: "Number", required: false, ..Default::default() }), ].into_iter().collect(), }) ).await?; ``` ``` client.create_collection('users', { 'fields': { 'name': {'field_type': 'String', 'required': True}, 'email': {'field_type': 'String', 'required': True}, 'age': {'field_type': 'Number'} } }) ``` ``` await client.createCollection("users", { fields: { name: { field_type: "String", required: true }, email: { field_type: "String", required: true }, age: { field_type: "Number" }, }, }); ``` ``` await client.createCollection("users", { fields: { name: { field_type: "String", required: true }, email: { field_type: "String", required: true }, age: { field_type: "Number" }, }, }); ``` ``` client.createCollection("users", mapOf( "fields" to mapOf( "name" to mapOf("type" to "String", "required" to true), "email" to mapOf("type" to "String", "required" to true), "age" to mapOf("type" to "Number") ) )) ``` ``` client.CreateCollection("users", map[string]interface{}{ "fields": map[string]interface{}{ "name": map[string]interface{}{"field_type": "String", "required": true}, "email": map[string]interface{}{"field_type": "String", "required": true}, "age": map[string]interface{}{"field_type": "Number"}, }, }) ``` ``` curl -X POST https://{EKODB_API_URL}/api/collections/users \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "fields": { "name": { "field_type": "String", "required": true }, "email": { "field_type": "String", "required": true }, "age": { "field_type": "Number" } } }' # Response { "status": "success", "message": "Collection created successfully" } ``` ## Get Collection Info[​](#get-collection-info "Direct link to Get Collection Info") Retrieve metadata about a collection. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let info = client.get_collection_info("users").await?; println!("Records: {}", info.record_count); println!("Schema: {:?}", info.schema); ``` ``` info = await client.get_collection_info('users') print(info['record_count']) # 1234 print(info['schema']) ``` ``` const info = await client.getCollectionInfo("users"); console.log(info.record_count); // 1234 console.log(info.schema); ``` ``` const info = await client.getCollectionInfo("users"); console.log(info.record_count); // 1234 console.log(info.schema); ``` ``` val info = client.getCollectionInfo("users") println("Records: ${info.recordCount}") println("Schema: ${info.schema}") ``` ``` info, err := client.GetCollectionInfo("users") fmt.Println("Records:", info.RecordCount) fmt.Println("Schema:", info.Schema) ``` ``` curl https://{EKODB_API_URL}/api/collections/users \ -H "Authorization: Bearer {TOKEN}" # Response { "name": "users", "record_count": 1234, "created_at": "2026-01-10T08:00:00Z", "schema": { "fields": { "name": {"field_type": "String", "required": true}, "email": {"field_type": "String", "required": true} } } } ``` ## List All Collections[​](#list-all-collections "Direct link to List All Collections") Get a list of all collections in the database. Admin Permission Required Listing all collections requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let collections = client.list_collections().await?; println!("{:?}", collections); ``` ``` collections = client.list_collections() print(collections) ``` ``` const collections = await client.listCollections(); console.log(collections); ``` ``` const collections = await client.listCollections(); console.log(collections); ``` ``` val collections = client.listCollections() println(collections) ``` ``` collections, err := client.ListCollections() fmt.Println(collections) ``` ``` curl https://{EKODB_API_URL}/api/collections \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "collections": [ "users", "products", "orders" ] } ``` ## Delete Collection[​](#delete-collection "Direct link to Delete Collection") Delete an entire collection and all its records permanently. Warning This operation requires Admin access and is **irreversible**. Use with extreme caution. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.delete_collection("users").await?; ``` ``` await client.delete_collection('users') ``` ``` await client.deleteCollection("users"); ``` ``` await client.deleteCollection("users"); ``` ``` client.deleteCollection("users") ``` ``` err := client.DeleteCollection("users") ``` ``` curl -X DELETE https://{EKODB_API_URL}/api/collections/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "success", "message": "Collection deleted successfully" } ``` ## Restore Record from Trash[​](#restore-record-from-trash "Direct link to Restore Record from Trash") Restore a previously deleted record from trash. Admin Permission Required Restoring records from trash requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.restore_from_trash("users", &record_id).await?; ``` ``` client.restore_from_trash('users', record_id) ``` ``` await client.restoreFromTrash("users", recordId); ``` ``` await client.restoreFromTrash("users", recordId); ``` ``` client.restoreFromTrash("users", recordId) ``` ``` err := client.RestoreFromTrash("users", recordId) ``` ``` curl -X POST https://{EKODB_API_URL}/api/trash/{collection}/{record_id} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "restored", "id": "record_id_123", "collection": "users" } ``` ## Restore Collection from Trash[​](#restore-collection-from-trash "Direct link to Restore Collection from Trash") Restore all deleted records in a collection from trash. Admin Permission Required Restoring collections from trash requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let result = client.restore_collection_from_trash("users").await?; println!("Restored: {} records", result.records_restored); ``` ``` result = client.restore_collection_from_trash('users') print(result['records_restored']) ``` ``` const result = await client.restoreCollectionFromTrash("users"); console.log(result.records_restored); ``` ``` const result = await client.restoreCollectionFromTrash("users"); console.log(result.records_restored); ``` ``` val result = client.restoreCollectionFromTrash("users") println("Restored: ${result.recordsRestored} records") ``` ``` result, err := client.RestoreCollectionFromTrash("users") fmt.Printf("Restored: %d records\n", result.RecordsRestored) ``` ``` curl -X POST https://{EKODB_API_URL}/api/trash/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "restored", "collection": "users", "records_restored": 42 } ``` Trash Retention Deleted records are kept in trash for 30 days. After this period, they are permanently deleted and cannot be restored. ## Get Schema[​](#get-schema "Direct link to Get Schema") Retrieve the current schema for a collection. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let schema = client.get_schema("users").await?; println!("{:?}", schema.fields); ``` ``` schema = await client.get_schema('users') print(schema['fields']) ``` ``` const schema = await client.getSchema("users"); console.log(schema.fields); ``` ``` const schema = await client.getSchema("users"); console.log(schema.fields); ``` ``` val schema = client.getSchema("users") println(schema.fields) ``` ``` schema, err := client.GetSchema("users") fmt.Println(schema.Fields) ``` ``` curl https://{EKODB_API_URL}/api/schemas/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "fields": { "name": { "field_type": "String", "required": true, "min": 2 }, "email": { "field_type": "String", "required": true }, "age": { "field_type": "Integer", "min": 0, "max": 150 } } } ``` ## Update Schema Constraints[​](#update-schema-constraints "Direct link to Update Schema Constraints") Update validation constraints for an existing schema. Admin Permission Required Updating schema constraints requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.update_schema("users", Schema { fields: [ ("email".to_string(), FieldSchema { field_type: "String", required: true, ..Default::default() }), ("age".to_string(), FieldSchema { field_type: "Integer", min: Some(18), max: Some(100), ..Default::default() }), ].into_iter().collect(), }).await?; ``` ``` await client.update_schema('users', { 'fields': { 'email': { 'field_type': 'String', 'required': True }, 'age': { 'field_type': 'Integer', 'min': 18, 'max': 100 } } }) ``` ``` await client.updateSchema("users", { fields: { email: { field_type: "String", required: true, }, age: { field_type: "Integer", min: 18, max: 100, }, }, }); ``` ``` await client.updateSchema("users", { fields: { email: { field_type: "String", required: true, }, age: { field_type: "Integer", min: 18, max: 100, }, }, }); ``` ``` client.updateSchema("users", mapOf( "fields" to mapOf( "email" to mapOf( "type" to "String", "required" to true ), "age" to mapOf( "type" to "Integer", "min" to 18, "max" to 100 ) ) )) ``` ``` client.UpdateSchema("users", map[string]interface{}{ "fields": map[string]interface{}{ "email": map[string]interface{}{ "field_type": "String", "required": true, }, "age": map[string]interface{}{ "field_type": "Integer", "min": 18, "max": 100, }, }, }) ``` ``` curl -X PUT https://{EKODB_API_URL}/api/schemas/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "fields": { "email": { "field_type": "String", "required": true }, "age": { "field_type": "Integer", "min": 18, "max": 100 } } }' # Response { "message": "Schema updated successfully" } ``` Schema Updates Schema updates only affect **new records**. Existing records are not automatically migrated or validated against the new schema. ## Set Primary Key Alias[​](#set-primary-key-alias "Direct link to Set Primary Key Alias") Configure a custom field to use as the primary key instead of the default `id`. Admin Permission Required Setting a primary key alias requires admin permissions. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.set_primary_key_alias("products", "sku").await?; ``` ``` await client.set_primary_key_alias('products', 'sku') ``` ``` await client.setPrimaryKeyAlias("products", "sku"); ``` ``` await client.setPrimaryKeyAlias("products", "sku"); ``` ``` client.setPrimaryKeyAlias("products", "sku") ``` ``` err := client.SetPrimaryKeyAlias("products", "sku") ``` ``` curl -X PUT https://{EKODB_API_URL}/api/schemas/{collection}/primary-key-alias \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "alias": "sku" }' # Response { "message": "Primary key alias set successfully", "alias": "sku" } ``` **Example - Using Custom Primary Key:** * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::Record; // Insert with custom primary key let mut user = Record::new(); user.insert("user_id", "custom_user_001"); user.insert("name", "John Doe"); user.insert("email", "john@example.com"); let result = client.insert("users", user, None).await?; // Retrieve using custom primary key let user = client.find_by_id("users", "custom_user_001").await?; println!("Found user: {:?}", user); ``` ``` # Insert with custom primary key user = { "user_id": "custom_user_001", "name": "John Doe", "email": "john@example.com" } result = await client.insert("users", user) # Retrieve using custom primary key user = await client.find_by_id("users", "custom_user_001") print(f"Found user: {user}") ``` ``` // Insert with custom primary key const user = { user_id: "custom_user_001", name: "John Doe", email: "john@example.com", }; const result = await client.insert("users", user); // Retrieve using custom primary key const found = await client.findById("users", "custom_user_001"); console.log("Found user:", found); ``` ``` // Insert with custom primary key const user = { user_id: "custom_user_001", name: "John Doe", email: "john@example.com", }; const result = await client.insert("users", user); // Retrieve using custom primary key const found = await client.findById("users", "custom_user_001"); console.log("Found user:", found); ``` ``` import io.ekodb.client.types.Record // Insert with custom primary key val user = Record.new() .insert("user_id", "custom_user_001") .insert("name", "John Doe") .insert("email", "john@example.com") val result = client.insert("users", user) // Retrieve using custom primary key val found = client.findById("users", "custom_user_001") println("Found user: $found") ``` ``` // Insert with custom primary key user := map[string]interface{}{ "user_id": "custom_user_001", "name": "John Doe", "email": "john@example.com", } result, err := client.Insert("users", user) if err != nil { log.Fatal(err) } // Retrieve using custom primary key found, err := client.FindByID("users", "custom_user_001") if err != nil { log.Fatal(err) } fmt.Printf("Found user: %+v\n", found) ``` ``` # Insert with custom primary key curl -X POST https://{EKODB_API_URL}/api/insert/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "user_id": "custom_user_001", "name": "John Doe", "email": "john@example.com" }' # Retrieve using custom primary key curl -X GET https://{EKODB_API_URL}/api/find/users/custom_user_001 \ -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` ## Schema Field Types[​](#schema-field-types "Direct link to Schema Field Types") The key is `field_type` Every field definition must carry a **`field_type`** key, and it is **required** — a field definition without it is rejected (`missing field 'field_type'`). The value is case-sensitive (`"String"`, `"Integer"`, `"Vector"`, …). Do not confuse this with the `type` key used in **typed value wrappers** on records (`{ "type": "String", "value": "Alice" }`) or in **index configs** (`{ "type": "vector", … }`). Those use `type`; a schema field definition uses `field_type`. ### Supported Data Types[​](#supported-data-types "Direct link to Supported Data Types") | Type | Description | Validation Options | | ---------- | ----------------------------- | --------------------------------------- | | `String` | Text values | `min`, `max` (length), `regex`, `enums` | | `Integer` | 64-bit signed integers | `min`, `max`, `enums` | | `Float` | 64-bit floating-point numbers | `min`, `max`, `enums` | | `Number` | Integers and floats | `min`, `max`, `enums` | | `Boolean` | True/false values | None | | `Array` | Lists of values | `min`, `max` (length) | | `Object` | Nested JSON objects | None | | `DateTime` | RFC 3339 datetime strings | `min`, `max` | | `Decimal` | Arbitrary-precision decimals | `min`, `max` | | `UUID` | RFC 4122 identifiers | None | | `Vector` | Numeric arrays for embeddings | `min`, `max` (length) | | `Set` | Unique value collections | `min`, `max` (length) | | `Binary` | Base64-encoded binary data | None | | `Bytes` | Raw byte arrays | None | | `Duration` | Time span values | None | ### Field Constraints[​](#field-constraints "Direct link to Field Constraints") **String Constraints:** ``` { "username": { "field_type": "String", "required": true, "min": 3, "max": 20, "regex": "^[a-zA-Z0-9_]+$" } } ``` **Number Constraints:** ``` { "age": { "field_type": "Number", "min": 0, "max": 150 }, "price": { "field_type": "Number", "min": 0.01, "max": 999999.99 } } ``` **Array Constraints:** ``` { "tags": { "field_type": "Array", "min": 1, "max": 10 } } ``` **Enum Validation:** ``` { "status": { "field_type": "String", "enums": ["draft", "published", "archived"], "default": "draft" } } ``` **Object Fields:** ``` { "address": { "field_type": "Object", "required": true } } ``` ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete workflow for creating and managing a collection: ``` #!/bin/bash # 1. Create collection with schema curl -X POST https://{EKODB_API_URL}/api/collections/products \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "fields": { "sku": { "field_type": "String", "required": true, "regex": "^[A-Z0-9-]+$" }, "name": { "field_type": "String", "required": true, "max": 200 }, "price": { "field_type": "Number", "min": 0.01, "required": true }, "category": { "field_type": "String", "enums": ["electronics", "clothing", "books", "food"] }, "in_stock": { "field_type": "Boolean", "default": true }, "tags": { "field_type": "Array", "max": 5 } } }' # 2. Set custom primary key curl -X PUT https://{EKODB_API_URL}/api/schemas/products/primary-key-alias \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "alias": "sku" }' # 3. Insert product (validates against schema) curl -X POST https://{EKODB_API_URL}/api/insert/products \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "sku": "ELEC-001", "name": "Wireless Mouse", "price": 29.99, "category": "electronics", "in_stock": true, "tags": ["wireless", "bluetooth", "ergonomic"] }' # 4. Get collection info curl -X GET https://{EKODB_API_URL}/api/collections/products \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # 5. Get current schema curl -X GET https://{EKODB_API_URL}/api/schemas/products \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # 6. Update schema constraints curl -X PUT https://{EKODB_API_URL}/api/schemas/products \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "fields": { "sku": { "field_type": "String", "required": true, "regex": "^[A-Z0-9-]+$" }, "name": { "field_type": "String", "required": true, "max": 300 }, "price": { "field_type": "Number", "min": 0.01, "max": 100000 } } }' ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Schema Design[​](#schema-design "Direct link to Schema Design") **Start Simple, Evolve Gradually:** ``` # Initial schema - minimal constraints { "fields": { "name": {"field_type": "String", "required": true}, "email": {"field_type": "String", "required": true} } } # Later - add more constraints as requirements clarify { "fields": { "name": { "field_type": "String", "required": true, "min": 2, "max": 100 }, "email": { "field_type": "String", "required": true } } } ``` ### Collection Naming[​](#collection-naming "Direct link to Collection Naming") **Use Clear, Plural Names:** ``` # Good /api/collections/users /api/collections/products /api/collections/order_items # Avoid /api/collections/user /api/collections/Product /api/collections/order-item ``` ### Schema Validation[​](#schema-validation "Direct link to Schema Validation") **Balance Strictness with Flexibility:** ``` { "fields": { "core_field": { "field_type": "String", "required": true, "max": 100 }, "optional_field": { "field_type": "String", "required": false } } } ``` ### Primary Key Strategy[​](#primary-key-strategy "Direct link to Primary Key Strategy") **Choose Based on Your Use Case:** | Use Case | Primary Key Strategy | | ------------------------- | ------------------------ | | Auto-generated IDs | Use default `id` field | | External system sync | Set alias to external ID | | Natural keys (SKU, email) | Set alias to natural key | | UUIDs | Set alias to UUID field | ## Validation Errors[​](#validation-errors "Direct link to Validation Errors") When a record violates the schema, you'll receive detailed error messages: ``` # Invalid insert attempt POST /api/insert/users { "name": "A", // Too short "email": "invalid-email", // Invalid format "age": 200 // Exceeds max } # Error response { "error": "Schema validation failed", "details": [ { "field": "name", "constraint": "min", "expected": 2, "actual": 1, "message": "Field 'name' must be at least 2 characters" }, { "field": "email", "constraint": "regex", "message": "Field 'email' does not match required pattern" }, { "field": "age", "constraint": "max", "expected": 150, "actual": 200, "message": "Field 'age' exceeds maximum value of 150" } ] } ``` ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - Insert, query, update, and delete records * [Indexes](/indexes.md) - Create indexes for query performance * [Authentication](/authentication.md) - API key and permission management * [Data Types](/data-types.md) - Detailed data type reference ## Example Code[​](#example-code "Direct link to Example Code") ### Direct HTTP/REST API Examples[​](#direct-httprest-api-examples "Direct link to Direct HTTP/REST API Examples") Raw HTTP examples demonstrating the REST API directly: * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/collection_management.js)**: `collection_management.js` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/collection_management.py)**: `collection_management.py` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/collection_management.go)**: `collection_management.go` * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/collection_management.rs)**: `collection_management.rs` ### Client Library Examples[​](#client-library-examples "Direct link to Client Library Examples") Production-ready examples using official client libraries: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_collection_management.rs)**: `client_collection_management.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_collection_management.py)**: `client_collection_management.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_collection_management.ts)**: `client_collection_management.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_collection_management.go)**: `client_collection_management.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientCollectionManagement.kt)**: `ClientCollectionManagement.kt` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_collection_management.js)**: `client_collection_management.js` --- # Configuration Configure your ekoDB instance at runtime using the REST API. All configuration changes take effect immediately without requiring a restart. Admin Access Required Configuration endpoints require admin permissions. ## API Endpoints[​](#api-endpoints "Direct link to API Endpoints") ### Get Configuration[​](#get-configuration "Direct link to Get Configuration") Retrieve the current configuration: ``` GET /api/config Authorization: Bearer {ADMIN_TOKEN} ``` ### Update Configuration[​](#update-configuration "Direct link to Update Configuration") Update one or more configuration options. Only the fields you include in the request body will be changed — all other settings remain unchanged. ``` PUT /api/config Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "use_typed_values": false, "storage_mode": "balanced", "durable_operations": true } ``` **Response:** ``` { "status": "success", "message": "Configuration updated successfully" } ``` *** ## Core Settings[​](#core-settings "Direct link to Core Settings") | Option | Type | Default | Range | Description | | -------------------- | ------- | -------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `use_typed_values` | Boolean | `true` | — | When `true`, responses include type metadata (e.g., `{"type": "String", "value": "Alice"}`). When `false`, returns plain values. | | `default_ttl` | Integer | `3600` | 0–604800 | Default time-to-live for new records in seconds. `0` disables expiration. Max 7 days (604,800 seconds). | | `token_expiry_hours` | Integer | `1` | 1–168 | JWT token lifetime in hours. Max 7 days (168 hours). | | `encrypt_ids` | Boolean | `true` | — | Encrypt record IDs with AES-GCM. Disable only for testing/benchmarks. | | `access_mode` | String | `"read_write"` | `"read"`, `"write"`, `"read_write"` | Node access permissions. Use `"read"` for read replicas, `"write"` for write-only nodes. | **Example — Configure a read-only replica:** ``` { "access_mode": "read", "token_expiry_hours": 24 } ``` *** ## Storage & Durability[​](#storage--durability "Direct link to Storage & Durability") ### Storage Mode[​](#storage-mode "Direct link to Storage Mode") | Option | Type | Default | Values | Description | | -------------- | ------ | -------- | ---------------------- | ---------------------------- | | `storage_mode` | String | `"fast"` | `"fast"`, `"balanced"` | Record persistence strategy. | | Mode | Description | Best For | | ---------- | --------------------------------------------------- | --------------------------- | | `fast` | In-memory primary with WAL-only persistence | Low-latency reads (default) | | `balanced` | LRU cache with periodic checkpoint batching to disk | General-purpose workloads | ### Durability & WAL[​](#durability--wal "Direct link to Durability & WAL") | Option | Type | Default | Range | Description | | ------------------------------- | ------- | ------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `durable_operations` | Boolean | `true` | — | When `true`, writes are fsynced to disk before responding. When `false`, writes are buffered and fsynced in the background for higher throughput. | | `insert_returns_record` | Boolean | `false` | — | When `true`, an insert echoes the full stored record (server-applied fields included) instead of just `{id}`. Costs a read; opt in per-request with `?return_record=true`. | | `update_returns_record` | Boolean | `true` | — | When `true` (default), an update returns the full merged record. When `false`, it returns `{id, updated:true}`. The merge already loads the record, so returning it is free. | | `delete_returns_record` | Boolean | `false` | — | When `false` (default), a delete returns the minimal `{id, deleted:true}` ack and skips loading the record body — the fast path. When `true`, it loads and returns the full deleted record. Override per-request with `?return_record=true`. | | `group_commits` | Boolean | `true` | — | Batch concurrent writes together with a single fsync. Dramatically improves throughput under concurrent load while maintaining durability. | | `group_commit_delay_ms` | Integer | `2` | 0–100 | Batching window in milliseconds for group commits. Lower = lower latency, higher = more batching throughput. | | `group_commit_mode` | String | `"leader_election"` | `"leader_election"`, `"channel"` | Group-commit completion strategy. `"leader_election"` (default, highest throughput): writers share a generation counter. `"channel"`: each writer gets its own per-operation write result (per-write error isolation). Both are equally durable. | | `wal_shards` | Integer | `4` | 1–16 | Number of parallel WAL shards for concurrent fsync. More shards = higher write throughput under concurrent load. | | `wal_max_retry_delay_ms` | Integer | `30000` | 1000–300000 | Maximum exponential backoff delay for WAL write retries, in milliseconds. Caps the delay to prevent excessive wait times when many retries accumulate. | | `wal_group_commit_timeout_secs` | Integer | `30` | 5–300 | Timeout in seconds for the WAL group-commit flusher task. Writers are unblocked automatically if the flusher crashes or stalls. | | `wal_ship_interval_secs` | Integer | `5` | 1–3600 | Interval (seconds) between WAL-shipment passes to WAL-mode replicas. A cheap no-op unless a WAL-mode ripple is configured. See [Ripples — WAL Mode](/ripples.md#wal-mode). | **Example — Maximum write throughput:** ``` { "durable_operations": true, "group_commits": true, "group_commit_delay_ms": 5, "group_commit_mode": "leader_election", "wal_shards": 8 } ``` **Example — Low-latency single-writer:** ``` { "durable_operations": true, "group_commits": false } ``` Disabling Durability Setting `durable_operations` to `false` means data written within the last flush window (typically 0–1 second) may be lost on crash. Only disable for non-critical data like analytics or bulk ingestion. ### Transactions[​](#transactions "Direct link to Transactions") | Option | Type | Default | Range | Description | | ------------------------- | ------ | ------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `default_isolation_level` | String | `"ReadUncommitted"` | `"ReadUncommitted"`, `"ReadCommitted"`, `"RepeatableRead"`, `"Serializable"` | Server-wide default [isolation level](/transactions.md#isolation-levels) for a transaction that does not specify one. A per-transaction value (passed to `begin`) overrides it. `ReadUncommitted` (default) is the zero-cost fast path; higher levels validate more at commit. Changeable at runtime. | The default applies only when a transaction does not request a level; the official client libraries send `ReadCommitted` per transaction. See [Transactions](/transactions.md) for the full semantics. *** ## Search[​](#search "Direct link to Search") | Option | Type | Default | Range | Description | | --------------- | ------- | ------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `max_vector_k` | Integer | `1000` | 1–1000000 | Upper bound a vector search's `k` (number of nearest neighbors) is clamped to. Bounds the result heap a single query can allocate. | | `max_ef_search` | Integer | `4000` | 1–1000000 | Upper bound a vector search's HNSW `ef_search` (beam width) is clamped to. Bounds the per-query work a single search can demand. | A vector search request asking for a larger `k` or `ef_search` is clamped to these caps, so a single query cannot exhaust memory or CPU. Raise them via `PUT /api/config` if your workload genuinely needs larger values. *** ## Resource Allocation[​](#resource-allocation "Direct link to Resource Allocation") | Option | Type | Default | Range | Description | | --------------------------- | ------- | ------- | ------ | --------------------------------------------------------- | | `memory_allocation_percent` | Integer | `50` | 1–80 | Percentage of system RAM to use for database caching. | | `cpu_utilization_percent` | Integer | `100` | 10–100 | Percentage of CPU cores to use for concurrent operations. | **Guidelines:** | Scenario | Memory | CPU | | --------------------------------- | ------ | ---- | | Dedicated database server | 50–70% | 100% | | Shared server with other services | 25–40% | 50% | | Background/minimal service | 10–20% | 25% | **Example — Shared server configuration:** ``` { "memory_allocation_percent": 30, "cpu_utilization_percent": 50 } ``` *** ## LLM Providers[​](#llm-providers "Direct link to LLM Providers") Configure LLM providers for [Chat/RAG](/chat-rag.md) functionality. You can configure multiple providers and select which to use per chat session. ### OpenAI[​](#openai "Direct link to OpenAI") | Option | Type | Default | Description | | ---------------------- | ------ | -------------------------- | --------------------------------------------------------------------- | | `openai_api_key` | String | — | API key for OpenAI. Required to use OpenAI models. | | `openai_api_url` | String | `"https://api.openai.com"` | Custom API URL for OpenAI-compatible APIs (e.g., LM Studio, Ollama). | | `openai_default_model` | String | — | Default model for OpenAI chat sessions (e.g., `"gpt-4"`, `"gpt-4o"`). | ### Anthropic[​](#anthropic "Direct link to Anthropic") | Option | Type | Default | Description | | --------------------------- | ------ | ------- | -------------------------------------------------------- | | `anthropic_api_key` | String | — | API key for Anthropic. Required to use Anthropic models. | | `anthropic_api_url` | String | — | Custom API URL for Anthropic. | | `anthropic_default_model` | String | — | Default model (e.g., `"claude-sonnet-4-5-20250929"`). | | `anthropic_default_version` | String | — | Anthropic API version string. | ### Perplexity[​](#perplexity "Direct link to Perplexity") | Option | Type | Default | Description | | -------------------------- | ------ | ------- | ---------------------------------------------------------- | | `perplexity_api_key` | String | — | API key for Perplexity. Required to use Perplexity models. | | `perplexity_api_url` | String | — | Custom API URL for Perplexity. | | `perplexity_default_model` | String | — | Default model for Perplexity chat sessions. | ### Gemini[​](#gemini "Direct link to Gemini") | Option | Type | Default | Description | | ---------------------- | ------ | ----------------------------------------------------------- | ----------------------------------------------------------------------------- | | `gemini_api_key` | String | — | API key for Google Gemini. Required to use Gemini models. | | `gemini_api_url` | String | `"https://generativelanguage.googleapis.com/v1beta/openai"` | Gemini's OpenAI-compatible endpoint. Override only if Google changes the URL. | | `gemini_default_model` | String | `"gemini-2.5-flash"` | Default model for Gemini chat sessions. | ### Embedding[​](#embedding "Direct link to Embedding") The embedding endpoint is configured separately from chat, so a local embedding model can serve search while a remote LLM serves chat (or the other way round). The key and the URL fall back to the OpenAI setting of the same name; the model has its own default. | Option | Type | Default | Description | | ------------------- | ------ | -------------------------- | ------------------------------------------------------------------------------------ | | `embedding_api_key` | String | `openai_api_key` | API key for the embedding provider. | | `embedding_api_url` | String | `openai_api_url` | OpenAI-compatible URL for embeddings (e.g. `"http://localhost:1234"` for LM Studio). | | `embedding_model` | String | `"text-embedding-ada-002"` | The embedding model (e.g. `"nomic-embed-text-v1.5"` for a local model). | **Example — Configure OpenAI with a local LM Studio endpoint:** ``` { "openai_api_key": "lm-studio", "openai_api_url": "http://localhost:1234", "openai_default_model": "llama-3.1-8b" } ``` **Example — Configure multiple providers:** ``` { "openai_api_key": "sk-...", "openai_default_model": "gpt-4o", "anthropic_api_key": "sk-ant-...", "anthropic_default_model": "claude-sonnet-4-5-20250929" } ``` ### Key verification on save[​](#key-verification-on-save "Direct link to Key verification on save") A provider's key or URL changed through `PUT /api/config` is probed the moment it is saved — one call to the provider's models endpoint where the provider has one (Perplexity does not, and reports `ok` with `verified: false`), no retries, a 15-second budget — and the outcome comes back in the response as `provider_checks`, keyed by provider (`openai`, `anthropic`, `perplexity`, `gemini`, and `embedding` for the embedding endpoint): ``` { "status": "success", "message": "Configuration updated successfully", "provider_checks": { "openai": { "status": "auth_failed", "verified": true, "http_status": 401, "message": "Failed to fetch OpenAI models: 401 Unauthorized: ..." } } } ``` The save itself never fails on a rejected key: a wrong key is something the response tells you, not something it refuses. The `status` vocabulary is the one `GET /api/chat_models` uses — see [Provider status](/chat-rag.md#provider-status). Perplexity has no models endpoint and reports `ok` with `verified: false`. *** ## Chat Settings[​](#chat-settings "Direct link to Chat Settings") Fine-tune chat/RAG behavior for each LLM provider. See [Chat & RAG](/chat-rag.md) for usage. ### General Chat Settings[​](#general-chat-settings "Direct link to General Chat Settings") | Option | Type | Default | Description | | --------------------------- | ------- | ------- | ------------------------------------------------------------------------------------ | | `chat_max_context_snippets` | Integer | `5` | Maximum number of context snippets returned from search results for each chat query. | | `chat_max_tokens` | Integer | `4096` | Maximum number of tokens for LLM responses. | | `chat_max_merged_messages` | Integer | `10000` | Maximum messages to merge when loading chat history. | ### Provider Temperature & Sampling[​](#provider-temperature--sampling "Direct link to Provider Temperature & Sampling") Control the creativity and determinism of LLM responses per provider. | Option | Type | Default | Range | Description | | ----------------------------- | ----- | ------- | ------- | -------------------------------------------------- | | `chat_openai_temperature` | Float | `0.6` | 0.0–2.0 | OpenAI response temperature. Lower = more focused. | | `chat_openai_top_p` | Float | `1.0` | 0.0–1.0 | OpenAI nucleus sampling. | | `chat_anthropic_temperature` | Float | `1.0` | 0.0–1.0 | Anthropic response temperature. | | `chat_anthropic_top_p` | Float | `0.99` | 0.0–1.0 | Anthropic nucleus sampling. | | `chat_perplexity_temperature` | Float | `0.2` | 0.0–2.0 | Perplexity response temperature. | | `chat_perplexity_top_p` | Float | `0.9` | 0.0–1.0 | Perplexity nucleus sampling. | ### Schema-Aware Search[​](#schema-aware-search "Direct link to Schema-Aware Search") These settings control how chat queries use collection schemas to improve search results. | Option | Type | Default | Description | | ------------------------------------ | ------- | ------- | -------------------------------------------------------------------------- | | `chat_schema_search_min_results` | Integer | `3` | Minimum text search results before triggering schema-based field matching. | | `chat_schema_search_min_word_length` | Integer | `3` | Minimum word length for query terms to be used in schema field matching. | | `chat_schema_field_match_score` | Float | `0.5` | Score added when a query term matches a schema field name. | | `chat_field_name_boost_multiplier` | Float | `0.3` | Score boost multiplier for results with matching field names. | | `chat_schema_sample_records` | Integer | `1` | Number of sample records to fetch when schema field names match the query. | ### Custom Collection Names[​](#custom-collection-names "Direct link to Custom Collection Names") Override the default collection names used by the chat system. | Option | Type | Default | Description | | -------------------------------- | ------ | --------------------------------- | --------------------------------------------------- | | `chat_configurations_collection` | String | `"chat_configurations_{db_name}"` | Collection for storing chat session configurations. | | `chat_messages_collection` | String | `"chat_messages_{db_name}"` | Collection for storing chat messages. | *** ## Write Performance Tuning[​](#write-performance-tuning "Direct link to Write Performance Tuning") Control which features run inline (synchronous) versus in the background during insert and update operations. Setting features to `false` defers them to background tasks for maximum write throughput. ### Insert Features[​](#insert-features "Direct link to Insert Features") | Option | Type | Default | Description | | ------------------------------ | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `insert_features.search_index` | Boolean | `true` | Update text/vector search indexes inline during insert. When `false`, search indexes update asynchronously. | | `insert_features.query_index` | Boolean | `true` | Update query indexes inline during insert. When `false`, query indexes update asynchronously. | | `insert_features.constraints` | Boolean | `true` | Run schema constraint validation (unique, required, enum, min, max, regex) during insert. When `false`, constraints are skipped entirely. | | `insert_features.disk_cache` | Boolean | `true` | Write to disk cache inline during insert. When `false`, disk cache writes are deferred (WAL still provides durability). | ### Update Features[​](#update-features "Direct link to Update Features") | Option | Type | Default | Description | | ------------------------------ | ------- | ------- | ------------------------------------------------------- | | `update_features.search_index` | Boolean | `true` | Update text/vector search indexes inline during update. | | `update_features.query_index` | Boolean | `true` | Update query indexes inline during update. | | `update_features.constraints` | Boolean | `true` | Run schema constraint validation during update. | | `update_features.disk_cache` | Boolean | `true` | Write to disk cache inline during update. | ### Read Features[​](#read-features "Direct link to Read Features") Optional per-read work, runtime-tunable via `PUT /api/config`. Correctness-critical work (record visibility, TTL, auth) is never affected by these flags. | Option | Type | Default | Description | | ------------------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `read_features.pattern_logging` | Boolean | `true` | Record access patterns on each read to feed cache warming. When `false`, per-read access logging is skipped — use for warming-insensitive, read-heavy workloads (e.g. a fully in-memory working set). | | `read_features.io_metrics` | Boolean | `true` | Record per-read I/O metrics for analytics. When `false`, reads are not counted in I/O analytics. | **Example — Maximum insert throughput (defer everything):** ``` { "insert_features": { "search_index": false, "query_index": false, "constraints": false, "disk_cache": false } } ``` **Example — Fast inserts with constraints still enforced:** ``` { "insert_features": { "search_index": false, "query_index": false, "constraints": true, "disk_cache": false } } ``` When to Defer Features * **Bulk ingestion**: Defer all features for maximum throughput, then rebuild indexes after * **No constraints needed**: Skip constraints if the collection has no unique/required fields * **Search not critical**: Defer search indexing if real-time search isn't required *** ## Security & Authentication[​](#security--authentication "Direct link to Security & Authentication") | Option | Type | Default | Range | Description | | -------------------- | ------- | ------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `jwt_leeway_seconds` | Integer | `5` | 0–60 | Clock-skew tolerance for JWT token validation, in seconds. Allows tokens that expired up to this many seconds ago to still be accepted. Limits the replay-attack window if node clocks drift. | *** ## Rate Limiting[​](#rate-limiting "Direct link to Rate Limiting") Configure rate limiting for public (unauthenticated) endpoints. | Option | Type | Default | Range | Description | | --------------------------------------- | ------- | ------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public_endpoint_rate_limit_enabled` | Boolean | `true` | — | Enable rate limiting on public endpoints (`/api/health`, `/api/auth/register`, `/api/auth/token`). | | `public_endpoint_rate_limit_per_minute` | Integer | `60` | 10–300 | Maximum requests per IP per minute for public endpoints. | | `admin_key_rate_limit_per_minute` | Integer | `5` | 1–60 | Maximum requests per IP per minute for admin key endpoints (`/api/auth/*/admin_key`). Kept intentionally low to resist brute-force attacks on admin credentials. | Authenticated Rate Limits Rate limits for authenticated API endpoints are determined by your deployment tier. See [Security - Rate Limiting](/security.md#rate-limiting) for tier details. *** ## Functions[​](#functions "Direct link to Functions") | Option | Type | Default | Description | | ---------------------- | ------ | ----------------------- | ---------------------------------------------------------------------- | | `functions_collection` | String | `"functions_{db_name}"` | Collection name for storing scheduled functions. | | `environment_vars` | Object | `{}` | Key-value pairs accessible in functions via `{{env.VAR_NAME}}` syntax. | **Example — Set function environment variables:** ``` { "environment_vars": { "WEBHOOK_URL": "https://api.example.com/webhook", "AUTH_TOKEN": "token123" } } ``` *** ## Maintenance[​](#maintenance "Direct link to Maintenance") | Option | Type | Default | Description | | --------------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------- | | `disable_maintenance` | Boolean | `false` | Disable periodic background maintenance tasks (cache cleanup, compaction). Only disable for benchmarking. | *** ## Full Configuration Example[​](#full-configuration-example "Direct link to Full Configuration Example") Here's an example configuring a production instance with multiple LLM providers, tuned durability, and resource limits: ``` curl -X PUT https://{EKODB_API_URL}/api/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "use_typed_values": true, "token_expiry_hours": 1, "default_ttl": 3600, "storage_mode": "fast", "durable_operations": true, "group_commits": true, "group_commit_delay_ms": 2, "wal_shards": 4, "wal_max_retry_delay_ms": 30000, "wal_group_commit_timeout_secs": 30, "jwt_leeway_seconds": 5, "admin_key_rate_limit_per_minute": 5, "memory_allocation_percent": 50, "cpu_utilization_percent": 100, "openai_api_key": "sk-...", "openai_default_model": "gpt-4o", "chat_max_context_snippets": 5, "chat_max_tokens": 4096, "public_endpoint_rate_limit_enabled": true, "public_endpoint_rate_limit_per_minute": 60, "insert_features": { "search_index": true, "query_index": true, "constraints": true, "disk_cache": true } }' ``` *** ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - CRUD operations and storage modes * [Chat & RAG](/chat-rag.md) - AI chat configuration and usage * [Security](/security.md) - Authentication, rate limits, and access control * [System Administration](/system-administration.md) - Health monitoring and WAL management --- # Data Types ekoDB supports 16 comprehensive data types with flexible type enforcement per collection and per field. ## Type Enforcement[​](#type-enforcement "Direct link to Type Enforcement") * **Write-time validation**: Types are checked when data is inserted or updated * **Optional at key-value level**: Key-value operations can be schema-less * **Required at document level**: Collections can enforce schemas * **Dynamic inference**: Types are automatically detected if not specified * **Explicit specification**: Define schemas via `/api/collections/{collection}` endpoint ## Response Formats[​](#response-formats "Direct link to Response Formats") ekoDB supports two response formats: ### Typed Responses (Default)[​](#typed-responses-default "Direct link to Typed Responses (Default)") Includes type metadata for strong typing: ``` { "name": { "type": "String", "value": "Alice" }, "age": { "type": "Integer", "value": 30 } } ``` **Benefits**: * Type safety * Explicit type information * Better for strongly-typed languages *** ### Non-Typed Responses[​](#non-typed-responses "Direct link to Non-Typed Responses") Traditional NoSQL format: ``` { "name": "Alice", "age": 30 } ``` **Benefits**: * Simpler JSON * Smaller payload size * Familiar format **Configuration**: Set via `/api/config` endpoint or ekoDB App *** ## Basic Types[​](#basic-types "Direct link to Basic Types") ### String[​](#string "Direct link to String") UTF-8 encoded text data for names, descriptions, and general content. ``` { "name": "Alice", "email": "alice@example.com", "description": "Software engineer" } ``` **Use Cases**: Names, emails, descriptions, URLs, JSON strings *** ### Integer[​](#integer "Direct link to Integer") 64-bit signed integers for whole numbers. **Range**: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 ``` { "age": 30, "count": 1000, "year": 2024 } ``` **Use Cases**: Counters, ages, quantities, IDs *** ### Float[​](#float "Direct link to Float") 64-bit IEEE 754 floating-point numbers for decimal values. ``` { "price": 29.99, "temperature": 98.6, "percentage": 0.85 } ``` **Use Cases**: Prices, measurements, percentages, scientific data Precision Limitation Floats have precision limitations. For financial calculations, use `Decimal` instead. *** ### Boolean[​](#boolean "Direct link to Boolean") Binary true/false values for logical operations. ``` { "isActive": true, "hasAccess": false, "verified": true } ``` **Use Cases**: Flags, toggles, status indicators *** Producing a non-inferable type on write ekoDB infers a field's type from the raw JSON you send: `true`/`false` → `Boolean`, a JSON string → `String`, a whole number → `Integer`, a fractional number → `Float`, an array → `Array`, an object → `Object`. The types below in **Advanced Numeric**, **Temporal**, and (for `Set`/`Vector`) **Collection** are **not** inferred from a raw value — a raw RFC 3339 string is stored as a `String`, a raw array as an `Array`, and so on. To store one of these, send the typed wrapper `{ "type": "", "value": }`. One exception, via schema coercion: if the collection's schema declares a field as `Number` or `Decimal`, a raw **numeric** value is upgraded to that type. There is no string→`DateTime`/`UUID`/`Decimal` or array→`Set`/`Vector` coercion. ## Advanced Numeric Types[​](#advanced-numeric-types "Direct link to Advanced Numeric Types") ### Number[​](#number "Direct link to Number") Flexible numeric type that holds either an integer or a float. A raw number is inferred as `Integer` or `Float`, so request `Number` explicitly with the `{ "type": "Number", "value": … }` wrapper (or declare the field as `Number` in the schema, which coerces a raw number to `Number`): ``` { "metric": { "type": "Number", "value": 42 }, // Number holding an integer "average": { "type": "Number", "value": 42.5 } // Number holding a float } ``` **Use Cases**: When you don't know if a value will be an integer or float **Benefits**: * Automatic type detection * Seamless conversion * Flexible queries *** ### Decimal[​](#decimal "Direct link to Decimal") Arbitrary-precision decimal numbers that avoid floating-point rounding errors. The wrapper's `value` is a **string** (to preserve precision); a raw string is stored as `String`, so use the wrapper (or declare the field as `Decimal` in the schema, which coerces a raw numeric value to `Decimal`): ``` { "balance": { "type": "Decimal", "value": "1234.56" }, "price": { "type": "Decimal", "value": "0.10" }, "total": { "type": "Decimal", "value": "999999.99" } } ``` **Use Cases**: * Financial calculations (currency, accounting) * Scientific computations requiring exact precision * Any scenario where `0.1 + 0.2` must equal exactly `0.3` **Why Decimal?** * No binary floating-point approximation * Exact decimal representation * Precision-preserving internal storage *** ## Temporal Types[​](#temporal-types "Direct link to Temporal Types") ### DateTime[​](#datetime "Direct link to DateTime") RFC 3339 formatted date-time values with timezone support. A raw RFC 3339 string is stored as a `String`; to store a true `DateTime` (which enables date math and range semantics), use the wrapper with an RFC 3339 string `value`: ``` { "createdAt": { "type": "DateTime", "value": "2026-01-01T00:00:00Z" }, "updatedAt": { "type": "DateTime", "value": "2026-12-31T23:59:59+00:00" }, "scheduledFor": { "type": "DateTime", "value": "2025-06-15T14:30:00-05:00" } } ``` **Format**: ISO 8601 / RFC 3339
**Timezone**: Always include timezone (Z for UTC or offset) **Use Cases**: Timestamps, scheduling, time-series data *** ### Duration[​](#duration "Direct link to Duration") Time-span values. The `Duration` wrapper's `value` is an integer number of **milliseconds** (a raw string like `"30s"` is stored as a `String`, not a `Duration`): ``` { "timeout": { "type": "Duration", "value": 30000 }, // 30 seconds "interval": { "type": "Duration", "value": 300000 }, // 5 minutes "ttl": { "type": "Duration", "value": 7200000 } // 2 hours } ``` Human-readable durations are a separate feature The shorthand forms `"30s"`, `"5m"`, `"1h"`, `"7d"` are accepted by the key-value store's `ttl` field, which has its own duration parser — they are **not** the `Duration` field type, whose wrapper takes milliseconds. **Use Cases**: TTL, timeouts, intervals, expiration *** ## Collection Types[​](#collection-types "Direct link to Collection Types") ### Array[​](#array "Direct link to Array") Ordered lists of heterogeneous elements, preserving insertion order. ``` { "tags": ["rust", "database", "nosql"], "scores": [95, 87, 92, 88], "mixed": ["text", 42, true, null] } ``` **Features**: * Maintains order * Allows duplicates * Can contain different types * Supports nested arrays **Use Cases**: Tags, lists, ordered data, multi-value fields *** ### Set[​](#set "Direct link to Set") Unordered collections of unique values with automatic deduplication. A raw array is stored as an `Array`, so use the wrapper to get a `Set`: ``` { "uniqueTags": { "type": "Set", "value": ["rust", "database", "nosql"] }, "permissions": { "type": "Set", "value": ["read", "write", "delete"] } } ``` **Features**: * Automatic deduplication * No guaranteed order * Fast membership testing * Unique values only **Use Cases**: Unique tags, permissions, categories *** ### Vector[​](#vector "Direct link to Vector") Fixed-dimension numeric arrays optimized for embeddings and vector similarity search. A raw array is stored as an `Array`, so use the wrapper to get a `Vector`: ``` { "embedding": { "type": "Vector", "value": [0.1, 0.2, 0.3, 0.4] }, "features": { "type": "Vector", "value": [1.5, 2.3, 0.8, 4.1] } } ``` **Features**: * Optimized for similarity search * High-performance indexing for fast similarity search * Cosine, Euclidean, Dot Product metrics * Fixed dimensions per field **Use Cases**: * AI/ML embeddings * Semantic search * Recommendation systems * Image/text similarity *** ### Object[​](#object "Direct link to Object") Nested documents/maps with key-value pairs for complex structured data. ``` { "address": { "street": "123 Main St", "city": "San Francisco", "state": "CA", "zip": "94102" }, "metadata": { "source": "api", "version": "1.0" } } ``` **Features**: * Nested structure * Flexible schema * Supports all types as values * Can be deeply nested **Use Cases**: Nested data, complex structures, JSON documents *** ## Specialized Types[​](#specialized-types "Direct link to Specialized Types") ### UUID[​](#uuid "Direct link to UUID") Universally unique identifiers (RFC 4122) for globally unique record identification. A raw string is stored as a `String`, so use the wrapper to get a `UUID`: ``` { "id": { "type": "UUID", "value": "550e8400-e29b-41d4-a716-446655440000" }, "userId": { "type": "UUID", "value": "6ba7b810-9dad-11d1-80b4-00c04fd430c8" } } ``` **Format**: 8-4-4-4-12 hexadecimal digits **Use Cases**: Unique identifiers, correlation IDs, distributed systems *** ### Binary[​](#binary "Direct link to Binary") Binary data for images, files, and other binary content. A raw string is stored as a `String`; use the wrapper with a base64-encoded string `value` to get `Binary`: ``` { "image": { "type": "Binary", "value": "iVBORw0KGgoAAAANSUhEUgAAAAUA..." }, "file": { "type": "Binary", "value": "SGVsbG8gV29ybGQh" } } ``` **Encoding**: base64 string on write (the wrapper also accepts a `[0-255]` byte array); reads return a byte array
**Storage**: Efficient binary storage **Use Cases**: Images, files, encrypted data, binary blobs *** ### Bytes[​](#bytes "Direct link to Bytes") Raw byte arrays for unencoded binary data storage. A raw array is stored as an `Array`, so use the wrapper with a `[0-255]` byte-value array to get `Bytes`: ``` { "rawData": { "type": "Bytes", "value": [72, 101, 108, 108, 111] } // "Hello" } ``` **Format**: Array of decimal byte values (0-255) **Use Cases**: Raw binary data, protocol buffers, custom encodings *** ### Null[​](#null "Direct link to Null") Explicit null/empty values for optional fields. ``` { "middleName": null, "deletedAt": null, "optionalField": null } ``` **Use Cases**: Optional fields, missing data, explicit absence *** ## Type Conversion[​](#type-conversion "Direct link to Type Conversion") ekoDB automatically handles type conversions where safe: ### Number Type Flexibility[​](#number-type-flexibility "Direct link to Number Type Flexibility") ``` // Number can match Integer or Float { "value": 42 } // Integer { "value": 42.5 } // Float ``` ### Automatic Coercion[​](#automatic-coercion "Direct link to Automatic Coercion") * Integer to Float (always safe, within f64's exact-integer range) * Integer to Decimal (lossless) * Float to Decimal (within Decimal's range) * Integer or Float to Number (automatic) *** ## Schema Definition[​](#schema-definition "Direct link to Schema Definition") Define schemas to enforce types when creating a collection: ``` POST /api/collections/users Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "fields": { "name": { "field_type": "String", "required": true }, "age": { "field_type": "Number", "min": 0, "max": 150 }, "email": { "field_type": "String", "unique": true }, "balance": { "field_type": "Decimal" }, "tags": { "field_type": "Array" }, "metadata": { "field_type": "Object" } } } ``` **Note:** See [Collections & Schemas](/collections-and-schemas.md) for complete schema management documentation. *** ## Best Practices[​](#best-practices "Direct link to Best Practices") ### ✅ Do[​](#-do "Direct link to ✅ Do") * Use `Decimal` for financial data * Use `DateTime` with timezones * Use `Vector` for embeddings * Use `Set` for unique collections * Define schemas for production collections ### ❌ Don't[​](#-dont "Direct link to ❌ Don't") * Use `Float` for money (use `Decimal`) * Store dates as strings (use `DateTime`) * Use `Array` when you need uniqueness (use `Set`) * Mix types inconsistently *** ## Next Steps[​](#next-steps "Direct link to Next Steps") * **Learn Schemas**: [Schema Validation](/collections-and-schemas.md) * **Query Types**: [Basic Operations](/basic-operations.md) * **Advanced Types**: [White Paper - Type System](/white-paper.md#23-type-system) *** **Questions?** Contact --- # Getting Started Want to get running fast? Check out the [Quick Start](/quick-start.md) guide for a 5-minute tutorial. This page provides comprehensive deployment and setup information. ## Sign Up for an Account[​](#sign-up-for-an-account "Direct link to Sign Up for an Account") To get started, please visit our application at [app.ekodb.io](https://app.ekodb.io) and sign up for an account. To do so, you will be required to provide us with some basic information and agree to ekoDB’s Privacy Policy and Terms of Use agreements before proceeding. For more information regarding either of these agreements, please visit the links to them at the bottom of our marketing website [ekodb.io](https://ekodb.io) or contact our legal team with questions via email at . For privacy-related inquiries, please contact us at . Once you have created an account and agreed to all terms, you can begin with your first ekoDB deployment! ## Deploying Your First Database[​](#deploying-your-first-database "Direct link to Deploying Your First Database") When you create a deployment, you first choose what to deploy: * **Database** — the core ekoDB instance, with built-in AI chat, RAG, and vector search. This guide deploys a database; it runs on its own and is all most projects need. * **AI Agent** — a separate deployment that connects to a database and adds autonomous AI (tools, goals, scheduled tasks). When you're ready for that, see [Build Your First AI Agent](/build-your-first-agent.md). Choose **Database**, then continue with the steps below. ### Selecting a Region[​](#selecting-a-region "Direct link to Selecting a Region") Start by selecting a region to deploy to - ekoDB has worldwide deployment capability, so select an area to localize your instance to provide the fastest connection for your application. ![Select A Region](/assets/images/SelectARegion-77479fb6a274e0de3b1fd54644f10d65.png) ### Selecting a Machine[​](#selecting-a-machine "Direct link to Selecting a Machine") Next, you select which machine (cloud provider and level of hardware) to deploy to. We have a selection of machines/hardware to choose from, though specific availability will be based on several factors including the region you are deploying to, selected cloud provider availability, etc. ![Select A Machine](/assets/images/SelectAMachine-102d84657e0ddbf60dac377577391aaf.png) ### Selecting an Environment and Subdomain[​](#selecting-an-environment-and-subdomain "Direct link to Selecting an Environment and Subdomain") Lastly, choose a custom subdomain and an environment (development, testing, or production) that you will deploy to. Note that this subdomain and environment combination (in addition to the cloud provider selected previously) will be the basis for the URL you will connect to, so subdomain, environment, and provider combinations must be unique. See [our API URL documentation](/api-urls.md) for more information. ![Select An Environment And Subdomain](/assets/images/SelectEnvAndSubdomain-945b8aa8ded385057c1a7b19c272aa9e.png) From here, click deploy to deploy your new database! You will be brought to the ekoDB app’s dashboard page, where you will be able to manage your deployment(s). It is typical for it to take at least several minutes for your new deployment to spin up - you will see your deployment’s status change from `Initializing`, to `Starting`, and finally to `Running` once it is fully up and running (you should also receive an email letting you know that your deployment is ready as well as any other status changes). ### Deployment Starting and Running[​](#deployment-starting-and-running "Direct link to Deployment Starting and Running") ![Deployment Starting](/assets/images/DeploymentStarting-0c2679f081832d66aebc70df4275833b.png) Now that you have your first deployment running, we recommend you navigate these developer documents further to learn how to connect to your database and interact with it. Please note that in our developer documents, any occurrence of `{VARIABLE}` is a placeholder for a variable value that you will need to provide. Also note that in order to connect to your database, you will need your API key, which can be found by selecting your deployment from the dashboard in the ekoDB app and navigating to the "Keys" section. For testing purposes, you can use an admin API key. However, we strongly recommend that admin API keys only be used for testing and development purposes. The next section, [Authentication](/authentication.md), will provide more details on provisioning API keys with collection rights. Want autonomous AI on top of this database? Your database already includes built-in [chat and RAG](/chat-rag.md). When you want an agent that can use tools, pursue goals, and run scheduled tasks on its own, deploy an **AI Agent** and connect it to this database — see [Build Your First AI Agent](/build-your-first-agent.md). ## Installing the Client Library[​](#installing-the-client-library "Direct link to Installing the Client Library") ekoDB provides official client libraries for multiple programming languages. We recommend using these libraries instead of making direct API calls, as they provide type safety, automatic authentication, retry logic, and a more intuitive developer experience. ### Step 3: Install Client Library[​](#step-3-install-client-library "Direct link to Step 3: Install Client Library") ekoDB provides official client libraries for **Rust, Python, TypeScript, Go, Kotlin, and JavaScript**. Installation Guide For complete installation instructions, configuration options, and troubleshooting for each language, see the **[Client Libraries Installation Guide](/client-libraries/installation.md)**. ### Quick Start Example[​](#quick-start-example "Direct link to Quick Start Example") Here's a quick example of connecting to your ekoDB instance: ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://{YOUR_SUBDOMAIN}.{ENVIRONMENT}.{PROVIDER}.ekodb.net", apiKey: "your-api-key" }); await client.init(); // Ready to use! const collections = await client.listCollections(); console.log("Collections:", collections); ``` Full Client Documentation For complete initialization options, configuration examples, and usage guides in all supported languages (TypeScript, Rust, Python, Go, Kotlin), see: * 📚 [Client Libraries Installation](/client-libraries/installation.md) * 🔧 [Client Libraries Basic Operations](/client-libraries/basic-operations.md) * 🚀 [Client Libraries Overview](/client-libraries/.md) * 💡 [GitHub Examples Repository](https://github.com/ekoDB/ekodb-client/tree/main/examples) - 129 (client library & direct HTTP examples) --- # Indexes Indexes dramatically improve query and search performance by creating optimized data structures for fast lookups. ekoDB supports two types of indexes: **query indexes** for exact matches and filters, and **search indexes** for full-text and vector similarity search. When to Use Indexes * **Query often** on specific fields (e.g., `status`, `email`, `user_id`) * **Search** text content or vector embeddings * **Performance issues** with large collections (> 10,000 records) * **Filter combinations** are common in your queries ## Query Indexes[​](#query-indexes "Direct link to Query Indexes") Query indexes optimize exact match and comparison queries using B-tree structures. ### Create Query Index[​](#create-query-index "Direct link to Create Query Index") ## Create Index[​](#create-index "Direct link to Create Index") Define an index in the collection schema. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::{Client, Schema, FieldTypeSchema, IndexConfig}; let schema = Schema::new() .add_field( "email", FieldTypeSchema::new("string") .with_index(IndexConfig::Hash) ); client.create_collection("users", schema).await?; ``` ``` client.create_schema('users', { 'indexes': [ { 'name': 'email_idx', 'type': 'hash', 'fields': ['email'] } ] }) ``` ``` await client.createSchema("users", { indexes: [ { name: "email_idx", type: "hash", fields: ["email"], }, ], }); ``` ``` await client.createSchema("users", { indexes: [ { name: "email_idx", type: "hash", fields: ["email"], }, ], }); ``` ``` import io.ekodb.client.EkoDBClient val schema = NewSchemaBuilder() .addField("email", NewFieldTypeSchemaBuilder("string") .hashIndex() .build() ) .build() client.createCollection("users", schema) ``` ``` import "github.com/ekoDB/ekodb-client-go" schema := ekodb.NewSchemaBuilder(). AddField("email", ekodb.NewFieldTypeSchemaBuilder("string"). HashIndex(). Build(), ). Build() err := client.CreateCollection("users", schema) ``` ``` curl -X POST https://{EKODB_API_URL}/api/schemas/users \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "indexes": [ { "name": "email_idx", "type": "hash", "fields": ["email"] } ] }' # Response { "status": "success", "message": "Schema created successfully" } ``` Create an index on one or more fields to speed up queries. ``` POST https://{EKODB_API_URL}/api/indexes/query/{collection} Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "field": "email", "index_type": "BTree", "label": "email_idx" } # Response { "status": "success", "message": "Index 'email_idx' created successfully", "index": { "label": "email_idx", "collection": "users", "field": "email", "index_type": "BTree", "unique": false, "sparse": false } } ``` **Optional Parameters:** * `unique` (boolean) - Enforce uniqueness constraint (default: false) * `sparse` (boolean) - Index only documents with the field (default: false) **Composite Index (Multiple Fields):** ``` { "label": "status_created_idx", "fields": ["status", "created_at"], "index_type": "btree" } ``` **Idempotency:** Re-creating an index that already exists returns `409 Conflict` and leaves the existing index unchanged, so it is safe to run your index setup on every deploy. A create with an unknown field or invalid type returns `400 Bad Request`. ### List Query Indexes[​](#list-query-indexes "Direct link to List Query Indexes") Get all query indexes for a collection. * Client Libraries (Recommended) * Direct API Client Library Support Index listing is not yet available in client libraries. Use the Direct API below. ``` curl https://{EKODB_API_URL}/api/indexes/query/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "collection": "users", "indexes": [ { "label": "users_email_index", "collection": "users", "field": "email", "index_type": "BTree", "unique": false, "sparse": false }, { "label": "users_status_index", "collection": "users", "field": "status", "index_type": "Hash", "unique": false, "sparse": false } ], "count": 2 } ``` ### Delete Query Index[​](#delete-query-index "Direct link to Delete Query Index") Remove an index when it's no longer needed. * Client Libraries (Recommended) * Direct API Client Library Support Index deletion is not yet available in client libraries. Use the Direct API below. ``` curl -X DELETE https://{EKODB_API_URL}/api/indexes/query/{collection}/{field} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "success", "message": "Index on field 'email' deleted successfully", "index": null } ``` ### Explain Query Execution[​](#explain-query-execution "Direct link to Explain Query Execution") Analyze how a query will be executed and whether indexes are used. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::QueryBuilder; use serde_json::json; let explanation = client.explain_query( "users", json!({ "filter": { "type": "Condition", "content": { "field": "email", "operator": "Eq", "value": "user@example.com" } } }) ).await?; println!("{:?}", explanation["execution_plan"]); ``` ``` explanation = client.explain_query('users', { 'filter': { 'type': 'Condition', 'content': { 'field': 'email', 'operator': 'Eq', 'value': 'user@example.com' } } }) print(explanation['execution_plan']) ``` ``` const explanation = await client.explainQuery("users", { filter: { type: "Condition", content: { field: "email", operator: "Eq", value: "user@example.com", }, }, }); console.log(explanation.execution_plan); ``` ``` const explanation = await client.explainQuery("users", { filter: { type: "Condition", content: { field: "email", operator: "Eq", value: "user@example.com", }, }, }); console.log(explanation.execution_plan); ``` ``` val explanation = client.explainQuery("users", mapOf( "filter" to mapOf( "type" to "Condition", "content" to mapOf( "field" to "email", "operator" to "Eq", "value" to "user@example.com" ) ) )) println(explanation["execution_plan"]) ``` ``` explanation, err := client.ExplainQuery("users", map[string]interface{}{ "filter": map[string]interface{}{ "type": "Condition", "content": map[string]interface{}{ "field": "email", "operator": "Eq", "value": "user@example.com", }, }, }) fmt.Println(explanation["execution_plan"]) ``` ``` curl -X POST https://{EKODB_API_URL}/api/query/{collection}/explain \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "filter": { "type": "Condition", "content": { "field": "email", "operator": "Eq", "value": "user@example.com" } } }' # Response { "query": { "filter": { "type": "Condition", "content": { "field": "email", "operator": "Eq", "value": "user@example.com" } } }, "execution_plan": { "scan_type": "index_scan", "indexes_used": [ { "field": "email", "index_type": "BTree", "unique": false, "sparse": false } ], "estimated_rows": 100, "filter_selectivity": 0.01 }, "estimated_cost": 5.5, "recommendations": [ "Index on 'email' field is optimal for this query", "Consider creating a unique index if email values are unique" ] } ``` ## Search Indexes[​](#search-indexes "Direct link to Search Indexes") Search indexes enable full-text search and vector similarity search. ### Create Search Index[​](#create-search-index "Direct link to Create Search Index") Create a search index for text or vector fields. **Text Search Index:** ``` POST https://{EKODB_API_URL}/api/indexes/search/{collection} Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "field": "description", "index_type": "text", "config": { "language": "english", "stemming": true, "stop_words": true, "min_word_length": 3 } } # Response { "status": "success", "message": "Search index created successfully on field 'description'", "field": "description", "index_type": "text", "documents_indexed": 5678 } ``` **Vector Search Index:** ``` POST https://{EKODB_API_URL}/api/indexes/search/{collection} Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "field": "embedding", "index_type": "vector", "config": { "metric": "cosine", "algorithm": "hnsw", "ef_construction": 200, "m": 16 } } # Response { "status": "success", "message": "Search index created successfully on field 'embedding'", "field": "embedding", "index_type": "vector", "documents_indexed": 10000 } ``` ### Explain Text Search[​](#explain-text-search "Direct link to Explain Text Search") Analyze text search query execution. ``` POST https://{EKODB_API_URL}/api/search/text/{collection}/explain Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "query": "machine learning algorithms", "field": "description", "limit": 10 } # Response { "query": "machine learning algorithms", "parsed_terms": ["machine", "learning", "algorithm"], "stemmed_terms": ["machin", "learn", "algorithm"], "execution_plan": { "index_used": "description_text", "search_type": "full_text", "estimated_matches": 156, "scoring_method": "tf_idf" }, "performance": { "estimated_time_ms": 15, "index_hit_rate": "high" } } ``` ### Explain Vector Search[​](#explain-vector-search "Direct link to Explain Vector Search") Analyze vector similarity search execution. ``` POST https://{EKODB_API_URL}/api/search/vector/{collection}/explain Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "vector": [0.1, 0.2, 0.3, ...], "field": "embedding", "limit": 10, "metric": "cosine" } # Response { "execution_plan": { "index_used": "embedding_vector", "algorithm": "hnsw", "dimension": 1536, "metric": "cosine", "estimated_comparisons": 234, "search_type": "approximate" }, "performance": { "estimated_time_ms": 8, "accuracy": "high", "speedup_vs_brute_force": "427x" }, "index_stats": { "total_vectors": 10000, "ef_search": 100, "levels": 4 } } ``` ### Explain Hybrid Search[​](#explain-hybrid-search "Direct link to Explain Hybrid Search") Analyze combined text and vector search. ``` POST https://{EKODB_API_URL}/api/search/hybrid/{collection}/explain Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} { "text_query": "neural networks", "vector": [0.1, 0.2, ...], "text_weight": 0.3, "vector_weight": 0.7, "limit": 10 } # Response { "execution_plan": { "text_search": { "index_used": "description_text", "estimated_matches": 89, "weight": 0.3 }, "vector_search": { "index_used": "embedding_vector", "estimated_matches": 10000, "weight": 0.7 }, "fusion_method": "reciprocal_rank" }, "performance": { "estimated_time_ms": 23, "combined_accuracy": "very_high" } } ``` ## Index Types[​](#index-types "Direct link to Index Types") ### Query Index Types[​](#query-index-types "Direct link to Query Index Types") | Type | Use Case | Performance | Space Usage | | ------- | ---------------------------- | ----------------- | ----------- | | `btree` | Exact matches, range queries | Very fast lookups | Moderate | | `hash` | Exact matches only | Fastest lookups | Low | ### Search Index Types[​](#search-index-types "Direct link to Search Index Types") | Type | Use Case | Performance | Space Usage | | -------- | -------------------------- | ----------- | ----------- | | `text` | Full-text search | Fast | Moderate | | `vector` | Semantic/similarity search | Very fast | High | > `index_type` accepts only `text` or `vector`. Hybrid (combined text + vector) is a query-time operation, not a standalone index type. Use the hybrid search endpoints rather than creating a `hybrid` index. ## Vector Index Algorithms[​](#vector-index-algorithms "Direct link to Vector Index Algorithms") ### HNSW (Hierarchical Navigable Small World)[​](#hnsw-hierarchical-navigable-small-world "Direct link to HNSW (Hierarchical Navigable Small World)") Best for most use cases - balances speed and accuracy. ``` { "algorithm": "hnsw", "config": { "ef_construction": 200, "m": 16, "ef_search": 100 } } ``` **Parameters:** * `ef_construction` - Quality during build (higher = better, slower) * `m` - Connections per layer (higher = better recall, more space) * `ef_search` - Search quality (higher = better, slower) ### IVF (Inverted File)[​](#ivf-inverted-file "Direct link to IVF (Inverted File)") Not yet implemented `ivf` is accepted as an `algorithm` value but currently falls back to HNSW behavior; a dedicated IVF index is not yet implemented. Use `hnsw` (the default) or `flat` for now. ## Distance Metrics[​](#distance-metrics "Direct link to Distance Metrics") ### Cosine Similarity[​](#cosine-similarity "Direct link to Cosine Similarity") Best for: Text embeddings, normalized vectors ``` { "metric": "cosine" } ``` **Range:** -1 to 1 (1 = identical, -1 = opposite) ### Euclidean Distance (L2)[​](#euclidean-distance-l2 "Direct link to Euclidean Distance (L2)") Best for: Spatial data, image embeddings ``` { "metric": "euclidean" } ``` **Range:** 0 to ∞ (0 = identical) ### Dot Product[​](#dot-product "Direct link to Dot Product") Best for: Pre-normalized embeddings, fast comparisons ``` { "metric": "dotproduct" } ``` ## Complete Example[​](#complete-example "Direct link to Complete Example") Create a high-performance search system: ``` #!/bin/bash # 1. Create collection for articles curl -X POST https://{EKODB_API_URL}/api/collections/articles \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "fields": { "title": {"field_type": "String", "required": true}, "content": {"field_type": "String", "required": true}, "author_id": {"field_type": "String", "required": true}, "status": {"field_type": "String", "enums": ["draft", "published"]}, "embedding": {"field_type": "Vector"} } }' # 2. Create query index for common filters curl -X POST https://{EKODB_API_URL}/api/indexes/query/articles \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "label": "articles_status_idx", "field": "status", "index_type": "btree" }' # 3. Create composite index for author + status curl -X POST https://{EKODB_API_URL}/api/indexes/query/articles \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "label": "articles_author_status_idx", "fields": ["author_id", "status"], "index_type": "btree" }' # 4. Create text search index curl -X POST https://{EKODB_API_URL}/api/indexes/search/articles \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "field": "content", "index_type": "text", "config": { "language": "english", "stemming": true, "stop_words": true } }' # 5. Create vector index for semantic search curl -X POST https://{EKODB_API_URL}/api/indexes/search/articles \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "field": "embedding", "index_type": "vector", "config": { "metric": "cosine", "algorithm": "hnsw", "ef_construction": 200, "m": 16 } }' # 6. Explain query to verify index usage curl -X POST https://{EKODB_API_URL}/api/query/articles/explain \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "published" } }, { "type": "Condition", "content": { "field": "author_id", "operator": "Eq", "value": "author_123" } } ] } } }' # 7. Test hybrid search explain curl -X POST https://{EKODB_API_URL}/api/search/hybrid/articles/explain \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "text_query": "machine learning", "vector": [0.1, 0.2, ...], "text_weight": 0.4, "vector_weight": 0.6, "limit": 10 }' ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Index Selection[​](#index-selection "Direct link to Index Selection") **Create Indexes For:** * ✅ Fields used in WHERE clauses frequently * ✅ Foreign keys and join fields * ✅ Fields used for sorting * ✅ Text fields for search * ✅ Vector embeddings for similarity search **Avoid Indexing:** * ❌ Low-cardinality fields (e.g., boolean with 2 values) * ❌ Fields that change frequently * ❌ Very large text fields (use search index instead) * ❌ Fields never used in queries ### Composite Indexes[​](#composite-indexes "Direct link to Composite Indexes") **Order Matters:** ``` # Good: Index fields in order of selectivity # (status has few values, created_at is more selective) fields: ["created_at", "status"] # Less optimal fields: ["status", "created_at"] ``` **Use for Common Query Combinations:** ``` # If you often query: WHERE author_id = X AND status = Y # Create composite index: fields: ["author_id", "status"] ``` ### Monitor Index Usage[​](#monitor-index-usage "Direct link to Monitor Index Usage") ``` # Check which indexes exist GET /api/indexes/query/articles # Response shows all indexes { "collection": "articles", "indexes": [ { "label": "articles_email_index", "collection": "articles", "field": "email", "index_type": "BTree", "unique": true, "sparse": false }, { "label": "articles_old_field_index", "collection": "articles", "field": "old_field", "index_type": "Hash", "unique": false, "sparse": false } ], "count": 2 } # Delete unused indexes to save space DELETE /api/indexes/query/articles/old_field ``` ### Vector Index Tuning[​](#vector-index-tuning "Direct link to Vector Index Tuning") **For Accuracy:** ``` { "algorithm": "hnsw", "ef_construction": 400, "m": 32, "ef_search": 200 } ``` **For Speed:** ``` { "algorithm": "hnsw", "ef_construction": 100, "m": 16, "ef_search": 50 } ``` **For Balance:** ``` { "algorithm": "hnsw", "ef_construction": 200, "m": 16, "ef_search": 100 } ``` ### Index Maintenance[​](#index-maintenance "Direct link to Index Maintenance") **Rebuild Indexes After:** * Large bulk imports * Schema changes * Performance degradation * Significant data updates ``` # Delete and recreate index DELETE /api/indexes/query/articles/email POST /api/indexes/query/articles { "field": "email", "index_type": "btree" } ``` ## Performance Impact[​](#performance-impact "Direct link to Performance Impact") ### Query Performance[​](#query-performance "Direct link to Query Performance") | Records | No Index | With Index | Speedup | | ------- | --------- | ---------- | ------- | | 1,000 | \~5ms | \~1ms | 5x | | 10,000 | \~50ms | \~2ms | 25x | | 100,000 | \~500ms | \~3ms | 166x | | 1M | \~5,000ms | \~5ms | 1,000x | ### Space Usage[​](#space-usage "Direct link to Space Usage") | Index Type | Space Overhead | Example (1M records) | | ------------- | -------------- | -------------------- | | Query (btree) | 10-20% | \~50-100 MB | | Text search | 20-40% | \~100-200 MB | | Vector (HNSW) | 50-100% | \~500MB-1GB | ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - Query and search records * [Query Expressions](/query-expressions.md) - Filter syntax for search queries * [Collections & Schemas](/collections-and-schemas.md) - Create collections * [System Administration](/system-administration.md) - Monitor performance ## Example Code[​](#example-code "Direct link to Example Code") Complete working examples for search and schema management: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_search.rs)**: `client_search.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_search.py)**: `client_search.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_search.ts)**: `client_search.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_search.go)**: `client_search.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientSearch.kt)**: `ClientSearch.kt` Schema management examples: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_schema.rs)**: `client_schema.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_schema.py)**: `client_schema.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_schema.ts)**: `client_schema.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_schema.go)**: `client_schema.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientSchemaManagement.kt)**: `ClientSchemaManagement.kt` --- # Introduction Welcome to ekoDB! We're glad you're here. As you explore these docs and start building, we'd love your feedback — email us anytime at . ## What ekoDB is[​](#what-ekodb-is "Direct link to What ekoDB is") ekoDB is a fast, real-time, **AI-native database**. It stores both document and key-value data as collections with optional schemas and constraints, serves reads and writes over REST and WebSocket Secure (WSS) with sub-millisecond point reads, and ships with AI built into the database runtime — **chat, retrieval-augmented generation (RAG), and vector search** — with no second vector database and no bolt-on RAG pipeline to maintain. It is secured end-to-end with collection-level permissions, and scales across instances with Ripple, ekoDB's built-in multi-instance synchronization. ## What you deploy[​](#what-you-deploy "Direct link to What you deploy") You deploy ekoDB from the dashboard at [app.ekodb.io](https://app.ekodb.io) as a managed cloud service — there is nothing to host or operate yourself. There are two kinds of deployment: * **Database** — the core ekoDB instance. It runs on its own and is fully featured, including the built-in AI chat, RAG, and vector search. Most projects start here, and many never need anything else. * **AI Agent** *(optional)* — a separate managed deployment that **connects to a database** and adds autonomous AI: tools, goals, scheduled tasks, persistent memory, and ready-made backend templates. An agent always works with a database; a database needs nothing else. In short: **deploy a database on its own, or add AI agent(s) that connect to it.** See [AI Agents](/ai-agents.md) for the agent story, or jump straight into the [Quick Start](/quick-start.md). ## Next Steps[​](#next-steps "Direct link to Next Steps") Ready to get started? * **[Quick Start](/quick-start.md)** — Get your first database running in 5 minutes * **[Getting Started](/getting-started.md)** — Comprehensive deployment and setup guide * **[Build Your First AI Agent](/build-your-first-agent.md)** — Deploy an agent against your own data in 10 minutes * **[Choose Your Path](/choose-your-path.md)** — Find the best approach for your use case If you have any questions or need a hand, reach us at . ### 📚 Learn by Example[​](#-learn-by-example "Direct link to 📚 Learn by Example") **Browse 129 Code Examples** — Complete, runnable examples in Rust, Python, TypeScript, JavaScript, Go, and Kotlin. Includes both **client library examples** (using official SDKs) and **direct HTTP/REST examples** (raw API calls). --- # Key-Value Store ekoDB includes a built-in key-value (KV) store optimized for fast lookups, caching, and metadata storage. Use KV operations for simple key-value pairs when you don't need the full document structure. When to Use KV Store * **Caching** - Store frequently accessed data * **Session Management** - User sessions and temporary data * **Configuration** - Application settings and flags * **Counters** - Track metrics and statistics * **Metadata** - Additional data linked to documents First, generate an access token using your API key: ``` curl -X POST https://{EKODB_API_URL}/api/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key": "YOUR_API_KEY"}' ``` **Response:** ``` { "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." } ``` ## Get Value[​](#get-value "Direct link to Get Value") Retrieve a value by key. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let value = client.kv_get("session:user123").await?; println!("{:?}", value); ``` ``` value = await client.kv_get('session:user123') print(value) # {'user_id': 'user_456', 'login_time': '2026-01-15T10:30:00Z'} ``` ``` const value = await client.kvGet('session:user123'); console.log(value); // { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' } ``` ``` const value = await client.kvGet('session:user123'); console.log(value); // { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' } ``` ``` val value = client.kvGet("session:user123") println(value) ``` ``` value, err := client.KVGet("session:user123") fmt.Println(value) ``` ``` curl https://{EKODB_API_URL}/api/kv/get/session:user123 \ -H "Authorization: Bearer {TOKEN}" # Response { "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" } ``` Response Format The get operation returns the value directly without wrapping it in an object. The key is part of the URL path. ## Set Value[​](#set-value "Direct link to Set Value") Store or update a key-value pair. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let value = serde_json::json!({ "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" }); client.kv_set("session:user123", value).await?; ``` ``` value = { 'user_id': 'user_456', 'login_time': '2026-01-15T10:30:00Z' } await client.kv_set('session:user123', value) ``` ``` const value = { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' }; await client.kvSet('session:user123', value); ``` ``` const value = { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' }; await client.kvSet('session:user123', value); ``` ``` val value = mapOf( "user_id" to "user_456", "login_time" to "2026-01-15T10:30:00Z" ) client.kvSet("session:user123", value) ``` ``` value := map[string]interface{}{ "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z", } err := client.KVSet("session:user123", value) ``` ``` curl -X POST https://{EKODB_API_URL}/api/kv/set/session:user123 \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" }' # Response { "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" } ``` Response Format The set operation returns the stored value directly, confirming what was saved. **Value Types:** KV store accepts any JSON value: ``` // String "simple value" // Number 42 // Boolean true // Object {"nested": "data", "count": 10} // Array ["item1", "item2", "item3"] ``` ## Set Value with TTL[​](#set-value-with-ttl "Direct link to Set Value with TTL") Store a key-value pair that automatically expires after a specified duration. This is ideal for caching, session management, and temporary data. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` // Set with 1 hour TTL (pass TTL as optional 3rd parameter) let value = serde_json::json!({ "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" }); client.kv_set("session:user123", value, Some("1h")).await?; ``` ``` # Set with 1 hour TTL (pass TTL as optional parameter) value = { 'user_id': 'user_456', 'login_time': '2026-01-15T10:30:00Z' } await client.kv_set('session:user123', value, ttl='1h') ``` ``` // Set with 1 hour TTL (TTL in seconds) const value = { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' }; await client.kvSet('session:user123', value, 3600); // 3600 seconds = 1 hour ``` ``` // Set with 1 hour TTL (TTL in seconds) const value = { user_id: 'user_456', login_time: '2026-01-15T10:30:00Z' }; await client.kvSet('session:user123', value, 3600); // 3600 seconds = 1 hour ``` ``` // Set with 1 hour TTL (Kotlin has a dedicated method) val value = buildJsonObject { put("user_id", "user_456") put("login_time", "2026-01-15T10:30:00Z") } client.kvSetWithTtl("session:user123", value, "1h") ``` ``` // Set with TTL using batch operation // Note: Go single KVSet doesn't support TTL directly // Use KVBatchSet for TTL support value := map[string]interface{}{ "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z", } entries := []map[string]interface{}{ {"key": "session:user123", "value": value, "ttl": int64(3600)}, // 1 hour } _, err := client.KVBatchSet(entries) ``` ``` curl -X POST https://{EKODB_API_URL}/api/kv/set/session:user123 \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "value": { "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" }, "ttl": "1h" }' # Response { "user_id": "user_456", "login_time": "2026-01-15T10:30:00Z" } ``` ### TTL Format[​](#ttl-format "Direct link to TTL Format") TTL supports human-readable duration formats: | Format | Description | Example | | ------ | ----------- | ---------- | | `30s` | Seconds | 30 seconds | | `5m` | Minutes | 5 minutes | | `1h` | Hours | 1 hour | | `2d` | Days | 2 days | | `1w` | Weeks | 1 week | TTL Behavior * Keys are automatically deleted when TTL expires * Updating a key resets or clears the TTL depending on whether you include the TTL parameter * Expired keys return `null`/`{}` on get operations (no error) * TTL is optional; keys without TTL persist indefinitely ## Delete Value[​](#delete-value "Direct link to Delete Value") Delete a key-value pair. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.kv_delete("session:user123").await?; ``` ``` await client.kv_delete('session:user123') ``` ``` await client.kvDelete('session:user123'); ``` ``` await client.kvDelete('session:user123'); ``` ``` client.kvDelete("session:user123") ``` ``` err := client.KVDelete("session:user123") ``` ``` curl -X DELETE https://{EKODB_API_URL}/api/kv/delete/session:user123 \ -H "Authorization: Bearer {TOKEN}" # Response { "message": "deleted" } ``` ## Batch Get[​](#batch-get "Direct link to Batch Get") Retrieve multiple values in a single request. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let keys = vec!["session_1", "session_2", "session_3"]; let values = client.kv_batch_get(&keys).await?; println!("{:?}", values); ``` ``` keys = ['session_1', 'session_2', 'session_3'] values = await client.kv_batch_get(keys) print(values) # [{"user_id": "user_1"}, {"user_id": "user_2"}, {}] ``` ``` const keys = ['session_1', 'session_2', 'session_3']; const values = await client.kvBatchGet(keys); console.log(values); // [{"user_id": "user_1"}, {"user_id": "user_2"}, {}] ``` ``` const keys = ['session_1', 'session_2', 'session_3']; const values = await client.kvBatchGet(keys); console.log(values); // [{"user_id": "user_1"}, {"user_id": "user_2"}, {}] ``` ``` val keys = listOf("session_1", "session_2", "session_3") val values = client.kvBatchGet(keys) println(values) // [{"user_id": "user_1"}, {"user_id": "user_2"}, {}] ``` ``` keys := []string{"session_1", "session_2", "session_3"} values, err := client.KVBatchGet(keys) fmt.Println(values) // [{"user_id": "user_1"}, {"user_id": "user_2"}, {}] ``` ``` curl -X POST https://{EKODB_API_URL}/api/kv/batch/get \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "keys": ["session_1", "session_2", "session_3"] }' # Response [ {"user_id": "user_1"}, {"user_id": "user_2"}, {} ] ``` Response Format Returns an array of values in the same order as the requested keys. Missing keys return an empty object `{}`. ## Batch Set[​](#batch-set "Direct link to Batch Set") Set multiple key-value pairs in a single operation. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let operations = vec![ ("counter_views", serde_json::json!(1000)), ("counter_clicks", serde_json::json!(250)), ("feature_flags", serde_json::json!({ "new_ui": true, "beta_features": false, })), ]; client.kv_batch_set(operations).await?; ``` ``` operations = [ {'key': 'counter_views', 'value': 1000}, {'key': 'counter_clicks', 'value': 250}, {'key': 'feature_flags', 'value': {'new_ui': True, 'beta_features': False}} ] await client.kv_batch_set(operations) ``` ``` const operations = [ { key: 'counter_views', value: 1000 }, { key: 'counter_clicks', value: 250 }, { key: 'feature_flags', value: { new_ui: true, beta_features: false } } ]; await client.kvBatchSet(operations); ``` ``` const operations = [ { key: 'counter_views', value: 1000 }, { key: 'counter_clicks', value: 250 }, { key: 'feature_flags', value: { new_ui: true, beta_features: false } } ]; await client.kvBatchSet(operations); ``` ``` val operations = listOf( mapOf("key" to "counter_views", "value" to 1000), mapOf("key" to "counter_clicks", "value" to 250), mapOf("key" to "feature_flags", "value" to mapOf("new_ui" to true, "beta_features" to false)) ) client.kvBatchSet(operations) ``` ``` operations := []map[string]interface{}{ {"key": "counter_views", "value": 1000}, {"key": "counter_clicks", "value": 250}, {"key": "feature_flags", "value": map[string]interface{}{"new_ui": true, "beta_features": false}}, } err := client.KVBatchSet(operations) ``` ``` curl -X POST https://{EKODB_API_URL}/api/kv/batch/set \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "operations": [ { "key": "counter_views", "value": 1000 }, { "key": "counter_clicks", "value": 250 }, { "key": "feature_flags", "value": { "new_ui": true, "beta_features": false } } ] }' # Response [ ["counter_views", true], ["counter_clicks", true], ["feature_flags", true] ] ``` Response Format Returns an array of `[key, success]` tuples indicating which keys were successfully set. ## Batch Delete[​](#batch-delete "Direct link to Batch Delete") Delete multiple keys in a single operation. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let keys = vec!["session:user123", "session:user456", "session:user789"]; let result = client.kv_batch_delete(&keys).await?; // Vec<(String, bool)> let deleted = result.iter().filter(|(_, ok)| *ok).count(); println!("Deleted: {}", deleted); ``` ``` result = await client.kv_batch_delete([ 'session:user123', 'session:user456', 'session:user789' ]) print(sum(1 for _, ok in result if ok)) # 3 (result is a list of [key, success]) ``` ``` const result = await client.kvBatchDelete([ 'session:user123', 'session:user456', 'session:user789' ]); console.log(result.filter(([, ok]) => ok).length); // 3 ``` ``` const result = await client.kvBatchDelete([ 'session:user123', 'session:user456', 'session:user789' ]); console.log(result.filter(([, ok]) => ok).length); // 3 ``` ``` val result = client.kvBatchDelete(listOf( "session:user123", "session:user456", "session:user789" )) val deleted = result.count { it.second } println("Deleted: $deleted") ``` ``` result, err := client.KVBatchDelete([]string{ "session:user123", "session:user456", "session:user789", }) fmt.Printf("Deleted: %v\n", result) ``` ``` curl -X DELETE https://{EKODB_API_URL}/api/kv/batch/delete \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "keys": [ "session:user123", "session:user456", "session:user789" ] }' # Response: array of [key, success] tuples [ ["session:user123", true], ["session:user456", true], ["session:user789", true] ] ``` Response Format Returns an array of `[key, success]` tuples indicating which keys were successfully deleted. ## Clear All[​](#clear-all "Direct link to Clear All") Delete all key-value pairs. Destructive Operation This operation permanently deletes all KV data and cannot be undone. ``` curl -X DELETE https://{EKODB_API_URL}/api/kv/clear \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "message": "success" } ``` Direct API Only The KV clear operation is available via the REST API only. Client libraries do not include a dedicated clear method. Use the API directly for this operation. ## Find Keys[​](#find-keys "Direct link to Find Keys") Query keys with a regular-expression `pattern`. Pagination and result limits go in `query_options`. ``` POST https://{EKODB_API_URL}/api/kv/find Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "pattern": "^user_session_", "query_options": { "limit": 10, "skip": 0 } } # Response [ {"key": "user_session_123", ...}, {"key": "user_session_456", ...}, {"key": "user_session_789", ...} ] ``` Response Format Returns an array of key-value records directly, not wrapped in an object. Each record contains the key and its associated value. **Request Options:** ``` { "pattern": "^cache_.*_temp$", // Regex matched against the key "include_expired": false, // Include entries past their TTL (default: false) "query_options": { "limit": 100, // Max results "skip": 0 // Number of results to skip (pagination) } } ``` Key matching is regex-based, so the common prefix/suffix/contains cases map to: prefix `^user_`, suffix `_temp$`, and contains `session`. ## Linking KV to Documents[​](#linking-kv-to-documents "Direct link to Linking KV to Documents") Link key-value pairs to document records for additional metadata. ### Create Link[​](#create-link "Direct link to Create Link") Link a KV pair to a document. ``` POST https://{EKODB_API_URL}/api/kv/{key}/links/{collection}/{record_id} Content-Type: application/json Authorization: Bearer {YOUR_API_TOKEN} { "metadata": { "created_by": "user_123", "purpose": "analytics", "expires": "2026-12-31" } } # Response null ``` Response Format Returns `null` on success. Check HTTP status code (200) to confirm the link was created. ### Get Links[​](#get-links "Direct link to Get Links") Retrieve all document links for a key. ``` GET https://{EKODB_API_URL}/api/kv/{key}/links Authorization: Bearer {YOUR_API_TOKEN} # Response [ { "collection": "reports", "record_id": "report_456", "metadata": { "created_by": "user_123", "purpose": "analytics" } }, { "collection": "dashboards", "record_id": "dashboard_789", "metadata": { "created_by": "user_456" } } ] ``` Response Format Returns an array of link objects directly, not wrapped in an object. ### Remove Link[​](#remove-link "Direct link to Remove Link") Remove a link between a KV pair and a document. ``` DELETE https://{EKODB_API_URL}/api/kv/{key}/links/{collection}/{record_id} Authorization: Bearer {YOUR_API_TOKEN} # Response null ``` Response Format Returns `null` on success. Check HTTP status code (200) to confirm the link was removed. ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete workflow using KV store for session management: ``` #!/bin/bash # 1. Create session on login session_id=$(uuidgen) curl -X POST https://{EKODB_API_URL}/api/kv/set/session_$session_id \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "user_id": "user_123", "login_time": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'", "ip_address": "192.168.1.1" }' # 2. Link session to user document curl -X POST https://{EKODB_API_URL}/api/kv/session_$session_id/links/users/user_123 \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "metadata": { "session_type": "web", "device": "desktop" } }' # 3. Get session data curl -X GET https://{EKODB_API_URL}/api/kv/get/session_$session_id \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # 4. Get all sessions for user (using find) curl -X POST https://{EKODB_API_URL}/api/kv/find \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{ "prefix": "session_", "limit": 100 }' # 5. Cleanup session on logout curl -X DELETE https://{EKODB_API_URL}/api/kv/delete/session_$session_id \ -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Choose KV vs Documents[​](#choose-kv-vs-documents "Direct link to Choose KV vs Documents") | Use Case | Recommendation | Why | | ------------------------- | -------------- | -------------------------------- | | Simple key-value lookup | KV Store | Faster, simpler | | Complex queries/filtering | Documents | Full query capabilities | | Temporary data/cache | KV Store | Fast writes, auto-cleanup | | Structured records | Documents | Schema validation, relationships | | Session management | KV Store | Quick lookups, TTL support | | User profiles | Documents | Complex queries, joins | ### Use Prefixes for Organization[​](#use-prefixes-for-organization "Direct link to Use Prefixes for Organization") ``` # Group related keys with prefixes POST /api/kv/set/cache:users:123 POST /api/kv/set/cache:posts:456 POST /api/kv/set/session:abc123 POST /api/kv/set/session:def456 POST /api/kv/set/config:feature_flags ``` ### Batch Operations for Efficiency[​](#batch-operations-for-efficiency "Direct link to Batch Operations for Efficiency") ``` # Good: Batch get for multiple keys POST /api/kv/batch/get {"keys": ["session_1", "session_2", "session_3"]} # Avoid: Multiple individual requests GET /api/kv/get/session_1 GET /api/kv/get/session_2 GET /api/kv/get/session_3 ``` ### Link KV to Documents[​](#link-kv-to-documents "Direct link to Link KV to Documents") ``` # Store analytics in KV, link to report document POST /api/kv/set/analytics_2024_01 {"views": 1000, "clicks": 250} POST /api/kv/analytics_2024_01/links/reports/report_123 {"metadata": {"period": "monthly"}} # Query report to get linked analytics GET /api/find/reports/report_123 # Then fetch linked KV data GET /api/kv/analytics_2024_01/links ``` ## Use Cases[​](#use-cases "Direct link to Use Cases") ### Caching API Responses[​](#caching-api-responses "Direct link to Caching API Responses") ``` # Cache expensive query results POST /api/kv/set/cache:trending_posts { "posts": [...], "generated_at": "2026-01-15T10:30:00Z", "ttl": 300 } # Retrieve cached data GET /api/kv/get/cache:trending_posts ``` ### Feature Flags[​](#feature-flags "Direct link to Feature Flags") ``` # Store feature flags POST /api/kv/set/feature_flags { "new_dashboard": true, "beta_search": false, "experimental_ai": true } # Quick lookup GET /api/kv/get/feature_flags ``` ### Rate Limiting Counters[​](#rate-limiting-counters "Direct link to Rate Limiting Counters") ``` # Increment API call counter GET /api/kv/get/rate_limit:user_123 # {"count": 95, "reset_at": "2026-01-15T11:00:00Z"} # Update counter POST /api/kv/set/rate_limit:user_123 {"count": 96, "reset_at": "2026-01-15T11:00:00Z"} ``` ### Session Storage[​](#session-storage "Direct link to Session Storage") ``` # Store user session POST /api/kv/set/session:abc123 { "user_id": "user_456", "roles": ["user", "premium"], } # Validate session GET /api/kv/get/session:abc123 ``` ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Basic Operations](/basic-operations.md) - Document CRUD operations * [Batch Operations](/batch-operations.md) - Bulk document operations * [Authentication](/authentication.md) - API key and JWT authentication ## Example Code[​](#example-code "Direct link to Example Code") ### Direct HTTP/REST API Examples[​](#direct-httprest-api-examples "Direct link to Direct HTTP/REST API Examples") Raw HTTP examples demonstrating the REST API directly: * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/kv_operations.js)**: `kv_operations.js` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/kv_operations.py)**: `kv_operations.py` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/kv_operations.go)**: `kv_operations.go` * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/kv_operations.rs)**: `kv_operations.rs` ### Client Library Examples[​](#client-library-examples "Direct link to Client Library Examples") Production-ready examples using official client libraries: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_kv_operations.rs)**: `client_kv_operations.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_kv_operations.py)**: `client_kv_operations.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_kv_operations.ts)**: `client_kv_operations.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_kv_operations.go)**: `client_kv_operations.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientKvOperations.kt)**: `ClientKvOperations.kt` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_kv_operations.js)**: `client_kv_operations.js` --- # Query Patterns ekoDB automatically tracks query access patterns to identify frequently accessed records and intelligently pre-loads them into cache on database startup. New in v0.30.0 Query pattern tracking with intelligent cache warming is available in ekoDB v0.30.0+. ## Overview[​](#overview "Direct link to Overview") The **Pattern Logger** monitors which records are accessed, how often, and how recently. On database restart, it uses this information to automatically warm the cache with your most-accessed data, eliminating cold-start delays. ### Key Benefits[​](#key-benefits "Direct link to Key Benefits") * **🚀 3x Faster Queries**: Cached records served in \~109µs vs \~328µs uncached * **📊 Zero Configuration**: Fully automatic - no setup required * **💾 Microsecond Lookups**: `find_by_id` completes in a few microseconds * **🔄 Persistent**: Patterns survive database restarts * **⚡ Smart Scoring**: Recent accesses weighted higher ## How It Works[​](#how-it-works "Direct link to How It Works") ### 1. Automatic Pattern Tracking[​](#1-automatic-pattern-tracking "Direct link to 1. Automatic Pattern Tracking") Every query operation automatically logs access patterns: ``` // JavaScript/TypeScript Client await client.findById("users", userId); // Tracked: FindById await client.find("users", { limit: 100 }); // Tracked: Find await client.textSearch("products", "laptop"); // Tracked: TextSearch ``` ``` # Python Client await client.find_by_id('users', user_id) # Tracked: FindById await client.find('users', limit=100) # Tracked: Find await client.text_search('products', 'laptop') # Tracked: TextSearch ``` ``` // Go Client client.FindByID("users", userId) // Tracked: FindById client.Find("users", filters) // Tracked: Find client.TextSearch("products", "laptop") // Tracked: TextSearch ``` Vector and hybrid searches (via `client.search` / `client.Search`) are tracked the same way, as `VectorSearch` and `HybridSearch`. ### 2. Hotness Scoring[​](#2-hotness-scoring "Direct link to 2. Hotness Scoring") Each record receives a **hotness score** based on how often it's accessed and how recently. Records accessed within the last hour are weighted most heavily, with the score decaying over time. This ensures the cache reflects current workload, not stale history. **Example**: * Record A: Accessed 100 times, last access 30 minutes ago → **Very Hot** * Record B: Accessed 50 times, last access 3 days ago → **Warm** * Record C: Accessed 10 times, last access 30 days ago → **Cold** ### 3. Cache Warming on Startup[​](#3-cache-warming-on-startup "Direct link to 3. Cache Warming on Startup") When your database restarts: 1. **Load Patterns**: Read patterns from disk 2. **Calculate Scores**: Compute hotness for all records 3. **Pre-load Hot Records**: Load top N records into cache 4. **Ready to Serve**: Hot queries are instant ``` Database Restart Flow: ├─ Load pattern file (~1-2ms) ├─ Calculate hotness scores (~5-10ms) ├─ Pre-load 50 hot records into cache (~100-200ms) └─ Database ready with warm cache ✅ Query Performance (from benchmarks): Uncached query (10 records): 328µs Cached query (10 records): 109µs → 3x faster! Single find_by_id: 2.3µs ``` ## Performance Impact[​](#performance-impact "Direct link to Performance Impact") ### Cache Performance[​](#cache-performance "Direct link to Cache Performance") Benchmarks compare query performance when records are pre-loaded in cache versus cold (uncached) queries. All timings are local embedded operations with no network overhead. | Query Size | Speedup | | ---------------- | --------------- | | **10 records** | **3.0x faster** | | **100 records** | **3.1x faster** | | **1000 records** | **3.6x faster** | Cache warming provides consistent 3x speedups across all query sizes. The improvement comes from avoiding disk I/O on hot paths—records are served directly from memory rather than reading from storage on each access. ### Single Record Lookups[​](#single-record-lookups "Direct link to Single Record Lookups") Single record operations show ekoDB's raw embedded performance. These are point lookups without any query parsing or filtering overhead. | Operation | Speedup | | ---------------- | ------------------------- | | **find\_by\_id** | Microsecond (\~2µs) | | **kv\_get** | Sub-microsecond (\~200ns) | | **kv\_exists** | Sub-microsecond (\~100ns) | These numbers reflect the advantage of an embedded database: no serialization, no network round-trips, no connection pooling. The `kv_exists` check at \~100ns is essentially a hash table lookup, while `find_by_id` includes document parsing overhead. For complete benchmark data including comparisons to other embedded databases, see [Performance Benchmarks](/reference/performance.md). Performance Trade-off Pattern logging adds minimal overhead but significantly improves cached query performance. See the [benchmark table above](#cache-performance) for details. ## Configuration[​](#configuration "Direct link to Configuration") ### Automatic Configuration[​](#automatic-configuration "Direct link to Automatic Configuration") Pattern tracking is **enabled by default** with automatic configuration: * Buffer size auto-scales with system resources * Flush frequency optimized for your hardware * Hot record limit adapts to available memory **No configuration needed!** Just use ekoDB normally. ### Disabling Pattern Tracking[​](#disabling-pattern-tracking "Direct link to Disabling Pattern Tracking") Pattern logging adds a small per-read cost (a sampled access-pattern record, roughly 1 in every 16 reads). For workloads where cache warming has no benefit — a fully in-memory working set, or throughput-sensitive read-heavy workloads — disable it at runtime via `PUT /api/config`: ``` { "read_features": { "pattern_logging": false } } ``` With it off, reads skip access-pattern logging entirely (no cache-warming data is collected). See [Configuration → Read Features](/configuration.md#read-features). ### Manual Tuning (Advanced)[​](#manual-tuning-advanced "Direct link to Manual Tuning (Advanced)") For advanced use cases, you can tune behavior via environment variables: ``` # Adjust cleanup interval (affects buffer size) export CLEANUP_INTERVAL_SECONDS=300 # Default: varies by tier # Buffer size calculation: # buffer = cleanup_interval * 10 # 300s → 3000 entries buffered before flush ``` ## Use Cases[​](#use-cases "Direct link to Use Cases") ### E-Commerce Platform[​](#e-commerce-platform "Direct link to E-Commerce Platform") ``` // Scenario: Product catalog with 1M products // Reality: 1000 products generate 80% of traffic // Before (Without Cache Warming): // - Database restart // - First queries: ~328µs each (uncached) // - User experience: Slightly slower initial loads // After (With Cache Warming): // - Database restart // - Pattern logger pre-loads 1000 hot products // - First queries served from cache immediately // - User experience: Fast from the start ✅ ``` ### Multi-Tenant SaaS[​](#multi-tenant-saas "Direct link to Multi-Tenant SaaS") ``` // Scenario: 10,000 tenants // Reality: 100 active tenants generate 90% of queries // Cache warming automatically identifies active tenants // Pre-loads their data on startup // Active tenants get instant response times // Inactive tenants use normal lazy-loading ``` ### Content Platform[​](#content-platform "Direct link to Content Platform") ``` // Scenario: News site with trending articles // Patterns adapt automatically: // - Morning: Breaking news articles hot // - Afternoon: Opinion pieces hot // - Evening: Sports content hot // Cache warming reflects current trends // No manual cache management needed ``` ## Monitoring[​](#monitoring "Direct link to Monitoring") ### Log Output[​](#log-output "Direct link to Log Output") Pattern Logger provides detailed logs on startup and during operation: ``` [INFO] PatternLogger: Buffer size set to 3000 entries (based on cleanup_interval: 300s) [INFO] Loaded 15234 pattern entries [INFO] Warming cache from query patterns... [INFO] Cache warming complete: 50 hot records loaded in 125ms ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### 1. Let It Run Automatically[​](#1-let-it-run-automatically "Direct link to 1. Let It Run Automatically") Don't disable pattern logging unless you have a specific reason. The benefits far outweigh the minimal overhead. ### 2. Monitor Hot Records[​](#2-monitor-hot-records "Direct link to 2. Monitor Hot Records") Periodically review which records are hot: * Optimize frequently accessed data * Add indexes for hot queries * Pre-compute expensive operations ### 3. Clean Old Patterns[​](#3-clean-old-patterns "Direct link to 3. Clean Old Patterns") Patterns are cleaned automatically based on the `cleanup_interval` setting. Patterns naturally age out over time, and a database restart will rebuild them from fresh access data. ### 4. Combine with Indexes[​](#4-combine-with-indexes "Direct link to 4. Combine with Indexes") Pattern tracking identifies hot **records**, indexes optimize hot **queries**: ``` // Pattern tracking: Pre-loads frequently accessed users // Index: Makes user lookups by email fast await client.createIndex("users", ["email"]); // Both work together for optimal performance ``` ## Storage & Maintenance[​](#storage--maintenance "Direct link to Storage & Maintenance") ### Pattern Storage[​](#pattern-storage "Direct link to Pattern Storage") Patterns are stored in an append-only log managed automatically by ekoDB. Each entry records the collection, record, operation type, and timestamp. ### Storage Requirements[​](#storage-requirements "Direct link to Storage Requirements") Pattern logging is lightweight — approximately 80 bytes per entry: | Entries | Disk Usage | | ---------- | ---------- | | 1,000,000 | \~80 MB | | 10,000,000 | \~800 MB | ### Cleanup Strategy[​](#cleanup-strategy "Direct link to Cleanup Strategy") Recommended cleanup schedule: | Data Size | Cleanup Frequency | Keep Days | | --------- | ----------------- | --------- | | < 1GB | Monthly | 30 days | | 1-10GB | Weekly | 14 days | | > 10GB | Daily | 7 days | ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Patterns Not Loading[​](#patterns-not-loading "Direct link to Patterns Not Loading") **Check logs for errors**: ``` [WARN] Failed to parse pattern entry: Invalid timestamp ``` **Solution**: If pattern data is corrupted, restart the database — patterns will rebuild automatically from new access data. ### High Memory Usage[​](#high-memory-usage "Direct link to High Memory Usage") **Check buffer size**: ``` # In database logs [INFO] PatternLogger: Buffer size set to 10000 entries ``` **Solution**: Reduce cleanup interval to decrease buffer: ``` export CLEANUP_INTERVAL_SECONDS=60 # Smaller buffer ``` ### Slow Startup[​](#slow-startup "Direct link to Slow Startup") **Check hot record count**: ``` # In database logs [INFO] Cache warming complete: 500 hot records loaded in 2500ms ``` **Solution**: Too many hot records being loaded. This is automatically tuned, but if startup is critical: * Patterns naturally age out * Clean old patterns more frequently * Consider if 500 hot records is appropriate for your use case ## Integration with Other Features[​](#integration-with-other-features "Direct link to Integration with Other Features") ### With Multi-Region (Ripple)[​](#with-multi-region-ripple "Direct link to With Multi-Region (Ripple)") Each region maintains its own patterns: * Region A: Tracks Region A's access patterns * Region B: Tracks Region B's access patterns * Benefit: Each region optimizes for its users ### With File Pool[​](#with-file-pool "Direct link to With File Pool") Uses global file descriptor management: * Pattern file = 1 FD * Respects system limits * Automatic retry/backoff ### With Disk Cache[​](#with-disk-cache "Direct link to With Disk Cache") Works seamlessly together: * Hot records → Memory cache (instant) * Warm records → Disk cache (fast) * Cold records → Database (acceptable) ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [System Administration](/system-administration.md) - Admin endpoints and monitoring * [Performance Benchmarks](/reference/performance.md) - Performance data and tuning * [Indexes](/indexes.md) - Create indexes for common queries * [White Paper](/white-paper.md) - Technical architecture details ## Summary[​](#summary "Direct link to Summary") Cache warming eliminates cold-start delays automatically. ekoDB learns which data matters, pre-loads it on restart, and adapts as access patterns change — no configuration required. --- # Query Expressions ekoDB's query language uses a **structured JSON format with tagged unions** instead of SQL strings or MongoDB-style query documents. This technical reference explains the design rationale, expression structure, and complete operator specification. ## Design Rationale[​](#design-rationale "Direct link to Design Rationale") ### Why Not SQL?[​](#why-not-sql "Direct link to Why Not SQL?") SQL was designed for relational databases with fixed schemas and table joins. ekoDB is a document database optimized for: 1. **Schema flexibility** - Documents can have different structures in the same collection 2. **Nested data** - No joins needed when related data is embedded 3. **HTTP-first API** - JSON over REST, not a separate query protocol 4. **Type-safe query builders** - Statically typed in Rust, TypeScript, Kotlin, Go; runtime checked in Python **SQL's limitations for document databases:** * String concatenation creates injection vulnerabilities * No type checking until runtime * Impedance mismatch between tabular results and nested documents * Query parsers are complex and error-prone ### Why Not MongoDB Query Language?[​](#why-not-mongodb-query-language "Direct link to Why Not MongoDB Query Language?") MongoDB uses a JSON-like query format, but with significant limitations: ``` // MongoDB - operators are magic strings at runtime { age: { $gte: 18, $lte: 65 }, status: "active" } ``` **Problems with this approach:** 1. **No type discrimination** - Cannot distinguish between a filter object and a data object 2. **Operator collision** - Field names starting with `$` are reserved 3. **Ambiguous semantics** - Is `{a: 1, b: 2}` an AND or just two fields? 4. **Harder to parse** - Must scan all keys to detect operators ### ekoDB's Tagged Union Approach[​](#ekodbs-tagged-union-approach "Direct link to ekoDB's Tagged Union Approach") ``` { "type": "Condition", "content": { "field": "age", "operator": "Gte", "value": 18 } } ``` **Benefits:** 1. **Unambiguous parsing** - `type` discriminator makes structure explicit 2. **Type-safe composition** - Logical operators nest cleanly with proper typing 3. **Minimal reserved field names** - Only `id` and `ttl` are reserved (vs MongoDB's `$` prefix for all operators) 4. **Better error messages** - Invalid queries caught during JSON schema validation 5. **Optimizable** - Query planner can analyze structure before execution 6. **Query intent is obvious** - Looking at the JSON, you immediately know what's being filtered. Compare: * ekoDB: `{"type": "Condition", "content": {"field": "age", "operator": "Gte", "value": 18}}` * MongoDB: `{"age": {"$gte": 18}}` (is `$gte` an operator or a field name?) 7. **Fine-grained propagation control** - Query options include `bypass_cache` and `bypass_ripple` for direct control over the data propagation waterfall. Most databases don't expose this level of control at the query level. **Trade-offs:** * **More verbose** - Requires `type` and `content` wrappers * **Unfamiliar syntax** - Not SQL or MongoDB query language * **More JSON payload** - Slightly larger wire format (mitigated by HTTP compression) **Why we chose verbosity:** Explicit query intent and type safety outweigh brevity. While the syntax may be unfamiliar initially, it's far more obvious what you're querying for—there's no ambiguity between data and query operators. Client libraries abstract the verbosity with fluent APIs like `QueryBuilder().gte("age", 18)`. ### Comparison to Other NoSQL Databases[​](#comparison-to-other-nosql-databases "Direct link to Comparison to Other NoSQL Databases") | Database | Query Format | Type Safety | Composability | Reserved Fields | | --------- | --------------------- | ------------------- | ------------------- | -------------------- | | **ekoDB** | Tagged unions | ✅ Strong | ✅ Explicit nesting | ⚠️ `id`, `ttl` only | | MongoDB | Dollar operators | ❌ Runtime only | ⚠️ Implicit | ✅ All `$` operators | | CouchDB | Map-Reduce | ❌ None | ⚠️ JavaScript | N/A | | DynamoDB | Condition expressions | ⚠️ Partial | ❌ String-based | ✅ Reserved words | | Firestore | Method chaining | ✅ Client-side only | ✅ Good | ⚠️ Minimal | ekoDB's approach is most similar to Firestore's, but with **explicit JSON representation** that works identically across all clients and the REST API. *** ## Expression Structure[​](#expression-structure "Direct link to Expression Structure") ekoDB uses a **tagged union** (also called discriminated union or sum type) for query expressions. The `type` field discriminates between different expression variants at parse time. ``` { "type": "Condition" | "Logical", "content": { ... } } ``` ### Type Discrimination[​](#type-discrimination "Direct link to Type Discrimination") The parser uses the `type` field to determine how to interpret `content`: ``` // Rust deserialization (simplified) enum QueryExpression { Condition { field: String, operator: Operator, value: Value }, Logical { operator: LogicOp, expressions: Vec } } ``` This enables: * **Compile-time type checking** in typed languages (Rust, TypeScript, Kotlin, Go) * **Validation before execution** - malformed queries rejected immediately * **Query plan optimization** - structure is known before accessing data ### Two Expression Types[​](#two-expression-types "Direct link to Two Expression Types") 1. **Condition** - Single field comparison (leaf node in expression tree) 2. **Logical** - Combine multiple expressions (branch node in expression tree) Expressions form a **tree structure** where: * Leaf nodes are `Condition` expressions * Branch nodes are `Logical` expressions containing child expressions * Tree depth is unlimited (for complex business logic) #### Visual Example: Query Expression Tree[​](#visual-example-query-expression-tree "Direct link to Visual Example: Query Expression Tree") Here's how the query `age >= 18 AND (status = "active" OR role = "admin")` is represented as a tree: **Legend:** * **Logical** expressions (AND, OR, NOT) = branch nodes that combine child expressions * **Condition** expressions (age >= 18, status = "active") = leaf nodes with field comparisons The parser evaluates this tree recursively: 1. Start at root AND node 2. Evaluate left child: `age >= 18` (Condition) 3. Evaluate right child: OR node * Evaluate `status = "active"` (Condition) * Evaluate `role = "admin"` (Condition) 4. Combine results with AND logic *** ## Comparison Operators[​](#comparison-operators "Direct link to Comparison Operators") All comparison operators follow this structure: ``` { "type": "Condition", "content": { "field": "field_name", "operator": "OperatorName", "value": "comparison_value" } } ``` ### Available Operators[​](#available-operators "Direct link to Available Operators") | Operator | Description | Aliases | Example Value | | -------- | --------------------- | ----------------------- | ------------------------- | | `Eq` | Equals | `Equal`, `Equals` | `"active"`, `25` | | `Ne` | Not equals | `NotEqual`, `NotEquals` | `"deleted"`, `0` | | `Gt` | Greater than | `GreaterThan` | `18`, `100.5` | | `Gte` | Greater than or equal | `GreaterThanOrEqual` | `18`, `0` | | `Lt` | Less than | `LessThan` | `65`, `1000` | | `Lte` | Less than or equal | `LessThanOrEqual` | `100`, `999.99` | | `In` | Value in array | - | `["active", "pending"]` | | `NotIn` | Value not in array | - | `["deleted", "archived"]` | *** ## String Operators[​](#string-operators "Direct link to String Operators") Special operators for string/text matching: | Operator | Description | Case Sensitive | Example | | ------------ | --------------- | -------------- | -------------- | | `Contains` | Substring match | Yes | `"@gmail.com"` | | `StartsWith` | Prefix match | Yes | `"Dr. "` | | `EndsWith` | Suffix match | Yes | `".com"` | String Matching All string operators are **case-sensitive**. To perform case-insensitive searches, normalize your data (e.g., lowercase) on insert. *** ## Logical Operators[​](#logical-operators "Direct link to Logical Operators") Combine multiple expressions with logical operators: Empty logical filters are rejected A `Logical` expression must contain at least one child in `expressions`. An empty array is rejected with HTTP 400 — an empty `And` would otherwise match every record and an empty `Or` none, which is almost always a malformed filter. To match all records, omit the filter entirely. ### AND - All conditions must match[​](#and---all-conditions-must-match "Direct link to AND - All conditions must match") ``` { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gte", "value": 18 } }, { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } } ] } } ``` ### OR - At least one condition must match[​](#or---at-least-one-condition-must-match "Direct link to OR - At least one condition must match") ``` { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "admin" } }, { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "moderator" } } ] } } ``` ### NOT - Negate a condition[​](#not---negate-a-condition "Direct link to NOT - Negate a condition") ``` { "type": "Logical", "content": { "operator": "Not", "expressions": [ { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "deleted" } } ] } } ``` ### Complex Nested Logic[​](#complex-nested-logic "Direct link to Complex Nested Logic") You can nest logical operators for complex queries: ``` { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gte", "value": 18 } }, { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "country", "operator": "Eq", "value": "US" } }, { "type": "Condition", "content": { "field": "country", "operator": "Eq", "value": "CA" } } ] } } ] } } ``` This finds records where: `age >= 18 AND (country = "US" OR country = "CA")` *** ## Query Options[​](#query-options "Direct link to Query Options") Beyond filters, ekoDB provides fine-grained control over sorting, pagination, and **data propagation** at the query level: ``` { "filter": { ... }, "sort": [ { "field": "created_at", "ascending": false }, { "field": "name", "ascending": true } ], "limit": 10, "skip": 20, "bypass_cache": false, "bypass_ripple": true } ``` ### Options Reference[​](#options-reference "Direct link to Options Reference") | Option | Type | Description | | --------------- | --------------- | --------------------------------------------------------- | | `filter` | QueryExpression | Filter condition (see above) | | `sort` | Array | Sort by one or more fields | | `limit` | Integer | Max results to return | | `skip` | Integer | Results to skip (pagination) | | `bypass_cache` | Boolean | Skip cache lookup (read directly from storage) | | `bypass_ripple` | Boolean | Don't propagate writes to ripple peers (for bulk imports) | ### Propagation Control (Unique to ekoDB)[​](#propagation-control-unique-to-ekodb "Direct link to Propagation Control (Unique to ekoDB)") Unlike most databases, ekoDB exposes **direct control over the data propagation waterfall** at the query level: **`bypass_cache`** - Control read path: * `false` (default): Check cache first, then storage if cache miss * `true`: Skip cache entirely, read directly from durable storage * **Use case**: When you need guaranteed fresh data and can tolerate slightly higher latency **`bypass_ripple`** - Control write replication: * `false` (default): Writes propagate to all configured ripple peers * `true`: Writes stay local, skip replication * **Use case**: Bulk data imports, maintenance operations, temporary data Most databases treat caching and replication as infrastructure concerns outside the query language. ekoDB makes these **first-class query options** for maximum control. *** ## Usage Examples[​](#usage-examples "Direct link to Usage Examples") For practical examples of using query expressions: * **Direct API** - See [Basic Operations - Query Examples](/basic-operations.md#query-examples) * **Client Libraries** - See [Querying with Filters](/client-libraries/querying-with-filters.md) *** ## Common Query Patterns[​](#common-query-patterns "Direct link to Common Query Patterns") ### Find All Active Records[​](#find-all-active-records "Direct link to Find All Active Records") ``` { "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } } } ``` ### Range Query (Between Values)[​](#range-query-between-values "Direct link to Range Query (Between Values)") ``` { "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "price", "operator": "Gte", "value": 10.0 } }, { "type": "Condition", "content": { "field": "price", "operator": "Lte", "value": 100.0 } } ] } } } ``` ### Exclude Values (NOT IN)[​](#exclude-values-not-in "Direct link to Exclude Values (NOT IN)") ``` { "filter": { "type": "Condition", "content": { "field": "status", "operator": "NotIn", "value": ["deleted", "archived", "spam"] } } } ``` ### Search Multiple Fields (OR)[​](#search-multiple-fields-or "Direct link to Search Multiple Fields (OR)") ``` { "filter": { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "title", "operator": "Contains", "value": "urgent" } }, { "type": "Condition", "content": { "field": "description", "operator": "Contains", "value": "urgent" } } ] } } } ``` ### Complex Business Logic[​](#complex-business-logic "Direct link to Complex Business Logic") Find premium users who are either: * Over 30 years old, OR * Have made more than 10 purchases ``` { "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "tier", "operator": "Eq", "value": "premium" } }, { "type": "Logical", "content": { "operator": "Or", "expressions": [ { "type": "Condition", "content": { "field": "age", "operator": "Gt", "value": 30 } }, { "type": "Condition", "content": { "field": "purchase_count", "operator": "Gt", "value": 10 } } ] } } ] } } } ``` *** ## Performance Tips[​](#performance-tips "Direct link to Performance Tips") ### Use Indexes[​](#use-indexes "Direct link to Use Indexes") Create indexes on fields you query frequently: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::QueryBuilder; // When a field has an index, ekoDB uses it automatically — no hint required. let query = QueryBuilder::new().eq("category", "electronics").build(); let results = client.find("products", query, None).await?; ``` ``` from ekodb_client import QueryBuilder # When a field has an index, ekoDB uses it automatically; no hint required. query = QueryBuilder().eq("category", "electronics").build() results = await client.find("products", query) ``` ``` import { QueryBuilder } from "@ekodb/ekodb-client"; // When a field has an index, ekoDB uses it automatically; no hint required. const query = new QueryBuilder().eq("category", "electronics").build(); const results = await client.find("products", query); ``` ``` const { QueryBuilder } = require("@ekodb/ekodb-client"); // When a field has an index, ekoDB uses it automatically; no hint required. const query = new QueryBuilder().eq("category", "electronics").build(); const results = await client.find("products", query); ``` ``` // When a field has an index, ekoDB uses it automatically; no hint required. val query = QueryBuilder().eq("category", "electronics").build() val results = client.find("products", query) ``` ``` // When a field has an index, ekoDB uses it automatically; no hint required. query := ekodb.NewQueryBuilder().Eq("category", "electronics").Build() results, err := client.Find("products", query) ``` ``` curl -X POST https://{EKODB_API_URL}/api/indexes/query/users \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "users_status_index", "field": "status", "index_type": "hash" }' ``` See [Indexes Documentation](/indexes.md) for more details. ### Limit Results[​](#limit-results "Direct link to Limit Results") Always use `limit` to prevent loading excessive data: ``` { "filter": { ... }, "limit": 100 } ``` ### Bypass Cache for Fresh Data[​](#bypass-cache-for-fresh-data "Direct link to Bypass Cache for Fresh Data") When you need the absolute latest data: ``` { "filter": { ... }, "bypass_cache": true } ``` ### Use Specific Fields[​](#use-specific-fields "Direct link to Use Specific Fields") Only select fields you need (reduces bandwidth): ``` { "filter": { ... }, "select_fields": ["id", "name", "email"] } ``` Projection works on both transports. The POST body uses a JSON array, either to include fields (`select_fields`) or to drop them (`exclude_fields`): ``` { "filter": { ... }, "exclude_fields": ["secret"] } ``` GET query strings use a comma-joined value: ``` GET /api/find/users?select_fields=id,name,email GET /api/find/users?exclude_fields=secret ``` An empty value (`?select_fields=`) means no projection. The primary key (`id`) is always returned. *** ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### 400 Bad Request - Invalid Filter Format[​](#400-bad-request---invalid-filter-format "Direct link to 400 Bad Request - Invalid Filter Format") ❌ **Wrong:** ``` { "filter": { "field": "status", "operator": "Eq", "value": "active" } } ``` ✅ **Correct:** ``` { "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "active" } } } ``` **Missing `type` and `content` wrapper!** ### 400 Bad Request - Unknown Operator[​](#400-bad-request---unknown-operator "Direct link to 400 Bad Request - Unknown Operator") ❌ **Wrong:** ``` { "operator": "equals" // lowercase } ``` ✅ **Correct:** ``` { "operator": "Eq" // PascalCase } ``` **Operators are case-sensitive and use PascalCase.** ### Empty Results When Data Exists[​](#empty-results-when-data-exists "Direct link to Empty Results When Data Exists") * Check if field names are correct (case-sensitive) * Verify value types match (string vs number) * For string matching, remember it's case-sensitive *** ## Next Steps[​](#next-steps "Direct link to Next Steps") * **[Basic Operations](/basic-operations.md)** - Complete CRUD API reference * **[Indexes](/indexes.md)** - Speed up your queries * **[Client Libraries](/client-libraries/.md)** - Type-safe query building * **[Batch Operations](/batch-operations.md)** - Query multiple records efficiently ## Need Help?[​](#need-help "Direct link to Need Help?") * 📖 **Client Library Examples**: [GitHub Examples](https://github.com/ekoDB/ekodb-client/tree/main/examples) (includes both client library and direct HTTP examples) * 🐛 **Issues**: [app.ekodb.io/support](https://app.ekodb.io/support) * 💬 **Support**: --- # Quick Start Guide Two ways to stand up a backend on ekoDB: **describe it to an agent** (three steps, no API code) or **build it by hand** (the dozen steps below). New here? Start with the [Introduction](/introduction.md). ## Build your backend with an agent (recommended)[​](#build-your-backend-with-an-agent-recommended "Direct link to Build your backend with an agent (recommended)") You do not have to hand-write the API below. Deploy an **AI Agent** next to your database, describe the backend you want, and it builds the collections and server-side functions for you. Your app calls those functions with one line. Then you go build your frontend. | | 🤖 With an agent | 🛠️ By hand | | ------------------- | ---------------------------------- | ---------------------------------------------------- | | Setup | deploy a database **and** an agent | deploy a database | | Build the backend | describe it in chat | write insert / get / query / update / batch yourself | | Add a feature later | ask for it | write another endpoint | | Your job | the frontend | the frontend **and** the whole API | ### Three steps[​](#three-steps "Direct link to Three steps") 1. **Deploy a database and an AI Agent.** Two deployments at [app.ekodb.io](https://app.ekodb.io), then connect the agent to the database. ([Build Your First AI Agent](/build-your-first-agent.md) has the click-by-click.) 2. **Describe your backend** in the agent's chat: > Build a user-management backend: a `users` collection with name, email, age, and role, plus a function `list_developers` that returns everyone whose role is developer. The agent creates the collection and the stored function directly in your database. 3. **Call it from your app** with the SDK and your database's API key: ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "YOUR_API_KEY" }); await client.init(); const developers = await client.callFunction("list_developers"); ``` That is the backend done. Go build your UI; ekoDB runs the rest. ### It does not stop at building[​](#it-does-not-stop-at-building "Direct link to It does not stop at building") Once your backend is live, keep talking to the agent to run it: * **Read the logs** and surface errors * **Diagnose a bug** by tracing it through the server logs to the cause * **Test a function** by calling it and checking the result * **Add or fix an index** when a query is slow * **Scale it**: spin up new deployments, convert a database to a multi-node cluster, tune node configs, and take backups * **Schedule jobs, evolve schemas, and add features**, all by asking From the first build through production scale, the agent runs the backend so you can stay on the frontend. Want raw control, or to see what the agent generates? The rest of this guide is the **by-hand path**: deploy a database, authenticate, and write each operation (insert, query, update) yourself. ## Step 1: Deploy Your Database (2 minutes)[​](#step-1-deploy-your-database-2-minutes "Direct link to Step 1: Deploy Your Database (2 minutes)") 1. **Sign up** at [app.ekodb.io](https://app.ekodb.io) 2. **Click "Create Deployment"** at the top of the dashboard page: [app.ekodb.io/dashboard](https://app.ekodb.io/dashboard) 3. **Choose the Database module** (the default). The other option, **AI Agent**, is covered in [Build Your First AI Agent](/build-your-first-agent.md). 4. **Select your region** (choose the region closest to your users) 5. **Choose a machine tier** (start with the free tier for testing) 6. **Set your subdomain and environment**: * Environment Type: `development` * Subdomain: `my-first-db` (or any unique name) 7. **Click Create Deployment** Your database will be ready in a few minutes. You'll receive an email once it's live. ## Step 2: Your Database URL[​](#step-2-your-database-url "Direct link to Step 2: Your Database URL") Once properly up and running, your deployment's status will show as "Running". Your database URL will be: ``` https://my-first-db.development.{provider}.ekodb.net ``` Replace `{provider}` with your selected cloud provider (ex. "google"). Viewing Your Deployments View all of your deployments at [app.ekodb.io/deployments](https://app.ekodb.io/deployments). Click on your deployment to navigate to that deployment's information page. That deployment's database URL can be found at the top under the subdomain (shown with "/api"). Here you can also find this deployment's API keys. You will need the Admin API key for the next step. To copy the Admin API key: Select your deployment → "Keys" tab → Click "Load Keys" → Copy the Admin API key ## Step 3: Authenticate (30 seconds)[​](#step-3-authenticate-30-seconds "Direct link to Step 3: Authenticate (30 seconds)") You authenticate with your **API key**. A client library takes the key and manages tokens for you, so you can skip ahead. For **raw HTTP**, the API key is not sent directly; you first exchange it for a short-lived access token: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("YOUR_API_KEY") .build()?; // Token is automatically managed by the client ``` ``` from ekodb_client import Client client = Client.new( 'https://my-first-db.development.google.ekodb.net', 'YOUR_API_KEY' ) # Token is automatically managed by the client ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "YOUR_API_KEY", }); await client.init(); // Token is automatically managed by the client ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "YOUR_API_KEY", }); await client.init(); // Token is automatically managed by the client ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://my-first-db.development.google.ekodb.net") .apiKey("YOUR_API_KEY") .build() // Token is automatically managed by the client ``` ``` import ekodb "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://my-first-db.development.google.ekodb.net", "YOUR_API_KEY", ) // Token is automatically managed by the client ``` ``` curl -X POST https://my-first-db.development.{provider}.ekodb.net/api/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key": "YOUR_API_KEY"}' ``` **Response:** ``` { "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." } ``` Save Your Token Save this token for subsequent requests. Tokens are valid for a limited time (default: 1 hour). Client libraries handle token refresh automatically. ## Step 4: Insert Your First Record (1 minute)[​](#step-4-insert-your-first-record-1-minute "Direct link to Step 4: Insert Your First Record (1 minute)") Let's create a user record: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::Record; let mut record = Record::new(); record.insert("name", "Alice"); record.insert("email", "alice@example.com"); record.insert("age", 30); record.insert("role", "developer"); let result = client.insert("users", record, None).await?; println!("Record ID: {}", result.get_string("id").unwrap_or_default()); ``` ``` result = await client.insert('users', { 'name': 'Alice', 'email': 'alice@example.com', 'age': 30, 'role': 'developer' }) print('Record ID:', result['id']) ``` ``` const result = await client.insert("users", { name: "Alice", email: "alice@example.com", age: 30, role: "developer", }); console.log("Record ID:", result.id); ``` ``` const result = await client.insert("users", { name: "Alice", email: "alice@example.com", age: 30, role: "developer", }); console.log("Record ID:", result.id); ``` ``` val result = client.insert("users", mapOf( "name" to "Alice", "email" to "alice@example.com", "age" to 30, "role" to "developer" )) println("Record ID: ${result.id}") ``` ``` result, err := client.Insert("users", map[string]interface{}{ "name": "Alice", "email": "alice@example.com", "age": 30, "role": "developer", }) if err != nil { log.Fatal(err) } fmt.Println("Record ID:", result.ID) ``` ``` curl -X POST https://my-first-db.development.{provider}.ekodb.net/api/insert/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -d '{ "name": "Alice", "email": "alice@example.com", "age": 30, "role": "developer" }' ``` **Response:** ``` { "id": "encrypted_record_id_here" } ``` Important The `id` returned is an **encrypted record ID**. Save this ID - you'll need it to fetch, update, or delete this specific record. ## Step 5: Get a Record (30 seconds)[​](#step-5-get-a-record-30-seconds "Direct link to Step 5: Get a Record (30 seconds)") Retrieve the record you just created: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let record = client.get("users", &record_id).await?; println!("{:?}", record); ``` ``` record = client.get('users', record_id) print(record) ``` ``` const record = await client.get("users", recordId); console.log(record); ``` ``` const record = await client.get("users", recordId); console.log(record); ``` ``` val record = client.get("users", recordId) println(record) ``` ``` record, err := client.Get("users", recordID) if err != nil { log.Fatal(err) } fmt.Println(record) ``` ``` curl -X GET https://my-first-db.development.{provider}.ekodb.net/api/find/users/{record_id} \ -H "Authorization: Bearer YOUR_API_TOKEN" ``` Replace `{record_id}` with the `id` from the insert response. **Response (Default - Typed):** ``` { "id": { "type": "String", "value": "encrypted_record_id_here" }, "name": { "type": "String", "value": "Alice" }, "email": { "type": "String", "value": "alice@example.com" }, "age": { "type": "Integer", "value": 30 }, "role": { "type": "String", "value": "developer" } } ``` Typed Responses By default, ekoDB returns **typed responses** with type metadata. This provides type safety and explicit type information. **Want simpler JSON?** You can switch to non-typed responses: * **Via App**: [app.ekodb.io/deployments](https://app.ekodb.io/deployments) → Select your deployment → Configurations → Edit → Toggle "Use Typed Values" * **Via API**: `PUT /api/config` with `{"use_typed_values": false}` **Non-typed response example:** ``` { "id": "encrypted_record_id_here", "name": "Alice", "email": "alice@example.com", "age": 30, "role": "developer" } ``` ## Step 6: Query with Filter (30 seconds)[​](#step-6-query-with-filter-30-seconds "Direct link to Step 6: Query with Filter (30 seconds)") You can also query records using filters instead of IDs: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::QueryBuilder; let query = QueryBuilder::new().eq("name", "Alice").build(); let results = client.find("users", query, None).await?; println!("{:?}", results); ``` ``` from ekodb_client import QueryBuilder query = QueryBuilder().eq("name", "Alice").build() results = await client.find("users", query) print(results) ``` ``` import { QueryBuilder } from "@ekodb/ekodb-client"; const query = new QueryBuilder().eq("name", "Alice").build(); const results = await client.find("users", query); console.log(results); ``` ``` const { QueryBuilder } = require("@ekodb/ekodb-client"); const query = new QueryBuilder().eq("name", "Alice").build(); const results = await client.find("users", query); console.log(results); ``` ``` val query = QueryBuilder().eq("name", "Alice").build() val results = client.find("users", query) println(results) ``` ``` query := ekodb.NewQueryBuilder().Eq("name", "Alice").Build() results, err := client.Find("users", query) if err != nil { log.Fatal(err) } fmt.Println(results) ``` ``` curl -X POST https://my-first-db.development.{provider}.ekodb.net/api/find/users \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -d '{ "filter": { "type": "Condition", "content": { "field": "name", "operator": "Eq", "value": "Alice" } } }' ``` **Response (Typed):** ``` [ { "id": { "type": "String", "value": "encrypted_record_id_here" }, "name": { "type": "String", "value": "Alice" }, "email": { "type": "String", "value": "alice@example.com" }, "age": { "type": "Integer", "value": 30 }, "role": { "type": "String", "value": "developer" } } ] ``` ## Step 7: Update a Record (30 seconds)[​](#step-7-update-a-record-30-seconds "Direct link to Step 7: Update a Record (30 seconds)") Update Alice's role using the encrypted ID: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.update("users", &record_id, serde_json::json!({ "role": "senior developer" })).await?; ``` ``` await client.update('users', record_id, { 'role': 'senior developer' }) ``` ``` await client.update("users", recordId, { role: "senior developer", }); ``` ``` await client.update("users", recordId, { role: "senior developer", }); ``` ``` client.update("users", recordId, mapOf( "role" to "senior developer" )) ``` ``` _, err := client.Update("users", recordID, map[string]interface{}{ "role": "senior developer", }) if err != nil { log.Fatal(err) } ``` ``` curl -X PUT https://my-first-db.development.{provider}.ekodb.net/api/update/users/{record_id} \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -d '{ "role": "senior developer" }' ``` Using Record IDs Replace `{record_id}` with the actual `id` from your insert response. This is the encrypted record ID that ekoDB uses to identify specific records. ## Step 8: Use a Client Library (Recommended)[​](#step-8-use-a-client-library-recommended "Direct link to Step 8: Use a Client Library (Recommended)") While direct API calls work great, we recommend using our official client libraries for a better developer experience: No Token Management Required Client libraries automatically handle token generation and refresh for you. Simply provide your API key once, and the client manages authentication behind the scenes! * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` cargo add ekodb_client # Optional: Add tokio if using #[tokio::main] in your code # cargo add tokio --features macros ``` ``` use ekodb_client::{Client, QueryBuilder, Record}; #[tokio::main] async fn main() -> Result<(), Box> { let client = Client::builder() .base_url("https://my-first-db.development.{provider}.ekodb.net") .api_key("YOUR_API_KEY") .build()?; // Insert let mut record = Record::new(); record.insert("name", "Bob"); record.insert("email", "bob@example.com"); record.insert("age", 25); record.insert("role", "designer"); let user = client.insert("users", record, None).await?; println!("Inserted: {}", user.get_string("id").unwrap_or_default()); // Query let query = QueryBuilder::new().eq("role", "designer").build(); let users = client.find("users", query, None).await?; println!("{:?}", users); Ok(()) } ``` ``` pip install ekodb-client ``` ``` from ekodb_client import Client client = Client.new( "https://my-first-db.development.{provider}.ekodb.net", "YOUR_API_KEY" ) # Insert user = await client.insert("users", { "name": "Bob", "email": "bob@example.com", "age": 25, "role": "designer" }) # Query users = await client.find("users", { "filter": { "type": "Condition", "content": { "field": "role", "operator": "Eq", "value": "designer" } } }) print(users) ``` ``` npm install @ekodb/ekodb-client ``` ``` import { EkoDBClient, QueryBuilder } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.{provider}.ekodb.net", apiKey: "YOUR_API_KEY", }); await client.init(); // Insert const user = await client.insert("users", { name: "Bob", email: "bob@example.com", age: 25, role: "designer", }); // Query const query = new QueryBuilder().eq("role", "designer").build(); const users = await client.find("users", query); console.log(users); ``` ``` npm install @ekodb/ekodb-client ``` ``` const { EkoDBClient, QueryBuilder } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: "https://my-first-db.development.{provider}.ekodb.net", apiKey: "YOUR_API_KEY", }); await client.init(); // Insert const user = await client.insert("users", { name: "Bob", email: "bob@example.com", age: 25, role: "designer", }); // Query const query = new QueryBuilder().eq("role", "designer").build(); const users = await client.find("users", query); console.log(users); ``` ``` // Add to build.gradle.kts dependencies { implementation("io.ekodb:ekodb-client-kt:0.23.0") } ``` ``` import io.ekodb.client.EkoDBClient suspend fun main() { val client = EkoDBClient.builder() .baseUrl("https://my-first-db.development.{provider}.ekodb.net") .apiKey("YOUR_API_KEY") .build() // Insert val user = client.insert("users", mapOf( "name" to "Bob", "email" to "bob@example.com", "age" to 25, "role" to "designer" )) // Query val users = client.find("users", mapOf( "filter" to mapOf( "type" to "Condition", "content" to mapOf( "field" to "role", "operator" to "Eq", "value" to "designer" ) ) )) println(users) } ``` ``` go get github.com/ekoDB/ekodb-client-go ``` ``` package main import ( "fmt" ekodb "github.com/ekoDB/ekodb-client-go" ) func main() { client := ekodb.NewClient( "https://my-first-db.development.{provider}.ekodb.net", "YOUR_API_KEY", ) // Insert user, err := client.Insert("users", map[string]interface{}{ "name": "Bob", "email": "bob@example.com", "age": 25, "role": "designer", }) if err != nil { panic(err) } // Query query := ekodb.NewQueryBuilder().Eq("role", "designer").Build() users, err := client.Find("users", query) if err != nil { panic(err) } fmt.Println(users) } ``` ## What You Just Learned[​](#what-you-just-learned "Direct link to What You Just Learned") ✅ How to deploy an ekoDB database
✅ How to insert records
✅ How to query data
✅ How to update records
✅ How to use client libraries ## Next Steps[​](#next-steps "Direct link to Next Steps") Now that you have ekoDB running, here's what to explore next: ### 📚 Learn by Example[​](#-learn-by-example "Direct link to 📚 Learn by Example") **[GitHub Examples Repository](https://github.com/ekoDB/ekodb-client/tree/main/examples)** - 129 in all languages: * Rust, Python, TypeScript, JavaScript, Go, Kotlin * **Client library examples** (using official SDKs) and **direct HTTP examples** (raw REST API calls) * CRUD operations, search, batch operations, functions, and more * Complete working code you can copy and run ### Choose Your Path[​](#choose-your-path "Direct link to Choose Your Path") **Building a Web/Mobile App?** → [Client Libraries Guide](/client-libraries/.md) - Type-safe SDKs for your language **Using REST API Directly?** → [Basic Operations](/basic-operations.md) - Complete API reference **Need Advanced Features?** → [Batch Operations](/batch-operations.md) - High-performance bulk operations
→ [Transactions](/transactions.md) - Atomic, durable multi-document transactions
→ [Functions](/client-libraries/advanced-operations.md#functions) - Stored procedures **Building an AI Application?** → [Chat & RAG](/chat-rag.md) - Chat with your data using the database's built-in RAG
→ [Vector Search & Embeddings](/vector-search.md) - Semantic similarity search
→ [Search](/indexes.md#search-indexes) - Full-text search capabilities **Want autonomous AI?** → [Build Your First AI Agent](/build-your-first-agent.md) - Deploy an **AI Agent** that connects to this database and runs tools, goals, and scheduled tasks
→ [AI Agents](/ai-agents.md) - What agents are and how they work **Want to Understand the Architecture?** → [White Paper](/white-paper.md) - Deep dive into ekoDB's design ### Common Next Tasks[​](#common-next-tasks "Direct link to Common Next Tasks") 1. **Set up proper authentication**: [Authentication Guide](/authentication.md) 2. **Define schemas**: Add validation and constraints to your collections 3. **Add indexes**: Optimize query performance 4. **Enable WebSockets**: Real-time data updates 5. **Set up Ripple**: Multi-region replication ## Need Help?[​](#need-help "Direct link to Need Help?") * 💬 Support: [app.ekodb.io/support](https://app.ekodb.io/support) * 📧 Email: *** **Ready to build something amazing?** Let's go! 🚀 --- # Real-Time Subscriptions ekoDB supports real-time push notifications when data changes. Subscribe to a collection and receive events for every insert, update, or delete — no polling required. Two transport options: * **WebSocket** (`/api/ws`) — bidirectional, supports multiple subscriptions per connection, best for applications that also perform CRUD over the same connection * **SSE** (`/api/subscribe/{collection}`) — server-push only, one subscription per connection, simpler to integrate from browsers Both deliver the same `MutationNotification` payload and support filtered subscriptions. ## WebSocket Subscriptions[​](#websocket-subscriptions "Direct link to WebSocket Subscriptions") ### Connecting[​](#connecting "Direct link to Connecting") Connect to the WebSocket endpoint with a Bearer token: ``` wss://{EKODB_HOST}/api/ws Authorization: Bearer {TOKEN} ``` The server assigns a unique connection ID and sends a 15-second heartbeat ping to keep the connection alive across proxies and NATs. ### Subscribe to a Collection[​](#subscribe-to-a-collection "Direct link to Subscribe to a Collection") Send a JSON message to subscribe: ``` { "type": "Subscribe", "messageId": "sub-1", "payload": { "collection": "orders" } } ``` **Response:** ``` { "type": "Success", "messageId": "sub-1", "payload": { "data": { "subscribed": true, "collection": "orders", "subscription_id": "sub_abc123" } } } ``` ### Filtered Subscriptions[​](#filtered-subscriptions "Direct link to Filtered Subscriptions") Reduce noise by subscribing only to mutations where a specific field matches a value: ``` { "type": "Subscribe", "messageId": "sub-2", "payload": { "collection": "orders", "filter_field": "status", "filter_value": "pending" } } ``` Only mutations where the `status` field equals `"pending"` are delivered. Delete events always pass through regardless of filter (to notify you when matching records are removed). ### Receiving Mutations[​](#receiving-mutations "Direct link to Receiving Mutations") When subscribed data changes, the server pushes a `MutationNotification`: ``` { "type": "MutationNotification", "payload": { "collection": "orders", "event": "insert", "record_ids": ["rec_abc123"], "records": { "id": "rec_abc123", "product": "Widget", "status": "pending", "quantity": 5 }, "timestamp": "2026-04-11T14:30:45.123456Z" } } ``` **Event types:** | Event | Description | | -------------- | ------------------------- | | `insert` | Single record inserted | | `update` | Single record updated | | `delete` | Single record deleted | | `batch_insert` | Multiple records inserted | | `batch_update` | Multiple records updated | | `batch_delete` | Multiple records deleted | For insert and update events, `records` contains the full record data. For delete events, `records` is null — only `record_ids` are provided. ### Schema Change Notifications[​](#schema-change-notifications "Direct link to Schema Change Notifications") When a collection's schema changes, the server pushes a `SchemaChanged` event to all subscribers of that collection: ``` { "type": "SchemaChanged", "payload": { "collection": "orders", "version": 3, "primary_key_alias": "id" } } ``` Client libraries use this to invalidate their schema cache automatically. ### Unsubscribe[​](#unsubscribe "Direct link to Unsubscribe") ``` { "type": "Unsubscribe", "payload": { "collection": "orders" } } ``` All subscriptions are automatically cleaned up when the WebSocket connection closes. ### Limits[​](#limits "Direct link to Limits") | Setting | Default | | -------------------------------- | ------------------- | | Max subscriptions per connection | 100 | | Event buffer per connection | 256 | | Rate limit | 100 messages/second | | Heartbeat interval | 15 seconds | If a subscriber falls behind (buffer full), events are dropped rather than blocking the mutation path. This is by design — ekoDB prioritizes write throughput over guaranteed delivery to slow subscribers. ## SSE Subscriptions[​](#sse-subscriptions "Direct link to SSE Subscriptions") For simpler integrations (especially browsers), use Server-Sent Events: ``` curl -N https://{EKODB_HOST}/api/subscribe/orders \ -H "Authorization: Bearer {TOKEN}" ``` With filtering: ``` curl -N "https://{EKODB_HOST}/api/subscribe/orders?filter_field=status&filter_value=pending" \ -H "Authorization: Bearer {TOKEN}" ``` **Events received:** ``` event: subscribed data: {"subscription_id":"sse_abc123","collection":"orders"} event: mutation data: {"collection":"orders","event":"insert","record_ids":["rec_abc123"],"timestamp":"2026-04-11T14:30:45Z"} event: schema_changed data: {"collection":"orders","version":3,"primary_key_alias":"id"} ``` ### Browser Integration[​](#browser-integration "Direct link to Browser Integration") ``` // The native browser EventSource API does NOT support custom request // headers. Because the ekoDB SSE endpoint requires `Authorization: // Bearer `, use a polyfill such as `event-source-polyfill` that // lets you pass headers (or proxy the request through your own backend // that injects the header server-side). import { EventSourcePolyfill } from "event-source-polyfill"; const source = new EventSourcePolyfill( "https://my-first-db.development.google.ekodb.net/api/subscribe/orders", { headers: { Authorization: "Bearer " + token } } ); source.addEventListener("mutation", (event) => { const notification = JSON.parse(event.data); console.log(notification.event, notification.record_ids); }); source.addEventListener("schema_changed", (event) => { const schema = JSON.parse(event.data); console.log("Schema updated to version", schema.version); }); source.addEventListener("error", () => { console.log("Connection lost, reconnecting..."); }); ``` SSE vs WebSocket SSE is simpler but limited to one subscription per connection. If you need to subscribe to multiple collections or perform CRUD over the same connection, use WebSocket instead. ## Client Library Usage[​](#client-library-usage "Direct link to Client Library Usage") All client libraries provide a high-level subscribe API that handles the WebSocket connection, message routing, and reconnection. * 🦀 Rust * 🐍 Python * 📘 TypeScript * 🔷 Go * 🟣 Kotlin ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("API_KEY") .build()?; let mut ws = client.websocket("wss://my-first-db.development.google.ekodb.net/api/ws").await?; // Subscribe to all mutations on "orders" let mut rx = ws.subscribe("orders", None, None).await?; // Subscribe with filter let mut rx_pending = ws.subscribe("orders", Some("status"), Some("pending")).await?; // Receive mutations while let Some(notification) = rx.recv().await { println!("{}: {} records", notification.event, notification.record_ids.len()); } ``` ``` from ekodb_client import Client client = Client.new("https://my-first-db.development.google.ekodb.net", "API_KEY") ws = await client.websocket("wss://my-first-db.development.google.ekodb.net/api/ws") # Subscribe to all mutations receiver = await ws.subscribe("orders") # Subscribe with filter receiver_pending = await ws.subscribe("orders", filter_field="status", filter_value="pending") # Receive mutations while True: notification = await receiver.recv() if notification is None: break print(f"{notification['event']}: {notification['record_ids']}") ``` ``` import { EkoDBClient } from '@ekodb/ekodb-client'; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "API_KEY" }); await client.init(); // Subscribe to all mutations const stream = await client.subscribe("orders"); // Subscribe with filter const filteredStream = await client.subscribe("orders", { filterField: "status", filterValue: "pending" }); // Receive mutations stream.on("mutation", (notification) => { console.log(`${notification.event}: ${notification.recordIds}`); }); stream.on("error", (error) => { console.error("Subscription error:", error); }); // Clean up stream.close(); ``` ``` import ( "fmt" "github.com/ekoDB/ekodb-client-go" ) client, _ := ekodb.NewClient("https://my-first-db.development.google.ekodb.net", "API_KEY") ws, _ := client.WebSocket("wss://my-first-db.development.google.ekodb.net/api/ws") // Subscribe to all mutations ch, _ := ws.Subscribe("orders") // Subscribe with filter chPending, _ := ws.Subscribe("orders", ekodb.SubscribeOptions{ FilterField: "status", FilterValue: "pending", }) // Receive mutations for notification := range ch { fmt.Printf("%s: %v\n", notification.Event, notification.RecordIDs) } ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient("https://my-first-db.development.google.ekodb.net", "API_KEY") val ws = client.webSocket("wss://my-first-db.development.google.ekodb.net/api/ws") // Subscribe to all mutations val flow = ws.subscribe("orders") // Subscribe with filter val filteredFlow = ws.subscribe("orders", SubscribeOptions( filterField = "status", filterValue = "pending" )) // Receive mutations flow.collect { notification -> println("${notification.event}: ${notification.recordIds}") } ``` ## Architecture Notes[​](#architecture-notes "Direct link to Architecture Notes") **Non-blocking broadcasts.** Mutations are never slowed down by subscribers. The server uses `try_send` — if a subscriber's buffer is full, the event is dropped and a warning is logged. This means real-time notifications are best-effort; for guaranteed delivery of every change, use [Ripples](/ripples.md) instead. **One channel per connection.** All subscriptions on a single WebSocket connection share one internal channel (capacity 256). This keeps memory usage constant regardless of how many collections you subscribe to. **Automatic cleanup.** When a WebSocket disconnects, all subscriptions are removed immediately. No stale subscriptions accumulate. **Permission enforcement.** Subscriptions require read permission on the collection. The server checks permissions at subscribe time and returns a 403 error if denied. --- # Architecture Patterns Real-world deployment patterns showing where ekoDB fits in your stack. Choose the pattern that matches your workload. *** ## The Core Principle[​](#the-core-principle "Direct link to The Core Principle") **Separate data processing from data serving.** Your app should run fast while your backend handles the complexity of large datasets. ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Data Tiering Model │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Data Lake/Warehouse → Projection Layer → App Layer │ │ (Snowflake, Databricks) (PostgreSQL, ekoDB) (ekoDB/Redis)│ │ │ │ ▪ All data ▪ Scoped subsets ▪ Hot data │ │ ▪ Batch processing ▪ By user/group/project ▪ Sub-ms reads│ │ ▪ Analytics ▪ Transformed for app ▪ Sessions │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` Not every app needs all three tiers. Match complexity to requirements. *** ## Pattern 1: Simple CRUD[​](#pattern-1-simple-crud "Direct link to Pattern 1: Simple CRUD") **Best for:** MVPs, internal tools, straightforward applications ``` ┌──────────────┐ ┌──────────────┐ │ │ │ │ │ Client │ ◄─────► │ ekoDB │ │ (Web/App) │ │ (Primary) │ │ │ │ │ └──────────────┘ └──────────────┘ ``` **Configuration:** * Storage Mode: `fast` or `balanced` * Durability: `durable` for production **When to use:** * Data fits on a single node * Queries are straightforward * No complex analytics requirements * Team wants minimal operational overhead **Example: Internal Dashboard** ``` ┌─────────────────┐ ┌─────────────────┐ │ React Admin │ │ ekoDB │ │ Dashboard │◄─────►│ │ │ │ │ ▪ Users │ │ ▪ User mgmt │ REST │ ▪ Settings │ │ ▪ Settings │ + │ ▪ Audit logs │ │ ▪ Reports │ WS │ ▪ Reports │ └─────────────────┘ └─────────────────┘ ``` > 📚 See [Basic Operations](/basic-operations.md) for CRUD examples and [Storage Modes](/white-paper.md#24-configuration-options) for configuration. *** ## Pattern 2: Social Media Platform[​](#pattern-2-social-media-platform "Direct link to Pattern 2: Social Media Platform") **Best for:** High read volume, real-time feeds, user-generated content ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────┐│ │ │ Warehouse│ │ PostgreSQL │ │ ekoDB │ │ Client││ │ │ (Spark) │───►│ (Primary) │───►│ (Cache + │◄──►│ Apps ││ │ │ │ │ │ │ Real-time) │ │ ││ │ └──────────┘ └──────────────┘ └──────────────┘ └───────┘│ │ │ │ │ │ │ Analytics & │ Live feeds │ │ └─────────ML Pipeline └────WebSocket │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` **Data Flow:** | Layer | Technology | Purpose | | ------------ | ---------------- | --------------------------------------------- | | Cold Storage | Spark/Databricks | Historical analytics, ML training | | Primary | PostgreSQL | User accounts, relationships, content | | Hot Layer | ekoDB | Feed cache, sessions, real-time notifications | | Client | Web/Mobile | User-facing applications | **ekoDB Role:** * **KV Store:** User sessions, auth tokens (Fast mode) * **Document Store:** Feed cache by user/group (scoped projections) * **WebSocket:** Real-time notifications, typing indicators * **Vector Search:** Content recommendations **Example Configuration:** ``` { "storage_mode": "fast", "durable_operations": false, "collections": { "sessions": { "ttl": "24h" }, "feed_cache": { "ttl": "5m" }, "notifications": { "ttl": "7d" } } } ``` **Why this works:** * PostgreSQL handles writes and complex queries * ekoDB serves read-heavy feed requests at high throughput * Scoped caching: each user sees only their feed, not the entire dataset * WebSocket eliminates polling for real-time features > 📚 See [Key-Value Store](/kv-store.md) for session management and [Basic Operations](/client-libraries/basic-operations.md) for real-time subscriptions. *** ## Pattern 3: Financial Services[​](#pattern-3-financial-services "Direct link to Pattern 3: Financial Services") **Best for:** Transactions, audit compliance, strong consistency requirements ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ PostgreSQL │ │ ekoDB │ │ ekoDB │ │ │ │ (Ledger) │───►│ (Read Cache)│ │ (Audit Logs) │ │ │ │ │ │ │ │ Balanced Mode│ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ API Gateway │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ Clients │ │ │ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` **Data Flow:** | Layer | Technology | Mode | Purpose | | ---------- | ---------- | ------------------ | ------------------------------- | | Ledger | PostgreSQL | ACID | Core transactions, balances | | Read Cache | ekoDB | Fast + Durable | Account lookups, balance checks | | Audit | ekoDB | Balanced + Durable | Immutable transaction history | **Critical Configuration:** ``` { "ledger_cache": { "storage_mode": "fast", "durable_operations": true }, "audit_logs": { "storage_mode": "balanced", "durable_operations": true } } ``` **Why Balanced Mode for Audit:** * Durable append-heavy writes with checkpoint batching to disk * Compact on-disk format for efficient disk usage * Immutable once written * Regulatory compliance friendly **Security Configuration:** * AES-256-GCM encryption at rest * TLS/SSL in transit (HTTPS/WSS only) * JWT authentication with short TTL * Field-level access control **Example: Transaction Flow** ``` 1. Client requests transfer 2. API validates via ekoDB (session check, rate limit) 3. PostgreSQL executes transaction (ACID) 4. PostgreSQL confirms commit 5. ekoDB cache invalidated/updated 6. ekoDB Balanced logs audit entry (immutable) 7. Client receives confirmation ``` > 📚 See [Transactions](/transactions.md) for ACID operations and [Security](/security.md) for encryption details. *** ## Pattern 4: E-Commerce Platform[​](#pattern-4-e-commerce-platform "Direct link to Pattern 4: E-Commerce Platform") **Best for:** Product catalogs, shopping carts, search, recommendations ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────────────────────────────┐│ │ │ Warehouse │ │ ekoDB ││ │ │ (Analytics) │ │ ┌────────────┐ ┌────────────────┐ ││ │ │ │────────►│ │ Products │ │ Sessions/Cart │ ││ │ │ ▪ Sales │ │ │ (Catalog) │ │ (KV TTL) │ ││ │ │ ▪ Inventory │ │ └────────────┘ └────────────────┘ ││ │ │ ▪ Trends │ │ ┌────────────┐ ┌────────────────┐ ││ │ └──────────────┘ │ │ Search │ │ Vectors for │ ││ │ │ │ (Full-text)│ │ Recommendations│ ││ │ │ └────────────┘ └────────────────┘ ││ │ └──────────────────────────────────────┘││ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ Storefront │ │ │ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` **ekoDB handles everything in one binary:** | Feature | Traditional Stack | ekoDB | | ------------------- | --------------------- | ----------------------- | | Product catalog | PostgreSQL | Document collections | | Product search | Elasticsearch | Built-in full-text | | Recommendations | Pinecone + ML service | Built-in vector search | | Shopping cart | Redis | KV with TTL | | Sessions | Redis | KV with TTL | | Real-time inventory | Kafka + Redis | WebSocket subscriptions | **Example: Product Document** ``` { "sku": "WIDGET-001", "name": "Premium Widget", "description": "High-quality widget for all your needs", "price": 29.99, "inventory": 150, "categories": ["widgets", "premium"], "embedding": [0.1, 0.2, 0.3, ...] // 384-dim for recommendations } ``` **Search + Recommendations in one query:** ``` // Full-text search const results = await client.search("products", { text: "premium widget", fields: ["name", "description"], fuzzy: true }); // Similar products (vector search via search method) const similar = await client.search("products", { vector: currentProduct.embedding, vector_field: "embedding", vector_k: 5 }); ``` **Cart with TTL:** ``` await client.kvSet(`cart:${userId}`, cartData, 86400); // TTL in seconds (24h) ``` **Storage Mode Configuration:** ``` { "products": { "storage_mode": "balanced", "durable_operations": true }, "sessions_cart": { "storage_mode": "fast", "durable_operations": false }, "order_history": { "storage_mode": "balanced", "durable_operations": true }, "inventory_updates": { "storage_mode": "fast", "durable_operations": true } } ``` | Data Type | Mode | Durability | Why | | ------------- | ---------- | ---------- | -------------------------------------------- | | Products | `balanced` | `true` | Read-heavy but updates matter, large catalog | | Sessions/Cart | `fast` | `false` | Ephemeral, TTL-based, regenerable | | Order History | `balanced` | `true` | Append-only, compliance, archival | | Inventory | `fast` | `true` | Real-time updates, must persist | > 📚 See [Storage Modes](/white-paper.md#24-configuration-options) for detailed mode comparisons and [Basic Operations](/basic-operations.md) for CRUD examples. *** ## Pattern 5: IoT / Time-Series[​](#pattern-5-iot--time-series "Direct link to Pattern 5: IoT / Time-Series") **Best for:** Sensor data, logs, metrics, high write throughput ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Devices │ │ ekoDB │ │ Warehouse │ │ │ │ (Sensors) │───►│ Fast Mode │───►│ (Archival) │ │ │ │ │ │ (Ingestion) │ │ │ │ │ └──────────────┘ └──────┬───────┘ └──────────────┘ │ │ │ │ │ WebSocket │ │ │ High throughput ▼ │ │ ┌──────────────┐ │ │ │ ekoDB │ │ │ │ Fast Mode │◄───── Dashboard/Alerts │ │ │ (Real-time) │ │ │ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` **Dual-Node Strategy:** | Node | Mode | Purpose | | --------- | ------------------ | ---------------------------------------------------------------------------------------------- | | Ingestion | Fast + Non-Durable | Highest write throughput of any mode — see [Performance Benchmarks](/reference/performance.md) | | Query | Fast + Durable | Real-time dashboards, alerts | **Why Fast Mode for Ingestion:** * Highest measured write throughput of any mode * Minimal disk I/O overhead with durability off * Automatic archival to warehouse via Ripple **Configuration:** ``` { "ingestion_node": { "storage_mode": "fast", "durable_operations": false, "ripple": { "targets": ["query_node", "warehouse"] } }, "query_node": { "storage_mode": "fast", "durable_operations": true } } ``` **Data Flow:** ``` Sensor → ekoDB Fast (ingest) → Ripple → ekoDB Fast (query) → Ripple → Snowflake (archive) ``` > 📚 See [Ripples](/ripples.md) for multi-node sync and [Storage Modes](/white-paper.md#24-configuration-options) for storage mode tuning. *** ## Pattern 6: AI / RAG Application[​](#pattern-6-ai--rag-application "Direct link to Pattern 6: AI / RAG Application") **Best for:** LLM applications, semantic search, chat with memory ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────────────────────────┐ │ │ │ Documents │ │ ekoDB │ │ │ │ (Upload) │───►│ ┌────────────┐ ┌────────────┐ │ │ │ │ │ │ │ Vectors │ │ Chat │ │ │ │ └──────────────┘ │ │ (Embeddings│ │ History │ │ │ │ │ │ + Search) │ │ (Branching)│ │ │ │ ┌──────────────┐ │ └────────────┘ └────────────┘ │ │ │ │ LLM │◄──►│ ┌────────────┐ ┌────────────┐ │ │ │ │ (OpenAI/ │ │ │ KV for │ │ Document │ │ │ │ │ Claude) │ │ │ Sessions │ │ Metadata │ │ │ │ └──────────────┘ │ └────────────┘ └────────────┘ │ │ │ └──────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ Chat UI │ │ │ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` **ekoDB provides the complete RAG stack:** | Component | Traditional | ekoDB | | -------------- | ----------------- | ---------------------------- | | Vector store | Pinecone/Weaviate | Built-in vector search | | Document store | PostgreSQL | Document collections | | Chat history | Redis/PostgreSQL | Built-in chat with branching | | Session state | Redis | KV with TTL | | Metadata | PostgreSQL | Same document | **RAG Workflow:** ``` // 1. Store documents with embeddings await client.insert("knowledge_base", { content: "ekoDB is a multi-model database...", embedding: await embed(content), // 384-dim source: "documentation", updated: new Date() }); // 2. Semantic search for context const context = await client.search("knowledge_base", { vector: await embed(userQuery), vector_field: "embedding", vector_k: 5 }); // 3. Chat with memory (built-in branching) const response = await client.chatMessage(sessionId, { message: userQuery, }); ``` **Chat Branching:** ekoDB's built-in chat system supports conversation branching: ``` Main conversation ├── Branch A (what-if scenario) │ └── Continue exploring... └── Branch B (alternative approach) └── Different direction... ``` **Storage Mode Configuration:** ``` { "knowledge_base": { "storage_mode": "balanced", "durable_operations": true }, "chat_history": { "storage_mode": "balanced", "durable_operations": true }, "sessions": { "storage_mode": "fast", "durable_operations": false }, "embeddings_cache": { "storage_mode": "fast", "durable_operations": true } } ``` | Data Type | Mode | Durability | Why | | -------------- | ---------- | ---------- | ---------------------------------------------------- | | Knowledge Base | `balanced` | `true` | Large documents, occasional updates, must persist | | Chat History | `balanced` | `true` | Append-only conversations, audit trail | | Sessions | `fast` | `false` | Ephemeral user state, TTL-based | | Embeddings | `fast` | `true` | Frequently accessed vectors, expensive to regenerate | > 📚 See [Chat & RAG](/chat-rag.md) for conversation management and [Vector Search](/vector-search.md) for embedding operations. *** ## Storage Mode Quick Reference[​](#storage-mode-quick-reference "Direct link to Storage Mode Quick Reference") Choose the right combination for your data: | Mode | Durable | Best For | Trade-off | | ---------- | ------- | ------------------------------------------ | ----------------------------------------------------- | | `fast` | `true` | User data, account state, critical records | Highest durability + speed | | `fast` | `false` | Sessions, caches, ephemeral data | Maximum speed, data loss on crash OK | | `balanced` | `true` | Large datasets, production workloads | Good balance for mixed read/write | | `balanced` | `false` | Development, bulk imports, migrations | Faster imports, not for production | | `balanced` | `true` | Audit logs, compliance, archival | The durable append-heavy choice | | `fast` | `false` | High-throughput ingestion, metrics, logs | Highest write speed of any mode; some loss acceptable | **When to use each mode:** * **`fast`**: Data fits in memory, sub-millisecond reads required * **`balanced`**: Datasets larger than RAM, mixed read/write workloads **Durability guidelines:** * **`durable: true`**: Financial data, user records, anything you can't regenerate * **`durable: false`**: Caches, sessions, computed data, anything with TTL > 📚 See [White Paper](/white-paper.md#24-configuration-options) for configuration options and [Performance Benchmarks](/reference/performance.md) for detailed metrics. *** ## Decision Matrix[​](#decision-matrix "Direct link to Decision Matrix") | If you need... | Use Pattern | Storage Mode | Durability | Key Features | | ------------------------- | --------------- | -------------------- | ------------------ | ------------------------------------- | | Simple app, minimal ops | 1: Simple CRUD | `fast` or `balanced` | `true` | Single node, straightforward queries | | High read volume, feeds | 2: Social Media | `fast` | `false` for cache | WebSocket, TTL-based sessions | | Strong consistency, audit | 3: Financial | `fast` + `balanced` | `true` always | Ledger cache + immutable audit logs | | Search + recommendations | 4: E-Commerce | `balanced` + `fast` | Mixed | Full-text + vectors + KV cart | | High write throughput | 5: IoT | `fast` | `false` for ingest | Highest write throughput, Ripple sync | | LLM/RAG application | 6: AI/RAG | `balanced` | `true` | Vectors + chat history + sessions | ### Storage Mode by Data Type[​](#storage-mode-by-data-type "Direct link to Storage Mode by Data Type") | Data Type | Recommended Mode | Durability | Examples | | ------------------ | ---------------- | ---------- | -------------------------------------- | | User accounts | `fast` | `true` | Profiles, preferences, auth tokens | | Sessions/cache | `fast` | `false` | Login sessions, API cache, rate limits | | Large documents | `balanced` | `true` | Product catalogs, knowledge bases | | Bulk imports | `balanced` | `false` | Data migrations, batch uploads | | Audit/compliance | `balanced` | `true` | Transaction logs, access logs | | High-volume writes | `fast` | `false` | Metrics, IoT sensors, clickstream | *** ## Anti-Patterns[​](#anti-patterns "Direct link to Anti-Patterns") **Don't do this:** | Anti-Pattern | Why It Fails | Instead | | ------------------------------------ | --------------------------------- | -------------------------------- | | Everything in one collection | No isolation, hard to scale | Separate by access pattern | | Non-durable for financial data | Data loss on crash | Always `durable: true` | | Warehouse queries from app | Slow, expensive | Project to ekoDB first | | Polling for real-time | Wasteful, laggy | Use WebSocket subscriptions | | Skipping the cache layer | Database overload at scale | Add ekoDB/Redis for hot data | | `fast` + `non-durable` for user data | Data loss on crash | Use `durable: true` | | Same mode for all collections | Missed optimization opportunities | Match mode to access pattern | | `balanced` for ephemeral cache | Unnecessary disk I/O | Use `fast` + `non-durable` + TTL | **Storage Mode Mistakes:** | Wrong Choice | Problem | Correct Choice | | ---------------------------- | ------------------------ | -------------------------- | | `fast` for 100GB+ datasets | Memory pressure | `balanced` | | `non-durable` for orders | Lost transactions | Always `durable: true` | | `durable` for computed cache | Unnecessary I/O overhead | `non-durable` + regenerate | *** ## Migration Path[​](#migration-path "Direct link to Migration Path") Start simple, add complexity as needed: ``` Stage 1: ekoDB only ↓ Stage 2: ekoDB + separate cache (if needed) ↓ Stage 3: PostgreSQL (writes) + ekoDB (reads/cache) ↓ Stage 4: Warehouse + PostgreSQL + ekoDB (full tiering) ``` Most applications never need Stage 4. Don't prematurely optimize. **Storage Mode Migration:** ``` Start: fast + durable (simple, safe default) ↓ Scale: balanced + durable (dataset grows beyond RAM) ↓ Optimize: Mixed modes per collection (different access patterns) ↓ Enterprise: balanced for audit + fast for cache + balanced for data ``` *** ## See Also[​](#see-also "Direct link to See Also") ### Core Documentation[​](#core-documentation "Direct link to Core Documentation") * [Basic Operations](/basic-operations.md) - CRUD operations and queries * [Client Libraries](/client-libraries/.md) - Language-specific SDKs and examples * [Storage Modes](/white-paper.md#24-configuration-options) - Fast vs Balanced ### Advanced Topics[​](#advanced-topics "Direct link to Advanced Topics") * [Performance Benchmarks](/reference/performance.md) - Detailed YCSB results * [Vector Search](/vector-search.md) - Semantic search and embeddings * [Chat & RAG](/chat-rag.md) - LLM integration and conversation management ### Scaling & Operations[​](#scaling--operations "Direct link to Scaling & Operations") * [Ripples](/ripples.md) - Multi-node synchronization and data propagation * [Transactions](/transactions.md) - ACID operations and isolation levels * [Key-Value Store](/kv-store.md) - High-performance caching with TTL --- # Error Codes Reference This document provides a complete reference for HTTP status codes and error responses returned by ekoDB's REST API. Understanding these errors helps you handle failures gracefully and troubleshoot issues efficiently. ## Error Response Format[​](#error-response-format "Direct link to Error Response Format") ekoDB uses two error response formats depending on the endpoint: ### Standard Format (Most Endpoints)[​](#standard-format-most-endpoints "Direct link to Standard Format (Most Endpoints)") ``` { "error": "Human-readable error description" } ``` ### Authentication/Rate Limit Format[​](#authenticationrate-limit-format "Direct link to Authentication/Rate Limit Format") ``` { "code": 401, "message": "Human-readable error description" } ``` ## HTTP Status Codes[​](#http-status-codes "Direct link to HTTP Status Codes") ### Success Codes (2xx)[​](#success-codes-2xx "Direct link to Success Codes (2xx)") | Code | Name | Description | | ------- | ------------ | ----------------------------------------------------------------------------- | | **200** | OK | Request succeeded. Response body contains requested data. | | **201** | Created | Resource successfully created (documents, collections, indexes). | | **207** | Multi-Status | Batch operation completed with mixed results. Check individual item statuses. | ### Client Error Codes (4xx)[​](#client-error-codes-4xx "Direct link to Client Error Codes (4xx)") #### 400 Bad Request[​](#400-bad-request "Direct link to 400 Bad Request") The request was malformed or contains invalid data. **Common Causes:** * Invalid JSON syntax in request body * Missing required fields * Invalid field types or values * Malformed query expressions * Invalid filter operators * Collection name validation failed **Example Response:** ``` { "error": "Invalid JSON: expected value at line 1 column 15" } ``` **Troubleshooting:** 1. Validate your JSON syntax using a JSON linter 2. Ensure all required fields are present 3. Verify filter expressions use correct `type` values (`Condition`, `Logical`) 4. Check operator names are spelled correctly (`Eq`, `Gt`, `Lt`, `Gte`, `Lte`, `Ne`) *** #### 401 Unauthorized[​](#401-unauthorized "Direct link to 401 Unauthorized") Authentication is required or credentials are invalid. **Common Causes:** * Missing `Authorization` header * Expired JWT token (default: 1-hour lifetime) * Invalid API key * Malformed bearer token **Example Response:** ``` { "code": 401, "message": "Invalid or expired token" } ``` **Troubleshooting:** 1. Ensure the `Authorization: Bearer {TOKEN}` header is present 2. Generate a fresh token if the current one has expired 3. Verify your API key is valid and has not been revoked 4. Check for typos in the bearer token **Token Refresh Example:** ``` curl -X POST https://{EKODB_API_URL}/api/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key": "YOUR_API_KEY"}' ``` *** #### 402 Payment Required[​](#402-payment-required "Direct link to 402 Payment Required") The LLM provider's account cannot pay for the call (AI/Chat features). **Common Causes:** * The provider's credit balance is exhausted * An organization or project spend limit at the provider was reached * Billing is not set up at the provider The body carries `error_kind: "provider_billing"` and names the provider — see [Provider failures](#provider-failures-llm-features). Never an ekoDB limit: ekoDB does not meter chat usage with a 402. **Example Response:** ``` { "error": "Anthropic API error: Your credit balance is too low to access the Anthropic API.", "error_kind": "provider_billing", "provider": "anthropic", "provider_status": 400 } ``` **Troubleshooting:** 1. Add credit, or check the spend and usage limits, in the named provider's billing console — the key itself was accepted 2. A retry will not succeed until the provider's balance or limit changes 3. `GET /api/chat_models` reports the same key as `billing` in its `providers` map, so you can confirm the fix without sending a chat *** #### 403 Forbidden[​](#403-forbidden "Direct link to 403 Forbidden") The authenticated user lacks permission for the requested operation. **Common Causes:** * Attempting to access a collection without permission * API key lacks required scope * Collection-level access restrictions (writable\_fields enforcement) * Admin-only operation attempted with non-admin key **Example Response:** ``` { "error": "Access denied: cannot modify fields: email, role" } ``` **Troubleshooting:** 1. Verify your API key has the required permissions 2. Check collection permission settings 3. Contact your administrator if you need elevated access 4. Review `collection_permissions` on your API key *** #### 404 Not Found[​](#404-not-found "Direct link to 404 Not Found") The requested resource does not exist. **Common Causes:** * Collection name is misspelled * Document ID does not exist * Endpoint URL is incorrect * Resource was deleted **Example Response:** ``` { "error": "Record not found" } ``` **Troubleshooting:** 1. Double-check the collection name (case-sensitive) 2. Verify the document ID exists before updating/deleting 3. Ensure the endpoint URL is correct 4. List collections to see available resources *** #### 429 Too Many Requests[​](#429-too-many-requests "Direct link to 429 Too Many Requests") Rate limit exceeded. The client has sent too many requests in a given time window. **Example Response:** ``` { "code": 429, "message": "Rate limit exceeded. Please try again in 60 seconds." } ``` **Response Headers:** | Header | Description | | ------------- | ----------------------------------------- | | `Retry-After` | Number of seconds to wait before retrying | **Troubleshooting:** 1. Implement exponential backoff retry logic 2. Respect the `Retry-After` header value 3. Batch multiple operations where possible 4. Consider upgrading your plan for higher limits **Retry Logic Example (JavaScript):** ``` async function retryableRequest(fn, maxRetries = 3) { for (let attempt = 0; attempt < maxRetries; attempt++) { try { return await fn(); } catch (error) { if (error.status === 429) { const retryAfter = parseInt(error.headers.get("Retry-After") || "60"); await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000)); } else { throw error; } } } throw new Error("Max retries exceeded"); } ``` ### Server Error Codes (5xx)[​](#server-error-codes-5xx "Direct link to Server Error Codes (5xx)") #### 500 Internal Server Error[​](#500-internal-server-error "Direct link to 500 Internal Server Error") An unexpected error occurred on the server. **Example Response:** ``` { "code": 500, "message": "Internal Server Error" } ``` **Troubleshooting:** 1. Retry the request after a brief delay 2. Check ekoDB status page for ongoing incidents 3. Contact support if the error persists *** #### 502 Bad Gateway[​](#502-bad-gateway "Direct link to 502 Bad Gateway") An LLM provider rejected ekoDB's request or is down (AI/Chat features): a rejected key, a permission problem, a 5xx from the provider, or no answer at all. The body says which — see [Provider failures](#provider-failures-llm-features). Never a problem with your ekoDB token or its permissions: those are a 401 or a 403. **Example Response:** ``` { "error": "OpenAI API error: Incorrect API key provided: sk-abc...", "error_kind": "provider_auth_failed", "provider": "openai", "provider_status": 401 } ``` *** #### 503 Service Unavailable[​](#503-service-unavailable "Direct link to 503 Service Unavailable") The service is temporarily unavailable. **Common Causes:** * Too many concurrent batch operations (server overload protection) * Server maintenance * Server misconfiguration on startup **Example Response:** ``` { "error": "Server is currently processing too many batch operations. Please try again later.", "active_operations": 64, "max_concurrent": 64, "retry_after_seconds": 5 } ``` **Troubleshooting:** 1. Check ekoDB status page for maintenance windows 2. Implement retry logic with exponential backoff 3. Wait a few minutes before retrying ## Provider Failures (LLM Features)[​](#provider-failures-llm-features "Direct link to Provider Failures (LLM Features)") When a chat call fails because of the LLM provider, the body is classified so a client can act on it without string-matching: | `error_kind` | HTTP status | Meaning | | ---------------------------- | ----------------------------------------------------- | --------------------------------------------------------- | | `provider_not_configured` | 400 on chat; 404 on `GET /api/chat_models/{provider}` | No key for that provider | | `provider_request_error` | 400 | The provider refused the request (bad model, parameters) | | `provider_billing` | 402 | The provider's account cannot pay | | `provider_rate_limited` | 429 | Provider rate limit; `Retry-After` is forwarded when sent | | `provider_auth_failed` | 502 | The provider rejected the configured key | | `provider_permission_denied` | 502 | The key may not use this resource or region | | `provider_unavailable` | 502 | The provider answered 5xx or an unusable body | | `provider_unreachable` | 502 | Nothing answered: DNS, connect, TLS, or a timeout | `provider` names the provider and `provider_status` is its own HTTP status; `retry_after_secs` is present on a rate limit when the provider named a wait. A provider failure is never reported as 401 or 403: at this layer those two statuses always mean the caller's own ekoDB token was rejected, never the provider's answer. Streaming chat (`text/event-stream`) carries the same fields in the payload of its `error` event, and the WebSocket `ChatStreamError` message carries them beside `error`. ## Operation-Specific Errors[​](#operation-specific-errors "Direct link to Operation-Specific Errors") ### Authentication Errors[​](#authentication-errors "Direct link to Authentication Errors") | Scenario | Code | Example Message | | --------------- | ---- | -------------------------- | | Missing API key | 401 | "API key is required" | | Invalid API key | 401 | "Invalid API key provided" | | Expired token | 401 | "Token has expired" | | Malformed token | 401 | "Invalid token format" | ### Collection Errors[​](#collection-errors "Direct link to Collection Errors") | Scenario | Code | Example Message | | ----------------------- | ---- | ------------------------------ | | Collection not found | 404 | "Collection 'users' not found" | | Invalid collection name | 400 | "Invalid collection name" | | Missing collection name | 400 | "Collection name is required" | ### Document Errors[​](#document-errors "Direct link to Document Errors") | Scenario | Code | Example Message | | ----------------------- | ---- | ----------------------------------- | | Document not found | 404 | "Record not found" | | Invalid document data | 400 | "Invalid document format" | | Field permission denied | 403 | "Cannot modify fields: email, role" | ### Batch Operation Errors[​](#batch-operation-errors "Direct link to Batch Operation Errors") Batch operations return `207 Multi-Status` when some operations succeed and others fail: ``` { "success_count": 8, "error_count": 2, "results": [ { "id": "doc1", "status": "success" }, { "id": "doc2", "status": "error", "error": "Validation failed" } ] } ``` ### Transaction Errors[​](#transaction-errors "Direct link to Transaction Errors") | Scenario | Code | Example Message | | ----------------------- | ---- | ------------------------- | | Transaction not found | 404 | "Transaction not found" | | Transaction expired | 400 | "Transaction has expired" | | Invalid isolation level | 400 | "Invalid isolation level" | ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Error Handling Pattern[​](#error-handling-pattern "Direct link to Error Handling Pattern") ``` try { const result = await client.insert("users", userData); } catch (error) { switch (error.status) { case 400: console.error("Invalid request:", error.message); // Fix request data and retry break; case 401: console.error("Authentication failed"); // Refresh token and retry await refreshToken(); break; case 403: console.error("Permission denied:", error.message); // Check permissions break; case 404: console.error("Resource not found:", error.message); // Check if collection/document exists break; case 429: console.error("Rate limited, waiting..."); // Wait and retry with backoff break; case 500: case 503: console.error("Server error, retrying..."); // Retry with exponential backoff break; default: console.error("Unexpected error:", error); } } ``` ### Implementing Exponential Backoff[​](#implementing-exponential-backoff "Direct link to Implementing Exponential Backoff") ``` import time import random def exponential_backoff_retry(func, max_retries=5, base_delay=1): for attempt in range(max_retries): try: return func() except Exception as e: if hasattr(e, 'status_code') and e.status_code in [429, 500, 503]: delay = base_delay * (2 ** attempt) + random.uniform(0, 1) time.sleep(delay) else: raise raise Exception("Max retries exceeded") ``` ### Logging Error Responses[​](#logging-error-responses "Direct link to Logging Error Responses") Always log the full error response for debugging: ``` match client.find("users", query, None).await { Ok(results) => { /* handle success */ } Err(e) => { error!( status = %e.status, error = %e.message, endpoint = "query", collection = "users", "API request failed" ); } } ``` ## Getting Help[​](#getting-help "Direct link to Getting Help") If you encounter persistent errors: 1. **Check the status page** - Verify ekoDB services are operational 2. **Review the documentation** - Ensure your request format is correct 3. **Search existing issues** - Your error may already have a known solution 4. **Contact support** - Include the full error response and request details Support Resources * **Documentation**: [docs.ekodb.io](https://docs.ekodb.io) * **Support Email**: --- # Functions Architecture Technical Overview This document provides an in-depth technical overview of ekoDB's Functions system. For practical usage examples, see [Advanced Operations - Functions](/client-libraries/advanced-operations.md#functions). ## Overview[​](#overview "Direct link to Overview") ekoDB's **Functions** system is a powerful server-side execution engine that enables you to define, version, and execute complex business logic as composable operations. Unlike traditional stored procedures, ekoDB Functions are JSON-based, parameterized, and designed for modern application architectures. ### Key Concepts[​](#key-concepts "Direct link to Key Concepts") **Function**: A stored procedure containing one or more operations (Query, Insert, Update, Delete, Chat, Embed, etc.) **Operation**: A single database action within a Function (referred to in the `functions` array) **Composability**: Functions can call other Functions using `CallFunction`, enabling unlimited nesting and reusability ## Why Functions?[​](#why-functions "Direct link to Why Functions?") ### Traditional Approach Problems[​](#traditional-approach-problems "Direct link to Traditional Approach Problems") ``` // ❌ Business logic scattered across application code async function getEnrichedUserData(userId: string) { // Query user const user = await db.findOne("users", { id: userId }); // Query orders const orders = await db.find("orders", { user_id: userId, status: "active" }); // Generate AI summary const summary = await openai.complete(`Summarize: ${JSON.stringify(orders)}`); // Update user with summary await db.update("users", userId, { ai_summary: summary }); return { user, orders, summary }; } ``` **Issues:** * Multiple network round-trips * Logic duplicated in every client * Hard to version and maintain * No transaction guarantees ### ekoDB Functions Approach[​](#ekodb-functions-approach "Direct link to ekoDB Functions Approach") ``` // ✅ Define once, call from anywhere { "label": "get_user_orders", "name": "Get User Orders with Count", "parameters": { "user_id": { "type": "string", "required": true }, "status": { "type": "string", "required": false, "default": "active" } }, "functions": [ { "type": "Query", "collection": "orders", "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "user_id", "operator": "Eq", "value": "{{user_id}}" } }, { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "{{status}}" } } ] } }, "limit": 50 }, { "type": "Count", "output_field": "total_orders" }, { "type": "Project", "fields": ["id", "amount", "status", "total_orders"], "exclude": false } ] } ``` **Benefits:** * ✅ Single network call * ✅ Defined once, used everywhere * ✅ Version controlled * ✅ Server-side execution * ✅ Automatic parameter validation ## Architecture[​](#architecture "Direct link to Architecture") ### Operation Types[​](#operation-types "Direct link to Operation Types") ekoDB Functions support these step types (the `type` field of each step). This is the complete catalog. #### Query & Read[​](#query--read "Direct link to Query & Read") * **FindAll** - Retrieve all records in a collection * **Query** - Filter, sort, and paginate (`filter`/`sort`/`limit`/`skip` are inline on this step) * **FindById** - Get one record by ID * **FindOne** - Find one record by key/value * **TextSearch** - Full-text search * **VectorSearch** - Semantic similarity (vector) search * **HybridSearch** - Combined text + vector search #### Write (CRUD)[​](#write-crud "Direct link to Write (CRUD)") * **Insert** - Insert one record * **BatchInsert** - Insert many records * **Update** - Update records matching a filter * **UpdateById** - Update one record by ID * **FindOneAndUpdate** - Atomic find-and-update * **UpdateWithAction** - Atomic field action (increment, push, append, pop, etc.) * **Upsert** - Atomic find-or-create by lookup key * **Delete** - Delete records matching a filter * **DeleteById** - Delete one record by ID * **BatchDelete** - Delete many records by ID #### In-pipeline record edits[​](#in-pipeline-record-edits "Direct link to In-pipeline record edits") * **SetField** - Set a literal value on the working record * **Increment** - Atomic counter add on a stored record * **Push** - Atomic array append on a stored record * **CurrentDatetime** - Stamp the current UTC time onto the working record #### Aggregation & shaping[​](#aggregation--shaping "Direct link to Aggregation & shaping") * **Group** - Group and aggregate (11 operations: Count, Sum, Average, Min, Max, First, Last, Push, AddToSet, StandardDeviation, ApproxDistinct) * **AddFields** - Add computed/derived fields (arithmetic, string, and conditional expressions) * **Project** - Select or exclude fields * **Count** - Add a record-count field #### Control flow[​](#control-flow "Direct link to Control flow") * **If** - Branch on a `FunctionCondition` * **ForEach** - Iterate over the working records, running sub-steps per record * **Parallel** - Run sub-steps concurrently and merge their results * **TryCatch** - Run steps; on error, run the catch steps (error bound to `output_error_field`) * **Sleep** - Pause (e.g. for rate limits or retry backoff) * **Return** - Build the final response object (optionally setting `status_code` for the HTTP status) * **Validate** - Assert a condition; fail the pipeline if it doesn't hold * **CallFunction** - Call another saved function by label (composition) #### Transactions[​](#transactions "Direct link to Transactions") * **CreateSavepoint** / **RollbackToSavepoint** / **ReleaseSavepoint** - Nested-transaction savepoints for partial rollback A saved function with `transaction_config.enabled = true` runs its steps as one **buffered, isolated** unit: each step stages into the transaction and the whole set applies atomically at commit, so a failure simply discards the staged writes (nothing is partially applied) and a concurrent operation never observes the function's intermediate writes. Filtered `Update`/`Delete` steps see writes staged earlier in the same function (read-your-writes), and savepoints rewind the buffer. This gives you full **ACID** — atomicity, consistency, isolation, and durability — with the isolation level enforced via optimistic concurrency (MVCC) and configurable up to `Serializable`. See [Transactions](/transactions.md) for the level semantics. #### AI[​](#ai "Direct link to AI") * **Chat** - LLM chat completion * **Embed** - Generate embeddings for a field #### Auth & crypto (build login/signup without external services)[​](#auth--crypto-build-loginsignup-without-external-services "Direct link to Auth & crypto (build login/signup without external services)") * **BcryptHash** / **BcryptVerify** - Password hashing and verification * **JwtSign** / **JwtVerify** - Issue and verify JWTs (HMAC, RSA, ECDSA, EdDSA — see [JWT Signing and Verification](#jwt-signing-and-verification)) * **HmacSign** / **HmacVerify** - HMAC sign and timing-safe verify * **AesEncrypt** / **AesDecrypt** - AES-GCM encrypt/decrypt * **TotpGenerate** / **TotpVerify** - TOTP MFA codes * **RandomToken** - CSPRNG random token * **UuidGenerate** - Generate a UUID * **Base64Encode** / **Base64Decode** - Base64 * **HexEncode** / **HexDecode** - Hex * **Slugify** - URL-safe slug #### Key-value store[​](#key-value-store "Direct link to Key-value store") * **KvGet** / **KvSet** / **KvDelete** / **KvExists** / **KvQuery** - KV cache operations (TTL, prefix scan) #### Caching & concurrency[​](#caching--concurrency "Direct link to Caching & concurrency") * **SWR** - Stale-while-revalidate cached pipeline (cache check → run → cache set) * **IdempotencyClaim** - Claim an idempotency key (once-only execution) * **RateLimit** - Fixed-window rate limit * **LockAcquire** / **LockRelease** - Distributed lock #### External integrations[​](#external-integrations "Direct link to External integrations") * **HttpRequest** - Call an external HTTP API (Stripe, SendGrid, etc.) * **EmailSend** - Send an email ### Parameter System[​](#parameter-system "Direct link to Parameter System") Functions support dynamic parameterization using `{{parameter_name}}` syntax: ``` { "parameters": { "user_id": { "type": "string", "required": true, "description": "User identifier" }, "limit": { "type": "number", "required": false, "default": 10, "description": "Result limit" }, "status": { "type": "string", "required": false, "default": "active", "enum": ["active", "inactive", "pending"] } }, "functions": [ { "type": "Query", "collection": "users", "filter": { "type": "Logical", "content": { "operator": "And", "expressions": [ { "type": "Condition", "content": { "field": "id", "operator": "Eq", "value": "{{user_id}}" } }, { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "{{status}}" } } ] } }, "limit": "{{limit}}" } ] } ``` **Parameter Resolution:** 1. User provides values when calling Function 2. ekoDB validates types and requirements 3. Parameters are substituted before execution 4. Results can reference previous Function outputs ### Result Chaining[​](#result-chaining "Direct link to Result Chaining") Operations execute sequentially, with results flowing from one to the next: ``` { "functions": [ { "type": "Query", "collection": "products", "filter": { "type": "Condition", "content": { "field": "category", "operator": "Eq", "value": "{{category}}" } } }, { "type": "Count", "output_field": "product_count" }, { "type": "Project", "fields": ["name", "price", "product_count"], "exclude": false } ] } ``` **How Results Flow:** * Query operation returns records * Count adds a count field to those records * Project selects only specified fields * Each operation works with the current set of records ## Execution Model[​](#execution-model "Direct link to Execution Model") ### Sequential Execution[​](#sequential-execution "Direct link to Sequential Execution") By default, Functions execute in sequence: ``` Function 1 → Function 2 → Function 3 → Result ``` Each Function waits for the previous to complete before starting. ### Conditional Execution[​](#conditional-execution "Direct link to Conditional Execution") Use the `If` Function type for conditional logic: ``` { "type": "If", "condition": { "type": "FieldEquals", "value": { "field": "plan", "value": "premium" } }, "then_functions": [{ "type": "Query", "collection": "premium_features" }], "else_functions": [{ "type": "Query", "collection": "basic_features" }] } ``` `condition` is a **FunctionCondition** object (adjacently tagged with `type` + `value`), evaluated against the working records — not a string expression. The branches are `then_functions` and `else_functions`. **FunctionCondition variants:** * Field comparisons: `FieldEquals`, `FieldExists`, `FieldGreaterThan`, `FieldLessThan`, `FieldGreaterThanOrEqual`, `FieldLessThanOrEqual` — e.g. `{ "type": "FieldGreaterThan", "value": { "field": "stock", "value": 0 } }` * Set/count: `HasRecords`, `CountEquals`, `CountGreaterThan`, `CountLessThan` * Boolean: `And`, `Or`, `Not` — e.g. `{ "type": "And", "value": { "conditions": [, ...] } }` ### Loop Execution[​](#loop-execution "Direct link to Loop Execution") Use the `ForEach` operation type to iterate over current records: ``` { "functions": [ { "type": "Query", "collection": "users", "filter": { "type": "Condition", "content": { "field": "status", "operator": "Eq", "value": "pending" } } }, { "type": "ForEach", "functions": [ { "type": "UpdateById", "collection": "users", "record_id": "{{id}}", "updates": { "notified": true } } ] } ] } ``` **How it works:** * Operates on records from previous operations * Executes nested operations for each record * Each record's fields are available as variables (e.g., `{{id}}`, `{{email}}`) ## Creating Functions[​](#creating-functions "Direct link to Creating Functions") ### What is a Function?[​](#what-is-a-function "Direct link to What is a Function?") A **Function** is a named collection of operations that can be leveraged to perform complex logic and transformations. Each function is versioned for friendly maintenance: * Stored in ekoDB * Called by label or ID * Nested using `CallFunction` * Parameterized with dynamic values ### Function Structure[​](#function-structure "Direct link to Function Structure") | Field | Description | Required | | ------------- | ---------------------- | -------- | | `label` | Unique identifier | Yes | | `name` | Human-readable name | Yes | | `description` | What the function does | No | | `version` | Semantic version | No | | `parameters` | Input parameters | No | | `functions` | Array of operations | Yes | | `tags` | Categorization tags | No | ### Creating a Function[​](#creating-a-function "Direct link to Creating a Function") **Minimal Example (Required Fields Only):** ``` { "label": "get_user", "name": "Get User by ID", "functions": [ { "type": "FindOne", "collection": "users", "filter": { "id": "{{user_id}}" } } ] } ``` **Full Example (With Optional Fields):** ``` { "label": "user_onboarding", "name": "User Onboarding Flow", "description": "Complete new user setup with AI welcome message", "version": "1.0.0", "tags": ["onboarding", "users", "ai"], "parameters": { "email": { "type": "string", "required": true }, "name": { "type": "string", "required": true } }, "functions": [ { "type": "Insert", "collection": "users", "record": { "email": "{{email}}", "name": "{{name}}" } }, { "type": "Chat", "messages": [ { "role": "system", "content": "You are a friendly onboarding assistant" }, { "role": "user", "content": "Write a welcome message for {{name}}" } ], "model": "gpt-4" } ] } ``` ### Calling a Function[​](#calling-a-function "Direct link to Calling a Function") **From Client Library:** ``` const result = await client.callFunction("user_onboarding", { email: "user@example.com", name: "John Doe", }); ``` **From REST API:** ``` POST /api/functions/user_onboarding Content-Type: application/json { "email": "user@example.com", "name": "John Doe" } ``` **Response format:** A call returns `{ "records": [...], "stats": {...} }`. The record field values follow the same `use_typed_values` rule as `find` and `query`: with `use_typed_values` enabled they are wrapped (`{"type":"String","value":"..."}`), and with it disabled they are bare. A client configured for typed values therefore gets the same shape from `callFunction` as from a read. This wrapping is applied only on the response to an external caller. Inside the pipeline, and when one function calls another with `CallFunction`, records flow as bare native values so that `{{record.field}}` access keeps working in the calling function. A `Return` step may set `status_code` to control the HTTP status of the response (for example `201` after a create, or `404` when a lookup misses). The code sets the actual HTTP status; it does not appear in the response body. ### Function Nesting[​](#function-nesting "Direct link to Function Nesting") Functions can call other Functions for modularity using `CallFunction`: ``` { "label": "process_order", "functions": [ { "type": "CallFunction", "function_label": "validate_inventory", "params": { "product_id": "{{product_id}}" } }, { "type": "CallFunction", "function_label": "charge_payment", "params": { "amount": "{{amount}}" } }, { "type": "CallFunction", "function_label": "send_confirmation", "params": { "order_id": "{{order_id}}" } } ] } ``` **Execution:** ``` process_order ├─ validate_inventory │ ├─ Query products │ └─ Update inventory ├─ charge_payment │ └─ HttpRequest to Stripe └─ send_confirmation ├─ Chat (AI message) └─ HttpRequest to email API ``` ## Versioning[​](#versioning "Direct link to Versioning") ### Why Version Functions?[​](#why-version-functions "Direct link to Why Version Functions?") * **Evolution** - Update logic without breaking existing callers * **Rollback** - Revert to previous versions if issues arise * **Testing** - Test new versions alongside old ones * **Auditing** - Track changes over time ### Version Strategy[​](#version-strategy "Direct link to Version Strategy") ``` { "label": "calculate_pricing", "version": "2.0", // Increment on breaking changes "functions": [...] } ``` **Best Practices:** * Use semantic versioning: `major.minor.patch` * Major version: Breaking changes * Minor version: New features (backward compatible) * Patch version: Bug fixes ### Calling Specific Versions[​](#calling-specific-versions "Direct link to Calling Specific Versions") ``` // Call latest version await client.callFunction("calculate_pricing", params); // Call specific version (if supported) await client.callFunction("calculate_pricing", params, { version: "1.0" }); ``` ## AI Integration[​](#ai-integration "Direct link to AI Integration") ### Chat Functions[​](#chat-functions "Direct link to Chat Functions") Execute LLM completions with context: ``` { "type": "Chat", "messages": [ { "role": "system", "content": "You are a data analyst" }, { "role": "user", "content": "Analyze this data: {{query_results}}" } ], "model": "gpt-4", "temperature": 0.7, "max_tokens": 1000 } ``` **Supported Models:** * OpenAI: `gpt-4`, `gpt-4-turbo`, `gpt-3.5-turbo` * Anthropic: `claude-3-opus`, `claude-3-sonnet` * Perplexity: `pplx-7b-online`, `pplx-70b-online` ### Embedding Functions[​](#embedding-functions "Direct link to Embedding Functions") Generate vector embeddings for text: ``` { "type": "Embed", "input_field": "description", "output_field": "description_vector", "model": "text-embedding-ada-002" } ``` **Use Cases:** * Semantic search * Document similarity * Recommendation systems * Clustering ### RAG Workflows[​](#rag-workflows "Direct link to RAG Workflows") Build complete RAG pipelines by combining search and AI: ``` { "label": "semantic_search", "parameters": { "query": { "type": "string", "required": true } }, "functions": [ { "type": "TextSearch", "collection": "knowledge_base", "query_text": "{{query}}", "limit": 5 }, { "type": "Chat", "messages": [ { "role": "system", "content": "Answer using only the provided context" }, { "role": "user", "content": "Context: {{search_results}}\n\nQuestion: {{query}}" } ], "model": "gpt-4" } ] } ``` **Note:** For true vector search RAG, first generate embeddings for your documents using the Embed operation, then use VectorSearch to find similar content. ## External Integrations[​](#external-integrations-1 "Direct link to External Integrations") ### HTTP Requests[​](#http-requests "Direct link to HTTP Requests") Call external APIs from Functions: ``` { "type": "HttpRequest", "method": "POST", "url": "https://api.stripe.com/v1/charges", "headers": { "Authorization": "Bearer {{stripe_key}}", "Content-Type": "application/x-www-form-urlencoded" }, "body": { "amount": "{{amount}}", "currency": "usd", "source": "{{token}}" }, "output_field": "charge_result" } ``` **Response Handling:** The HTTP response is stored in the specified `output_field` (default: `http_response`): * **JSON responses** are automatically parsed into native Objects/Arrays * **Non-JSON responses** (HTML, plain text) are stored as Strings You can reference the response in subsequent stages using the field name specified in `output_field` (e.g., `{{charge_result}}` in the example above). **Use Cases:** * Payment processing (Stripe, PayPal) * Email services (SendGrid, Mailgun) * SMS notifications (Twilio) * Webhook triggers * Third-party API integration ## Performance Considerations[​](#performance-considerations "Direct link to Performance Considerations") ### Execution Time[​](#execution-time "Direct link to Execution Time") * **Operation overhead**: \~1-5ms per operation * **Network calls**: Variable (external APIs, AI) * **Database operations**: Sub-millisecond for queries * **Total Function time**: Sum of all operation times ### Optimization Strategies[​](#optimization-strategies "Direct link to Optimization Strategies") 1. **Batch operations** instead of loops when possible 2. **Limit result sets** with filters and projections 3. **Filter early** - Reduce data before expensive operations 4. **Cache Function definitions** (auto-cached by ekoDB) 5. **Minimize external HTTP calls** ### Stale-While-Revalidate (SWR) Pattern[​](#stale-while-revalidate-swr-pattern "Direct link to Stale-While-Revalidate (SWR) Pattern") Functions are perfect for the **SWR pattern**, which dramatically improves perceived performance by serving cached results while fetching fresh data in the background. **How it works:** ``` 1. Client requests Function execution 2. Return cached result immediately (if exists) 3. Execute Function in background 4. Update cache with fresh result 5. Next request gets updated data ``` **Benefits:** * ⚡ **Instant response** - Users see data immediately * 🔄 **Always fresh** - Background updates keep data current * 📉 **Reduced load** - Fewer Function executions * 🎯 **Better UX** - No loading spinners for cached data **When to use:** * ✅ Dashboard summaries that don't change frequently * ✅ User profile data (name, email, settings) * ✅ Product catalogs with periodic updates * ✅ Analytics data that's expensive to compute * ❌ Real-time data (stock prices, live scores) * ❌ User-specific transactional data (cart, orders) **Example pattern:** ``` // Client-side SWR with Functions async function getUserDashboard(userId: string) { // 1. Return cached data immediately const cached = cache.get(`dashboard:${userId}`); if (cached) { // Serve stale data instantly displayDashboard(cached); // 2. Revalidate in background client.callFunction("user_dashboard", { user_id: userId }).then((fresh) => { cache.set(`dashboard:${userId}`, fresh); displayDashboard(fresh); // Update UI with fresh data }); return cached; } // 3. No cache - fetch and display const data = await client.callFunction("user_dashboard", { user_id: userId }); cache.set(`dashboard:${userId}`, data); return data; } ``` **Server-side caching:** Functions can also implement internal caching for even better performance: ``` { "label": "expensive_analytics", "functions": [ { "type": "Query", "collection": "events", "filter": { "type": "Condition", "content": { "field": "date", "operator": "Eq", "value": "{{today}}" } } }, { "type": "Group", "by_fields": ["category"], "functions": [{ "output_field": "count", "operation": "Count" }] } ] } ``` Call this Function with a cache key and TTL, and ekoDB can cache results server-side for all clients. Example Code **SWR Pattern Examples in all languages:** * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/swr_pattern.rs)**: `swr_pattern.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/swr_pattern.py)**: `swr_pattern.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_swr_pattern.ts)**: `client_swr_pattern.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/swr_pattern.go)**: `swr_pattern.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/SwrPattern.kt)**: `SwrPattern.kt` **Edge Cache Pattern** - Use ekoDB as an edge cache with external API caching: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_edge_cache.rs)**: `client_edge_cache.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_edge_cache.py)**: `client_edge_cache.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_edge_cache.ts)**: `client_edge_cache.ts` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_edge_cache.js)**: `client_edge_cache.js` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_edge_cache.go)**: `client_edge_cache.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientEdgeCache.kt)**: `ClientEdgeCache.kt` ### Best Practices[​](#best-practices "Direct link to Best Practices") **DO:** * ✅ Keep Functions focused and single-purpose * ✅ Use meaningful labels and descriptions * ✅ Document parameters clearly * ✅ Version Functions properly * ✅ Test with sample data before production **DON'T:** * ❌ Create mega-Functions with 50+ operations * ❌ Use Functions for simple single operations * ❌ Hard-code values (use parameters) * ❌ Nest Functions more than 3-4 levels deep * ❌ Ignore error handling ## Error Handling[​](#error-handling "Direct link to Error Handling") ### Operation-Level Errors[​](#operation-level-errors "Direct link to Operation-Level Errors") If an operation fails: 1. Execution stops immediately 2. Error details are returned 3. No rollback (unless in transaction) 4. Caller receives error response ``` { "error": true, "operation_index": 2, "operation_type": "Chat", "message": "API rate limit exceeded", "details": {...} } ``` ### Error Recovery[​](#error-recovery "Direct link to Error Recovery") To catch a failing step and run fallback logic, use `TryCatch` (the step's error is bound to `output_error_field`): ``` { "type": "TryCatch", "try_functions": [ { "type": "Chat", "messages": [{ "role": "user", "content": "{{question}}" }] } ], "catch_functions": [ { "type": "Insert", "collection": "error_log", "record": { "error": "{{error}}" } } ], "output_error_field": "error" } ``` `If` branches on a `FunctionCondition` evaluated against the working records — note `condition` is an object (not a string), and the branches are `then_functions` / `else_functions`: ``` { "type": "If", "condition": { "type": "HasRecords" }, "then_functions": [ { "type": "Insert", "collection": "audit", "record": { "event": "found" } } ], "else_functions": [] } ``` ### Transactions[​](#transactions-1 "Direct link to Transactions") Wrap operations in transactions for atomicity: ``` { "transaction_config": { "enabled": true }, "functions": [ { "type": "Update", "collection": "inventory", "..." }, { "type": "Insert", "collection": "orders", "..." } ] } ``` If any operation fails, all changes are rolled back. ## Security[​](#security "Direct link to Security") ### Permission Model[​](#permission-model "Direct link to Permission Model") Stored functions run with **definer (instance) rights**, similar to a SQL `SECURITY DEFINER` routine: * **Calling a function requires authentication** — the caller must present a valid token. Saving or replacing a function definition is admin-gated. * **Once authenticated, the function's steps execute with the instance's full access**, not the caller's per-collection or per-field API-key scope. A function can therefore read and write any collection in the instance, which is what lets you expose a narrow, audited operation (for example `items_create`) to a caller who has no direct write access to the underlying collection. * Because ekoDB is deployed one instance per customer, definer rights stay within a single tenant's data boundary. This is the intended model: you control access by choosing **which functions exist and who may call them**, not by relying on the caller's collection grants to scope a function's internals. Per-call collection-permission scoping (grant-on-create plus enforce-on-call) is on the roadmap as an additional layer. ### Parameter Validation[​](#parameter-validation "Direct link to Parameter Validation") All parameters are validated before execution: * Type checking (string, number, boolean, array, object) * Required field validation * Enum value checking * Custom validation rules ### Injection Prevention[​](#injection-prevention "Direct link to Injection Prevention") ekoDB automatically: * ✅ Sanitizes all parameter inputs * ✅ Prevents NoSQL injection * ✅ Escapes special characters * ✅ Validates JSON structure **Safe:** ``` { "filter": { "email": "{{user_email}}" } } ``` User cannot inject malicious queries. ### JWT Signing and Verification[​](#jwt-signing-and-verification "Direct link to JWT Signing and Verification") `JwtSign` and `JwtVerify` support both symmetric and asymmetric algorithms via the `algorithm` parameter, which defaults to `HS256`: | Family | Algorithms | `secret` holds | | ----------------- | ------------------------- | ---------------------------------------------------------------- | | HMAC | `HS256`, `HS384`, `HS512` | A shared secret | | RSA (PKCS#1 v1.5) | `RS256`, `RS384`, `RS512` | A PEM private key to sign, the matching PEM public key to verify | | RSA-PSS | `PS256`, `PS384`, `PS512` | Same as RSA above | | ECDSA | `ES256`, `ES384` | A PEM private/public key pair | | EdDSA | `EdDSA` | A PEM private/public key pair | `none` is rejected outright — there is no way to issue or accept an unsigned token. Asymmetric algorithms are what let you verify a token you didn't issue yourself without standing up a separate signing service — a GitHub App installation token or a Google service-account JWT both arrive signed `RS256`, and `JwtVerify` checks them against the issuer's public key directly. ``` { "type": "JwtVerify", "algorithm": "RS256", "token_field": "installation_token", "secret": "{{env.GITHUB_APP_PUBLIC_KEY}}", "output_field": "claims" } ``` **Key formats.** RSA (`RS*` / `PS*`) accepts both PKCS#1 (`BEGIN RSA PRIVATE KEY`, the format GitHub Apps hand out) and PKCS#8 (`BEGIN PRIVATE KEY`). **`ES*` and `EdDSA` are PKCS#8 only.** The SEC1 form (`BEGIN EC PRIVATE KEY`) — what a plain `openssl ecparam -genkey -name prime256v1` produces — is rejected with a clear error rather than silently mishandled. Convert an existing SEC1 key with: ``` openssl pkcs8 -topk8 -nocrypt -in ec_key_sec1.pem -out ec_key_pkcs8.pem ``` **Both `secret` and `algorithm` must be operator-supplied, not caller-supplied.** Neither accepts a call-time parameter — `secret` may be a literal or `{{env.NAME}}`, and `algorithm` must be a literal (not even `{{env.NAME}}`). This is enforced because a caller who could choose either could forge a token: naming `HS256` against a stage whose `secret` holds an asymmetric *public* key turns that public key into an HMAC secret, and a caller-supplied signing key needs no forging at all. ## Use Cases[​](#use-cases "Direct link to Use Cases") ### 1. User Authentication Flow[​](#1-user-authentication-flow "Direct link to 1. User Authentication Flow") ``` { "label": "get_user_by_email", "parameters": { "email": { "type": "string", "required": true } }, "functions": [ { "type": "FindOne", "collection": "users", "key": "email", "value": "{{email}}" } ] } ``` ### 2. E-commerce Order Processing[​](#2-e-commerce-order-processing "Direct link to 2. E-commerce Order Processing") ``` { "label": "create_order", "parameters": { "user_id": { "type": "string", "required": true }, "product_id": { "type": "string", "required": true } }, "functions": [ { "type": "FindById", "collection": "products", "record_id": "{{product_id}}" }, { "type": "Insert", "collection": "orders", "record": { "user_id": "{{user_id}}", "product_id": "{{product_id}}", "status": "pending" } } ] } ``` ### 3. AI Content Moderation[​](#3-ai-content-moderation "Direct link to 3. AI Content Moderation") ``` { "label": "moderate_content", "parameters": { "content": { "type": "string", "required": true } }, "functions": [ { "type": "Chat", "messages": [ { "role": "system", "content": "Analyze if content is appropriate and respond with 'SAFE' or 'UNSAFE'" }, { "role": "user", "content": "{{content}}" } ], "model": "gpt-4" } ] } ``` ### 4. Data Analytics Pipeline[​](#4-data-analytics-pipeline "Direct link to 4. Data Analytics Pipeline") ``` { "label": "event_summary", "parameters": { "date": { "type": "string", "required": true } }, "functions": [ { "type": "Query", "collection": "events", "filter": { "type": "Condition", "content": { "field": "date", "operator": "Eq", "value": "{{date}}" } } }, { "type": "Count", "output_field": "total_events" } ] } ``` ## Comparison with Alternatives[​](#comparison-with-alternatives "Direct link to Comparison with Alternatives") ### vs. Stored Procedures (SQL)[​](#vs-stored-procedures-sql "Direct link to vs. Stored Procedures (SQL)") | Feature | ekoDB Functions | SQL Stored Procedures | | -------------- | ------------------ | --------------------- | | Language | JSON (declarative) | SQL (procedural) | | Versioning | Built-in | Manual | | AI Integration | Native | None | | External APIs | Yes | Limited | | Portability | JSON format | Database-specific | ### vs. Cloud Functions (AWS Lambda, etc.)[​](#vs-cloud-functions-aws-lambda-etc "Direct link to vs. Cloud Functions (AWS Lambda, etc.)") | Feature | ekoDB Functions | Cloud Functions | | ---------- | -------------------- | ------------------ | | Execution | Server-side in ekoDB | Separate compute | | Network | Single call | Multiple calls | | State | Direct DB access | Must connect to DB | | Deployment | JSON definition | Code deployment | | Cost | Included | Per-invocation | ### vs. Application Code[​](#vs-application-code "Direct link to vs. Application Code") | Feature | ekoDB Functions | Application Code | | ----------- | --------------- | -------------------- | | Location | Server-side | Client-side | | Reusability | All clients | Single codebase | | Performance | Low latency | Multiple round-trips | | Versioning | Built-in | Git/deployment | | Testing | Isolated | Full app context | ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * **[Advanced Operations - Functions](/client-libraries/advanced-operations.md#functions)** - Practical usage examples * **[Transactions Architecture](/reference/transaction-architecture.md)** - ACID transaction details * **[White Paper](/white-paper.md)** - Overall ekoDB architecture * **[Basic Operations](/basic-operations.md)** - REST API reference ## Next Steps[​](#next-steps "Direct link to Next Steps") ### 📚 Explore Function Examples[​](#-explore-function-examples "Direct link to 📚 Explore Function Examples") **Complete working examples in all languages:** * **Function Composition Examples** - Full-featured examples: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_function_composition.rs)**: `client_function_composition.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_function_composition.py)**: `client_function_composition.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_function_composition.ts)**: `client_function_composition.ts` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_function_composition.js)**: `client_function_composition.js` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_functions.go)**: `client_functions.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientFunctionComposition.kt)**: `ClientFunctionComposition.kt` * **RAG & AI Examples** - Retrieval-augmented generation: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/rag_conversation_system.rs)**: `rag_conversation_system.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/rag_conversation_system.py)**: `rag_conversation_system.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/rag_conversation_system.ts)**: `rag_conversation_system.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/rag_conversation_system.go)**: `rag_conversation_system.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/RagConversationSystem.kt)**: `RagConversationSystem.kt` * **SWR Pattern** - Stale-while-revalidate with Functions: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/swr_pattern.rs)**: `swr_pattern.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/swr_pattern.py)**: `swr_pattern.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_swr_pattern.ts)**: `client_swr_pattern.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/swr_pattern.go)**: `swr_pattern.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/SwrPattern.kt)**: `SwrPattern.kt` ### Learning Path[​](#learning-path "Direct link to Learning Path") 1. **Learn by example** - Browse the GitHub examples above 2. **Start simple** - Create a basic Query Function 3. **Add parameters** - Make Functions dynamic 4. **Compose Functions** - Combine Functions for workflows 5. **Integrate AI** - Add Chat and Embed Functions **Need help?** [Submit a support ticket](https://app.ekodb.io/support) or [open an issue](https://github.com/ekoDB/ekodb-client/issues) --- # Performance Benchmarks > **TL;DR:** ekoDB is measured on the industry-standard YCSB suite against PostgreSQL, MongoDB, MySQL and Redis, **while encrypting and decrypting every record**, against engines encrypting nothing. A single Rust binary covers both the durable and cache tiers, plus auth, encryption, search and real-time subscriptions. Across every head-to-head throughput matchup in this run, ekoDB leads **19 of 20**. The single exception is read-only against PostgreSQL (0.9x). **ekoDB v0.70.1 (5af3aca9)**, run 2026-09-02, 1,000,000 records, 64 threads. Every figure on this page is rendered from [the published artifact](https://storage.googleapis.com/ekodb-cdn/benchmarks/v0.70.1/ycsb.json), not typed in. **Disclosed with this run:** * ekoDB encrypts at rest; the competitors in this run do not. Read-side results are encrypted-vs-plaintext, not like-for-like ## When to Use ekoDB (and When Not To)[​](#when-to-use-ekodb-and-when-not-to "Direct link to When to Use ekoDB (and When Not To)") **ekoDB excels when you need:** write-heavy workloads with durability, read-heavy throughput, sub-millisecond latencies, CPU-efficient deployments, or a unified platform replacing multiple services (document store + KV + FTS + vector search + auth + real-time subs) in a single binary. **Consider alternatives when:** you need specialized OLAP with complex window functions and CTEs (ClickHouse, DuckDB). Quick Decision Guide *Based on YCSB benchmarks: 1M records, 64 threads, Fast storage mode. See [Results Summary](#ycsb-results-summary) for numbers.* | Your Workload | Best Choice | Why | | -------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Write-heavy + durable** | **ekoDB** | Fastest across all competitors with sub-millisecond latency | | **Read-heavy + durable** | **ekoDB**, with read-only a coin toss | Leads MongoDB and MySQL on every workload, and PostgreSQL on mixed reads. On 100% read-only the two trade the lead run to run — see [Results Summary](#ycsb-results-summary) | | **Non-durable (any mix)** | **ekoDB** | Competitive or leading, while carrying auth/encryption overhead competitors skip | | **Low-latency (any mode)** | **ekoDB** | Consistent sub-millisecond across all workloads | | **CPU cost-sensitive** | **ekoDB** | Better efficiency across durable and non-durable modes | | **Unified platform** | **ekoDB** | Single binary replaces 4–5 services (auth, search, cache, DB) | *** ## Why ekoDB's Performance Is Remarkable[​](#why-ekodbs-performance-is-remarkable "Direct link to Why ekoDB's Performance Is Remarkable") ekoDB isn't just fast. It's fast *despite doing more work per request* than competitors. Understanding this context makes the benchmark results more meaningful. ### The Overhead ekoDB Carries[​](#the-overhead-ekodb-carries "Direct link to The Overhead ekoDB Carries") Every ekoDB request includes capabilities that competitors skip entirely: | Layer | What Happens | Competitors | | -------------- | ------------------------------------------------------- | --------------------------------- | | **Security** | JWT/API key auth, AES-GCM encryption on all stored data | Redis: none, PG: separate | | **Data** | Schema validation, automatic indexing | PG: yes, Redis: none | | **Search** | Full-text + vector search index updates | Requires Elasticsearch + Pinecone | | **Durability** | Configurable persistence guarantees per operation | PG: batched, Redis: async | | **Real-time** | Subscription notifications | Requires additional services | Durability Configuration ekoDB supports two durability modes via the `durable_operations` config setting: | Mode | Setting | Behavior | Use Case | | --------------------- | --------------------------- | ------------------------------------------------------- | ----------------------------------------------- | | **Durable** (default) | `durable_operations: true` | Every confirmed write persisted to disk before response | Most workloads, guaranteed persistence | | **Non-Durable** | `durable_operations: false` | Writes persisted asynchronously | High-throughput, eventual durability acceptable | ### Durability: The Hidden Performance Tax[​](#durability-the-hidden-performance-tax "Direct link to Durability: The Hidden Performance Tax") Most benchmarks don't match durability settings, making some databases appear faster than they are in production. A database that skips durability guarantees will always benchmark faster — but that speed comes at the cost of data loss on crash. All ekoDB benchmarks use equivalent durability settings across databases for a fair comparison. *** ## YCSB Results Summary[​](#ycsb-results-summary "Direct link to YCSB Results Summary") Industry-standard [YCSB](https://github.com/brianfrankcooper/YCSB) benchmarks: 1M records, 1M operations, 64 threads, Fast storage mode. The run, its version and its commit are stated above and rendered from the artifact rather than written here, so this paragraph cannot drift from the tables under it. Reproduce with `make bench-ycsb-compare`. Each engine is compared **in the mode it is actually run in**. Forcing a cache into per-operation fsync, or a system of record out of it, produces a number nobody can act on. ### Run conditions[​](#run-conditions "Direct link to Run conditions") Throughput only means something alongside what each engine was actually doing per operation, and how it was configured. **Both tables below are rendered from the run's own recorded conditions**, not from a description of them — a benchmark page is exactly where a setting should never be restated by hand. | Engine | Durability setting recorded for this run | | --------------------- | ---------------------------------------------------------------------------------------- | | **ekoDB** | `durable_operations=true, group_commits=true, group_commit_delay_ms=2` | | **MongoDB** | `w=1, journal=true, journalCommitInterval=2ms` | | **PostgreSQL** | `synchronous_commit=on, fsync=on, commit_delay=2000µs, commit_siblings=2` | | **MySQL** | `innodb_flush_log_at_trx_commit=1, sync_binlog=1, binlog_group_commit_sync_delay=2000µs` | | **ekoDB (Key-Value)** | `kv_durable_operations=true, group_commits=true, group_commit_delay_ms=2` | | **Redis** | `appendonly=yes, appendfsync=always` | | Engine | Encryption at rest recorded for this run | | --------------------- | ------------------------------------------------------------ | | **ekoDB** | `encrypt_ids=true, encrypt_manifests=true, encrypt_wal=true` | | **MongoDB** | `none` | | **PostgreSQL** | `none` | | **MySQL** | `none` | | **ekoDB (Key-Value)** | `encrypt_ids=true, encrypt_manifests=true, encrypt_wal=true` | | **Redis** | `none` | This run verified how each compared engine was configured for encryption at rest, so the comparison above is stated rather than assumed. What each engine offers here is a property of the engine rather than of the run, so it is stated once and linked: | Engine | Encryption at rest available? | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | | **ekoDB** | Built in. Record-body encryption is unconditional; ids, WAL and manifests are on by default. | | PostgreSQL | Not in core — [transparent data encryption is not implemented](https://wiki.postgresql.org/wiki/Transparent_Data_Encryption). | | MongoDB | [Enterprise only](https://www.mongodb.com/docs/manual/core/security-encryption-at-rest/). | | MySQL | Available in Community via [`component_keyring_file`](https://dev.mysql.com/doc/refman/8.0/en/keyring-component-installation.html). | | Redis | [No built-in at-rest encryption](https://redis.io/faq/doc/2j6kk4yf86/does-redis-support-encryption/) in open source. | Every ekoDB figure on this page is measured **while encrypting and decrypting every record**. Whether that is a like-for-like comparison depends on what the run verified, which the table above states rather than assumes. ### Durable tier: systems of record[​](#durable-tier-systems-of-record "Direct link to Durable tier: systems of record") Every engine on **per-commit fsync** with matched group-commit windows. The exact setting each one ran with is in [run conditions](#run-conditions) below, read from the artifact rather than described. | Workload | ekoDB | MongoDB | PostgreSQL | MySQL | | --------------------------------------------- | ---------- | ------- | ----------- | ------ | | **Update-heavy**
50% reads, 50% updates | **31,530** | 10,849 | 3,658 | 2,641 | | **Read-heavy**
95% reads, 5% updates | **83,913** | 68,451 | 33,992 | 27,166 | | **Read-only**
100% reads, point lookups | 99,492 | 74,140 | **104,427** | 58,620 | | **Read-latest**
95% reads, 5% inserts | **84,817** | 71,362 | 70,721 | 45,328 | | **Read-modify-write**
50% reads, 50% RMW | **35,373** | 10,060 | 3,479 | 3,352 | operations per second, higher is better. Best in each row in bold. | Workload | vs MongoDB | vs PostgreSQL | vs MySQL | | --------------------------------------------- | ------------------ | --------------------- | ---------------- | | **Update-heavy**
50% reads, 50% updates | **2.9x** | **8.6x** | **11.9x** | | **Read-heavy**
95% reads, 5% updates | **1.2x** | **2.4x** | **3x** | | **Read-only**
100% reads, point lookups | **1.3x** | 0.9x | **1.6x** | | **Read-latest**
95% reads, 5% inserts | **1.1x** | **1.1x** | **1.8x** | | **Read-modify-write**
50% reads, 50% RMW | **3.5x** | **10.1x** | **10.5x** | ekoDB throughput divided by the other engine's. Above 1 means ekoDB is ahead and is shown in bold; below 1 means the other engine is. Where PostgreSQL leads, it leads on **pure point-lookup reads**, and that is the one place the encryption trade shows. On write-heavy workloads the fsync dominates and ekoDB's per-record encryption disappears into it; on 1M point-lookups by key there is no fsync to hide behind, so decrypting every record is the whole difference. **The honest statement:** ekoDB serves *encrypted* point reads at close to the throughput of an *unencrypted* PostgreSQL instance — while also carrying auth, search indexing and subscription dispatch that PostgreSQL is not doing. ### Cache tier: in-memory stores[​](#cache-tier-in-memory-stores "Direct link to Cache tier: in-memory stores") ekoDB's key-value path against Redis. **Read the configuration before the ratio**: the multiple here moves by an order of magnitude depending on whether Redis is running durable, and the run's actual setting is in [run conditions](#run-conditions). | Workload | ekoDB (Key-Value) | Redis | | --------------------------------------------- | ----------------- | ------ | | **Update-heavy**
50% reads, 50% updates | **38,130** | 5,966 | | **Read-heavy**
95% reads, 5% updates | **104,264** | 6,414 | | **Read-only**
100% reads, point lookups | **101,895** | 92,267 | | **Read-latest**
95% reads, 5% inserts | **89,261** | 5,950 | | **Read-modify-write**
50% reads, 50% RMW | **27,359** | 4,075 | operations per second, higher is better. Best in each row in bold. | Workload | vs Redis | | --------------------------------------------- | ---------------- | | **Update-heavy**
50% reads, 50% updates | **6.3x** | | **Read-heavy**
95% reads, 5% updates | **16.2x** | | **Read-only**
100% reads, point lookups | **1.1x** | | **Read-latest**
95% reads, 5% inserts | **15x** | | **Read-modify-write**
50% reads, 50% RMW | **6.7x** | ekoDB throughput divided by the other engine's. Above 1 means ekoDB is ahead and is shown in bold; below 1 means the other engine is. Why this number differs from the one on ekodb.io Both surfaces read the same artifact, so when they show different multiples against Redis it is because a run configured Redis differently, not because one of them is wrong. Redis with `appendonly=no` is how it is normally deployed and is a fair cache-tier comparison; Redis with `appendfsync=always` answers a different question — what durability costs Redis — and produces a far larger multiple. The table above states which one this run used. ### Tail latency and CPU[​](#tail-latency-and-cpu "Direct link to Tail latency and CPU") Throughput is one axis and not the one a deployment is usually sized on. These are the same cells measured the other two ways, with the best value in each row in bold, whichever engine holds it. **Each tier is shown separately**, because a durable p99 and a cache p99 are not comparable figures and a table that followed the cache section without saying so would read as though they were. #### Durable tier — documents, JSON, and rows[​](#durable-tier--documents-json-and-rows "Direct link to Durable tier — documents, JSON, and rows") | Workload | ekoDB | MongoDB | PostgreSQL | MySQL | | --------------------------------------------- | ---------- | ------- | ---------- | ------ | | **Update-heavy**
50% reads, 50% updates | **0.90ms** | 6.48ms | 5.56ms | 5.97ms | | **Read-heavy**
95% reads, 5% updates | 3.07ms | 2.95ms | **0.59ms** | 1.35ms | | **Read-only**
100% reads, point lookups | 2.59ms | 4.85ms | **1.19ms** | 5.95ms | | **Read-latest**
95% reads, 5% inserts | 3.28ms | 2.62ms | **0.94ms** | 7.85ms | | **Read-modify-write**
50% reads, 50% RMW | **1.12ms** | 6.50ms | 6.01ms | 5.36ms | p99 latency, lower is better. Best in each row in bold. | Workload | ekoDB | MongoDB | PostgreSQL | MySQL | | --------------------------------------------- | -------- | ------- | ---------- | ------- | | **Update-heavy**
50% reads, 50% updates | 145% | 172% | 70% | **54%** | | **Read-heavy**
95% reads, 5% updates | 187% | 468% | **159%** | 200% | | **Read-only**
100% reads, point lookups | **171%** | 476% | 328% | 316% | | **Read-latest**
95% reads, 5% inserts | **178%** | 460% | 295% | 318% | | **Read-modify-write**
50% reads, 50% RMW | 222% | 151% | 82% | **74%** | mean CPU, 100% is one saturated core, lower is better. Best in each row in bold. #### Cache tier — key-value[​](#cache-tier--key-value "Direct link to Cache tier — key-value") | Workload | ekoDB (Key-Value) | Redis | | --------------------------------------------- | ----------------- | ---------- | | **Update-heavy**
50% reads, 50% updates | **0.96ms** | 21.62ms | | **Read-heavy**
95% reads, 5% updates | **1.32ms** | 22.28ms | | **Read-only**
100% reads, point lookups | 1.52ms | **1.09ms** | | **Read-latest**
95% reads, 5% inserts | **6.73ms** | 20.68ms | | **Read-modify-write**
50% reads, 50% RMW | **5.46ms** | 21.73ms | p99 latency, lower is better. Best in each row in bold. | Workload | ekoDB (Key-Value) | Redis | | --------------------------------------------- | ----------------- | ------- | | **Update-heavy**
50% reads, 50% updates | 134% | **25%** | | **Read-heavy**
95% reads, 5% updates | 290% | **24%** | | **Read-only**
100% reads, point lookups | 200% | **76%** | | **Read-latest**
95% reads, 5% inserts | 221% | **23%** | | **Read-modify-write**
50% reads, 50% RMW | 175% | **25%** | mean CPU, 100% is one saturated core, lower is better. Best in each row in bold. ### What you are comparing against[​](#what-you-are-comparing-against "Direct link to What you are comparing against") The tables read as engine against engine, which understates the difference. A production PostgreSQL reaching the same capability is PostgreSQL **plus** an encryption layer, an auth layer, a cache, a search index, and the code joining them. Each is a component someone writes, deploys, patches, monitors and is paged about. ekoDB is one binary, with encryption beneath all of it rather than bolted alongside. The measured throughput above **already includes** that overhead; a competitor number that excludes it is not measuring the same system, only the same benchmark. ### Analysis: Why ekoDB Leads[​](#analysis-why-ekodb-leads "Direct link to Analysis: Why ekoDB Leads") The 1M record, 64-thread YCSB results reveal three architectural advantages: **1. Write Performance Under Durability** ekoDB maintains fast writes even with full persistence guarantees. Where competitors see throughput collapse when durability is enabled (Redis drops to single-digit K ops/sec), ekoDB's write path is designed for durable workloads from the ground up — not bolted on as an afterthought. **2. Scalable Concurrency** At 64 threads, ekoDB scales efficiently with minimal lock contention. Both the Key-Value and Collections engines distribute write load across cores, maintaining consistent performance as thread count increases. **3. CPU Efficiency** Why CPU Efficiency Matters On cloud platforms, you pay for CPU cores. A database that delivers 50K ops/sec using 200% CPU (2 cores) is more cost-effective than one delivering 60K ops/sec using 600% CPU (6 cores). **Efficiency = ops/sec ÷ CPU%**: Higher is better. Lower CPU per operation means smaller instances, lower cloud costs, and more headroom for co-located services. The Bottom Line ekoDB achieves these results while carrying [significantly more overhead per request](#the-overhead-ekodb-carries) than every competitor — auth, encryption, indexing, and real-time subscriptions on every operation. A \~50MB Rust binary with zero garbage collection pauses. *** ## Detailed YCSB Benchmarks by Storage Mode[​](#detailed-ycsb-benchmarks-by-storage-mode "Direct link to Detailed YCSB Benchmarks by Storage Mode") These are earlier runs, kept deliberately **The current figures are the derived ones above.** Everything in this section is hand-recorded from earlier comparison runs and is **not** regenerated from the published artifact, so it will not move when a new run lands. It is kept rather than deleted because it covers configurations the current artifact does not measure — **fast mode non-durable**, and **balanced mode** — and those are real measurements that exist nowhere else. Each block names the run it came from. Where a configuration appears both here and above, the derived version above is the current one. Industry-standard [YCSB (Yahoo! Cloud Serving Benchmark)](https://github.com/brianfrankcooper/YCSB) results comparing ekoDB against PostgreSQL, MongoDB, MySQL, Redis, and ekoDB (Key-Value) with matching durability settings. YCSB Workloads Explained | Workload | Mix | Real-World Example | | -------- | --------------------- | --------------------------------------- | | **A** | 50% read, 50% update | Session stores, shopping carts | | **B** | 95% read, 5% update | Social media profiles, photo tagging | | **C** | 100% read | Product catalogs, configuration lookups | | **D** | 95% read, 5% insert | News feeds, activity streams | | **F** | 50% read-modify-write | Bank transactions, inventory counters | Test Configuration * **Hardware:** Apple M1 Max, 64GB RAM, NVMe SSD * **Scale:** 1,000,000 records | 1,000,000 operations | 64 threads * **Batch Size:** 1 operation per request (raw network performance test) * **Databases:** 6 databases tested — ekoDB (Collections), ekoDB (Key-Value), PostgreSQL 15, MongoDB 6.0, MySQL 8.0, Redis 7.x All databases tested on the same machine, same day, for accurate comparison per configuration. Driver Asymmetry PostgreSQL benchmarks use **optimized JDBC drivers** with connection pooling, prepared statement caching, and batch optimizations built into the driver layer. These are production-grade drivers tuned over decades. ekoDB benchmarks use **raw TCP protocol** with a basic YCSB client: no driver-level optimizations, no connection pooling magic, no prepared statement cache. ekoDB carries this disadvantage into every figure on the page. The current record is counted from the artifact at the top of this page rather than restated here, so the two cannot disagree. The exception is read-only workload C, covered in [Results Summary](#ycsb-results-summary). *** ### Fast Mode (Recommended)[​](#fast-mode-recommended "Direct link to Fast Mode (Recommended)") > **Note:** All benchmarks include AES-GCM encryption on all stored data. * Fast + Durable * Fast + Non-Durable Source run Everything from here to the end of this section is from the **2026-02-16** comparison run, which put Redis in `appendfsync=always`. That configuration is kept because it answers a specific question — what Redis costs you when you demand durability from it — but it is **not** how Redis is normally deployed, so it is not the basis for the headline comparison at the top of this page, which is rendered from the current published artifact and names its own run. **Storage Mode:** Fast | **Durability:** Durable (guaranteed persistence) #### Durability Settings[​](#durability-settings "Direct link to Durability Settings") | Database | Durable Setting | fsync Behavior | Group Commit Delay | | -------------- | ------------------------------------------------- | ---------------------------------- | ---------------------------------------- | | **ekoDB** | `durable_operations=true` | WAL fsync, `group_commits=true` | 2ms (explicit) | | **PostgreSQL** | `synchronous_commit=on, fsync=on` | Per-commit WAL fsync | `commit_delay=2000us, commit_siblings=2` | | **MongoDB** | `w=1, journal=true` | WiredTiger journal fsync | `journalCommitInterval=2` | | **MySQL** | `innodb_flush_log_at_trx_commit=1, sync_binlog=1` | Per-commit redo log + binlog fsync | `binlog_group_commit_sync_delay=2000us` | | **Redis** | `appendfsync=always` | Per-operation AOF fsync | N/A (no group commit) | This is the recommended production configuration. Every confirmed write is persisted to disk before the client receives a response — no data loss on crash, power failure, or unexpected shutdown. In real-world terms: a session store handling 39K mixed read/update operations per second, a product catalog serving 124K lookups per second, or a financial ledger processing 38K atomic read-modify-write transactions per second — all with full durability, encryption, and auth on every operation. #### Throughput (ops/sec)[​](#throughput-opssec "Direct link to Throughput (ops/sec)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ----------- | ----------- | ----------- | ---------- | | **ekoDB (Key-Value)** | **39,369** | 112,854 | 116,442 | **127,389** | **37,929** | | **ekoDB (Collections)** | 36,964 | **115,128** | **123,655** | 118,315 | 36,937 | | MongoDB | 10,511 | 72,327 | 77,095 | 76,017 | 10,130 | | MySQL | 2,505 | 26,730 | 63,016 | 45,409 | 2,324 | | PostgreSQL | 5,420 | 54,555 | 107,227 | 94,949 | 4,790 | | Redis | 6,022 | 6,403 | 103,864 | 5,999 | 4,200 | **Key findings** — all figures in this subsection are from the **2026-02-16** run described in the note above, not the derived summary at the top of this page: * **ekoDB leads every workload in that run**, both paths scoring **A grade** (84 and 81 out of 100) * **Workload A** (50% read/50% update): ekoDB (Key-Value) **7.3x faster** than PostgreSQL, **3.7x faster** than MongoDB * **Workload B** (95% read/5% update): ekoDB (Collections) **2.1x faster** than PostgreSQL, **1.6x faster** than MongoDB * **Workload C** (100% read): ekoDB (Collections) **15% faster** than PostgreSQL (124K vs 107K), **1.6x faster** than MongoDB * **Workload D** (95% read latest/5% insert): ekoDB (Key-Value) **127K — fastest across all databases and workloads**, 1.3x faster than PostgreSQL, 1.7x faster than MongoDB * **Workload F** (Read-Modify-Write): ekoDB (Key-Value) **7.9x faster** than PostgreSQL, **3.7x faster** than MongoDB * **Redis struggles with durable writes** (`appendfsync=always`): 10ms+ latency on non-read workloads, only 4–6K ops/sec on A/D/F #### Average Latency (ms)[​](#average-latency-ms "Direct link to Average Latency (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Key-Value)** | 0.53 | 0.46 | 0.59 | 0.40 | 0.43 | | **ekoDB (Collections)** | **0.43** | 0.40 | 0.45 | **0.39** | **0.41** | | MongoDB | 1.17 | 0.44 | 0.79 | 0.43 | 1.38 | | MySQL | 1.54 | 0.25 | 0.94 | 0.58 | 1.87 | | PostgreSQL | 1.01 | **0.17** | **0.56** | 0.31 | 1.16 | | Redis | 10.60 | 9.96 | 0.61 | 10.12 | 10.10 | **Key findings:** * ekoDB maintains **sub-millisecond latencies** across all workloads (0.39–0.59ms) * **Lowest latency** on write-heavy Workloads A (0.43ms) and F (0.41ms) * PostgreSQL has lower latency on read-heavy workloads B (0.17ms) and C (0.56ms) * **Redis latency explodes** on write-heavy workloads: 10.60ms on Workload A vs ekoDB's 0.43ms — a 24x difference due to per-operation AOF fsync #### Tail Latency — P95 (ms)[​](#tail-latency--p95-ms "Direct link to Tail Latency — P95 (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB** | **0.75** | **1.38** | **1.72** | **1.64** | **1.37** | | MongoDB | 6.16 | 1.97 | 2.77 | 1.86 | 5.15 | | PostgreSQL | 3.56 | 0.44 | 1.06 | 1.24 | 3.88 | #### Tail Latency — P99 (ms)[​](#tail-latency--p99-ms "Direct link to Tail Latency — P99 (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB** | **1.18** | 2.50 | **3.14** | 3.11 | **3.46** | | MongoDB | 12.03 | 3.21 | 6.69 | 2.90 | 9.79 | | PostgreSQL | 6.28 | **0.84** | 5.20 | 7.24 | 6.89 | **Key findings (tail latency):** * **Write-heavy P99 (A, F):** ekoDB's P99 stays under 3.5ms — **5–10x lower** than MongoDB (9.8–12ms) and **2–5x lower** than PostgreSQL (6.3–6.9ms). This means fewer SLA breaches under production load * **Read-heavy P99 (B, C, D):** ekoDB delivers consistent 2.5–3.1ms P99. PostgreSQL has lower P99 on Workload B (0.84ms) but spikes to 5.2–7.2ms on C and D * **Predictable tail behavior:** ekoDB's P95-to-P99 spread is tight (typically 1.5–2x), indicating consistent performance without long-tail spikes. MongoDB's spread can reach 2–3x on write-heavy workloads #### CPU Efficiency (ops/sec per CPU %)[​](#cpu-efficiency-opssec-per-cpu- "Direct link to CPU Efficiency (ops/sec per CPU %)") > Note: ekoDB intentionally caps CPU usage at 95% to leave headroom for system stability. Other databases typically try to use 100% of available CPU during benchmarks. | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Key-Value)** | **248** | **499** | 548 | **479** | **226** | | **ekoDB (Collections)** | 145 | 357 | 447 | 412 | 109 | | MongoDB | 46 | 103 | 111 | 116 | 50 | | MySQL | 46 | 150 | 198 | 155 | 38 | | PostgreSQL | 45 | 172 | 238 | 222 | 40 | | Redis | 257 | 280 | **1,336** | 271 | 175 | **Key findings:** * ekoDB (Key-Value) achieves **2–5x better CPU efficiency** than PostgreSQL and MongoDB across all workloads * Best efficiency on Workload D: ekoDB (Key-Value) **479 ops/CPU%** (2.2x PostgreSQL, 4.1x MongoDB) * Redis achieves high CPU efficiency on Workload C (1,336 ops/CPU%) due to minimal CPU usage on pure reads, but collapses on write-heavy workloads #### CPU Utilization Details[​](#cpu-utilization-details "Direct link to CPU Utilization Details") | Database | A Avg | A Peak | B Avg | B Peak | C Avg | C Peak | D Avg | D Peak | F Avg | F Peak | | ----------------------- | ----- | ------ | ----- | ------ | ----- | ------ | ----- | ------ | ----- | ------ | | **ekoDB (Key-Value)** | 158.4 | 289.8 | 226.1 | 312.5 | 212.1 | 305.0 | 265.4 | 356.1 | 167.8 | 229.5 | | **ekoDB (Collections)** | 254.8 | 346.6 | 322.0 | 431.4 | 276.4 | 383.1 | 287.1 | 371.0 | 337.3 | 430.4 | | MongoDB | 224.0 | 344.4 | 697.2 | 819.8 | 690.7 | 780.6 | 651.3 | 763.7 | 201.8 | 552.1 | | MySQL | 53.3 | 85.8 | 177.4 | 253.9 | 317.6 | 366.2 | 292.3 | 365.6 | 61.1 | 96.3 | | PostgreSQL | 120.2 | 182.9 | 316.3 | 527.6 | 450.4 | 657.0 | 426.7 | 630.8 | 118.8 | 173.3 | | Redis | 23.4 | 27.9 | 22.8 | 26.8 | 77.7 | 94.1 | 22.1 | 27.0 | 23.9 | 27.8 | *Measured on 10-core Apple M1 Max. CPU% > 100 indicates multi-core usage.* **Storage Mode:** Fast | **Durability:** Non-Durable (async persistence) #### Durability Settings[​](#durability-settings-1 "Direct link to Durability Settings") | Database | Durability Setting | Guarantee | | -------------- | ----------------------------------- | ------------------------------- | | **ekoDB** | `durable_operations=false` | Writes persisted asynchronously | | **PostgreSQL** | `synchronous_commit=off, fsync=off` | No persistence guarantee | | **MongoDB** | `w=0` | No write acknowledgment | Non-durable mode disables persistence guarantees across all databases for a fair throughput ceiling comparison. This configuration suits ephemeral workloads — session caches, rate limiters, real-time analytics counters, and development environments where maximum throughput matters more than surviving a crash. Even in this mode, ekoDB still performs JWT auth, AES-GCM encryption, and full search indexing on every operation — overhead that PostgreSQL and MongoDB skip entirely. #### Throughput (ops/sec)[​](#throughput-opssec-1 "Direct link to Throughput (ops/sec)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ----------- | ----------- | ----------- | ---------- | | **ekoDB (Collections)** | 87,827 | **117,536** | **121,788** | **118,203** | **66,124** | | MongoDB | **88,472** | 79,517 | 81,024 | 81,380 | 60,449 | | PostgreSQL | 80,425 | 98,561 | 103,670 | 105,943 | 57,707 | **Key findings:** * **Workload A** (50% read/50% update): Three-way race — MongoDB (88K), ekoDB (88K), PostgreSQL (80K). ekoDB and MongoDB essentially tied on session stores and shopping carts, with ekoDB carrying auth/encryption overhead that MongoDB skips * **Workloads B–D** (read-heavy): ekoDB leads clearly at 118–122K ops/sec, 12–20% ahead of PostgreSQL (99–106K) and 45–50% ahead of MongoDB (79–81K). These are the most common production patterns — social media profiles, product catalogs, news feeds * **Workload F** (Read-Modify-Write): ekoDB leads (66K) over MongoDB (60K) and PostgreSQL (58K) — important for financial transactions, inventory counters, and any atomic read-then-update pattern #### Average Latency (ms)[​](#average-latency-ms-1 "Direct link to Average Latency (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Collections)** | 0.56 | **0.47** | **0.46** | **0.47** | **0.51** | | PostgreSQL | **0.54** | 0.54 | 0.58 | 0.54 | 0.60 | | MongoDB | 1.39 | 0.79 | 0.74 | 0.78 | 1.02 | **Key findings:** * ekoDB and PostgreSQL deliver **near-identical sub-millisecond latencies** (0.46–0.56ms vs 0.54–0.60ms) — ekoDB edges ahead on read-heavy workloads (B/C/D), PostgreSQL marginally leads on Workload A * MongoDB trails with **significantly higher latencies** on write-heavy workloads (1.02–1.39ms on A and F) * The latency gap matters most for user-facing applications: at p50, both ekoDB and PostgreSQL deliver responses in under 600µs regardless of workload mix #### CPU Efficiency (ops/sec per CPU %)[​](#cpu-efficiency-opssec-per-cpu--1 "Direct link to CPU Efficiency (ops/sec per CPU %)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Collections)** | **174** | **319** | **348** | **339** | **131** | | PostgreSQL | 146 | 204 | 206 | 209 | 116 | | MongoDB | 122 | 104 | 111 | 110 | 75 | * ekoDB leads CPU efficiency across all workloads — **1.1–1.7x better than PostgreSQL**, **1.4–3.1x better than MongoDB** * On cloud platforms, this efficiency advantage directly translates to lower compute costs: ekoDB delivers more throughput per CPU core than competitors while also handling auth, encryption, and search indexing #### CPU Utilization Details[​](#cpu-utilization-details-1 "Direct link to CPU Utilization Details") | Database | A Avg | A Peak | B Avg | B Peak | C Avg | C Peak | D Avg | D Peak | F Avg | F Peak | | ----------------------- | ----- | ------ | ----- | ------ | ----- | ------ | ----- | ------ | ----- | ------ | | **ekoDB (Collections)** | 502.5 | 587.8 | 368.3 | 428.2 | 349.3 | 397.4 | 348.1 | 411.5 | 504.2 | 564.2 | | MongoDB | 719.5 | 876.9 | 762.9 | 818.2 | 728.2 | 825.6 | 738.4 | 835.4 | 802.9 | 875.1 | | PostgreSQL | 548.2 | 683.9 | 482.3 | 667.1 | 502.7 | 707.7 | 505.2 | 717.8 | 496.2 | 683.6 | *Measured on 10-core Apple M1 Max. CPU% > 100 indicates multi-core usage.* Durability vs Performance Trade-off **Non-durable mode** is faster but risky: * **Data loss window**: Uncommitted writes lost on crash (typically 0–1 seconds) * **Use case**: Caches, temp data, dev/test environments, bulk ingestion **Durable mode** (default) is recommended for production: * In durable mode, ekoDB leads write-heavy and mixed YCSB workloads with equivalent durability settings, and trades pure read-only with PostgreSQL * Use durable for production, non-durable only for caches/temp data *** ### Cache Mode: ekoDB KV vs Redis[​](#cache-mode-ekodb-kv-vs-redis "Direct link to Cache Mode: ekoDB KV vs Redis") A head-to-head comparison of ekoDB's Key-Value engine against Redis in pure cache mode — no persistence, no durability overhead. This isolates raw in-memory throughput performance. Test Configuration * **Hardware:** Apple M1 Max, 64GB RAM, NVMe SSD * **Scale:** 1,000,000 records | 1,000,000 operations | 64 threads * **Storage Mode:** Fast | **Durability:** Non-Durable (no fsync, no persistence) * **ekoDB:** `durable_operations=false` (async WAL) * **Redis:** `appendonly=no` (no persistence) #### Throughput (ops/sec)[​](#throughput-opssec-2 "Direct link to Throughput (ops/sec)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | --------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Key-Value)** | 87,078 | 90,498 | 90,058 | 80,782 | **61,660** | | Redis | **91,701** | **87,627** | 88,582 | **84,182** | 58,851 | #### Average Latency (ms)[​](#average-latency-ms-2 "Direct link to Average Latency (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | --------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Key-Value)** | 0.70 | **0.69** | 0.72 | 0.77 | **0.67** | | Redis | **0.69** | 0.72 | **0.71** | **0.71** | 0.72 | #### CPU Efficiency (ops/sec per CPU %)[​](#cpu-efficiency-opssec-per-cpu--2 "Direct link to CPU Efficiency (ops/sec per CPU %)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | --------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Key-Value)** | 419 | 550 | 588 | 432 | 305 | | Redis | **1,189** | **1,139** | **1,141** | **1,087** | **766** | #### Performance Scores[​](#performance-scores "Direct link to Performance Scores") | Database | Score | Grade | | --------------------- | ------ | ------ | | Redis | **97** | **A+** | | **ekoDB (Key-Value)** | **83** | **A** | What This Means ekoDB matches Redis on throughput (\~82K avg ops/sec) and latency (\~0.72ms avg) — despite carrying the [full overhead described above](#the-overhead-ekodb-carries) on every operation. Redis wins decisively on CPU efficiency due to its single-threaded, minimal-overhead design. If you're choosing between Redis and ekoDB for caching, ekoDB delivers equivalent speed plus built-in auth, encryption, search, and real-time subscriptions — without additional services. *** ### Balanced Mode[​](#balanced-mode "Direct link to Balanced Mode") Source run From the **2026-02-16** comparison run. Balanced mode is not measured by the current published artifact, so these figures are not regenerated and describe that run only. Balanced storage mode is designed for **datasets larger than available RAM**, automatically managing which records stay in memory and which are stored on disk. * Balanced + Durable * Balanced + Non-Durable **Storage Mode:** Balanced | **Durability:** Durable (guaranteed persistence) Balanced mode is designed for datasets that exceed available RAM — user profile stores, content management systems, and product catalogs where the total dataset is large but the active working set fits in memory. ekoDB leads on write-heavy and mixed workloads (A, B, F) with 1.6–2.9x better CPU efficiency than the three durable engines, while PostgreSQL edges ahead on pure-read workloads (C, D) where its mature query optimizer is most effective. #### Throughput (ops/sec)[​](#throughput-opssec-3 "Direct link to Throughput (ops/sec)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ----------- | ---------- | ---------- | | **ekoDB (Collections)** | **30,462** | **75,267** | 87,222 | 81,011 | **31,818** | | PostgreSQL | 5,207 | 53,112 | **103,370** | **94,769** | 5,020 | | MongoDB | 11,071 | 68,714 | 71,003 | 68,611 | 10,423 | | MySQL | 4,603 | 42,806 | 61,005 | 46,155 | 4,804 | #### Average Latency (ms)[​](#average-latency-ms-3 "Direct link to Average Latency (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Collections)** | **0.60** | 0.64 | 0.67 | 0.58 | **0.58** | | PostgreSQL | 1.06 | **0.17** | **0.59** | **0.34** | 1.17 | | MongoDB | 1.17 | 0.49 | 0.87 | 0.51 | 1.17 | | MySQL | 1.26 | 0.30 | 0.99 | 0.66 | 1.20 | #### CPU Efficiency (ops/sec per CPU %)[​](#cpu-efficiency-opssec-per-cpu--3 "Direct link to CPU Efficiency (ops/sec per CPU %)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Collections)** | **108** | **281** | **346** | **342** | **104** | | PostgreSQL | 43 | 176 | 179 | 199 | 42 | | MongoDB | 49 | 96 | 103 | 94 | 43 | | MySQL | 61 | 144 | 196 | 143 | 60 | **vs Fast + Durable:** Balanced mode trails across all workloads — Fast + Durable delivers 39K (A), 115K (B), 124K (C), 127K (D), 38K (F). The gap is most pronounced on read-heavy workloads where Fast mode's in-memory primary storage eliminates disk access entirely. **Storage Mode:** Balanced | **Durability:** Non-Durable (async persistence) Non-durable Balanced mode targets development, staging, and batch-processing workloads where datasets exceed RAM and crash recovery is not critical. Without persistence overhead, all databases compete closer to their throughput ceilings. #### Throughput (ops/sec)[​](#throughput-opssec-4 "Direct link to Throughput (ops/sec)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | ekoDB (Collections) | 51,760 | 51,560 | 59,354 | 80,626 | 42,746 | | PostgreSQL | 52,059 | 73,665 | **85,690** | **93,318** | **55,894** | | MongoDB | **82,563** | **75,483** | 77,036 | 75,982 | 51,634 | | MySQL | 29,932 | 54,145 | 59,837 | 49,044 | 25,319 | #### Average Latency (ms)[​](#average-latency-ms-4 "Direct link to Average Latency (ms)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | ekoDB (Collections) | 1.14 | 1.18 | 1.02 | 0.73 | 1.11 | | PostgreSQL | **0.93** | **0.81** | **0.72** | **0.63** | **0.67** | | MongoDB | 1.49 | 0.85 | 0.79 | 0.85 | 1.21 | | MySQL | 1.20 | 0.94 | 0.99 | 1.05 | 1.29 | #### CPU Efficiency (ops/sec per CPU %)[​](#cpu-efficiency-opssec-per-cpu--4 "Direct link to CPU Efficiency (ops/sec per CPU %)") | Database | Workload A | Workload B | Workload C | Workload D | Workload F | | ----------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | | **ekoDB (Collections)** | **141** | **199** | **245** | **329** | 119 | | PostgreSQL | 131 | 178 | 154 | 168 | **123** | | MongoDB | 122 | 100 | 105 | 99 | 71 | | MySQL | 84 | 149 | 195 | 153 | 69 | **Note:** In Balanced + Non-Durable mode, PostgreSQL leads on read-heavy workloads (C, D, F) and MongoDB leads on Workload A. ekoDB maintains a CPU efficiency advantage across most workloads. For maximum non-durable throughput, use Fast mode (66–122K ops/sec vs Balanced 43–81K). Storage Mode Guide | Mode | Optimized For | Best For | | ------------ | ------------------------------------------ | ----------------------------------------- | | **Fast** | Maximum throughput, in-memory performance | Low-latency reads, data fits in RAM | | **Balanced** | Memory-efficient, larger-than-RAM datasets | Datasets larger than RAM, general-purpose | *** ## Feature Comparison[​](#feature-comparison "Direct link to Feature Comparison") Beyond raw performance, ekoDB consolidates capabilities that typically require multiple external services into a single \~50MB binary. The table below shows what ships built-in versus what requires extensions or separate infrastructure. | Feature | ekoDB | PostgreSQL | MongoDB | MySQL | Redis | | --------------------------- | ----------------- | ---------------- | -------------------- | ------------------ | ---------------------- | | **Document Queries** | ✅ Built-in JSON | ✅ SQL + JSONB | ✅ Built-in BSON | ⚠️ JSON columns | ⚠️ RedisJSON module | | **Vector Search** | ✅ Built-in | ⚠️ pgvector ext | ⚠️ Atlas only | ❌ Needs Pinecone | ⚠️ RediSearch module | | **Full-Text Search** | ✅ Built-in | ⚠️ FTS config | ⚠️ Atlas Search | ⚠️ FULLTEXT index | ⚠️ RediSearch module | | **Built-in Auth** | ✅ JWT + API keys | ⚠️ Roles only | ⚠️ SCRAM only | ⚠️ Roles only | ⚠️ ACL only | | **Per-op Encryption** | ✅ AES-GCM | ⚠️ pgcrypto ext | ⚠️ CSFLE client-side | ⚠️ TDE / app-level | ❌ TLS in-transit only | | **Single Binary** | ✅ Yes | ❌ Multi-process | ❌ mongod + mongos | ❌ Multi-process | ✅ Yes | | **Real-time Subscriptions** | ✅ WebSocket | ⚠️ LISTEN/NOTIFY | ⚠️ Change Streams | ❌ Needs external | ⚠️ Pub/Sub | | **Durable Writes** | ✅ Per-operation | ✅ Per-commit | ✅ Journal | ✅ Per-commit | ⚠️ AOF fsync | *** ## Embedded Benchmarks (Local Performance)[​](#embedded-benchmarks-local-performance "Direct link to Embedded Benchmarks (Local Performance)") These benchmarks measure ekoDB's core Rust database engine running in-process — no network round-trips, no serialization, no TCP overhead. This represents the performance floor for applications that embed ekoDB directly as a library rather than connecting over the network, and provides a direct comparison against established embedded databases like SQLite, RocksDB, and LevelDB. Benchmark Environment All benchmarks run on the embedded Rust database engine. Production performance via REST/WebSocket/TCP APIs includes additional network latency. ### Summary[​](#summary "Direct link to Summary") | Category | Operation | ekoDB | SQLite | RocksDB | LevelDB | | ------------- | ------------------- | ---------- | -------- | ---------- | ------- | | **Key-Value** | Get | **185 ns** | \~5 µs | \~3 µs | \~4 µs | | **Key-Value** | Set | **3.5 µs** | \~50 µs | \~6 µs | \~8 µs | | **Query** | find\_by\_id | **2.3 µs** | \~10 µs | N/A | N/A | | **Query** | Simple (10 records) | **108 µs** | \~200 µs | N/A | N/A | | **Insert** | Single record | 101 µs | \~50 µs | **\~6 µs** | \~8 µs | | **Update** | Single record | 27 µs | \~50 µs | **\~6 µs** | \~8 µs | **When to use ekoDB:** Excellent reads and KV operations, plus document queries that RocksDB/LevelDB can't do. The KV layer delivers 185 ns Get and 3.5 µs Set with high-concurrency access. **When to consider alternatives:** RocksDB/LevelDB are faster at raw writes. They're pure KV stores without document parsing or indexing overhead. ### Key-Value Operations[​](#key-value-operations "Direct link to Key-Value Operations") | Operation | ekoDB | SQLite | RocksDB | LevelDB | | ---------- | ---------- | ------- | ------- | ------- | | **Get** | **185 ns** | \~5 µs | \~3 µs | \~4 µs | | **Set** | **3.5 µs** | \~50 µs | \~6 µs | \~8 µs | | **Exists** | **109 ns** | \~5 µs | \~3 µs | \~4 µs | | **Delete** | **85 ns** | \~50 µs | \~6 µs | \~8 µs | ekoDB's KV layer uses lock-free concurrent storage optimized for reads. RocksDB/LevelDB edge ahead on writes due to LSM-tree architecture. ### Query Operations[​](#query-operations "Direct link to Query Operations") | Records | ekoDB (Simple) | ekoDB (Complex) | SQLite | DuckDB | | ------- | -------------- | --------------- | -------- | -------- | | 10 | **108 µs** | **109 µs** | \~200 µs | \~300 µs | | 100 | **113 µs** | 1.16 ms | \~500 µs | \~600 µs | | 1000 | **120 µs** | 12.8 ms | \~2 ms | \~15 ms | ekoDB maintains consistent performance as result sets grow. For complex analytical queries, DuckDB is purpose-built for that use case. ### Cache Warming[​](#cache-warming "Direct link to Cache Warming") ekoDB's pattern-based cache warming pre-loads frequently accessed records: | Records | Uncached | Cached | Speedup | | ------- | -------- | ------- | -------- | | 10 | 329 µs | 109 µs | **3.0x** | | 100 | 3.38 ms | 1.08 ms | **3.1x** | | 1000 | 41.4 ms | 11.4 ms | **3.6x** | ### Write Operations[​](#write-operations "Direct link to Write Operations") | Operation | ekoDB | SQLite | RocksDB | LevelDB | | -------------------- | ------ | ------- | ------------ | -------- | | **Single insert** | 101 µs | \~50 µs | **\~6 µs** | \~8 µs | | **Batch insert 100** | 5.2 ms | \~5 ms | **\~600 µs** | \~800 µs | | **Single update** | 27 µs | \~50 µs | **\~6 µs** | \~8 µs | | **Single delete** | 72 µs | \~50 µs | **\~6 µs** | \~8 µs | RocksDB/LevelDB dominate write performance. Their LSM-tree design converts random writes to sequential I/O. ekoDB includes indexing overhead that slows writes but accelerates reads and enables queries. ### Join Operations[​](#join-operations "Direct link to Join Operations") | Join Type | ekoDB (10) | ekoDB (100) | ekoDB (1000) | SQLite (1000) | | -------------------- | ---------- | ----------- | ------------ | ------------- | | **Simple Join** | 22 µs | 528 µs | 42.9 ms | \~80 ms | | **Multi-Collection** | 22 µs | 530 µs | 40.9 ms | \~100 ms | | **Filtered Join** | 25 µs | 617 µs | 45.6 ms | \~90 ms | ekoDB outperforms SQLite on joins due to in-memory processing. For complex join strategies on large datasets, SQLite's query planner offers more sophistication. ### Authentication & Encryption[​](#authentication--encryption "Direct link to Authentication & Encryption") | Operation | Time | | -------------------- | ------- | | **Validate API key** | 87 ns | | **Generate token** | 1.08 µs | | **Validate token** | 1.61 µs | | **Encrypt (small)** | 2.0 µs | | **Encrypt (large)** | 72.3 µs | Sub-microsecond auth overhead means authentication is negligible in request latency. ### Feature Comparison (Embedded)[​](#feature-comparison-embedded "Direct link to Feature Comparison (Embedded)") | Feature | ekoDB | SQLite | RocksDB | LevelDB | | --------------------- | ----- | --------- | ------- | ------- | | **Document queries** | ✅ | ✅ | ❌ | ❌ | | **Full-text search** | ✅ | ✅ (FTS5) | ❌ | ❌ | | **Vector search** | ✅ | ❌ | ❌ | ❌ | | **Built-in auth** | ✅ | ❌ | ❌ | ❌ | | **ACID transactions** | ✅ | ✅ | ✅ | ❌ | ekoDB trades some raw write performance for a richer feature set. If you need pure KV speed, RocksDB wins. If you need queries, search, and auth in one package, ekoDB is the only embedded option. *** ## See Also[​](#see-also "Direct link to See Also") * [Query Patterns & Cache Warming](/patterns-cache-warming.md) - Intelligent caching for 3x faster queries * [Transactions Architecture](/reference/transaction-architecture.md) - ACID transaction performance * [Error Codes](/reference/error-codes.md) - API error reference --- # Transactions Architecture This document provides a comprehensive overview of ekoDB's transaction support, including ACID guarantees, isolation levels, savepoints, WAL integration, and both function-based and REST API transaction management. ## Overview[​](#overview "Direct link to Overview") ekoDB transactions are **buffered** and guarantee **atomicity** (all-or-nothing), **isolation** (enforced, configurable), and **durability** (WAL). They provide: * **Atomicity** - each statement stages into the transaction; the whole set applies atomically at commit, and a failure or rollback discards the staged writes (nothing partially applied) * **Isolation** - enforced via optimistic concurrency over per-record version stamps (MVCC), configurable per transaction up to `Serializable` * **Durability** - committed transactions are persisted via the WAL * **Deferred Replication** - ripple propagation is deferred to commit and respects per-operation `bypass_ripple` * **Nested Transactions** - Savepoints for partial rollback (rewind the buffer) * **WAL Audit Trail** - All transaction events logged for recovery and auditing * **Dual Interface** - Both function-based and REST API transaction management * **Automatic Cleanup** - Expired transaction management (discards the buffer) Isolation is enforced ekoDB enforces **isolation** between concurrent transactions via optimistic concurrency on per-record version stamps (MVCC). A transaction's staged writes are invisible to others until commit (no dirty reads) and visible to its own reads that carry the transaction id (read-your-writes); at commit, ekoDB validates that the records it read or wrote are unchanged and otherwise rejects the commit as a retryable conflict (HTTP `409`) — nothing is applied. No locks are held across statements. See [Isolation Levels](#isolation-levels) for what each level validates. ## Table of Contents[​](#table-of-contents "Direct link to Table of Contents") 1. [Transaction Types](#transaction-types) 2. [Isolation Levels](#isolation-levels) 3. [Savepoints (Nested Transactions)](#savepoints-nested-transactions) 4. [Function-Based Transactions](#function-based-transactions) 5. [REST API Transactions](#rest-api-transactions) 6. [WAL Integration](#wal-integration) 7. [Usage Examples](#usage-examples) 8. [Best Practices](#best-practices) ## Transaction Types[​](#transaction-types "Direct link to Transaction Types") ### 1. Function-Based Transactions[​](#1-function-based-transactions "Direct link to 1. Function-Based Transactions") Transactions embedded within callable functions with automatic lifecycle management. **Features:** * Automatic begin/commit/rollback * WAL logging built-in * Savepoint support via Function enum * Configurable isolation levels ### 2. REST API Transactions[​](#2-rest-api-transactions "Direct link to 2. REST API Transactions") Direct transaction management via HTTP endpoints: ``` POST /api/transactions # Begin GET /api/transactions/{id} # Status POST /api/transactions/{id}/commit # Commit POST /api/transactions/{id}/rollback # Rollback POST /api/transactions/{id}/savepoints # Create savepoint POST /api/transactions/{id}/savepoints/{name}/rollback DELETE /api/transactions/{id}/savepoints/{name} ``` **Features:** * Manual transaction control * Long-running transaction support * Timeout management * Active transaction tracking ## Isolation Levels[​](#isolation-levels "Direct link to Isolation Levels") The four levels can be passed to `TransactionConfig` and the REST API. The level selects **how much ekoDB validates at commit** under its optimistic-concurrency model (version-stamp checks over per-record stamps; no locks held across statements). It is resolved per transaction as the value you pass, else the server-wide `default_isolation_level` setting (`ReadUncommitted` out of the box; the official clients send `ReadCommitted`), and the default is runtime-tunable via `/api/config`. ekoDB validates at **commit** rather than taking per-statement read snapshots, so the levels map to validation strength (which conflicts abort the commit), not classic lock-based read semantics. ### ReadUncommitted[​](#readuncommitted "Direct link to ReadUncommitted") * No commit validation; concurrent commits are last-writer-wins. The zero-cost default fast path. * **Use case:** Analytics, non-critical / high-throughput writes ### ReadCommitted (client default)[​](#readcommitted-client-default "Direct link to ReadCommitted (client default)") * **Lost-update prevention** — the commit is rejected if a record this transaction **wrote** was changed by another committed transaction since it was observed. * **Use case:** Most general-purpose transactions ### RepeatableRead[​](#repeatableread "Direct link to RepeatableRead") * Adds **read-set validation** on top of `ReadCommitted`: the commit is rejected if a record this transaction **read** (not only one it wrote) changed before commit. Prevents non-repeatable reads / read skew. Phantoms are still possible. * **Use case:** Reads that must stay valid through commit (financial calculations, reporting) ### Serializable[​](#serializable "Direct link to Serializable") * Strongest — adds **collection-membership validation** on top of `RepeatableRead`: the commit is rejected if the membership of any collection this transaction **queried** changed before commit (an insert / delete / update that shifts a predicate's result set). Prevents phantoms, and with them write skew; the committed history is serializable. * **Use case:** Banking, critical financial operations ## Savepoints (Nested Transactions)[​](#savepoints-nested-transactions "Direct link to Savepoints (Nested Transactions)") Savepoints allow partial rollback within a transaction: ### REST API[​](#rest-api "Direct link to REST API") ``` # Create savepoint POST /api/transactions/{id}/savepoints {"name": "before_debit"} # Rollback to savepoint POST /api/transactions/{id}/savepoints/before_debit/rollback # Release savepoint DELETE /api/transactions/{id}/savepoints/before_debit ``` ### How It Works[​](#how-it-works "Direct link to How It Works") 1. **Create** - Marks a restore point within the transaction 2. **Rollback** - Reverts all operations performed after the savepoint 3. **Release** - Releases the savepoint when no longer needed **Benefits:** * Partial rollback without full transaction abort * Multi-stage error recovery * Complex workflow support ## Function-Based Transactions[​](#function-based-transactions "Direct link to Function-Based Transactions") ### Creating a Transactional Function[​](#creating-a-transactional-function "Direct link to Creating a Transactional Function") ``` POST /api/functions { "label": "transfer_funds", "description": "Transfer money between accounts", "transaction_config": { "enabled": true, "auto_rollback": true, "isolation_level": "Serializable" }, "functions": [ { "type": "Update", "collection": "accounts", "filter": {"type": "Condition", "content": {"field": "account_id", "operator": "Eq", "value": "{{from_account}}"}}, "update": {"$decrement": {"balance": "{{amount}}"}} }, { "type": "CreateSavepoint", "name": "after_debit" }, { "type": "Update", "collection": "accounts", "filter": {"type": "Condition", "content": {"field": "account_id", "operator": "Eq", "value": "{{to_account}}"}}, "update": {"$increment": {"balance": "{{amount}}"}} } ] } ``` ### Executing the Function[​](#executing-the-function "Direct link to Executing the Function") ``` POST /api/functions/transfer_funds { "from_account": "ACC001", "to_account": "ACC002", "amount": 100 } ``` ### Automatic Behavior[​](#automatic-behavior "Direct link to Automatic Behavior") * **Begin** - Logged to WAL on function start * **Commit** - Automatic if all functions succeed * **Rollback** - Automatic if `auto_rollback: true` and error occurs * **WAL Logging** - All lifecycle events recorded ## REST API Transactions[​](#rest-api-transactions "Direct link to REST API Transactions") ### Full Transaction Lifecycle[​](#full-transaction-lifecycle "Direct link to Full Transaction Lifecycle") #### 1. Begin Transaction[​](#1-begin-transaction "Direct link to 1. Begin Transaction") ``` POST /api/transactions Content-Type: application/json { "isolation_level": "Serializable", "timeout_seconds": 600 } # Response { "transaction_id": "tx-abc123", "isolation_level": "Serializable", "created_at": 1699814400 } ``` #### 2. Perform Operations[​](#2-perform-operations "Direct link to 2. Perform Operations") ``` # Use standard CRUD APIs PUT /api/update/accounts/account1 {"balance": 900} PUT /api/update/accounts/account2 {"balance": 1100} ``` **Note:** Pass `?transaction_id=` on each statement (insert/update/delete/ batch and point reads) to stage it into the transaction. Staged writes are applied atomically at commit; reads that carry the id see the transaction's own staged writes (read-your-writes). Reads without the id see only committed data. #### 3. Create Savepoint[​](#3-create-savepoint "Direct link to 3. Create Savepoint") ``` POST /api/transactions/tx-abc123/savepoints {"name": "checkpoint1"} # Response { "status": "created", "transaction_id": "tx-abc123", "savepoint_name": "checkpoint1" } ``` #### 4. Commit or Rollback[​](#4-commit-or-rollback "Direct link to 4. Commit or Rollback") ``` # Commit POST /api/transactions/tx-abc123/commit # Response { "status": "committed", "transaction_id": "tx-abc123", "operations_count": 5 } # OR Rollback POST /api/transactions/tx-abc123/rollback # Response { "status": "rolled_back", "transaction_id": "tx-abc123", "operations_rolled_back": 5 } ``` #### 5. Check Status[​](#5-check-status "Direct link to 5. Check Status") ``` GET /api/transactions/tx-abc123 # Response { "transaction_id": "tx-abc123", "state": "Active", "isolation_level": "Serializable", "operations_count": 5, "savepoints_count": 2, "created_at": 1699814400, "last_activity": 1699814450, "timeout_seconds": 600 } ``` ### Transaction Manager[​](#transaction-manager "Direct link to Transaction Manager") The transaction manager provides: * In-memory tracking of active transactions * Automatic cleanup of expired transactions (every 60 seconds) * Thread-safe operation * Timeout management ## WAL Integration[​](#wal-integration "Direct link to WAL Integration") ### Transaction Lifecycle Events[​](#transaction-lifecycle-events "Direct link to Transaction Lifecycle Events") All transaction events are logged to the Write-Ahead Log, including: * **Transaction lifecycle**: Begin, Commit, Rollback * **Savepoint events**: Create, Rollback, Release * **Timestamps**: For audit trail and recovery * **Operation counts**: For tracking and monitoring ### Replay Behavior[​](#replay-behavior "Direct link to Replay Behavior") Transaction events are **audit logs only**: * Not replayed on WAL recovery * Provide complete audit trail * Enable transaction history queries * Support compliance requirements ## Usage Examples[​](#usage-examples "Direct link to Usage Examples") ### Example 1: Simple Function Transaction[​](#example-1-simple-function-transaction "Direct link to Example 1: Simple Function Transaction") ``` // Create function POST /api/functions { "label": "create_user_with_profile", "transaction_config": { "enabled": true, "auto_rollback": true, "isolation_level": "ReadCommitted" }, "functions": [ { "type": "Insert", "collection": "users", "record": {"email": "{{email}}", "name": "{{name}}"} }, { "type": "Insert", "collection": "profiles", "record": {"user_email": "{{email}}", "bio": "{{bio}}"} } ] } // Call function POST /api/functions/create_user_with_profile { "email": "user@example.com", "name": "John Doe", "bio": "Software Engineer" } ``` **Result:** Both records created atomically or neither created if error occurs. ### Example 2: Multi-Stage Transfer with Savepoints[​](#example-2-multi-stage-transfer-with-savepoints "Direct link to Example 2: Multi-Stage Transfer with Savepoints") ``` # 1. Begin transaction POST /api/transactions {"isolation_level": "Serializable", "timeout_seconds": 600} # -> {"transaction_id": "tx-001", ...} # 2. Create initial savepoint POST /api/transactions/tx-001/savepoints {"name": "start"} # 3. Debit source account PUT /api/update/accounts/ACC001 {"$decrement": {"balance": 100}} # 4. Create savepoint after debit POST /api/transactions/tx-001/savepoints {"name": "after_debit"} # 5. Credit destination account PUT /api/update/accounts/ACC002 {"$increment": {"balance": 100}} # 6. If credit fails, rollback to after_debit and retry POST /api/transactions/tx-001/savepoints/after_debit/rollback # Retry credit operation... # 7. Commit transaction POST /api/transactions/tx-001/commit ``` ### Example 3: Long-Running Workflow[​](#example-3-long-running-workflow "Direct link to Example 3: Long-Running Workflow") ``` // Function with multiple checkpoints { "label": "order_processing", "transaction_config": { "enabled": true, "auto_rollback": false, // Manual control "isolation_level": "RepeatableRead" }, "functions": [ {"type": "CreateSavepoint", "name": "order_created"}, {"type": "Insert", "collection": "orders", "record": {"...": "..."}}, {"type": "CreateSavepoint", "name": "inventory_reserved"}, {"type": "Update", "collection": "inventory", "...": "..."}, {"type": "CreateSavepoint", "name": "payment_processed"}, {"type": "Insert", "collection": "payments", "record": {"...": "..."}}, {"type": "Update", "collection": "orders", "update": {"status": "completed"}} ] } ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### 1. Declare Intended Isolation Level[​](#1-declare-intended-isolation-level "Direct link to 1. Declare Intended Isolation Level") The table below describes which level to **declare** for a given workload. Isolation is enforced (see [Isolation Levels](#isolation-levels)): a higher level applies stricter conflict validation, and conflicting concurrent transactions are rejected with a 409 rather than committing inconsistent state. | Use Case | Declare | Why | | ---------------------- | --------------- | ---------------------------- | | Analytics/Reporting | ReadUncommitted | Performance over consistency | | General CRUD | ReadCommitted | Intended default | | Financial Calculations | RepeatableRead | Consistent reads | | Banking/Critical | Serializable | Maximum safety | ### 2. Use Savepoints for Multi-Stage Operations[​](#2-use-savepoints-for-multi-stage-operations "Direct link to 2. Use Savepoints for Multi-Stage Operations") ``` // Good: Savepoints for partial rollback [ {"type": "CreateSavepoint", "name": "before_risky_op"}, {"type": "Update", /* risky operation */}, // If fails, can rollback to savepoint ] // Bad: All-or-nothing without savepoints [ {"type": "Update", /* operation 1 */}, {"type": "Update", /* operation 2 */}, {"type": "Update", /* operation 3 */}, // If operation 3 fails, all operations rollback ] ``` ### 3. Set Appropriate Timeouts[​](#3-set-appropriate-timeouts "Direct link to 3. Set Appropriate Timeouts") ``` // Short transaction {"timeout_seconds": 60} // 1 minute // Long-running workflow {"timeout_seconds": 1800} // 30 minutes // Critical operation {"timeout_seconds": 300} // 5 minutes (default) ``` ### 4. Handle Errors Gracefully[​](#4-handle-errors-gracefully "Direct link to 4. Handle Errors Gracefully") ``` // Function-based: Use auto_rollback { "transaction_config": { "enabled": true, "auto_rollback": true // Automatic rollback on error } } // API-based: Manual error handling try { // Perform operations POST /api/transactions/{id}/commit } catch (error) { POST /api/transactions/{id}/rollback } ``` ### 5. Monitor Active Transactions[​](#5-monitor-active-transactions "Direct link to 5. Monitor Active Transactions") ``` # List all active transactions GET /api/transactions # Check specific transaction GET /api/transactions/{id} ``` ### 6. Use WAL for Auditing[​](#6-use-wal-for-auditing "Direct link to 6. Use WAL for Auditing") Transaction events in WAL provide: * Complete transaction history * Audit trail for compliance * Debugging failed transactions * Performance analysis ## Performance Characteristics[​](#performance-characteristics "Direct link to Performance Characteristics") ### Transaction Overhead[​](#transaction-overhead "Direct link to Transaction Overhead") Transactions buffer their writes in memory and apply them at commit, so overhead is the cost of buffering plus WAL logging of begin/commit/rollback events plus commit-time version-stamp validation. Concurrency control is **optimistic** — no locks are held across statements — so the per-level cost is just how much the commit validates: | Cost component | Overhead | Notes | | -------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | Write buffering | Low | In-memory staging until commit | | WAL begin/commit/rollback events | Low | Async, batched, non-blocking | | Commit validation | Per-level | None at `ReadUncommitted`; write-set at `ReadCommitted`; read+write-set at `RepeatableRead`; read+write-set plus collection membership at `Serializable` | ### Savepoint Overhead[​](#savepoint-overhead "Direct link to Savepoint Overhead") * **Create**: Snapshots the in-memory buffer * **Rollback**: Restores the buffer snapshot (no live-store compensation) * **Release**: Sub-microsecond ### WAL Logging[​](#wal-logging "Direct link to WAL Logging") * Transaction events: Async, non-blocking * Batched writes for efficiency * Minimal impact on transaction performance ## Common Use Cases[​](#common-use-cases "Direct link to Common Use Cases") ### 1. Real-Time Financial Systems[​](#1-real-time-financial-systems "Direct link to 1. Real-Time Financial Systems") * Strong consistency for account updates * Real-time WebSocket updates for UI * Comprehensive audit logging ### 2. E-Commerce Platforms[​](#2-e-commerce-platforms "Direct link to 2. E-Commerce Platforms") * Order creation with inventory reservation (savepoints) * Payment processing with rollback capability * Multi-step checkout flows with error recovery ### 3. Multi-Tenant SaaS[​](#3-multi-tenant-saas "Direct link to 3. Multi-Tenant SaaS") * Tenant data isolation with transactions * Batch operations with partial rollback * Audit trail for compliance ### 4. Workflow Orchestration[​](#4-workflow-orchestration "Direct link to 4. Workflow Orchestration") * Long-running workflows with checkpoints * Partial rollback on step failure * State persistence via WAL ## Architecture Diagram[​](#architecture-diagram "Direct link to Architecture Diagram") ``` ┌─────────────────────────────────────────────────────────┐ │ Transaction Layer │ ├─────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────┐ ┌──────────────────┐ │ │ │ Function-Based │ │ REST API │ │ │ │ Transactions │ │ Transactions │ │ │ └────────┬────────┘ └────────┬─────────┘ │ │ │ │ │ │ └───────────┬───────────────┘ │ │ │ │ │ ┌───────────▼──────────────┐ │ │ │ TransactionManager │ │ │ │ - Active tracking │ │ │ │ - Timeout management │ │ │ │ - Savepoint handling │ │ │ └───────────┬──────────────┘ │ │ │ │ │ ┌───────────▼──────────────┐ │ │ │ Transaction │ │ │ │ - Operations tracking │ │ │ │ - Savepoint management │ │ │ │ - Isolation level │ │ │ └───────────┬──────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────┐ │ │ │ WAL (Write-Ahead Log) │ │ │ │ - Transaction events │ │ │ │ - Savepoint events │ │ │ │ - Audit trail │ │ │ └───────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────┘ ``` ## Transaction State Machine[​](#transaction-state-machine "Direct link to Transaction State Machine") ``` ┌─────────┐ │ BEGIN │ └────┬────┘ │ ▼ ┌──────────────────┐ │ ACTIVE │◄─────┐ └──────┬───┬───────┘ │ │ │ │ ┌──────────┘ └──────────┐ │ │ │ │ ▼ ▼ │ ┌──────────┐ ┌──────────────┐ │ COMMIT │ │ SAVEPOINT │ └──────────┘ │ ROLLBACK │ └──────┬───────┘ │ └───────┘ ``` ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [White Paper](/white-paper.md) - ACID compliance and architecture overview * [Functions](/client-libraries/advanced-operations.md#functions) - Reusable stored procedures --- # Ripples - Data Propagation Ripples enable real-time data propagation across multiple ekoDB instances, providing horizontal scaling, geographic distribution, and high availability through automatic replication. ## What are Ripples?[​](#what-are-ripples "Direct link to What are Ripples?") Ripples are ekoDB's built-in data propagation system that replicates database operations (inserts, updates, deletes) across connected ekoDB instances, either as application-level operations (Operations mode) or by shipping committed WAL entries (WAL mode). Implementation status Ripples today is an **ekoDB→ekoDB** system. Two modes are wired and production-ready: * **Operations mode** — real-time, one-way CRUD propagation with loop prevention. * **WAL mode** — interval-based committed-WAL shipping to replicas (see [WAL Mode](#wal-mode)). The other modes and destinations described below are **roadmap, not current behavior**: Snapshot, Search, Analytics, Stream, Webhook, Embedding, and Chat modes — and all foreign destinations (Elasticsearch, BigQuery, Kafka, PostgreSQL, etc.) — are scaffolded but not wired to the write path. `PropagationFilter` / `TransformConfig` are accepted but not yet enforced. **Bidirectional / active-active is supported** (both nodes writable, with automatic loop prevention), but there is **no conflict resolution** — concurrent writes to the *same* record on different nodes diverge (last write by arrival), so active-active is safe only when writes are partitioned by node or contention is low. Ripples handles **forward change propagation** only. **Base-state seeding, backups, and disaster recovery are provided by the managed deployment platform**, not by Ripples: new nodes boot their data disk from a snapshot, deployments take automated disk-snapshot backups, and a deployment can be cloned from a snapshot. So a read replica is seeded from a snapshot (base state) and then kept current by WAL-mode shipping (forward changes) — see [System Administration](/system-administration.md) for backup/restore. **Key Features (today):** * **Real-Time Propagation**: Operations replicate immediately as they occur (Operations mode) * **WAL shipping**: committed WAL entries streamed to replicas on an interval (WAL mode) * **Selective Replication**: Configure which nodes receive which operations * **Loop Prevention**: Automatic deduplication prevents infinite loops * **Bidirectional Sync / Active-Active**: Nodes can both send and receive (loop-prevented). Works as active-active — but there's no conflict resolution, so partition writes by node to avoid same-record divergence ## How Ripples Work[​](#how-ripples-work "Direct link to How Ripples Work") When a write operation occurs on a node: 1. **Operation Executes**: Record is inserted/updated/deleted locally 2. **Ripple Generated**: Operation metadata is packaged with unique ID 3. **Propagation**: Ripple is sent to configured peer nodes 4. **Remote Execution**: Peer nodes receive and execute the operation 5. **Loop Prevention**: Request ID prevents duplicate processing ``` ┌─────────────┐ │ Node 1 │ ← Client writes here │ (Primary) │ └──────┬──────┘ │ Ripple (INSERT) ├───────────────────┐ ↓ ↓ ┌─────────────┐ ┌─────────────┐ │ Node 2 │ │ Node 3 │ │ (Replica) │ │ (Replica) │ └─────────────┘ └─────────────┘ ``` ## Ripple Propagation Modes[​](#ripple-propagation-modes "Direct link to Ripple Propagation Modes") Ripples support multiple propagation modes for different use cases. The `mode` field in the configuration determines how data is formatted and transmitted. ### Quick Mode Selection Guide[​](#quick-mode-selection-guide "Direct link to Quick Mode Selection Guide") | Mode | Status | Primary Use Case | Destination | | ------------------------ | ------------ | ---------------------------------- | -------------------------- | | **Operations** (default) | ✅ Available | Multi-node database replication | ekoDB instances | | **WAL** | ✅ Available | High-volume efficient replication | ekoDB instances | | **Snapshot** | 🚧 Roadmap | Periodic backups, data warehousing | ekoDB, storage systems | | **Search** | 🚧 Roadmap | Full-text search integration | Elasticsearch, Meilisearch | | **Analytics** | 🚧 Roadmap | Business intelligence, reporting | BigQuery, Snowflake | | **Stream** | 🚧 Roadmap | Event-driven architectures | Kafka, RabbitMQ | | **Webhook** | 🚧 Roadmap | Third-party integrations | HTTP endpoints | | **Embedding** | 🚧 Roadmap | Semantic search, AI/ML | Vector databases | | **Chat** | 🚧 Roadmap | AI chatbots, LLM context | OpenAI, custom LLMs | | **Custom** | 🚧 Roadmap | Specialized integrations | Custom systems | Recommended Mode Use **Operations mode** (the default) for standard multi-node replication, or **WAL mode** for high-volume forward replication to read replicas. The 🚧 Roadmap modes above are the intended design but are not yet wired — see the implementation-status note at the top. ### Operations Mode (Default)[​](#operations-mode-default "Direct link to Operations Mode (Default)") **Purpose:** Real-time CRUD operation replication between ekoDB instances **Use Case:** Multi-node clusters, horizontal scaling, high availability **How it works:** Each insert/update/delete is sent as an individual operation to peer nodes in real-time. **Example Configuration:** ``` POST /api/ripples/config { "name": "replica1", "url": "https://replica1.production.google.ekodb.net:8080", "api_key": "replica1-admin-key", "mode": "Operations", # Default mode "enabled": true } ``` **Best for:** Standard database replication and read replicas. Also supports **active-active / bidirectional** (both nodes writable, loop-prevented) — just partition writes by node, since there's no conflict resolution for concurrent same-record writes (see the implementation-status note). *** ### WAL Mode[​](#wal-mode "Direct link to WAL Mode") **Purpose:** Batch Write-Ahead Log shipping for efficient replication **Use Case:** Large-scale replication with lower network overhead **How it works:** A background loop on the primary ships committed WAL entries to WAL-mode replicas on the `wal_ship_interval_secs` interval (default 5s, set via `PUT /api/config`). Shipping is incremental and crash-safe — the primary tracks a persisted position and only sends entries committed since the last successful ship — and the replica deduplicates by sequence, so a re-ship after a crash is idempotent. The replica applies inserts/updates/deletes (KV entries replicate via Operations mode instead). **Scope:** WAL mode ships **forward changes** one-way (primary→replica). Seeding a replica's **base state** is handled by the managed deployment platform — new nodes boot their disk from a snapshot and deployments take automated snapshot backups (see [System Administration](/system-administration.md)) — so the usual flow is: snapshot-seed the replica, then WAL-ship forward changes onto it. **Example Configuration:** ``` POST /api/ripples/config { "name": "wal_replica", "url": "https://replica.production.google.ekodb.net:8080", "api_key": "replica-admin-key", "mode": "WAL", "strategy": "batched", "options": { "batch_size": 5000, "interval_secs": 10 }, "enabled": true } ``` **Best for:** High-volume replication, reducing network traffic, eventual consistency scenarios *** Roadmap modes The modes from **Snapshot** through **Custom** below (and all non-ekoDB destinations) describe the **intended design and are not yet wired to the write path** — see the implementation-status note above. The configuration examples show the planned shape, not behavior you can rely on today. Use **Operations** or **WAL** mode for real replication. ### Snapshot Mode[​](#snapshot-mode "Direct link to Snapshot Mode") **Purpose:** Periodic full database snapshots **Use Case:** Backup nodes, reporting databases, point-in-time recovery **How it works:** Sends complete snapshots of collections on a schedule rather than individual operations. **Example Configuration:** ``` POST /api/ripples/config { "name": "backup_snapshot", "url": "https://backup.production.google.ekodb.net:8080", "api_key": "backup-admin-key", "mode": "Snapshot", "strategy": "scheduled", "options": { "cron": "0 2 * * *", # Daily at 2 AM "collections": ["users", "orders"] }, "enabled": true } ``` **Best for:** Backup systems, data warehousing, periodic sync to reporting databases *** ### Search Mode[​](#search-mode "Direct link to Search Mode") **Purpose:** Send data formatted for search indexing **Use Case:** Elasticsearch, Meilisearch, or other search engine integration **How it works:** Transforms operations into search-optimized format and sends to search indices. **Example Configuration:** ``` POST /api/ripples/config { "name": "elasticsearch_index", "url": "https://elasticsearch.example.com:9200", "api_key": "es-api-key", "mode": "Search", "destination": "elasticsearch", "transform": { "include_fields": ["title", "content", "author", "tags"], "exclude_fields": ["internal_id"] }, "enabled": true } ``` **Best for:** Full-text search, faceted search, search-as-you-type features *** ### Analytics Mode[​](#analytics-mode "Direct link to Analytics Mode") **Purpose:** Send data to analytics platforms **Use Case:** BigQuery, Snowflake, data warehouses, BI tools **How it works:** Formats data for analytics schemas and sends to analytics platforms in batches. **Example Configuration:** ``` POST /api/ripples/config { "name": "bigquery_analytics", "url": "https://bigquery.googleapis.com/v2/projects/my-project", "api_key": "bq-service-account-key", "mode": "Analytics", "destination": "bigquery", "filter": { "collections": ["events", "metrics", "user_activity"] }, "options": { "dataset": "production_analytics", "batch_size": 10000 }, "enabled": true } ``` **Best for:** Business intelligence, data analytics, reporting dashboards *** ### Stream Mode[​](#stream-mode "Direct link to Stream Mode") **Purpose:** Send data to message streams **Use Case:** Kafka, RabbitMQ, event-driven architectures **How it works:** Publishes operations as messages to streaming platforms for event processing. **Example Configuration:** ``` POST /api/ripples/config { "name": "kafka_stream", "url": "https://kafka.example.com:9092", "api_key": "kafka-api-key", "mode": "Stream", "destination": "kafka", "options": { "topic": "ekodb-events", "partition_key": "collection_name" }, "enabled": true } ``` **Best for:** Event sourcing, microservices communication, real-time data pipelines *** ### Webhook Mode[​](#webhook-mode "Direct link to Webhook Mode") **Purpose:** Trigger HTTP webhooks on data changes **Use Case:** Third-party integrations, automation, notifications **How it works:** Sends HTTP POST requests to configured webhook URLs when operations occur. **Example Configuration:** ``` POST /api/ripples/config { "name": "slack_notifications", "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL", "mode": "Webhook", "filter": { "collections": ["orders"], "operations": ["insert"] }, "transform": { "transform_fn": "format_slack_message" }, "enabled": true } ``` **Best for:** Slack/Discord notifications, Zapier integration, custom automation *** ### Embedding Mode[​](#embedding-mode "Direct link to Embedding Mode") **Purpose:** Generate and send vector embeddings **Use Case:** Vector databases, semantic search, AI/ML pipelines **How it works:** Processes text fields, generates embeddings, and sends to vector databases. **Example Configuration:** ``` POST /api/ripples/config { "name": "pinecone_vectors", "url": "https://your-index.pinecone.io", "api_key": "pinecone-api-key", "mode": "Embedding", "destination": "vectordb", "options": { "embedding_field": "content", "embedding_model": "text-embedding-ada-002", "dimension": 1536 }, "enabled": true } ``` **Best for:** Semantic search, RAG (Retrieval Augmented Generation), similarity matching *** ### Chat Mode[​](#chat-mode "Direct link to Chat Mode") **Purpose:** Send data to chat/LLM systems **Use Case:** AI assistants, chatbots, context-aware LLMs **How it works:** Formats data for chat context and sends to LLM providers or chat databases. **Example Configuration:** ``` POST /api/ripples/config { "name": "openai_context", "url": "https://api.openai.com/v1/assistants/asst_xxx/messages", "api_key": "openai-api-key", "mode": "Chat", "filter": { "collections": ["support_tickets", "knowledge_base"] }, "transform": { "include_fields": ["question", "answer", "category"] }, "enabled": true } ``` **Best for:** AI chatbots, customer support automation, context-aware assistants *** ### Custom Mode[​](#custom-mode "Direct link to Custom Mode") **Purpose:** User-defined custom propagation logic **Use Case:** Specialized integrations, custom data pipelines **How it works:** Allows you to define custom propagation behavior via configuration. **Example Configuration:** ``` POST /api/ripples/config { "name": "custom_pipeline", "url": "https://custom-endpoint.example.com/ingest", "api_key": "custom-api-key", "mode": { "Custom": "my_custom_handler" }, "options": { "custom_param1": "value1", "custom_param2": "value2" }, "enabled": true } ``` **Best for:** Proprietary systems, specialized data transformations, unique integrations *** ## Replication Patterns (Send/Receive/Both)[​](#replication-patterns-sendreceiveboth "Direct link to Replication Patterns (Send/Receive/Both)") Beyond the propagation mode, nodes can be configured in different replication patterns: ### Send Mode[​](#send-mode "Direct link to Send Mode") Node **sends** ripples to peers but doesn't receive. **Use Case:** Primary write nodes that propagate to replicas **Configure each peer separately:** ``` # Add first replica peer POST /api/ripples/config { "name": "replica1", "url": "https://replica1.production.google.ekodb.net:8080", "api_key": "replica1-admin-key", "mode": "Operations", "enabled": true } # Add second replica peer POST /api/ripples/config { "name": "replica2", "url": "https://replica2.production.google.ekodb.net:8080", "api_key": "replica2-admin-key", "mode": "Operations", "enabled": true } ``` ### Receive Mode[​](#receive-mode "Direct link to Receive Mode") Node **receives** ripples from peers but doesn't send. **Use Case:** Read replicas that stay synchronized **Note:** Receive-only nodes don't need to configure peer connections. They simply process incoming ripple requests from nodes that have them configured as peers. ### Both Mode (Full Mesh)[​](#both-mode-full-mesh "Direct link to Both Mode (Full Mesh)") Node both sends and receives ripples. **Use Case:** Multi-master deployments, peer-to-peer sync **Configure each peer separately:** ``` # On Node 1: Add Node 2 as peer POST /api/ripples/config { "name": "node2", "url": "https://node2.production.google.ekodb.net:8080", "api_key": "node2-admin-key", "mode": "Operations", "enabled": true } # On Node 2: Add Node 1 as peer POST /api/ripples/config { "name": "node1", "url": "https://node1.production.google.ekodb.net:8080", "api_key": "node1-admin-key", "mode": "Operations", "enabled": true } ``` ### None Mode[​](#none-mode "Direct link to None Mode") Node operates independently with no ripples. **Use Case:** Isolated analytics nodes, testing environments **No configuration needed:** Simply don't add any ripple peers to the node. ## Real-World Use Cases[​](#real-world-use-cases "Direct link to Real-World Use Cases") ### 1. Geographic Distribution (Multi-Region)[​](#1-geographic-distribution-multi-region "Direct link to 1. Geographic Distribution (Multi-Region)") **Scenario:** E-commerce platform with users in US, EU, and Asia **Architecture:** * Primary node in each region (both mode for bidirectional sync) * Cross-region ripples for data consistency * Read replicas in each region (receive mode) ``` US Region EU Region Asia Region ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Primary │◄──────────►│ Primary │◄──────────►│ Primary │ │ (both) │ │ (both) │ │ (both) │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ ├─────┐ ├────┴────┐ ├────┴────┐ ↓ ↓ ↓ ↓ ↓ ↓ ┌────────┐┌────────┐┌────────┐┌────────┐┌────────┐┌────────┐ │Replica1││Replica2││Replica1││Replica2││Replica1││Replica2│ │(receive)││(receive)││(receive)││(receive)││(receive)││(receive)│ └────────┘└────────┘└────────┘└────────┘└────────┘└────────┘ ``` **Benefits:** * Users read from nearest region (low latency) * Writes propagate globally (eventual consistency) * Failover to another region if one goes down ### 2. Read Scaling (Analytics Workload)[​](#2-read-scaling-analytics-workload "Direct link to 2. Read Scaling (Analytics Workload)") **Scenario:** SaaS application with heavy reporting queries **Architecture:** * Primary write node with durable WAL (send mode) * Multiple read replicas with fast WAL (receive mode) * Analytics nodes isolated from production (none mode) ``` Production Traffic Analytics Traffic │ │ ↓ ↓ ┌─────────────┐ ┌─────────────┐ │ Primary │──────────────►│ Analytics │ │ (send) │ WAL Export │ (none) │ │ Durable WAL │ │ Fast WAL │ └──────┬──────┘ └─────────────┘ │ Ripples ├──────────┬──────────┐ ↓ ↓ ↓ ┌──────────┐┌──────────┐┌──────────┐ │ Replica1 ││ Replica2 ││ Replica3 │ │(receive) ││(receive) ││(receive) │ │ Fast WAL ││ Fast WAL ││ Fast WAL │ └──────────┘└──────────┘└──────────┘ ``` **Benefits:** * Production writes don't slow down (durable primary) * Replicas handle read traffic (fast WAL, high throughput) * Analytics isolated (no ripple overhead, batch data loads) **Configuration Example:** ``` # Primary (write node) - Add each replica as a peer curl -X POST https://primary.production.google.ekodb.net/api/ripples/config \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "replica1", "url": "https://replica1.production.google.ekodb.net:8080", "api_key": "replica1-admin-key", "mode": "Operations", "enabled": true }' curl -X POST https://primary.production.google.ekodb.net/api/ripples/config \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "replica2", "url": "https://replica2.production.google.ekodb.net:8080", "api_key": "replica2-admin-key", "mode": "Operations", "enabled": true }' curl -X POST https://primary.production.google.ekodb.net/api/ripples/config \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "replica3", "url": "https://replica3.production.google.ekodb.net:8080", "api_key": "replica3-admin-key", "mode": "Operations", "enabled": true }' # Read Replicas (no peer config needed - they only receive) # Analytics node (isolated - no peer config needed) ``` ### 3. High Availability (Active-Active)[​](#3-high-availability-active-active "Direct link to 3. High Availability (Active-Active)") Active-active works, but there is no conflict resolution You **can** run two writable nodes peering at each other — bidirectional propagation with automatic loop prevention (request-ID dedup, origin rejection, hop-count cap) is implemented and works. What's missing is **conflict resolution**: a received write is blind-applied (no last-write-wins, no vector clocks), so two nodes writing the **same record** concurrently will **diverge** (each keeps the other's value). This is safe when writes are **partitioned by node** (each node owns distinct keys/regions) or contention is low; avoid it for hot-key contention. Note also that Ripples does not auto-promote on failover — route writes with an external load balancer/health check. **Scenario:** Multi-region app where each region owns its own keyspace **Architecture:** * Two writable nodes, peered bidirectionally (`both` direction) * Writes partitioned by region/key so the same record isn't written on both * External load balancer for routing (no automatic promotion in Ripples) * Conflict handling: none — last write by arrival (partition writes to avoid same-key conflicts) ``` ┌──────────────┐ │Load Balancer │ └──────┬───────┘ │ ┌───┴────┐ ↓ ↓ ┌────────┐┌────────┐ │Primary1││Primary2│ │ (both) ││ (both) │ └────┬───┘└───┬────┘ └────┬───┘ │ Bidirectional Ripples ``` **Benefits:** * Both nodes accept writes (active-active) * Automatic failover (no manual intervention) * Zero downtime for maintenance **Example Workflow:** ``` # Node 1 configuration - Add Node 2 as peer curl -X POST https://node1.production.google.ekodb.net/api/ripples/config \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "node2", "url": "https://node2.production.google.ekodb.net:8080", "api_key": "node2-admin-key", "mode": "Operations", "enabled": true }' # Node 2 configuration - Add Node 1 as peer curl -X POST https://node2.production.google.ekodb.net/api/ripples/config \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "node1", "url": "https://node1.production.google.ekodb.net:8080", "api_key": "node1-admin-key", "mode": "Operations", "enabled": true }' # Write to Node 1 curl -X POST https://node1.production.google.ekodb.net/api/insert/orders \ -H "Authorization: Bearer $TOKEN" \ -d '{"data": {"product": "Widget", "amount": 100}}' # Immediately available on Node 2 via ripple curl -X POST https://node2.production.google.ekodb.net/api/find/orders \ -H "Authorization: Bearer $TOKEN" \ -d '{"filter": {"type": "Condition", "content": {"field": "product", "operator": "Eq", "value": "Widget"}}}' ``` ### 4. Data Ingestion Pipeline[​](#4-data-ingestion-pipeline "Direct link to 4. Data Ingestion Pipeline") **Scenario:** IoT platform collecting sensor data from thousands of devices **Architecture:** * Ingestion nodes accept writes only (send mode) * Storage nodes receive and persist (receive mode, durable WAL) * Processing nodes compute analytics (receive mode) ``` IoT Devices ├────┬────┬────┐ ↓ ↓ ↓ ↓ ┌────┐┌────┐┌────┐┌────┐ │Ing1││Ing2││Ing3││Ing4│ ← Fast WAL (write-optimized) │send││send││send││send│ └──┬─┘└──┬─┘└──┬─┘└──┬─┘ │ Ripples │ │ └──────┬──────┴─────┘ │ ┌─────┼─────┐ ↓ ↓ ↓ ┌────────┐┌────────┐┌────────┐ │Storage1││Storage2││Process │ ← Durable WAL (data safety) │receive ││receive ││receive │ └────────┘└────────┘└────────┘ ``` **Benefits:** * Ingestion nodes optimized for write throughput * Storage nodes ensure data durability * Processing nodes compute real-time analytics * Horizontal scaling at each layer ### 5. Development/Staging Environments[​](#5-developmentstaging-environments "Direct link to 5. Development/Staging Environments") **Scenario:** Testing environment that mirrors production data **Architecture:** * Production primary (send mode) * Staging replica (receive mode) * Staging accepts test writes locally (send mode disabled) ``` Production Staging ┌──────────┐ ┌──────────┐ │ Primary │──────────────►│ Replica │ │ (send) │ Ripples │(receive) │ └──────────┘ └──────────┘ ↑ │ Test writes │ (not rippled back) ``` **Benefits:** * Staging has real production data * Test writes don't affect production * Safe environment for testing features ## Configuring Ripples[​](#configuring-ripples "Direct link to Configuring Ripples") ### Basic Configuration[​](#basic-configuration "Direct link to Basic Configuration") Configure ripples on a node via the ripples API. Each peer requires a separate configuration: * Client Libraries (Recommended) * Direct API - 🦀 Rust - 📘 TypeScript - 📦 JavaScript - 🐍 Python - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::{Client, RippleConfig}; let client = Client::builder() .base_url("https://your-ekodb-instance.com") .api_key("your-admin-token") .build()?; let config = RippleConfig { name: "peer-node-1".to_string(), url: "https://peer1.production.google.ekodb.net:8080".to_string(), api_key: "peer1-admin-key".to_string(), mode: "Operations".to_string(), enabled: true, }; client.configure_ripple(config).await?; ``` ``` import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://your-ekodb-instance.com", apiKey: "your-admin-token", }); await client.init(); await client.configureRipple({ name: "peer-node-1", url: "https://peer1.production.google.ekodb.net:8080", api_key: "peer1-admin-key", mode: "Operations", enabled: true, }); ``` ``` const { EkoDBClient } = require("@ekodb/ekodb-client"); const client = new EkoDBClient({ baseURL: "https://your-ekodb-instance.com", apiKey: "your-admin-token", }); await client.init(); await client.configureRipple({ name: "peer-node-1", url: "https://peer1.production.google.ekodb.net:8080", api_key: "peer1-admin-key", mode: "Operations", enabled: true, }); ``` ``` from ekodb_client import Client client = Client.new( 'https://your-ekodb-instance.com', 'your-admin-token' ) client.configure_ripple({ 'name': 'peer-node-1', 'url': 'https://peer1.production.google.ekodb.net:8080', 'api_key': 'peer1-admin-key', 'mode': 'Operations', 'enabled': True }) ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://your-ekodb-instance.com") .apiKey("your-admin-token") .build() client.configureRipple(mapOf( "name" to "peer-node-1", "url" to "https://peer1.production.google.ekodb.net:8080", "api_key" to "peer1-admin-key", "mode" to "Operations", "enabled" to true )) ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://your-ekodb-instance.com", "your-admin-token", ) err := client.ConfigureRipple(map[string]interface{}{ "name": "peer-node-1", "url": "https://peer1.production.google.ekodb.net:8080", "api_key": "peer1-admin-key", "mode": "Operations", "enabled": true, }) if err != nil { log.Fatal(err) } ``` ``` curl -X POST https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "peer-node-1", "url": "https://peer1.production.google.ekodb.net:8080", "api_key": "peer1-admin-key", "mode": "Operations", "enabled": true }' # Response { "status": "success", "peer_name": "peer-node-1", "ripple_enabled": true } ``` ### List Configured Ripples[​](#list-configured-ripples "Direct link to List Configured Ripples") note Ripple management is an admin-only operation. Use the Direct API with an admin token. * Direct API ``` curl https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response (array of RippleConfig objects) [ { "name": "peer1", "url": "https://peer1.production.google.ekodb.net:8080", "mode": "operations", "enabled": true }, { "name": "peer2", "url": "https://peer2.production.google.ekodb.net:8080", "mode": "operations", "enabled": true } ] ``` ### Remove Ripple Configuration[​](#remove-ripple-configuration "Direct link to Remove Ripple Configuration") Not Yet Implemented The DELETE endpoint for ripple configurations is not yet available. To disable a ripple, update its configuration with `"enabled": false`: ``` curl -X PUT https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "peer-node-1", "enabled": false }' ``` ## Bypassing Ripples[​](#bypassing-ripples "Direct link to Bypassing Ripples") For bulk imports or maintenance operations, you can bypass ripple propagation: ### Single Operations[​](#single-operations "Direct link to Single Operations") ``` POST https://{EKODB_API_URL}/api/insert/users?bypass_ripple=true Authorization: Bearer {TOKEN} Content-Type: application/json { "data": { "name": "John Doe", "email": "john@example.com" } } ``` ### Batch Operations[​](#batch-operations "Direct link to Batch Operations") ``` POST https://{EKODB_API_URL}/api/batch/insert/users?bypass_ripple=true Authorization: Bearer {TOKEN} Content-Type: application/json { "inserts": [ {"data": {"name": "User 1"}}, {"data": {"name": "User 2"}}, {"data": {"name": "User 3"}} ] } ``` **When to Bypass:** * Initial data seeding * Large bulk imports * Database migrations * Maintenance operations * Temporary disconnection scenarios ### Ripples in Transactions[​](#ripples-in-transactions "Direct link to Ripples in Transactions") Transactional writes ripple at commit. Each write staged inside a transaction carries its own `bypass_ripple` intent (default `false` = propagate to peers; `true` = suppress), and propagation happens once the transaction commits. Staged writes are never rippled before commit, and a rolled-back or conflicted (`409`) transaction ripples nothing. A single transaction may mix per-operation choices. ## Loop Prevention[​](#loop-prevention "Direct link to Loop Prevention") ekoDB implements automatic loop prevention to avoid infinite ripple cycles in multi-node deployments. Operations are automatically deduplicated across all nodes, ensuring each operation is processed exactly once regardless of network topology. **Key guarantees:** * Operations are never processed more than once per node * No infinite loops in bidirectional or full-mesh deployments * Automatic deduplication with no manual configuration required * Works transparently across all replication patterns ## Performance Considerations[​](#performance-considerations "Direct link to Performance Considerations") ### Ripple Overhead[​](#ripple-overhead "Direct link to Ripple Overhead") **Minimal for typical workloads:** * Negligible CPU overhead per operation * Network latency depends on topology **Batch operations scale well:** * Single ripple for entire batch * Amortized overhead across records ### Network Topology[​](#network-topology "Direct link to Network Topology") **Full Mesh (N nodes):** * Each write ripples to (N-1) peers * Best for: Small clusters (2-5 nodes) * Network traffic: O(N²) **Hub-and-Spoke:** * Primary sends to all replicas * Replicas don't send to each other * Best for: Read scaling (1 primary, N replicas) * Network traffic: O(N) **Hybrid:** * Primaries in full mesh * Replicas receive only * Best for: Multi-region with local replicas * Network traffic: O(P²) + O(R) where P=primaries, R=replicas ## Monitoring Ripples[​](#monitoring-ripples "Direct link to Monitoring Ripples") ### Check Ripple Status[​](#check-ripple-status "Direct link to Check Ripple Status") ``` curl -X GET https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` ### Monitor Replication Lag[​](#monitor-replication-lag "Direct link to Monitor Replication Lag") ``` # Check WAL health for replication status curl -X GET https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '.last_entry_timestamp' ``` ### Health Checks[​](#health-checks "Direct link to Health Checks") ``` # Verify all peers are healthy for peer in peer1 peer2 peer3; do echo "Checking $peer..." curl -s https://$peer.production.google.ekodb.net/api/health | jq '.status' done ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### 1. Configure Primary Nodes to Send to Replicas[​](#1-configure-primary-nodes-to-send-to-replicas "Direct link to 1. Configure Primary Nodes to Send to Replicas") **Why:** Prevents write conflicts and simplifies conflict resolution **Add each replica as a peer on the primary node:** ``` POST /api/ripples/config { "name": "replica1", "url": "https://replica1.production.google.ekodb.net:8080", "api_key": "replica1-admin-key", "mode": "Operations", "enabled": true } ``` ### 2. Bypass Ripples for Bulk Operations[​](#2-bypass-ripples-for-bulk-operations "Direct link to 2. Bypass Ripples for Bulk Operations") **Why:** Reduces network overhead during large imports ``` # Initial data load curl -X POST .../api/batch/insert/users?bypass_ripple=true \ -d '{"inserts": [...]}' # After import, sync manually via WAL ``` ### 3. Monitor Ripple Health[​](#3-monitor-ripple-health "Direct link to 3. Monitor Ripple Health") **Why:** Detect failures early, prevent data divergence ``` # Daily health check */5 * * * * /scripts/check_ripple_health.sh ``` ### 4. Read Replicas Need No Configuration[​](#4-read-replicas-need-no-configuration "Direct link to 4. Read Replicas Need No Configuration") **Why:** Prevents accidental writes from replicating back **Read replicas simply receive ripples from primary nodes that have them configured as peers.** No peer configuration is needed on the replica itself. ### 5. Isolate Analytics Nodes[​](#5-isolate-analytics-nodes "Direct link to 5. Isolate Analytics Nodes") **Why:** Prevents ripple overhead from impacting analytics queries **Simply don't add any ripple peers to analytics nodes.** They will operate independently with no replication overhead. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Ripples Not Propagating[​](#ripples-not-propagating "Direct link to Ripples Not Propagating") **Check ripple configuration:** ``` curl -X GET https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` **Verify network connectivity:** ``` # Test from source node to peer curl -k https://peer.production.google.ekodb.net:8080/api/health ``` **Check authentication:** ``` # Ensure peer has valid admin token curl -X POST https://peer.production.google.ekodb.net/api/auth/token \ -d '{"api_key": "your-admin-key"}' ``` ### Replication Lag[​](#replication-lag "Direct link to Replication Lag") **Check WAL entries:** ``` # Compare timestamps between primary and replica curl -s https://primary.production.google.ekodb.net/api/wal/health | jq '.last_entry_timestamp' curl -s https://replica.production.google.ekodb.net/api/wal/health | jq '.last_entry_timestamp' ``` **Monitor network latency:** ``` ping peer.production.google.ekodb.net ``` ### Duplicate Operations[​](#duplicate-operations "Direct link to Duplicate Operations") **Verify loop prevention:** * Ensure all nodes are running and healthy * Verify ripple configuration is correct on each node * Check network connectivity between nodes ## Security[​](#security "Direct link to Security") ### Authentication[​](#authentication "Direct link to Authentication") All ripple endpoints require admin authentication: ``` Authorization: Bearer {ADMIN_TOKEN} ``` ### TLS/SSL Required[​](#tlsssl-required "Direct link to TLS/SSL Required") Ripples always use HTTPS: ``` ✓ https://peer.production.google.ekodb.net:8080 (secure) ✗ http://peer.production.google.ekodb.net:8080 (rejected) ``` ### Network Isolation[​](#network-isolation "Direct link to Network Isolation") **Recommended:** Use private networks (VPC) for ripple traffic ``` Production: 10.0.0.0/16 (VPC) Peer URLs: https://10.0.0.2:8080, https://10.0.0.3:8080 ``` ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [System Administration](/system-administration.md) - WAL management and replication * [Batch Operations](/batch-operations.md) - Bypass ripples for bulk operations * [Basic Operations](/basic-operations.md) - Single-record ripple behavior * [White Paper](/white-paper.md) - Architecture deep dive *** *For questions or support with ripple configuration, contact * --- # Scheduled Functions Automate recurring tasks, maintenance operations, and time-based workflows with ekoDB's built-in scheduler. Background Execution Scheduled functions run automatically in the background without blocking database operations. ## Quick Start[​](#quick-start "Direct link to Quick Start") * Direct API (Recommended) * Client Libraries ``` # 1. Create a function to schedule curl -X POST https://{EKODB_API_URL}/api/functions \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "daily_cleanup", "description": "Remove old log records", "functions": [{ "type": "Delete", "collection": "logs", "filter": { "type": "Condition", "content": { "field": "created_at", "operator": "Lt", "value": "{{cutoff_date}}" } } }] }' # 2. Schedule it to run daily at midnight curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Daily Log Cleanup", "function_label": "daily_cleanup", "cron_expression": "0 0 0 * * *", "parameters": {"cutoff_date": "2026-01-01T00:00:00Z"}, "enabled": true, "timezone": "UTC" }' ``` Schedule management is available in all client libraries. See the [Advanced Operations](/client-libraries/advanced-operations.md#schedule-management) guide for examples in Rust, Python, TypeScript, Go, and Kotlin. **Available methods:** `createSchedule`, `listSchedules`, `getSchedule`, `updateSchedule`, `deleteSchedule`, `pauseSchedule`, `resumeSchedule` ## Core Concepts[​](#core-concepts "Direct link to Core Concepts") ### Cron Expressions[​](#cron-expressions "Direct link to Cron Expressions") ekoDB uses **6-field cron expressions**: ``` ┌───────────── second (0-59) │ ┌───────────── minute (0-59) │ │ ┌───────────── hour (0-23) │ │ │ ┌───────────── day of month (1-31) │ │ │ │ ┌───────────── month (1-12 or JAN-DEC) │ │ │ │ │ ┌───────────── day of week (0-6 or SUN-SAT) │ │ │ │ │ │ * * * * * * ``` ### Common Schedules[​](#common-schedules "Direct link to Common Schedules") | Expression | Description | Runs At | | ------------------- | ----------------- | ------------------------------- | | `0 * * * * *` | Every minute | :00 seconds of every minute | | `0 0 * * * *` | Every hour | Top of every hour | | `0 0 0 * * *` | Daily at midnight | 00:00:00 every day | | `0 0 9 * * MON-FRI` | Weekdays at 9 AM | 9:00 AM Monday-Friday | | `0 0 0 1 * *` | Monthly on 1st | Midnight on 1st of month | | `0 */15 * * * *` | Every 15 minutes | :00, :15, :30, :45 of each hour | | `0 30 2 * * *` | Daily at 2:30 AM | 2:30 AM every day | ### Special Characters[​](#special-characters "Direct link to Special Characters") * `*` - Any value (every) * `,` - Value list separator (`1,15,30`) * `-` - Range (`MON-FRI`, `9-17`) * `/` - Step values (`*/15` = every 15 units) ## Creating Schedules[​](#creating-schedules "Direct link to Creating Schedules") ### Step 1: Create a Function[​](#step-1-create-a-function "Direct link to Step 1: Create a Function") First, create the function you want to schedule: ``` curl -X POST https://{EKODB_API_URL}/api/functions \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "generate_report", "description": "Generate daily analytics report", "functions": [ { "type": "Query", "collection": "events", "filter": { "type": "Condition", "content": { "field": "timestamp", "operator": "Gt", "value": "{{start_date}}" } } }, { "type": "Group", "by_fields": ["event_type"], "functions": [{ "output_field": "total", "operation": "Count" }] }, { "type": "ForEach", "functions": [ { "type": "Insert", "collection": "reports", "record": { "event_type": "{{event_type}}", "count": "{{total}}", "since": "{{start_date}}" } } ] } ] }' ``` This actually persists the report: `Query` loads the events, `Group` produces one record per `event_type` shaped like `{ "event_type": "click", "total": 12 }` (the grouping key is stored under the by-field name), and `ForEach` iterates those grouped records — exposing each record's fields as `{{event_type}}` and `{{total}}` placeholders — so the nested `Insert` writes one row per group into `reports`. `{{start_date}}` is the schedule parameter, available in every stage. Persisting computed/aggregated results A stage like `Insert` writes the **literal** record in its definition; it does not automatically carry the pipeline's computed data. To persist what an earlier stage produced (a `Query`/`Group`/`Project` result), wrap the write in `ForEach` and reference the current record's fields with **text `{{field}}`** placeholders. Use the text form (not the structural `{ "type": "Parameter", "name": "field" }` form) for ForEach-injected fields: they are resolved per iteration, whereas structural parameters must be declared function parameters and are rejected by the save-time validator. ### Step 2: Schedule the Function[​](#step-2-schedule-the-function "Direct link to Step 2: Schedule the Function") ``` curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Daily Analytics Report", "description": "Generate analytics every day at 1 AM", "function_label": "generate_report", "cron_expression": "0 0 1 * * *", "parameters": { "start_date": "2026-01-01T00:00:00Z" }, "enabled": true, "timezone": "UTC" }' # Response includes schedule id and next_execution timestamp ``` Schedule parameters are passed verbatim The `parameters` object is stored as-is and passed to the function unchanged on every run — there is no execution-time `{{NOW}}`/current-time substitution. Do not put a fixed timestamp in `parameters` expecting it to track each run. To record when a run executed, stamp it inside the function with a `CurrentDatetime` stage (which writes the current UTC time onto the working record) rather than passing it as a parameter. Cron is evaluated in the schedule's timezone The cron expression is evaluated in the schedule's `timezone` (`calculate_next_execution` resolves the next occurrence in that zone, then stores it as an absolute UTC instant). The `timezone` accepts any IANA name (e.g. `UTC`, `America/New_York`, `Europe/London`) and defaults to `UTC`. Daylight-saving transitions are handled automatically, so a `0 0 9 * * *` schedule in `America/New_York` fires at 9am New York wall-clock time year round. An invalid timezone is rejected at create/update time. ## Managing Schedules[​](#managing-schedules "Direct link to Managing Schedules") ### List All Schedules[​](#list-all-schedules "Direct link to List All Schedules") ``` curl -X GET https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" ``` ### Get Schedule by ID[​](#get-schedule-by-id "Direct link to Get Schedule by ID") ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" ``` ### Update Schedule[​](#update-schedule "Direct link to Update Schedule") ``` # Change schedule time curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "cron_expression": "0 0 2 * * *" }' # Update parameters curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "parameters": { "start_date": "2026-06-01T00:00:00Z" } }' ``` ### Enable/Disable Schedule[​](#enabledisable-schedule "Direct link to Enable/Disable Schedule") ``` # Disable temporarily curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{"enabled": false}' # Re-enable curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{"enabled": true}' ``` ### Delete Schedule[​](#delete-schedule "Direct link to Delete Schedule") ``` curl -X DELETE https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" ``` ## Execution & Monitoring[​](#execution--monitoring "Direct link to Execution & Monitoring") ### Execution Lifecycle[​](#execution-lifecycle "Direct link to Execution Lifecycle") 1. **Schedule Check** (every 60 seconds) * ScheduleManager loads all enabled schedules * Calculates if `next_execution <= now` 2. **Function Execution** * Calls the associated function with parameters * Measures execution time * Updates `last_execution` timestamp 3. **Statistics Update** * Increments `total_executions` * Updates success/failure counts * Calculates average execution time * Logs errors if execution fails 4. **Next Execution Calculation** * Determines next run based on cron expression * Updates `next_execution` field ### Execution Statistics[​](#execution-statistics "Direct link to Execution Statistics") ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" # Response includes stats object: # { # "stats": { # "total_executions": 42, # "successful_executions": 41, # "failed_executions": 1, # "avg_execution_time_ms": 125.3, # "last_error": null # } # } ``` You can also retrieve aggregate stats across all schedules: ``` curl -X GET https://{EKODB_API_URL}/api/schedules/stats \ -H "Authorization: Bearer {TOKEN}" ``` ### Error Handling[​](#error-handling "Direct link to Error Handling") When a scheduled function fails, check the schedule's `stats.last_error` field: ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" # Check stats.failed_executions and stats.last_error in the response ``` ## Real-World Use Cases[​](#real-world-use-cases "Direct link to Real-World Use Cases") ### Data Cleanup[​](#data-cleanup "Direct link to Data Cleanup") ``` # Remove old records daily at 3 AM curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Delete Old Logs", "function_label": "cleanup_old_logs", "cron_expression": "0 0 3 * * *", "parameters": {"days_to_keep": 30}, "enabled": true }' ``` ### Periodic Reports[​](#periodic-reports "Direct link to Periodic Reports") ``` # Generate weekly summary on Sundays at 9 AM curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Weekly Summary", "function_label": "generate_weekly_summary", "cron_expression": "0 0 9 * * SUN", "parameters": {"email_to": "team@company.com"}, "enabled": true }' ``` ### Data Synchronization[​](#data-synchronization "Direct link to Data Synchronization") ``` # Sync with external API every hour curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Sync External Data", "function_label": "sync_external_api", "cron_expression": "0 0 * * * *", "parameters": {"api_endpoint": "https://api.example.com/data"}, "enabled": true }' ``` ### Cache Warming[​](#cache-warming "Direct link to Cache Warming") ``` # Pre-load frequently accessed data every 30 minutes curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Warm Cache", "function_label": "warm_product_cache", "cron_expression": "0 */30 * * * *", "parameters": {"top_n": 100}, "enabled": true }' ``` ### Backup & Archival[​](#backup--archival "Direct link to Backup & Archival") ``` # Archive old data on the 1st of each month curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Monthly Archive", "function_label": "archive_old_data", "cron_expression": "0 0 0 1 * *", "parameters": { "archive_collection": "archived_data", "months_old": 6 }, "enabled": true }' ``` ## Parameters[​](#parameters "Direct link to Parameters") A schedule passes a fixed `parameters` object to the function on every run. The values are stored verbatim and bound to the function's declared parameters when it executes. ``` curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Daily Report", "function_label": "generate_report", "cron_expression": "0 0 1 * * *", "parameters": { "start_date": "2026-01-01T00:00:00Z", "recipient": "reports@company.com" }, "enabled": true }' ``` Generating "now" inside the function There is no `{{NOW}}` / `{{NOW-1d}}` schedule-parameter substitution. The function parameter substituter only resolves call-time `{{param}}` placeholders and `{{env.VAR}}` environment values. To stamp the current time inside a function, use the `CurrentDatetime` stage (writes UTC "now" to a working-record field) rather than a magic schedule parameter. ## Best Practices[​](#best-practices "Direct link to Best Practices") ### 1. Choose Appropriate Times[​](#1-choose-appropriate-times "Direct link to 1. Choose Appropriate Times") ``` # ✅ Good - Run during low-traffic hours "cron_expression": "0 0 3 * * *" # 3 AM # ❌ Avoid - Peak business hours "cron_expression": "0 0 12 * * *" # Noon ``` ### 2. Set Realistic Intervals[​](#2-set-realistic-intervals "Direct link to 2. Set Realistic Intervals") ``` # ✅ Good - Hourly for data sync "cron_expression": "0 0 * * * *" # ❌ Avoid - Every second (too frequent) "cron_expression": "* * * * * *" ``` ### 3. Monitor Execution Times[​](#3-monitor-execution-times "Direct link to 3. Monitor Execution Times") Retrieve a schedule and check `stats.avg_execution_time_ms`. If execution time is growing, consider optimizing the underlying function. ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" # Check stats.avg_execution_time_ms in the response ``` ### 4. Handle Failures Gracefully[​](#4-handle-failures-gracefully "Direct link to 4. Handle Failures Gracefully") Build error handling into your function definition with the `TryCatch` stage — it runs `try_functions`, and if any fail, runs `catch_functions` instead (capturing the error into `output_error_field`): ``` curl -X POST https://{EKODB_API_URL}/api/functions \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "safe_cleanup", "functions": [ { "type": "TryCatch", "try_functions": [ { "type": "Delete", "collection": "logs", "filter": { "type": "Condition", "content": { "field": "created_at", "operator": "Lt", "value": "{{cutoff_date}}" } } } ], "catch_functions": [ { "type": "Insert", "collection": "cleanup_errors", "record": { "stage": "delete_logs", "error": "{{error}}" } } ], "output_error_field": "error" } ] }' ``` ### 5. Set the Timezone Instead of Pre-Converting to UTC[​](#5-set-the-timezone-instead-of-pre-converting-to-utc "Direct link to 5. Set the Timezone Instead of Pre-Converting to UTC") Cron expressions are evaluated in the schedule's `timezone`, so write the expression in local wall-clock time and let the scheduler handle the UTC conversion (including daylight saving). Set `timezone` to the IANA name for the location. ``` # A 9 AM Eastern weekday report — no manual UTC math, DST handled for you curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "Business Hours Report", "function_label": "report", "cron_expression": "0 0 9 * * MON-FRI", "timezone": "America/New_York", "enabled": true }' ``` ### 6. Test With a Wide Cron, Then Tighten[​](#6-test-with-a-wide-cron-then-tighten "Direct link to 6. Test With a Wide Cron, Then Tighten") The manual trigger endpoint only fires schedules that are currently enabled — the scheduler caches enabled schedules in memory and `trigger` looks the schedule up there, so a disabled schedule returns `Schedule not found`. To dry-run a new schedule, create it enabled with a far-future or infrequent cron, trigger it manually to verify, then update the cron to the real cadence. ``` # 1. Create enabled with an infrequent cron so it won't auto-fire while testing curl -X POST https://{EKODB_API_URL}/api/schedules \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "New Schedule", "function_label": "new_function", "cron_expression": "0 0 0 1 1 *", "enabled": true }' # 2. Manually trigger to test (works only on enabled schedules) curl -X POST https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID}/trigger \ -H "Authorization: Bearer {TOKEN}" # 3. Update to the real cadence after verification curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{"cron_expression": "0 0 * * * *"}' ``` ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Schedule Not Running[​](#schedule-not-running "Direct link to Schedule Not Running") **Problem**: Schedule exists but function never executes **Solutions**: 1. Check `enabled` is `true` 2. Verify cron expression is valid 3. Check `next_execution` timestamp 4. Ensure function exists with correct label ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" # Verify enabled, next_execution, and function_label in the response ``` ### Function Fails[​](#function-fails "Direct link to Function Fails") **Problem**: Schedule runs but function execution fails **Solutions**: 1. Check `stats.last_error` for error message 2. Verify function parameters are correct 3. Test function manually 4. Check function permissions ``` curl -X GET https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" # Check stats.last_error and stats.failed_executions in the response ``` ### Schedule Runs at the Wrong Time[​](#schedule-runs-at-the-wrong-time "Direct link to Schedule Runs at the Wrong Time") **Problem**: Schedule fires earlier or later than expected **Cause**: The schedule's `timezone` is not what you expect, so the cron expression resolves against the wrong zone. The default is `UTC`. **Solution**: Set `timezone` to the IANA name for your location. The cron expression is then interpreted in that zone with daylight saving handled automatically — no manual UTC conversion needed. ``` # Run at 9am New York time, DST-aware curl -X PUT https://{EKODB_API_URL}/api/schedules/{SCHEDULE_ID} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{"cron_expression": "0 0 9 * * MON-FRI", "timezone": "America/New_York"}' ``` ## API Reference[​](#api-reference "Direct link to API Reference") ### Create Schedule[​](#create-schedule "Direct link to Create Schedule") ``` POST /api/schedules ``` **Request body:** | Field | Type | Required | Description | | ----------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Yes | Display name for the schedule | | `description` | string | No | Optional description | | `function_label` | string | Yes | Label of the function to execute | | `cron_expression` | string | Yes | 6-field cron expression | | `parameters` | object | No | Parameters to pass to the function | | `enabled` | boolean | Yes | Whether the schedule is active | | `timezone` | string | No | IANA timezone the cron expression is evaluated in, DST-aware (e.g. `America/New_York`); invalid names are rejected (default: `UTC`) | Admin authentication All `/api/schedules` endpoints require an admin token (they are behind the server's admin auth filter). ### List Schedules[​](#list-schedules "Direct link to List Schedules") ``` GET /api/schedules ``` ### Get Schedule[​](#get-schedule "Direct link to Get Schedule") ``` GET /api/schedules/{id} ``` ### Update Schedule[​](#update-schedule-1 "Direct link to Update Schedule") ``` PUT /api/schedules/{id} ``` **Request body:** Any subset of the fields from Create Schedule. ### Delete Schedule[​](#delete-schedule-1 "Direct link to Delete Schedule") ``` DELETE /api/schedules/{id} ``` ### Trigger Schedule[​](#trigger-schedule "Direct link to Trigger Schedule") ``` POST /api/schedules/{id}/trigger ``` Manually triggers immediate execution of a schedule, regardless of cron timing. Works only on schedules that are currently enabled — the scheduler resolves the trigger against its in-memory set of enabled schedules, so a disabled schedule returns `Schedule not found`. ### Schedule Stats[​](#schedule-stats "Direct link to Schedule Stats") ``` GET /api/schedules/stats ``` Returns aggregate statistics across all schedules. ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Functions](/client-libraries/advanced-operations.md#functions) - Create executable functions * [System Administration](/system-administration.md) - Monitor execution * [Batch Operations](/batch-operations.md) - Efficient bulk operations ## Summary[​](#summary "Direct link to Summary") Scheduled functions in ekoDB enable: ✅ **Automation** - Recurring tasks without manual intervention ✅ **Flexible scheduling** - Cron-based with 6-field precision ✅ **Background execution** - Non-blocking operation ✅ **Monitoring** - Built-in execution statistics ✅ **Error tracking** - Automatic error logging ✅ **Production-ready** - Reliable and maintainable Note: cron expressions are evaluated in the schedule's `timezone` (any IANA name, DST-aware; defaults to `UTC`). --- # Security Security is a top priority for ekoDB. Your data is protected with enterprise-grade features designed for production workloads. ## Encryption[​](#encryption "Direct link to Encryption") ### Data at Rest[​](#data-at-rest "Direct link to Data at Rest") All data stored in ekoDB is encrypted using **AES-256-GCM** encryption: * **Record data** - All field values are encrypted before storage * **Encrypted IDs** - Record identifiers are encrypted to prevent enumeration attacks * **WAL (Write-Ahead Log)** - Transaction logs are encrypted * **Backups** - All backup files inherit encryption ### Data in Transit[​](#data-in-transit "Direct link to Data in Transit") * **TLS 1.3** - All connections use modern TLS encryption * **Certificate validation** - Strict certificate verification * **Perfect Forward Secrecy** - Key exchange protects past sessions ### Encryption Keys[​](#encryption-keys "Direct link to Encryption Keys") ekoDB automatically manages encryption keys: ``` Database Key Structure: ├── Master Key (derived from your deployment) ├── Collection Keys (per-collection encryption) └── Record Keys (per-record for sensitive data) ``` Key Management The admin key generated at the time of deploying your database is automatically encrypted and rotated by app.ekodb.io to secure your account. All other API keys (that you generate) are your responsibility to rotate as needed. See [Creating Scoped API Keys](#creating-scoped-api-keys) below for how to generate new keys and revoke old ones. ## Authentication[​](#authentication "Direct link to Authentication") ### API Keys[​](#api-keys "Direct link to API Keys") ekoDB uses API keys for authentication. Each key is associated with specific permissions. **Key Types:** | Type | Description | Use Case | | ----------------- | ------------------------------- | ---------------------------------- | | **Admin Key** | Full access to all operations | Development, deployment management | | **Scoped Key** | Limited to specific collections | Production applications | | **Read-Only Key** | Read access only | Analytics, reporting | ### Creating Scoped API Keys[​](#creating-scoped-api-keys "Direct link to Creating Scoped API Keys") API key management is performed via the REST API. Use an admin token to create new scoped keys. (For first-run bootstrap you can instead pass the admin key directly in the request body as `api_key`; see [Authentication](/authentication.md#register-api-key).) ``` POST /api/auth/register Authorization: Bearer {ADMIN_TOKEN} Content-Type: application/json { "label": "production-backend", "description": "Backend service API key", "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true }, "products": { "read": true, "write": false }, "orders": { "read": true, "write": true } } } ``` **Response:** ``` { "key": "eko_prod_abc123...", "label": "production-backend", "created_at": "2026-01-28T12:00:00Z" } ``` **Example with curl:** ``` curl -X POST https://{EKODB_API_URL}/api/auth/register \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "production-backend", "description": "Backend service API key", "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true }, "products": { "read": true, "write": false }, "orders": { "read": true, "write": true } } }' ``` Client Library Usage Once you have an API key, use it to initialize any client library: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::Client; let client = Client::builder() .base_url("https://mydb.production.google.ekodb.net") .api_key("eko_prod_abc123...") .build()?; ``` ``` from ekodb_client import Client client = Client.new( 'https://mydb.production.google.ekodb.net', 'eko_prod_abc123...' ) ``` ``` import { EkoDBClient } from '@ekodb/ekodb-client'; const client = new EkoDBClient({ baseURL: 'https://mydb.production.google.ekodb.net', apiKey: 'eko_prod_abc123...' }); await client.init(); ``` ``` const { EkoDBClient } = require('@ekodb/ekodb-client'); const client = new EkoDBClient({ baseURL: 'https://mydb.production.google.ekodb.net', apiKey: 'eko_prod_abc123...' }); await client.init(); ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl("https://mydb.production.google.ekodb.net") .apiKey("eko_prod_abc123...") .build() ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://mydb.production.google.ekodb.net", "eko_prod_abc123...", ) ``` ### JWT Tokens[​](#jwt-tokens "Direct link to JWT Tokens") API keys are exchanged for JWT tokens for API access: 1. **Token Generation** - Client libraries automatically handle this 2. **Token Lifetime** - Default 1 hour (configurable) 3. **Token Refresh** - Automatic refresh before expiration 4. **Token Revocation** - Delete the API key to revoke all associated tokens ## Access Control[​](#access-control "Direct link to Access Control") ### Collection-Level Permissions[​](#collection-level-permissions "Direct link to Collection-Level Permissions") Control access at the collection level: | Permission | Description | | ---------- | ---------------------------------- | | `read` | Query, find, and search records | | `write` | Insert, update, and delete records | ### Field-Level Access Control[​](#field-level-access-control "Direct link to Field-Level Access Control") Restrict access to specific fields within a collection: ``` { "collection_permissions": { "users": { "read": true, "write": true, "select_fields": ["id", "name", "email"], "exclude_fields": ["password_hash", "ssn"], "writable_fields": ["name", "email", "preferences"] } } } ``` | Field | Description | | ----------------- | ------------------------------------- | | `select_fields` | Only return these fields (allowlist) | | `exclude_fields` | Never return these fields (blocklist) | | `writable_fields` | Only allow writes to these fields | Sensitive Data Use `exclude_fields` to ensure sensitive data like password hashes, SSNs, or internal flags are never exposed to API consumers. ## Network Security[​](#network-security "Direct link to Network Security") ### Supported Protocols[​](#supported-protocols "Direct link to Supported Protocols") ekoDB **only** supports secure connections: | Protocol | Status | Description | | -------- | ------------ | ------------------------- | | HTTPS | ✅ Supported | REST API connections | | WSS | ✅ Supported | WebSocket connections | | HTTP | ❌ Blocked | Rejected at load balancer | | WS | ❌ Blocked | Rejected at load balancer | ### Rate Limiting[​](#rate-limiting "Direct link to Rate Limiting") Protect your deployment from abuse. On the managed platform, both **authenticated** API requests (made with your API key) and **public** (unauthenticated) endpoints are rate-limited per plan. The platform sets these limits for each plan; they are not changed through `/api/config`: | Plan | Authenticated (req/min) | Public endpoints (req/min) | | ----------------------- | ----------------------- | -------------------------- | | **Free** | 200 | 400 | | **Developer** | 500 | 1,000 | | **Startup** | 1,000 | 2,000 | | **Business** | 2,000 | 4,000 | | **Professional** | 5,000 | 10,000 | | **Enterprise Basic** | 10,000 | 20,000 | | **Enterprise Standard** | 15,000 | 30,000 | Public endpoints — `/api/health`, `/api/auth/register`, `/api/auth/token`, and `/api/auth/{api_key}/admin_key` — are limited by client IP. Self-hosted deployments set the public limit with the `PUBLIC_RATE_LIMIT_PER_MINUTE` environment variable (default 60); see [System Administration](/system-administration.md#public-endpoint-rate-limiting). **Rate Limit Responses:** ``` HTTP/1.1 429 Too Many Requests Retry-After: 60 X-RateLimit-Limit: 200 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1706456400 ``` ### IP Allowlisting (Enterprise)[​](#ip-allowlisting-enterprise "Direct link to IP Allowlisting (Enterprise)") Restrict API access to specific IP addresses or CIDR ranges: ``` { "allowed_ips": [ "203.0.113.0/24", "198.51.100.42" ] } ``` ## API Key Management[​](#api-key-management "Direct link to API Key Management") ### Listing Keys[​](#listing-keys "Direct link to Listing Keys") View all API keys for your deployment: ``` GET /api/auth/keys Authorization: Bearer {ADMIN_TOKEN} ``` ### Rotating Keys[​](#rotating-keys "Direct link to Rotating Keys") Best practice: Rotate API keys periodically. 1. **Create new key** with same permissions via the REST API 2. **Update applications** to use new key 3. **Delete old key** after confirming new key works ``` # Step 1: Create new key curl -X POST https://{EKODB_API_URL}/api/auth/register \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "label": "production-backend-v2", "description": "Rotated key - January 2026", "is_admin": false, "collection_permissions": { "users": { "read": true, "write": true }, "products": { "read": true, "write": false } } }' # Step 2: Update your application configuration with new key # Step 3: Delete old key curl -X DELETE https://{EKODB_API_URL}/api/auth/keys/{old_key_id} \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` ### Deleting Keys[​](#deleting-keys "Direct link to Deleting Keys") Immediately revoke access by deleting an API key: ``` DELETE /api/auth/keys/{key_id} Authorization: Bearer {ADMIN_TOKEN} ``` Key Deletion Deleting an API key immediately invalidates all JWT tokens generated from that key. Ensure you've updated all applications using that key first. ## Security Best Practices[​](#security-best-practices "Direct link to Security Best Practices") ### 1. Use Environment Variables[​](#1-use-environment-variables "Direct link to 1. Use Environment Variables") Never hardcode API keys in source code: ``` # .env file (never commit to git!) EKODB_API_KEY=eko_prod_abc123... EKODB_URL=https://mydb.production.google.ekodb.net ``` * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use std::env; use ekodb_client::Client; let client = Client::builder() .base_url(&env::var("EKODB_URL").expect("EKODB_URL not set")) .api_key(&env::var("EKODB_API_KEY").expect("EKODB_API_KEY not set")) .build()?; ``` ``` import os from ekodb_client import Client client = Client.new( os.environ['EKODB_URL'], os.environ['EKODB_API_KEY'] ) ``` ``` import { EkoDBClient } from '@ekodb/ekodb-client'; const client = new EkoDBClient({ baseURL: process.env.EKODB_URL!, apiKey: process.env.EKODB_API_KEY! }); await client.init(); ``` ``` const { EkoDBClient } = require('@ekodb/ekodb-client'); const client = new EkoDBClient({ baseURL: process.env.EKODB_URL, apiKey: process.env.EKODB_API_KEY }); await client.init(); ``` ``` import io.ekodb.client.EkoDBClient val client = EkoDBClient.builder() .baseUrl(System.getenv("EKODB_URL")) .apiKey(System.getenv("EKODB_API_KEY")) .build() ``` ``` import ( "os" "github.com/ekoDB/ekodb-client-go" ) client := ekodb.NewClient( os.Getenv("EKODB_URL"), os.Getenv("EKODB_API_KEY"), ) ``` ### 2. Implement Least Privilege[​](#2-implement-least-privilege "Direct link to 2. Implement Least Privilege") Create separate API keys for different services: | Service | Permissions | | --------------- | ----------------------------- | | User Service | `users: read, write` | | Analytics | `users: read`, `events: read` | | Admin Dashboard | Admin key (restricted access) | ### 3. Monitor API Usage[​](#3-monitor-api-usage "Direct link to 3. Monitor API Usage") Review access patterns in your ekoDB dashboard: * Failed authentication attempts * Unusual query patterns * High-volume operations ### 4. Secure Your Infrastructure[​](#4-secure-your-infrastructure "Direct link to 4. Secure Your Infrastructure") * Use private networks when possible * Enable firewall rules * Implement audit logging ### 5. Handle Errors Securely[​](#5-handle-errors-securely "Direct link to 5. Handle Errors Securely") Never expose internal error details to clients: * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` match client.find("users", query, None).await { Ok(result) => result, Err(e) => { // Log full error internally eprintln!("Database error: {:?}", e); // Return generic message to client return Err(AppError::new("An error occurred. Please try again.")); } } ``` ``` try: result = await client.find('users', query) except Exception as e: # Log full error internally logger.error(f"Database error: {e}") # Return generic message to client raise HTTPException(status_code=500, detail="An error occurred. Please try again.") ``` ``` try { const result = await client.find('users', query); } catch (error) { // Log full error internally console.error('Database error:', error); // Return generic message to client throw new Error('An error occurred. Please try again.'); } ``` ``` try { const result = await client.find('users', query); } catch (error) { // Log full error internally console.error('Database error:', error); // Return generic message to client throw new Error('An error occurred. Please try again.'); } ``` ``` try { val result = client.find("users", query) } catch (e: Exception) { // Log full error internally logger.error("Database error", e) // Return generic message to client throw ResponseStatusException(HttpStatus.INTERNAL_SERVER_ERROR, "An error occurred. Please try again.") } ``` ``` result, err := client.Find("users", query) if err != nil { // Log full error internally log.Printf("Database error: %v", err) // Return generic message to client http.Error(w, "An error occurred. Please try again.", http.StatusInternalServerError) return } ``` ## Compliance[​](#compliance "Direct link to Compliance") ekoDB is designed with compliance in mind: | Standard | Status | | --------- | ------------------------------------ | | **GDPR** | ✅ Data encryption, deletion support | | **SOC 2** | 🔄 In progress | | **HIPAA** | 📋 Available on Enterprise plan | ### Data Deletion[​](#data-deletion "Direct link to Data Deletion") Support GDPR "right to be forgotten": * 🦀 Rust * 🐍 Python * 📘 TypeScript * 📦 JavaScript * 🟣 Kotlin * 🔷 Go ``` use ekodb_client::QueryBuilder; // Hard delete user data client.delete("users", &user_id, None).await?; // Delete related records by first finding their IDs, then batch-deleting let query = QueryBuilder::new().eq("user_id", &user_id).build(); let events = client.find("user_events", query, None).await?; let ids: Vec = events .iter() .filter_map(|r| r.get_string("id")) .collect(); client.batch_delete("user_events", ids, None).await?; ``` ``` from ekodb_client import QueryBuilder # Hard delete user data await client.delete('users', user_id) # Delete related records by first finding IDs, then batch-deleting query = QueryBuilder().eq('user_id', user_id).build() events = await client.find('user_events', query) ids = [r['id'] for r in events] await client.batch_delete('user_events', ids) ``` ``` import { QueryBuilder } from '@ekodb/ekodb-client'; // Hard delete user data await client.delete('users', userId); // Delete related records by first finding IDs, then batch-deleting const query = new QueryBuilder().eq('user_id', userId).build(); const events = await client.find('user_events', query); const ids = events.map((r: Record) => r.id as string); await client.batchDelete('user_events', ids); ``` ``` // Hard delete user data await client.delete('users', userId); // Delete related data with filter await client.batchDelete('user_events', { filter: { type: 'Condition', content: { field: 'user_id', operator: 'Eq', value: userId } } }); ``` ``` // Hard delete user data client.delete("users", userId) // Delete related data with filter client.batchDelete("user_events", mapOf( "filter" to mapOf( "type" to "Condition", "content" to mapOf( "field" to "user_id", "operator" to "Eq", "value" to userId ) ) )) ``` ``` // Hard delete user data if err := client.Delete("users", userID); err != nil { log.Fatal(err) } // Delete related records by first finding IDs, then batch-deleting query := ekodb.NewQueryBuilder().Eq("user_id", userID).Build() events, err := client.Find("user_events", query) if err != nil { log.Fatal(err) } ids := make([]string, 0, len(events)) for _, r := range events { if id, ok := r["id"].(string); ok { ids = append(ids, id) } } if _, err := client.BatchDelete("user_events", ids); err != nil { log.Fatal(err) } ``` ## Vulnerability Reporting[​](#vulnerability-reporting "Direct link to Vulnerability Reporting") Found a security issue? We take security seriously. **Report vulnerabilities to:** **Response times:** * Critical: 24 hours * High: 48 hours * Medium: 1 week * Low: 2 weeks We follow responsible disclosure practices and will acknowledge your contribution. --- # Support & Feedback ## Getting Help[​](#getting-help "Direct link to Getting Help") ### 1. Search the Documentation[​](#1-search-the-documentation "Direct link to 1. Search the Documentation") If you have technical questions regarding ekoDB, we recommend first searching for answers in our developer documents. Our team regularly updates these resources. ### 2. In-App Support[​](#2-in-app-support "Direct link to 2. In-App Support") Try chatting with our in-app support bot, which is capable of assisting with some common situations and questions. ### 3. Support Tickets[​](#3-support-tickets "Direct link to 3. Support Tickets") If these resources prove insufficient, or if you are experiencing deployment issues or technical issues with the application or software, we welcome you to contact our support team: * **[app.ekodb.io/support](https://app.ekodb.io/support)** - Create a support ticket * **** - Email our support team ## Contact Information[​](#contact-information "Direct link to Contact Information") ### Technical Support[​](#technical-support "Direct link to Technical Support") * **Email**: * **Support Portal**: [app.ekodb.io/support](https://app.ekodb.io/support) ### Billing Issues[​](#billing-issues "Direct link to Billing Issues") If you are experiencing billing-related issues: * **Email**: ### Feedback[​](#feedback "Direct link to Feedback") We warmly welcome your feedback on: * The ekoDB database * Our deployment management application * Our marketing website * Your product support experience * Any other interaction you may have with ekoDB or our team **Please reach out to us at**: Your feedback is very important to us, and we appreciate the opportunity to learn how we can improve directly from our users. ## Community & Resources[​](#community--resources "Direct link to Community & Resources") * **Examples**: Client library & direct HTTP examples are available in the ekoDB client repository * **Website**: [ekodb.io](https://ekodb.io) ## Next Steps[​](#next-steps "Direct link to Next Steps") * **[Quick Start](/quick-start.md)** - Get your first database running in 5 minutes * **[Getting Started](/getting-started.md)** - Comprehensive deployment and setup guide * **[Client Libraries](/client-libraries/.md)** - Install and use ekoDB client libraries --- # System Administration Administrative endpoints for monitoring, maintenance, and system analysis. Admin Access Required All endpoints in this section require admin permissions. ## Health Monitoring[​](#health-monitoring "Direct link to Health Monitoring") ### Check System Health[​](#check-system-health "Direct link to Check System Health") Get overall system health status. This endpoint is partially public — returns basic status without auth, detailed metrics with admin auth. * Client Libraries * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` // Returns Ok(()) if healthy, Err if not client.health_check().await?; ``` ``` # Returns True if healthy is_healthy = client.health_check() ``` ``` const isHealthy = await client.health(); ``` ``` const isHealthy = await client.health(); ``` ``` val isHealthy = client.health() ``` ``` err := client.Health() if err != nil { log.Fatal("Server unhealthy:", err) } ``` info Client libraries provide a simple health check that returns success/failure. For detailed system metrics, use the Direct API with an admin token. ``` # Public (no auth) - basic status curl https://{EKODB_API_URL}/api/health # Response { "status": "ok" } # Admin auth - detailed metrics curl https://{EKODB_API_URL}/api/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` **Response (with admin auth — detailed):** ``` { "status": "ok", "version": "1.2.3", "timestamp": "2026-01-15T15:45:30Z", "capacity": { "can_accept_requests": true, "status": "available", "retry_after_ms": 0, "connections_available": 450, "utilization_percent": 10.0 }, "system_resources": { "cpu_count": 8, "total_memory_gb": 16, "disk_free_gb": 100 }, "batch_settings": { "parallel_max": 1000, "sequential_max": 10000, "insert_batch_size": 500, "update_batch_size": 200, "delete_batch_size": 300, "query_batch_size": 1000 }, "active_operations": { "has_active": true, "total": 5, "by_type": { "insert": 2, "query": 3 }, "under_stress": [] }, "file_pool": { "fd_in_use": 50, "fd_max": 500, "fd_available": 450, "fd_utilization_percent": 10.0, "eviction_count": 0, "total_permit_wait_time_ms": 0 }, "retry_after_ms": null, "adaptive_strategy": { "parallelism_for_100_items": { "preparation_cpu_bound": 8, "validation_mixed": 4, "disk_io_bound": 16 }, "chunk_sizing": { "records_1000": 250, "records_10000": 500 } }, "performance_settings": { "max_concurrent_ops": 100, "operation_timeout_secs": 30, "compression_level": 6 }, "storage_limits": { "file_pool_max_size_mb": 256, "disk_cache_max_size_mb": 1024, "wal_max_size_mb": 512, "memory_cache_size_mb": 4096 } } ``` **Capacity Statuses:** * `available` — Normal operation, accepting requests * `busy` — High utilization (70–90%), still accepting requests * `overloaded` — Over capacity, check `retry_after_ms` ## Write-Ahead Log (WAL)[​](#write-ahead-log-wal "Direct link to Write-Ahead Log (WAL)") The Write-Ahead Log ensures data durability and enables replication. ### Get WAL Health[​](#get-wal-health "Direct link to Get WAL Health") Check the status and health of the WAL system. ``` curl https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "is_healthy": true, "last_flush": "2026-01-15T15:45:30Z", "buffer_utilization": 0.15, "retry_rate": 0.0, "file_size_mb": 100.0, "max_size_mb": 512, "permanent_failures": 0 } ``` **Fields:** | Field | Description | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `is_healthy` | `true` if the WAL file is below 90% of `max_size_mb` **and** `permanent_failures` is 0 | | `last_flush` | Timestamp of the most recent WAL flush to disk | | `buffer_utilization` | Percentage of WAL write buffer in use (0.0–1.0) | | `retry_rate` | Rate of write retries (0.0 = no retries) | | `file_size_mb` | Current WAL file size in megabytes | | `max_size_mb` | Maximum WAL file size before rotation is recommended | | `permanent_failures` | Count of WAL writes permanently lost (all retries exhausted) over this process's lifetime. **Must stay 0** — any nonzero value means durability was violated and `is_healthy` is forced `false`. An `event=wal_permanent_failure` critical log is also emitted; alert on it immediately. | Durability loss A nonzero `permanent_failures` means committed data could not be written to the WAL and was lost. Treat it as a SOC2-reportable incident: investigate disk health, check the structured `event=wal_permanent_failure` logs, and verify replica/backup state. ### Rotate WAL[​](#rotate-wal "Direct link to Rotate WAL") Manually rotate the WAL to a new file. ``` curl -X POST https://{EKODB_API_URL}/api/wal/rotate \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "success", "message": "WAL rotated successfully" } ``` When to Rotate * Before backups * When WAL `file_size_mb` is approaching `max_size_mb` (check via `/api/wal/health`) * During maintenance windows * For replication synchronization ### Get WAL Entries[​](#get-wal-entries "Direct link to Get WAL Entries") Retrieve WAL entries within a time range (used for replication and gap filling). ``` curl "https://{EKODB_API_URL}/api/wal/entries?from_timestamp=1705329600&to_timestamp=1705333200" \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "entries": [...], "count": 150, "from_timestamp": 1705329600, "to_timestamp": 1705333200 } ``` **Query Parameters:** * `from_timestamp` - Start Unix timestamp in seconds (required) * `to_timestamp` - End Unix timestamp in seconds (required) ## Replication[​](#replication "Direct link to Replication") ### Receive WAL Shipment[​](#receive-wal-shipment "Direct link to Receive WAL Shipment") Receive and apply WAL entries from a peer instance (for replication and gap filling). This endpoint requires the `x-ripple-request-id` header to verify the request comes from a configured peer. ``` POST https://{EKODB_API_URL}/api/replication/wal Content-Type: application/json x-ripple-request-id: {REQUEST_ID} { "entries": [...], "timestamp": 1705329600, "source_deployment_id": "primary-db-01" } # Response { "status": "ok", "entries_applied": 2, "entries_failed": 0, "total_entries": 2 } ``` ## System Analysis[​](#system-analysis "Direct link to System Analysis") ### Get System Analysis[​](#get-system-analysis "Direct link to Get System Analysis") Analyze system hardware capabilities and get recommended configuration settings. ``` GET https://{EKODB_API_URL}/api/system/analysis Authorization: Bearer {ADMIN_TOKEN} # Response { "current_specs": { "cpu_count": 8, "cpu_frequency": 2400, "total_memory": 17179869184, "disk_free": 107374182400, "has_gpu": false, "gpu_count": 0, "total_gpu_memory": 0 }, "recommended_tier": "performance", "current_settings": { "batch_parallel_max": 1000, "batch_sequential_max": 10000, "insert_batch_size": 500, "update_batch_size": 200, "delete_batch_size": 300, "query_batch_size": 1000, "max_concurrent_ops": 100, "operation_timeout": 30, "compression_level": 6 }, "recommended_settings": { "batch_parallel_max": 2000, "batch_sequential_max": 20000, "insert_batch_size": 1000, "update_batch_size": 500, "delete_batch_size": 500, "query_batch_size": 2000, "max_concurrent_ops": 200, "operation_timeout": 60, "compression_level": 3 }, "recommendations": [ "System has 8 CPU cores - optimal for parallel operations", "16GB RAM available - can handle large batch operations", "Consider increasing batch sizes for better throughput", "Fast compression recommended for better performance" ] } ``` System Analysis vs Analytics * **System Analysis** (`/api/system/analysis`) - Hardware specs and configuration recommendations * **Analytics** (`/api/analytics`) - Database statistics, collection sizes, and performance metrics ### Get Analytics Data[​](#get-analytics-data "Direct link to Get Analytics Data") Get database statistics, collection metrics, and performance data. ``` GET https://{EKODB_API_URL}/api/analytics Authorization: Bearer {ADMIN_TOKEN} # Response { "cache": { "hits": 50000, "misses": 5000, "total_requests": 55000, "hit_rate": 0.909 }, "io": { "reads": 100000, "writes": 50000 }, "cpu": { "cpu_count": 8, "usage_ratio": 0.35 }, "memory": { "total_memory": 17179869184, "used_memory": 6442450944 }, "network": { "ingress_bytes": 1073741824, "egress_bytes": 2147483648 }, "database": { "total_size": 10737418240, "available_size": 96636764160, "system_used_space": 5368709120, "db_used_space": 5368709120, "max_size": 107374182400 }, "collections": [ ["users", {"record_count": 10000, "total_size": 1048576}], ["posts", {"record_count": 50000, "total_size": 5242880}], ["events", {"record_count": 500000, "total_size": 52428800}] ], "performance_logs": [] } ``` ### Get System Logs[​](#get-system-logs "Direct link to Get System Logs") Retrieve server logs for troubleshooting. Returns the most recent log lines in chronological order. ``` curl https://{EKODB_API_URL}/api/system/logs \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "logs": [ "2026-01-15T15:40:00Z INFO [ekodb_server] Database started successfully", "2026-01-15T15:40:05Z INFO [ekodb_server::handlers] Insert completed for collection 'users'", "2026-01-15T15:45:30Z ERROR [ekodb_server::handlers] Query timeout after 30s for collection 'events'" ] } ``` The response contains an array of log line strings. The number of lines returned is controlled by the `MAX_LOG_LINES` environment variable (default: 1000). ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete system monitoring and maintenance workflow: ``` #!/bin/bash # 1. Check system health health=$(curl -s https://{EKODB_API_URL}/api/health \ -H "Authorization: Bearer {ADMIN_TOKEN}") echo "System Status: $(echo $health | jq -r '.status')" echo "Capacity: $(echo $health | jq -r '.capacity.status')" echo "CPU Cores: $(echo $health | jq -r '.system_resources.cpu_count')" # 2. Check WAL health wal_health=$(curl -s https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}") wal_size=$(echo $wal_health | jq -r '.file_size_mb') wal_max=$(echo $wal_health | jq -r '.max_size_mb') echo "WAL Size: ${wal_size}MB / ${wal_max}MB" # 3. Rotate WAL if needed (approaching max size) is_healthy=$(echo $wal_health | jq -r '.is_healthy') if [ "$is_healthy" = "false" ]; then echo "WAL approaching limit, rotating..." curl -X POST https://{EKODB_API_URL}/api/wal/rotate \ -H "Authorization: Bearer {ADMIN_TOKEN}" fi # 4. Get system analysis analysis=$(curl -s https://{EKODB_API_URL}/api/system/analysis \ -H "Authorization: Bearer {ADMIN_TOKEN}") echo "Recommendations:" echo $analysis | jq -r '.recommendations[]' # 5. Check recent logs for errors logs=$(curl -s https://{EKODB_API_URL}/api/system/logs \ -H "Authorization: Bearer {ADMIN_TOKEN}") error_count=$(echo $logs | jq '[.logs[] | select(contains("ERROR"))] | length') echo "Recent Errors: $error_count" # 6. Get WAL entries for backup (last hour) end_time=$(date +%s) start_time=$((end_time - 3600)) curl -s "https://{EKODB_API_URL}/api/wal/entries?from_timestamp=$start_time&to_timestamp=$end_time" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ > wal_backup_$(date +%Y%m%d_%H%M%S).json echo "WAL backup complete" ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Health Monitoring[​](#health-monitoring-1 "Direct link to Health Monitoring") **Set Up Regular Health Checks:** ``` # Check health every 5 minutes */5 * * * * curl -s https://{EKODB_API_URL}/api/health \ | jq -r '.status' | \ grep -q "ok" || echo "ALERT: Database unhealthy!" ``` ### WAL Management[​](#wal-management "Direct link to WAL Management") **Monitor WAL Size:** ``` # Alert if WAL is approaching max size (is_healthy = false means >90% of max) is_healthy=$(curl -s https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" | jq -r '.is_healthy') if [ "$is_healthy" = "false" ]; then echo "WARNING: WAL approaching size limit" fi ``` **Scheduled Rotation:** ``` # Rotate WAL daily at 3 AM 0 3 * * * curl -X POST https://{EKODB_API_URL}/api/wal/rotate \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` ### Log Retention[​](#log-retention "Direct link to Log Retention") **Archive Logs:** ``` # Archive current server logs curl -s https://{EKODB_API_URL}/api/system/logs \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ > archived_logs_$(date +%Y%m%d).json ``` ### Replication Setup[​](#replication-setup "Direct link to Replication Setup") **Configure Primary-Replica:** ``` # On replica: Fetch and apply WAL entries every minute */1 * * * * bash /scripts/replicate_wal.sh # replicate_wal.sh #!/bin/bash PRIMARY_URL="https://primary.production.google.ekodb.net" REPLICA_URL="https://replica.production.google.ekodb.net" # Get latest WAL entries from primary end_time=$(date +%s) start_time=$((end_time - 120)) # Last 2 minutes entries=$(curl -s "$PRIMARY_URL/api/wal/entries?from_timestamp=$start_time&to_timestamp=$end_time" \ -H "Authorization: Bearer {ADMIN_TOKEN}") # Apply to replica curl -X POST "$REPLICA_URL/api/replication/wal" \ -H "Content-Type: application/json" \ -H "x-ripple-request-id: manual-sync-$(date +%s)" \ -d "$entries" ``` ### Performance Optimization[​](#performance-optimization "Direct link to Performance Optimization") **Act on Recommendations:** ``` # Get and apply system recommendations recommendations=$(curl -s .../api/system/analysis | jq -r '.recommendations[]') # Example: Create recommended indexes echo "$recommendations" | grep "Create index" | while read -r rec; do echo "Creating index: $rec" # Parse and create index via API done # Example: Delete unused indexes echo "$recommendations" | grep "Delete unused index" | while read -r rec; do echo "Cleanup needed: $rec" done ``` ## Monitoring Alerts[​](#monitoring-alerts "Direct link to Monitoring Alerts") ### Critical Metrics to Monitor[​](#critical-metrics-to-monitor "Direct link to Critical Metrics to Monitor") | Metric | Warning Threshold | Critical Threshold | Action | | ------------------- | ----------------- | ------------------ | -------------------------------- | | Disk usage | > 75% | > 90% | Rotate WAL, archive old data | | Memory usage | > 80% | > 95% | Restart, scale up | | Query time (p95) | > 100ms | > 500ms | Create indexes, optimize queries | | WAL file size | > 800MB | > 1GB | Force rotation | | Error rate | > 1% | > 5% | Investigate logs | | Active transactions | > 100 | > 500 | Check for stuck transactions | ### Sample Monitoring Script[​](#sample-monitoring-script "Direct link to Sample Monitoring Script") ``` #!/bin/bash ALERT_EMAIL="ops@example.com" THRESHOLD_ERRORS=10 # Get health data health=$(curl -s https://{EKODB_API_URL}/api/health \ -H "Authorization: Bearer {ADMIN_TOKEN}") # Check capacity status capacity=$(echo $health | jq -r '.capacity.status') if [ "$capacity" = "overloaded" ]; then echo "ALERT: Server overloaded" | mail -s "ekoDB Alert" $ALERT_EMAIL fi # Check WAL health wal_healthy=$(curl -s https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" | jq -r '.is_healthy') if [ "$wal_healthy" = "false" ]; then echo "ALERT: WAL approaching size limit" | mail -s "ekoDB Alert" $ALERT_EMAIL fi # Check recent errors in logs error_count=$(curl -s https://{EKODB_API_URL}/api/system/logs \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '[.logs[] | select(contains("ERROR"))] | length') if [ $error_count -gt $THRESHOLD_ERRORS ]; then echo "ALERT: $error_count errors in recent logs" | mail -s "ekoDB Alert" $ALERT_EMAIL fi ``` ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### High Memory Usage[​](#high-memory-usage "Direct link to High Memory Usage") ``` # Check analytics for collection sizes curl -s https://{EKODB_API_URL}/api/analytics \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '.collections[] | {name: .[0], size: .[1].total_size, records: .[1].record_count}' \ | jq -s 'sort_by(.size) | reverse | .[0]' # Check memory metrics curl -s https://{EKODB_API_URL}/api/analytics \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '.memory' # Check health status for memory usage curl -s https://{EKODB_API_URL}/api/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '.system_resources.total_memory_gb' # Consider: # - Archiving old data from large collections # - Implementing pagination # - Adding query result limits # - Clearing unused indexes ``` ### Slow Query Performance[​](#slow-query-performance "Direct link to Slow Query Performance") ``` # Check if indexes exist for frequently queried fields curl -s https://{EKODB_API_URL}/api/indexes/query/{collection} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Use explain to analyze query performance curl -X POST https://{EKODB_API_URL}/api/query/{collection}/explain \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{ "filter": { "type": "Condition", "content": { "field": "your_field", "operator": "Eq", "value": "your_value" } } }' # Create indexes for fields without them curl -X POST https://{EKODB_API_URL}/api/indexes/query/{collection} \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -d '{"field": "frequently_queried_field", "index_type": "btree"}' # Check batch settings and adjust if needed curl -s https://{EKODB_API_URL}/api/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '.batch_settings' ``` ### WAL Disk Space Issues[​](#wal-disk-space-issues "Direct link to WAL Disk Space Issues") ``` # Check WAL health and size curl -s https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq '{is_healthy, file_size_mb, max_size_mb}' # Rotate immediately if unhealthy is_healthy=$(curl -s https://{EKODB_API_URL}/api/wal/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ | jq -r '.is_healthy') if [ "$is_healthy" = "false" ]; then echo "WAL approaching limit, rotating..." curl -X POST https://{EKODB_API_URL}/api/wal/rotate \ -H "Authorization: Bearer {ADMIN_TOKEN}" fi # Get WAL entries for backup before cleanup end_time=$(date +%s) start_time=$((end_time - 86400)) # Last 24 hours curl -s "https://{EKODB_API_URL}/api/wal/entries?from_timestamp=$start_time&to_timestamp=$end_time" \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ > wal_backup_$(date +%Y%m%d).json # After backing up, rotation will free space ``` ## Ripple Configuration[​](#ripple-configuration "Direct link to Ripple Configuration") For multi-node deployments, configure data propagation between instances: ``` # Configure ripples on a node (add each peer separately) curl -X POST https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "peer1", "url": "https://peer1.production.google.ekodb.net:8080", "api_key": "peer1-admin-key", "mode": "Operations", "enabled": true }' curl -X POST https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "peer2", "url": "https://peer2.production.google.ekodb.net:8080", "api_key": "peer2-admin-key", "mode": "Operations", "enabled": true }' # List configured ripples curl -X GET https://{EKODB_API_URL}/api/ripples/config \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Check ripple health curl -X GET https://{EKODB_API_URL}/api/ripples/health \ -H "Authorization: Bearer {ADMIN_TOKEN}" ``` **Replication Roles:** * `primary` - Propagate writes to peers (primary nodes) * `replica` - Receive updates from peers (read replicas) * `peer` - Full bidirectional sync (multi-master) * `standalone` - No replication (isolated nodes) Ripple Use Cases See [Ripples - Data Propagation](/ripples.md) for comprehensive guides on multi-region deployments, read scaling, high availability architectures, and data pipeline patterns. ## Manifest Recovery[​](#manifest-recovery "Direct link to Manifest Recovery") ### Rebuild Collection Manifests[​](#rebuild-collection-manifests "Direct link to Rebuild Collection Manifests") Rebuild collection manifests from individual record files. Useful for recovering from schema evolution issues, stale manifest data after upgrades, or collection loading problems after crashes. ``` POST https://{EKODB_API_URL}/api/admin/rebuild-manifests Content-Type: application/json Authorization: Bearer {ADMIN_TOKEN} # Rebuild specific collections { "collections": ["users", "orders"] } # Rebuild ALL collections (omit collections field) {} # Response { "rebuilt": ["users", "orders"], "message": "Successfully rebuilt 2 collection(s)" } ``` Backup First This operation automatically backs up existing manifests (`.backup` extension) before rebuilding, but it's recommended to have a full backup before running this in production. **Use Cases:** * **Schema Evolution Issues** - Records missing after adding new required fields * **Stale Manifest Data** - Manifest doesn't reflect actual records on disk * **Post-Crash Recovery** - Collection loading problems after unexpected shutdown * **Version Upgrades** - Manifest format changes between versions **What It Does:** 1. Backs up existing manifest files 2. Scans individual record files 3. Rebuilds manifest from actual record data 4. Updates in-memory collections ## Public Endpoint Rate Limiting[​](#public-endpoint-rate-limiting "Direct link to Public Endpoint Rate Limiting") ekoDB includes configurable IP-based rate limiting for public endpoints that don't require authentication. This protects against DDoS and brute force attacks. ### Protected Endpoints[​](#protected-endpoints "Direct link to Protected Endpoints") * `/api/health` - Health check * `/api/auth/register` - API key registration * `/api/auth/token` - Token generation * `/api/auth/{api_key}/admin_key` - Admin key retrieval ### Configuration[​](#configuration "Direct link to Configuration") The public-endpoint limit is platform-owned and set via environment variables — it is **not** a `/api/config` field, so a customer cannot raise their own abuse-protection limit at runtime. On the managed platform the value is set per plan; for self-hosting, set it in your `.env`: | Environment Variable | Default | Description | | ------------------------------ | ------- | ------------------------------------------------------ | | `PUBLIC_RATE_LIMIT_PER_MINUTE` | `60` | Max requests per IP per minute | | `PUBLIC_RATE_LIMIT_DISABLED` | `false` | Set to `true` to disable public-endpoint rate limiting | On the managed platform this limit scales per plan; see [Security → Rate Limiting](/security.md#rate-limiting). ### Rate Limit Response[​](#rate-limit-response "Direct link to Rate Limit Response") When rate limited, the server returns: ``` HTTP/1.1 429 Too Many Requests Retry-After: 45 Content-Type: application/json { "error": "Rate limit exceeded", "retry_after_secs": 45 } ``` ### IP Detection[​](#ip-detection "Direct link to IP Detection") The rate limiter detects client IP from (in order): 1. `X-Forwarded-For` header (first IP in list) 2. `X-Real-IP` header 3. Direct connection IP Load Balancer Configuration If using a load balancer, ensure it forwards the original client IP via `X-Forwarded-For` or `X-Real-IP` headers for accurate rate limiting. ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Configuration](/configuration.md) - All database configuration options * [Ripples - Data Propagation](/ripples.md) - Multi-node data propagation and use cases * [Indexes](/indexes.md) - Create indexes for performance * [Transactions](/transactions.md) - Understand transaction impact on WAL * [Collections & Schemas](/collections-and-schemas.md) - Collection management * [Authentication](/authentication.md) - Admin key management --- # Transactions Manage atomic, durable, isolated transactions using ekoDB's REST API with support for savepoints and full rollback capabilities. Enforced, configurable isolation ekoDB transactions are **buffered and isolated**. Statements you issue with a `transaction_id` are staged server-side and applied atomically only at commit, so they are invisible to other transactions until then (no dirty reads) and visible to your own reads that pass the same `transaction_id` (read-your-writes). Isolation is **enforced** by optimistic concurrency over per-record version stamps (MVCC): at commit, ekoDB validates that the records this transaction read or wrote have not been changed under it by another committed transaction — how much it validates depends on the [isolation level](#isolation-levels). If they have, the commit is rejected with a retryable conflict (HTTP `409`) and **nothing** is applied. This applies to REST/client transactions and to transactional stored functions alike. **Atomicity** (all-or-nothing; a rollback or failure discards the staged writes) and **durability** (WAL) are guaranteed as before. Architecture & Concepts For detailed information about transactions architecture, isolation levels, and best practices, see the [Transactions Architecture Reference](/reference/transaction-architecture.md). REST API Transaction Considerations When using the REST API directly (without client libraries), be aware of these important limitations: 1. **Manual Transaction ID Management** - You must track and include the `transaction_id` with every operation within a transaction 2. **Connection Stateless** - REST is stateless; transaction state is maintained server-side with timeout expiration 3. **No Automatic Rollback** - If your client crashes, the transaction will remain pending until timeout (default: 5 minutes) 4. **Network Failures** - A failed commit response doesn't guarantee the commit failed; use idempotency keys for critical operations **Recommendation:** Use the official client libraries which handle transaction lifecycle automatically. ## Isolation Levels[​](#isolation-levels "Direct link to Isolation Levels") The isolation level selects **how much ekoDB validates at commit**. ekoDB uses *optimistic* concurrency: no locks are held across statements, and a conflict surfaces as a retryable `409` at commit rather than blocking. The level is resolved per transaction as the value you pass, else the server-wide `default_isolation_level` setting (`ReadUncommitted` out of the box; the official client libraries send `ReadCommitted` by default). It can be changed at runtime via `/api/config`. * **ReadUncommitted** — no conflict detection; concurrent commits are last-writer-wins. Cheapest, and the server default unless `default_isolation_level` is changed or a level is passed. * **ReadCommitted** — **lost-update prevention**: the commit is rejected if a record this transaction **wrote** was changed by another committed transaction since this transaction observed it. * **RepeatableRead** — adds **read-set validation**: the commit is rejected if a record this transaction **read** (not only one it wrote) changed before commit. Prevents non-repeatable reads / read skew. Phantoms are still possible. * **Serializable** — strongest: adds **collection-membership validation** on top — the commit is rejected if the membership of any collection this transaction **queried** changed before commit (a concurrently committed insert / delete / update that shifts a predicate's result set). Prevents phantoms, and with them write skew; the committed history is serializable. How the levels map to behavior ekoDB validates version stamps at **commit**; it does not take per-statement read snapshots. A level's guarantee is "a transaction that would exhibit the anomaly **aborts** at commit," not lock-based read isolation during execution — reads always observe live committed data. The ladder is strict: each level prevents a superset of the anomalies below it (lost updates → non-repeatable reads → phantoms). Phantom protection (`Serializable`) is gated, so it costs other writers nothing unless a `Serializable` transaction is in flight. Isolation is also coherent with **direct** (non-transactional) writes: every durable write bumps the same per-record version, so a concurrent transaction detects it. ## Ripple Propagation in Transactions[​](#ripple-propagation-in-transactions "Direct link to Ripple Propagation in Transactions") Transactional writes ripple at commit. Each write staged inside a transaction carries its own `bypass_ripple` intent (default `false` = propagate to peers; `true` = suppress), and propagation happens once the transaction commits. Staged writes are never rippled before commit, and a rolled-back or conflicted (`409`) transaction ripples nothing. A single transaction may mix per-operation choices. ## Begin Transaction[​](#begin-transaction "Direct link to Begin Transaction") Start a new transaction with optional isolation level and timeout. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let tx_id = client.begin_transaction(Some(TransactionOptions { isolation_level: IsolationLevel::Serializable, timeout_seconds: Some(600), })).await?; println!("{}", tx_id); // 'tx-abc-123' ``` ``` tx_id = await client.begin_transaction( isolation_level='Serializable' ) print(tx_id) # 'tx-abc-123' ``` ``` const txId = await client.beginTransaction({ isolation_level: "Serializable", }); console.log(txId); // 'tx-abc-123' ``` ``` const txId = await client.beginTransaction({ isolation_level: "Serializable", }); console.log(txId); // 'tx-abc-123' ``` ``` val txId = client.beginTransaction( isolationLevel = "Serializable" ) println(txId) // 'tx-abc-123' ``` ``` txId, err := client.BeginTransaction(ekodb.TransactionOptions{ IsolationLevel: "Serializable", }) fmt.Println(txId) // 'tx-abc-123' ``` ``` curl -X POST https://{EKODB_API_URL}/api/transactions \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "isolation_level": "Serializable", "timeout_seconds": 600 }' # Response { "transaction_id": "tx-abc-123", "isolation_level": "Serializable", "created_at": 1699814400 } ``` Working Examples **Prefer a client library?** See examples in Rust, Python, TypeScript, Go, or Kotlin **Want to use the REST API directly?** See examples in JavaScript, Python, Go, or Rust **Parameters:** * `isolation_level` (optional) - `ReadUncommitted`, `ReadCommitted`, `RepeatableRead`, `Serializable` * `timeout_seconds` (optional) - Default: 300 (5 minutes) ## Commit Transaction[​](#commit-transaction "Direct link to Commit Transaction") Commit all operations within the transaction. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.commit_transaction(&tx_id).await?; ``` ``` await client.commit_transaction(tx_id) ``` ``` await client.commitTransaction(txId); ``` ``` await client.commitTransaction(txId); ``` ``` client.commitTransaction(txId) ``` ``` err := client.CommitTransaction(txId) ``` ``` curl -X POST https://{EKODB_API_URL}/api/transactions/{transaction_id}/commit \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # Response { "status": "committed", "transaction_id": "tx-abc123", "operations_count": 5 } ``` ## Rollback Transaction[​](#rollback-transaction "Direct link to Rollback Transaction") Rollback all operations and abort the transaction. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.rollback_transaction(&tx_id).await?; ``` ``` await client.rollback_transaction(tx_id) ``` ``` await client.rollbackTransaction(txId); ``` ``` await client.rollbackTransaction(txId); ``` ``` client.rollbackTransaction(txId) ``` ``` err := client.RollbackTransaction(txId) ``` ``` curl -X POST https://{EKODB_API_URL}/api/transactions/{transaction_id}/rollback \ -H "Authorization: Bearer {YOUR_API_TOKEN}" # Response { "status": "rolled_back", "transaction_id": "tx-abc123", "operations_rolled_back": 5 } ``` ## Create Savepoint[​](#create-savepoint "Direct link to Create Savepoint") Create a savepoint for partial rollback within a transaction. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.create_savepoint(&tx_id, "checkpoint1").await?; ``` ``` await client.create_savepoint(tx_id, 'checkpoint1') ``` ``` await client.createSavepoint(txId, "checkpoint1"); ``` ``` await client.createSavepoint(txId, "checkpoint1"); ``` ``` client.createSavepoint(txId, "checkpoint1") ``` ``` err := client.CreateSavepoint(txId, "checkpoint1") ``` ``` curl -X POST https://{EKODB_API_URL}/api/transactions/{transaction_id}/savepoints \ -H "Authorization: Bearer {ADMIN_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "checkpoint1" }' # Response { "status": "created", "transaction_id": "tx-abc123", "savepoint_name": "checkpoint1" } ``` ## Rollback to Savepoint[​](#rollback-to-savepoint "Direct link to Rollback to Savepoint") Rollback to a specific savepoint, undoing operations after it. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.rollback_to_savepoint(&tx_id, "checkpoint1").await?; ``` ``` await client.rollback_to_savepoint(tx_id, 'checkpoint1') ``` ``` await client.rollbackToSavepoint(txId, "checkpoint1"); ``` ``` await client.rollbackToSavepoint(txId, "checkpoint1"); ``` ``` client.rollbackToSavepoint(txId, "checkpoint1") ``` ``` err := client.RollbackToSavepoint(txId, "checkpoint1") ``` ``` curl -X POST https://{EKODB_API_URL}/api/transactions/{transaction_id}/savepoints/{savepoint_name}/rollback \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "rolled_back", "transaction_id": "tx-abc123", "savepoint_name": "checkpoint1", "operations_rolled_back": 3 } ``` ## Release Savepoint[​](#release-savepoint "Direct link to Release Savepoint") Remove a savepoint that's no longer needed. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` client.release_savepoint(&tx_id, "checkpoint1").await?; ``` ``` await client.release_savepoint(tx_id, 'checkpoint1') ``` ``` await client.releaseSavepoint(txId, "checkpoint1"); ``` ``` await client.releaseSavepoint(txId, "checkpoint1"); ``` ``` client.releaseSavepoint(txId, "checkpoint1") ``` ``` err := client.ReleaseSavepoint(txId, "checkpoint1") ``` ``` curl -X DELETE https://{EKODB_API_URL}/api/transactions/{transaction_id}/savepoints/{savepoint_name} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "status": "released", "transaction_id": "tx-abc123", "savepoint_name": "checkpoint1" } ``` ## Get Transaction Status[​](#get-transaction-status "Direct link to Get Transaction Status") Check the current status of a transaction. * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` let status = client.get_transaction_status(&tx_id).await?; println!("{:?}", status); ``` ``` status = await client.get_transaction_status(tx_id) print(status['state']) print(status['operations_count']) ``` ``` const status = await client.getTransactionStatus(txId); console.log(status.state); console.log(status.operations_count); ``` ``` const status = await client.getTransactionStatus(txId); console.log(status.state); console.log(status.operations_count); ``` ``` val status = client.getTransactionStatus(txId) println(status) ``` ``` status, err := client.GetTransactionStatus(txId) fmt.Println(status) ``` ``` curl https://{EKODB_API_URL}/api/transactions/{transaction_id} \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "transaction_id": "tx-abc123", "state": "Active", "isolation_level": "Serializable", "operations_count": 5, "savepoints_count": 2, "created_at": 1699814400, "last_activity": 1699814450, "timeout_seconds": 600 } ``` ## List Active Transactions[​](#list-active-transactions "Direct link to List Active Transactions") List all currently active transactions. * Client Libraries (Recommended) * Direct API - 📘 TypeScript - 📦 JavaScript - 🐍 Python - 🦀 Rust - 🟣 Kotlin - 🔷 Go ``` const transactions = await client.listActiveTransactions(); console.log(transactions); /* [ { transaction_id: 'tx-abc-123', isolation_level: 'Serializable', created_at: 1699814400 }, { transaction_id: 'tx-def-456', isolation_level: 'ReadCommitted', created_at: 1699814500 } ] */ ``` ``` const transactions = await client.listActiveTransactions(); console.log(transactions); ``` ``` transactions = await client.list_active_transactions() print(transactions) # [{'transaction_id': 'tx-abc-123', ...}] ``` ``` let transactions = client.list_active_transactions().await?; println!("{:?}", transactions); ``` ``` val transactions = client.listActiveTransactions() println(transactions) ``` ``` transactions, err := client.ListActiveTransactions() fmt.Println(transactions) ``` ``` curl https://{EKODB_API_URL}/api/transactions \ -H "Authorization: Bearer {ADMIN_TOKEN}" # Response { "transactions": [ { "transaction_id": "tx-abc123", "state": "Active", "isolation_level": "Serializable", "operations_count": 5, "savepoints_count": 2, "created_at": 1699814400, "last_activity": 1699814450, "timeout_seconds": 600 }, { "transaction_id": "tx-def456", "state": "Active", "isolation_level": "ReadCommitted", "operations_count": 2, "savepoints_count": 0, "created_at": 1699814500, "last_activity": 1699814510, "timeout_seconds": 300 } ], "count": 2 } ``` ## Complete Example[​](#complete-example "Direct link to Complete Example") Here's a complete example of a multi-step transaction with savepoints: ``` # 1. Begin transaction curl -X POST https://{EKODB_API_URL}/api/transactions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"isolation_level": "Serializable", "timeout_seconds": 600}' # Response: {"transaction_id": "tx-001"} # 2. Create initial savepoint curl -X POST https://{EKODB_API_URL}/api/transactions/tx-001/savepoints \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"name": "start"}' # 3. Debit source account curl -X PUT https://{EKODB_API_URL}/api/update/accounts/ACC001 \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"balance": 900}' # 4. Create savepoint after debit curl -X POST https://{EKODB_API_URL}/api/transactions/tx-001/savepoints \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"name": "after_debit"}' # 5. Credit destination account curl -X PUT https://{EKODB_API_URL}/api/update/accounts/ACC002 \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {YOUR_API_TOKEN}" \ -d '{"balance": 1100}' # 6. If credit fails, rollback to after_debit # curl -X POST https://{EKODB_API_URL}/api/transactions/tx-001/savepoints/after_debit/rollback \ # -H "Authorization: Bearer {YOUR_API_TOKEN}" # Then retry credit operation... # 7. Commit transaction curl -X POST https://{EKODB_API_URL}/api/transactions/tx-001/commit \ -H "Authorization: Bearer {YOUR_API_TOKEN}" ``` ## Best Practices[​](#best-practices "Direct link to Best Practices") ### Choose an Isolation Level[​](#choose-an-isolation-level "Direct link to Choose an Isolation Level") Pick the level by how strictly the commit must validate (see [Isolation Levels](#isolation-levels)). Higher levels reject more conflicts at commit, so be ready to **retry** a transaction that fails with a `409`. | Use Case | Level | Why | | --------------------- | --------------- | ---------------------------------------------- | | Analytics/Reporting | ReadUncommitted | Performance over consistency; no validation | | General CRUD | ReadCommitted | Prevents lost updates; client default | | Reads must stay valid | RepeatableRead | Also prevents non-repeatable reads / read skew | | Banking/Critical | Serializable | Also prevents phantoms (and write skew) | ### Use Savepoints for Multi-Stage Operations[​](#use-savepoints-for-multi-stage-operations "Direct link to Use Savepoints for Multi-Stage Operations") Savepoints allow partial rollback without aborting the entire transaction: ``` # Create savepoint before risky operation POST /api/transactions/{id}/savepoints {"name": "before_risky_op"} # Perform risky operation PUT /api/update/... # If operation fails, rollback to savepoint POST /api/transactions/{id}/savepoints/before_risky_op/rollback # Retry or continue with alternative approach ``` ### Set Appropriate Timeouts[​](#set-appropriate-timeouts "Direct link to Set Appropriate Timeouts") ``` // Short transaction (1 minute) {"timeout_seconds": 60} // Long-running workflow (30 minutes) {"timeout_seconds": 1800} // Critical operation (5 minutes, default) {"timeout_seconds": 300} ``` ### Handle Errors Gracefully[​](#handle-errors-gracefully "Direct link to Handle Errors Gracefully") Always implement proper error handling with rollback: ``` # Begin transaction curl -X POST .../api/transactions ... # -> {"transaction_id": "tx-001"} # Try operations if operation_fails; then # Rollback on error curl -X POST .../api/transactions/tx-001/rollback ... else # Commit on success curl -X POST .../api/transactions/tx-001/commit ... fi ``` ### Monitor Active Transactions[​](#monitor-active-transactions "Direct link to Monitor Active Transactions") Track and debug active transactions: ``` # List all active transactions GET /api/transactions # Check specific transaction status GET /api/transactions/{id} # Useful for: # - Debugging stuck transactions # - Monitoring long-running operations # - Identifying transaction bottlenecks ``` ## Use Cases[​](#use-cases "Direct link to Use Cases") ### Financial Transfers[​](#financial-transfers "Direct link to Financial Transfers") Multi-account updates with atomicity guarantees: ``` # Begin transaction POST /api/transactions {"isolation_level": "Serializable"} # Debit source PUT /api/update/accounts/{from_id} {"$decrement": {"balance": 100}} # Credit destination PUT /api/update/accounts/{to_id} {"$increment": {"balance": 100}} # Commit both or rollback both POST /api/transactions/{id}/commit ``` ### E-Commerce Orders[​](#e-commerce-orders "Direct link to E-Commerce Orders") Order creation with inventory reservation: ``` # Begin transaction POST /api/transactions {"isolation_level": "RepeatableRead"} # Create savepoint before inventory check POST /api/transactions/{id}/savepoints {"name": "before_inventory"} # Reserve inventory PUT /api/update/inventory/{product_id} {"$decrement": {"stock": 1}} # If insufficient stock, rollback to savepoint # Otherwise create order POST /api/insert/orders {...} # Commit transaction POST /api/transactions/{id}/commit ``` ### Batch Processing[​](#batch-processing "Direct link to Batch Processing") Process multiple records with error recovery: ``` # Begin transaction POST /api/transactions # Process records in batches for batch in batches: # Create savepoint before batch POST /api/transactions/{id}/savepoints {"name": "batch_{i}"} # Process batch POST /api/batch/insert/... # If batch fails, rollback and continue if error: POST /api/transactions/{id}/savepoints/batch_{i}/rollback continue # Commit all successful batches POST /api/transactions/{id}/commit ``` ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Transactions Architecture Reference](/reference/transaction-architecture.md) - Detailed architecture, isolation levels, and best practices * [Functions Architecture](/reference/functions-architecture.md) - Function-based transactions (REST API) * [Functions via Client Libraries](/client-libraries/advanced-operations.md#functions) - Function-based transactions with client SDKs * [Basic Operations](/basic-operations.md) - Standard CRUD operations * [Batch Operations](/batch-operations.md) - Bulk operations within transactions * [White Paper](/white-paper.md) - ACID compliance and architecture ## Example Code[​](#example-code "Direct link to Example Code") ### Direct HTTP/REST API Examples[​](#direct-httprest-api-examples "Direct link to Direct HTTP/REST API Examples") Raw HTTP examples demonstrating the REST API directly: * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/transactions.js)**: `transactions.js` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/transactions.py)**: `transactions.py` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/transactions.go)**: `transactions.go` * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/transactions.rs)**: `transactions.rs` ### Client Library Examples[​](#client-library-examples "Direct link to Client Library Examples") Production-ready examples using official client libraries: * **[Rust](https://github.com/ekoDB/ekodb-client/blob/main/examples/rust/examples/client_transactions.rs)**: `client_transactions.rs` * **[Python](https://github.com/ekoDB/ekodb-client/blob/main/examples/python/client_transactions.py)**: `client_transactions.py` * **[TypeScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/typescript/client_transactions.ts)**: `client_transactions.ts` * **[Go](https://github.com/ekoDB/ekodb-client/blob/main/examples/go/client_transactions.go)**: `client_transactions.go` * **[Kotlin](https://github.com/ekoDB/ekodb-client/blob/main/examples/kotlin/examples/ClientTransactions.kt)**: `ClientTransactions.kt` * **[JavaScript](https://github.com/ekoDB/ekodb-client/blob/main/examples/javascript/client_transactions.js)**: `client_transactions.js` --- # Using ekoDB This section covers ekoDB's core concepts: have an AI Agent build your backend, or work with the data directly through the client libraries and REST API. Start here to understand how to structure and interact with your data. New to ekoDB? If you're not sure where to start, check out the [Choose Your Path](/choose-your-path.md) guide or jump straight to the [Quick Start](/quick-start.md) tutorial. *** ## 📚 What's in This Section[​](#-whats-in-this-section "Direct link to 📚 What's in This Section") *** ### Build with an AI Agent (fastest)[​](#build-with-an-ai-agent-fastest "Direct link to Build with an AI Agent (fastest)") Describe your backend and an AI Agent builds the collections and server-side functions for you, then operates and scales it. You call what it built from your app with the SDK. * **[AI Agents](/ai-agents.md)** - What agents are and how they work * **[Build Your First AI Agent](/build-your-first-agent.md)** - From signup to a working agent in 10 minutes **Best for:** Shipping a backend fast and staying focused on your frontend ### Client Libraries[​](#client-libraries "Direct link to Client Libraries") Type-safe SDKs for your favorite languages with built-in authentication, retry logic, and WebSocket support. * **[Overview](/client-libraries/.md)** - Introduction to client libraries * **[Installation](/client-libraries/installation.md)** - Install for TypeScript, Python, Rust, Go, Kotlin * **[Basic Operations](/client-libraries/basic-operations.md)** - CRUD operations with type safety * **[Advanced Operations](/client-libraries/advanced-operations.md)** - Vector search, functions, real-time subscriptions, TTL **Best for:** Application development with type safety and auto-complete ### Direct API Usage[​](#direct-api-usage "Direct link to Direct API Usage") Complete REST API reference for HTTP-based integration. * **[Basic Operations](/basic-operations.md)** - Core CRUD endpoints * **[Batch Operations](/batch-operations.md)** - Bulk operations for high performance * **[Collections & Schemas](/collections-and-schemas.md)** - Structure your data with collections and schemas * **[Indexes](/indexes.md)** - Hash, B-tree, text search, and vector search (HNSW) * **[Key-Value Store](/kv-store.md)** - Fast in-memory caching with TTL * **[Transactions](/transactions.md)** - Atomic, durable multi-document transactions * **[System Administration](/system-administration.md)** - Health checks, metrics, and troubleshooting **Best for:** Custom integrations, serverless functions, or languages without official clients *** ## 🚀 Quick Links[​](#-quick-links "Direct link to 🚀 Quick Links") **Want the backend built for you?** → [Build Your First AI Agent](/build-your-first-agent.md) **Building an app?** → [Client Libraries](/client-libraries/.md) **Using REST API?** → [Basic Operations](/basic-operations.md) **Need performance?** → [Batch Operations](/batch-operations.md) **Building with AI?** → [Advanced Operations](/client-libraries/advanced-operations.md) **Understanding data structure?** → [Data Types](/data-types.md) + [Collections & Schemas](/collections-and-schemas.md) *** ## 📞 Need Help?[​](#-need-help "Direct link to 📞 Need Help?") * **Quick Start**: [5-Minute Tutorial](/quick-start.md) * **Find Your Path**: [Choose Your Path Guide](/choose-your-path.md) * **Email Support**: --- # Vector Search & Embeddings Build intelligent search and recommendation systems with ekoDB's integrated vector search capabilities. Production-Ready Vector search is built directly into ekoDB - no separate deployment needed. Performance competitive with specialized vector databases. ## Quick Start[​](#quick-start "Direct link to Quick Start") * Client Libraries (Recommended) * Direct API - 🦀 Rust - 🐍 Python - 📘 TypeScript - 📦 JavaScript - 🟣 Kotlin - 🔷 Go ``` use ekodb_client::{Client, Record, SearchQuery}; let client = Client::builder() .base_url("https://my-first-db.development.google.ekodb.net") .api_key("your-api-key") .build()?; // 1. Store vectors with your data let mut record = Record::new(); record.insert("name", "Ergonomic Chair"); record.insert("description", "Comfortable office chair..."); record.insert("embedding", vec![0.12, 0.34, 0.56]); // 1536-dim vector client.insert("products", record, None).await?; // 2. Search by similarity let query_embedding: Vec = vec![0.12, 0.34, 0.56]; // from your embedding model let query = SearchQuery { query: String::new(), vector: Some(query_embedding), vector_field: Some("embedding".to_string()), vector_metric: Some("cosine".to_string()), vector_k: Some(10), ..Default::default() }; let results = client.search("products", query).await?; ``` ``` from ekodb_client import Client client = Client.new("https://my-first-db.development.google.ekodb.net", "your-api-key") # 1. Store vectors with your data await client.insert('products', { 'name': 'Ergonomic Chair', 'description': 'Comfortable office chair...', 'embedding': [0.12, 0.34, 0.56] # 1536-dim vector from OpenAI }) # 2. Search by similarity query_embedding = [0.12, 0.34, 0.56] # from your embedding model results = await client.search( 'products', query='', # empty query for vector-only search vector=query_embedding, vector_field='embedding', vector_metric='cosine', vector_k=10 ) ``` ``` import { EkoDBClient, SearchQueryBuilder } from '@ekodb/ekodb-client'; const client = new EkoDBClient({ baseURL: 'https://my-first-db.development.google.ekodb.net', apiKey: 'your-api-key' }); await client.init(); // 1. Store vectors with your data await client.insert('products', { name: 'Ergonomic Chair', description: 'Comfortable office chair...', embedding: [0.12, 0.34, 0.56 /* ...1536 dims */], // from OpenAI }); // 2. Search by similarity const queryEmbedding = [0.12, 0.34, 0.56 /* ...1536 dims */]; // from your embedding model const query = new SearchQueryBuilder('') // empty text query for vector-only search .vector(queryEmbedding) .vectorField('embedding') .vectorMetric('cosine') .vectorK(10) .build(); const results = await client.search('products', query); ``` ``` const { EkoDBClient, SearchQueryBuilder } = require('@ekodb/ekodb-client'); const client = new EkoDBClient({ baseURL: 'https://my-first-db.development.google.ekodb.net', apiKey: 'your-api-key' }); await client.init(); // 1. Store vectors with your data await client.insert('products', { name: 'Ergonomic Chair', description: 'Comfortable office chair...', embedding: [0.12, 0.34, 0.56 /* ...1536 dims */], // from OpenAI }); // 2. Search by similarity const queryEmbedding = [0.12, 0.34, 0.56 /* ...1536 dims */]; // from your embedding model const query = new SearchQueryBuilder('') // empty text query for vector-only search .vector(queryEmbedding) .vectorField('embedding') .vectorMetric('cosine') .vectorK(10) .build(); const results = await client.search('products', query); ``` ``` import io.ekodb.client.EkoDBClient import kotlinx.serialization.json.* val client = EkoDBClient.builder() .baseUrl("https://my-first-db.development.google.ekodb.net") .apiKey("your-api-key") .build() // 1. Store vectors with your data client.insert("products", mapOf( "name" to "Ergonomic Chair", "description" to "Comfortable office chair...", "embedding" to listOf(0.12, 0.34, 0.56) // 1536-dim vector )) // 2. Search by similarity val queryEmbedding = listOf(0.12, 0.34, 0.56) // from your embedding model val searchQuery = buildJsonObject { put("query", "") // empty text query for vector-only search put("vector", buildJsonArray { queryEmbedding.forEach { add(it) } }) put("vector_field", "embedding") put("vector_metric", "cosine") put("vector_k", 10) } val results = client.search("products", searchQuery) ``` ``` import "github.com/ekoDB/ekodb-client-go" client := ekodb.NewClient( "https://my-first-db.development.google.ekodb.net", "your-api-key", ) // 1. Store vectors with your data client.Insert("products", map[string]interface{}{ "name": "Ergonomic Chair", "description": "Comfortable office chair...", "embedding": []float64{0.12, 0.34, 0.56}, // 1536-dim vector }) // 2. Search by similarity queryEmbedding := []float64{0.12, 0.34, 0.56} // from your embedding model vectorField := "embedding" vectorMetric := "cosine" vectorK := 10 results, err := client.Search("products", ekodb.SearchQuery{ Query: "", Vector: queryEmbedding, VectorField: &vectorField, VectorMetric: &vectorMetric, VectorK: &vectorK, }) ``` ``` # 1. Store vectors with your data curl -X POST https://{EKODB_API_URL}/api/insert/products \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": {"type": "String", "value": "Ergonomic Chair"}, "description": {"type": "String", "value": "Comfortable office chair..."}, "embedding": {"type": "Vector", "value": [0.12, 0.34, 0.56, ...]} }' # 2. Search by similarity curl -X POST https://{EKODB_API_URL}/api/search/{collection} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "query": "", "vector": [0.12, 0.34, 0.56, ...], "vector_field": "embedding", "vector_metric": "cosine", "vector_k": 10 }' ``` ## Performance[​](#performance "Direct link to Performance") | Dataset Size | Average Latency | Throughput | | ----------------- | ---------------------- | -------------------- | | **1K vectors** | 51ms | 379 RPS | | **10K vectors** | 322ms | 56 RPS | | **Hybrid search** | Text + vector combined | Configurable weights | **Competitive Performance:** * ✅ **6x faster** than PostgreSQL pgvector * ✅ **Competitive** with Milvus, Elasticsearch * ✅ **Integrated** - no separate deployment ## Vector Types[​](#vector-types "Direct link to Vector Types") ekoDB supports vector fields for storing embeddings from AI models: ``` // Standard vector field const record = { title: 'Database Performance Tips', content: 'This article discusses...', embedding: { type: 'Vector', value: [0.12, 0.34, 0.56, ...], // Array of numbers metadata: { model: 'text-embedding-ada-002', dimensions: 1536 } } }; ``` ## Schema Definition[​](#schema-definition "Direct link to Schema Definition") Define vector fields in your schema. ekoDB infers the vector dimension from the first inserted vector and rejects later vectors of a different length, so you do not declare the dimension yourself: ``` const schema = { fields: { title: { field_type: 'String', required: true }, content: { field_type: 'String', required: true }, embedding: { field_type: 'Vector', required: true, // Optional: Vector index configuration index: { type: 'vector', algorithm: 'flat', // exact search metric: 'cosine', // similarity metric } } } }; await client.createCollection('articles', schema); ``` ## Distance Metrics[​](#distance-metrics "Direct link to Distance Metrics") Choose the right metric for your use case: ### Cosine Similarity (Recommended)[​](#cosine-similarity-recommended "Direct link to Cosine Similarity (Recommended)") Measures the angle between vectors. Range: \[-1, 1] ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(10) .build(); const results = await client.search('articles', query); ``` **Best for:** * ✅ Text embeddings and semantic search * ✅ When vector magnitude should be ignored * ✅ Most AI/ML embeddings (OpenAI, Cohere, etc.) ### Euclidean Distance[​](#euclidean-distance "Direct link to Euclidean Distance") Measures straight-line (L2) distance. Lower = more similar. ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('euclidean') .vectorK(10) .build(); const results = await client.search('locations', query); ``` **Best for:** * ✅ Spatial data and coordinates * ✅ When both magnitude and direction matter * ✅ Physical measurements ### Dot Product[​](#dot-product "Direct link to Dot Product") Calculates inner product. Higher = more similar. ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('dotproduct') .vectorK(10) .build(); const results = await client.search('recommendations', query); ``` **Best for:** * ✅ When vectors are pre-normalized * ✅ Certain recommendation systems * ✅ When magnitude contains meaningful information ## Search Methods[​](#search-methods "Direct link to Search Methods") ekoDB's vector search finds the true nearest neighbors for your query, with performance optimized by vector indexes when defined in your schema. ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(10) .vectorThreshold(0.7) // minimum similarity score .build(); const results = await client.search('products', query); ``` **Performance:** * Low-latency similarity search across collections * Accurate results guaranteed * Further optimized with vector indexes (when defined in schema) ## Hybrid Search[​](#hybrid-search "Direct link to Hybrid Search") Combine text search with vector similarity for powerful hybrid queries: ``` // Hybrid: Text + Vector search const query = new SearchQueryBuilder('database performance') .fields(['title', 'content']) .vector(queryEmbedding) .vectorField('embedding') .textWeight(0.3) // 30% text relevance .vectorWeight(0.7) // 70% semantic similarity .limit(10) .build(); const results = await client.search('articles', query); ``` **Use cases:** * Semantic search with keyword filtering * Recommendations with category constraints * RAG systems with both keyword and semantic matching ## Filtering by Metadata[​](#filtering-by-metadata "Direct link to Filtering by Metadata") Restrict vector and hybrid search to a subset of records with a metadata pre-filter. Only records matching the filter are considered as candidates before similarity ranking, so a query like "find the nearest in-stock electronics over $100" never scores the rest of the collection. ``` import { SearchQueryBuilder, QueryBuilder } from '@ekodb/ekodb-client'; const filter = new QueryBuilder() .eq('category', 'electronics') .gte('price', 100) .build().filter; const query = new SearchQueryBuilder('') .vector(queryVector) .vectorK(10) .filters(filter) // only electronics priced >= 100 are ranked .build(); const results = await client.search('products', query); ``` The filter uses the same [Query Expression](/query-expressions.md) syntax as `find`, including `Logical` `And` / `Or` / `Not` combinations. The pre-filter is uniform across every search mode: full-text search, brute-force vector search, the indexed vector paths, and hybrid search. It is always applied to the candidate set **before** ranking, so results are never silently truncated by an index window. In hybrid search it governs the entire result, so a record that matches the text query but fails the filter is excluded, not surfaced on its text score alone. > **Indexed vector search with a filter is exact.** A `Flat` index ranks only matching records directly. An `HNSW` index runs a fast approximate filtered traversal, but if a selective filter starves that traversal of candidates, ekoDB automatically falls back to an exact scan — so you always get the true matches, never a silently truncated set. The only cost of a highly selective filter is a little extra latency on that fallback. The same `.filters(...)` works on a pure text search too: ``` // Full-text search restricted to one category const query = new SearchQueryBuilder("introduction") .fields(["title", "content"]) .filters(new QueryBuilder().eq("category", "ml").build().filter) .build(); const results = await client.search("documents", query); ``` ``` // Hybrid search constrained to a single tenant const query = new SearchQueryBuilder('annual report') .fields(['title', 'body']) .vector(queryVector) .textWeight(0.3) .vectorWeight(0.7) .filters(new QueryBuilder().eq('tenant_id', tenantId).build().filter) .build(); const results = await client.search('documents', query); ``` ## Real-World Examples[​](#real-world-examples "Direct link to Real-World Examples") ### Semantic Search[​](#semantic-search "Direct link to Semantic Search") * Client Libraries (Recommended) * Direct API ``` // Generate embedding from OpenAI const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); async function semanticSearch(query: string) { // 1. Get query embedding const response = await openai.embeddings.create({ model: 'text-embedding-ada-002', input: query, }); const queryVector = response.data[0].embedding; // 2. Search by similarity const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(5) .build(); const results = await client.search('articles', query); return results; } // Search: "How to optimize database queries" const articles = await semanticSearch('database optimization tips'); ``` ``` # 1. Get embedding from OpenAI (or your embedding service) QUERY_VECTOR=$(curl https://api.openai.com/v1/embeddings \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "text-embedding-ada-002", "input": "database optimization tips"}' \ | jq '.data[0].embedding') # 2. Search ekoDB curl -X POST https://{EKODB_API_URL}/api/search/articles \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d "{ \"query\": \"\", \"vector\": $QUERY_VECTOR, \"vector_metric\": \"cosine\", \"vector_k\": 5 }" ``` ### Product Recommendations[​](#product-recommendations "Direct link to Product Recommendations") ``` async function getSimilarProducts(productId: string) { // 1. Get product's embedding const product = await client.findById('products', productId); const embedding = product.embedding; // 2. Find similar products const query = new SearchQueryBuilder('') .vector(embedding) .vectorMetric('cosine') .vectorK(10) .build(); const similar = await client.search('products', query); // Filter out the original product return similar.filter(p => p.id !== productId); } ``` ### Image Similarity[​](#image-similarity "Direct link to Image Similarity") ``` // Store image embeddings from CLIP or similar model async function findSimilarImages(imageEmbedding: number[], category?: string) { const query = new SearchQueryBuilder('') .vector(imageEmbedding) .vectorMetric('cosine') .vectorK(20) .vectorThreshold(0.7) .build(); const results = await client.search('images', query); return results.map(r => ({ url: r.url, similarity: r.score, metadata: r.metadata })); } ``` ## Deletion & Index Maintenance[​](#deletion--index-maintenance "Direct link to Deletion & Index Maintenance") ### Automatic Deletion from Search[​](#automatic-deletion-from-search "Direct link to Automatic Deletion from Search") When you delete a record that contains a vector field, ekoDB immediately removes it from vector search results. There's no need to manually update the index — deleted records won't appear in searches. ``` // Insert a record with a vector await client.insert('products', { name: 'Discontinued Widget', embedding: [0.12, 0.34, 0.56, ...] }); // Delete it — immediately excluded from vector search results await client.delete('products', recordId); ``` ### Reindexing[​](#reindexing "Direct link to Reindexing") Over time, frequent deletions can degrade search performance, because a deleted vector is marked rather than removed from the graph. After heavy delete churn, reindex to rebuild the search graph and restore optimal performance. Reindexing is manual (call the reindex endpoint or your client's reindex method); ekoDB does not auto-trigger it. ``` # Reindex a collection's vector index curl -X POST https://{EKODB_API_URL}/api/indexes/search/{collection}/reindex \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{}' # Optionally specify which vector field to reindex curl -X POST https://{EKODB_API_URL}/api/indexes/search/{collection}/reindex \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{"field": "embedding"}' ``` **Response:** ``` { "status": "ok", "collection": "products", "field": "embedding", "vectors_reindexed": 4850, "duration_ms": 127.5 } ``` When to Reindex Reindexing is only needed after heavy deletion workloads. For most applications with occasional deletes, the index stays performant without manual intervention. ## Search Tuning[​](#search-tuning "Direct link to Search Tuning") ### ef\_search (Beam Width)[​](#ef_search-beam-width "Direct link to ef_search (Beam Width)") The `ef_search` parameter controls the search beam width — higher values explore more of the graph, improving accuracy at the cost of latency. ekoDB resolves `ef_search` with a 3-tier fallback: 1. **Per-query override** — pass `ef_search` in the search request 2. **Collection-level config** — set in the vector index configuration 3. **Heuristic default** — `max(k * 2, 64)` Both `vector_k` and `ef_search` are clamped to server caps (`max_vector_k`, default 1000; `max_ef_search`, default 4000) so a single query cannot exhaust memory or CPU. Raise them via `PUT /api/config`. See [Configuration — Search](/configuration.md#search). ``` // Per-query override for a high-precision search. // The builder covers the common parameters; ef_search is set via the Direct API (below). const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(10) .build(); const results = await client.search('articles', query); ``` ``` # Direct API with ef_search override curl -X POST https://{EKODB_API_URL}/api/search/{collection} \ -H "Authorization: Bearer {TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "query": "", "vector": [0.12, 0.34, 0.56, ...], "vector_metric": "cosine", "vector_k": 10, "ef_search": 200 }' ``` | ef\_search | Accuracy | Latency | Use Case | | ---------- | --------- | ------- | ------------------------------- | | 32-64 | Good | Low | Real-time recommendations | | 64-128 | High | Medium | Semantic search (default range) | | 200+ | Very High | Higher | Precision-critical applications | ## Performance Optimization[​](#performance-optimization "Direct link to Performance Optimization") ### 1. Use Vector Indexes[​](#1-use-vector-indexes "Direct link to 1. Use Vector Indexes") Define vector indexes in your schema for automatic indexing: ``` // Define index in schema for automatic indexing const schema = { fields: { embedding: { field_type: 'Vector', index: { type: 'vector', algorithm: 'flat', // exact search metric: 'cosine', } } } }; ``` ### 2. Set Similarity Threshold[​](#2-set-similarity-threshold "Direct link to 2. Set Similarity Threshold") Filter out low-relevance results: ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(10) .vectorThreshold(0.75) // only return results with > 75% similarity .build(); const results = await client.search('articles', query); ``` ### 3. Batch Insert Vectors[​](#3-batch-insert-vectors "Direct link to 3. Batch Insert Vectors") ``` // More efficient than individual inserts await client.batchInsert('articles', articlesWithEmbeddings); ``` ### 4. Use Field Projection[​](#4-use-field-projection "Direct link to 4. Use Field Projection") Return only needed fields to reduce data transfer: ``` const query = new SearchQueryBuilder('') .vector(queryVector) .vectorMetric('cosine') .vectorK(10) .selectFields(['title', 'price', 'image_url']) // only these fields .build(); const results = await client.search('products', query); ``` ## Embedding Models[​](#embedding-models "Direct link to Embedding Models") ekoDB works with any embedding model. Popular choices: ### OpenAI[​](#openai "Direct link to OpenAI") ``` import OpenAI from 'openai'; const openai = new OpenAI(); const response = await openai.embeddings.create({ model: 'text-embedding-ada-002', // 1536 dimensions input: 'Your text here', }); const embedding = response.data[0].embedding; ``` ### Cohere[​](#cohere "Direct link to Cohere") ``` import { CohereClient } from 'cohere-ai'; const cohere = new CohereClient({ token: process.env.COHERE_API_KEY }); const response = await cohere.embed({ texts: ['Your text here'], model: 'embed-english-v3.0', // 1024 dimensions }); const embedding = response.embeddings[0]; ``` ### Local Models (Transformers.js)[​](#local-models-transformersjs "Direct link to Local Models (Transformers.js)") ``` import { pipeline } from '@xenova/transformers'; const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2'); const output = await extractor('Your text here', { pooling: 'mean', normalize: true }); const embedding = Array.from(output.data); // 384 dimensions ``` ## API Reference[​](#api-reference "Direct link to API Reference") Vector, text, and hybrid search all go through `client.search(collection, query)`, where `query` is built with `SearchQueryBuilder`. Which fields you set decides the search type. ### Vector search[​](#vector-search "Direct link to Vector search") ``` import { SearchQueryBuilder } from '@ekodb/ekodb-client'; const query = new SearchQueryBuilder('') // text query ('' for vector-only) .vector(queryVector) // query vector (presence triggers vector search) .vectorField('embedding') // field holding vectors (default: 'embedding') .vectorMetric('cosine') // 'cosine' | 'euclidean' | 'dotproduct' .vectorK(10) // number of nearest neighbors .vectorThreshold(0.7) // minimum similarity score (0.0-1.0) .filters(metadataFilter) // optional metadata pre-filter (QueryExpression) .selectFields(['title', 'price']) // field projection (or .excludeFields([...])) .build(); const results = await client.search('products', query); ``` ### Hybrid search[​](#hybrid-search-1 "Direct link to Hybrid search") Set a text query plus a vector and weight them: ``` const query = new SearchQueryBuilder('database performance') .fields(['title', 'content']) .vector(queryVector) .textWeight(0.3) .vectorWeight(0.7) .limit(10) .build(); const results = await client.search('articles', query); ``` `ef_search` and other less-common parameters can be sent directly in the body of `POST /api/search/{collection}` (see the Direct API examples above). ## Best Practices[​](#best-practices "Direct link to Best Practices") 1. **Match Dimensions**: Keep every vector in a field the same length as your embedding model's output — ekoDB locks the dimension to the first inserted vector and rejects mismatched lengths 2. **Use Schemas**: Declare the field as `Vector` and attach a vector index for automatic similarity indexing 3. **Choose Right Metric**: Cosine for most AI embeddings (OpenAI, Cohere, etc.) 4. **Batch Operations**: Use `batchInsert` for multiple vectors 5. **Set Thresholds**: Filter low-relevance results with `threshold` parameter 6. **Tune `ef_search`**: Raise it for precision-critical queries, lower it for latency-sensitive ones 7. **Field Projection**: Only return fields you need 8. **Monitor Performance**: Track query latency and optimize with vector indexes as needed ## Related Documentation[​](#related-documentation "Direct link to Related Documentation") * [Indexes](/indexes.md) - Create indexes for optimal performance * [Query Expressions](/query-expressions.md) - Filter syntax * [Client Libraries](/client-libraries/advanced-operations.md) - Full API examples * [System Administration](/system-administration.md) - Monitor performance ## Summary[​](#summary "Direct link to Summary") Vector search in ekoDB enables: ✅ **Semantic search** - Find by meaning, not just keywords ✅ **Recommendations** - Product, content, and user similarity ✅ **Image search** - Visual similarity matching ✅ **RAG systems** - Retrieval-augmented generation ✅ **Integrated** - No separate vector database needed ✅ **Production-ready** - Competitive performance and reliability --- # ekoDB White Paper **Last Revision:** [PDF](/downloads/ekodb-white-paper.pdf)[MD](/downloads/ekodb-white-paper.md)[TXT](/downloads/ekodb-white-paper.txt) **A Multi-Model Database System** *By Sean M. Vazquez, Creator of ekoDB* *** ## Executive Summary[​](#executive-summary "Direct link to Executive Summary") ekoDB is a \~50MB binary that replaces your document store, cache, search engine, vector database, server-side functions, and auth service. Built from scratch in Rust for memory safety and bare-metal performance, it combines in-memory speed with full durability guarantees through Write-Ahead Logging (WAL) — no garbage collector pauses, no null pointer crashes, no data races. No separate Redis for sessions, no Elasticsearch for search, no Pinecone for vectors, no custom auth middleware. One binary, one API, one set of client libraries across six languages. **Performance that matters:** ekoDB doesn't just match the databases it replaces — it outperforms them while doing more work per request: | What ekoDB Replaces | ekoDB | Competitor | Advantage | | ------------------- | --------------------------------- | ------------- | ----------------------------- | | PostgreSQL (writes) | 39K ops/sec | 5K ops/sec | **7.3x faster** | | MongoDB (writes) | 39K ops/sec | 11K ops/sec | **3.7x faster** | | PostgreSQL (reads) | 124K ops/sec | 107K ops/sec | **15% faster, \~2x less CPU** | | Redis (writes) | 39K ops/sec | 6K ops/sec | **6.5x faster** | | MySQL (writes) | 39K ops/sec | 2.5K ops/sec | **15.7x faster** | | Redis (features) | Auth, encryption, search built-in | None included | **Full stack in one binary** | *YCSB benchmarks, 2026-02-16 comparison run: 1M records, 64 threads, full durability, 6 databases tested, with Redis in appendfsync=always so every engine is measured under the same persistence guarantee. Every ekoDB request includes JWT auth, AES-GCM encryption, full-text indexing, and vector indexing — work that competitors skip entirely. The current run and the full workload-by-workload breakdown are at .* **Use less, get more:** ekoDB achieves 2-5x better CPU efficiency than PostgreSQL, MongoDB and MySQL across all workloads. Redis is the exception and wins on this measure: it saturates a single core by design, where ekoDB spreads across several to carry auth, encryption and search on every operation. That means smaller cloud instances, lower monthly bills, and more headroom for your application — not your database. ### Scale Your Way[​](#scale-your-way "Direct link to Scale Your Way") Run one ekoDB for your entire application, or run many. Connect multiple ekoDB instances via **Ripple** for cross-region replication, use **Functions** for server-side business logic — from simple queries to full workflow orchestration with AI, caching, and graph traversal — or orchestrate from your own application API. Start with a single node and scale horizontally as your needs grow — no re-architecture required. ### Less to Learn, More to Build[​](#less-to-learn-more-to-build "Direct link to Less to Learn, More to Build") One query language. One set of client libraries. All capabilities listed below accessible through the same API. Your team learns one database instead of five. **Key Capabilities:** * **Multi-Model Architecture**: Document storage, key-value, full-text search, vector search, real-time subscriptions, and server-side functions in a single system * **Proven Performance**: 37-127K ops/sec with full durability, leading every YCSB workload against 4 competitors in that run, 2-5x better CPU efficiency * **Configurable Durability**: Non-durable mode for high throughput or durable mode for guaranteed persistence * **Adaptive Scaling**: Deployments from IoT devices to enterprise servers * **Secure by Default**: HTTPS/WSS-only, AES-GCM encryption at rest, TLS/SSL in transit * **Adaptive Memory**: 1%-80% of available RAM with automatic management * **Distributed**: Ripple system for horizontal scaling and cross-database propagation * **Server-Side Logic**: Composable Functions for business logic, AI workflows, SWR caching, graph traversal, and external API calls — executed server-side in a single request * **Memory-Safe Foundation**: Built entirely in Rust — no garbage collector pauses, guaranteed memory safety *** ## 1. Introduction[​](#1-introduction "Direct link to 1. Introduction") ### 1.1 The Problem[​](#11-the-problem "Direct link to 1.1 The Problem") Applications often require multiple database systems to handle different workloads: * MongoDB for document storage * Redis for caching and real-time data * Elasticsearch for full-text search * Pinecone for vector search * PostgreSQL for relational data This multi-database approach introduces complexity, operational overhead, and integration challenges. ### 1.2 The Solution[​](#12-the-solution "Direct link to 1.2 The Solution") ekoDB consolidates the capabilities described in the [Executive Summary](#executive-summary) into a single Rust binary — managed through a unified API with client libraries across six languages. ### 1.3 History[​](#13-history "Direct link to 1.3 History") ekoDB originated from a practical challenge: integrating multiple databases required complex layers of abstraction to achieve feature parity and consistent usage patterns. **Development Timeline:** * **2013**: Initial development as SOLO (Single Object Language Operator), an API gateway for multi-database integration * **2013-2022**: SOLO operated as an API gateway connecting various database systems * **2022**: Decision to eliminate the abstraction layer and build a unified database * **2022-2025**: Complete from-scratch rewrite as ekoDB * **Current**: Active development and production use *** ## 2. Architecture[​](#2-architecture "Direct link to 2. Architecture") ### 2.1 Storage Architecture[​](#21-storage-architecture "Direct link to 2.1 Storage Architecture") ekoDB uses a **hybrid in-memory architecture** with disk persistence: ``` ┌─────────────────────────────────────────┐ │ In-Memory Layer (Hot Data) │ │ ┌────────────────────────────────────┐ │ │ │ High-performance storage │ │ │ │ optimized for fast access │ │ │ └────────────────────────────────────┘ │ └─────────────────────────────────────────┘ ↕ ┌─────────────────────────────────────────┐ │ Cache Layer (Warm Data) │ │ ┌────────────────────────────────────┐ │ │ │ Automatic caching with │ │ │ │ intelligent eviction │ │ │ └────────────────────────────────────┘ │ └─────────────────────────────────────────┘ ↕ ┌─────────────────────────────────────────┐ │ Disk Persistence (Cold Data) │ │ ┌────────────────────────────────────┐ │ │ │ Durable storage │ │ │ │ - Encrypted (AES-256-GCM) │ │ │ │ - Compressed │ │ │ │ - Configurable durability │ │ │ └────────────────────────────────────┘ │ └─────────────────────────────────────────┘ ``` **Key Features:** * **Larger-than-Memory Support**: Datasets can exceed available RAM * **Smart Caching**: Frequently accessed data stays in memory for fast access * **Automatic Eviction**: Intelligent cache eviction when memory limits are reached ### 2.2 Data Model[​](#22-data-model "Direct link to 2.2 Data Model") ekoDB supports multiple data models in a unified system: #### Document Model[​](#document-model "Direct link to Document Model") ``` { "name": "John Doe", "email": "john@example.com", "age": 30, "tags": ["developer", "rust"], "address": { "city": "San Francisco", "country": "USA" } } ``` #### Key-Value Model[​](#key-value-model "Direct link to Key-Value Model") ``` { "session:user123": { "token": "abc123", "expires": "2025-10-15T00:00:00Z" } } ``` #### Vector Model[​](#vector-model "Direct link to Vector Model") ``` { "content": "ekoDB is a high-performance database", "embedding": [0.1, 0.2, 0.3, ...], // 384-dimensional vector "metadata": { "source": "documentation" } } ``` ### 2.3 Type System[​](#23-type-system "Direct link to 2.3 Type System") ekoDB provides a flexible type system with per-collection per-field type enforcement: > Note: > > * Type enforcement occurs at the time of write and is not enforced at the time of read. > * Type enforcement is optional at key-value level vs required at document level. > * All types are dynamically inferred at write time or can be explicitly specified via `/schemas` endpoint. **Supported Types:** ekoDB supports 16 comprehensive data types organized into four categories: #### Basic Types[​](#basic-types "Direct link to Basic Types") * **`String`** - UTF-8 encoded text data for names, descriptions, and general text content * **`Integer`** - 64-bit signed integers (-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) * **`Float`** - 64-bit IEEE 754 floating-point numbers for decimal values * **`Boolean`** - Binary true/false values for logical operations #### Advanced Numeric Types[​](#advanced-numeric-types "Direct link to Advanced Numeric Types") * **`Number`** - Flexible numeric type that automatically handles both integers and floats, inferred at write time * **`Decimal`** - Arbitrary-precision decimal numbers that avoid floating-point rounding errors. Essential for financial calculations (e.g., currency, accounting), scientific computations requiring exact decimal representation, and any scenario where `0.1 + 0.2` must equal exactly `0.3`. Unlike `Float`, `Decimal` uses precision-preserving internal storage, ensuring mathematical precision without binary floating-point approximation issues #### Temporal Types[​](#temporal-types "Direct link to Temporal Types") * **`DateTime`** - RFC 3339 formatted date-time values with timezone support (e.g., `2024-01-01T00:00:00Z`) * **`Duration`** - Time duration values for representing time spans (e.g., `30s`, `5m`, `2h`) #### Collection Types[​](#collection-types "Direct link to Collection Types") * **`Array`** - Ordered lists of heterogeneous elements, preserving insertion order * **`Set`** - Unordered collections of unique values with automatic deduplication * **`Vector`** - Fixed-dimension numeric arrays optimized for embeddings and vector similarity search * **`Object`** - Nested documents/maps with key-value pairs for complex structured data #### Specialized Types[​](#specialized-types "Direct link to Specialized Types") * **`UUID`** - Universally unique identifiers (RFC 4122) for globally unique record identification * **`Binary`** - Base64-encoded binary data for images, files, and other binary content * **`Bytes`** - Raw byte arrays for unencoded binary data storage * **`Null`** - Explicit null/empty values for optional fields **Response Formats:** * **Typed**: Includes type metadata (e.g., `{"type": "String", "value": "text"}`) * **Non-Typed**: Traditional NoSQL format (e.g., `"text"`) ### 2.4 Configuration Options[​](#24-configuration-options "Direct link to 2.4 Configuration Options") ekoDB provides flexible configuration options to tune behavior for different use cases: #### Response Format Configuration[​](#response-format-configuration "Direct link to Response Format Configuration") * **Typed Responses**: Include type metadata for strong typing * **Non-Typed Responses**: Traditional NoSQL format for simplicity * **Default**: Typed responses (includes type metadata) * **Configuration**: Configure via configuration API or ekoDB App () #### Durability Configuration[​](#durability-configuration "Direct link to Durability Configuration") * **Durable** (`durable_operations: true`): Guaranteed persistence, every write confirmed to disk * **Non-Durable** (`durable_operations: false`): Higher throughput, async persistence #### Storage Mode Configuration[​](#storage-mode-configuration "Direct link to Storage Mode Configuration") ekoDB provides two storage modes optimized for different workloads: | Mode | Description | Best For | | ------------ | ---------------------------------------- | ------------------------------------- | | **Fast** | In-memory with WAL durability | Maximum throughput, general workloads | | **Balanced** | In-memory with periodic disk persistence | General purpose, mixed workloads | **Fast Mode** (`storage_mode: "fast"`): * Fastest write and read performance * Data recoverable from WAL on restart * Ideal for caches, sessions, and high-throughput ingestion **Balanced Mode** (`storage_mode: "balanced"`): * Near fast-mode performance with eventual disk persistence * Good balance of speed and storage efficiency **Configuration Example:** ``` { "storage_mode": "balanced", "durable_operations": true } ``` Storage mode can be changed at runtime via the configuration API. Each mode works with both durability settings (durable or async). *** ## 3. ACID Properties[​](#3-acid-properties "Direct link to 3. ACID Properties") ekoDB provides **full ACID** — Atomicity, Consistency, Isolation, and Durability. Single operations and multi-operation transactions are atomic and durable, and isolation between concurrent transactions is **enforced** with a configurable level (up to Serializable). The per-property status is below; the [Transactions](/transactions.md) page is the precise reference. ### What is ACID?[​](#what-is-acid "Direct link to What is ACID?") **ACID** stands for Atomicity, Consistency, Isolation, and Durability — the four properties of reliable database transactions: * **Atomicity**: Operations either complete fully or not at all. On failure, changes are rolled back. * **Consistency**: Data moves from one valid state to another; schema constraints and validation are enforced. * **Isolation**: Concurrent transactions don't observe each other's in-progress writes. * **Durability**: Once committed, data survives a crash. Write-Ahead Logging (WAL) guarantees it. ### How ekoDB implements ACID today[​](#how-ekodb-implements-acid-today "Direct link to How ekoDB implements ACID today") * **Atomicity ✅** — Single operations are atomic. Multi-operation work (the transaction API and transactional stored functions) is **buffered**: each statement stages into the transaction and the whole set applies atomically at commit. A failure or rollback discards the staged writes — nothing is partially applied. * **Consistency ✅** — 16 field types and constraint validation (required, unique, min/max, enum, regex, default, null handling) enforce data integrity when a schema is configured. * **Durability ✅** — Write-Ahead Logging with configurable durability modes. In durable mode (the default), a write is fsynced to disk before it is acknowledged, and a failed durable write **fails-stop** — it is never reported as committed. * **Isolation ✅** — Enforced by **optimistic concurrency** over per-record version stamps (MVCC). A transaction's uncommitted writes are invisible to others (no dirty reads) and visible to its own reads that carry the transaction id (read-your-writes); at commit ekoDB validates that the records the transaction read or wrote are unchanged and otherwise rejects the commit as a retryable conflict — nothing is applied. The level is per-transaction configurable up to `Serializable`, with `ReadUncommitted` as the zero-cost default fast path. No locks are held across statements. This applies to both the transaction API and transactional stored functions. ### Transactions[​](#transactions "Direct link to Transactions") ekoDB supports multi-document transactions with: * Atomic, all-or-nothing commit with automatic rollback on error * Savepoints for partial rollback within a transaction * A configurable, **enforced** isolation level (up to Serializable) per transaction * Full audit trail for all operations For usage and the precise semantics of each isolation level, see the [Transactions](/transactions.md) page. *** ## 4. Performance[​](#4-performance "Direct link to 4. Performance") ### 4.1 Write Performance[​](#41-write-performance "Direct link to 4.1 Write Performance") Write throughput depends on the [durability mode](#24-configuration-options) selected. In durable mode, ekoDB leads all tested competitors on write-heavy and mixed YCSB workloads with equivalent persistence guarantees; on pure read-only, PostgreSQL and ekoDB trade the lead. In non-durable mode, throughput increases further for workloads where crash recovery is not critical. **Dual-Node Strategy**: Deploy a primary node in non-durable mode and a secondary node in durable mode (via Ripple replication) to achieve both high performance and durability. ### 4.2 Read Performance[​](#42-read-performance "Direct link to 4.2 Read Performance") * **Indexed Lookups**: Fast hash and B-tree index lookups * **Point Queries**: Sub-millisecond latency * **Range Queries**: Efficient range scans via B-tree indexes * **Full-Text Search**: Fast term-based retrieval * **Vector Search**: Efficient nearest neighbor search ### 4.3 Memory Efficiency[​](#43-memory-efficiency "Direct link to 4.3 Memory Efficiency") * **Base Memory**: Low memory footprint optimized for efficiency * **Compression**: Significant space savings with configurable compression * **Adaptive Allocation**: 1%-80% of available RAM * **Automatic Eviction**: Intelligent memory management *** ## 5. Indexing[​](#5-indexing "Direct link to 5. Indexing") ### 5.1 Index Types[​](#51-index-types "Direct link to 5.1 Index Types") ekoDB implements multiple index types optimized for different query patterns: #### Hash Indexes (Default)[​](#hash-indexes-default "Direct link to Hash Indexes (Default)") * **Use Case**: `WHERE field = value` — fast equality lookups * **Automatic**: Created for frequently queried fields #### B-Tree Indexes[​](#b-tree-indexes "Direct link to B-Tree Indexes") * **Use Case**: `WHERE field > value`, sorting, range scans * **Features**: Supports `<`, `>`, `<=`, `>=`, `BETWEEN` #### Inverted Indexes[​](#inverted-indexes "Direct link to Inverted Indexes") * **Use Case**: Full-text search * **Features**: Stemming, fuzzy matching, tokenization #### Vector Indexes[​](#vector-indexes "Direct link to Vector Indexes") * **Use Case**: Semantic similarity search, embeddings, AI/ML workloads * **Metrics**: Cosine similarity, Euclidean distance, dot product * **Search Modes**: Approximate nearest neighbor and exact search ### 5.2 Index Management[​](#52-index-management "Direct link to 5.2 Index Management") * **Automatic Creation**: Indexes created based on query patterns * **Automatic Maintenance**: Updated on insert/update/delete * **Concurrent Access**: Thread-safe operations * **Memory Efficient**: Automatic memory management and eviction *** ## 6. Concurrency & Isolation[​](#6-concurrency--isolation "Direct link to 6. Concurrency & Isolation") ### 6.1 Concurrency Control[​](#61-concurrency-control "Direct link to 6.1 Concurrency Control") ekoDB uses a lock-free concurrent in-memory store (sharded maps): * **Concurrent reads**: Many readers access data simultaneously without blocking * **Per-key write atomicity**: A single record write is applied atomically — never seen half-applied * **Cross-collection independence**: Operations on different collections don't block each other ### 6.2 Isolation[​](#62-isolation "Direct link to 6.2 Isolation") Isolation between concurrent **transactions** is **enforced** by optimistic concurrency over per-record version stamps (MVCC). A transaction buffers its writes and applies them atomically at commit, so its uncommitted writes are invisible to other transactions (no dirty reads) and visible only to its own reads that carry the transaction id (read-your-writes). At commit, ekoDB validates that the records the transaction read or wrote have not been changed under it by another committed transaction; if they have, the commit is rejected as a **retryable conflict** and nothing is applied. No locks are held across statements — conflicts surface at commit rather than blocking. The level is per-transaction configurable and selects how much the commit validates: * `ReadUncommitted` — no validation (last-writer-wins); the zero-cost default fast path. * `ReadCommitted` — reject the commit on a write-write conflict (lost-update prevention). * `RepeatableRead` — additionally reject on a read-set change (prevents non-repeatable reads / read skew). * `Serializable` — additionally reject on a collection-membership change (prevents phantoms / write skew; serializable history). The default level is configurable at runtime (`default_isolation_level`), and direct (non-transactional) writes bump the same per-record version, so they are coherent with concurrent transactions by construction. The same enforced model backs both the transaction API and transactional stored functions. *** ## 7. Durability & Recovery[​](#7-durability--recovery "Direct link to 7. Durability & Recovery") ### 7.1 Write-Ahead Logging (WAL)[​](#71-write-ahead-logging-wal "Direct link to 7.1 Write-Ahead Logging (WAL)") All writes are recorded to a Write-Ahead Log before execution. The [durability mode](#24-configuration-options) controls whether each WAL entry is fsynced before acknowledging the client (durable) or batched asynchronously (non-durable). **WAL Management:** * Automatic rotation and compaction * Manual rotation available * Automatic cleanup after replication ### 7.2 Recovery Process[​](#72-recovery-process "Direct link to 7.2 Recovery Process") ekoDB performs automatic crash recovery. The system validates data integrity and restores the database to its last consistent state. Recovery time depends on data volume and system resources. *** ## 8. Search Capabilities[​](#8-search-capabilities "Direct link to 8. Search Capabilities") Built on the [index types described in Section 5](#51-index-types), ekoDB exposes three search modes through a unified query API: ### 8.1 Full-Text Search[​](#81-full-text-search "Direct link to 8.1 Full-Text Search") * **Field Weighting**: Prioritize specific fields in relevance scoring * **Minimum Score**: Filter results by relevance threshold * **Fuzzy Matching**: Typo tolerance via Levenshtein distance ### 8.2 Vector Search[​](#82-vector-search "Direct link to 8.2 Vector Search") * **Embedding Support**: 384, 768, 1536 dimensions (configurable) * **Metadata Filtering**: Combine vector similarity with field filters * **Top-K Results**: Efficient nearest-neighbor selection ### 8.3 Hybrid Search[​](#83-hybrid-search "Direct link to 8.3 Hybrid Search") Combine full-text and vector search in a single query — keyword matching for precision, semantic similarity for recall, with unified scoring and ranking. *** ## 9. Distributed Architecture[​](#9-distributed-architecture "Direct link to 9. Distributed Architecture") ### 9.1 Ripple System[​](#91-ripple-system "Direct link to 9.1 Ripple System") ekoDB's distributed architecture uses the **Ripple** system for real-time data propagation and horizontal scaling: ``` ┌─────────────────────────────────────────────────┐ │ Regional Cluster │ │ ┌─────────────────────────────────────────┐ │ │ │ Instance Groups │ │ │ │ ┌──────────┐ ┌──────────┐ │ │ │ │ │ Node 1 │ │ Node 2 │ ... │ │ │ │ │ (Primary)│ │(Secondary)│ │ │ │ │ └──────────┘ └──────────┘ │ │ │ └─────────────────────────────────────────┘ │ └─────────────────────────────────────────────────┘ ↕ Ripple ┌─────────────────────────────────────────────────┐ │ Multi-Tenant Single Nodes │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Tenant A │ │ Tenant B │ │ Tenant C │ │ │ └──────────┘ └──────────┘ └──────────┘ │ └─────────────────────────────────────────────────┘ ``` **Core Features:** * **Real-Time Propagation**: Operations replicate immediately as they occur * **Cross-Database Propagation**: Replicate data across ekoDB instances * **Horizontal Scaling**: Add nodes for increased read/write capacity * **Regional Distribution**: Deploy across geographic regions for low-latency access * **Automatic Failover**: Secondary nodes take over on primary failure * **Loop Prevention**: Automatic deduplication prevents infinite operation loops in multi-node deployments * **Configurable Modes**: Send-only, receive-only, bidirectional, or isolated nodes **Real-World Use Cases:** **Geographic Distribution (Multi-Region)** Deploy primary nodes in US, EU, and Asia with cross-region ripples. Users read from their nearest region for low latency while writes propagate globally for consistency. **Read Scaling (Analytics)** Configure a primary write node in durable mode sending ripples to multiple read replicas in non-durable mode. Production traffic doesn't slow down while replicas handle heavy reporting queries. **High Availability (Active-Active)** Two primary nodes in bidirectional mode accept writes simultaneously. If one fails, the other continues operations without manual intervention. **Data Pipelines (IoT)** Dedicated ingestion nodes in non-durable mode propagate to storage nodes in durable mode and processing nodes for real-time analytics. Each layer scales independently. **Development/Staging** Production primary sends ripples to staging replica for realistic testing. Test writes in staging don't propagate back to production. See [Ripples - Data Propagation](/ripples.md) documentation for comprehensive architecture guides and configuration examples ### 9.2 Replication[​](#92-replication "Direct link to 9.2 Replication") * **Asynchronous Replication**: Non-blocking writes * **Configurable Targets**: Replicate to multiple destinations * **Selective Replication**: Choose which collections to replicate * **Conflict Resolution**: Last-write-wins strategy *** ## 10. Functions[​](#10-functions "Direct link to 10. Functions") ekoDB's **Functions** system is a server-side execution engine for composable business logic. A single Function call can query data, transform results, invoke AI models, and call external APIs — all in one request, with no additional network round-trips. ### 10.1 Composable Server-Side Logic[​](#101-composable-server-side-logic "Direct link to 10.1 Composable Server-Side Logic") Functions are JSON-defined, parameterized, versioned, and callable from any client library or the REST API. Each Function contains one or more operations (Query, Insert, Update, Chat, Embed, HttpRequest, and more) that execute sequentially with results flowing between steps. Functions compose freely: a Function can call other Functions via `CallFunction`, enabling modular workflows without duplicating logic across clients. Built-in conditional execution (`If`), iteration (`ForEach`), and error handling give Functions the control flow needed for real business logic — not just simple queries. ### 10.2 Caching & Stale-While-Revalidate[​](#102-caching--stale-while-revalidate "Direct link to 10.2 Caching & Stale-While-Revalidate") Functions integrate naturally with the **stale-while-revalidate (SWR)** pattern. Cached Function results are served instantly while fresh data is computed in the background. This turns expensive multi-step operations — dashboard summaries, analytics aggregations, enriched user profiles — into instant responses with automatic freshness. For workloads that don't change on every request, SWR eliminates perceived latency entirely. ### 10.3 Graph Traversal with Vectors[​](#103-graph-traversal-with-vectors "Direct link to 10.3 Graph Traversal with Vectors") By combining **vector search** with **conditional logic** and **Function composition**, you can build graph-like traversals entirely within ekoDB. Use vector similarity to discover related nodes, chain Functions to walk relationships, and apply conditional branching to shape the traversal — all server-side, in a single call. This enables recommendation engines, knowledge graphs, and relationship discovery without a separate graph database. ### 10.4 AI Integration[​](#104-ai-integration "Direct link to 10.4 AI Integration") Functions natively support **Chat** (LLM completions) and **Embed** (vector generation) operations alongside standard database operations. Build complete RAG pipelines, content moderation workflows, automated summarization, or any AI-augmented logic — without leaving the database layer or adding external orchestration. For detailed architecture, operation types, and code examples, see [Functions Architecture](/reference/functions-architecture.md). *** ## 11. Security[​](#11-security "Direct link to 11. Security") ### 11.1 Network Security[​](#111-network-security "Direct link to 11.1 Network Security") **HTTPS/WSS Only**: Unlike traditional databases that use direct TCP connections, ekoDB exclusively uses HTTPS and WSS (Secure WebSocket) protocols. **Benefits:** * No direct TCP exposure * Built-in encryption (TLS/SSL) * Standard web protocols * Firewall friendly (port 443) * Certificate-based security * Protection against man-in-the-middle attacks ### 11.2 Encryption[​](#112-encryption "Direct link to 11.2 Encryption") **At Rest:** * AES-256-GCM encryption for all stored data * Volume-level encryption provided by ekoDB managed infrastructure **In Transit:** * TLS/SSL for all network communication * Certificate validation * Perfect forward secrecy ### 11.3 Authentication[​](#113-authentication "Direct link to 11.3 Authentication") * **JWT Tokens**: Industry-standard authentication * **API Keys**: Simple authentication for services * **Role-Based Access**: Collection-level and field-level permissions * **Token Expiration**: Configurable TTL *** ## 12. Use Cases[​](#12-use-cases "Direct link to 12. Use Cases") ### 12.1 AI Agents & Workflows[​](#121-ai-agents--workflows "Direct link to 12.1 AI Agents & Workflows") * **Vector Search**: Store and query embeddings * **Chat History**: Built-in chat session management * **Context Management**: Efficient retrieval of relevant context * **Real-Time Updates**: WebSocket for live agent interactions ### 12.2 Real-Time Analytics[​](#122-real-time-analytics "Direct link to 12.2 Real-Time Analytics") * **High Throughput**: High-speed ingestion for real-time analytics * **In-Memory Processing**: Sub-millisecond queries * **Time-Series Data**: Efficient storage and retrieval * **Aggregations**: Fast analytical queries ### 12.3 IoT Data Processing[​](#123-iot-data-processing "Direct link to 12.3 IoT Data Processing") * **Adaptive Scaling**: Runs on resource-constrained devices * **Edge Computing**: Deploy close to data sources * **Efficient Storage**: Compression reduces disk usage * **Batch Operations**: Handle bursts of sensor data ### 12.4 Session Management[​](#124-session-management "Direct link to 12.4 Session Management") * **TTL Support**: Automatic session expiration * **Fast Lookups**: O(1) key-value operations * **High Concurrency**: Handle thousands of sessions * **Persistence**: Optional durability for sessions ### 12.5 Content Delivery[​](#125-content-delivery "Direct link to 12.5 Content Delivery") * **Distributed Caching**: Ripple for multi-region deployment * **Fast Reads**: In-memory performance * **Compression**: Reduce bandwidth usage * **Real-Time Updates**: WebSocket for live content *** ## 13. Getting Started[​](#13-getting-started "Direct link to 13. Getting Started") ### 13.1 Quick Start[​](#131-quick-start "Direct link to 13.1 Quick Start") ``` # Install client library npm install @ekodb/ekodb-client # Connect to ekoDB import { EkoDBClient } from "@ekodb/ekodb-client"; const client = new EkoDBClient({ baseURL: "https://my-first-db.development.google.ekodb.net", apiKey: "your-api-key" }); await client.init(); // Insert a document await client.insert("users", { name: "John Doe", email: "john@example.com" }); // Query documents using ekoDB query builder const query = { filter: { type: "Condition", content: { field: "age", operator: "Gt", value: 25 } } }; const users = await client.find("users", query); ``` ### 13.2 Resources[​](#132-resources "Direct link to 13.2 Resources") * **Homepage**: * **Documentation**: * **Management Console**: * **Support**: *** ## 14. Architecture Deep Dives[​](#14-architecture-deep-dives "Direct link to 14. Architecture Deep Dives") For detailed technical documentation on specific subsystems: * **[Functions Architecture](/reference/functions-architecture.md)** - Server-side execution, operation types, workflows, and composition patterns * **[Transactions Architecture](/reference/transaction-architecture.md)** - Atomic, durable, isolated transactions (enforced MVCC isolation), savepoints, and rollback * **[Advanced Operations](/client-libraries/advanced-operations.md)** - Client library usage, search capabilities, chat integration, and real-time features *** ## 15. About the Author[​](#15-about-the-author "Direct link to 15. About the Author") **Sean M. Vazquez** is the creator and lead developer of ekoDB. His journey in technology began at age 10, writing HTML, CSS, and JavaScript on an old Compaq computer. Those early years were filled with screen-by-screen adventure games and tinkering with 8-bit and 16-bit consoles (Game Boy, SNES, Sega Genesis), sparking a lifelong passion for building software and the endless wonder of console modding. To this day, he still treasures his N64, playing Mario Kart, Ocarina of Time, 007, and Super Smash Bros with game mods. As he grew, Sean expanded into C++ and JavaScript, and discovered Linux, a discovery that would shape his approach to systems engineering. He enrolled at Stevens Institute of Technology as a mechanical engineer, but the pull of code was too strong. Remembering the joy of those early programming days, he switched to computer science. **Career Progression:** * **Stevens Institute of Technology**: B.S. in Computer Science, 2013 * **Thomson Reuters**: QA Engineer Intern, working on C# tax software where he learned the importance of quality and testing at scale * **UBS**: eLearning software implementation and automation, building systems that reached thousands of employees * **TMP Worldwide**: Full Stack Engineer focused on front-end development for enterprise recruitment platforms * **Metacake**: Full Stack Engineer driving growth engineering and building products from scratch for contract clients * **American Express**: Grew from Full Stack Engineer to Senior Engineering Manager and Lead Software Engineer/Architect, leading both product development and internal tooling engineering with a focus on automation **The Origin of ekoDB:** ekoDB began in 2013 as SOLO (Single Object Language Operator), an ambitious vision to create "one API gateway to rule them all." But Sean's vision went further: why just abstract databases when you could make all data universally accessible? The goal became empowering engineers to build their best backends without being constrained by database choices. After nearly a decade as an API gateway, SOLO was rewritten from scratch as ekoDB, a unified database platform that eliminates the complexity of multi-database architectures. Today, Sean works from a setup that reflects his roots: a Filco tenkeyless keyboard, Raspberry Pis running Arch Linux alongside Debian and Ubuntu, and macOS for daily work. He still has a soft spot for the iPod. **Contact**: *** *Features and specifications are subject to change as ekoDB continues to evolve.* [PDF](/downloads/ekodb-white-paper.pdf)[MD](/downloads/ekodb-white-paper.md)[TXT](/downloads/ekodb-white-paper.txt) --- # Why ekoDB? ekoDB is an **agentic backend platform**: a fast, AI-native database, plus AI agents that build and operate your backend so you can focus on the frontend. It runs great as a standalone database too. Where ekoDB shines: ## 🤖 The backend that builds itself[​](#-the-backend-that-builds-itself "Direct link to 🤖 The backend that builds itself") Deploy a database and an AI Agent, describe the backend you want, and the agent builds it for you: * **Describe it, don't code it** - The agent turns a plain-English description into real collections and server-side functions in your database * **Your app just calls it** - Invoke what the agent built through the SDK with `client.callFunction(...)`, and spend your time on the frontend * **It operates and scales it** - The agent reads logs, diagnoses issues, tests functions, and adds indexes, and it can deploy, convert a database to a cluster, and take backups * **Runs next to your data** - The agent connects to your own database using your own keys ## 🚀 Unrivaled Performance[​](#-unrivaled-performance "Direct link to 🚀 Unrivaled Performance") Powered by Rust, ekoDB has exceptional real-time performance capabilities: * **Sub-millisecond point reads** and up to \~127K durable operations/sec (YCSB) * **High-throughput ingestion** for demanding workloads * **Wins 19 of 20 YCSB head-to-head matchups** against PostgreSQL, MongoDB, MySQL and Redis, at 2-5x better CPU efficiency than PostgreSQL, MongoDB and MySQL. Pure read-only against PostgreSQL is the one the two trade run to run, and Redis is more CPU-efficient by design — see [Performance Benchmarks](/reference/performance.md) * Optimized for both CRUD operations and complex queries ## 🤖 AI-Native Architecture[​](#-ai-native-architecture "Direct link to 🤖 AI-Native Architecture") ekoDB's AI integration is a significant differentiator: * **No second database needed** - Vector search, embeddings, and RAG built-in * **Multiple LLM support** - OpenAI, Claude, Perplexity, and more * **Controlled environment** - Secure AI data access without complexity * **Proven methods** - Industry-standard algorithms for building AI applications ## 🌐 Built for Scale[​](#-built-for-scale "Direct link to 🌐 Built for Scale") Engineered from inception with scalability in mind: * **Ripple system** - Manage multiple database instances effortlessly * **Data propagation** - Automatic sync between distributed nodes * **Authentication management** - Built-in token and distributed record handling * **Multi-protocol** - Supports both REST API and WebSocket Secure ## 💡 Developer-First Experience[​](#-developer-first-experience "Direct link to 💡 Developer-First Experience") Simple, transparent, and straightforward: * **Developer experience flexibility** - Build with provided client libraries or fully supported REST APIs and WebSocket Secure * **Tackle multiple problems with one database** - Caching, storage, messaging, queues, functional logic, AI access - all in one database * **Flexible schema** - NoSQL flexibility with optional constraints as you build and transactional when you need it * **Quick to deploy** - From hobby projects to enterprise-scale applications Unleash the limitless potential of ekoDB and build what you could never build before! ---