Deployment

All CROW services are deployed as Cloudflare Workers using Wrangler. Each service has its own wrangler.jsonc configuration file with environment-specific overrides for local, dev, and prod.

Workspace Bootstrap

Use the local-dev repo to clone and synchronize the multi-repo workspace before deploying individual services:

git clone https://github.com/CROW-B3/local-dev.git
cd local-dev
bun install
bun run clone

See Local Development for sync, checkout, cleanup, and optional repo workflows.

Environments

Environment	Domain Pattern	Database Suffix	Notes
local	`localhost:800x`	`-local`	Local Wrangler dev server
dev	`dev.*.crowai.dev`	`-dev`	Staging environment
prod	`*.crowai.dev`	(none)	Production

Domain Mapping

Complete domain-to-service mapping for dev and prod environments:

Public-Facing Domains

Dev Domain	Prod Domain	Service
`dev.app.crowai.dev`	`app.crowai.dev`	dashboard-client
`dev.auth.crowai.dev`	`auth.crowai.dev`	auth-client
`dev.crowai.dev`	`crowai.dev`	landing-client
`dev.rogue.crowai.dev`	`rogue.crowai.dev`	rogue-store
`dev.api.crowai.dev`	`api.crowai.dev`	core-api-gateway
`dev.mcp.crowai.dev`	`mcp.crowai.dev`	mcp-service
`dev.a2a.crowai.dev`	`a2a.crowai.dev`	a2a-service
`dev.cctv.crowai.dev`	--	cctv-ingest-service

Internal Service Domains

Dev Domain	Prod Domain	Service
`dev.internal.auth-api.crowai.dev`	`internal.auth-api.crowai.dev`	core-auth-service
`dev.internal.users.crowai.dev`	`internal.users.crowai.dev`	core-user-service
`dev.internal.orgs.crowai.dev`	`internal.orgs.crowai.dev`	core-organization-service
`dev.internal.products.crowai.dev`	`internal.products.crowai.dev`	core-product-service
`dev.interactions.crowai.dev`	`interactions.crowai.dev`	core-interaction-service
`dev.internal.interactions.crowai.dev`	--	core-interaction-service (internal)
`dev.internal.billing.crowai.dev`	`internal.billing.crowai.dev`	core-billing-service
`dev.internal.notifications.crowai.dev`	`internal.notifications.crowai.dev`	core-notification-service
`dev.internal.analytics.crowai.dev`	`internal.analytics.crowai.dev`	core-analytics-service
`dev.internal.chat.crowai.dev`	`internal.chat.crowai.dev`	core-chat-service / bff-chat-service
`dev.internal.qna.crowai.dev`	`internal.qna.crowai.dev`	bff-qna-service
`dev.patterns.crowai.dev`	`patterns.crowai.dev`	core-pattern-service
`dev.internal.patterns.crowai.dev`	--	core-pattern-service (internal)
`dev.internal.ingest-worker.crowai.dev`	--	web-ingest-service
`dev.ingest.crowai.dev`	--	web-ingest-service (public)
`dev.social-collector.crowai.dev`	`social-collector.crowai.dev`	core-social-collector
`dev.internal.social-collector.crowai.dev`	--	core-social-collector (internal)

Deploying a Service

Deploy to Dev

cd core-auth-service
npx wrangler deploy --env dev

Deploy to Prod

cd core-auth-service
npx wrangler deploy

The --env flag selects the environment block from wrangler.jsonc. When omitted, the top-level (prod) configuration is used.

Tailing Logs

npx wrangler tail --env dev

This streams real-time logs from the deployed worker. Useful for debugging requests in dev.

Local Development

npx wrangler dev --env local

Starts a local development server. D1 databases run locally, but AI, Browser Rendering, and Vectorize bindings may require --remote for full functionality.

Workers Containers

Several services deploy as Workers Containers, which run Docker images alongside the Worker. The container configuration is in wrangler.jsonc under the containers key.

Services Using Containers

Service	Container Class	Image	Max Instances	Purpose
core-pattern-service	`PatternAnalyzerContainer`	`./Dockerfile`	2	Python with sklearn, numpy for pattern analysis
core-interaction-service	`InteractionAnalyzerContainer`	`./Dockerfile`	2	Interaction analysis
infra-crawl-service	`CrawlerContainer`	`./container_src/Dockerfile`	10	Web crawling with headless browser

Container Configuration Example

"containers": [
  {
    "class_name": "PatternAnalyzerContainer",
    "image": "./Dockerfile",
    "max_instances": 2
  }
]

Deploying Container Services

Container images are built and pushed to Cloudflare's container registry as part of wrangler deploy:

cd core-pattern-service
npx wrangler deploy --env dev

Wrangler builds the Docker image from the specified Dockerfile, pushes it to the registry, and deploys the Worker alongside it.

Checking Container Health

# Tail logs for the container service
npx wrangler tail crow-core-pattern-service --env dev

# Check deployment status
npx wrangler deployments list --env dev

Container instances are managed as Durable Objects internally. Each container class has a corresponding durable_objects binding and SQLite migration for persistent state.

Frontend Deployment

The dashboard, auth, and rogue-store clients are deployed via OpenNext on Cloudflare. Their wrangler.jsonc references .open-next/worker.js as the main entrypoint and .open-next/assets as the assets directory. The landing client uses Astro's Cloudflare adapter and deploys the generated dist/_worker.js/index.js worker bundle directly.

Services Using OpenNext

Service	Dev Domain	Prod Domain
dashboard-client	`dev.app.crowai.dev`	`app.crowai.dev`
auth-client	`dev.auth.crowai.dev`	`auth.crowai.dev`
rogue-store	`dev.rogue.crowai.dev`	`rogue.crowai.dev`

Build & Deploy

cd dashboard-client

# Build Next.js, then build for Cloudflare, then deploy
npx next build
npx opennextjs-cloudflare build
npx wrangler deploy --env dev

Or as a single chain:

npx next build && npx opennextjs-cloudflare build && npx wrangler deploy --env dev

Landing Client (Astro)

cd landing-client
bun run build
npx wrangler deploy --env dev

OpenNext Configuration Notes

Assets are served via the ASSETS binding pointing to .open-next/assets
KV namespaces (NEXT_INC_CACHE_KV) are used for Next.js incremental cache
Smart placement ("placement": { "mode": "smart" }) is enabled for optimal edge routing
Each Next.js client has its own custom domain configuration in routes

Durable Objects

Durable Objects provide persistent, single-threaded execution contexts. Several services use them for stateful workloads.

Services Using Durable Objects

Service	DO Class	Purpose	Migration Tag
core-pattern-service	`PatternAnalyzerContainer`	Container-backed pattern analysis	v1 (new_sqlite_classes)
core-interaction-service	`InteractionAnalyzerContainer`	Container-backed interaction analysis	v1 (new_sqlite_classes)
web-ingest-service	`CrowWebSession`	Persistent web session tracking	v1/v3 (renamed from MyDurableObject)
cctv-ingest-service	`IngestSession`	CCTV ingest sessions (deleted in v2)	v1/v2
infra-crawl-service	`CrawlerContainer`	Container-backed web crawler	v1 (new_sqlite_classes)

Migration Configuration

Durable Object migrations are defined in wrangler.jsonc under the migrations key:

"migrations": [
  { "tag": "v1", "new_sqlite_classes": ["PatternAnalyzerContainer"] }
]

For Workers Containers, the Durable Object class is the container class itself, using new_sqlite_classes for SQLite-backed persistent state.

Standard Durable Objects (non-container) use new_classes:

"migrations": [
  { "tag": "v1", "new_classes": ["IngestSession"] },
  { "tag": "v2", "deleted_classes": ["IngestSession"] }
]

Migrations are applied automatically on deploy. Ensure dev and prod migration arrays match to avoid drift.

D1 Database Migrations

Services Using D1

Service	Binding	Migrations Dir	Dev Database Name
core-auth-service	`DB`	`drizzle/migrations`	`crow-core-auth-service-db-dev`
core-user-service	`DB`	`drizzle/migrations`	`crow-core-user-service-db-dev`
core-organization-service	`DB`	`drizzle/migrations`	`crow-core-organization-service-db-dev`
core-product-service	`DB`	`drizzle/migrations`	`crow-core-product-service-db-dev`
core-interaction-service	`DB`	`drizzle/migrations`	`crow-core-interaction-service-db-dev`
core-pattern-service	`DB`	`drizzle/migrations`	`crow-core-pattern-service-db-dev`
core-billing-service	`DB`	`drizzle/migrations`	`crow-core-billing-service-db-dev`
core-analytics-service	`DB`	`drizzle/migrations`	`crow-core-analytics-service-db-dev`
core-chat-service	`DB`	`drizzle/migrations`	`crow-core-chat-service-db-dev`
mcp-service	`DB`	(default)	`crow-mcp-service-db-dev`
bff-chat-service	`DB`	`migrations`	`crow-bff-chat-service-db-dev`
bff-qna-service	`DB`	(default)	`crow-bff-qna-service-db-dev`
cctv-ingest-service	`DB`	`migrations`	`crow-cctv-ingest-db-dev`
web-ingest-service	`DB`	(default)	`crow-web-ingest-service-db-dev`
core-social-collector	`DB`	`drizzle/migrations`	`crow-social-collector-db-dev`
dashboard-client	`DB_MAIN`	(default)	`crow-dashboard-client-db-dev`
landing-client	`DB_MAIN`	(default)	`crow-landing-client-db-dev`

Creating a New D1 Database

npx wrangler d1 create crow-my-service-db-dev

Copy the returned database_id into the service's wrangler.jsonc.

Running Migrations

Migrations are stored in each service's drizzle/migrations/ directory (or migrations/ for bff-chat-service, cctv-ingest-service):

npx wrangler d1 migrations apply DB --env dev --remote

The --remote flag applies migrations to the remote D1 database (not local).

Generating Migrations

After modifying src/db/schema.ts:

npx drizzle-kit generate

This generates a new SQL migration file in the migrations directory.

Querying a Database

npx wrangler d1 execute DB --env dev --command "SELECT * FROM user LIMIT 5"

Queue Workers

Queues provide asynchronous message passing between services. Queue names include the environment suffix (e.g., crow-interaction-queue-dev).

Queue Topology

Queue Name	Producer(s)	Consumer	DLQ
`crow-interaction-queue`	web-ingest-service, core-social-processor	core-interaction-service	--
`crow-cctv-batch-queue`	cctv-ingest-service, core-interaction-service	core-interaction-service	--
`crow-product-crawl-queue`	core-auth-service	core-product-service	--
`crow-organization-context-queue`	core-organization-service	core-organization-service	`crow-organization-context-dlq`
`crow-social-processing-queue`	core-social-collector	core-social-processor	--
`crow-session-expiry-queue`	web-ingest-service	web-ingest-service	--
`crow-web-session-export`	web-ingest-service	--	--
`email-queue`	core-notification-service	core-notification-service	`email-dlq`
`crow-notification-queue`	--	core-notification-service	--

Consumer Configuration

Consumers are configured with batch size, timeout, retries, and optional DLQ:

"consumers": [
  {
    "queue": "crow-interaction-queue-dev",
    "max_batch_size": 10,
    "max_batch_timeout": 30,
    "max_retries": 3,
    "retry_delay": 60
  }
]

Dead Letter Queues

DLQs are configured for critical queues:

crow-organization-context-dlq / crow-organization-context-dlq-dev
email-dlq / email-dlq-dev

Failed messages that exhaust all retries are routed to the DLQ for manual inspection.

Cron Triggers

Service	Schedule	Purpose
core-pattern-service	`0 * * * ` (hourly), `0 2 * ` (daily 2am), `0 3 * 1` (weekly Mon 3am), `0 4 1 * ` (monthly 1st 4am), `0 5 1 1 ` (yearly Jan 1st 5am)	Pattern analysis at multiple intervals
bff-qna-service	`0 /12 * *` (every 12 hours)	QnA index refresh
core-social-collector	`0 /2 * *` (every 2 hours)	Social media data collection

Secrets Management

Secrets are never stored in wrangler.jsonc. They are set via the Wrangler CLI:

npx wrangler secret put SECRET_NAME --env dev

This prompts for the secret value interactively. The secret is stored in Cloudflare's encrypted secrets store and injected into the worker environment at runtime.

Listing Secrets

npx wrangler secret list --env dev

Common Secrets by Service

Service	Secrets
core-api-gateway	`INTERNAL_GATEWAY_KEY`, `SERVICE_API_KEY_ORG_SERVICE`, `CRAWLER_SERVICE_SECRET`
core-auth-service	`BETTER_AUTH_SECRET`, `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `SERVICE_API_KEY_GATEWAY`, `SERVICE_API_KEY_WEB_INGEST`, `SERVICE_API_KEY_*`
core-user-service	`BETTER_AUTH_SECRET`, `INTERNAL_GATEWAY_KEY`, `SERVICE_API_KEY_AUTH`, `SERVICE_API_KEY_ORGANIZATION`, `SERVICE_API_KEY_BILLING`
core-organization-service	`INTERNAL_GATEWAY_KEY`, `SERVICE_API_KEY_GATEWAY`, `SERVICE_API_KEY_ORGANIZATION`
core-product-service	`CRAWLER_SERVICE_SECRET`
core-interaction-service	`SYSTEM_SECRET`, `INTERNAL_GATEWAY_KEY`
core-pattern-service	`SYSTEM_SECRET`, `INTERNAL_GATEWAY_KEY`
core-billing-service	`BETTER_AUTH_SECRET`, `INTERNAL_GATEWAY_KEY`, `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`
core-notification-service	`BETTER_AUTH_SECRET`, `INTERNAL_GATEWAY_KEY`, `RESEND_API_KEY`
core-analytics-service	`INTERNAL_GATEWAY_KEY`
bff-qna-service	`INTERNAL_GATEWAY_KEY`, `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_R2_ACCESS_KEY_ID`, `CLOUDFLARE_R2_SECRET_ACCESS_KEY`
mcp-service	`SERVICE_API_KEY`, `INTERNAL_GATEWAY_KEY`
core-chat-service	`INTERNAL_GATEWAY_KEY`
bff-chat-service	`INTERNAL_GATEWAY_KEY`
a2a-service	`INTERNAL_GATEWAY_KEY`
web-ingest-service	`INTERNAL_GATEWAY_KEY`, `SERVICE_API_KEY`
core-social-collector	`TAVILY_API_KEY`, `SYSTEM_SECRET`
core-social-processor	(inherits via queue binding)
infra-crawl-service	`CRAWLER_SERVICE_SECRET`

Wrangler Configuration Structure

Every service's wrangler.jsonc follows this pattern:

{
  "name": "crow-{service-name}",
  "main": "src/index.ts",
  "compatibility_date": "2025-12-17",
  // Top-level = prod config
  "vars": { "ENVIRONMENT": "prod", ... },
  "d1_databases": [{ "binding": "DB", ... }],
  "routes": [{ "pattern": "*.crowai.dev", "custom_domain": true }],
  "env": {
    "dev": {
      "vars": { "ENVIRONMENT": "dev", ... },
      "d1_databases": [{ ... }],
      "routes": [{ "pattern": "dev.*.crowai.dev", ... }]
    },
    "local": {
      "vars": { "ENVIRONMENT": "local", ... },
      "d1_databases": [{ ... }]
    }
  }
}

Observability

All services have observability enabled in their wrangler config:

"observability": {
  "enabled": true,
  "head_sampling_rate": 1
}

This enables Cloudflare's built-in observability with 100% sampling. Logs can be viewed via wrangler tail or the Cloudflare dashboard.

Monitoring Commands

npx wrangler tail --env dev                    # Stream live logs
npx wrangler tail --env dev --format json      # JSON formatted logs
npx wrangler d1 info DB --env dev              # Database info
npx wrangler secret list --env dev             # List all secrets
npx wrangler deployments list --env dev        # Deployment history

Workspace Bootstrap​

Environments​

Domain Mapping​

Public-Facing Domains​

Internal Service Domains​

Deploying a Service​

Deploy to Dev​

Deploy to Prod​

Tailing Logs​

Local Development​

Workers Containers​

Services Using Containers​

Container Configuration Example​

Deploying Container Services​

Checking Container Health​

Frontend Deployment​

Services Using OpenNext​

Build & Deploy​

Landing Client (Astro)​

OpenNext Configuration Notes​

Durable Objects​

Services Using Durable Objects​

Migration Configuration​

D1 Database Migrations​

Services Using D1​

Creating a New D1 Database​

Running Migrations​

Generating Migrations​

Querying a Database​

Queue Workers​

Queue Topology​

Consumer Configuration​

Dead Letter Queues​

Cron Triggers​

Secrets Management​

Listing Secrets​

Common Secrets by Service​

Wrangler Configuration Structure​

Observability​

Monitoring Commands​