core-social-processor

Queue-driven social content processor that enriches raw social media items with AI-generated sentiment analysis, topic extraction, entity recognition, and summaries. Consumes messages from the social processing queue and produces enriched interaction messages to the main interaction queue.

Worker name: crow-social-processor Domain: No HTTP domain (queue consumer). Health endpoint available via fetch handler.

Architecture

The processor sits between the social collector and the core interaction service, adding AI enrichment to raw social data before it enters the main interaction pipeline.

Social Collector --> SOCIAL_PROCESSING_QUEUE --> Social Processor --> INTERACTION_QUEUE --> Interaction Service

Each queue message contains a batch of collected social items (URLs, titles, content, platform info). The processor:

1. Parses the raw message data into CollectedItem[]
2. Sends the batch to the AI enrichment service
3. Constructs an enriched message with sentiment, topics, entities, tags, and product mentions
4. Forwards the enriched message to the INTERACTION_QUEUE

AI Enrichment

The enrichment service uses Workers AI (@cf/meta/llama-3.1-8b-instruct-fp8) to analyze social content and produce:

Field	Type	Description
sentiment	{ label, score }	positive, negative, neutral, or mixed with confidence score
topics	string[]	Up to 10 extracted topics
entities	{ name, type }[]	Up to 20 entity mentions (brand, product, person, organization, location)
summary	string	2-3 sentence summary of the content batch
confidence	number	Overall enrichment confidence (0.8 for AI results, 0.1 for fallback)

If AI enrichment fails, a fallback result is generated with neutral sentiment and minimal metadata.

Routes

Method	Path	Description
GET	/	Health check
POST	/api/v1/process	Manual processing endpoint (accepts items directly)
GET	/docs	OpenAPI documentation

Manual Process Endpoint

POST /api/v1/process accepts a JSON body with:

{
  "organizationId": "org_123",
  "items": [
    {
      "url": "https://...",
      "title": "Post title",
      "content": "Post content",
      "platform": "twitter",
      "author": "username",
      "publishedAt": "2026-01-01",
      "source": "search"
    }
  ]
}

Queue Configuration

Setting	Value
Queue consumed	crow-social-processing-queue
Max batch size	5
Max batch timeout	30 seconds
Max retries	3
Retry delay	60 seconds

Messages that fail processing are retried. After max retries, they are dead-lettered.

Environment Variables

Variable	Example
ENVIRONMENT	dev
AI_GATEWAY_ID	crow-ai-gateway

Bindings

Binding	Type	Name
AI	Workers AI	LLM inference (@cf/meta/llama-3.1-8b-instruct-fp8)
INTERACTION_QUEUE	Queue (producer)	crow-interaction-queue

Dependencies

• Inbound: social collector (via crow-social-processing-queue)
• Outbound: core interaction service (via crow-interaction-queue)

Key Behaviors

• Queue consumer: Primary workload comes from queue messages, not HTTP requests
• Batch processing: Each queue message can contain multiple social items processed together
• AI fallback: If AI enrichment fails, the processor still forwards the message with a low-confidence fallback
• Tag generation: Automatically tags enriched messages with sentiment label and top 3 topics
• Product extraction: Extracts brand and product entity mentions for product-level analytics

Deployment

cd core-social-processor
npx wrangler deploy          # prod
npx wrangler deploy --env dev  # dev