lainess
← All projects

Data processing

Type-011

Type-011 collects posts from configured sources, preserves their revision history, and generates vector embeddings. An event-driven pipeline separates collection from inference, with workers running locally or on remote machines.

  • Python
  • FastAPI
  • Pydantic
  • SQLAlchemy
  • Alembic
  • PostgreSQL
  • pgvector
  • RabbitMQ
  • Debezium
  • PyTorch
  • Qwen3
  • MLOps
  • Docker
  • CI/CD
  • GitHub Actions
  • Prometheus
  • Grafana
  • Clean Architecture
  • Distributed Computing

Key features

  • Historical post collection and ongoing synchronization, including edits and deletion tracking.
  • Web interface for managing sources and checking synchronization progress.
  • Batched Qwen3 embedding generation, with vectors linked to post versions and models in pgvector.
  • Idempotent vector storage, delayed retries, failed-task isolation, and reconciliation for missing embeddings.

Architecture

The collector saves each new post version and its event in one PostgreSQL transaction. Debezium forwards committed events to RabbitMQ, where independent workers pick up embedding tasks and store the resulting vectors.

Architecture

Loading diagram…

Posts flow from a source through the collector into PostgreSQL. Debezium forwards events to RabbitMQ; GPU workers generate embeddings and save them in pgvector.

Remote workers

GPU workers run separately from the application and share a task queue, with per-worker device, resource, and batch settings. Remote workers connect to the service through SSH tunnels.

Observability

Prometheus metrics feed two Grafana dashboards:

  • Collection: source activity, stored posts and revisions, incoming events, processing results, and latency.
  • RabbitMQ operations: queue backlog, task throughput and age, active consumers, and redeliveries.