pixel/README.md

Pixel - AI Video Creation Platform

中文文档

Pixel is an intelligent platform for creating comics and videos from scripts using AI. It streamlines the workflow from scriptwriting to asset management, storyboard generation, and final video production, leveraging the power of Multi-Agent Systems and advanced generative models.

✨ Features

• Intelligent Script Analysis:
  • Automatically parse scripts to identify characters, scenes, and props using LLMs.
  • Agent-based Workflow: Utilizes specialized agents (powered by AgentScope) for deep script understanding and breakdown.
• Multi-Provider AI Support:
  • LLM: DashScope (Qwen), Google (Gemini), VolcEngine (Doubao).
  • Image Generation: Flux 1.1 Pro, Wanx, Kolors (ModelScope).
  • Video Generation: Kling 1.5, Hailuo (MiniMax), CogVideoX, Wan 2.1.
• Asset Management: Centralized "Material Center" to manage and edit creative assets.
• AI Storyboarding: Generate visual storyboards from script descriptions using AI image generation with style consistency control.
• Video Generation & Editing:
  • Transform static storyboards into dynamic videos.
  • Fine-grained Control: First/Last frame control, Camera Motion (Zoom, Pan, Tilt), and Motion Bucket settings.
• Infinite Canvas: A node-based free creation workspace (powered by React Flow) supporting multi-selection, smooth zooming, and flexible node connections.
• Project Management: Organize your creative works in a structured workspace with support for Episodes and Scenes.

🏗 Architecture

The project is structured as a monorepo with a clean three-layer architecture:

• frontend/: A Next.js 15 (App Router) application using React 19, Tailwind CSS, and @xyflow/react for the canvas interface.
• backend/: A FastAPI service with three-layer architecture (API Layer, Service Layer, Repository Layer). It uses AgentScope for agent orchestration and supports multiple model providers via a plugin system.

📚 Documentation

Comprehensive documentation is available:

• docs/API.md: Complete API reference with examples
• docs/development-guide.md: Development best practices and guidelines
• docs/FRONTEND_OPTIMIZATION.md: Frontend optimization strategies

🚀 Quick Start

Prerequisites

• Node.js: v20 or higher (Required for Next.js 15)
• Python: v3.12 or higher
• Package Manager: pnpm (Frontend), uv (Backend - Recommended)
• Redis: v7+ (Optional but recommended for caching)
• PostgreSQL: v15+ (For production, SQLite for development)
• API Keys: Aliyun DashScope, VolcEngine, or Google AI Studio keys depending on models used

Installation

1. Clone the Repository

git clone <repository-url>
cd pixel

2. Backend Setup

cd backend

# Install uv if you haven't already
pip install uv

# Install dependencies
uv sync

# Configure environment variables
cp .env.example .env
# Edit .env with your API keys

# Run database migrations
alembic upgrade head

# Start the backend server
./start.sh
# Or manually: uv run uvicorn src.main:app --reload --port 8000

The backend will be available at http://localhost:8000

3. Frontend Setup

cd frontend

# Install dependencies
pnpm install

# Start the development server
pnpm dev

The frontend will be available at http://localhost:3000

Environment Configuration

Create a .env file in the backend/ directory:

# AI Providers
DASHSCOPE_API_KEY=your_dashscope_key
VOLCENGINE_ACCESS_KEY=your_volcengine_key
VOLCENGINE_SECRET_KEY=your_volcengine_secret
GOOGLE_API_KEY=your_google_key
KLING_API_KEY=your_kling_key

# Storage
OSS_ACCESS_KEY_ID=your_oss_key
OSS_ACCESS_KEY_SECRET=your_oss_secret

# Database (Optional - defaults to SQLite)
DATABASE_URL=postgresql://user:pass@localhost/pixel

# Redis (Optional but recommended)
REDIS_URL=redis://localhost:6379
REDIS_ENABLED=1

# CORS
# Production: set explicit comma-separated origins
CORS_ALLOWED_ORIGINS=https://your-app.example.com
# Development fallback origins
CORS_DEV_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000

# Monitoring
ENABLE_METRICS=true
LOG_LEVEL=INFO

Docker Deployment (Optional)

# Build and start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

🔌 API Documentation

The backend provides RESTful APIs with comprehensive documentation:

• API Documentation: Complete API reference with examples
• Interactive Docs: http://localhost:8000/docs (Swagger UI)
• ReDoc: http://localhost:8000/redoc
• OpenAPI Spec: http://localhost:8000/openapi.json

Core Endpoints

• Image Generation: POST /api/v1/generations/image
• Video Generation: POST /api/v1/generations/video
• Script Analysis: POST /api/v1/script/analyze
• Task Status: GET /api/v1/tasks/{task_id}
• Project Management: /api/v1/projects/*
• Canvas Operations: /api/v1/canvas/*

Interactive Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc
• OpenAPI Spec: http://localhost:8000/openapi.json

All generation endpoints support an extra_params field to pass model-specific arguments directly to the underlying SDK.

🧪 Development

Running Tests

Backend:

cd backend
pytest                          # Run all tests
pytest tests/test_integration.py  # Run integration tests
pytest --cov=src                # Run with coverage

Frontend:

cd frontend
pnpm test                       # Run all tests
pnpm test:watch                 # Run in watch mode
pnpm test:coverage              # Run with coverage

Code Quality

Backend:

# Format code
black src/
isort src/

# Lint
flake8 src/
mypy src/

Frontend:

# Lint
pnpm lint

# Type check
pnpm type-check

# Format
pnpm format

Database Migrations

cd backend

# Create a new migration
alembic revision --autogenerate -m "description"

# Apply migrations
alembic upgrade head

# Rollback one migration
alembic downgrade -1

API Type Generation

When backend API changes, regenerate frontend types:

cd frontend
pnpm gen:api

📊 Monitoring

Health Check

curl http://localhost:8000/health

Metrics (Prometheus)

curl http://localhost:8000/metrics

Logs

Backend logs are in JSON format for easy parsing:

tail -f backend/logs/app.log | jq

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Follow the three-layer architecture (API, Service, Repository)
• Write tests for new features
• Update documentation
• Follow code style guidelines
• Use conventional commits

🐛 Troubleshooting

Backend Issues

Redis Connection Error:

# Check if Redis is running
redis-cli ping
# Should return: PONG

Database Connection Error:

# Check database connection
psql -h localhost -U user -d pixel

Import Errors:

# Reinstall dependencies
uv sync --reinstall

Frontend Issues

Module Not Found:

# Clear cache and reinstall
rm -rf node_modules .next
pnpm install

API Type Mismatch:

# Regenerate API types
pnpm gen:api

📦 Project Structure

pixel/
├── backend/                 # FastAPI backend
│   ├── src/
│   │   ├── controllers/    # API Layer (HTTP handlers)
│   │   ├── services/       # Service Layer (business logic)
│   │   ├── repositories/   # Repository Layer (data access)
│   │   ├── models/         # Data models (entities & schemas)
│   │   ├── middlewares/    # API middlewares
│   │   └── utils/          # Utilities
│   ├── tests/              # Test suite
│   └── alembic/            # Database migrations
├── frontend/               # Next.js frontend
│   ├── src/
│   │   ├── app/           # Next.js pages (App Router)
│   │   ├── components/    # React components
│   │   ├── lib/           # Utilities and services
│   │   └── store/         # State management (Zustand)
│   └── tests/             # Test suite
├── docs/                   # Documentation
├── ARCHITECTURE.md         # Architecture documentation
└── docker-compose.yml      # Docker configuration

📄 License

MIT