feat: update README with multi-server architecture and Redis integration details

This commit is contained in:
M1ngdaXie
2026-02-05 15:21:08 -08:00
parent 6fac2f7997
commit 3fa623c975

135
README.md
View File

@@ -6,16 +6,19 @@ A full-stack real-time collaborative application supporting text editing and Kan
- **Collaborative Text Editor**: Real-time document editing with TipTap - **Collaborative Text Editor**: Real-time document editing with TipTap
- **Collaborative Kanban Boards**: Drag-and-drop task management - **Collaborative Kanban Boards**: Drag-and-drop task management
- **Multi-Server Synchronization**: Redis pub/sub enables horizontal scaling across multiple backend instances
- **User Authentication**: OAuth2 login (Google, GitHub) - **User Authentication**: OAuth2 login (Google, GitHub)
- **Document Sharing**: Share documents with users or via public links - **Document Sharing**: Share documents with users or via public links
- **Offline Support**: IndexedDB persistence for offline editing - **Offline Support**: IndexedDB persistence for offline editing
- **User Presence**: Live cursors and user awareness - **User Presence**: Live cursors and user awareness with cross-server sync
- **High Availability**: Automatic fallback mode when Redis is unavailable
## Tech Stack ## Tech Stack
**Frontend:** React 19, TypeScript, Vite, TipTap, Yjs, y-websocket, y-indexeddb **Frontend:** React 19, TypeScript, Vite, TipTap, Yjs, y-websocket, y-indexeddb
**Backend:** Go 1.25, Gin, Gorilla WebSocket, PostgreSQL 16 **Backend:** Go 1.25, Gin, Gorilla WebSocket, PostgreSQL 16, Redis 7 (pub/sub)
**Infrastructure:** Docker Compose **Infrastructure:** Docker Compose
**Logging:** Zap (structured logging)
## Quick Start ## Quick Start
@@ -70,10 +73,45 @@ npm run dev
│ React + Yjs │ WS+HTTP │ Gin + Hub │ SQL │ Documents │ │ React + Yjs │ WS+HTTP │ Gin + Hub │ SQL │ Documents │
│ TipTap │ │ WebSocket │ │ Users │ │ TipTap │ │ WebSocket │ │ Users │
│ IndexedDB │ │ REST API │ │ Sessions │ │ IndexedDB │ │ REST API │ │ Sessions │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ Pub/Sub
┌─────────────┐
│ Redis │
│ │
│ Message │
│ Bus for │
│ Multi- │
│ Server │
└─────────────┘
``` ```
### Real-Time Collaboration Flow ### Multi-Server Architecture (Horizontal Scaling)
```
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Client A │ │ Client B │ │ Client C │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
│ WS │ WS │ WS
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Server 1 │ │ Server 2 │ │ Server 3 │
│ :8080 │ │ :8081 │ │ :8082 │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└────────────────┼────────────────┘
Redis Pub/Sub
(room:doc-123)
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Publish Subscribe Awareness
Updates Updates Caching
```
### Real-Time Collaboration Flow (Single Server)
``` ```
Client A Backend Hub Client B Client A Backend Hub Client B
│ │ │ │ │ │
@@ -88,6 +126,23 @@ Client A Backend Hub Client B
[Apply CRDT merge] [Relay only] [Apply CRDT merge] [Apply CRDT merge] [Relay only] [Apply CRDT merge]
``` ```
### Cross-Server Collaboration Flow (with Redis)
```
Client A Server 1 Redis Server 2 Client B
│ │ │ │ │
│──1. Edit───────►│ │ │ │
│ │──2. Broadcast │ │ │
│ │ to local─────────┼────────────────►│──3. Forward────►│
│ │ │ │ to local │
│ │──4. Publish to─────► │ │
│ │ Redis │ │ │
│ │ (room:doc-123) │──5. Distribute─►│ │
│ │ │ to subscribers│ │
│ │ │ │──6. Broadcast──►│
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
```
### Authentication Flow ### Authentication Flow
``` ```
User OAuth Provider Backend Database User OAuth Provider Backend Database
@@ -111,6 +166,8 @@ realtime-collab/
│ │ ├── auth/ # JWT & OAuth middleware │ │ ├── auth/ # JWT & OAuth middleware
│ │ ├── handlers/ # HTTP/WebSocket handlers │ │ ├── handlers/ # HTTP/WebSocket handlers
│ │ ├── hub/ # WebSocket hub & rooms │ │ ├── hub/ # WebSocket hub & rooms
│ │ ├── logger/ # Structured logging (Zap)
│ │ ├── messagebus/ # Redis pub/sub abstraction
│ │ ├── models/ # Domain models │ │ ├── models/ # Domain models
│ │ └── store/ # PostgreSQL data layer │ │ └── store/ # PostgreSQL data layer
│ └── scripts/init.sql # Database schema │ └── scripts/init.sql # Database schema
@@ -152,6 +209,16 @@ JWT_SECRET=your-secret-key-here
FRONTEND_URL=http://localhost:5173 FRONTEND_URL=http://localhost:5173
ALLOWED_ORIGINS=http://localhost:5173,http://localhost:3000 ALLOWED_ORIGINS=http://localhost:5173,http://localhost:3000
# Redis Configuration (required for multi-server setup)
REDIS_URL=redis://localhost:6379/0
# Server Instance ID (auto-generated if not set)
# Only needed when running multiple instances
SERVER_ID=
# Environment (development, production)
ENVIRONMENT=development
# OAuth (optional) # OAuth (optional)
GOOGLE_CLIENT_ID=your-google-client-id GOOGLE_CLIENT_ID=your-google-client-id
GOOGLE_CLIENT_SECRET=your-google-client-secret GOOGLE_CLIENT_SECRET=your-google-client-secret
@@ -191,6 +258,35 @@ docker-compose down -v # Reset database
docker-compose up -d # Start fresh docker-compose up -d # Start fresh
``` ```
### Multi-Server Testing
```bash
# Terminal 1: Start first server
cd backend
PORT=8080 SERVER_ID=server-1 REDIS_URL=redis://localhost:6379/0 go run cmd/server/main.go
# Terminal 2: Start second server
cd backend
PORT=8081 SERVER_ID=server-2 REDIS_URL=redis://localhost:6379/0 go run cmd/server/main.go
# Terminal 3: Monitor Redis pub/sub activity
redis-cli monitor | grep -E "PUBLISH|HSET"
```
### Redis Commands
```bash
# Monitor real-time Redis activity
redis-cli monitor
# Check active subscriptions
redis-cli PUBSUB CHANNELS
# View awareness cache for a room
redis-cli HGETALL "room:doc-{id}:awareness"
# Clear Redis (for testing)
redis-cli FLUSHDB
```
## How It Works ## How It Works
### CRDT-Based Collaboration ### CRDT-Based Collaboration
@@ -205,13 +301,32 @@ docker-compose up -d # Start fresh
- All conflict resolution happens client-side via Yjs - All conflict resolution happens client-side via Yjs
- Rooms are isolated by document ID - Rooms are isolated by document ID
### Data Flow ### Redis Pub/Sub for Multi-Server Sync
- **MessageBus Interface**: Abstraction layer for pub/sub operations
- **Redis Implementation**: Broadcasts Yjs updates across server instances
- **Awareness Caching**: User presence synced via Redis HSET/HGET
- **Health Monitoring**: Automatic fallback to local-only mode if Redis fails
- **Binary Safety**: Preserves Yjs binary data through Redis string encoding
- **Room Isolation**: Each document gets its own Redis channel (`room:doc-{id}:messages`)
### Data Flow (Multi-Server)
1. User edits → Yjs generates binary update 1. User edits → Yjs generates binary update
2. Update sent via WebSocket to backend 2. Update sent via WebSocket to backend server (e.g., Server 1)
3. Backend broadcasts to all clients in room 3. **Server 1** broadcasts to:
4. Each client's Yjs applies and merges update - All local clients in the room (via WebSocket)
5. IndexedDB persists state locally - Redis pub/sub channel (for cross-server sync)
6. Periodic backup to PostgreSQL (optional) 4. **Server 2** receives update from Redis subscription
5. **Server 2** forwards to its local clients in the same room
6. Each client's Yjs applies and merges update (CRDT magic)
7. IndexedDB persists state locally
8. Periodic backup to PostgreSQL (optional)
### Fallback Mode
- When Redis is unavailable, servers enter **fallback mode**
- Continues to work for single-server deployments
- Cross-server sync disabled until Redis recovers
- Automatic recovery when Redis comes back online
- Health checks run every 10 seconds
## Testing ## Testing