feat: update README with multi-server architecture and Redis integration details
This commit is contained in:
135
README.md
135
README.md
@@ -6,16 +6,19 @@ A full-stack real-time collaborative application supporting text editing and Kan
|
||||
|
||||
- **Collaborative Text Editor**: Real-time document editing with TipTap
|
||||
- **Collaborative Kanban Boards**: Drag-and-drop task management
|
||||
- **Multi-Server Synchronization**: Redis pub/sub enables horizontal scaling across multiple backend instances
|
||||
- **User Authentication**: OAuth2 login (Google, GitHub)
|
||||
- **Document Sharing**: Share documents with users or via public links
|
||||
- **Offline Support**: IndexedDB persistence for offline editing
|
||||
- **User Presence**: Live cursors and user awareness
|
||||
- **User Presence**: Live cursors and user awareness with cross-server sync
|
||||
- **High Availability**: Automatic fallback mode when Redis is unavailable
|
||||
|
||||
## Tech Stack
|
||||
|
||||
**Frontend:** React 19, TypeScript, Vite, TipTap, Yjs, y-websocket, y-indexeddb
|
||||
**Backend:** Go 1.25, Gin, Gorilla WebSocket, PostgreSQL 16
|
||||
**Backend:** Go 1.25, Gin, Gorilla WebSocket, PostgreSQL 16, Redis 7 (pub/sub)
|
||||
**Infrastructure:** Docker Compose
|
||||
**Logging:** Zap (structured logging)
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -70,10 +73,45 @@ npm run dev
|
||||
│ React + Yjs │ WS+HTTP │ Gin + Hub │ SQL │ Documents │
|
||||
│ TipTap │ │ WebSocket │ │ Users │
|
||||
│ IndexedDB │ │ REST API │ │ Sessions │
|
||||
└─────────────┘ └──────────────┘ └─────────────┘
|
||||
└─────────────┘ └──────┬───────┘ └─────────────┘
|
||||
│
|
||||
│ Pub/Sub
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ Redis │
|
||||
│ │
|
||||
│ Message │
|
||||
│ Bus for │
|
||||
│ Multi- │
|
||||
│ Server │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
### Real-Time Collaboration Flow
|
||||
### Multi-Server Architecture (Horizontal Scaling)
|
||||
```
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Client A │ │ Client B │ │ Client C │
|
||||
└────┬─────┘ └────┬─────┘ └────┬─────┘
|
||||
│ │ │
|
||||
│ WS │ WS │ WS
|
||||
▼ ▼ ▼
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Server 1 │ │ Server 2 │ │ Server 3 │
|
||||
│ :8080 │ │ :8081 │ │ :8082 │
|
||||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
└────────────────┼────────────────┘
|
||||
│
|
||||
Redis Pub/Sub
|
||||
(room:doc-123)
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Publish Subscribe Awareness
|
||||
Updates Updates Caching
|
||||
```
|
||||
|
||||
### Real-Time Collaboration Flow (Single Server)
|
||||
```
|
||||
Client A Backend Hub Client B
|
||||
│ │ │
|
||||
@@ -88,6 +126,23 @@ Client A Backend Hub Client B
|
||||
[Apply CRDT merge] [Relay only] [Apply CRDT merge]
|
||||
```
|
||||
|
||||
### Cross-Server Collaboration Flow (with Redis)
|
||||
```
|
||||
Client A Server 1 Redis Server 2 Client B
|
||||
│ │ │ │ │
|
||||
│──1. Edit───────►│ │ │ │
|
||||
│ │──2. Broadcast │ │ │
|
||||
│ │ to local─────────┼────────────────►│──3. Forward────►│
|
||||
│ │ │ │ to local │
|
||||
│ │──4. Publish to─────► │ │
|
||||
│ │ Redis │ │ │
|
||||
│ │ (room:doc-123) │──5. Distribute─►│ │
|
||||
│ │ │ to subscribers│ │
|
||||
│ │ │ │──6. Broadcast──►│
|
||||
│ │ │ │ │
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
```
|
||||
|
||||
### Authentication Flow
|
||||
```
|
||||
User OAuth Provider Backend Database
|
||||
@@ -111,6 +166,8 @@ realtime-collab/
|
||||
│ │ ├── auth/ # JWT & OAuth middleware
|
||||
│ │ ├── handlers/ # HTTP/WebSocket handlers
|
||||
│ │ ├── hub/ # WebSocket hub & rooms
|
||||
│ │ ├── logger/ # Structured logging (Zap)
|
||||
│ │ ├── messagebus/ # Redis pub/sub abstraction
|
||||
│ │ ├── models/ # Domain models
|
||||
│ │ └── store/ # PostgreSQL data layer
|
||||
│ └── scripts/init.sql # Database schema
|
||||
@@ -152,6 +209,16 @@ JWT_SECRET=your-secret-key-here
|
||||
FRONTEND_URL=http://localhost:5173
|
||||
ALLOWED_ORIGINS=http://localhost:5173,http://localhost:3000
|
||||
|
||||
# Redis Configuration (required for multi-server setup)
|
||||
REDIS_URL=redis://localhost:6379/0
|
||||
|
||||
# Server Instance ID (auto-generated if not set)
|
||||
# Only needed when running multiple instances
|
||||
SERVER_ID=
|
||||
|
||||
# Environment (development, production)
|
||||
ENVIRONMENT=development
|
||||
|
||||
# OAuth (optional)
|
||||
GOOGLE_CLIENT_ID=your-google-client-id
|
||||
GOOGLE_CLIENT_SECRET=your-google-client-secret
|
||||
@@ -191,6 +258,35 @@ docker-compose down -v # Reset database
|
||||
docker-compose up -d # Start fresh
|
||||
```
|
||||
|
||||
### Multi-Server Testing
|
||||
```bash
|
||||
# Terminal 1: Start first server
|
||||
cd backend
|
||||
PORT=8080 SERVER_ID=server-1 REDIS_URL=redis://localhost:6379/0 go run cmd/server/main.go
|
||||
|
||||
# Terminal 2: Start second server
|
||||
cd backend
|
||||
PORT=8081 SERVER_ID=server-2 REDIS_URL=redis://localhost:6379/0 go run cmd/server/main.go
|
||||
|
||||
# Terminal 3: Monitor Redis pub/sub activity
|
||||
redis-cli monitor | grep -E "PUBLISH|HSET"
|
||||
```
|
||||
|
||||
### Redis Commands
|
||||
```bash
|
||||
# Monitor real-time Redis activity
|
||||
redis-cli monitor
|
||||
|
||||
# Check active subscriptions
|
||||
redis-cli PUBSUB CHANNELS
|
||||
|
||||
# View awareness cache for a room
|
||||
redis-cli HGETALL "room:doc-{id}:awareness"
|
||||
|
||||
# Clear Redis (for testing)
|
||||
redis-cli FLUSHDB
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### CRDT-Based Collaboration
|
||||
@@ -205,13 +301,32 @@ docker-compose up -d # Start fresh
|
||||
- All conflict resolution happens client-side via Yjs
|
||||
- Rooms are isolated by document ID
|
||||
|
||||
### Data Flow
|
||||
### Redis Pub/Sub for Multi-Server Sync
|
||||
- **MessageBus Interface**: Abstraction layer for pub/sub operations
|
||||
- **Redis Implementation**: Broadcasts Yjs updates across server instances
|
||||
- **Awareness Caching**: User presence synced via Redis HSET/HGET
|
||||
- **Health Monitoring**: Automatic fallback to local-only mode if Redis fails
|
||||
- **Binary Safety**: Preserves Yjs binary data through Redis string encoding
|
||||
- **Room Isolation**: Each document gets its own Redis channel (`room:doc-{id}:messages`)
|
||||
|
||||
### Data Flow (Multi-Server)
|
||||
1. User edits → Yjs generates binary update
|
||||
2. Update sent via WebSocket to backend
|
||||
3. Backend broadcasts to all clients in room
|
||||
4. Each client's Yjs applies and merges update
|
||||
5. IndexedDB persists state locally
|
||||
6. Periodic backup to PostgreSQL (optional)
|
||||
2. Update sent via WebSocket to backend server (e.g., Server 1)
|
||||
3. **Server 1** broadcasts to:
|
||||
- All local clients in the room (via WebSocket)
|
||||
- Redis pub/sub channel (for cross-server sync)
|
||||
4. **Server 2** receives update from Redis subscription
|
||||
5. **Server 2** forwards to its local clients in the same room
|
||||
6. Each client's Yjs applies and merges update (CRDT magic)
|
||||
7. IndexedDB persists state locally
|
||||
8. Periodic backup to PostgreSQL (optional)
|
||||
|
||||
### Fallback Mode
|
||||
- When Redis is unavailable, servers enter **fallback mode**
|
||||
- Continues to work for single-server deployments
|
||||
- Cross-server sync disabled until Redis recovers
|
||||
- Automatic recovery when Redis comes back online
|
||||
- Health checks run every 10 seconds
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user