523 lines
16 KiB
Markdown
523 lines
16 KiB
Markdown
# DocEngine — Document Template Recognition & Reconstruction System
|
|
|
|
A production-ready system for scanning documents, detecting layouts, extracting content, generating reusable templates, matching future uploads against stored templates, and reconstructing original layouts as PDF output.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌───────────────┐ ┌───────────────┐ ┌──────────────┐
|
|
│ FastAPI App │─────▶│ Celery │─────▶│ Redis │
|
|
│ (Port 7989) │ │ Worker(s) │ │ (Broker) │
|
|
└───────┬───────┘ └───────┬───────┘ └──────────────┘
|
|
│ │
|
|
▼ ▼
|
|
┌───────────────────────────────────────┐
|
|
│ PostgreSQL (Schema: admin) │
|
|
│ 192.168.0.111:7925 │
|
|
└───────────────────────────────────────┘
|
|
```
|
|
|
|
**Stack**: Python 3.12, FastAPI, SQLAlchemy 2.x, Pydantic V2, Celery, Redis, PaddleOCR, PyMuPDF, OpenCV, ReportLab, PostgreSQL 16.
|
|
|
|
## Features
|
|
|
|
| Capability | Implementation |
|
|
|----------------------------------|--------------------------------------------|
|
|
| Scanned image OCR | PaddleOCR (CPU/GPU) |
|
|
| Native PDF text extraction | PyMuPDF (fitz) |
|
|
| Layout detection | OpenCV + LayoutParser |
|
|
| Table extraction | Camelot-py + OpenCV contour detection |
|
|
| Header/footer detection | Positional heuristics |
|
|
| Watermark detection | Transparency + large-font analysis |
|
|
| Font info extraction | PyMuPDF text dict parsing |
|
|
| Template generation & storage | PostgreSQL (admin schema) |
|
|
| Template fingerprinting | SHA-256 structural hashing |
|
|
| Template matching | Multi-signal similarity scoring |
|
|
| PDF reconstruction | ReportLab from template definitions |
|
|
| Async processing | Celery + Redis |
|
|
| Authentication | JWT (access + refresh tokens, bcrypt) |
|
|
| Monitoring | Prometheus + structlog JSON logging |
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
docengine/
|
|
├── app/
|
|
│ ├── main.py # FastAPI application entry
|
|
│ ├── api/
|
|
│ │ ├── router.py # Top-level API router
|
|
│ │ └── v1/
|
|
│ │ ├── auth.py # Auth endpoints
|
|
│ │ ├── documents.py # Document endpoints
|
|
│ │ ├── health.py # Health check
|
|
│ │ └── templates.py # Template endpoints
|
|
│ ├── core/
|
|
│ │ ├── config.py # Pydantic Settings
|
|
│ │ ├── database.py # SQLAlchemy engine & session
|
|
│ │ ├── dependencies.py # FastAPI DI
|
|
│ │ ├── exceptions.py # Custom exception hierarchy
|
|
│ │ ├── logging_config.py # structlog configuration
|
|
│ │ └── security.py # JWT & bcrypt helpers
|
|
│ ├── models/ # SQLAlchemy ORM models
|
|
│ ├── schemas/ # Pydantic request/response schemas
|
|
│ ├── repositories/ # Data access layer
|
|
│ ├── services/ # Business logic
|
|
│ │ ├── document_service.py # Orchestration pipeline
|
|
│ │ ├── ocr_service.py # PaddleOCR integration
|
|
│ │ ├── pdf_service.py # PyMuPDF native PDF parsing
|
|
│ │ ├── layout_service.py # OpenCV layout detection
|
|
│ │ ├── template_service.py # Template generation
|
|
│ │ ├── fingerprint_service.py
|
|
│ │ ├── matching_service.py
|
|
│ │ └── reconstruction_service.py
|
|
│ ├── middleware/ # CORS, audit, metrics, rate limit
|
|
│ ├── storage/ # File storage abstraction
|
|
│ ├── tasks/ # Celery async tasks
|
|
│ ├── workers/ # Celery app configuration
|
|
│ └── events/ # App lifecycle handlers
|
|
├── alembic/ # Database migrations
|
|
├── sql/ # Raw SQL scripts
|
|
├── tests/ # Test suite
|
|
├── docker-compose.yml # Dev stack
|
|
├── docker-compose.prod.yml # Production stack
|
|
├── Dockerfile # Multi-stage build
|
|
├── requirements.txt
|
|
└── .env
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
- Python 3.12+
|
|
- PostgreSQL 16 (running at `192.168.0.111:7925`)
|
|
- Redis (for Celery)
|
|
- `poppler-utils` and `ghostscript` (for pdf2image/camelot)
|
|
|
|
### Local Setup
|
|
|
|
```bash
|
|
# Clone & enter
|
|
cd docengine
|
|
|
|
# Create virtual environment
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Create storage directories
|
|
mkdir -p storage/{documents,templates,images,temp,rendered}
|
|
|
|
# Run database migrations
|
|
alembic upgrade head
|
|
|
|
# (Optional) Seed default data
|
|
psql -h 192.168.0.111 -p 7925 -U postgres -d document_engine -f sql/003_seed_data.sql
|
|
psql -h 192.168.0.111 -p 7925 -U postgres -d document_engine -f sql/004_indexes.sql
|
|
|
|
# Start the application
|
|
python -m app.main
|
|
```
|
|
|
|
The API is now available at `http://localhost:7989`. Interactive docs at `http://localhost:7989/docs`.
|
|
|
|
### Start Celery Worker (separate terminal)
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
celery -A app.workers.celery_app worker --loglevel=info --concurrency=4
|
|
```
|
|
|
|
### Docker Setup
|
|
|
|
```bash
|
|
# Build and start all services (app + worker + db + redis)
|
|
docker compose up --build -d
|
|
|
|
# Run migrations inside the container
|
|
docker compose exec app alembic upgrade head
|
|
|
|
# Seed data
|
|
docker compose exec app bash -c "psql -h db -U postgres -d document_engine -f sql/003_seed_data.sql"
|
|
```
|
|
|
|
---
|
|
|
|
## API Reference
|
|
|
|
Base URL: `http://localhost:7989/api/v1`
|
|
|
|
### Health
|
|
|
|
```bash
|
|
curl http://localhost:7989/api/v1/health
|
|
```
|
|
|
|
### Authentication
|
|
|
|
#### Register
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/auth/register \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"username": "john",
|
|
"email": "john@example.com",
|
|
"password": "SecurePass123!",
|
|
"full_name": "John Doe"
|
|
}'
|
|
```
|
|
|
|
#### Login
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/auth/login \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"username": "john",
|
|
"password": "SecurePass123!"
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"access_token": "eyJhbGciOiJIUzI1NiIs...",
|
|
"refresh_token": "eyJhbGciOiJIUzI1NiIs...",
|
|
"token_type": "bearer",
|
|
"expires_in": 1800
|
|
}
|
|
```
|
|
|
|
#### Get Current User
|
|
|
|
```bash
|
|
curl http://localhost:7989/api/v1/auth/me \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Refresh Token
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/auth/refresh \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"refresh_token": "<refresh_token>"}'
|
|
```
|
|
|
|
#### Change Password
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/auth/change-password \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"current_password": "SecurePass123!",
|
|
"new_password": "NewSecurePass456!"
|
|
}'
|
|
```
|
|
|
|
#### Logout
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/auth/logout \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"refresh_token": "<refresh_token>"}'
|
|
```
|
|
|
|
### Documents
|
|
|
|
#### Upload Document
|
|
|
|
```bash
|
|
# Upload a PDF
|
|
curl -X POST http://localhost:7989/api/v1/documents/upload \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-F "file=@/path/to/document.pdf"
|
|
|
|
# Upload a scanned image
|
|
curl -X POST http://localhost:7989/api/v1/documents/upload \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-F "file=@/path/to/scan.jpg"
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000",
|
|
"filename": "abc123_document.pdf",
|
|
"original_filename": "document.pdf",
|
|
"content_type": "application/pdf",
|
|
"file_size": 245760,
|
|
"checksum": "e3b0c44298fc1c149afbf4c8996fb924...",
|
|
"status": "pending",
|
|
"created_at": "2026-06-01T12:00:00Z"
|
|
}
|
|
```
|
|
|
|
#### Get Document
|
|
|
|
```bash
|
|
curl http://localhost:7989/api/v1/documents/<document_id> \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### List Documents
|
|
|
|
```bash
|
|
# With pagination
|
|
curl "http://localhost:7989/api/v1/documents?page=1&page_size=20" \
|
|
-H "Authorization: Bearer <access_token>"
|
|
|
|
# Filter by status
|
|
curl "http://localhost:7989/api/v1/documents?status=completed" \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Delete Document
|
|
|
|
```bash
|
|
curl -X DELETE http://localhost:7989/api/v1/documents/<document_id> \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Get Template Matches for Document
|
|
|
|
```bash
|
|
curl http://localhost:7989/api/v1/documents/<document_id>/template \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
### Templates
|
|
|
|
#### List Templates
|
|
|
|
```bash
|
|
curl "http://localhost:7989/api/v1/templates?page=1&page_size=20" \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Get Template
|
|
|
|
```bash
|
|
curl http://localhost:7989/api/v1/templates/<template_id> \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Delete (Deactivate) Template
|
|
|
|
```bash
|
|
curl -X DELETE http://localhost:7989/api/v1/templates/<template_id> \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
#### Match Document to Templates
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/templates/match \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"document_id": "<document_id>",
|
|
"min_confidence": 0.5,
|
|
"max_results": 5
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"id": "...",
|
|
"document_id": "...",
|
|
"format_id": "...",
|
|
"confidence_score": 0.92,
|
|
"match_details": { "dimension_score": 1.0, "header_score": 0.85 },
|
|
"selected": true,
|
|
"template_name": "Invoice Template v1",
|
|
"created_at": "2026-06-01T12:00:00Z"
|
|
}
|
|
]
|
|
```
|
|
|
|
#### Render Template to PDF
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7989/api/v1/templates/render \
|
|
-H "Authorization: Bearer <access_token>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"template_id": "<template_id>",
|
|
"data": {
|
|
"company_name": "Acme Corp",
|
|
"invoice_number": "INV-2026-001",
|
|
"date": "2026-06-01",
|
|
"total": "$1,250.00"
|
|
},
|
|
"output_filename": "invoice_output.pdf"
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"output_path": "rendered/invoice_output.pdf",
|
|
"filename": "invoice_output.pdf",
|
|
"file_size": 32768,
|
|
"page_count": 1,
|
|
"rendered_at": "2026-06-01T12:05:00Z"
|
|
}
|
|
```
|
|
|
|
#### Download Rendered PDF
|
|
|
|
```bash
|
|
curl -O http://localhost:7989/api/v1/templates/<template_id>/download?filename=invoice_output.pdf \
|
|
-H "Authorization: Bearer <access_token>"
|
|
```
|
|
|
|
---
|
|
|
|
## Processing Pipeline
|
|
|
|
When a document is uploaded, the following Celery task pipeline executes asynchronously:
|
|
|
|
1. **File Type Detection** — Determine if the document is a native PDF or scanned image.
|
|
2. **Page Extraction** — Convert PDF pages to images (for scanned docs) or parse directly (for native PDFs).
|
|
3. **OCR** — Run PaddleOCR on scanned pages to extract text blocks with coordinates, confidence, and bounding boxes.
|
|
4. **Native PDF Parsing** — Use PyMuPDF to extract text, fonts, images, and tables from native PDFs.
|
|
5. **Layout Analysis** — Detect headers, footers, watermarks, tables, and image regions using OpenCV heuristics.
|
|
6. **Template Generation** — Build a reusable template definition from the detected layout, stored in PostgreSQL.
|
|
7. **Fingerprint Generation** — Compute a structural fingerprint (SHA-256) for future matching.
|
|
8. **Status Update** — Mark the document as `completed` (or `failed` with error details).
|
|
|
|
---
|
|
|
|
## Database
|
|
|
|
**Connection**: `postgresql://postgres:***@192.168.0.111:7925/document_engine`
|
|
**Schema**: `admin`
|
|
|
|
### Migrations
|
|
|
|
```bash
|
|
# Create a new migration
|
|
alembic revision --autogenerate -m "description"
|
|
|
|
# Apply migrations
|
|
alembic upgrade head
|
|
|
|
# Rollback one step
|
|
alembic downgrade -1
|
|
```
|
|
|
|
### Tables
|
|
|
|
| Table | Purpose |
|
|
|--------------------------|----------------------------------------------|
|
|
| `users` | User accounts |
|
|
| `roles` | Role definitions (admin, user, viewer) |
|
|
| `user_roles` | User-role mapping (M2M) |
|
|
| `refresh_tokens` | JWT refresh token storage |
|
|
| `audit_logs` | Action audit trail |
|
|
| `documents` | Uploaded document records |
|
|
| `document_pages` | Per-page data (dimensions, images) |
|
|
| `document_text_blocks` | Extracted text with position & font info |
|
|
| `document_images` | Extracted images with position |
|
|
| `document_tables` | Extracted tables with cell data (JSONB) |
|
|
| `document_formats` | Template definitions |
|
|
| `document_cells` | Template cell layout definitions |
|
|
| `document_regions` | Template region definitions |
|
|
| `table_formats` | Template table structure definitions |
|
|
| `table_columns` | Template table column definitions |
|
|
| `table_rows` | Template table row definitions |
|
|
| `watermarks` | Template watermark definitions |
|
|
| `image_regions` | Template image region definitions |
|
|
| `template_fingerprints` | Structural fingerprints for matching |
|
|
| `template_matches` | Document-to-template match results |
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Install dev dependencies
|
|
pip install -r requirements-dev.txt
|
|
|
|
# Run all tests
|
|
pytest
|
|
|
|
# Run with coverage
|
|
pytest --cov=app --cov-report=term-missing
|
|
|
|
# Run specific test categories
|
|
pytest tests/unit/
|
|
pytest tests/api/
|
|
pytest tests/repositories/
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
All configuration is via environment variables (`.env` file). Key settings:
|
|
|
|
| Variable | Default | Description |
|
|
|------------------------------------|------------------------|---------------------------------|
|
|
| `APP_PORT` | `7989` | Application port |
|
|
| `DB_HOST` | `192.168.0.111` | PostgreSQL host |
|
|
| `DB_PORT` | `7925` | PostgreSQL port |
|
|
| `DB_NAME` | `document_engine` | Database name |
|
|
| `DB_SCHEMA` | `admin` | PostgreSQL schema |
|
|
| `REDIS_HOST` | `localhost` | Redis host |
|
|
| `CELERY_BROKER_URL` | `redis://localhost:6379/0` | Celery broker |
|
|
| `JWT_SECRET_KEY` | *(see .env)* | JWT signing key |
|
|
| `JWT_ACCESS_TOKEN_EXPIRE_MINUTES` | `30` | Access token TTL |
|
|
| `STORAGE_LOCAL_PATH` | `./storage` | Local file storage path |
|
|
| `STORAGE_MAX_FILE_SIZE_MB` | `100` | Max upload size |
|
|
| `OCR_LANGUAGE` | `en` | PaddleOCR language |
|
|
| `OCR_USE_GPU` | `false` | Enable GPU for OCR |
|
|
|
|
---
|
|
|
|
## Production Deployment
|
|
|
|
```bash
|
|
# Using production compose file
|
|
docker compose -f docker-compose.prod.yml up --build -d
|
|
|
|
# Scale workers
|
|
docker compose -f docker-compose.prod.yml up --scale worker=4 -d
|
|
```
|
|
|
|
Production compose includes:
|
|
- Resource limits (CPU/memory)
|
|
- Redis authentication
|
|
- App replicas
|
|
- Persistent named volumes
|
|
- Auto-restart policies
|
|
|
|
---
|
|
|
|
## Default Credentials
|
|
|
|
| Username | Password | Role |
|
|
|----------|---------------|-------|
|
|
| `admin` | `Admin@123!` | admin |
|
|
|
|
> ⚠️ **Change the default admin password immediately in production.**
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
Proprietary — All rights reserved.
|