Compare commits
4 Commits
ai_backed_
...
c17bb57a3c
| Author | SHA1 | Date | |
|---|---|---|---|
| c17bb57a3c | |||
| 0bcd7821f2 | |||
| 71d8f0fab5 | |||
| c76d533783 |
13
.env.prod
Normal file
13
.env.prod
Normal file
@@ -0,0 +1,13 @@
|
||||
# Database Configuration
|
||||
# Connecting to existing 'postgres-db' container in 'arbit-app_arbit-network'
|
||||
DB_USER=postgres
|
||||
DB_PASSWORD='M@tr!x#149@dm!N'
|
||||
DB_NAME=ocr
|
||||
DB_HOST=postgres-db
|
||||
DB_PORT=5432
|
||||
|
||||
# Mail Configuration (IMAP)
|
||||
MAIL_SERVER=imap.gmail.com
|
||||
MAIL_USERNAME=matrixinfotech.it@gmail.com
|
||||
MAIL_PASSWORD=qtxsthbxbisgcmqu
|
||||
MAIL_PORT=993
|
||||
3
.vscode/settings.json
vendored
3
.vscode/settings.json
vendored
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"dotrush.roslyn.projectOrSolutionFiles": []
|
||||
}
|
||||
95
DEPLOYMENT.md
Normal file
95
DEPLOYMENT.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# OCR Application Deployment Guide
|
||||
|
||||
This guide describes how to deploy the OCR application on a Linux host (e.g., Ubuntu/Debian).
|
||||
|
||||
## Architecture
|
||||
- **Backend**: Containerized (FastAPI, Python 3.9).
|
||||
- **Database**: Containerized (PostgreSQL 15).
|
||||
- **Frontend**: Static files (Angular) served by Host Nginx.
|
||||
- **Reverse Proxy**: Host Nginx proxies requests to Frontend (Static) and Backend (API).
|
||||
|
||||
## Prerequisites
|
||||
- **Deployment Host**: Docker & Docker Compose, Nginx.
|
||||
- **Build Machine**: Node.js & NPM (to run the package script).
|
||||
|
||||
---
|
||||
|
||||
## 1. Full Stack Deployment (Docker)
|
||||
|
||||
The app connects to your **existing Postgres container** (`postgres-db`) in the `arbit-app_arbit-network`.
|
||||
|
||||
1. **Create the Database**:
|
||||
Since we are using an existing postgres instance, we must manually create the `ocr_db`.
|
||||
```bash
|
||||
# Run this on your host to create the DB inside the existing container
|
||||
docker exec -it postgres-db psql -U postgres -c "CREATE DATABASE ocr_db;"
|
||||
```
|
||||
|
||||
2. **Navigate to the packaged directory**:
|
||||
```bash
|
||||
cd ocr_build
|
||||
```
|
||||
|
||||
3. **Verify Environment**:
|
||||
Ensure `.env.prod` exists and points to `DB_HOST=postgres-db`.
|
||||
**Also configure your Email credentials** in `.env.prod` if you want the Mailbox feature to work (Gmail requires an App Password).
|
||||
```bash
|
||||
cat .env.prod
|
||||
```
|
||||
|
||||
4. **Start App Containers**:
|
||||
This will start `ocr_backend` and `ocr_frontend`.
|
||||
```bash
|
||||
# Build and start in detached mode
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
5. **Verify Status**:
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
You should see `ocr_backend` and `ocr_frontend` running.
|
||||
|
||||
---
|
||||
|
||||
## 2. Nginx Configuration (Host Reverse Proxy)
|
||||
|
||||
Since the Frontend is now running in a container on port 8080 (serving `/ocrf/`), we configure the Host Nginx to proxy traffic to it.
|
||||
|
||||
1. **Create Config File**:
|
||||
Copy the provided config to `/etc/nginx/sites-available/ocr`.
|
||||
```bash
|
||||
sudo cp nginx_host.conf /etc/nginx/sites-available/ocr
|
||||
```
|
||||
|
||||
2. **Enable Site**:
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/ocr /etc/nginx/sites-enabled/
|
||||
```
|
||||
|
||||
3. **Test & Reload**:
|
||||
```bash
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Verification
|
||||
|
||||
- **Url**: `http://app.technobeesolutions.in/ocrf/` (Proxies to Frontend Container)
|
||||
- **API**: `http://app.technobeesolutions.in/ocrb/` (Proxies to Backend Container)
|
||||
|
||||
---
|
||||
|
||||
## 4. Troubleshooting
|
||||
|
||||
- **Logs**:
|
||||
```bash
|
||||
docker-compose logs -f backend
|
||||
```
|
||||
- **Database**:
|
||||
Connect via the existing container:
|
||||
```bash
|
||||
docker exec -it postgres-db psql -U postgres -d ocr_db
|
||||
```
|
||||
30
backend/Dockerfile
Normal file
30
backend/Dockerfile
Normal file
@@ -0,0 +1,30 @@
|
||||
# Use official lightweight Python image
|
||||
FROM python:3.9-slim
|
||||
|
||||
# Install system dependencies
|
||||
# tesseract-ocr: for pytesseract
|
||||
# poppler-utils: for pdf2image
|
||||
# libtesseract-dev: development headers
|
||||
RUN apt-get update && apt-get install -y \
|
||||
tesseract-ocr \
|
||||
poppler-utils \
|
||||
libtesseract-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy requirements first to leverage Docker cache
|
||||
COPY requirements.txt .
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy the rest of the application code
|
||||
COPY . .
|
||||
|
||||
# Expose port (default for Uvicorn)
|
||||
EXPOSE 8000
|
||||
|
||||
# Run the application
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -10,10 +10,10 @@ from dotenv import load_dotenv
|
||||
load_dotenv()
|
||||
|
||||
DB_USER = os.getenv("DB_USER", "postgres")
|
||||
DB_PASSWORD = os.getenv("DB_PASSWORD", "M@tr!x#149@dm!N")
|
||||
DB_HOST = os.getenv("DB_HOST", "192.168.0.111")
|
||||
DB_PORT = os.getenv("DB_PORT", "7925")
|
||||
DB_NAME = os.getenv("DB_NAME", "ocr")
|
||||
DB_PASSWORD = os.getenv("DB_PASSWORD", "password")
|
||||
DB_HOST = os.getenv("DB_HOST", "localhost")
|
||||
DB_PORT = os.getenv("DB_PORT", "5432")
|
||||
DB_NAME = os.getenv("DB_NAME", "ocr_db")
|
||||
|
||||
encoded_user = urllib.parse.quote_plus(DB_USER)
|
||||
encoded_password = urllib.parse.quote_plus(DB_PASSWORD)
|
||||
|
||||
@@ -1,116 +0,0 @@
|
||||
import ollama
|
||||
import json
|
||||
import base64
|
||||
|
||||
INVOICE_SCHEMA = {
|
||||
"document_type": None,
|
||||
"invoice_number": None,
|
||||
"invoice_date": None,
|
||||
"due_date": None,
|
||||
"purchase_order_number": None,
|
||||
"vendor": {
|
||||
"name": None,
|
||||
"address": None,
|
||||
"email": None,
|
||||
"phone": None,
|
||||
"gstin": None,
|
||||
"tax_id": None,
|
||||
"website": None
|
||||
},
|
||||
"customer": {
|
||||
"name": None,
|
||||
"address": None,
|
||||
"gstin": None
|
||||
},
|
||||
"amounts": {
|
||||
"subtotal": None,
|
||||
"tax": None,
|
||||
"discount": None,
|
||||
"shipping": None,
|
||||
"round_off": None,
|
||||
"total": None,
|
||||
"amount_paid": None,
|
||||
"balance_due": None,
|
||||
"currency": None
|
||||
},
|
||||
"tax_breakdown": [
|
||||
{
|
||||
"type": None,
|
||||
"rate": None,
|
||||
"amount": None
|
||||
}
|
||||
],
|
||||
"line_items": [
|
||||
{
|
||||
"line_no": None,
|
||||
"description": None,
|
||||
"product_code": None,
|
||||
"hsn_sac": None,
|
||||
"quantity": None,
|
||||
"unit": None,
|
||||
"unit_price": None,
|
||||
"discount": None,
|
||||
"tax_rate": None,
|
||||
"tax_amount": None,
|
||||
"total": None
|
||||
}
|
||||
],
|
||||
"payment_information": {
|
||||
"bank_name": None,
|
||||
"account_number": None,
|
||||
"ifsc": None,
|
||||
"upi_id": None
|
||||
},
|
||||
"metadata": {
|
||||
"pages": None,
|
||||
"ocr_confidence": None,
|
||||
"language": None
|
||||
}
|
||||
}
|
||||
|
||||
def extract_data(text: str = None, image_path: str = None, model_type: str = "text") -> dict:
|
||||
"""
|
||||
Extracts structured data using either Text (Gemma) or Vision (Qwen) models.
|
||||
"""
|
||||
|
||||
prompt = f"""
|
||||
You are an expert data extraction assistant.
|
||||
Extract every possible detail from the provided document and return it strictly as a SINGLE VALID JSON OBJECT matching the following schema structure:
|
||||
|
||||
{json.dumps(INVOICE_SCHEMA, indent=4)}
|
||||
|
||||
IMPORTANT:
|
||||
- Return ONLY the JSON. No markdown formatting, no explanations, no prefix.
|
||||
- If a field is not found or data is not available, use null.
|
||||
"""
|
||||
|
||||
messages = [{'role': 'user', 'content': prompt}]
|
||||
model = 'gemma:2b'
|
||||
|
||||
if model_type == 'vision':
|
||||
if not image_path:
|
||||
return {"error": "Image path required for vision mode"}
|
||||
|
||||
# Qwen-VL handles images passed in the message
|
||||
model = 'qwen2.5vl:7b' # Using the installed model ID
|
||||
messages[0]['images'] = [image_path]
|
||||
messages[0]['content'] = "Analyze this image. " + prompt
|
||||
else:
|
||||
# Text Mode
|
||||
if not text:
|
||||
return {"error": "Text required for text mode"}
|
||||
messages[0]['content'] += f"\n\n---\n{text}\n---"
|
||||
|
||||
try:
|
||||
response = ollama.chat(model=model, messages=messages)
|
||||
content = response['message']['content']
|
||||
|
||||
# Clean up markdown
|
||||
content = content.replace("```json", "").replace("```", "").strip()
|
||||
|
||||
return json.loads(content)
|
||||
|
||||
except Exception as e:
|
||||
print(f"LLM Extraction Error ({model_type}): {e}")
|
||||
return {"error": str(e), "raw_output": content if 'content' in locals() else ""}
|
||||
|
||||
105
backend/main.py
105
backend/main.py
@@ -44,7 +44,7 @@ def extract_text_from_pdf(file_bytes: bytes) -> str:
|
||||
return ""
|
||||
|
||||
# Internal modules
|
||||
from database import get_db, Email, Vendor, Document
|
||||
from database import get_db, Email
|
||||
from scheduler import start_scheduler, stop_scheduler
|
||||
from mail_service import fetch_and_store_emails
|
||||
|
||||
@@ -85,9 +85,15 @@ class LoginResponse(BaseModel):
|
||||
token: str
|
||||
message: str
|
||||
|
||||
@app.post("/api/login", response_model=LoginResponse)
|
||||
def login(request: LoginRequest):
|
||||
if request.username == "admin" and request.password == "admin":
|
||||
return LoginResponse(token="fake-super-secret-token", message="Success")
|
||||
raise HTTPException(status_code=401, detail="Invalid username or password")
|
||||
|
||||
class NERResponse(BaseModel):
|
||||
text: str
|
||||
file_path: str
|
||||
|
||||
|
||||
|
||||
def extract_text_from_image(file_bytes: bytes) -> str:
|
||||
@@ -104,41 +110,13 @@ async def extract_text(file: UploadFile = File(...)):
|
||||
content = await file.read()
|
||||
filename = file.filename.lower()
|
||||
|
||||
# Save file for Vision mode
|
||||
file_path = f"uploads/{file.filename}"
|
||||
with open(file_path, "wb") as f:
|
||||
f.write(content)
|
||||
|
||||
extracted_text = ""
|
||||
|
||||
if filename.endswith(".pdf"):
|
||||
# Try text extraction first
|
||||
with pdfplumber.open(io.BytesIO(content)) as pdf:
|
||||
try:
|
||||
text = ""
|
||||
for page in pdf.pages:
|
||||
page_text = page.extract_text(layout=True)
|
||||
if page_text:
|
||||
text += page_text + "\n"
|
||||
if text.strip():
|
||||
extracted_text = text.strip()
|
||||
except Exception:
|
||||
pass
|
||||
extracted_text = extract_text_from_pdf(content)
|
||||
|
||||
if not extracted_text:
|
||||
try:
|
||||
# Fallback to pypdf
|
||||
reader = PdfReader(io.BytesIO(content))
|
||||
text = ""
|
||||
for page in reader.pages:
|
||||
page_text = page.extract_text()
|
||||
if page_text:
|
||||
text += page_text + "\n"
|
||||
extracted_text = text.strip()
|
||||
except:
|
||||
pass
|
||||
|
||||
# If text is still empty, it might be a scanned PDF.
|
||||
# If text is empty, it might be a scanned PDF.
|
||||
if not extracted_text.strip():
|
||||
try:
|
||||
images = convert_from_bytes(content)
|
||||
@@ -153,35 +131,7 @@ async def extract_text(file: UploadFile = File(...)):
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Unsupported file type")
|
||||
|
||||
return NERResponse(text=extracted_text, file_path=file_path)
|
||||
|
||||
# 3. AI Extraction Module
|
||||
from llm_service import extract_data
|
||||
from pdf2image import convert_from_path
|
||||
|
||||
class AITextRequest(BaseModel):
|
||||
text: Optional[str] = None
|
||||
file_path: Optional[str] = None
|
||||
model_type: str = "text"
|
||||
|
||||
@app.post("/api/extract/ai")
|
||||
def extract_ai_data(request: AITextRequest):
|
||||
final_image_path = request.file_path
|
||||
|
||||
if request.model_type == "vision" and request.file_path and request.file_path.endswith(".pdf"):
|
||||
# Convert PDF first page to image
|
||||
try:
|
||||
images = convert_from_path(request.file_path)
|
||||
if images:
|
||||
# Save temp image
|
||||
temp_img_path = request.file_path + ".jpg"
|
||||
images[0].save(temp_img_path, "JPEG")
|
||||
final_image_path = temp_img_path
|
||||
except Exception as e:
|
||||
print(f"Error converting PDF for vision: {e}")
|
||||
|
||||
data = extract_data(text=request.text, image_path=final_image_path, model_type=request.model_type)
|
||||
return data
|
||||
return NERResponse(text=extracted_text)
|
||||
|
||||
import zipfile
|
||||
import mimetypes
|
||||
@@ -311,36 +261,3 @@ def sync_emails():
|
||||
@app.get("/")
|
||||
def read_root():
|
||||
return {"message": "OCR Backend API is running"}
|
||||
|
||||
class DocumentSaveRequest(BaseModel):
|
||||
vendor_name: str
|
||||
file_path: str
|
||||
model_type: str
|
||||
data: dict
|
||||
|
||||
@app.post("/api/documents/save")
|
||||
def save_document(request: DocumentSaveRequest, db: Session = Depends(get_db)):
|
||||
# 1. Find or Create Vendor
|
||||
vendor = db.query(Vendor).filter(Vendor.name == request.vendor_name).first()
|
||||
if not vendor:
|
||||
vendor = Vendor(name=request.vendor_name, default_model=request.model_type)
|
||||
db.add(vendor)
|
||||
db.commit()
|
||||
db.refresh(vendor)
|
||||
else:
|
||||
# Update preference
|
||||
vendor.default_model = request.model_type
|
||||
db.commit()
|
||||
|
||||
# 2. Save Document
|
||||
filename = request.file_path.split('/')[-1]
|
||||
doc = Document(
|
||||
vendor_id=vendor.id,
|
||||
filename=filename,
|
||||
status="verified",
|
||||
processed_data=request.data
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
|
||||
return {"message": "Document saved and Vendor preference updated", "vendor_id": vendor.id}
|
||||
|
||||
@@ -10,4 +10,4 @@ imap-tools
|
||||
apscheduler
|
||||
python-dotenv
|
||||
pdfplumber
|
||||
ollama
|
||||
pdf2image
|
||||
|
||||
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 228 KiB |
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 224 KiB |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 112 KiB |
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 116 KiB |
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 116 KiB |
22
deployment/nginx.conf
Normal file
22
deployment/nginx.conf
Normal file
@@ -0,0 +1,22 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name app.technobeesolutions.in;
|
||||
|
||||
# Proxy Angular Frontend Container
|
||||
location /ocrf/ {
|
||||
proxy_pass http://localhost:8080/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Proxy API requests to the Docker Backend
|
||||
location /ocrb/ {
|
||||
proxy_pass http://localhost:8000/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
BIN
docengine/.DS_Store
vendored
Normal file
BIN
docengine/.DS_Store
vendored
Normal file
Binary file not shown.
@@ -1,26 +0,0 @@
|
||||
__pycache__
|
||||
*.pyc
|
||||
*.pyo
|
||||
.Python
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
*.egg-info
|
||||
dist/
|
||||
build/
|
||||
.git
|
||||
.gitignore
|
||||
.dockerignore
|
||||
*.md
|
||||
*.rst
|
||||
docs/
|
||||
tests/
|
||||
htmlcov/
|
||||
.coverage
|
||||
.pytest_cache
|
||||
.mypy_cache
|
||||
.ruff_cache
|
||||
*.log
|
||||
.idea/
|
||||
.vscode/
|
||||
@@ -8,28 +8,28 @@ APP_PORT=7989
|
||||
APP_WORKERS=4
|
||||
|
||||
# Database
|
||||
DB_HOST=localhost
|
||||
DB_HOST=103.125.129.116
|
||||
DB_PORT=5432
|
||||
DB_NAME=document_engine
|
||||
DB_NAME=ocr
|
||||
DB_USER=postgres
|
||||
DB_PASSWORD=changeme
|
||||
DB_PASSWORD=M@triXPostgr3s@6202
|
||||
DB_SCHEMA=admin
|
||||
DB_POOL_SIZE=20
|
||||
DB_MAX_OVERFLOW=10
|
||||
DB_ECHO=false
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_HOST=103.125.129.116
|
||||
REDIS_PORT=7901
|
||||
REDIS_DB=0
|
||||
REDIS_PASSWORD=
|
||||
REDIS_PASSWORD=M@triXR3d1s@6202
|
||||
|
||||
# Celery
|
||||
CELERY_BROKER_URL=redis://localhost:6379/0
|
||||
CELERY_RESULT_BACKEND=redis://localhost:6379/1
|
||||
CELERY_BROKER_URL=redis://:M%40triXR3d1s%406202@103.125.129.116:7901/0
|
||||
CELERY_RESULT_BACKEND=redis://:M%40triXR3d1s%406202@103.125.129.116:7901/1
|
||||
|
||||
# JWT
|
||||
JWT_SECRET_KEY=change-this-to-a-secure-random-string
|
||||
JWT_SECRET_KEY=a7f3c9e1d4b8f2a6c0e5d7b3a9f1c4e8d2b6a0f5c3e7d1b9a4f8c2e6d0b5a3
|
||||
JWT_ALGORITHM=HS256
|
||||
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30
|
||||
JWT_REFRESH_TOKEN_EXPIRE_DAYS=7
|
||||
@@ -48,7 +48,7 @@ LOG_LEVEL=INFO
|
||||
LOG_FORMAT=json
|
||||
|
||||
# CORS
|
||||
CORS_ORIGINS=["http://localhost:3000","http://localhost:8080"]
|
||||
CORS_ORIGINS=["http://localhost:3000","http://localhost:8080","http://localhost:4200"]
|
||||
CORS_ALLOW_CREDENTIALS=true
|
||||
|
||||
# Rate Limiting
|
||||
64
docengine/.gitignore
vendored
64
docengine/.gitignore
vendored
@@ -1,64 +0,0 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
*.manifest
|
||||
*.spec
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
*.py,cover
|
||||
.hypothesis/
|
||||
.pytest_cache/
|
||||
*.mo
|
||||
*.pot
|
||||
*.log
|
||||
local_settings.py
|
||||
db.sqlite3
|
||||
db.sqlite3-journal
|
||||
instance/
|
||||
.webassets-cache
|
||||
.scrapy
|
||||
docs/_build/
|
||||
target/
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
.env
|
||||
!.env.example
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
storage/
|
||||
*.pid
|
||||
celerybeat-schedule
|
||||
celerybeat.pid
|
||||
@@ -1,39 +0,0 @@
|
||||
FROM python:3.12-slim AS base
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
libpq-dev \
|
||||
libgl1-mesa-glx \
|
||||
libglib2.0-0 \
|
||||
libsm6 \
|
||||
libxext6 \
|
||||
libxrender-dev \
|
||||
libgomp1 \
|
||||
poppler-utils \
|
||||
ghostscript \
|
||||
libmagic1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
RUN mkdir -p /app/storage/documents /app/storage/templates /app/storage/images /app/storage/temp
|
||||
|
||||
FROM base AS app
|
||||
EXPOSE 7989
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7989", "--workers", "4"]
|
||||
|
||||
FROM base AS worker
|
||||
CMD ["celery", "-A", "app.workers.celery_app", "worker", "--loglevel=info", "--concurrency=4"]
|
||||
|
||||
FROM base AS beat
|
||||
CMD ["celery", "-A", "app.workers.celery_app", "beat", "--loglevel=info"]
|
||||
@@ -1,522 +0,0 @@
|
||||
# DocEngine — Document Template Recognition & Reconstruction System
|
||||
|
||||
A production-ready system for scanning documents, detecting layouts, extracting content, generating reusable templates, matching future uploads against stored templates, and reconstructing original layouts as PDF output.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────┐ ┌───────────────┐ ┌──────────────┐
|
||||
│ FastAPI App │─────▶│ Celery │─────▶│ Redis │
|
||||
│ (Port 7989) │ │ Worker(s) │ │ (Broker) │
|
||||
└───────┬───────┘ └───────┬───────┘ └──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ PostgreSQL (Schema: admin) │
|
||||
│ 192.168.0.111:7925 │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Stack**: Python 3.12, FastAPI, SQLAlchemy 2.x, Pydantic V2, Celery, Redis, PaddleOCR, PyMuPDF, OpenCV, ReportLab, PostgreSQL 16.
|
||||
|
||||
## Features
|
||||
|
||||
| Capability | Implementation |
|
||||
|----------------------------------|--------------------------------------------|
|
||||
| Scanned image OCR | PaddleOCR (CPU/GPU) |
|
||||
| Native PDF text extraction | PyMuPDF (fitz) |
|
||||
| Layout detection | OpenCV + LayoutParser |
|
||||
| Table extraction | Camelot-py + OpenCV contour detection |
|
||||
| Header/footer detection | Positional heuristics |
|
||||
| Watermark detection | Transparency + large-font analysis |
|
||||
| Font info extraction | PyMuPDF text dict parsing |
|
||||
| Template generation & storage | PostgreSQL (admin schema) |
|
||||
| Template fingerprinting | SHA-256 structural hashing |
|
||||
| Template matching | Multi-signal similarity scoring |
|
||||
| PDF reconstruction | ReportLab from template definitions |
|
||||
| Async processing | Celery + Redis |
|
||||
| Authentication | JWT (access + refresh tokens, bcrypt) |
|
||||
| Monitoring | Prometheus + structlog JSON logging |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
docengine/
|
||||
├── app/
|
||||
│ ├── main.py # FastAPI application entry
|
||||
│ ├── api/
|
||||
│ │ ├── router.py # Top-level API router
|
||||
│ │ └── v1/
|
||||
│ │ ├── auth.py # Auth endpoints
|
||||
│ │ ├── documents.py # Document endpoints
|
||||
│ │ ├── health.py # Health check
|
||||
│ │ └── templates.py # Template endpoints
|
||||
│ ├── core/
|
||||
│ │ ├── config.py # Pydantic Settings
|
||||
│ │ ├── database.py # SQLAlchemy engine & session
|
||||
│ │ ├── dependencies.py # FastAPI DI
|
||||
│ │ ├── exceptions.py # Custom exception hierarchy
|
||||
│ │ ├── logging_config.py # structlog configuration
|
||||
│ │ └── security.py # JWT & bcrypt helpers
|
||||
│ ├── models/ # SQLAlchemy ORM models
|
||||
│ ├── schemas/ # Pydantic request/response schemas
|
||||
│ ├── repositories/ # Data access layer
|
||||
│ ├── services/ # Business logic
|
||||
│ │ ├── document_service.py # Orchestration pipeline
|
||||
│ │ ├── ocr_service.py # PaddleOCR integration
|
||||
│ │ ├── pdf_service.py # PyMuPDF native PDF parsing
|
||||
│ │ ├── layout_service.py # OpenCV layout detection
|
||||
│ │ ├── template_service.py # Template generation
|
||||
│ │ ├── fingerprint_service.py
|
||||
│ │ ├── matching_service.py
|
||||
│ │ └── reconstruction_service.py
|
||||
│ ├── middleware/ # CORS, audit, metrics, rate limit
|
||||
│ ├── storage/ # File storage abstraction
|
||||
│ ├── tasks/ # Celery async tasks
|
||||
│ ├── workers/ # Celery app configuration
|
||||
│ └── events/ # App lifecycle handlers
|
||||
├── alembic/ # Database migrations
|
||||
├── sql/ # Raw SQL scripts
|
||||
├── tests/ # Test suite
|
||||
├── docker-compose.yml # Dev stack
|
||||
├── docker-compose.prod.yml # Production stack
|
||||
├── Dockerfile # Multi-stage build
|
||||
├── requirements.txt
|
||||
└── .env
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12+
|
||||
- PostgreSQL 16 (running at `192.168.0.111:7925`)
|
||||
- Redis (for Celery)
|
||||
- `poppler-utils` and `ghostscript` (for pdf2image/camelot)
|
||||
|
||||
### Local Setup
|
||||
|
||||
```bash
|
||||
# Clone & enter
|
||||
cd docengine
|
||||
|
||||
# Create virtual environment
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Create storage directories
|
||||
mkdir -p storage/{documents,templates,images,temp,rendered}
|
||||
|
||||
# Run database migrations
|
||||
alembic upgrade head
|
||||
|
||||
# (Optional) Seed default data
|
||||
psql -h 192.168.0.111 -p 7925 -U postgres -d document_engine -f sql/003_seed_data.sql
|
||||
psql -h 192.168.0.111 -p 7925 -U postgres -d document_engine -f sql/004_indexes.sql
|
||||
|
||||
# Start the application
|
||||
python -m app.main
|
||||
```
|
||||
|
||||
The API is now available at `http://localhost:7989`. Interactive docs at `http://localhost:7989/docs`.
|
||||
|
||||
### Start Celery Worker (separate terminal)
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
celery -A app.workers.celery_app worker --loglevel=info --concurrency=4
|
||||
```
|
||||
|
||||
### Docker Setup
|
||||
|
||||
```bash
|
||||
# Build and start all services (app + worker + db + redis)
|
||||
docker compose up --build -d
|
||||
|
||||
# Run migrations inside the container
|
||||
docker compose exec app alembic upgrade head
|
||||
|
||||
# Seed data
|
||||
docker compose exec app bash -c "psql -h db -U postgres -d document_engine -f sql/003_seed_data.sql"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
Base URL: `http://localhost:7989/api/v1`
|
||||
|
||||
### Health
|
||||
|
||||
```bash
|
||||
curl http://localhost:7989/api/v1/health
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
#### Register
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/auth/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"username": "john",
|
||||
"email": "john@example.com",
|
||||
"password": "SecurePass123!",
|
||||
"full_name": "John Doe"
|
||||
}'
|
||||
```
|
||||
|
||||
#### Login
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"username": "john",
|
||||
"password": "SecurePass123!"
|
||||
}'
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "eyJhbGciOiJIUzI1NiIs...",
|
||||
"refresh_token": "eyJhbGciOiJIUzI1NiIs...",
|
||||
"token_type": "bearer",
|
||||
"expires_in": 1800
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Current User
|
||||
|
||||
```bash
|
||||
curl http://localhost:7989/api/v1/auth/me \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Refresh Token
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/auth/refresh \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"refresh_token": "<refresh_token>"}'
|
||||
```
|
||||
|
||||
#### Change Password
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/auth/change-password \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"current_password": "SecurePass123!",
|
||||
"new_password": "NewSecurePass456!"
|
||||
}'
|
||||
```
|
||||
|
||||
#### Logout
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/auth/logout \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"refresh_token": "<refresh_token>"}'
|
||||
```
|
||||
|
||||
### Documents
|
||||
|
||||
#### Upload Document
|
||||
|
||||
```bash
|
||||
# Upload a PDF
|
||||
curl -X POST http://localhost:7989/api/v1/documents/upload \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-F "file=@/path/to/document.pdf"
|
||||
|
||||
# Upload a scanned image
|
||||
curl -X POST http://localhost:7989/api/v1/documents/upload \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-F "file=@/path/to/scan.jpg"
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"filename": "abc123_document.pdf",
|
||||
"original_filename": "document.pdf",
|
||||
"content_type": "application/pdf",
|
||||
"file_size": 245760,
|
||||
"checksum": "e3b0c44298fc1c149afbf4c8996fb924...",
|
||||
"status": "pending",
|
||||
"created_at": "2026-06-01T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Document
|
||||
|
||||
```bash
|
||||
curl http://localhost:7989/api/v1/documents/<document_id> \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### List Documents
|
||||
|
||||
```bash
|
||||
# With pagination
|
||||
curl "http://localhost:7989/api/v1/documents?page=1&page_size=20" \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
|
||||
# Filter by status
|
||||
curl "http://localhost:7989/api/v1/documents?status=completed" \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Delete Document
|
||||
|
||||
```bash
|
||||
curl -X DELETE http://localhost:7989/api/v1/documents/<document_id> \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Get Template Matches for Document
|
||||
|
||||
```bash
|
||||
curl http://localhost:7989/api/v1/documents/<document_id>/template \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
### Templates
|
||||
|
||||
#### List Templates
|
||||
|
||||
```bash
|
||||
curl "http://localhost:7989/api/v1/templates?page=1&page_size=20" \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Get Template
|
||||
|
||||
```bash
|
||||
curl http://localhost:7989/api/v1/templates/<template_id> \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Delete (Deactivate) Template
|
||||
|
||||
```bash
|
||||
curl -X DELETE http://localhost:7989/api/v1/templates/<template_id> \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
#### Match Document to Templates
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/templates/match \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"document_id": "<document_id>",
|
||||
"min_confidence": 0.5,
|
||||
"max_results": 5
|
||||
}'
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "...",
|
||||
"document_id": "...",
|
||||
"format_id": "...",
|
||||
"confidence_score": 0.92,
|
||||
"match_details": { "dimension_score": 1.0, "header_score": 0.85 },
|
||||
"selected": true,
|
||||
"template_name": "Invoice Template v1",
|
||||
"created_at": "2026-06-01T12:00:00Z"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### Render Template to PDF
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7989/api/v1/templates/render \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"template_id": "<template_id>",
|
||||
"data": {
|
||||
"company_name": "Acme Corp",
|
||||
"invoice_number": "INV-2026-001",
|
||||
"date": "2026-06-01",
|
||||
"total": "$1,250.00"
|
||||
},
|
||||
"output_filename": "invoice_output.pdf"
|
||||
}'
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"output_path": "rendered/invoice_output.pdf",
|
||||
"filename": "invoice_output.pdf",
|
||||
"file_size": 32768,
|
||||
"page_count": 1,
|
||||
"rendered_at": "2026-06-01T12:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### Download Rendered PDF
|
||||
|
||||
```bash
|
||||
curl -O http://localhost:7989/api/v1/templates/<template_id>/download?filename=invoice_output.pdf \
|
||||
-H "Authorization: Bearer <access_token>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
When a document is uploaded, the following Celery task pipeline executes asynchronously:
|
||||
|
||||
1. **File Type Detection** — Determine if the document is a native PDF or scanned image.
|
||||
2. **Page Extraction** — Convert PDF pages to images (for scanned docs) or parse directly (for native PDFs).
|
||||
3. **OCR** — Run PaddleOCR on scanned pages to extract text blocks with coordinates, confidence, and bounding boxes.
|
||||
4. **Native PDF Parsing** — Use PyMuPDF to extract text, fonts, images, and tables from native PDFs.
|
||||
5. **Layout Analysis** — Detect headers, footers, watermarks, tables, and image regions using OpenCV heuristics.
|
||||
6. **Template Generation** — Build a reusable template definition from the detected layout, stored in PostgreSQL.
|
||||
7. **Fingerprint Generation** — Compute a structural fingerprint (SHA-256) for future matching.
|
||||
8. **Status Update** — Mark the document as `completed` (or `failed` with error details).
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
**Connection**: `postgresql://postgres:***@192.168.0.111:7925/document_engine`
|
||||
**Schema**: `admin`
|
||||
|
||||
### Migrations
|
||||
|
||||
```bash
|
||||
# Create a new migration
|
||||
alembic revision --autogenerate -m "description"
|
||||
|
||||
# Apply migrations
|
||||
alembic upgrade head
|
||||
|
||||
# Rollback one step
|
||||
alembic downgrade -1
|
||||
```
|
||||
|
||||
### Tables
|
||||
|
||||
| Table | Purpose |
|
||||
|--------------------------|----------------------------------------------|
|
||||
| `users` | User accounts |
|
||||
| `roles` | Role definitions (admin, user, viewer) |
|
||||
| `user_roles` | User-role mapping (M2M) |
|
||||
| `refresh_tokens` | JWT refresh token storage |
|
||||
| `audit_logs` | Action audit trail |
|
||||
| `documents` | Uploaded document records |
|
||||
| `document_pages` | Per-page data (dimensions, images) |
|
||||
| `document_text_blocks` | Extracted text with position & font info |
|
||||
| `document_images` | Extracted images with position |
|
||||
| `document_tables` | Extracted tables with cell data (JSONB) |
|
||||
| `document_formats` | Template definitions |
|
||||
| `document_cells` | Template cell layout definitions |
|
||||
| `document_regions` | Template region definitions |
|
||||
| `table_formats` | Template table structure definitions |
|
||||
| `table_columns` | Template table column definitions |
|
||||
| `table_rows` | Template table row definitions |
|
||||
| `watermarks` | Template watermark definitions |
|
||||
| `image_regions` | Template image region definitions |
|
||||
| `template_fingerprints` | Structural fingerprints for matching |
|
||||
| `template_matches` | Document-to-template match results |
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Install dev dependencies
|
||||
pip install -r requirements-dev.txt
|
||||
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=app --cov-report=term-missing
|
||||
|
||||
# Run specific test categories
|
||||
pytest tests/unit/
|
||||
pytest tests/api/
|
||||
pytest tests/repositories/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration is via environment variables (`.env` file). Key settings:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|------------------------------------|------------------------|---------------------------------|
|
||||
| `APP_PORT` | `7989` | Application port |
|
||||
| `DB_HOST` | `192.168.0.111` | PostgreSQL host |
|
||||
| `DB_PORT` | `7925` | PostgreSQL port |
|
||||
| `DB_NAME` | `document_engine` | Database name |
|
||||
| `DB_SCHEMA` | `admin` | PostgreSQL schema |
|
||||
| `REDIS_HOST` | `localhost` | Redis host |
|
||||
| `CELERY_BROKER_URL` | `redis://localhost:6379/0` | Celery broker |
|
||||
| `JWT_SECRET_KEY` | *(see .env)* | JWT signing key |
|
||||
| `JWT_ACCESS_TOKEN_EXPIRE_MINUTES` | `30` | Access token TTL |
|
||||
| `STORAGE_LOCAL_PATH` | `./storage` | Local file storage path |
|
||||
| `STORAGE_MAX_FILE_SIZE_MB` | `100` | Max upload size |
|
||||
| `OCR_LANGUAGE` | `en` | PaddleOCR language |
|
||||
| `OCR_USE_GPU` | `false` | Enable GPU for OCR |
|
||||
|
||||
---
|
||||
|
||||
## Production Deployment
|
||||
|
||||
```bash
|
||||
# Using production compose file
|
||||
docker compose -f docker-compose.prod.yml up --build -d
|
||||
|
||||
# Scale workers
|
||||
docker compose -f docker-compose.prod.yml up --scale worker=4 -d
|
||||
```
|
||||
|
||||
Production compose includes:
|
||||
- Resource limits (CPU/memory)
|
||||
- Redis authentication
|
||||
- App replicas
|
||||
- Persistent named volumes
|
||||
- Auto-restart policies
|
||||
|
||||
---
|
||||
|
||||
## Default Credentials
|
||||
|
||||
| Username | Password | Role |
|
||||
|----------|---------------|-------|
|
||||
| `admin` | `Admin@123!` | admin |
|
||||
|
||||
> ⚠️ **Change the default admin password immediately in production.**
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Proprietary — All rights reserved.
|
||||
@@ -1,41 +0,0 @@
|
||||
[alembic]
|
||||
script_location = alembic
|
||||
prepend_sys_path = .
|
||||
version_path_separator = os
|
||||
sqlalchemy.url = postgresql+psycopg2://postgres:M%%40tr%%21x%%23149%%40dm%%21N@192.168.0.111:7925/document_engine
|
||||
|
||||
[post_write_hooks]
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
BIN
docengine/alembic/__pycache__/env.cpython-313.pyc
Normal file
BIN
docengine/alembic/__pycache__/env.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,67 +0,0 @@
|
||||
from logging.config import fileConfig
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config, pool, text
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.database import Base
|
||||
|
||||
# Import all models so Alembic can detect them
|
||||
import app.models # noqa: F401
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
# Override the database URL from settings
|
||||
config.set_main_option("sqlalchemy.url", settings.database_url)
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode."""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
version_table_schema=settings.db_schema,
|
||||
include_schemas=True,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode."""
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
# Ensure schema exists
|
||||
connection.execute(text(f"CREATE SCHEMA IF NOT EXISTS {settings.db_schema}"))
|
||||
connection.execute(text(f"SET search_path TO {settings.db_schema}, public"))
|
||||
connection.commit()
|
||||
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
version_table_schema=settings.db_schema,
|
||||
include_schemas=True,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
@@ -1,26 +0,0 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -1,398 +0,0 @@
|
||||
"""initial schema
|
||||
|
||||
Revision ID: 001_initial
|
||||
Revises:
|
||||
Create Date: 2026-05-31 18:00:00.000000
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision: str = "001_initial"
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
SCHEMA = "admin"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Create schema
|
||||
op.execute(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
|
||||
|
||||
# Users table
|
||||
op.create_table(
|
||||
"users",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("username", sa.String(150), unique=True, nullable=False, index=True),
|
||||
sa.Column("email", sa.String(255), unique=True, nullable=False, index=True),
|
||||
sa.Column("hashed_password", sa.String(255), nullable=False),
|
||||
sa.Column("full_name", sa.String(255), nullable=True),
|
||||
sa.Column("is_active", sa.Boolean, default=True, nullable=False),
|
||||
sa.Column("is_superuser", sa.Boolean, default=False, nullable=False),
|
||||
sa.Column("last_login", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Roles table
|
||||
op.create_table(
|
||||
"roles",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("name", sa.String(50), unique=True, nullable=False, index=True),
|
||||
sa.Column("description", sa.Text, nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# User roles (many-to-many)
|
||||
op.create_table(
|
||||
"user_roles",
|
||||
sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), primary_key=True),
|
||||
sa.Column("role_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.roles.id", ondelete="CASCADE"), primary_key=True),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Refresh tokens
|
||||
op.create_table(
|
||||
"refresh_tokens",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("token", sa.String(512), unique=True, nullable=False, index=True),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("revoked", sa.Boolean, default=False, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Audit logs
|
||||
op.create_table(
|
||||
"audit_logs",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True),
|
||||
sa.Column("action", sa.String(100), nullable=False, index=True),
|
||||
sa.Column("resource_type", sa.String(100), nullable=False, index=True),
|
||||
sa.Column("resource_id", sa.String(255), nullable=True),
|
||||
sa.Column("details", sa.Text, nullable=True),
|
||||
sa.Column("ip_address", sa.String(45), nullable=True),
|
||||
sa.Column("user_agent", sa.String(512), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False, index=True),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Documents
|
||||
op.create_table(
|
||||
"documents",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("filename", sa.String(500), nullable=False),
|
||||
sa.Column("original_filename", sa.String(500), nullable=False),
|
||||
sa.Column("content_type", sa.String(100), nullable=False),
|
||||
sa.Column("file_size", sa.BigInteger, nullable=False),
|
||||
sa.Column("checksum", sa.String(128), nullable=False, index=True),
|
||||
sa.Column("storage_path", sa.String(1024), nullable=False),
|
||||
sa.Column("status", sa.String(50), default="pending", nullable=False, index=True),
|
||||
sa.Column("page_count", sa.Integer, nullable=True),
|
||||
sa.Column("is_scanned", sa.Boolean, nullable=True),
|
||||
sa.Column("document_metadata", postgresql.JSONB, nullable=True),
|
||||
sa.Column("error_message", sa.Text, nullable=True),
|
||||
sa.Column("uploaded_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document pages
|
||||
op.create_table(
|
||||
"document_pages",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("image_path", sa.String(1024), nullable=True),
|
||||
sa.Column("text_content", sa.Text, nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document text blocks
|
||||
op.create_table(
|
||||
"document_text_blocks",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("text", sa.Text, nullable=False),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("confidence", sa.Float, nullable=True),
|
||||
sa.Column("font_family", sa.String(255), nullable=True),
|
||||
sa.Column("font_size", sa.Float, nullable=True),
|
||||
sa.Column("font_color", sa.String(50), nullable=True),
|
||||
sa.Column("font_style", sa.String(50), nullable=True),
|
||||
sa.Column("block_type", sa.String(50), default="text", nullable=False),
|
||||
sa.Column("sequence", sa.Integer, default=0, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document images
|
||||
op.create_table(
|
||||
"document_images",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("image_path", sa.String(1024), nullable=False),
|
||||
sa.Column("image_type", sa.String(50), default="figure", nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document tables
|
||||
op.create_table(
|
||||
"document_tables",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("rows", sa.Integer, nullable=False),
|
||||
sa.Column("columns", sa.Integer, nullable=False),
|
||||
sa.Column("data", postgresql.JSONB, nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document formats (templates)
|
||||
op.create_table(
|
||||
"document_formats",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("name", sa.String(255), nullable=False, index=True),
|
||||
sa.Column("description", sa.Text, nullable=True),
|
||||
sa.Column("page_width", sa.Float, nullable=False),
|
||||
sa.Column("page_height", sa.Float, nullable=False),
|
||||
sa.Column("page_count", sa.Integer, default=1, nullable=False),
|
||||
sa.Column("margin_top", sa.Float, default=72.0, nullable=False),
|
||||
sa.Column("margin_right", sa.Float, default=72.0, nullable=False),
|
||||
sa.Column("margin_bottom", sa.Float, default=72.0, nullable=False),
|
||||
sa.Column("margin_left", sa.Float, default=72.0, nullable=False),
|
||||
sa.Column("fingerprint", postgresql.JSONB, nullable=True),
|
||||
sa.Column("source_document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="SET NULL"), nullable=True, index=True),
|
||||
sa.Column("version", sa.Integer, default=1, nullable=False),
|
||||
sa.Column("is_active", sa.Boolean, default=True, nullable=False, index=True),
|
||||
sa.Column("created_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document cells
|
||||
op.create_table(
|
||||
"document_cells",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=False),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("row_no", sa.Integer, default=0, nullable=False),
|
||||
sa.Column("column_no", sa.Integer, default=0, nullable=False),
|
||||
sa.Column("data_type", sa.String(50), default="text", nullable=False),
|
||||
sa.Column("font_family", sa.String(255), nullable=True),
|
||||
sa.Column("font_size", sa.Float, nullable=True),
|
||||
sa.Column("font_style", sa.String(50), nullable=True),
|
||||
sa.Column("font_color", sa.String(50), nullable=True),
|
||||
sa.Column("background_color", sa.String(50), nullable=True),
|
||||
sa.Column("border_top", sa.String(100), nullable=True),
|
||||
sa.Column("border_right", sa.String(100), nullable=True),
|
||||
sa.Column("border_bottom", sa.String(100), nullable=True),
|
||||
sa.Column("border_left", sa.String(100), nullable=True),
|
||||
sa.Column("padding_top", sa.Float, default=0.0, nullable=False),
|
||||
sa.Column("padding_right", sa.Float, default=0.0, nullable=False),
|
||||
sa.Column("padding_bottom", sa.Float, default=0.0, nullable=False),
|
||||
sa.Column("padding_left", sa.Float, default=0.0, nullable=False),
|
||||
sa.Column("alignment", sa.String(20), default="left", nullable=False),
|
||||
sa.Column("vertical_alignment", sa.String(20), default="top", nullable=False),
|
||||
sa.Column("rowspan", sa.Integer, default=1, nullable=False),
|
||||
sa.Column("colspan", sa.Integer, default=1, nullable=False),
|
||||
sa.Column("static_text", sa.Text, nullable=True),
|
||||
sa.Column("field_name", sa.String(255), nullable=True),
|
||||
sa.Column("sequence", sa.Integer, default=0, nullable=False),
|
||||
sa.Column("is_dynamic", sa.Boolean, default=False, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Document regions
|
||||
op.create_table(
|
||||
"document_regions",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=False),
|
||||
sa.Column("region_type", sa.String(50), nullable=False),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("content", postgresql.JSONB, nullable=True),
|
||||
sa.Column("sequence", sa.Integer, default=0, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Table formats
|
||||
op.create_table(
|
||||
"table_formats",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=False),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("rows", sa.Integer, nullable=False),
|
||||
sa.Column("columns", sa.Integer, nullable=False),
|
||||
sa.Column("border_style", sa.String(50), default="solid", nullable=False),
|
||||
sa.Column("border_width", sa.Float, default=1.0, nullable=False),
|
||||
sa.Column("border_color", sa.String(50), default="#000000", nullable=False),
|
||||
sa.Column("header_rows", sa.Integer, default=1, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Table columns
|
||||
op.create_table(
|
||||
"table_columns",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("column_index", sa.Integer, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("header_text", sa.String(500), nullable=True),
|
||||
sa.Column("data_type", sa.String(50), default="text", nullable=False),
|
||||
sa.Column("alignment", sa.String(20), default="left", nullable=False),
|
||||
sa.Column("font_family", sa.String(255), nullable=True),
|
||||
sa.Column("font_size", sa.Float, nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Table rows
|
||||
op.create_table(
|
||||
"table_rows",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("row_index", sa.Integer, nullable=False),
|
||||
sa.Column("height", sa.Float, default=20.0, nullable=False),
|
||||
sa.Column("is_header", sa.Boolean, default=False, nullable=False),
|
||||
sa.Column("background_color", sa.String(50), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Watermarks
|
||||
op.create_table(
|
||||
"watermarks",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=True),
|
||||
sa.Column("text", sa.String(500), nullable=True),
|
||||
sa.Column("image_path", sa.String(1024), nullable=True),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("opacity", sa.Float, default=0.3, nullable=False),
|
||||
sa.Column("rotation", sa.Float, default=0.0, nullable=False),
|
||||
sa.Column("font_family", sa.String(255), nullable=True),
|
||||
sa.Column("font_size", sa.Float, nullable=True),
|
||||
sa.Column("font_color", sa.String(50), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Image regions
|
||||
op.create_table(
|
||||
"image_regions",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("page_number", sa.Integer, nullable=False),
|
||||
sa.Column("x", sa.Float, nullable=False),
|
||||
sa.Column("y", sa.Float, nullable=False),
|
||||
sa.Column("width", sa.Float, nullable=False),
|
||||
sa.Column("height", sa.Float, nullable=False),
|
||||
sa.Column("image_path", sa.String(1024), nullable=True),
|
||||
sa.Column("image_type", sa.String(50), default="figure", nullable=False),
|
||||
sa.Column("is_static", sa.Boolean, default=True, nullable=False),
|
||||
sa.Column("field_name", sa.String(255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Template fingerprints
|
||||
op.create_table(
|
||||
"template_fingerprints",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, unique=True, index=True),
|
||||
sa.Column("page_dimensions", postgresql.JSONB, nullable=True),
|
||||
sa.Column("logo_coordinates", postgresql.JSONB, nullable=True),
|
||||
sa.Column("header_coordinates", postgresql.JSONB, nullable=True),
|
||||
sa.Column("footer_coordinates", postgresql.JSONB, nullable=True),
|
||||
sa.Column("table_coordinates", postgresql.JSONB, nullable=True),
|
||||
sa.Column("cell_coordinates", postgresql.JSONB, nullable=True),
|
||||
sa.Column("fingerprint_hash", sa.String(256), nullable=False, index=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Template matches
|
||||
op.create_table(
|
||||
"template_matches",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
|
||||
sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
|
||||
sa.Column("confidence_score", sa.Float, nullable=False),
|
||||
sa.Column("match_details", postgresql.JSONB, nullable=True),
|
||||
sa.Column("selected", sa.Boolean, default=False, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
|
||||
schema=SCHEMA,
|
||||
)
|
||||
|
||||
# Additional indexes
|
||||
op.create_index("ix_documents_status_created", "documents", ["status", "created_at"], schema=SCHEMA)
|
||||
op.create_index("ix_document_pages_doc_page", "document_pages", ["document_id", "page_number"], schema=SCHEMA)
|
||||
op.create_index("ix_document_text_blocks_type", "document_text_blocks", ["page_id", "block_type"], schema=SCHEMA)
|
||||
op.create_index("ix_document_cells_format_page", "document_cells", ["format_id", "page_number"], schema=SCHEMA)
|
||||
op.create_index("ix_template_matches_doc_score", "template_matches", ["document_id", "confidence_score"], schema=SCHEMA)
|
||||
op.create_index("ix_audit_logs_resource", "audit_logs", ["resource_type", "resource_id"], schema=SCHEMA)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("template_matches", schema=SCHEMA)
|
||||
op.drop_table("template_fingerprints", schema=SCHEMA)
|
||||
op.drop_table("image_regions", schema=SCHEMA)
|
||||
op.drop_table("watermarks", schema=SCHEMA)
|
||||
op.drop_table("table_rows", schema=SCHEMA)
|
||||
op.drop_table("table_columns", schema=SCHEMA)
|
||||
op.drop_table("table_formats", schema=SCHEMA)
|
||||
op.drop_table("document_regions", schema=SCHEMA)
|
||||
op.drop_table("document_cells", schema=SCHEMA)
|
||||
op.drop_table("document_formats", schema=SCHEMA)
|
||||
op.drop_table("document_tables", schema=SCHEMA)
|
||||
op.drop_table("document_images", schema=SCHEMA)
|
||||
op.drop_table("document_text_blocks", schema=SCHEMA)
|
||||
op.drop_table("document_pages", schema=SCHEMA)
|
||||
op.drop_table("documents", schema=SCHEMA)
|
||||
op.drop_table("audit_logs", schema=SCHEMA)
|
||||
op.drop_table("refresh_tokens", schema=SCHEMA)
|
||||
op.drop_table("user_roles", schema=SCHEMA)
|
||||
op.drop_table("roles", schema=SCHEMA)
|
||||
op.drop_table("users", schema=SCHEMA)
|
||||
Binary file not shown.
@@ -1,11 +0,0 @@
|
||||
|
||||
from fastapi import FastAPI
|
||||
app = FastAPI()
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status":"UP"}
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=7989)
|
||||
@@ -1 +0,0 @@
|
||||
# DocEngine - Document Template Recognition and Reconstruction System
|
||||
BIN
docengine/app/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/__pycache__/main.cpython-313.pyc
Normal file
BIN
docengine/app/__pycache__/main.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/api/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/__pycache__/router.cpython-313.pyc
Normal file
BIN
docengine/app/api/__pycache__/router.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,15 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.api.v1.auth import router as auth_router
|
||||
from app.api.v1.documents import router as documents_router
|
||||
from app.api.v1.health import router as health_router
|
||||
from app.api.v1.templates import router as templates_router
|
||||
|
||||
api_v1_router = APIRouter(prefix="/api/v1")
|
||||
|
||||
api_v1_router.include_router(health_router)
|
||||
api_v1_router.include_router(auth_router)
|
||||
api_v1_router.include_router(documents_router)
|
||||
api_v1_router.include_router(templates_router)
|
||||
BIN
docengine/app/api/v1/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/api/v1/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/v1/__pycache__/auth.cpython-313.pyc
Normal file
BIN
docengine/app/api/v1/__pycache__/auth.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/v1/__pycache__/documents.cpython-313.pyc
Normal file
BIN
docengine/app/api/v1/__pycache__/documents.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/v1/__pycache__/health.cpython-313.pyc
Normal file
BIN
docengine/app/api/v1/__pycache__/health.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/api/v1/__pycache__/templates.cpython-313.pyc
Normal file
BIN
docengine/app/api/v1/__pycache__/templates.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,246 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.database import get_db
|
||||
from app.core.dependencies import CurrentUser
|
||||
from app.core.security import (
|
||||
create_access_token,
|
||||
create_refresh_token,
|
||||
decode_token,
|
||||
hash_password,
|
||||
verify_password,
|
||||
InvalidTokenError,
|
||||
)
|
||||
from app.repositories.user_repository import RefreshTokenRepository, UserRepository
|
||||
from app.schemas.auth import (
|
||||
ChangePasswordRequest,
|
||||
LoginRequest,
|
||||
RefreshTokenRequest,
|
||||
RegisterRequest,
|
||||
TokenResponse,
|
||||
)
|
||||
from app.schemas.common import SuccessResponse
|
||||
from app.schemas.user import UserResponse
|
||||
|
||||
router = APIRouter(prefix="/auth", tags=["Authentication"])
|
||||
|
||||
|
||||
@router.post(
|
||||
"/register",
|
||||
response_model=UserResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
summary="Register User",
|
||||
description="Register a new user account.",
|
||||
)
|
||||
def register(
|
||||
payload: RegisterRequest,
|
||||
db: Session = Depends(get_db),
|
||||
) -> UserResponse:
|
||||
"""Register a new user."""
|
||||
user_repo = UserRepository(db)
|
||||
|
||||
# Check for existing user
|
||||
if user_repo.get_by_username(payload.username):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=f"Username '{payload.username}' is already taken",
|
||||
)
|
||||
if user_repo.get_by_email(payload.email):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=f"Email '{payload.email}' is already registered",
|
||||
)
|
||||
|
||||
hashed = hash_password(payload.password)
|
||||
user = user_repo.create_user(
|
||||
username=payload.username,
|
||||
email=payload.email,
|
||||
hashed_password=hashed,
|
||||
full_name=payload.full_name,
|
||||
role_names=["user"],
|
||||
)
|
||||
db.commit()
|
||||
db.refresh(user)
|
||||
|
||||
return UserResponse.model_validate(user)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/login",
|
||||
response_model=TokenResponse,
|
||||
summary="Login",
|
||||
description="Authenticate with username and password to obtain JWT tokens.",
|
||||
)
|
||||
def login(
|
||||
payload: LoginRequest,
|
||||
db: Session = Depends(get_db),
|
||||
) -> TokenResponse:
|
||||
"""Authenticate user and return JWT tokens."""
|
||||
user_repo = UserRepository(db)
|
||||
refresh_repo = RefreshTokenRepository(db)
|
||||
|
||||
user = user_repo.get_by_username(payload.username)
|
||||
if not user or not verify_password(payload.password, user.hashed_password):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid username or password",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
if not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="User account is deactivated",
|
||||
)
|
||||
|
||||
# Generate tokens
|
||||
access_token = create_access_token(data={"sub": str(user.id), "username": user.username})
|
||||
refresh_token_str = create_refresh_token(data={"sub": str(user.id)})
|
||||
|
||||
# Store refresh token
|
||||
expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days)
|
||||
refresh_repo.create_token(
|
||||
user_id=user.id,
|
||||
token=refresh_token_str,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
|
||||
# Update last login
|
||||
user_repo.update_last_login(user)
|
||||
db.commit()
|
||||
|
||||
return TokenResponse(
|
||||
access_token=access_token,
|
||||
refresh_token=refresh_token_str,
|
||||
token_type="bearer",
|
||||
expires_in=settings.jwt_access_token_expire_minutes * 60,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/refresh",
|
||||
response_model=TokenResponse,
|
||||
summary="Refresh Token",
|
||||
description="Obtain a new access token using a valid refresh token.",
|
||||
)
|
||||
def refresh_token(
|
||||
payload: RefreshTokenRequest,
|
||||
db: Session = Depends(get_db),
|
||||
) -> TokenResponse:
|
||||
"""Refresh access token using a refresh token."""
|
||||
refresh_repo = RefreshTokenRepository(db)
|
||||
user_repo = UserRepository(db)
|
||||
|
||||
# Validate the refresh token
|
||||
try:
|
||||
token_payload = decode_token(payload.refresh_token)
|
||||
except InvalidTokenError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid or expired refresh token",
|
||||
)
|
||||
|
||||
if token_payload.get("type") != "refresh":
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token type",
|
||||
)
|
||||
|
||||
# Check if token exists in database and is not revoked
|
||||
stored_token = refresh_repo.get_by_token(payload.refresh_token)
|
||||
if not stored_token:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Refresh token not found or revoked",
|
||||
)
|
||||
|
||||
user = user_repo.get_by_id(token_payload["sub"])
|
||||
if not user or not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or deactivated",
|
||||
)
|
||||
|
||||
# Revoke old refresh token
|
||||
refresh_repo.revoke_token(payload.refresh_token)
|
||||
|
||||
# Generate new tokens
|
||||
new_access_token = create_access_token(data={"sub": str(user.id), "username": user.username})
|
||||
new_refresh_token = create_refresh_token(data={"sub": str(user.id)})
|
||||
|
||||
expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days)
|
||||
refresh_repo.create_token(
|
||||
user_id=user.id,
|
||||
token=new_refresh_token,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
db.commit()
|
||||
|
||||
return TokenResponse(
|
||||
access_token=new_access_token,
|
||||
refresh_token=new_refresh_token,
|
||||
token_type="bearer",
|
||||
expires_in=settings.jwt_access_token_expire_minutes * 60,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/logout",
|
||||
response_model=SuccessResponse,
|
||||
summary="Logout",
|
||||
description="Revoke the current refresh token.",
|
||||
)
|
||||
def logout(
|
||||
payload: RefreshTokenRequest,
|
||||
current_user: CurrentUser,
|
||||
db: Session = Depends(get_db),
|
||||
) -> SuccessResponse:
|
||||
"""Logout by revoking the refresh token."""
|
||||
refresh_repo = RefreshTokenRepository(db)
|
||||
refresh_repo.revoke_token(payload.refresh_token)
|
||||
db.commit()
|
||||
return SuccessResponse(message="Successfully logged out")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/change-password",
|
||||
response_model=SuccessResponse,
|
||||
summary="Change Password",
|
||||
description="Change the current user's password.",
|
||||
)
|
||||
def change_password(
|
||||
payload: ChangePasswordRequest,
|
||||
current_user: CurrentUser,
|
||||
db: Session = Depends(get_db),
|
||||
) -> SuccessResponse:
|
||||
"""Change user password."""
|
||||
if not verify_password(payload.current_password, current_user.hashed_password):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="Current password is incorrect",
|
||||
)
|
||||
|
||||
current_user.hashed_password = hash_password(payload.new_password)
|
||||
|
||||
# Revoke all refresh tokens for security
|
||||
refresh_repo = RefreshTokenRepository(db)
|
||||
refresh_repo.revoke_all_user_tokens(current_user.id)
|
||||
db.commit()
|
||||
|
||||
return SuccessResponse(message="Password changed successfully")
|
||||
|
||||
|
||||
@router.get(
|
||||
"/me",
|
||||
response_model=UserResponse,
|
||||
summary="Get Current User",
|
||||
description="Get the currently authenticated user's profile.",
|
||||
)
|
||||
def get_me(current_user: CurrentUser) -> UserResponse:
|
||||
"""Get current authenticated user profile."""
|
||||
return UserResponse.model_validate(current_user)
|
||||
@@ -1,268 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Query, UploadFile, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.database import get_db
|
||||
from app.core.dependencies import CurrentUser
|
||||
from app.core.exceptions import FileSizeError, UnsupportedFileTypeError
|
||||
from app.core.logging_config import get_logger
|
||||
from app.models.document import Document
|
||||
from app.repositories.document_repository import DocumentRepository
|
||||
from app.schemas.common import PaginatedResponse, SuccessResponse
|
||||
from app.schemas.document import (
|
||||
DocumentListResponse,
|
||||
DocumentResponse,
|
||||
DocumentUploadResponse,
|
||||
TemplateMatchRequest,
|
||||
TemplateMatchResponse,
|
||||
)
|
||||
from app.storage.provider import LocalStorageProvider, get_storage_provider
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/documents", tags=["Documents"])
|
||||
|
||||
ALLOWED_CONTENT_TYPES = {
|
||||
"image/jpeg": "jpg",
|
||||
"image/png": "png",
|
||||
"image/tiff": "tiff",
|
||||
"application/pdf": "pdf",
|
||||
}
|
||||
|
||||
ALLOWED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tiff", ".tif", ".pdf"}
|
||||
|
||||
|
||||
def _validate_file(file: UploadFile) -> str:
|
||||
"""Validate uploaded file type and size. Returns the content type."""
|
||||
if not file.filename:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="Filename is required",
|
||||
)
|
||||
|
||||
# Check extension
|
||||
from pathlib import Path
|
||||
ext = Path(file.filename).suffix.lower()
|
||||
if ext not in ALLOWED_EXTENSIONS:
|
||||
raise UnsupportedFileTypeError(ext)
|
||||
|
||||
# Determine content type
|
||||
content_type = file.content_type or ""
|
||||
if content_type not in ALLOWED_CONTENT_TYPES:
|
||||
# Try to infer from extension
|
||||
ext_to_ct = {
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".png": "image/png",
|
||||
".tiff": "image/tiff",
|
||||
".tif": "image/tiff",
|
||||
".pdf": "application/pdf",
|
||||
}
|
||||
content_type = ext_to_ct.get(ext, "")
|
||||
if not content_type:
|
||||
raise UnsupportedFileTypeError(file.content_type or "unknown")
|
||||
|
||||
return content_type
|
||||
|
||||
|
||||
@router.post(
|
||||
"/upload",
|
||||
response_model=DocumentUploadResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
summary="Upload Document",
|
||||
description="Upload a document (JPG, JPEG, PNG, TIFF, or PDF) for processing.",
|
||||
)
|
||||
async def upload_document(
|
||||
file: UploadFile = File(..., description="Document file to upload"),
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> DocumentUploadResponse:
|
||||
"""Upload a document for processing."""
|
||||
content_type = _validate_file(file)
|
||||
|
||||
# Read file data
|
||||
file_data = await file.read()
|
||||
|
||||
# Check file size
|
||||
if len(file_data) > settings.storage_max_file_size_bytes:
|
||||
raise FileSizeError(settings.storage_max_file_size_mb)
|
||||
|
||||
# Store file
|
||||
storage = get_storage_provider()
|
||||
checksum = storage.compute_checksum(file_data)
|
||||
safe_filename = file.filename or "unknown"
|
||||
stored_filename = storage.generate_filename(safe_filename)
|
||||
storage_path = storage.save_file(file_data, "documents", stored_filename)
|
||||
|
||||
# Create document record
|
||||
doc_repo = DocumentRepository(db)
|
||||
document = Document(
|
||||
filename=stored_filename,
|
||||
original_filename=safe_filename,
|
||||
content_type=content_type,
|
||||
file_size=len(file_data),
|
||||
checksum=checksum,
|
||||
storage_path=storage_path,
|
||||
status="pending",
|
||||
uploaded_by=current_user.id if current_user else None,
|
||||
)
|
||||
doc_repo.create(document)
|
||||
db.commit()
|
||||
db.refresh(document)
|
||||
|
||||
logger.info(
|
||||
"document_uploaded",
|
||||
document_id=str(document.id),
|
||||
filename=safe_filename,
|
||||
size=len(file_data),
|
||||
content_type=content_type,
|
||||
)
|
||||
|
||||
# Trigger async processing via Celery
|
||||
try:
|
||||
from app.tasks.document_tasks import process_document_task
|
||||
process_document_task.delay(str(document.id))
|
||||
except Exception as e:
|
||||
logger.warning("celery_dispatch_failed", error=str(e), document_id=str(document.id))
|
||||
|
||||
return DocumentUploadResponse.model_validate(document)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{document_id}",
|
||||
response_model=DocumentResponse,
|
||||
summary="Get Document",
|
||||
description="Retrieve a document by its ID with all extracted content.",
|
||||
)
|
||||
def get_document(
|
||||
document_id: uuid.UUID,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> DocumentResponse:
|
||||
"""Get a document by ID."""
|
||||
doc_repo = DocumentRepository(db)
|
||||
document = doc_repo.get_with_pages(document_id)
|
||||
if not document:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Document '{document_id}' not found",
|
||||
)
|
||||
return DocumentResponse.model_validate(document)
|
||||
|
||||
|
||||
@router.get(
|
||||
"",
|
||||
response_model=PaginatedResponse[DocumentListResponse],
|
||||
summary="List Documents",
|
||||
description="List all documents with pagination.",
|
||||
)
|
||||
def list_documents(
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
status_filter: str | None = Query(default=None, alias="status"),
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> PaginatedResponse[DocumentListResponse]:
|
||||
"""List documents with pagination and optional status filter."""
|
||||
doc_repo = DocumentRepository(db)
|
||||
offset = (page - 1) * page_size
|
||||
filters = {}
|
||||
if status_filter:
|
||||
filters["status"] = status_filter
|
||||
|
||||
documents = doc_repo.get_all(
|
||||
offset=offset,
|
||||
limit=page_size,
|
||||
filters=filters,
|
||||
order_by="created_at",
|
||||
order_desc=True,
|
||||
)
|
||||
total = doc_repo.count(filters=filters)
|
||||
|
||||
items = [DocumentListResponse.model_validate(doc) for doc in documents]
|
||||
return PaginatedResponse.create(
|
||||
items=items,
|
||||
total=total,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{document_id}/template",
|
||||
response_model=list[TemplateMatchResponse],
|
||||
summary="Get Document Template Matches",
|
||||
description="Get template matching results for a document.",
|
||||
)
|
||||
def get_document_template_matches(
|
||||
document_id: uuid.UUID,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> list[TemplateMatchResponse]:
|
||||
"""Get template matches for a document."""
|
||||
from app.repositories.document_repository import TemplateMatchRepository
|
||||
|
||||
doc_repo = DocumentRepository(db)
|
||||
document = doc_repo.get_by_id(document_id)
|
||||
if not document:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Document '{document_id}' not found",
|
||||
)
|
||||
|
||||
match_repo = TemplateMatchRepository(db)
|
||||
matches = match_repo.get_document_matches(document_id)
|
||||
|
||||
results = []
|
||||
for match in matches:
|
||||
resp = TemplateMatchResponse(
|
||||
id=match.id,
|
||||
document_id=match.document_id,
|
||||
format_id=match.format_id,
|
||||
confidence_score=match.confidence_score,
|
||||
match_details=match.match_details,
|
||||
selected=match.selected,
|
||||
template_name=match.template.name if match.template else None,
|
||||
created_at=match.created_at,
|
||||
)
|
||||
results.append(resp)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/{document_id}",
|
||||
response_model=SuccessResponse,
|
||||
summary="Delete Document",
|
||||
description="Delete a document and its associated data.",
|
||||
)
|
||||
def delete_document(
|
||||
document_id: uuid.UUID,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> SuccessResponse:
|
||||
"""Delete a document."""
|
||||
doc_repo = DocumentRepository(db)
|
||||
document = doc_repo.get_by_id(document_id)
|
||||
if not document:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Document '{document_id}' not found",
|
||||
)
|
||||
|
||||
# Delete stored file
|
||||
try:
|
||||
storage = get_storage_provider()
|
||||
storage.delete_file(document.storage_path)
|
||||
except Exception as e:
|
||||
logger.warning("file_delete_failed", error=str(e), path=document.storage_path)
|
||||
|
||||
doc_repo.delete(document)
|
||||
db.commit()
|
||||
|
||||
return SuccessResponse(message=f"Document '{document_id}' deleted successfully")
|
||||
@@ -1,51 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
import redis
|
||||
from fastapi import APIRouter, status
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.database import check_database_connection
|
||||
from app.schemas.common import HealthResponse
|
||||
|
||||
router = APIRouter(tags=["Health"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"/health",
|
||||
response_model=HealthResponse,
|
||||
status_code=status.HTTP_200_OK,
|
||||
summary="Health Check",
|
||||
description="Check the health status of the application and its dependencies.",
|
||||
)
|
||||
async def health_check() -> HealthResponse:
|
||||
"""Perform health check on all system components."""
|
||||
# Check database
|
||||
db_status = "healthy" if check_database_connection() else "unhealthy"
|
||||
|
||||
# Check Redis
|
||||
redis_status = "healthy"
|
||||
try:
|
||||
r = redis.Redis(
|
||||
host=settings.redis_host,
|
||||
port=settings.redis_port,
|
||||
db=settings.redis_db,
|
||||
password=settings.redis_password or None,
|
||||
socket_timeout=3,
|
||||
)
|
||||
r.ping()
|
||||
r.close()
|
||||
except Exception:
|
||||
redis_status = "unhealthy"
|
||||
|
||||
overall_status = "healthy" if db_status == "healthy" and redis_status == "healthy" else "degraded"
|
||||
|
||||
return HealthResponse(
|
||||
status=overall_status,
|
||||
version=settings.app_version,
|
||||
environment=settings.app_env,
|
||||
database=db_status,
|
||||
redis=redis_status,
|
||||
timestamp=datetime.utcnow(),
|
||||
)
|
||||
@@ -1,223 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.database import get_db
|
||||
from app.core.dependencies import CurrentUser
|
||||
from app.core.logging_config import get_logger
|
||||
from app.repositories.document_repository import DocumentRepository, TemplateMatchRepository
|
||||
from app.repositories.template_repository import TemplateRepository
|
||||
from app.schemas.common import PaginatedResponse, SuccessResponse
|
||||
from app.schemas.document import TemplateMatchRequest, TemplateMatchResponse
|
||||
from app.schemas.template import (
|
||||
TemplateListResponse,
|
||||
TemplateRenderRequest,
|
||||
TemplateRenderResponse,
|
||||
TemplateResponse,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/templates", tags=["Templates"])
|
||||
|
||||
|
||||
@router.get(
|
||||
"",
|
||||
response_model=PaginatedResponse[TemplateListResponse],
|
||||
summary="List Templates",
|
||||
description="List all active templates with pagination.",
|
||||
)
|
||||
def list_templates(
|
||||
page: int = Query(default=1, ge=1),
|
||||
page_size: int = Query(default=20, ge=1, le=100),
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> PaginatedResponse[TemplateListResponse]:
|
||||
"""List all active templates."""
|
||||
template_repo = TemplateRepository(db)
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
templates = template_repo.get_active_templates(offset=offset, limit=page_size)
|
||||
total = template_repo.count_active()
|
||||
|
||||
items = [TemplateListResponse.model_validate(t) for t in templates]
|
||||
return PaginatedResponse.create(
|
||||
items=items,
|
||||
total=total,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{template_id}",
|
||||
response_model=TemplateResponse,
|
||||
summary="Get Template",
|
||||
description="Retrieve a template by ID with all its components.",
|
||||
)
|
||||
def get_template(
|
||||
template_id: uuid.UUID,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> TemplateResponse:
|
||||
"""Get a template by ID."""
|
||||
template_repo = TemplateRepository(db)
|
||||
template = template_repo.get_by_id(template_id)
|
||||
if not template:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Template '{template_id}' not found",
|
||||
)
|
||||
return TemplateResponse.model_validate(template)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/{template_id}",
|
||||
response_model=SuccessResponse,
|
||||
summary="Delete Template",
|
||||
description="Soft-delete a template by deactivating it.",
|
||||
)
|
||||
def delete_template(
|
||||
template_id: uuid.UUID,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> SuccessResponse:
|
||||
"""Soft-delete a template."""
|
||||
template_repo = TemplateRepository(db)
|
||||
template = template_repo.get_by_id(template_id)
|
||||
if not template:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Template '{template_id}' not found",
|
||||
)
|
||||
|
||||
template_repo.deactivate_template(template_id)
|
||||
db.commit()
|
||||
|
||||
return SuccessResponse(message=f"Template '{template_id}' deactivated successfully")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/match",
|
||||
response_model=list[TemplateMatchResponse],
|
||||
summary="Match Document to Templates",
|
||||
description="Match a document against existing templates and return ranked results.",
|
||||
)
|
||||
def match_template(
|
||||
payload: TemplateMatchRequest,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> list[TemplateMatchResponse]:
|
||||
"""Match a document against existing templates."""
|
||||
doc_repo = DocumentRepository(db)
|
||||
document = doc_repo.get_by_id(payload.document_id)
|
||||
if not document:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Document '{payload.document_id}' not found",
|
||||
)
|
||||
|
||||
if document.status != "completed":
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Document must be in 'completed' status. Current status: '{document.status}'",
|
||||
)
|
||||
|
||||
# Perform template matching
|
||||
from app.services.matching_service import MatchingService
|
||||
matching_service = MatchingService(db)
|
||||
matches = matching_service.match_document(
|
||||
document_id=payload.document_id,
|
||||
min_confidence=payload.min_confidence,
|
||||
max_results=payload.max_results,
|
||||
)
|
||||
db.commit()
|
||||
|
||||
results = []
|
||||
for match in matches:
|
||||
resp = TemplateMatchResponse(
|
||||
id=match.id,
|
||||
document_id=match.document_id,
|
||||
format_id=match.format_id,
|
||||
confidence_score=match.confidence_score,
|
||||
match_details=match.match_details,
|
||||
selected=match.selected,
|
||||
template_name=match.template.name if match.template else None,
|
||||
created_at=match.created_at,
|
||||
)
|
||||
results.append(resp)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@router.post(
|
||||
"/render",
|
||||
response_model=TemplateRenderResponse,
|
||||
summary="Render Template to PDF",
|
||||
description="Generate a PDF from a stored template with supplied data.",
|
||||
)
|
||||
def render_template(
|
||||
payload: TemplateRenderRequest,
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> TemplateRenderResponse:
|
||||
"""Render a template to PDF."""
|
||||
template_repo = TemplateRepository(db)
|
||||
template = template_repo.get_by_id(payload.template_id)
|
||||
if not template:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Template '{payload.template_id}' not found",
|
||||
)
|
||||
|
||||
if not template.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="Template is deactivated",
|
||||
)
|
||||
|
||||
from app.services.reconstruction_service import ReconstructionService
|
||||
reconstruction_service = ReconstructionService(db)
|
||||
result = reconstruction_service.render_template(
|
||||
template=template,
|
||||
data=payload.data,
|
||||
output_filename=payload.output_filename,
|
||||
images=payload.images,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{template_id}/download",
|
||||
summary="Download Rendered PDF",
|
||||
description="Download a previously rendered PDF.",
|
||||
)
|
||||
def download_rendered_pdf(
|
||||
template_id: uuid.UUID,
|
||||
filename: str = Query(..., description="Filename of the rendered PDF"),
|
||||
current_user: CurrentUser = None,
|
||||
db: Session = Depends(get_db),
|
||||
) -> FileResponse:
|
||||
"""Download a rendered PDF."""
|
||||
from app.storage.provider import get_storage_provider
|
||||
|
||||
storage = get_storage_provider()
|
||||
storage_path = f"rendered/{filename}"
|
||||
|
||||
if not storage.file_exists(storage_path):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Rendered PDF '{filename}' not found",
|
||||
)
|
||||
|
||||
absolute_path = storage.get_absolute_path(storage_path)
|
||||
return FileResponse(
|
||||
path=absolute_path,
|
||||
media_type="application/pdf",
|
||||
filename=filename,
|
||||
)
|
||||
BIN
docengine/app/core/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/aes.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/aes.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/config.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/config.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/database.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/database.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/dependencies.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/dependencies.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/exceptions.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/exceptions.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/logging_config.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/logging_config.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/core/__pycache__/security.cpython-313.pyc
Normal file
BIN
docengine/app/core/__pycache__/security.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,119 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from pydantic import field_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application configuration loaded from environment variables."""
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
case_sensitive=False,
|
||||
extra="ignore",
|
||||
)
|
||||
|
||||
# Application
|
||||
app_name: str = "DocEngine"
|
||||
app_version: str = "1.0.0"
|
||||
app_env: str = "development"
|
||||
app_debug: bool = True
|
||||
app_host: str = "0.0.0.0"
|
||||
app_port: int = 7989
|
||||
app_workers: int = 4
|
||||
|
||||
# Database
|
||||
db_host: str = "192.168.0.111"
|
||||
db_port: int = 7925
|
||||
db_name: str = "document_engine"
|
||||
db_user: str = "postgres"
|
||||
db_password: str = "M@tr!x#149@dm!N"
|
||||
db_schema: str = "admin"
|
||||
db_pool_size: int = 20
|
||||
db_max_overflow: int = 10
|
||||
db_echo: bool = False
|
||||
|
||||
# Redis
|
||||
redis_host: str = "localhost"
|
||||
redis_port: int = 6379
|
||||
redis_db: int = 0
|
||||
redis_password: str = ""
|
||||
|
||||
# Celery
|
||||
celery_broker_url: str = "redis://localhost:6379/0"
|
||||
celery_result_backend: str = "redis://localhost:6379/1"
|
||||
|
||||
# JWT
|
||||
jwt_secret_key: str = "a7f3c9e1d4b8f2a6c0e5d7b3a9f1c4e8d2b6a0f5c3e7d1b9a4f8c2e6d0b5a3"
|
||||
jwt_algorithm: str = "HS256"
|
||||
jwt_access_token_expire_minutes: int = 30
|
||||
jwt_refresh_token_expire_days: int = 7
|
||||
|
||||
# Storage
|
||||
storage_provider: str = "local"
|
||||
storage_local_path: str = "./storage"
|
||||
storage_max_file_size_mb: int = 100
|
||||
|
||||
# OCR
|
||||
ocr_language: str = "en"
|
||||
ocr_use_gpu: bool = False
|
||||
|
||||
# Logging
|
||||
log_level: str = "INFO"
|
||||
log_format: str = "json"
|
||||
|
||||
# CORS
|
||||
cors_origins: list[str] = ["http://localhost:3000", "http://localhost:8080"]
|
||||
cors_allow_credentials: bool = True
|
||||
|
||||
# Rate Limiting
|
||||
rate_limit_requests: int = 100
|
||||
rate_limit_window_seconds: int = 60
|
||||
|
||||
# Prometheus
|
||||
prometheus_enabled: bool = True
|
||||
|
||||
@field_validator("cors_origins", mode="before")
|
||||
@classmethod
|
||||
def parse_cors_origins(cls, v: Any) -> list[str]:
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
parsed = json.loads(v)
|
||||
if isinstance(parsed, list):
|
||||
return parsed
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return [origin.strip() for origin in v.split(",") if origin.strip()]
|
||||
return v
|
||||
|
||||
@property
|
||||
def database_url(self) -> str:
|
||||
from urllib.parse import quote_plus
|
||||
password = quote_plus(self.db_password)
|
||||
return f"postgresql+psycopg2://{self.db_user}:{password}@{self.db_host}:{self.db_port}/{self.db_name}"
|
||||
|
||||
@property
|
||||
def async_database_url(self) -> str:
|
||||
from urllib.parse import quote_plus
|
||||
password = quote_plus(self.db_password)
|
||||
return f"postgresql+asyncpg://{self.db_user}:{password}@{self.db_host}:{self.db_port}/{self.db_name}"
|
||||
|
||||
@property
|
||||
def redis_url(self) -> str:
|
||||
if self.redis_password:
|
||||
return f"redis://:{self.redis_password}@{self.redis_host}:{self.redis_port}/{self.redis_db}"
|
||||
return f"redis://{self.redis_host}:{self.redis_port}/{self.redis_db}"
|
||||
|
||||
@property
|
||||
def is_production(self) -> bool:
|
||||
return self.app_env == "production"
|
||||
|
||||
@property
|
||||
def storage_max_file_size_bytes(self) -> int:
|
||||
return self.storage_max_file_size_mb * 1024 * 1024
|
||||
|
||||
|
||||
settings = Settings()
|
||||
@@ -1,88 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Generator
|
||||
from contextlib import contextmanager
|
||||
|
||||
from sqlalchemy import MetaData, create_engine, event, text
|
||||
from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
NAMING_CONVENTION = {
|
||||
"ix": "ix_%(column_0_label)s",
|
||||
"uq": "uq_%(table_name)s_%(column_0_name)s",
|
||||
"ck": "ck_%(table_name)s_%(constraint_name)s",
|
||||
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
|
||||
"pk": "pk_%(table_name)s",
|
||||
}
|
||||
|
||||
metadata = MetaData(
|
||||
naming_convention=NAMING_CONVENTION,
|
||||
schema=settings.db_schema,
|
||||
)
|
||||
|
||||
engine = create_engine(
|
||||
settings.database_url,
|
||||
pool_size=settings.db_pool_size,
|
||||
max_overflow=settings.db_max_overflow,
|
||||
echo=settings.db_echo,
|
||||
pool_pre_ping=True,
|
||||
pool_recycle=3600,
|
||||
connect_args={
|
||||
"options": f"-c search_path={settings.db_schema},public"
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def set_search_path(dbapi_connection: object, connection_record: object) -> None:
|
||||
cursor = dbapi_connection.cursor() # type: ignore[union-attr]
|
||||
cursor.execute(f"SET search_path TO {settings.db_schema}, public")
|
||||
cursor.close()
|
||||
dbapi_connection.commit() # type: ignore[union-attr]
|
||||
|
||||
|
||||
SessionLocal = sessionmaker(
|
||||
autocommit=False,
|
||||
autoflush=False,
|
||||
bind=engine,
|
||||
)
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
"""Base class for all SQLAlchemy models."""
|
||||
|
||||
metadata = metadata
|
||||
|
||||
|
||||
def get_db() -> Generator[Session, None, None]:
|
||||
"""Dependency to get database session."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def get_db_context() -> Generator[Session, None, None]:
|
||||
"""Context manager for database session (used outside request scope)."""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def check_database_connection() -> bool:
|
||||
"""Verify database connectivity."""
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("SELECT 1"))
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
@@ -1,97 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.database import get_db
|
||||
from app.core.security import InvalidTokenError, decode_token
|
||||
from app.models.user import User
|
||||
from app.repositories.user_repository import UserRepository
|
||||
|
||||
security_scheme = HTTPBearer(auto_error=True)
|
||||
|
||||
|
||||
def get_current_user(
|
||||
credentials: Annotated[HTTPAuthorizationCredentials, Depends(security_scheme)],
|
||||
db: Annotated[Session, Depends(get_db)],
|
||||
) -> User:
|
||||
"""Extract and validate the current user from the JWT token."""
|
||||
try:
|
||||
payload = decode_token(credentials.credentials)
|
||||
except InvalidTokenError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid or expired token",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
token_type = payload.get("type")
|
||||
if token_type != "access":
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token type. Access token required.",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
user_id: str | None = payload.get("sub")
|
||||
if user_id is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Token payload missing subject",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
user_repo = UserRepository(db)
|
||||
user = user_repo.get_by_id(user_id)
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
if not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="User account is deactivated",
|
||||
)
|
||||
|
||||
return user
|
||||
|
||||
|
||||
def get_current_active_user(
|
||||
current_user: Annotated[User, Depends(get_current_user)],
|
||||
) -> User:
|
||||
"""Ensure the current user is active."""
|
||||
if not current_user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="User account is deactivated",
|
||||
)
|
||||
return current_user
|
||||
|
||||
|
||||
def require_role(required_roles: list[str]): # noqa: ANN201
|
||||
"""Dependency factory to require specific roles."""
|
||||
|
||||
def role_checker(
|
||||
current_user: Annotated[User, Depends(get_current_user)],
|
||||
) -> User:
|
||||
user_roles = {role.name for role in current_user.roles}
|
||||
if not user_roles.intersection(required_roles):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=f"One of the following roles required: {', '.join(required_roles)}",
|
||||
)
|
||||
return current_user
|
||||
|
||||
return role_checker
|
||||
|
||||
|
||||
CurrentUser = Annotated[User, Depends(get_current_user)]
|
||||
ActiveUser = Annotated[User, Depends(get_current_active_user)]
|
||||
AdminUser = Annotated[User, Depends(require_role(["admin"]))]
|
||||
DBSession = Annotated[Session, Depends(get_db)]
|
||||
@@ -1,106 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
class DocEngineException(Exception):
|
||||
"""Base exception for DocEngine application."""
|
||||
|
||||
def __init__(self, detail: str, status_code: int = 500, extra: dict[str, Any] | None = None) -> None:
|
||||
self.detail = detail
|
||||
self.status_code = status_code
|
||||
self.extra = extra or {}
|
||||
super().__init__(self.detail)
|
||||
|
||||
|
||||
class NotFoundError(DocEngineException):
|
||||
"""Resource not found."""
|
||||
|
||||
def __init__(self, resource: str, identifier: str) -> None:
|
||||
super().__init__(
|
||||
detail=f"{resource} with identifier '{identifier}' not found",
|
||||
status_code=404,
|
||||
)
|
||||
self.resource = resource
|
||||
self.identifier = identifier
|
||||
|
||||
|
||||
class DuplicateError(DocEngineException):
|
||||
"""Resource already exists."""
|
||||
|
||||
def __init__(self, resource: str, field: str, value: str) -> None:
|
||||
super().__init__(
|
||||
detail=f"{resource} with {field} '{value}' already exists",
|
||||
status_code=409,
|
||||
)
|
||||
|
||||
|
||||
class ValidationError(DocEngineException):
|
||||
"""Input validation error."""
|
||||
|
||||
def __init__(self, detail: str, errors: list[dict[str, Any]] | None = None) -> None:
|
||||
super().__init__(detail=detail, status_code=422)
|
||||
self.errors = errors or []
|
||||
|
||||
|
||||
class AuthenticationError(DocEngineException):
|
||||
"""Authentication failed."""
|
||||
|
||||
def __init__(self, detail: str = "Authentication failed") -> None:
|
||||
super().__init__(detail=detail, status_code=401)
|
||||
|
||||
|
||||
class AuthorizationError(DocEngineException):
|
||||
"""Authorization failed."""
|
||||
|
||||
def __init__(self, detail: str = "Insufficient permissions") -> None:
|
||||
super().__init__(detail=detail, status_code=403)
|
||||
|
||||
|
||||
class StorageError(DocEngineException):
|
||||
"""Storage operation failed."""
|
||||
|
||||
def __init__(self, detail: str) -> None:
|
||||
super().__init__(detail=detail, status_code=500)
|
||||
|
||||
|
||||
class ProcessingError(DocEngineException):
|
||||
"""Document processing failed."""
|
||||
|
||||
def __init__(self, detail: str, document_id: str | None = None) -> None:
|
||||
super().__init__(detail=detail, status_code=500)
|
||||
self.document_id = document_id
|
||||
|
||||
|
||||
class TemplateError(DocEngineException):
|
||||
"""Template operation failed."""
|
||||
|
||||
def __init__(self, detail: str) -> None:
|
||||
super().__init__(detail=detail, status_code=500)
|
||||
|
||||
|
||||
class RateLimitError(DocEngineException):
|
||||
"""Rate limit exceeded."""
|
||||
|
||||
def __init__(self, detail: str = "Rate limit exceeded. Please try again later.") -> None:
|
||||
super().__init__(detail=detail, status_code=429)
|
||||
|
||||
|
||||
class FileSizeError(DocEngineException):
|
||||
"""File exceeds maximum allowed size."""
|
||||
|
||||
def __init__(self, max_size_mb: int) -> None:
|
||||
super().__init__(
|
||||
detail=f"File size exceeds maximum allowed size of {max_size_mb}MB",
|
||||
status_code=413,
|
||||
)
|
||||
|
||||
|
||||
class UnsupportedFileTypeError(DocEngineException):
|
||||
"""File type not supported."""
|
||||
|
||||
def __init__(self, file_type: str) -> None:
|
||||
super().__init__(
|
||||
detail=f"File type '{file_type}' is not supported. Supported types: jpg, jpeg, png, tiff, pdf",
|
||||
status_code=415,
|
||||
)
|
||||
@@ -1,61 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import structlog
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
def setup_logging() -> None:
|
||||
"""Configure structlog for structured JSON logging."""
|
||||
shared_processors: list[structlog.types.Processor] = [
|
||||
structlog.contextvars.merge_contextvars,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
]
|
||||
|
||||
if settings.log_format == "json":
|
||||
renderer: structlog.types.Processor = structlog.processors.JSONRenderer()
|
||||
else:
|
||||
renderer = structlog.dev.ConsoleRenderer(colors=True)
|
||||
|
||||
structlog.configure(
|
||||
processors=[
|
||||
*shared_processors,
|
||||
structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
|
||||
],
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
wrapper_class=structlog.stdlib.BoundLogger,
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
|
||||
formatter = structlog.stdlib.ProcessorFormatter(
|
||||
processors=[
|
||||
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
|
||||
renderer,
|
||||
],
|
||||
foreign_pre_chain=shared_processors,
|
||||
)
|
||||
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(formatter)
|
||||
|
||||
root_logger = logging.getLogger()
|
||||
root_logger.handlers.clear()
|
||||
root_logger.addHandler(handler)
|
||||
root_logger.setLevel(getattr(logging, settings.log_level.upper(), logging.INFO))
|
||||
|
||||
# Reduce noise from third-party libraries
|
||||
for logger_name in ("uvicorn.access", "sqlalchemy.engine", "celery"):
|
||||
logging.getLogger(logger_name).setLevel(logging.WARNING)
|
||||
|
||||
|
||||
def get_logger(name: str | None = None) -> structlog.stdlib.BoundLogger:
|
||||
"""Get a structlog logger instance."""
|
||||
return structlog.get_logger(name)
|
||||
@@ -1,59 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from jose import JWTError, jwt
|
||||
from passlib.context import CryptContext
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
"""Hash a password using bcrypt."""
|
||||
return pwd_context.hash(password)
|
||||
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a plain password against a hashed password."""
|
||||
return pwd_context.verify(plain_password, hashed_password)
|
||||
|
||||
|
||||
def create_access_token(data: dict[str, Any], expires_delta: timedelta | None = None) -> str:
|
||||
"""Create a JWT access token."""
|
||||
to_encode = data.copy()
|
||||
expire = datetime.now(UTC) + (expires_delta or timedelta(minutes=settings.jwt_access_token_expire_minutes))
|
||||
to_encode.update({"exp": expire, "type": "access"})
|
||||
return jwt.encode(to_encode, settings.jwt_secret_key, algorithm=settings.jwt_algorithm)
|
||||
|
||||
|
||||
def create_refresh_token(data: dict[str, Any], expires_delta: timedelta | None = None) -> str:
|
||||
"""Create a JWT refresh token."""
|
||||
to_encode = data.copy()
|
||||
expire = datetime.now(UTC) + (expires_delta or timedelta(days=settings.jwt_refresh_token_expire_days))
|
||||
to_encode.update({
|
||||
"exp": expire,
|
||||
"type": "refresh",
|
||||
"jti": str(uuid.uuid4()),
|
||||
})
|
||||
return jwt.encode(to_encode, settings.jwt_secret_key, algorithm=settings.jwt_algorithm)
|
||||
|
||||
|
||||
def decode_token(token: str) -> dict[str, Any]:
|
||||
"""Decode and validate a JWT token."""
|
||||
try:
|
||||
payload = jwt.decode(token, settings.jwt_secret_key, algorithms=[settings.jwt_algorithm])
|
||||
return payload
|
||||
except JWTError as e:
|
||||
raise InvalidTokenError(str(e)) from e
|
||||
|
||||
|
||||
class InvalidTokenError(Exception):
|
||||
"""Raised when a JWT token is invalid or expired."""
|
||||
|
||||
def __init__(self, detail: str = "Invalid or expired token") -> None:
|
||||
self.detail = detail
|
||||
super().__init__(self.detail)
|
||||
BIN
docengine/app/events/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/events/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/events/__pycache__/handlers.cpython-313.pyc
Normal file
BIN
docengine/app/events/__pycache__/handlers.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,39 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from app.core.logging_config import get_logger, setup_logging
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def on_startup() -> None:
|
||||
"""Application startup event handler."""
|
||||
setup_logging()
|
||||
logger.info("application_starting", event="startup")
|
||||
|
||||
# Ensure storage directories exist
|
||||
from app.storage.provider import get_storage_provider
|
||||
try:
|
||||
get_storage_provider()
|
||||
logger.info("storage_initialized")
|
||||
except Exception as e:
|
||||
logger.error("storage_init_failed", error=str(e))
|
||||
|
||||
# Verify database connection
|
||||
from app.core.database import check_database_connection
|
||||
if check_database_connection():
|
||||
logger.info("database_connected")
|
||||
else:
|
||||
logger.error("database_connection_failed")
|
||||
|
||||
logger.info("application_started", event="startup_complete")
|
||||
|
||||
|
||||
def on_shutdown() -> None:
|
||||
"""Application shutdown event handler."""
|
||||
logger.info("application_shutting_down", event="shutdown")
|
||||
|
||||
# Cleanup resources
|
||||
from app.core.database import engine
|
||||
engine.dispose()
|
||||
|
||||
logger.info("application_stopped", event="shutdown_complete")
|
||||
@@ -1,107 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from collections.abc import AsyncGenerator
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI, Request, status
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.api.router import api_v1_router
|
||||
from app.core.config import settings
|
||||
from app.core.exceptions import DocEngineException
|
||||
from app.events.handlers import on_shutdown, on_startup
|
||||
from app.middleware.audit import AuditMiddleware
|
||||
from app.middleware.cors import setup_cors
|
||||
from app.middleware.metrics import setup_metrics
|
||||
from app.middleware.rate_limit import RateLimitMiddleware
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Application lifespan manager."""
|
||||
on_startup()
|
||||
yield
|
||||
on_shutdown()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title=settings.app_name,
|
||||
description="Document Template Recognition and Reconstruction System",
|
||||
version=settings.app_version,
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
openapi_url="/openapi.json",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
# Setup middleware (order matters: last added = first executed)
|
||||
setup_cors(app)
|
||||
app.add_middleware(AuditMiddleware)
|
||||
app.add_middleware(RateLimitMiddleware)
|
||||
|
||||
# Setup Prometheus metrics
|
||||
setup_metrics(app)
|
||||
|
||||
# Include API routes
|
||||
app.include_router(api_v1_router)
|
||||
|
||||
|
||||
# Exception handlers
|
||||
@app.exception_handler(DocEngineException)
|
||||
async def docengine_exception_handler(request: Request, exc: DocEngineException) -> JSONResponse:
|
||||
"""Handle application-specific exceptions."""
|
||||
return JSONResponse(
|
||||
status_code=exc.status_code,
|
||||
content={
|
||||
"detail": exc.detail,
|
||||
"error_code": type(exc).__name__,
|
||||
"extra": exc.extra if exc.extra else None,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_exception_handler(request: Request, exc: RequestValidationError) -> JSONResponse:
|
||||
"""Handle request validation errors."""
|
||||
errors = []
|
||||
for error in exc.errors():
|
||||
errors.append({
|
||||
"field": ".".join(str(loc) for loc in error.get("loc", [])),
|
||||
"message": error.get("msg", ""),
|
||||
"type": error.get("type", ""),
|
||||
})
|
||||
|
||||
return JSONResponse(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
|
||||
content={
|
||||
"detail": "Request validation failed",
|
||||
"error_code": "ValidationError",
|
||||
"errors": errors,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def general_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Handle unexpected exceptions."""
|
||||
return JSONResponse(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
content={
|
||||
"detail": "An unexpected error occurred" if settings.is_production else str(exc),
|
||||
"error_code": "InternalServerError",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
"app.main:app",
|
||||
host=settings.app_host,
|
||||
port=settings.app_port,
|
||||
reload=not settings.is_production,
|
||||
workers=1 if settings.app_debug else settings.app_workers,
|
||||
)
|
||||
BIN
docengine/app/middleware/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/middleware/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/middleware/__pycache__/audit.cpython-313.pyc
Normal file
BIN
docengine/app/middleware/__pycache__/audit.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/middleware/__pycache__/cors.cpython-313.pyc
Normal file
BIN
docengine/app/middleware/__pycache__/cors.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/middleware/__pycache__/metrics.cpython-313.pyc
Normal file
BIN
docengine/app/middleware/__pycache__/metrics.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/middleware/__pycache__/rate_limit.cpython-313.pyc
Normal file
BIN
docengine/app/middleware/__pycache__/rate_limit.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,71 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
|
||||
|
||||
from app.core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AuditMiddleware(BaseHTTPMiddleware):
|
||||
"""Middleware to log all API requests for audit purposes."""
|
||||
|
||||
EXCLUDED_PATHS = {"/api/v1/health", "/metrics", "/docs", "/openapi.json", "/redoc"}
|
||||
|
||||
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
|
||||
if request.url.path in self.EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
start_time = time.monotonic()
|
||||
|
||||
# Extract client info
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
user_agent = request.headers.get("user-agent", "unknown")
|
||||
|
||||
# Add request ID to request state
|
||||
request.state.request_id = request_id
|
||||
|
||||
logger.info(
|
||||
"request_started",
|
||||
request_id=request_id,
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
client_ip=client_ip,
|
||||
user_agent=user_agent[:200],
|
||||
)
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
duration_ms = (time.monotonic() - start_time) * 1000
|
||||
|
||||
logger.info(
|
||||
"request_completed",
|
||||
request_id=request_id,
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
status_code=response.status_code,
|
||||
duration_ms=round(duration_ms, 2),
|
||||
client_ip=client_ip,
|
||||
)
|
||||
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
response.headers["X-Process-Time"] = f"{duration_ms:.2f}ms"
|
||||
return response
|
||||
|
||||
except Exception as exc:
|
||||
duration_ms = (time.monotonic() - start_time) * 1000
|
||||
logger.exception(
|
||||
"request_failed",
|
||||
request_id=request_id,
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
duration_ms=round(duration_ms, 2),
|
||||
error=str(exc),
|
||||
)
|
||||
raise
|
||||
@@ -1,31 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
def setup_cors(app: FastAPI) -> None:
|
||||
"""Configure CORS middleware."""
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.cors_origins,
|
||||
allow_credentials=settings.cors_allow_credentials,
|
||||
allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"],
|
||||
allow_headers=[
|
||||
"Authorization",
|
||||
"Content-Type",
|
||||
"Accept",
|
||||
"X-Request-ID",
|
||||
"X-Requested-With",
|
||||
],
|
||||
expose_headers=[
|
||||
"X-Request-ID",
|
||||
"X-Process-Time",
|
||||
"X-RateLimit-Limit",
|
||||
"X-RateLimit-Remaining",
|
||||
"X-RateLimit-Reset",
|
||||
],
|
||||
max_age=600,
|
||||
)
|
||||
@@ -1,29 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import FastAPI
|
||||
from prometheus_fastapi_instrumentator import Instrumentator
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
def setup_metrics(app: FastAPI) -> None:
|
||||
"""Configure Prometheus metrics instrumentation."""
|
||||
if not settings.prometheus_enabled:
|
||||
return
|
||||
|
||||
instrumentator = Instrumentator(
|
||||
should_group_status_codes=True,
|
||||
should_ignore_untemplated=True,
|
||||
should_respect_env_var=False,
|
||||
excluded_handlers=["/metrics", "/api/v1/health", "/docs", "/openapi.json"],
|
||||
env_var_name="PROMETHEUS_ENABLED",
|
||||
inprogress_name="docengine_inprogress_requests",
|
||||
inprogress_labels=True,
|
||||
)
|
||||
|
||||
instrumentator.instrument(app).expose(
|
||||
app,
|
||||
endpoint="/metrics",
|
||||
include_in_schema=False,
|
||||
should_gzip=True,
|
||||
)
|
||||
@@ -1,72 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import defaultdict
|
||||
|
||||
from fastapi import Request, Response, status
|
||||
from fastapi.responses import JSONResponse
|
||||
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class RateLimitMiddleware(BaseHTTPMiddleware):
|
||||
"""Token bucket rate limiter per client IP."""
|
||||
|
||||
EXCLUDED_PATHS = {"/api/v1/health", "/metrics", "/docs", "/openapi.json", "/redoc"}
|
||||
|
||||
def __init__(self, app, max_requests: int | None = None, window_seconds: int | None = None) -> None: # noqa: ANN001
|
||||
super().__init__(app)
|
||||
self.max_requests = max_requests or settings.rate_limit_requests
|
||||
self.window_seconds = window_seconds or settings.rate_limit_window_seconds
|
||||
self._requests: dict[str, list[float]] = defaultdict(list)
|
||||
|
||||
def _clean_old_requests(self, client_ip: str, now: float) -> None:
|
||||
"""Remove requests outside the current window."""
|
||||
cutoff = now - self.window_seconds
|
||||
self._requests[client_ip] = [
|
||||
ts for ts in self._requests[client_ip] if ts > cutoff
|
||||
]
|
||||
|
||||
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
|
||||
if request.url.path in self.EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
now = time.monotonic()
|
||||
|
||||
self._clean_old_requests(client_ip, now)
|
||||
|
||||
if len(self._requests[client_ip]) >= self.max_requests:
|
||||
logger.warning(
|
||||
"rate_limit_exceeded",
|
||||
client_ip=client_ip,
|
||||
path=request.url.path,
|
||||
request_count=len(self._requests[client_ip]),
|
||||
)
|
||||
return JSONResponse(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
content={
|
||||
"detail": "Rate limit exceeded. Please try again later.",
|
||||
"retry_after_seconds": self.window_seconds,
|
||||
},
|
||||
headers={
|
||||
"Retry-After": str(self.window_seconds),
|
||||
"X-RateLimit-Limit": str(self.max_requests),
|
||||
"X-RateLimit-Remaining": "0",
|
||||
"X-RateLimit-Reset": str(int(now + self.window_seconds)),
|
||||
},
|
||||
)
|
||||
|
||||
self._requests[client_ip].append(now)
|
||||
remaining = self.max_requests - len(self._requests[client_ip])
|
||||
|
||||
response = await call_next(request)
|
||||
response.headers["X-RateLimit-Limit"] = str(self.max_requests)
|
||||
response.headers["X-RateLimit-Remaining"] = str(remaining)
|
||||
response.headers["X-RateLimit-Reset"] = str(int(now + self.window_seconds))
|
||||
|
||||
return response
|
||||
@@ -1,43 +0,0 @@
|
||||
from app.models.user import AuditLog, RefreshToken, Role, User, user_roles_table
|
||||
from app.models.document import (
|
||||
Document,
|
||||
DocumentImage,
|
||||
DocumentPage,
|
||||
DocumentTable,
|
||||
DocumentTextBlock,
|
||||
TemplateMatch,
|
||||
)
|
||||
from app.models.template import (
|
||||
DocumentCell,
|
||||
DocumentFormat,
|
||||
DocumentRegion,
|
||||
ImageRegion,
|
||||
TableColumn,
|
||||
TableFormat,
|
||||
TableRow,
|
||||
TemplateFingerprint,
|
||||
Watermark,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
"Role",
|
||||
"RefreshToken",
|
||||
"AuditLog",
|
||||
"user_roles_table",
|
||||
"Document",
|
||||
"DocumentPage",
|
||||
"DocumentTextBlock",
|
||||
"DocumentImage",
|
||||
"DocumentTable",
|
||||
"TemplateMatch",
|
||||
"DocumentFormat",
|
||||
"DocumentCell",
|
||||
"DocumentRegion",
|
||||
"TableFormat",
|
||||
"TableColumn",
|
||||
"TableRow",
|
||||
"Watermark",
|
||||
"ImageRegion",
|
||||
"TemplateFingerprint",
|
||||
]
|
||||
BIN
docengine/app/models/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/models/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/models/__pycache__/base.cpython-313.pyc
Normal file
BIN
docengine/app/models/__pycache__/base.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/models/__pycache__/document.cpython-313.pyc
Normal file
BIN
docengine/app/models/__pycache__/document.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/models/__pycache__/template.cpython-313.pyc
Normal file
BIN
docengine/app/models/__pycache__/template.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/models/__pycache__/user.cpython-313.pyc
Normal file
BIN
docengine/app/models/__pycache__/user.cpython-313.pyc
Normal file
Binary file not shown.
@@ -1,39 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import DateTime, func
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.core.database import Base
|
||||
|
||||
|
||||
class TimestampMixin:
|
||||
"""Mixin providing created_at and updated_at timestamps."""
|
||||
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(UTC),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(UTC),
|
||||
server_default=func.now(),
|
||||
onupdate=lambda: datetime.now(UTC),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
|
||||
class UUIDPrimaryKeyMixin:
|
||||
"""Mixin providing a UUID primary key."""
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
nullable=False,
|
||||
)
|
||||
@@ -1,244 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import BigInteger, Boolean, DateTime, Float, ForeignKey, Integer, String, Text, func
|
||||
from sqlalchemy.dialects.postgresql import JSONB, UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.core.database import Base
|
||||
from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin
|
||||
|
||||
|
||||
class Document(Base, UUIDPrimaryKeyMixin, TimestampMixin):
|
||||
"""Uploaded document record."""
|
||||
|
||||
__tablename__ = "documents"
|
||||
|
||||
filename: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
original_filename: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
content_type: Mapped[str] = mapped_column(String(100), nullable=False)
|
||||
file_size: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||||
checksum: Mapped[str] = mapped_column(String(128), nullable=False, index=True)
|
||||
storage_path: Mapped[str] = mapped_column(String(1024), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(50),
|
||||
default="pending",
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
is_scanned: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
|
||||
document_metadata: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
uploaded_by: Mapped[uuid.UUID | None] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
index=True,
|
||||
)
|
||||
|
||||
pages: Mapped[list[DocumentPage]] = relationship(
|
||||
"DocumentPage",
|
||||
back_populates="document",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DocumentPage.page_number",
|
||||
lazy="selectin",
|
||||
)
|
||||
template_matches: Mapped[list[TemplateMatch]] = relationship(
|
||||
"TemplateMatch",
|
||||
back_populates="document",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="dynamic",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Document(id={self.id}, filename={self.original_filename}, status={self.status})>"
|
||||
|
||||
|
||||
class DocumentPage(Base, UUIDPrimaryKeyMixin):
|
||||
"""Individual page within a document."""
|
||||
|
||||
__tablename__ = "document_pages"
|
||||
|
||||
document_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("documents.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True)
|
||||
text_content: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
document: Mapped[Document] = relationship("Document", back_populates="pages")
|
||||
text_blocks: Mapped[list[DocumentTextBlock]] = relationship(
|
||||
"DocumentTextBlock",
|
||||
back_populates="page",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DocumentTextBlock.sequence",
|
||||
lazy="selectin",
|
||||
)
|
||||
images: Mapped[list[DocumentImage]] = relationship(
|
||||
"DocumentImage",
|
||||
back_populates="page",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
tables: Mapped[list[DocumentTable]] = relationship(
|
||||
"DocumentTable",
|
||||
back_populates="page",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentPage(id={self.id}, document_id={self.document_id}, page={self.page_number})>"
|
||||
|
||||
|
||||
class DocumentTextBlock(Base, UUIDPrimaryKeyMixin):
|
||||
"""Extracted text block from a document page."""
|
||||
|
||||
__tablename__ = "document_text_blocks"
|
||||
|
||||
page_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_pages.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
text: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
font_family: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
font_size: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
font_color: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
font_style: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
block_type: Mapped[str] = mapped_column(
|
||||
String(50),
|
||||
default="text",
|
||||
nullable=False,
|
||||
)
|
||||
sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="text_blocks")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentTextBlock(id={self.id}, type={self.block_type}, text={self.text[:50]})>"
|
||||
|
||||
|
||||
class DocumentImage(Base, UUIDPrimaryKeyMixin):
|
||||
"""Extracted image from a document page."""
|
||||
|
||||
__tablename__ = "document_images"
|
||||
|
||||
page_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_pages.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
image_path: Mapped[str] = mapped_column(String(1024), nullable=False)
|
||||
image_type: Mapped[str] = mapped_column(
|
||||
String(50),
|
||||
default="figure",
|
||||
nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="images")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentImage(id={self.id}, type={self.image_type})>"
|
||||
|
||||
|
||||
class DocumentTable(Base, UUIDPrimaryKeyMixin):
|
||||
"""Extracted table from a document page."""
|
||||
|
||||
__tablename__ = "document_tables"
|
||||
|
||||
page_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_pages.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
rows: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
columns: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
data: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="tables")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentTable(id={self.id}, rows={self.rows}, cols={self.columns})>"
|
||||
|
||||
|
||||
class TemplateMatch(Base, UUIDPrimaryKeyMixin):
|
||||
"""Template matching result for a document."""
|
||||
|
||||
__tablename__ = "template_matches"
|
||||
|
||||
document_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("documents.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
confidence_score: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
match_details: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
selected: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
document: Mapped[Document] = relationship("Document", back_populates="template_matches")
|
||||
template: Mapped[DocumentFormat] = relationship("DocumentFormat")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<TemplateMatch(id={self.id}, doc={self.document_id}, score={self.confidence_score})>"
|
||||
@@ -1,378 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Integer, String, Text, func
|
||||
from sqlalchemy.dialects.postgresql import JSONB, UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.core.database import Base
|
||||
from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin
|
||||
|
||||
|
||||
class DocumentFormat(Base, UUIDPrimaryKeyMixin, TimestampMixin):
|
||||
"""Reusable document template format."""
|
||||
|
||||
__tablename__ = "document_formats"
|
||||
|
||||
name: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
|
||||
description: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
page_width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
page_height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
page_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
margin_top: Mapped[float] = mapped_column(Float, default=72.0, nullable=False)
|
||||
margin_right: Mapped[float] = mapped_column(Float, default=72.0, nullable=False)
|
||||
margin_bottom: Mapped[float] = mapped_column(Float, default=72.0, nullable=False)
|
||||
margin_left: Mapped[float] = mapped_column(Float, default=72.0, nullable=False)
|
||||
fingerprint: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
source_document_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("documents.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
index=True,
|
||||
)
|
||||
version: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True)
|
||||
created_by: Mapped[uuid.UUID | None] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
)
|
||||
|
||||
cells: Mapped[list[DocumentCell]] = relationship(
|
||||
"DocumentCell",
|
||||
back_populates="format",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DocumentCell.sequence",
|
||||
lazy="selectin",
|
||||
)
|
||||
regions: Mapped[list[DocumentRegion]] = relationship(
|
||||
"DocumentRegion",
|
||||
back_populates="format",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="DocumentRegion.sequence",
|
||||
lazy="selectin",
|
||||
)
|
||||
table_formats: Mapped[list[TableFormat]] = relationship(
|
||||
"TableFormat",
|
||||
back_populates="format",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
watermarks: Mapped[list[Watermark]] = relationship(
|
||||
"Watermark",
|
||||
back_populates="format",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
image_regions: Mapped[list[ImageRegion]] = relationship(
|
||||
"ImageRegion",
|
||||
back_populates="format",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
fingerprint_record: Mapped[TemplateFingerprint | None] = relationship(
|
||||
"TemplateFingerprint",
|
||||
back_populates="format",
|
||||
uselist=False,
|
||||
cascade="all, delete-orphan",
|
||||
lazy="selectin",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentFormat(id={self.id}, name={self.name}, v{self.version})>"
|
||||
|
||||
|
||||
class DocumentCell(Base, UUIDPrimaryKeyMixin):
|
||||
"""Cell definition within a document template."""
|
||||
|
||||
__tablename__ = "document_cells"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
row_no: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
column_no: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
data_type: Mapped[str] = mapped_column(String(50), default="text", nullable=False)
|
||||
font_family: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
font_size: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
font_style: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
font_color: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
background_color: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
border_top: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
border_right: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
border_bottom: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
border_left: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
padding_top: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
|
||||
padding_right: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
|
||||
padding_bottom: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
|
||||
padding_left: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
|
||||
alignment: Mapped[str] = mapped_column(String(20), default="left", nullable=False)
|
||||
vertical_alignment: Mapped[str] = mapped_column(String(20), default="top", nullable=False)
|
||||
rowspan: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
colspan: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
static_text: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
field_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
is_dynamic: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="cells")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentCell(id={self.id}, page={self.page_number}, row={self.row_no}, col={self.column_no})>"
|
||||
|
||||
|
||||
class DocumentRegion(Base, UUIDPrimaryKeyMixin):
|
||||
"""Region definition within a document template."""
|
||||
|
||||
__tablename__ = "document_regions"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
region_type: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
content: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="regions")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<DocumentRegion(id={self.id}, type={self.region_type}, page={self.page_number})>"
|
||||
|
||||
|
||||
class TableFormat(Base, UUIDPrimaryKeyMixin):
|
||||
"""Table definition within a document template."""
|
||||
|
||||
__tablename__ = "table_formats"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
rows: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
columns: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
border_style: Mapped[str] = mapped_column(String(50), default="solid", nullable=False)
|
||||
border_width: Mapped[float] = mapped_column(Float, default=1.0, nullable=False)
|
||||
border_color: Mapped[str] = mapped_column(String(50), default="#000000", nullable=False)
|
||||
header_rows: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="table_formats")
|
||||
table_columns: Mapped[list[TableColumn]] = relationship(
|
||||
"TableColumn",
|
||||
back_populates="table_format",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="TableColumn.column_index",
|
||||
lazy="selectin",
|
||||
)
|
||||
table_rows: Mapped[list[TableRow]] = relationship(
|
||||
"TableRow",
|
||||
back_populates="table_format",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="TableRow.row_index",
|
||||
lazy="selectin",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<TableFormat(id={self.id}, rows={self.rows}, cols={self.columns})>"
|
||||
|
||||
|
||||
class TableColumn(Base, UUIDPrimaryKeyMixin):
|
||||
"""Column definition within a table format."""
|
||||
|
||||
__tablename__ = "table_columns"
|
||||
|
||||
table_format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("table_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
column_index: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
header_text: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
data_type: Mapped[str] = mapped_column(String(50), default="text", nullable=False)
|
||||
alignment: Mapped[str] = mapped_column(String(20), default="left", nullable=False)
|
||||
font_family: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
font_size: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
table_format: Mapped[TableFormat] = relationship("TableFormat", back_populates="table_columns")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<TableColumn(id={self.id}, index={self.column_index}, header={self.header_text})>"
|
||||
|
||||
|
||||
class TableRow(Base, UUIDPrimaryKeyMixin):
|
||||
"""Row definition within a table format."""
|
||||
|
||||
__tablename__ = "table_rows"
|
||||
|
||||
table_format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("table_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
row_index: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, default=20.0, nullable=False)
|
||||
is_header: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
background_color: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
table_format: Mapped[TableFormat] = relationship("TableFormat", back_populates="table_rows")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<TableRow(id={self.id}, index={self.row_index}, is_header={self.is_header})>"
|
||||
|
||||
|
||||
class Watermark(Base, UUIDPrimaryKeyMixin):
|
||||
"""Watermark definition within a document template."""
|
||||
|
||||
__tablename__ = "watermarks"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
text: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
opacity: Mapped[float] = mapped_column(Float, default=0.3, nullable=False)
|
||||
rotation: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
|
||||
font_family: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
font_size: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
font_color: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="watermarks")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Watermark(id={self.id}, text={self.text})>"
|
||||
|
||||
|
||||
class ImageRegion(Base, UUIDPrimaryKeyMixin):
|
||||
"""Image region within a document template."""
|
||||
|
||||
__tablename__ = "image_regions"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
height: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True)
|
||||
image_type: Mapped[str] = mapped_column(String(50), default="figure", nullable=False)
|
||||
is_static: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
field_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="image_regions")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<ImageRegion(id={self.id}, type={self.image_type}, page={self.page_number})>"
|
||||
|
||||
|
||||
class TemplateFingerprint(Base, UUIDPrimaryKeyMixin):
|
||||
"""Layout fingerprint for template matching."""
|
||||
|
||||
__tablename__ = "template_fingerprints"
|
||||
|
||||
format_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("document_formats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
unique=True,
|
||||
index=True,
|
||||
)
|
||||
page_dimensions: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
logo_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
header_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
footer_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
table_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
cell_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
|
||||
fingerprint_hash: Mapped[str] = mapped_column(String(256), nullable=False, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="fingerprint_record")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<TemplateFingerprint(id={self.id}, format_id={self.format_id}, hash={self.fingerprint_hash[:16]})>"
|
||||
@@ -1,136 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, String, Table, Text, func
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.core.database import Base
|
||||
from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin
|
||||
|
||||
user_roles_table = Table(
|
||||
"user_roles",
|
||||
Base.metadata,
|
||||
mapped_column("user_id", UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True),
|
||||
mapped_column("role_id", UUID(as_uuid=True), ForeignKey("roles.id", ondelete="CASCADE"), primary_key=True),
|
||||
)
|
||||
|
||||
|
||||
class User(Base, UUIDPrimaryKeyMixin, TimestampMixin):
|
||||
"""User account model."""
|
||||
|
||||
__tablename__ = "users"
|
||||
|
||||
username: Mapped[str] = mapped_column(String(150), unique=True, nullable=False, index=True)
|
||||
email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False, index=True)
|
||||
hashed_password: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
full_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
is_superuser: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
last_login: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
roles: Mapped[list[Role]] = relationship(
|
||||
"Role",
|
||||
secondary=user_roles_table,
|
||||
back_populates="users",
|
||||
lazy="joined",
|
||||
)
|
||||
refresh_tokens: Mapped[list[RefreshToken]] = relationship(
|
||||
"RefreshToken",
|
||||
back_populates="user",
|
||||
cascade="all, delete-orphan",
|
||||
lazy="dynamic",
|
||||
)
|
||||
audit_logs: Mapped[list[AuditLog]] = relationship(
|
||||
"AuditLog",
|
||||
back_populates="user",
|
||||
lazy="dynamic",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<User(id={self.id}, username={self.username})>"
|
||||
|
||||
|
||||
class Role(Base, UUIDPrimaryKeyMixin):
|
||||
"""User role model."""
|
||||
|
||||
__tablename__ = "roles"
|
||||
|
||||
name: Mapped[str] = mapped_column(String(50), unique=True, nullable=False, index=True)
|
||||
description: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
users: Mapped[list[User]] = relationship(
|
||||
"User",
|
||||
secondary=user_roles_table,
|
||||
back_populates="roles",
|
||||
lazy="dynamic",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Role(id={self.id}, name={self.name})>"
|
||||
|
||||
|
||||
class RefreshToken(Base, UUIDPrimaryKeyMixin):
|
||||
"""JWT refresh token storage."""
|
||||
|
||||
__tablename__ = "refresh_tokens"
|
||||
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
token: Mapped[str] = mapped_column(String(512), unique=True, nullable=False, index=True)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
revoked: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
user: Mapped[User] = relationship("User", back_populates="refresh_tokens")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<RefreshToken(id={self.id}, user_id={self.user_id}, revoked={self.revoked})>"
|
||||
|
||||
|
||||
class AuditLog(Base, UUIDPrimaryKeyMixin):
|
||||
"""Audit trail for user actions."""
|
||||
|
||||
__tablename__ = "audit_logs"
|
||||
|
||||
user_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
index=True,
|
||||
)
|
||||
action: Mapped[str] = mapped_column(String(100), nullable=False, index=True)
|
||||
resource_type: Mapped[str] = mapped_column(String(100), nullable=False, index=True)
|
||||
resource_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
details: Mapped[dict | None] = mapped_column(type_=Text, nullable=True)
|
||||
ip_address: Mapped[str | None] = mapped_column(String(45), nullable=True)
|
||||
user_agent: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=func.now(),
|
||||
server_default=func.now(),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
|
||||
user: Mapped[User | None] = relationship("User", back_populates="audit_logs")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<AuditLog(id={self.id}, action={self.action}, resource={self.resource_type})>"
|
||||
BIN
docengine/app/repositories/__pycache__/__init__.cpython-313.pyc
Normal file
BIN
docengine/app/repositories/__pycache__/__init__.cpython-313.pyc
Normal file
Binary file not shown.
BIN
docengine/app/repositories/__pycache__/base.cpython-313.pyc
Normal file
BIN
docengine/app/repositories/__pycache__/base.cpython-313.pyc
Normal file
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user