commit

.env.prod

@@ -1,13 +0,0 @@
-# Database Configuration
-# Connecting to existing 'postgres-db' container in 'arbit-app_arbit-network'
-DB_USER=postgres
-DB_PASSWORD='M@tr!x#149@dm!N'
-DB_NAME=ocr
-DB_HOST=postgres-db
-DB_PORT=5432
-
-# Mail Configuration (IMAP)
-MAIL_SERVER=imap.gmail.com
-MAIL_USERNAME=matrixinfotech.it@gmail.com
-MAIL_PASSWORD=qtxsthbxbisgcmqu
-MAIL_PORT=993

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "dotrush.roslyn.projectOrSolutionFiles": []
+}

DEPLOYMENT.md

@@ -1,95 +0,0 @@
-# OCR Application Deployment Guide
-
-This guide describes how to deploy the OCR application on a Linux host (e.g., Ubuntu/Debian).
-
-## Architecture
-- **Backend**: Containerized (FastAPI, Python 3.9).
-- **Database**: Containerized (PostgreSQL 15).
-- **Frontend**: Static files (Angular) served by Host Nginx.
-- **Reverse Proxy**: Host Nginx proxies requests to Frontend (Static) and Backend (API).
-
-## Prerequisites
-- **Deployment Host**: Docker & Docker Compose, Nginx.
-- **Build Machine**: Node.js & NPM (to run the package script).
-
----
-
-## 1. Full Stack Deployment (Docker)
-
-The app connects to your **existing Postgres container** (`postgres-db`) in the `arbit-app_arbit-network`.
-
-1.  **Create the Database**:
-    Since we are using an existing postgres instance, we must manually create the `ocr_db`.
-    ```bash
-    # Run this on your host to create the DB inside the existing container
-    docker exec -it postgres-db psql -U postgres -c "CREATE DATABASE ocr_db;"
-    ```
-
-2.  **Navigate to the packaged directory**:
-    ```bash
-    cd ocr_build
-    ```
-
-3.  **Verify Environment**:
-    Ensure `.env.prod` exists and points to `DB_HOST=postgres-db`.
-    **Also configure your Email credentials** in `.env.prod` if you want the Mailbox feature to work (Gmail requires an App Password).
-    ```bash
-    cat .env.prod
-    ```
-
-4.  **Start App Containers**:
-    This will start `ocr_backend` and `ocr_frontend`.
-    ```bash
-    # Build and start in detached mode
-    docker-compose up -d --build
-    ```
-
-5.  **Verify Status**:
-    ```bash
-    docker-compose ps
-    ```
-    You should see `ocr_backend` and `ocr_frontend` running.
-
----
-
-## 2. Nginx Configuration (Host Reverse Proxy)
-
-Since the Frontend is now running in a container on port 8080 (serving `/ocrf/`), we configure the Host Nginx to proxy traffic to it.
-
-1.  **Create Config File**:
-    Copy the provided config to `/etc/nginx/sites-available/ocr`.
-    ```bash
-    sudo cp nginx_host.conf /etc/nginx/sites-available/ocr
-    ```
-
-2.  **Enable Site**:
-    ```bash
-    sudo ln -s /etc/nginx/sites-available/ocr /etc/nginx/sites-enabled/
-    ```
-
-3.  **Test & Reload**:
-    ```bash
-    sudo nginx -t
-    sudo systemctl reload nginx
-    ```
-
----
-
-## 3. Verification
-
-- **Url**: `http://app.technobeesolutions.in/ocrf/` (Proxies to Frontend Container)
-- **API**: `http://app.technobeesolutions.in/ocrb/` (Proxies to Backend Container)
-
----
-
-## 4. Troubleshooting
-
-- **Logs**:
-    ```bash
-    docker-compose logs -f backend
-    ```
-- **Database**:
-    Connect via the existing container:
-    ```bash
-    docker exec -it postgres-db psql -U postgres -d ocr_db
-    ```

SESSION_HANDOFF.md

@@ -0,0 +1,42 @@
+# Session Handoff & Context
+
+**Date Saved**: June 7, 2026
+**Conversation ID**: 43f3aa27-9124-446e-9713-4e31e083b4f5
+
+## Project: Template Mapping Engine (OCR)
+A system designed to extract document layouts via OCR and allow users to map specific bounding box regions to template fields using a drag-and-drop UI. Future documents from the same vendor are automatically recognized and mapped using a heuristic template engine.
+
+## What Has Been Completed So Far
+
+### 1. Backend Startup & Dependency Fixes:
+- **Auto-Bootstrapping**: Updated `run_docengine.sh` and `run_backend.sh` to automatically set up virtual environments and install missing dependencies if key modules (`uvicorn`, `numpy`, etc.) are missing.
+- **Python 3.13 Compatibility**: Upgraded `paddlepaddle` in `docengine/requirements.txt` to `>=3.0.0` to support Python 3.13 on macOS ARM64. Added missing dependencies (`numpy`, `opencv-python-headless`, `pdf2image`) to `backend/requirements.txt`.
+- **CORS Config Fix**: Corrected a missing double-quote in `CORS_ORIGINS` in both `.env` and `.env.example` in `docengine/`.
+
+### 2. Backend Code Integrity & Framework Alignment:
+- **SQLAlchemy 2.0 / Imperative Mapping Fix**: Updated `user_roles_table` in `docengine/app/models/user.py` to use SQLAlchemy `Column` objects instead of `mapped_column`, preventing a database initialization crash.
+- **Structlog Parameter Fix**: Replaced the `event` keyword argument with `phase` in `docengine/app/events/handlers.py` to prevent a duplicate parameter `TypeError` from the structlog library.
+- **Optional Development Auth Bypass**: Modified `get_current_user` in `docengine/app/core/dependencies.py` to automatically fallback to the first database user (or a fake dev admin model) when no authorization token is supplied and `APP_ENV` is set to `"development"`. This prevents `401 Unauthorized` errors when testing endpoints locally.
+
+### 3. Frontend Angular Implementation:
+- **API Endpoint Alignments**: Removed the duplicate `/api` prefix from endpoints in `TemplateService.ts` (e.g. changing `${this.apiUrl}/api/documents/upload` to `${this.apiUrl}/documents/upload`). This resolves the `404 Not Found` error by aligning with the backend prefix `/api/v1/documents/upload`.
+
+---
+
+## Where to Pick Up Next
+1. **Initialize Database Tables**:
+   - The database currently lacks the necessary schema tables (evidenced by the error: `relation "admin.users" does not exist`).
+   - Run the setup command to build the tables and apply migrations:
+     ```bash
+     ./run_docengine.sh setup
+     ```
+2. **Retest Upload Flow**:
+   - Once database tables exist, perform a document upload from the Angular frontend to visually verify the OCR bounding boxes and layout mapper.
+3. **Verify Heuristic Template Recognition**:
+   - Test the template matching and layout coordinate storage workflows.
+
+---
+
+## How to Resume with AI Assistant
+To resume this session in a new chat, you can tell the AI:
+*"Please read `SESSION_HANDOFF.md` in the root of the OCR project to get context on the Template Engine we were building and the startup fixes, and let's continue."*

backend/.env

@@ -1,7 +1,7 @@
 DB_USER=postgres
-DB_PASSWORD=M@tr!x#149@dm!N
-DB_HOST=192.168.0.111
-DB_PORT=7925
+DB_PASSWORD=M@triXPostgr3s@6202
+DB_HOST=103.125.129.116
+DB_PORT=5432
 DB_NAME=ocr
 
 # Mail Configuration (Gmail)

backend/Dockerfile

@@ -1,30 +0,0 @@
-# Use official lightweight Python image
-FROM python:3.9-slim
-
-# Install system dependencies
-# tesseract-ocr: for pytesseract
-# poppler-utils: for pdf2image
-# libtesseract-dev: development headers
-RUN apt-get update && apt-get install -y \
-    tesseract-ocr \
-    poppler-utils \
-    libtesseract-dev \
-    && rm -rf /var/lib/apt/lists/*
-
-# Set working directory
-WORKDIR /app
-
-# Copy requirements first to leverage Docker cache
-COPY requirements.txt .
-
-# Install Python dependencies
-RUN pip install --no-cache-dir -r requirements.txt
-
-# Copy the rest of the application code
-COPY . .
-
-# Expose port (default for Uvicorn)
-EXPOSE 8000
-
-# Run the application
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

backend/alembic.ini

@@ -0,0 +1,149 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts.
+# this is typically a path given in POSIX (e.g. forward slashes)
+# format, relative to the token %(here)s which refers to the location of this
+# ini file
+script_location = %(here)s/alembic
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+# Or organize into date-based subdirectories (requires recursive_version_locations = true)
+# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.  for multiple paths, the path separator
+# is defined by "path_separator" below.
+prepend_sys_path = .
+
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the tzdata library which can be installed by adding
+# `alembic[tz]` to the pip requirements.
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to <script_location>/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "path_separator"
+# below.
+# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
+
+# path_separator; This indicates what character is used to split lists of file
+# paths, including version_locations and prepend_sys_path within configparser
+# files such as alembic.ini.
+# The default rendered in new alembic.ini files is "os", which uses os.pathsep
+# to provide os-dependent path splitting.
+#
+# Note that in order to support legacy alembic.ini files, this default does NOT
+# take place if path_separator is not present in alembic.ini.  If this
+# option is omitted entirely, fallback logic is as follows:
+#
+# 1. Parsing of the version_locations option falls back to using the legacy
+#    "version_path_separator" key, which if absent then falls back to the legacy
+#    behavior of splitting on spaces and/or commas.
+# 2. Parsing of the prepend_sys_path option falls back to the legacy
+#    behavior of splitting on spaces, commas, or colons.
+#
+# Valid values for path_separator are:
+#
+# path_separator = :
+# path_separator = ;
+# path_separator = space
+# path_separator = newline
+#
+# Use os.pathsep. Default configuration used for new projects.
+path_separator = os
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# database URL.  This is consumed by the user-maintained env.py script only.
+# other means of configuring database URLs may be customized within the env.py
+# file.
+sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
+# hooks = ruff
+# ruff.type = module
+# ruff.module = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Alternatively, use the exec runner to execute a binary found on your PATH
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration.  This is also consumed by the user-maintained
+# env.py script only.
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARNING
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARNING
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

backend/alembic/README

@@ -0,0 +1 @@
+Generic single-database configuration.

backend/alembic/env.py

@@ -0,0 +1,87 @@
+from logging.config import fileConfig
+
+from sqlalchemy import engine_from_config
+from sqlalchemy import pool
+
+from alembic import context
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+if config.config_file_name is not None:
+    fileConfig(config.config_file_name)
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+import os
+import sys
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from database import Base, DATABASE_URL
+import models.template_models
+
+target_metadata = Base.metadata
+config.set_main_option("sqlalchemy.url", DATABASE_URL.replace("%", "%%"))
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline() -> None:
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+    connectable = engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix="sqlalchemy.",
+        poolclass=pool.NullPool,
+    )
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection=connection, 
+            target_metadata=target_metadata,
+            include_schemas=True,
+            version_table_schema='public'
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

backend/alembic/script.py.mako

@@ -0,0 +1,28 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
+branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
+depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    ${downgrades if downgrades else "pass"}

backend/alembic/versions/91b2545f5a30_add_templates_schema.py

@@ -0,0 +1,28 @@
+"""Add templates schema
+
+Revision ID: 91b2545f5a30
+Revises: 
+Create Date: 2026-06-01 16:28:57.181829
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import os
+
+# revision identifiers, used by Alembic.
+revision: str = '91b2545f5a30'
+down_revision: Union[str, None] = None
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+def upgrade() -> None:
+    # Read and execute the SQL schema script
+    sql_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), 'sql', '01_templates_schema.sql')
+    with open(sql_path, 'r') as f:
+        sql = f.read()
+    op.execute(sql)
+
+def downgrade() -> None:
+    op.execute("DROP SCHEMA IF EXISTS templates CASCADE;")

backend/api/documents.py

@@ -0,0 +1,54 @@
+import json
+from fastapi import APIRouter, Depends, HTTPException, status, UploadFile, File
+from sqlalchemy.orm import Session
+from typing import List
+
+from database import get_db
+from models.template_models import TemplateDocument, DocumentLayout
+from schemas.template_schemas import DocumentSchema, DocumentLayoutSchema, TemplateRecognitionResult
+from engine.ocr.document_processor import DocumentProcessor
+from engine.recognition.template_engine import TemplateRecognitionEngine
+
+router = APIRouter(prefix="/api/documents", tags=["Documents"])
+
+@router.post("/upload", response_model=DocumentSchema, status_code=status.HTTP_201_CREATED)
+async def upload_document(file: UploadFile = File(...), db: Session = Depends(get_db)):
+    # 1. Create document record
+    content = await file.read()
+    file_hash = hash(content) # Simple hash for demo
+    
+    db_doc = TemplateDocument(
+        document_name=file.filename,
+        document_type=file.content_type,
+        file_name=file.filename,
+        file_hash=str(file_hash),
+        status="processing"
+    )
+    db.add(db_doc)
+    db.commit()
+    db.refresh(db_doc)
+    
+    # 2. Process File (OCR & Layout)
+    processor = DocumentProcessor(db)
+    processor.process_file(db_doc.pk_document_id, file.filename, content)
+    
+    db.refresh(db_doc)
+    return db_doc
+
+@router.get("/{document_id}", response_model=DocumentSchema)
+def get_document(document_id: int, db: Session = Depends(get_db)):
+    doc = db.query(TemplateDocument).filter(TemplateDocument.pk_document_id == document_id).first()
+    if not doc:
+        raise HTTPException(status_code=404, detail="Document not found")
+    return doc
+
+@router.get("/{document_id}/layout", response_model=List[DocumentLayoutSchema])
+def get_document_layout(document_id: int, db: Session = Depends(get_db)):
+    layouts = db.query(DocumentLayout).filter(DocumentLayout.fk_document_id == document_id).order_by(DocumentLayout.sequence_no).all()
+    return layouts
+
+@router.post("/{document_id}/recognize", response_model=TemplateRecognitionResult)
+def recognize_template(document_id: int, db: Session = Depends(get_db)):
+    engine = TemplateRecognitionEngine(db)
+    return engine.recognize_template(document_id)
+

backend/api/templates.py

@@ -0,0 +1,53 @@
+from fastapi import APIRouter, Depends, HTTPException, status
+from sqlalchemy.orm import Session
+from typing import List
+
+from database import get_db
+from schemas.template_schemas import (
+    TemplateCreate, TemplateSchema, 
+    TemplateFieldCreate, TemplateFieldSchema,
+    TemplateFieldMappingCreate, TemplateFieldMappingSchema
+)
+from services.template_service import TemplateService
+
+router = APIRouter(prefix="/api/templates", tags=["Templates"])
+
+@router.post("", response_model=TemplateSchema, status_code=status.HTTP_201_CREATED)
+def create_template(template_data: TemplateCreate, db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    return service.create_template(template_data)
+
+@router.get("", response_model=List[TemplateSchema])
+def get_templates(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    return service.get_all_templates(skip, limit)
+
+@router.get("/{template_id}", response_model=TemplateSchema)
+def get_template(template_id: int, db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    template = service.get_template(template_id)
+    if not template:
+        raise HTTPException(status_code=404, detail="Template not found")
+    return template
+
+@router.delete("/{template_id}", status_code=status.HTTP_204_NO_CONTENT)
+def delete_template(template_id: int, db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    if not service.delete_template(template_id):
+        raise HTTPException(status_code=404, detail="Template not found")
+    return None
+
+@router.post("/{template_id}/fields", response_model=TemplateFieldSchema, status_code=status.HTTP_201_CREATED)
+def add_template_field(template_id: int, field_data: TemplateFieldCreate, db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    # Check if template exists
+    if not service.get_template(template_id):
+        raise HTTPException(status_code=404, detail="Template not found")
+    return service.add_template_field(template_id, field_data)
+
+@router.post("/{template_id}/mappings/save", response_model=List[TemplateFieldMappingSchema])
+def save_mappings(template_id: int, mappings: List[TemplateFieldMappingCreate], db: Session = Depends(get_db)):
+    service = TemplateService(db)
+    if not service.get_template(template_id):
+        raise HTTPException(status_code=404, detail="Template not found")
+    return service.save_mapping(template_id, mappings)

backend/database.py

@@ -10,10 +10,10 @@ from dotenv import load_dotenv
 load_dotenv()
 
 DB_USER = os.getenv("DB_USER", "postgres")
-DB_PASSWORD = os.getenv("DB_PASSWORD", "password")
-DB_HOST = os.getenv("DB_HOST", "localhost")
-DB_PORT = os.getenv("DB_PORT", "5432")
-DB_NAME = os.getenv("DB_NAME", "ocr_db")
+DB_PASSWORD = os.getenv("DB_PASSWORD", "M@tr!x#149@dm!N")
+DB_HOST = os.getenv("DB_HOST", "192.168.0.111")
+DB_PORT = os.getenv("DB_PORT", "7925")
+DB_NAME = os.getenv("DB_NAME", "ocr")
 
 encoded_user = urllib.parse.quote_plus(DB_USER)
 encoded_password = urllib.parse.quote_plus(DB_PASSWORD)

backend/engine/ocr/document_processor.py

@@ -0,0 +1,217 @@
+import io
+import re
+from typing import List, Dict, Any
+import numpy as np
+import cv2
+import pdfplumber
+import pytesseract
+from PIL import Image
+from pdf2image import convert_from_bytes
+from sqlalchemy.orm import Session
+from models.template_models import DocumentLayout, TemplateDocument
+from schemas.template_schemas import DocumentLayoutSchema
+
+class DocumentProcessor:
+    def __init__(self, db: Session):
+        self.db = db
+
+    def process_file(self, document_id: int, filename: str, file_bytes: bytes) -> List[DocumentLayout]:
+        document = self.db.query(TemplateDocument).filter(TemplateDocument.pk_document_id == document_id).first()
+        if not document:
+            raise ValueError("Document not found")
+            
+        layouts = []
+        if filename.lower().endswith('.pdf'):
+            layouts = self._process_pdf(document_id, file_bytes)
+        elif filename.lower().endswith(('.png', '.jpg', '.jpeg', '.tiff')):
+            layouts = self._process_image(document_id, file_bytes)
+            
+        # Update page count
+        if layouts:
+            document.page_count = max([l.page_no for l in layouts])
+            document.status = "processed"
+            self.db.commit()
+            
+        return layouts
+
+    def _process_pdf(self, document_id: int, file_bytes: bytes) -> List[DocumentLayout]:
+        layouts = []
+        try:
+            with pdfplumber.open(io.BytesIO(file_bytes)) as pdf:
+                sequence_no = 1
+                for page_no, page in enumerate(pdf.pages, start=1):
+                    # Check if it has text
+                    text = page.extract_text()
+                    if text and text.strip():
+                        # Extract words with bounding boxes
+                        words = page.extract_words()
+                        
+                        # Group words into blocks using simple proximity heuristics
+                        blocks = self._group_words_to_blocks(words, page.width, page.height)
+                        
+                        for block in blocks:
+                            db_layout = DocumentLayout(
+                                fk_document_id=document_id,
+                                page_no=page_no,
+                                text_value=block['text'],
+                                block_type=block['type'],
+                                x_coordinate=block['x0'],
+                                y_coordinate=block['top'],
+                                width=block['x1'] - block['x0'],
+                                height=block['bottom'] - block['top'],
+                                confidence=100.0,
+                                sequence_no=sequence_no
+                            )
+                            self.db.add(db_layout)
+                            layouts.append(db_layout)
+                            sequence_no += 1
+                    else:
+                        # Scanned PDF page -> convert to image and process
+                        # Handled separately to avoid complexity in this mock
+                        pass
+        except Exception as e:
+            print(f"Error processing PDF with pdfplumber: {e}")
+            
+        if not layouts:
+             # Fallback to image-based processing for scanned PDFs
+             images = convert_from_bytes(file_bytes, dpi=300)
+             sequence_no = 1
+             for page_no, img in enumerate(images, start=1):
+                  img_byte_arr = io.BytesIO()
+                  img.save(img_byte_arr, format='PNG')
+                  page_layouts = self._process_image(document_id, img_byte_arr.getvalue(), page_no=page_no, start_sequence=sequence_no)
+                  layouts.extend(page_layouts)
+                  sequence_no += len(page_layouts)
+                  
+        self.db.commit()
+        return layouts
+
+    def _process_image(self, document_id: int, file_bytes: bytes, page_no: int = 1, start_sequence: int = 1) -> List[DocumentLayout]:
+        layouts = []
+        try:
+            # Decode image
+            nparr = np.frombuffer(file_bytes, np.uint8)
+            img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
+            
+            # Preprocessing
+            gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+            # Denoise & Threshold
+            gray = cv2.medianBlur(gray, 3)
+            gray = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
+            
+            # OCR using PyTesseract with bounding boxes
+            data = pytesseract.image_to_data(gray, output_type=pytesseract.Output.DICT)
+            
+            sequence_no = start_sequence
+            
+            # Grouping words into lines/blocks simplified
+            height, width = gray.shape
+            blocks = self._tesseract_data_to_blocks(data, width, height)
+            
+            for block in blocks:
+                db_layout = DocumentLayout(
+                    fk_document_id=document_id,
+                    page_no=page_no,
+                    text_value=block['text'],
+                    block_type=block['type'],
+                    x_coordinate=block['x'],
+                    y_coordinate=block['y'],
+                    width=block['w'],
+                    height=block['h'],
+                    confidence=block['conf'],
+                    sequence_no=sequence_no
+                )
+                self.db.add(db_layout)
+                layouts.append(db_layout)
+                sequence_no += 1
+                
+        except Exception as e:
+            print(f"Error processing image with tesseract: {e}")
+            
+        self.db.commit()
+        return layouts
+
+    def _group_words_to_blocks(self, words: List[Dict], page_width: float, page_height: float) -> List[Dict]:
+        """Group nearby words into logical blocks and classify them."""
+        # Extremely simplified grouping by Y coordinate proximity
+        blocks = []
+        if not words:
+            return blocks
+            
+        # Sort words by Y then X
+        words.sort(key=lambda w: (w['top'], w['x0']))
+        
+        current_block = {
+            'text': words[0]['text'],
+            'x0': words[0]['x0'],
+            'top': words[0]['top'],
+            'x1': words[0]['x1'],
+            'bottom': words[0]['bottom']
+        }
+        
+        for word in words[1:]:
+            # If word is roughly on the same line (y diff is small) and close horizontally
+            if abs(word['top'] - current_block['top']) < 10 and (word['x0'] - current_block['x1']) < 50:
+                current_block['text'] += ' ' + word['text']
+                current_block['x1'] = word['x1']
+                current_block['top'] = min(current_block['top'], word['top'])
+                current_block['bottom'] = max(current_block['bottom'], word['bottom'])
+            else:
+                current_block['type'] = self._classify_block(current_block, page_height)
+                blocks.append(current_block)
+                current_block = {
+                    'text': word['text'],
+                    'x0': word['x0'],
+                    'top': word['top'],
+                    'x1': word['x1'],
+                    'bottom': word['bottom']
+                }
+        
+        current_block['type'] = self._classify_block(current_block, page_height)
+        blocks.append(current_block)
+        return blocks
+
+    def _tesseract_data_to_blocks(self, data: Dict, page_width: float, page_height: float) -> List[Dict]:
+        blocks = []
+        n_boxes = len(data['text'])
+        current_line = []
+        
+        for i in range(n_boxes):
+            if int(data['conf'][i]) > 10 and data['text'][i].strip():
+                # We can group by line_num
+                blocks.append({
+                    'text': data['text'][i],
+                    'x': data['left'][i],
+                    'y': data['top'][i],
+                    'w': data['width'][i],
+                    'h': data['height'][i],
+                    'conf': float(data['conf'][i]),
+                    'type': self._classify_block({'top': data['top'][i], 'text': data['text'][i]}, page_height)
+                })
+        return blocks
+
+    def _classify_block(self, block: Dict, page_height: float) -> str:
+        y = block.get('top') or block.get('y') or 0
+        text = block.get('text', '').lower()
+        
+        if y < page_height * 0.15:
+            return "HEADER"
+        elif y > page_height * 0.85:
+            return "FOOTER"
+            
+        if "total" in text:
+            return "TOTAL"
+        if "tax" in text or "gst" in text or "vat" in text:
+            return "TAX"
+        if re.search(r'\b(vendor|from)\b', text):
+            return "VENDOR"
+        if re.search(r'\b(bill to|sold to)\b', text):
+            return "BILL_TO"
+        if re.search(r'\b(ship to)\b', text):
+            return "SHIP_TO"
+            
+        # Very naive fallback for table check
+        if re.search(r'\b(qty|rate|amount|price|item)\b', text):
+            return "TABLE_HEADER"
+            
+        return "TEXT"

backend/engine/recognition/template_engine.py

@@ -0,0 +1,174 @@
+import hashlib
+import json
+from typing import List, Dict, Any, Tuple
+from sqlalchemy.orm import Session
+from models.template_models import Template, TemplateDocument, DocumentLayout, TemplateRecognitionHistory
+from schemas.template_schemas import TemplateRecognitionResult
+
+class TemplateRecognitionEngine:
+    def __init__(self, db: Session):
+        self.db = db
+        
+    def generate_fingerprint(self, layouts: List[DocumentLayout]) -> str:
+        """
+        Generate a structural fingerprint based on the document's layout.
+        We use relative positions of HEADER, VENDOR, TABLE_HEADER blocks.
+        """
+        key_blocks = []
+        for layout in layouts:
+            if layout.block_type in ["HEADER", "VENDOR", "TABLE_HEADER", "TOTAL"]:
+                key_blocks.append({
+                    "type": layout.block_type,
+                    "text": layout.text_value[:50] if layout.text_value else "", # first 50 chars
+                    "rx": round(float(layout.x_coordinate) / 100) if layout.x_coordinate else 0, # relative bucket x
+                    "ry": round(float(layout.y_coordinate) / 100) if layout.y_coordinate else 0  # relative bucket y
+                })
+        
+        # Sort top to bottom, left to right
+        key_blocks.sort(key=lambda b: (b['ry'], b['rx']))
+        
+        fingerprint_data = json.dumps(key_blocks)
+        return hashlib.sha256(fingerprint_data.encode('utf-8')).hexdigest()
+
+    def recognize_template(self, document_id: int) -> TemplateRecognitionResult:
+        """
+        Matches a document against existing templates using a 4-level weighted score:
+        Level 1: Vendor Match (40%)
+        Level 2: Header Similarity (20%)
+        Level 3: Layout/Fingerprint Similarity (20%)
+        Level 4: Coordinate Similarity (20%)
+        Threshold: 85%
+        """
+        layouts = self.db.query(DocumentLayout).filter(DocumentLayout.fk_document_id == document_id).all()
+        if not layouts:
+            return TemplateRecognitionResult(templateMatched=False)
+            
+        doc_fingerprint = self.generate_fingerprint(layouts)
+        
+        # Extract features for scoring
+        doc_vendor = self._extract_vendor_name(layouts)
+        doc_headers = self._extract_headers(layouts)
+        
+        templates = self.db.query(Template).filter(Template.active_flag == True).all()
+        best_match = None
+        highest_score = 0
+        
+        for template in templates:
+            score = 0.0
+            
+            # Level 1: Fingerprint Exact Match (counts for Layout + Vendor + Header if exact)
+            if template.template_fingerprint == doc_fingerprint:
+                score += 100.0
+            else:
+                # Need to load template sample mapping to compare heuristics
+                # In a real system, we'd compare against the mapped fields' coordinates
+                score += self._calculate_heuristic_score(template, doc_vendor, doc_headers, layouts)
+                
+            if score > highest_score:
+                highest_score = score
+                best_match = template
+                
+        # Store history
+        if best_match:
+            history = TemplateRecognitionHistory(
+                fk_template_id=best_match.pk_template_id,
+                fk_document_id=document_id,
+                recognition_score=highest_score,
+                matched_flag=(highest_score >= 85.0)
+            )
+            self.db.add(history)
+            self.db.commit()
+            
+        if best_match and highest_score >= 85.0:
+            extracted_fields = self._auto_map_fields(best_match, layouts)
+            return TemplateRecognitionResult(
+                templateMatched=True,
+                templateId=best_match.pk_template_id,
+                confidence=highest_score,
+                extractedFields=extracted_fields
+            )
+            
+        return TemplateRecognitionResult(templateMatched=False)
+
+    def _extract_vendor_name(self, layouts: List[DocumentLayout]) -> str:
+        for layout in layouts:
+            if layout.block_type == "VENDOR":
+                return layout.text_value.lower()
+        return ""
+        
+    def _extract_headers(self, layouts: List[DocumentLayout]) -> List[str]:
+        headers = []
+        for layout in layouts:
+            if layout.block_type == "TABLE_HEADER" and layout.text_value:
+                headers.append(layout.text_value.lower())
+        return headers
+
+    def _calculate_heuristic_score(self, template: Template, doc_vendor: str, doc_headers: List[str], layouts: List[DocumentLayout]) -> float:
+        score = 0.0
+        
+        # This requires the template to have some stored metadata or we check its fields
+        # E.g., if template has a field "Vendor Name" mapped to a specific text
+        
+        # 1. Vendor Match (40%)
+        # For this mockup, we check if the template name matches the vendor
+        if template.template_name.lower() in doc_vendor or doc_vendor in template.template_name.lower():
+            score += 40.0
+            
+        # 2. Header Similarity (20%)
+        # Check if template fields exist that match doc_headers
+        field_labels = [f.field_label.lower() for f in template.fields]
+        header_matches = sum(1 for h in doc_headers if any(h in fl or fl in h for fl in field_labels))
+        if len(doc_headers) > 0:
+             score += (header_matches / len(doc_headers)) * 20.0
+             
+        # 3. Layout / Coordinate Similarity
+        # We can check the template's previous mappings coordinates against current document
+        # If coordinates are within a tolerance, we add score
+        mapping_count = 0
+        match_count = 0
+        for mapping in template.mappings:
+            mapping_count += 1
+            # Find a layout block in current document that is near mapping coordinates
+            for layout in layouts:
+                if layout.page_no == mapping.page_no:
+                    # check distance
+                    if mapping.x_coordinate and mapping.y_coordinate and layout.x_coordinate and layout.y_coordinate:
+                        dx = abs(float(mapping.x_coordinate) - float(layout.x_coordinate))
+                        dy = abs(float(mapping.y_coordinate) - float(layout.y_coordinate))
+                        if dx < 50 and dy < 20: # arbitrary tolerance
+                            match_count += 1
+                            break
+        
+        if mapping_count > 0:
+            coord_score = (match_count / mapping_count) * 40.0 # Layout 20% + Coord 20%
+            score += coord_score
+            
+        return score
+
+    def _auto_map_fields(self, template: Template, layouts: List[DocumentLayout]) -> List[dict]:
+        extracted = []
+        for mapping in template.mappings:
+            # Find closest block in new document
+            best_block = None
+            min_dist = float('inf')
+            
+            for layout in layouts:
+                if layout.page_no == mapping.page_no:
+                    if mapping.x_coordinate and mapping.y_coordinate and layout.x_coordinate and layout.y_coordinate:
+                        dx = abs(float(mapping.x_coordinate) - float(layout.x_coordinate))
+                        dy = abs(float(mapping.y_coordinate) - float(layout.y_coordinate))
+                        dist = dx**2 + dy**2
+                        if dist < min_dist and dist < 5000: # tolerance squared
+                            min_dist = dist
+                            best_block = layout
+                            
+            if best_block:
+                extracted.append({
+                    "field_id": mapping.fk_template_field_id,
+                    "field_label": mapping.field.field_label if mapping.field else "",
+                    "value": best_block.text_value,
+                    "confidence": 95.0, # arbitrary high confidence for coordinate match
+                    "layout_id": best_block.pk_document_data_id
+                })
+                
+        return extracted

backend/llm_service.py

@@ -0,0 +1,116 @@
+import ollama
+import json
+import base64
+
+INVOICE_SCHEMA = {
+    "document_type": None,
+    "invoice_number": None,
+    "invoice_date": None,
+    "due_date": None,
+    "purchase_order_number": None,
+    "vendor": {
+        "name": None,
+        "address": None,
+        "email": None,
+        "phone": None,
+        "gstin": None,
+        "tax_id": None,
+        "website": None
+    },
+    "customer": {
+        "name": None,
+        "address": None,
+        "gstin": None
+    },
+    "amounts": {
+        "subtotal": None,
+        "tax": None,
+        "discount": None,
+        "shipping": None,
+        "round_off": None,
+        "total": None,
+        "amount_paid": None,
+        "balance_due": None,
+        "currency": None
+    },
+    "tax_breakdown": [
+        {
+            "type": None,
+            "rate": None,
+            "amount": None
+        }
+    ],
+    "line_items": [
+        {
+            "line_no": None,
+            "description": None,
+            "product_code": None,
+            "hsn_sac": None,
+            "quantity": None,
+            "unit": None,
+            "unit_price": None,
+            "discount": None,
+            "tax_rate": None,
+            "tax_amount": None,
+            "total": None
+        }
+    ],
+    "payment_information": {
+        "bank_name": None,
+        "account_number": None,
+        "ifsc": None,
+        "upi_id": None
+    },
+    "metadata": {
+        "pages": None,
+        "ocr_confidence": None,
+        "language": None
+    }
+}
+
+def extract_data(text: str = None, image_path: str = None, model_type: str = "text") -> dict:
+    """
+    Extracts structured data using either Text (Gemma) or Vision (Qwen) models.
+    """
+    
+    prompt = f"""
+    You are an expert data extraction assistant. 
+    Extract every possible detail from the provided document and return it strictly as a SINGLE VALID JSON OBJECT matching the following schema structure:
+    
+    {json.dumps(INVOICE_SCHEMA, indent=4)}
+
+    IMPORTANT: 
+    - Return ONLY the JSON. No markdown formatting, no explanations, no prefix.
+    - If a field is not found or data is not available, use null.
+    """
+    
+    messages = [{'role': 'user', 'content': prompt}]
+    model = 'gemma:2b'
+    
+    if model_type == 'vision':
+        if not image_path:
+            return {"error": "Image path required for vision mode"}
+        
+        # Qwen-VL handles images passed in the message
+        model = 'qwen2.5vl:7b' # Using the installed model ID
+        messages[0]['images'] = [image_path]
+        messages[0]['content'] = "Analyze this image. " + prompt
+    else:
+        # Text Mode
+        if not text:
+            return {"error": "Text required for text mode"}
+        messages[0]['content'] += f"\n\n---\n{text}\n---"
+
+    try:
+        response = ollama.chat(model=model, messages=messages)
+        content = response['message']['content']
+        
+        # Clean up markdown
+        content = content.replace("```json", "").replace("```", "").strip()
+        
+        return json.loads(content)
+
+    except Exception as e:
+        print(f"LLM Extraction Error ({model_type}): {e}")
+        return {"error": str(e), "raw_output": content if 'content' in locals() else ""}
+

backend/main.py

@@ -44,10 +44,14 @@ def extract_text_from_pdf(file_bytes: bytes) -> str:
         return ""
 
 # Internal modules
-from database import get_db, Email
+from database import get_db, Email, Vendor, Document
 from scheduler import start_scheduler, stop_scheduler
 from mail_service import fetch_and_store_emails
 
+# Import the new routers
+from api.templates import router as templates_router
+from api.documents import router as documents_router
+
 # Lifespan for Scheduler
 @asynccontextmanager
 async def lifespan(app: FastAPI):
@@ -59,6 +63,10 @@ async def lifespan(app: FastAPI):
 
 app = FastAPI(lifespan=lifespan)
 
+# Include Routers
+app.include_router(templates_router)
+app.include_router(documents_router)
+
 # CORS configuration
 origins = [
     "http://localhost",
@@ -85,15 +93,9 @@ class LoginResponse(BaseModel):
     token: str
     message: str
 
-@app.post("/api/login", response_model=LoginResponse)
-def login(request: LoginRequest):
-    if request.username == "admin" and request.password == "admin":
-        return LoginResponse(token="fake-super-secret-token", message="Success")
-    raise HTTPException(status_code=401, detail="Invalid username or password")
-
 class NERResponse(BaseModel):
     text: str
-
+    file_path: str
 
 
 def extract_text_from_image(file_bytes: bytes) -> str:
@@ -110,13 +112,41 @@ async def extract_text(file: UploadFile = File(...)):
     content = await file.read()
     filename = file.filename.lower()
     
+    # Save file for Vision mode
+    file_path = f"uploads/{file.filename}"
+    with open(file_path, "wb") as f:
+        f.write(content)
+    
     extracted_text = ""
     
     if filename.endswith(".pdf"):
         # Try text extraction first
-        extracted_text = extract_text_from_pdf(content)
-        
-        # If text is empty, it might be a scanned PDF. 
+        with pdfplumber.open(io.BytesIO(content)) as pdf:
+            try:
+                text = ""
+                for page in pdf.pages:
+                    page_text = page.extract_text(layout=True)
+                    if page_text:
+                        text += page_text + "\n"
+                if text.strip():
+                    extracted_text = text.strip()
+            except Exception:
+                pass
+
+        if not extracted_text:
+             try:
+                 # Fallback to pypdf
+                 reader = PdfReader(io.BytesIO(content))
+                 text = ""
+                 for page in reader.pages:
+                    page_text = page.extract_text()
+                    if page_text:
+                        text += page_text + "\n"
+                 extracted_text = text.strip()
+             except:
+                 pass
+
+        # If text is still empty, it might be a scanned PDF. 
         if not extracted_text.strip():
              try:
                  images = convert_from_bytes(content)
@@ -131,7 +161,35 @@ async def extract_text(file: UploadFile = File(...)):
     else:
         raise HTTPException(status_code=400, detail="Unsupported file type")
     
-    return NERResponse(text=extracted_text)
+    return NERResponse(text=extracted_text, file_path=file_path)
+
+# 3. AI Extraction Module
+from llm_service import extract_data
+from pdf2image import convert_from_path
+
+class AITextRequest(BaseModel):
+    text: Optional[str] = None
+    file_path: Optional[str] = None
+    model_type: str = "text"
+
+@app.post("/api/extract/ai")
+def extract_ai_data(request: AITextRequest):
+    final_image_path = request.file_path
+    
+    if request.model_type == "vision" and request.file_path and request.file_path.endswith(".pdf"):
+        # Convert PDF first page to image
+        try:
+            images = convert_from_path(request.file_path)
+            if images:
+                # Save temp image
+                temp_img_path = request.file_path + ".jpg"
+                images[0].save(temp_img_path, "JPEG")
+                final_image_path = temp_img_path
+        except Exception as e:
+            print(f"Error converting PDF for vision: {e}")
+            
+    data = extract_data(text=request.text, image_path=final_image_path, model_type=request.model_type)
+    return data
 
 import zipfile
 import mimetypes
@@ -261,3 +319,36 @@ def sync_emails():
 @app.get("/")
 def read_root():
     return {"message": "OCR Backend API is running"}
+
+class DocumentSaveRequest(BaseModel):
+    vendor_name: str
+    file_path: str
+    model_type: str
+    data: dict
+
+@app.post("/api/documents/save")
+def save_document(request: DocumentSaveRequest, db: Session = Depends(get_db)):
+    # 1. Find or Create Vendor
+    vendor = db.query(Vendor).filter(Vendor.name == request.vendor_name).first()
+    if not vendor:
+        vendor = Vendor(name=request.vendor_name, default_model=request.model_type)
+        db.add(vendor)
+        db.commit()
+        db.refresh(vendor)
+    else:
+        # Update preference
+        vendor.default_model = request.model_type
+        db.commit()
+    
+    # 2. Save Document
+    filename = request.file_path.split('/')[-1]
+    doc = Document(
+        vendor_id=vendor.id,
+        filename=filename,
+        status="verified",
+        processed_data=request.data
+    )
+    db.add(doc)
+    db.commit()
+    
+    return {"message": "Document saved and Vendor preference updated", "vendor_id": vendor.id}

backend/models/template_models.py

@@ -0,0 +1,105 @@
+from sqlalchemy import Column, Integer, String, Text, DateTime, Boolean, ForeignKey, Numeric
+from sqlalchemy.sql import func
+from sqlalchemy.orm import relationship
+from database import Base
+
+class TemplateDocument(Base):
+    __tablename__ = "documents"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_document_id = Column(Integer, primary_key=True, index=True)
+    document_name = Column(String(500))
+    document_type = Column(String(100))
+    file_name = Column(String(500))
+    file_hash = Column(String(500), index=True)
+    page_count = Column(Integer)
+    status = Column(String(50))
+    created_at = Column(DateTime, server_default=func.now())
+    updated_at = Column(DateTime, server_default=func.now(), onupdate=func.now())
+    
+    # Relationships
+    layouts = relationship("DocumentLayout", back_populates="document", cascade="all, delete")
+
+class DocumentLayout(Base):
+    __tablename__ = "document_layout"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_document_data_id = Column(Integer, primary_key=True, index=True)
+    fk_document_id = Column(Integer, ForeignKey("templates.documents.pk_document_id", ondelete="CASCADE"), nullable=False, index=True)
+    page_no = Column(Integer, nullable=False, index=True)
+    text_value = Column(Text)
+    block_type = Column(String(100))
+    parent_block_id = Column(Integer, ForeignKey("templates.document_layout.pk_document_data_id", ondelete="SET NULL"))
+    x_coordinate = Column(Numeric)
+    y_coordinate = Column(Numeric)
+    width = Column(Numeric)
+    height = Column(Numeric)
+    confidence = Column(Numeric)
+    sequence_no = Column(Integer)
+    layout_path = Column(Text)
+    created_at = Column(DateTime, server_default=func.now())
+    
+    document = relationship("TemplateDocument", back_populates="layouts")
+    parent = relationship("DocumentLayout", remote_side=[pk_document_data_id])
+    mappings = relationship("TemplateFieldMapping", back_populates="document_layout", cascade="all, delete")
+
+class Template(Base):
+    __tablename__ = "templates"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_template_id = Column(Integer, primary_key=True, index=True)
+    template_name = Column(String(255), nullable=False, index=True)
+    template_fingerprint = Column(Text)
+    active_flag = Column(Boolean, default=True)
+    created_at = Column(DateTime, server_default=func.now())
+    updated_at = Column(DateTime, server_default=func.now(), onupdate=func.now())
+    
+    fields = relationship("TemplateField", back_populates="template", cascade="all, delete")
+    mappings = relationship("TemplateFieldMapping", back_populates="template", cascade="all, delete")
+
+class TemplateField(Base):
+    __tablename__ = "template_fields"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_template_field_id = Column(Integer, primary_key=True, index=True)
+    fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False, index=True)
+    field_label = Column(String(255), nullable=False)
+    field_type = Column(String(100), nullable=False)
+    display_order = Column(Integer)
+    required_flag = Column(Boolean, default=False)
+    created_at = Column(DateTime, server_default=func.now())
+    
+    template = relationship("Template", back_populates="fields")
+    mappings = relationship("TemplateFieldMapping", back_populates="field", cascade="all, delete")
+
+class TemplateFieldMapping(Base):
+    __tablename__ = "template_fields_mapping"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_mapping_id = Column(Integer, primary_key=True, index=True)
+    fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False, index=True)
+    fk_template_field_id = Column(Integer, ForeignKey("templates.template_fields.pk_template_field_id", ondelete="CASCADE"), nullable=False, index=True)
+    fk_document_data_id = Column(Integer, ForeignKey("templates.document_layout.pk_document_data_id", ondelete="SET NULL"))
+    page_no = Column(Integer)
+    x_coordinate = Column(Numeric)
+    y_coordinate = Column(Numeric)
+    width = Column(Numeric)
+    height = Column(Numeric)
+    mapping_confidence = Column(Numeric)
+    layout_path = Column(Text)
+    created_at = Column(DateTime, server_default=func.now())
+    
+    template = relationship("Template", back_populates="mappings")
+    field = relationship("TemplateField", back_populates="mappings")
+    document_layout = relationship("DocumentLayout", back_populates="mappings")
+
+class TemplateRecognitionHistory(Base):
+    __tablename__ = "template_recognition_history"
+    __table_args__ = {"schema": "templates"}
+    
+    pk_history_id = Column(Integer, primary_key=True, index=True)
+    fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False)
+    fk_document_id = Column(Integer, ForeignKey("templates.documents.pk_document_id", ondelete="CASCADE"), nullable=False, index=True)
+    recognition_score = Column(Numeric)
+    matched_flag = Column(Boolean)
+    created_at = Column(DateTime, server_default=func.now())

backend/repositories/template_repository.py

@@ -0,0 +1,75 @@
+from sqlalchemy.orm import Session
+from sqlalchemy import desc
+from models.template_models import Template, TemplateField, TemplateFieldMapping, TemplateRecognitionHistory
+from schemas.template_schemas import TemplateCreate, TemplateFieldCreate, TemplateFieldMappingCreate
+
+class TemplateRepository:
+    def __init__(self, db: Session):
+        self.db = db
+
+    def get_template(self, template_id: int):
+        return self.db.query(Template).filter(Template.pk_template_id == template_id).first()
+
+    def get_templates(self, skip: int = 0, limit: int = 100):
+        return self.db.query(Template).offset(skip).limit(limit).all()
+
+    def create_template(self, template_data: TemplateCreate):
+        db_template = Template(template_name=template_data.template_name)
+        self.db.add(db_template)
+        self.db.commit()
+        self.db.refresh(db_template)
+        
+        # Create fields if provided
+        for field in template_data.fields:
+            db_field = TemplateField(
+                fk_template_id=db_template.pk_template_id,
+                field_label=field.field_label,
+                field_type=field.field_type,
+                display_order=field.display_order,
+                required_flag=field.required_flag
+            )
+            self.db.add(db_field)
+            
+        self.db.commit()
+        self.db.refresh(db_template)
+        return db_template
+
+    def add_template_field(self, template_id: int, field_data: TemplateFieldCreate):
+        db_field = TemplateField(
+            fk_template_id=template_id,
+            field_label=field_data.field_label,
+            field_type=field_data.field_type,
+            display_order=field_data.display_order,
+            required_flag=field_data.required_flag
+        )
+        self.db.add(db_field)
+        self.db.commit()
+        self.db.refresh(db_field)
+        return db_field
+
+    def delete_template(self, template_id: int):
+        template = self.get_template(template_id)
+        if template:
+            self.db.delete(template)
+            self.db.commit()
+            return True
+        return False
+
+    def save_mapping(self, template_id: int, mappings: list[TemplateFieldMappingCreate]):
+        saved_mappings = []
+        for mapping in mappings:
+            db_mapping = TemplateFieldMapping(
+                fk_template_id=template_id,
+                fk_template_field_id=mapping.fk_template_field_id,
+                fk_document_data_id=mapping.fk_document_data_id,
+                page_no=mapping.page_no,
+                x_coordinate=mapping.x_coordinate,
+                y_coordinate=mapping.y_coordinate,
+                width=mapping.width,
+                height=mapping.height,
+                mapping_confidence=mapping.mapping_confidence
+            )
+            self.db.add(db_mapping)
+            saved_mappings.append(db_mapping)
+        self.db.commit()
+        return saved_mappings

backend/requirements.txt

@@ -10,4 +10,8 @@ imap-tools
 apscheduler
 python-dotenv
 pdfplumber
+ollama
+numpy
+opencv-python-headless
 pdf2image
+

backend/schemas/template_schemas.py

@@ -0,0 +1,80 @@
+from pydantic import BaseModel, ConfigDict
+from typing import List, Optional
+from datetime import datetime
+
+# Common
+class BaseSchema(BaseModel):
+    model_config = ConfigDict(from_attributes=True)
+
+# Document
+class DocumentLayoutSchema(BaseSchema):
+    pk_document_data_id: int
+    fk_document_id: int
+    page_no: int
+    text_value: Optional[str]
+    block_type: Optional[str]
+    parent_block_id: Optional[int]
+    x_coordinate: Optional[float]
+    y_coordinate: Optional[float]
+    width: Optional[float]
+    height: Optional[float]
+    confidence: Optional[float]
+    sequence_no: Optional[int]
+
+class DocumentSchema(BaseSchema):
+    pk_document_id: int
+    document_name: Optional[str]
+    document_type: Optional[str]
+    file_name: Optional[str]
+    page_count: Optional[int]
+    status: Optional[str]
+    created_at: datetime
+    layouts: List[DocumentLayoutSchema] = []
+
+# Template Field
+class TemplateFieldCreate(BaseSchema):
+    field_label: str
+    field_type: str
+    display_order: Optional[int] = 0
+    required_flag: Optional[bool] = False
+
+class TemplateFieldSchema(TemplateFieldCreate):
+    pk_template_field_id: int
+    fk_template_id: int
+    created_at: datetime
+
+# Template Field Mapping
+class TemplateFieldMappingCreate(BaseSchema):
+    fk_template_field_id: int
+    fk_document_data_id: Optional[int] = None
+    page_no: Optional[int] = None
+    x_coordinate: Optional[float] = None
+    y_coordinate: Optional[float] = None
+    width: Optional[float] = None
+    height: Optional[float] = None
+    mapping_confidence: Optional[float] = None
+
+class TemplateFieldMappingSchema(TemplateFieldMappingCreate):
+    pk_mapping_id: int
+    fk_template_id: int
+    created_at: datetime
+
+# Template
+class TemplateCreate(BaseSchema):
+    template_name: str
+    fields: Optional[List[TemplateFieldCreate]] = []
+
+class TemplateSchema(BaseSchema):
+    pk_template_id: int
+    template_name: str
+    template_fingerprint: Optional[str] = None
+    active_flag: bool
+    created_at: datetime
+    fields: List[TemplateFieldSchema] = []
+    mappings: List[TemplateFieldMappingSchema] = []
+
+class TemplateRecognitionResult(BaseSchema):
+    templateMatched: bool
+    templateId: Optional[int] = None
+    confidence: Optional[float] = None
+    extractedFields: List[dict] = []

backend/services/template_service.py

@@ -0,0 +1,25 @@
+from sqlalchemy.orm import Session
+from schemas.template_schemas import TemplateCreate, TemplateFieldCreate, TemplateFieldMappingCreate
+from repositories.template_repository import TemplateRepository
+
+class TemplateService:
+    def __init__(self, db: Session):
+        self.repository = TemplateRepository(db)
+
+    def get_template(self, template_id: int):
+        return self.repository.get_template(template_id)
+
+    def get_all_templates(self, skip: int = 0, limit: int = 100):
+        return self.repository.get_templates(skip, limit)
+
+    def create_template(self, template_data: TemplateCreate):
+        return self.repository.create_template(template_data)
+
+    def add_template_field(self, template_id: int, field_data: TemplateFieldCreate):
+        return self.repository.add_template_field(template_id, field_data)
+
+    def delete_template(self, template_id: int):
+        return self.repository.delete_template(template_id)
+
+    def save_mapping(self, template_id: int, mappings: list[TemplateFieldMappingCreate]):
+        return self.repository.save_mapping(template_id, mappings)

backend/sql/01_templates_schema.sql

@@ -0,0 +1,107 @@
+-- Schema for Template Engine
+CREATE SCHEMA IF NOT EXISTS templates;
+
+-- 1. documents table
+CREATE TABLE IF NOT EXISTS templates.documents (
+    pk_document_id BIGSERIAL PRIMARY KEY,
+    document_name VARCHAR(500),
+    document_type VARCHAR(100),
+    file_name VARCHAR(500),
+    file_hash VARCHAR(500),
+    page_count INTEGER,
+    status VARCHAR(50),
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+-- Index for quick lookups
+CREATE INDEX idx_documents_hash ON templates.documents(file_hash);
+
+-- 2. document_layout table
+CREATE TABLE IF NOT EXISTS templates.document_layout (
+    pk_document_data_id BIGSERIAL PRIMARY KEY,
+    fk_document_id BIGINT NOT NULL,
+    page_no INTEGER NOT NULL,
+    text_value TEXT,
+    block_type VARCHAR(100),
+    parent_block_id BIGINT,
+    x_coordinate NUMERIC,
+    y_coordinate NUMERIC,
+    width NUMERIC,
+    height NUMERIC,
+    confidence NUMERIC,
+    sequence_no INTEGER,
+    layout_path TEXT,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    
+    CONSTRAINT fk_dl_document FOREIGN KEY (fk_document_id) REFERENCES templates.documents(pk_document_id) ON DELETE CASCADE,
+    CONSTRAINT fk_dl_parent FOREIGN KEY (parent_block_id) REFERENCES templates.document_layout(pk_document_data_id) ON DELETE SET NULL
+);
+
+CREATE INDEX idx_doc_layout_doc_id ON templates.document_layout(fk_document_id);
+CREATE INDEX idx_doc_layout_page ON templates.document_layout(page_no);
+
+-- 3. templates table
+CREATE TABLE IF NOT EXISTS templates.templates (
+    pk_template_id BIGSERIAL PRIMARY KEY,
+    template_name VARCHAR(255) NOT NULL,
+    template_fingerprint TEXT,
+    active_flag BOOLEAN DEFAULT TRUE,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE INDEX idx_templates_name ON templates.templates(template_name);
+
+-- 4. template_fields table
+CREATE TABLE IF NOT EXISTS templates.template_fields (
+    pk_template_field_id BIGSERIAL PRIMARY KEY,
+    fk_template_id BIGINT NOT NULL,
+    field_label VARCHAR(255) NOT NULL,
+    field_type VARCHAR(100) NOT NULL,
+    display_order INTEGER,
+    required_flag BOOLEAN DEFAULT FALSE,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    
+    CONSTRAINT fk_tf_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE
+);
+
+CREATE INDEX idx_template_fields_tmpl_id ON templates.template_fields(fk_template_id);
+
+-- 5. template_fields_mapping table
+CREATE TABLE IF NOT EXISTS templates.template_fields_mapping (
+    pk_mapping_id BIGSERIAL PRIMARY KEY,
+    fk_template_id BIGINT NOT NULL,
+    fk_template_field_id BIGINT NOT NULL,
+    fk_document_data_id BIGINT,
+    page_no INTEGER,
+    x_coordinate NUMERIC,
+    y_coordinate NUMERIC,
+    width NUMERIC,
+    height NUMERIC,
+    mapping_confidence NUMERIC,
+    layout_path TEXT,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    
+    CONSTRAINT fk_tfm_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE,
+    CONSTRAINT fk_tfm_field FOREIGN KEY (fk_template_field_id) REFERENCES templates.template_fields(pk_template_field_id) ON DELETE CASCADE,
+    CONSTRAINT fk_tfm_doc_data FOREIGN KEY (fk_document_data_id) REFERENCES templates.document_layout(pk_document_data_id) ON DELETE SET NULL
+);
+
+CREATE INDEX idx_mapping_tmpl_id ON templates.template_fields_mapping(fk_template_id);
+CREATE INDEX idx_mapping_field_id ON templates.template_fields_mapping(fk_template_field_id);
+
+-- 6. template_recognition_history table
+CREATE TABLE IF NOT EXISTS templates.template_recognition_history (
+    pk_history_id BIGSERIAL PRIMARY KEY,
+    fk_template_id BIGINT NOT NULL,
+    fk_document_id BIGINT NOT NULL,
+    recognition_score NUMERIC,
+    matched_flag BOOLEAN,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    
+    CONSTRAINT fk_trh_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE,
+    CONSTRAINT fk_trh_document FOREIGN KEY (fk_document_id) REFERENCES templates.documents(pk_document_id) ON DELETE CASCADE
+);
+
+CREATE INDEX idx_recognition_history_doc ON templates.template_recognition_history(fk_document_id);

check_matches.py

@@ -0,0 +1,16 @@
+import sys
+import os
+sys.path.append(os.path.join(os.getcwd(), 'docengine'))
+
+from app.core.database import SessionLocal
+from app.models.document import TemplateMatch
+from app.models.template import DocumentFormat
+
+db = SessionLocal()
+matches = db.query(TemplateMatch).order_by(TemplateMatch.created_at.desc()).limit(10).all()
+print(f"Found {len(matches)} matches.")
+for m in matches:
+    fmt = db.query(DocumentFormat).filter(DocumentFormat.id == m.format_id).first()
+    fmt_name = fmt.name if fmt else 'Unknown'
+    print(f"Match: doc_id={m.document_id}, format_id={m.format_id}, name={fmt_name}, score={m.confidence_score}")
+    print(f"Details: {m.match_details}")

debug_match.py

@@ -0,0 +1,26 @@
+import sys
+import os
+sys.path.append(os.path.join(os.getcwd(), 'docengine'))
+
+from app.core.database import SessionLocal
+from app.models.document import Document
+from app.services.matching_service import MatchingService
+
+db = SessionLocal()
+doc = db.query(Document).order_by(Document.created_at.desc()).first()
+if not doc:
+    print("No documents found.")
+    sys.exit(0)
+
+print(f"Latest document: {doc.id} (status: {doc.status})")
+
+svc = MatchingService(db)
+try:
+    # Match with min_confidence=0.0 so it returns EVERYTHING
+    matches = svc.match_document(doc.id, min_confidence=0.0)
+    print(f"Returned {len(matches)} matches.")
+    for m in matches:
+        print(f"Match: format_id={m.format_id}, score={m.confidence_score}")
+        print(f"Details: {m.match_details}")
+except Exception as e:
+    print(f"Error matching: {e}")

deployment/nginx.conf

@@ -1,22 +0,0 @@
-server {
-    listen 80;
-    server_name app.technobeesolutions.in;
-
-    # Proxy Angular Frontend Container
-    location /ocrf/ {
-        proxy_pass http://localhost:8080/;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-    }
-
-    # Proxy API requests to the Docker Backend
-    location /ocrb/ {
-        proxy_pass http://localhost:8000/;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-    }
-}

docengine/.dockerignore

@@ -0,0 +1,26 @@
+__pycache__
+*.pyc
+*.pyo
+.Python
+.env
+.venv
+env/
+venv/
+*.egg-info
+dist/
+build/
+.git
+.gitignore
+.dockerignore
+*.md
+*.rst
+docs/
+tests/
+htmlcov/
+.coverage
+.pytest_cache
+.mypy_cache
+.ruff_cache
+*.log
+.idea/
+.vscode/

docengine/.env.example

@@ -8,28 +8,28 @@ APP_PORT=7989
 APP_WORKERS=4
 
 # Database
-DB_HOST=103.125.129.116
+DB_HOST=localhost
 DB_PORT=5432
 DB_NAME=ocr
 DB_USER=postgres
-DB_PASSWORD=M@triXPostgr3s@6202
+DB_PASSWORD=changeme
 DB_SCHEMA=admin
 DB_POOL_SIZE=20
 DB_MAX_OVERFLOW=10
 DB_ECHO=false
 
 # Redis
-REDIS_HOST=103.125.129.116
-REDIS_PORT=7901
+REDIS_HOST=localhost
+REDIS_PORT=6379
 REDIS_DB=0
-REDIS_PASSWORD=M@triXR3d1s@6202
+REDIS_PASSWORD=
 
 # Celery
-CELERY_BROKER_URL=redis://:M%40triXR3d1s%406202@103.125.129.116:7901/0
-CELERY_RESULT_BACKEND=redis://:M%40triXR3d1s%406202@103.125.129.116:7901/1
+CELERY_BROKER_URL=redis://localhost:6379/0
+CELERY_RESULT_BACKEND=redis://localhost:6379/1
 
 # JWT
-JWT_SECRET_KEY=a7f3c9e1d4b8f2a6c0e5d7b3a9f1c4e8d2b6a0f5c3e7d1b9a4f8c2e6d0b5a3
+JWT_SECRET_KEY=change-this-to-a-secure-random-string
 JWT_ALGORITHM=HS256
 JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30
 JWT_REFRESH_TOKEN_EXPIRE_DAYS=7

docengine/.gitignore

@@ -0,0 +1,64 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+*.manifest
+*.spec
+pip-log.txt
+pip-delete-this-directory.txt
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+*.mo
+*.pot
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+instance/
+.webassets-cache
+.scrapy
+docs/_build/
+target/
+.venv
+env/
+venv/
+ENV/
+.env
+!.env.example
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+.DS_Store
+Thumbs.db
+/storage/
+*.pid
+celerybeat-schedule
+celerybeat.pid

docengine/.pids/server.pid

@@ -0,0 +1 @@
+3173

docengine/.pids/worker.pid

@@ -0,0 +1 @@
+3176

docengine/Dockerfile

@@ -0,0 +1,39 @@
+FROM python:3.12-slim AS base
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    libpq-dev \
+    libgl1-mesa-glx \
+    libglib2.0-0 \
+    libsm6 \
+    libxext6 \
+    libxrender-dev \
+    libgomp1 \
+    poppler-utils \
+    ghostscript \
+    libmagic1 \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . .
+
+RUN mkdir -p /app/storage/documents /app/storage/templates /app/storage/images /app/storage/temp
+
+FROM base AS app
+EXPOSE 7989
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7989", "--workers", "4"]
+
+FROM base AS worker
+CMD ["celery", "-A", "app.workers.celery_app", "worker", "--loglevel=info", "--concurrency=4"]
+
+FROM base AS beat
+CMD ["celery", "-A", "app.workers.celery_app", "beat", "--loglevel=info"]

docengine/README.md

@@ -0,0 +1,522 @@
+# DocEngine — Document Template Recognition & Reconstruction System
+
+A production-ready system for scanning documents, detecting layouts, extracting content, generating reusable templates, matching future uploads against stored templates, and reconstructing original layouts as PDF output.
+
+## Architecture
+
+```
+┌───────────────┐      ┌───────────────┐      ┌──────────────┐
+│  FastAPI App   │─────▶│    Celery      │─────▶│   Redis      │
+│  (Port 7989)   │      │  Worker(s)     │      │  (Broker)    │
+└───────┬───────┘      └───────┬───────┘      └──────────────┘
+        │                      │
+        ▼                      ▼
+┌───────────────────────────────────────┐
+│       PostgreSQL (Schema: admin)       │
+│       192.168.0.111:5432               │
+└───────────────────────────────────────┘
+```
+
+**Stack**: Python 3.12, FastAPI, SQLAlchemy 2.x, Pydantic V2, Celery, Redis, PaddleOCR, PyMuPDF, OpenCV, ReportLab, PostgreSQL 16.
+
+## Features
+
+| Capability                       | Implementation                             |
+|----------------------------------|--------------------------------------------|
+| Scanned image OCR                | PaddleOCR (CPU/GPU)                        |
+| Native PDF text extraction       | PyMuPDF (fitz)                             |
+| Layout detection                 | OpenCV + LayoutParser                      |
+| Table extraction                 | Camelot-py + OpenCV contour detection      |
+| Header/footer detection          | Positional heuristics                      |
+| Watermark detection              | Transparency + large-font analysis         |
+| Font info extraction             | PyMuPDF text dict parsing                  |
+| Template generation & storage    | PostgreSQL (admin schema)                  |
+| Template fingerprinting          | SHA-256 structural hashing                 |
+| Template matching                | Multi-signal similarity scoring            |
+| PDF reconstruction               | ReportLab from template definitions        |
+| Async processing                 | Celery + Redis                             |
+| Authentication                   | JWT (access + refresh tokens, bcrypt)      |
+| Monitoring                       | Prometheus + structlog JSON logging        |
+
+## Project Structure
+
+```
+docengine/
+├── app/
+│   ├── main.py                  # FastAPI application entry
+│   ├── api/
+│   │   ├── router.py            # Top-level API router
+│   │   └── v1/
+│   │       ├── auth.py          # Auth endpoints
+│   │       ├── documents.py     # Document endpoints
+│   │       ├── health.py        # Health check
+│   │       └── templates.py     # Template endpoints
+│   ├── core/
+│   │   ├── config.py            # Pydantic Settings
+│   │   ├── database.py          # SQLAlchemy engine & session
+│   │   ├── dependencies.py      # FastAPI DI
+│   │   ├── exceptions.py        # Custom exception hierarchy
+│   │   ├── logging_config.py    # structlog configuration
+│   │   └── security.py          # JWT & bcrypt helpers
+│   ├── models/                  # SQLAlchemy ORM models
+│   ├── schemas/                 # Pydantic request/response schemas
+│   ├── repositories/            # Data access layer
+│   ├── services/                # Business logic
+│   │   ├── document_service.py  # Orchestration pipeline
+│   │   ├── ocr_service.py       # PaddleOCR integration
+│   │   ├── pdf_service.py       # PyMuPDF native PDF parsing
+│   │   ├── layout_service.py    # OpenCV layout detection
+│   │   ├── template_service.py  # Template generation
+│   │   ├── fingerprint_service.py
+│   │   ├── matching_service.py
+│   │   └── reconstruction_service.py
+│   ├── middleware/               # CORS, audit, metrics, rate limit
+│   ├── storage/                  # File storage abstraction
+│   ├── tasks/                    # Celery async tasks
+│   ├── workers/                  # Celery app configuration
+│   └── events/                   # App lifecycle handlers
+├── alembic/                      # Database migrations
+├── sql/                          # Raw SQL scripts
+├── tests/                        # Test suite
+├── docker-compose.yml            # Dev stack
+├── docker-compose.prod.yml       # Production stack
+├── Dockerfile                    # Multi-stage build
+├── requirements.txt
+└── .env
+```
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.12+
+- PostgreSQL 16 (running at `192.168.0.111:5432`)
+- Redis (for Celery)
+- `poppler-utils` and `ghostscript` (for pdf2image/camelot)
+
+### Local Setup
+
+```bash
+# Clone & enter
+cd docengine
+
+# Create virtual environment
+python -m venv .venv
+source .venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Create storage directories
+mkdir -p storage/{documents,templates,images,temp,rendered}
+
+# Run database migrations
+alembic upgrade head
+
+# (Optional) Seed default data
+psql -h 192.168.0.111 -p 5432 -U postgres -d ocr -f sql/003_seed_data.sql
+psql -h 192.168.0.111 -p 5432 -U postgres -d ocr -f sql/004_indexes.sql
+
+# Start the application
+python -m app.main
+```
+
+The API is now available at `http://localhost:7989`. Interactive docs at `http://localhost:7989/docs`.
+
+### Start Celery Worker (separate terminal)
+
+```bash
+source .venv/bin/activate
+celery -A app.workers.celery_app worker --loglevel=info --concurrency=4
+```
+
+### Docker Setup
+
+```bash
+# Build and start all services (app + worker + db + redis)
+docker compose up --build -d
+
+# Run migrations inside the container
+docker compose exec app alembic upgrade head
+
+# Seed data
+docker compose exec app bash -c "psql -h db -U postgres -d ocr -f sql/003_seed_data.sql"
+```
+
+---
+
+## API Reference
+
+Base URL: `http://localhost:7989/api/v1`
+
+### Health
+
+```bash
+curl http://localhost:7989/api/v1/health
+```
+
+### Authentication
+
+#### Register
+
+```bash
+curl -X POST http://localhost:7989/api/v1/auth/register \
+  -H "Content-Type: application/json" \
+  -d '{
+    "username": "john",
+    "email": "john@example.com",
+    "password": "SecurePass123!",
+    "full_name": "John Doe"
+  }'
+```
+
+#### Login
+
+```bash
+curl -X POST http://localhost:7989/api/v1/auth/login \
+  -H "Content-Type: application/json" \
+  -d '{
+    "username": "john",
+    "password": "SecurePass123!"
+  }'
+```
+
+Response:
+
+```json
+{
+  "access_token": "eyJhbGciOiJIUzI1NiIs...",
+  "refresh_token": "eyJhbGciOiJIUzI1NiIs...",
+  "token_type": "bearer",
+  "expires_in": 1800
+}
+```
+
+#### Get Current User
+
+```bash
+curl http://localhost:7989/api/v1/auth/me \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Refresh Token
+
+```bash
+curl -X POST http://localhost:7989/api/v1/auth/refresh \
+  -H "Content-Type: application/json" \
+  -d '{"refresh_token": "<refresh_token>"}'
+```
+
+#### Change Password
+
+```bash
+curl -X POST http://localhost:7989/api/v1/auth/change-password \
+  -H "Authorization: Bearer <access_token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "current_password": "SecurePass123!",
+    "new_password": "NewSecurePass456!"
+  }'
+```
+
+#### Logout
+
+```bash
+curl -X POST http://localhost:7989/api/v1/auth/logout \
+  -H "Authorization: Bearer <access_token>" \
+  -H "Content-Type: application/json" \
+  -d '{"refresh_token": "<refresh_token>"}'
+```
+
+### Documents
+
+#### Upload Document
+
+```bash
+# Upload a PDF
+curl -X POST http://localhost:7989/api/v1/documents/upload \
+  -H "Authorization: Bearer <access_token>" \
+  -F "file=@/path/to/document.pdf"
+
+# Upload a scanned image
+curl -X POST http://localhost:7989/api/v1/documents/upload \
+  -H "Authorization: Bearer <access_token>" \
+  -F "file=@/path/to/scan.jpg"
+```
+
+Response:
+
+```json
+{
+  "id": "550e8400-e29b-41d4-a716-446655440000",
+  "filename": "abc123_document.pdf",
+  "original_filename": "document.pdf",
+  "content_type": "application/pdf",
+  "file_size": 245760,
+  "checksum": "e3b0c44298fc1c149afbf4c8996fb924...",
+  "status": "pending",
+  "created_at": "2026-06-01T12:00:00Z"
+}
+```
+
+#### Get Document
+
+```bash
+curl http://localhost:7989/api/v1/documents/<document_id> \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### List Documents
+
+```bash
+# With pagination
+curl "http://localhost:7989/api/v1/documents?page=1&page_size=20" \
+  -H "Authorization: Bearer <access_token>"
+
+# Filter by status
+curl "http://localhost:7989/api/v1/documents?status=completed" \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Delete Document
+
+```bash
+curl -X DELETE http://localhost:7989/api/v1/documents/<document_id> \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Get Template Matches for Document
+
+```bash
+curl http://localhost:7989/api/v1/documents/<document_id>/template \
+  -H "Authorization: Bearer <access_token>"
+```
+
+### Templates
+
+#### List Templates
+
+```bash
+curl "http://localhost:7989/api/v1/templates?page=1&page_size=20" \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Get Template
+
+```bash
+curl http://localhost:7989/api/v1/templates/<template_id> \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Delete (Deactivate) Template
+
+```bash
+curl -X DELETE http://localhost:7989/api/v1/templates/<template_id> \
+  -H "Authorization: Bearer <access_token>"
+```
+
+#### Match Document to Templates
+
+```bash
+curl -X POST http://localhost:7989/api/v1/templates/match \
+  -H "Authorization: Bearer <access_token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "document_id": "<document_id>",
+    "min_confidence": 0.5,
+    "max_results": 5
+  }'
+```
+
+Response:
+
+```json
+[
+  {
+    "id": "...",
+    "document_id": "...",
+    "format_id": "...",
+    "confidence_score": 0.92,
+    "match_details": { "dimension_score": 1.0, "header_score": 0.85 },
+    "selected": true,
+    "template_name": "Invoice Template v1",
+    "created_at": "2026-06-01T12:00:00Z"
+  }
+]
+```
+
+#### Render Template to PDF
+
+```bash
+curl -X POST http://localhost:7989/api/v1/templates/render \
+  -H "Authorization: Bearer <access_token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "template_id": "<template_id>",
+    "data": {
+      "company_name": "Acme Corp",
+      "invoice_number": "INV-2026-001",
+      "date": "2026-06-01",
+      "total": "$1,250.00"
+    },
+    "output_filename": "invoice_output.pdf"
+  }'
+```
+
+Response:
+
+```json
+{
+  "output_path": "rendered/invoice_output.pdf",
+  "filename": "invoice_output.pdf",
+  "file_size": 32768,
+  "page_count": 1,
+  "rendered_at": "2026-06-01T12:05:00Z"
+}
+```
+
+#### Download Rendered PDF
+
+```bash
+curl -O http://localhost:7989/api/v1/templates/<template_id>/download?filename=invoice_output.pdf \
+  -H "Authorization: Bearer <access_token>"
+```
+
+---
+
+## Processing Pipeline
+
+When a document is uploaded, the following Celery task pipeline executes asynchronously:
+
+1. **File Type Detection** — Determine if the document is a native PDF or scanned image.
+2. **Page Extraction** — Convert PDF pages to images (for scanned docs) or parse directly (for native PDFs).
+3. **OCR** — Run PaddleOCR on scanned pages to extract text blocks with coordinates, confidence, and bounding boxes.
+4. **Native PDF Parsing** — Use PyMuPDF to extract text, fonts, images, and tables from native PDFs.
+5. **Layout Analysis** — Detect headers, footers, watermarks, tables, and image regions using OpenCV heuristics.
+6. **Template Generation** — Build a reusable template definition from the detected layout, stored in PostgreSQL.
+7. **Fingerprint Generation** — Compute a structural fingerprint (SHA-256) for future matching.
+8. **Status Update** — Mark the document as `completed` (or `failed` with error details).
+
+---
+
+## Database
+
+**Connection**: `postgresql://postgres:***@192.168.0.111:5432/ocr`
+**Schema**: `admin`
+
+### Migrations
+
+```bash
+# Create a new migration
+alembic revision --autogenerate -m "description"
+
+# Apply migrations
+alembic upgrade head
+
+# Rollback one step
+alembic downgrade -1
+```
+
+### Tables
+
+| Table                    | Purpose                                      |
+|--------------------------|----------------------------------------------|
+| `users`                  | User accounts                                |
+| `roles`                  | Role definitions (admin, user, viewer)        |
+| `user_roles`             | User-role mapping (M2M)                      |
+| `refresh_tokens`         | JWT refresh token storage                    |
+| `audit_logs`             | Action audit trail                           |
+| `documents`              | Uploaded document records                    |
+| `document_pages`         | Per-page data (dimensions, images)           |
+| `document_text_blocks`   | Extracted text with position & font info     |
+| `document_images`        | Extracted images with position               |
+| `document_tables`        | Extracted tables with cell data (JSONB)      |
+| `document_formats`       | Template definitions                         |
+| `document_cells`         | Template cell layout definitions             |
+| `document_regions`       | Template region definitions                  |
+| `table_formats`          | Template table structure definitions         |
+| `table_columns`          | Template table column definitions            |
+| `table_rows`             | Template table row definitions               |
+| `watermarks`             | Template watermark definitions               |
+| `image_regions`          | Template image region definitions            |
+| `template_fingerprints`  | Structural fingerprints for matching         |
+| `template_matches`       | Document-to-template match results           |
+
+---
+
+## Testing
+
+```bash
+# Install dev dependencies
+pip install -r requirements-dev.txt
+
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=app --cov-report=term-missing
+
+# Run specific test categories
+pytest tests/unit/
+pytest tests/api/
+pytest tests/repositories/
+```
+
+---
+
+## Configuration
+
+All configuration is via environment variables (`.env` file). Key settings:
+
+| Variable                           | Default                | Description                     |
+|------------------------------------|------------------------|---------------------------------|
+| `APP_PORT`                         | `7989`                 | Application port                |
+| `DB_HOST`                          | `192.168.0.111`        | PostgreSQL host                 |
+| `DB_PORT`                          | `5432`                 | PostgreSQL port                 |
+| `DB_NAME`                          | `ocr`                  | Database name                   |
+| `DB_SCHEMA`                        | `admin`                | PostgreSQL schema               |
+| `REDIS_HOST`                       | `192.168.0.111`        | Redis host                      |
+| `CELERY_BROKER_URL`                | `redis://:***@192.168.0.111:7901/0` | Celery broker         |
+| `JWT_SECRET_KEY`                   | *(see .env)*           | JWT signing key                 |
+| `JWT_ACCESS_TOKEN_EXPIRE_MINUTES`  | `30`                   | Access token TTL                |
+| `STORAGE_LOCAL_PATH`               | `./storage`            | Local file storage path         |
+| `STORAGE_MAX_FILE_SIZE_MB`         | `100`                  | Max upload size                 |
+| `OCR_LANGUAGE`                     | `en`                   | PaddleOCR language              |
+| `OCR_USE_GPU`                      | `false`                | Enable GPU for OCR              |
+
+---
+
+## Production Deployment
+
+```bash
+# Using production compose file
+docker compose -f docker-compose.prod.yml up --build -d
+
+# Scale workers
+docker compose -f docker-compose.prod.yml up --scale worker=4 -d
+```
+
+Production compose includes:
+- Resource limits (CPU/memory)
+- Redis authentication
+- App replicas
+- Persistent named volumes
+- Auto-restart policies
+
+---
+
+## Default Credentials
+
+| Username | Password      | Role  |
+|----------|---------------|-------|
+| `admin`  | `Admin@123!`  | admin |
+
+> ⚠️ **Change the default admin password immediately in production.**
+
+---
+
+## License
+
+Proprietary — All rights reserved.

docengine/alembic.ini

@@ -0,0 +1,41 @@
+[alembic]
+script_location = alembic
+prepend_sys_path = .
+version_path_separator = os
+sqlalchemy.url = postgresql+psycopg2://postgres:M%%40triXPostgr3s%%406202@192.168.0.111:5432/ocr
+
+[post_write_hooks]
+
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

docengine/alembic/env.py

@@ -0,0 +1,67 @@
+from logging.config import fileConfig
+
+from alembic import context
+from sqlalchemy import engine_from_config, pool, text
+
+from app.core.config import settings
+from app.core.database import Base
+
+# Import all models so Alembic can detect them
+import app.models  # noqa: F401
+
+config = context.config
+
+if config.config_file_name is not None:
+    fileConfig(config.config_file_name)
+
+target_metadata = Base.metadata
+
+# Override the database URL from settings
+config.set_main_option("sqlalchemy.url", settings.database_url.replace('%', '%%'))
+
+
+def run_migrations_offline() -> None:
+    """Run migrations in 'offline' mode."""
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+        version_table_schema=settings.db_schema,
+        include_schemas=True,
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode."""
+    connectable = engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix="sqlalchemy.",
+        poolclass=pool.NullPool,
+    )
+
+    with connectable.connect() as connection:
+        # Ensure schema exists
+        connection.execute(text(f"CREATE SCHEMA IF NOT EXISTS {settings.db_schema}"))
+        connection.execute(text(f"SET search_path TO {settings.db_schema}, public"))
+        connection.commit()
+
+        context.configure(
+            connection=connection,
+            target_metadata=target_metadata,
+            version_table_schema=settings.db_schema,
+            include_schemas=True,
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

docengine/alembic/script.py.mako

@@ -0,0 +1,26 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: Union[str, None] = ${repr(down_revision)}
+branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
+depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    ${downgrades if downgrades else "pass"}

docengine/alembic/versions/001_initial.py

@@ -0,0 +1,398 @@
+"""initial schema
+
+Revision ID: 001_initial
+Revises:
+Create Date: 2026-05-31 18:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+revision: str = "001_initial"
+down_revision: Union[str, None] = None
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+SCHEMA = "admin"
+
+
+def upgrade() -> None:
+    # Create schema
+    op.execute(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
+
+    # Users table
+    op.create_table(
+        "users",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("username", sa.String(150), unique=True, nullable=False, index=True),
+        sa.Column("email", sa.String(255), unique=True, nullable=False, index=True),
+        sa.Column("hashed_password", sa.String(255), nullable=False),
+        sa.Column("full_name", sa.String(255), nullable=True),
+        sa.Column("is_active", sa.Boolean, default=True, nullable=False),
+        sa.Column("is_superuser", sa.Boolean, default=False, nullable=False),
+        sa.Column("last_login", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Roles table
+    op.create_table(
+        "roles",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("name", sa.String(50), unique=True, nullable=False, index=True),
+        sa.Column("description", sa.Text, nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # User roles (many-to-many)
+    op.create_table(
+        "user_roles",
+        sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), primary_key=True),
+        sa.Column("role_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.roles.id", ondelete="CASCADE"), primary_key=True),
+        schema=SCHEMA,
+    )
+
+    # Refresh tokens
+    op.create_table(
+        "refresh_tokens",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("token", sa.String(512), unique=True, nullable=False, index=True),
+        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("revoked", sa.Boolean, default=False, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Audit logs
+    op.create_table(
+        "audit_logs",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True),
+        sa.Column("action", sa.String(100), nullable=False, index=True),
+        sa.Column("resource_type", sa.String(100), nullable=False, index=True),
+        sa.Column("resource_id", sa.String(255), nullable=True),
+        sa.Column("details", sa.Text, nullable=True),
+        sa.Column("ip_address", sa.String(45), nullable=True),
+        sa.Column("user_agent", sa.String(512), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False, index=True),
+        schema=SCHEMA,
+    )
+
+    # Documents
+    op.create_table(
+        "documents",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("filename", sa.String(500), nullable=False),
+        sa.Column("original_filename", sa.String(500), nullable=False),
+        sa.Column("content_type", sa.String(100), nullable=False),
+        sa.Column("file_size", sa.BigInteger, nullable=False),
+        sa.Column("checksum", sa.String(128), nullable=False, index=True),
+        sa.Column("storage_path", sa.String(1024), nullable=False),
+        sa.Column("status", sa.String(50), default="pending", nullable=False, index=True),
+        sa.Column("page_count", sa.Integer, nullable=True),
+        sa.Column("is_scanned", sa.Boolean, nullable=True),
+        sa.Column("document_metadata", postgresql.JSONB, nullable=True),
+        sa.Column("error_message", sa.Text, nullable=True),
+        sa.Column("uploaded_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document pages
+    op.create_table(
+        "document_pages",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("image_path", sa.String(1024), nullable=True),
+        sa.Column("text_content", sa.Text, nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document text blocks
+    op.create_table(
+        "document_text_blocks",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("text", sa.Text, nullable=False),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("confidence", sa.Float, nullable=True),
+        sa.Column("font_family", sa.String(255), nullable=True),
+        sa.Column("font_size", sa.Float, nullable=True),
+        sa.Column("font_color", sa.String(50), nullable=True),
+        sa.Column("font_style", sa.String(50), nullable=True),
+        sa.Column("block_type", sa.String(50), default="text", nullable=False),
+        sa.Column("sequence", sa.Integer, default=0, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document images
+    op.create_table(
+        "document_images",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("image_path", sa.String(1024), nullable=False),
+        sa.Column("image_type", sa.String(50), default="figure", nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document tables
+    op.create_table(
+        "document_tables",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("rows", sa.Integer, nullable=False),
+        sa.Column("columns", sa.Integer, nullable=False),
+        sa.Column("data", postgresql.JSONB, nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document formats (templates)
+    op.create_table(
+        "document_formats",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("name", sa.String(255), nullable=False, index=True),
+        sa.Column("description", sa.Text, nullable=True),
+        sa.Column("page_width", sa.Float, nullable=False),
+        sa.Column("page_height", sa.Float, nullable=False),
+        sa.Column("page_count", sa.Integer, default=1, nullable=False),
+        sa.Column("margin_top", sa.Float, default=72.0, nullable=False),
+        sa.Column("margin_right", sa.Float, default=72.0, nullable=False),
+        sa.Column("margin_bottom", sa.Float, default=72.0, nullable=False),
+        sa.Column("margin_left", sa.Float, default=72.0, nullable=False),
+        sa.Column("fingerprint", postgresql.JSONB, nullable=True),
+        sa.Column("source_document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="SET NULL"), nullable=True, index=True),
+        sa.Column("version", sa.Integer, default=1, nullable=False),
+        sa.Column("is_active", sa.Boolean, default=True, nullable=False, index=True),
+        sa.Column("created_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document cells
+    op.create_table(
+        "document_cells",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=False),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("row_no", sa.Integer, default=0, nullable=False),
+        sa.Column("column_no", sa.Integer, default=0, nullable=False),
+        sa.Column("data_type", sa.String(50), default="text", nullable=False),
+        sa.Column("font_family", sa.String(255), nullable=True),
+        sa.Column("font_size", sa.Float, nullable=True),
+        sa.Column("font_style", sa.String(50), nullable=True),
+        sa.Column("font_color", sa.String(50), nullable=True),
+        sa.Column("background_color", sa.String(50), nullable=True),
+        sa.Column("border_top", sa.String(100), nullable=True),
+        sa.Column("border_right", sa.String(100), nullable=True),
+        sa.Column("border_bottom", sa.String(100), nullable=True),
+        sa.Column("border_left", sa.String(100), nullable=True),
+        sa.Column("padding_top", sa.Float, default=0.0, nullable=False),
+        sa.Column("padding_right", sa.Float, default=0.0, nullable=False),
+        sa.Column("padding_bottom", sa.Float, default=0.0, nullable=False),
+        sa.Column("padding_left", sa.Float, default=0.0, nullable=False),
+        sa.Column("alignment", sa.String(20), default="left", nullable=False),
+        sa.Column("vertical_alignment", sa.String(20), default="top", nullable=False),
+        sa.Column("rowspan", sa.Integer, default=1, nullable=False),
+        sa.Column("colspan", sa.Integer, default=1, nullable=False),
+        sa.Column("static_text", sa.Text, nullable=True),
+        sa.Column("field_name", sa.String(255), nullable=True),
+        sa.Column("sequence", sa.Integer, default=0, nullable=False),
+        sa.Column("is_dynamic", sa.Boolean, default=False, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Document regions
+    op.create_table(
+        "document_regions",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=False),
+        sa.Column("region_type", sa.String(50), nullable=False),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("content", postgresql.JSONB, nullable=True),
+        sa.Column("sequence", sa.Integer, default=0, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Table formats
+    op.create_table(
+        "table_formats",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=False),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("rows", sa.Integer, nullable=False),
+        sa.Column("columns", sa.Integer, nullable=False),
+        sa.Column("border_style", sa.String(50), default="solid", nullable=False),
+        sa.Column("border_width", sa.Float, default=1.0, nullable=False),
+        sa.Column("border_color", sa.String(50), default="#000000", nullable=False),
+        sa.Column("header_rows", sa.Integer, default=1, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Table columns
+    op.create_table(
+        "table_columns",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("column_index", sa.Integer, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("header_text", sa.String(500), nullable=True),
+        sa.Column("data_type", sa.String(50), default="text", nullable=False),
+        sa.Column("alignment", sa.String(20), default="left", nullable=False),
+        sa.Column("font_family", sa.String(255), nullable=True),
+        sa.Column("font_size", sa.Float, nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Table rows
+    op.create_table(
+        "table_rows",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("row_index", sa.Integer, nullable=False),
+        sa.Column("height", sa.Float, default=20.0, nullable=False),
+        sa.Column("is_header", sa.Boolean, default=False, nullable=False),
+        sa.Column("background_color", sa.String(50), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Watermarks
+    op.create_table(
+        "watermarks",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=True),
+        sa.Column("text", sa.String(500), nullable=True),
+        sa.Column("image_path", sa.String(1024), nullable=True),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("opacity", sa.Float, default=0.3, nullable=False),
+        sa.Column("rotation", sa.Float, default=0.0, nullable=False),
+        sa.Column("font_family", sa.String(255), nullable=True),
+        sa.Column("font_size", sa.Float, nullable=True),
+        sa.Column("font_color", sa.String(50), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Image regions
+    op.create_table(
+        "image_regions",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("page_number", sa.Integer, nullable=False),
+        sa.Column("x", sa.Float, nullable=False),
+        sa.Column("y", sa.Float, nullable=False),
+        sa.Column("width", sa.Float, nullable=False),
+        sa.Column("height", sa.Float, nullable=False),
+        sa.Column("image_path", sa.String(1024), nullable=True),
+        sa.Column("image_type", sa.String(50), default="figure", nullable=False),
+        sa.Column("is_static", sa.Boolean, default=True, nullable=False),
+        sa.Column("field_name", sa.String(255), nullable=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Template fingerprints
+    op.create_table(
+        "template_fingerprints",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, unique=True, index=True),
+        sa.Column("page_dimensions", postgresql.JSONB, nullable=True),
+        sa.Column("logo_coordinates", postgresql.JSONB, nullable=True),
+        sa.Column("header_coordinates", postgresql.JSONB, nullable=True),
+        sa.Column("footer_coordinates", postgresql.JSONB, nullable=True),
+        sa.Column("table_coordinates", postgresql.JSONB, nullable=True),
+        sa.Column("cell_coordinates", postgresql.JSONB, nullable=True),
+        sa.Column("fingerprint_hash", sa.String(256), nullable=False, index=True),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Template matches
+    op.create_table(
+        "template_matches",
+        sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
+        sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True),
+        sa.Column("confidence_score", sa.Float, nullable=False),
+        sa.Column("match_details", postgresql.JSONB, nullable=True),
+        sa.Column("selected", sa.Boolean, default=False, nullable=False),
+        sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
+        schema=SCHEMA,
+    )
+
+    # Additional indexes
+    op.create_index("ix_documents_status_created", "documents", ["status", "created_at"], schema=SCHEMA)
+    op.create_index("ix_document_pages_doc_page", "document_pages", ["document_id", "page_number"], schema=SCHEMA)
+    op.create_index("ix_document_text_blocks_type", "document_text_blocks", ["page_id", "block_type"], schema=SCHEMA)
+    op.create_index("ix_document_cells_format_page", "document_cells", ["format_id", "page_number"], schema=SCHEMA)
+    op.create_index("ix_template_matches_doc_score", "template_matches", ["document_id", "confidence_score"], schema=SCHEMA)
+    op.create_index("ix_audit_logs_resource", "audit_logs", ["resource_type", "resource_id"], schema=SCHEMA)
+
+
+def downgrade() -> None:
+    op.drop_table("template_matches", schema=SCHEMA)
+    op.drop_table("template_fingerprints", schema=SCHEMA)
+    op.drop_table("image_regions", schema=SCHEMA)
+    op.drop_table("watermarks", schema=SCHEMA)
+    op.drop_table("table_rows", schema=SCHEMA)
+    op.drop_table("table_columns", schema=SCHEMA)
+    op.drop_table("table_formats", schema=SCHEMA)
+    op.drop_table("document_regions", schema=SCHEMA)
+    op.drop_table("document_cells", schema=SCHEMA)
+    op.drop_table("document_formats", schema=SCHEMA)
+    op.drop_table("document_tables", schema=SCHEMA)
+    op.drop_table("document_images", schema=SCHEMA)
+    op.drop_table("document_text_blocks", schema=SCHEMA)
+    op.drop_table("document_pages", schema=SCHEMA)
+    op.drop_table("documents", schema=SCHEMA)
+    op.drop_table("audit_logs", schema=SCHEMA)
+    op.drop_table("refresh_tokens", schema=SCHEMA)
+    op.drop_table("user_roles", schema=SCHEMA)
+    op.drop_table("roles", schema=SCHEMA)
+    op.drop_table("users", schema=SCHEMA)

docengine/app.py

@@ -0,0 +1,11 @@
+
+from fastapi import FastAPI
+app = FastAPI()
+
+@app.get("/health")
+def health():
+    return {"status":"UP"}
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=7989)

docengine/app/__init__.py

@@ -0,0 +1 @@
+# DocEngine - Document Template Recognition and Reconstruction System

docengine/app/api/router.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from fastapi import APIRouter
+
+from app.api.v1.auth import router as auth_router
+from app.api.v1.documents import router as documents_router
+from app.api.v1.health import router as health_router
+from app.api.v1.templates import router as templates_router
+
+api_v1_router = APIRouter(prefix="/api/v1")
+
+api_v1_router.include_router(health_router)
+api_v1_router.include_router(auth_router)
+api_v1_router.include_router(documents_router)
+api_v1_router.include_router(templates_router)

docengine/app/api/v1/auth.py

@@ -0,0 +1,246 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from sqlalchemy.orm import Session
+
+from app.core.config import settings
+from app.core.database import get_db
+from app.core.dependencies import CurrentUser
+from app.core.security import (
+    create_access_token,
+    create_refresh_token,
+    decode_token,
+    hash_password,
+    verify_password,
+    InvalidTokenError,
+)
+from app.repositories.user_repository import RefreshTokenRepository, UserRepository
+from app.schemas.auth import (
+    ChangePasswordRequest,
+    LoginRequest,
+    RefreshTokenRequest,
+    RegisterRequest,
+    TokenResponse,
+)
+from app.schemas.common import SuccessResponse
+from app.schemas.user import UserResponse
+
+router = APIRouter(prefix="/auth", tags=["Authentication"])
+
+
+@router.post(
+    "/register",
+    response_model=UserResponse,
+    status_code=status.HTTP_201_CREATED,
+    summary="Register User",
+    description="Register a new user account.",
+)
+def register(
+    payload: RegisterRequest,
+    db: Session = Depends(get_db),
+) -> UserResponse:
+    """Register a new user."""
+    user_repo = UserRepository(db)
+
+    # Check for existing user
+    if user_repo.get_by_username(payload.username):
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"Username '{payload.username}' is already taken",
+        )
+    if user_repo.get_by_email(payload.email):
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"Email '{payload.email}' is already registered",
+        )
+
+    hashed = hash_password(payload.password)
+    user = user_repo.create_user(
+        username=payload.username,
+        email=payload.email,
+        hashed_password=hashed,
+        full_name=payload.full_name,
+        role_names=["user"],
+    )
+    db.commit()
+    db.refresh(user)
+
+    return UserResponse.model_validate(user)
+
+
+@router.post(
+    "/login",
+    response_model=TokenResponse,
+    summary="Login",
+    description="Authenticate with username and password to obtain JWT tokens.",
+)
+def login(
+    payload: LoginRequest,
+    db: Session = Depends(get_db),
+) -> TokenResponse:
+    """Authenticate user and return JWT tokens."""
+    user_repo = UserRepository(db)
+    refresh_repo = RefreshTokenRepository(db)
+
+    user = user_repo.get_by_username(payload.username)
+    if not user or not verify_password(payload.password, user.hashed_password):
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid username or password",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+
+    if not user.is_active:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="User account is deactivated",
+        )
+
+    # Generate tokens
+    access_token = create_access_token(data={"sub": str(user.id), "username": user.username})
+    refresh_token_str = create_refresh_token(data={"sub": str(user.id)})
+
+    # Store refresh token
+    expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days)
+    refresh_repo.create_token(
+        user_id=user.id,
+        token=refresh_token_str,
+        expires_at=expires_at,
+    )
+
+    # Update last login
+    user_repo.update_last_login(user)
+    db.commit()
+
+    return TokenResponse(
+        access_token=access_token,
+        refresh_token=refresh_token_str,
+        token_type="bearer",
+        expires_in=settings.jwt_access_token_expire_minutes * 60,
+    )
+
+
+@router.post(
+    "/refresh",
+    response_model=TokenResponse,
+    summary="Refresh Token",
+    description="Obtain a new access token using a valid refresh token.",
+)
+def refresh_token(
+    payload: RefreshTokenRequest,
+    db: Session = Depends(get_db),
+) -> TokenResponse:
+    """Refresh access token using a refresh token."""
+    refresh_repo = RefreshTokenRepository(db)
+    user_repo = UserRepository(db)
+
+    # Validate the refresh token
+    try:
+        token_payload = decode_token(payload.refresh_token)
+    except InvalidTokenError:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid or expired refresh token",
+        )
+
+    if token_payload.get("type") != "refresh":
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid token type",
+        )
+
+    # Check if token exists in database and is not revoked
+    stored_token = refresh_repo.get_by_token(payload.refresh_token)
+    if not stored_token:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Refresh token not found or revoked",
+        )
+
+    user = user_repo.get_by_id(token_payload["sub"])
+    if not user or not user.is_active:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="User not found or deactivated",
+        )
+
+    # Revoke old refresh token
+    refresh_repo.revoke_token(payload.refresh_token)
+
+    # Generate new tokens
+    new_access_token = create_access_token(data={"sub": str(user.id), "username": user.username})
+    new_refresh_token = create_refresh_token(data={"sub": str(user.id)})
+
+    expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days)
+    refresh_repo.create_token(
+        user_id=user.id,
+        token=new_refresh_token,
+        expires_at=expires_at,
+    )
+    db.commit()
+
+    return TokenResponse(
+        access_token=new_access_token,
+        refresh_token=new_refresh_token,
+        token_type="bearer",
+        expires_in=settings.jwt_access_token_expire_minutes * 60,
+    )
+
+
+@router.post(
+    "/logout",
+    response_model=SuccessResponse,
+    summary="Logout",
+    description="Revoke the current refresh token.",
+)
+def logout(
+    payload: RefreshTokenRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> SuccessResponse:
+    """Logout by revoking the refresh token."""
+    refresh_repo = RefreshTokenRepository(db)
+    refresh_repo.revoke_token(payload.refresh_token)
+    db.commit()
+    return SuccessResponse(message="Successfully logged out")
+
+
+@router.post(
+    "/change-password",
+    response_model=SuccessResponse,
+    summary="Change Password",
+    description="Change the current user's password.",
+)
+def change_password(
+    payload: ChangePasswordRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> SuccessResponse:
+    """Change user password."""
+    if not verify_password(payload.current_password, current_user.hashed_password):
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Current password is incorrect",
+        )
+
+    current_user.hashed_password = hash_password(payload.new_password)
+
+    # Revoke all refresh tokens for security
+    refresh_repo = RefreshTokenRepository(db)
+    refresh_repo.revoke_all_user_tokens(current_user.id)
+    db.commit()
+
+    return SuccessResponse(message="Password changed successfully")
+
+
+@router.get(
+    "/me",
+    response_model=UserResponse,
+    summary="Get Current User",
+    description="Get the currently authenticated user's profile.",
+)
+def get_me(current_user: CurrentUser) -> UserResponse:
+    """Get current authenticated user profile."""
+    return UserResponse.model_validate(current_user)

docengine/app/api/v1/documents.py

@@ -0,0 +1,307 @@
+from __future__ import annotations
+
+import uuid
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, File, HTTPException, Query, UploadFile, status
+from sqlalchemy.orm import Session
+
+from app.core.config import settings
+from app.core.database import get_db
+from app.core.dependencies import CurrentUser
+from app.core.exceptions import FileSizeError, UnsupportedFileTypeError
+from app.core.logging_config import get_logger
+from app.models.document import Document
+from app.repositories.document_repository import DocumentRepository
+from app.schemas.common import PaginatedResponse, SuccessResponse
+from app.schemas.document import (
+    DocumentListResponse,
+    DocumentResponse,
+    DocumentUploadResponse,
+    TemplateMatchRequest,
+    TemplateMatchResponse,
+)
+from app.storage.provider import LocalStorageProvider, get_storage_provider
+
+logger = get_logger(__name__)
+
+router = APIRouter(prefix="/documents", tags=["Documents"])
+
+ALLOWED_CONTENT_TYPES = {
+    "image/jpeg": "jpg",
+    "image/png": "png",
+    "image/tiff": "tiff",
+    "application/pdf": "pdf",
+}
+
+ALLOWED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tiff", ".tif", ".pdf"}
+
+
+def _validate_file(file: UploadFile) -> str:
+    """Validate uploaded file type and size. Returns the content type."""
+    if not file.filename:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Filename is required",
+        )
+
+    # Check extension
+    from pathlib import Path
+    ext = Path(file.filename).suffix.lower()
+    if ext not in ALLOWED_EXTENSIONS:
+        raise UnsupportedFileTypeError(ext)
+
+    # Determine content type
+    content_type = file.content_type or ""
+    if content_type not in ALLOWED_CONTENT_TYPES:
+        # Try to infer from extension
+        ext_to_ct = {
+            ".jpg": "image/jpeg",
+            ".jpeg": "image/jpeg",
+            ".png": "image/png",
+            ".tiff": "image/tiff",
+            ".tif": "image/tiff",
+            ".pdf": "application/pdf",
+        }
+        content_type = ext_to_ct.get(ext, "")
+        if not content_type:
+            raise UnsupportedFileTypeError(file.content_type or "unknown")
+
+    return content_type
+
+
+@router.post(
+    "/upload",
+    response_model=DocumentUploadResponse,
+    status_code=status.HTTP_201_CREATED,
+    summary="Upload Document",
+    description="Upload a document (JPG, JPEG, PNG, TIFF, or PDF) for processing.",
+)
+async def upload_document(
+    file: UploadFile = File(..., description="Document file to upload"),
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> DocumentUploadResponse:
+    """Upload a document for processing."""
+    content_type = _validate_file(file)
+
+    # Read file data
+    file_data = await file.read()
+
+    # Check file size
+    if len(file_data) > settings.storage_max_file_size_bytes:
+        raise FileSizeError(settings.storage_max_file_size_mb)
+
+    # Store file
+    storage = get_storage_provider()
+    checksum = storage.compute_checksum(file_data)
+    safe_filename = file.filename or "unknown"
+    stored_filename = storage.generate_filename(safe_filename)
+    storage_path = storage.save_file(file_data, "documents", stored_filename)
+
+    # Create document record
+    doc_repo = DocumentRepository(db)
+    document = Document(
+        filename=stored_filename,
+        original_filename=safe_filename,
+        content_type=content_type,
+        file_size=len(file_data),
+        checksum=checksum,
+        storage_path=storage_path,
+        status="pending",
+        uploaded_by=current_user.id if current_user else None,
+    )
+    doc_repo.create(document)
+    db.commit()
+    db.refresh(document)
+
+    logger.info(
+        "document_uploaded",
+        document_id=str(document.id),
+        filename=safe_filename,
+        size=len(file_data),
+        content_type=content_type,
+    )
+
+    # Trigger async processing via Celery
+    try:
+        from app.tasks.document_tasks import process_document_task
+        process_document_task.delay(str(document.id))
+    except Exception as e:
+        logger.warning("celery_dispatch_failed", error=str(e), document_id=str(document.id))
+
+    return DocumentUploadResponse.model_validate(document)
+
+
+@router.get(
+    "/{document_id}",
+    response_model=DocumentResponse,
+    summary="Get Document",
+    description="Retrieve a document by its ID with all extracted content.",
+)
+def get_document(
+    document_id: uuid.UUID,
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> DocumentResponse:
+    """Get a document by ID."""
+    doc_repo = DocumentRepository(db)
+    document = doc_repo.get_with_pages(document_id)
+    if not document:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Document '{document_id}' not found",
+        )
+    return DocumentResponse.model_validate(document)
+
+
+@router.get(
+    "",
+    response_model=PaginatedResponse[DocumentListResponse],
+    summary="List Documents",
+    description="List all documents with pagination.",
+)
+def list_documents(
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=20, ge=1, le=100),
+    status_filter: str | None = Query(default=None, alias="status"),
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> PaginatedResponse[DocumentListResponse]:
+    """List documents with pagination and optional status filter."""
+    doc_repo = DocumentRepository(db)
+    offset = (page - 1) * page_size
+    filters = {}
+    if status_filter:
+        filters["status"] = status_filter
+
+    documents = doc_repo.get_all(
+        offset=offset,
+        limit=page_size,
+        filters=filters,
+        order_by="created_at",
+        order_desc=True,
+    )
+    total = doc_repo.count(filters=filters)
+
+    items = [DocumentListResponse.model_validate(doc) for doc in documents]
+    return PaginatedResponse.create(
+        items=items,
+        total=total,
+        page=page,
+        page_size=page_size,
+    )
+
+
+@router.get(
+    "/{document_id}/template",
+    response_model=list[TemplateMatchResponse],
+    summary="Get Document Template Matches",
+    description="Get template matching results for a document.",
+)
+def get_document_template_matches(
+    document_id: uuid.UUID,
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> list[TemplateMatchResponse]:
+    """Get template matches for a document."""
+    from app.repositories.document_repository import TemplateMatchRepository
+
+    doc_repo = DocumentRepository(db)
+    document = doc_repo.get_by_id(document_id)
+    if not document:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Document '{document_id}' not found",
+        )
+
+    match_repo = TemplateMatchRepository(db)
+    matches = match_repo.get_document_matches(document_id)
+
+    results = []
+    for match in matches:
+        resp = TemplateMatchResponse(
+            id=match.id,
+            document_id=match.document_id,
+            format_id=match.format_id,
+            confidence_score=match.confidence_score,
+            match_details=match.match_details,
+            selected=match.selected,
+            template_name=match.template.name if match.template else None,
+            created_at=match.created_at,
+        )
+        results.append(resp)
+
+    return results
+
+
+@router.post(
+    "/{document_id}/extraction",
+    response_model=dict,
+    summary="Extract Document Data",
+    description="Match a document against templates and extract key-value & table data.",
+)
+def extract_document_data(
+    document_id: uuid.UUID,
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> dict:
+    """Extract document data using matched template mappings."""
+    doc_repo = DocumentRepository(db)
+    document = doc_repo.get_by_id(document_id)
+    if not document:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Document '{document_id}' not found",
+        )
+
+    if document.status != "completed":
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Document must be in 'completed' status. Current status: '{document.status}'",
+        )
+
+    from app.services.extraction_service import ExtractionService
+    extraction_service = ExtractionService(db)
+    try:
+        result = extraction_service.extract_document_data(document_id)
+        return result
+    except Exception as e:
+        logger.exception("extraction_endpoint_failed", document_id=str(document_id), error=str(e))
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail=f"Extraction failed: {str(e)}",
+        )
+
+
+@router.delete(
+    "/{document_id}",
+    response_model=SuccessResponse,
+    summary="Delete Document",
+    description="Delete a document and its associated data.",
+)
+def delete_document(
+    document_id: uuid.UUID,
+    current_user: CurrentUser = None,
+    db: Session = Depends(get_db),
+) -> SuccessResponse:
+    """Delete a document."""
+    doc_repo = DocumentRepository(db)
+    document = doc_repo.get_by_id(document_id)
+    if not document:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Document '{document_id}' not found",
+        )
+
+    # Delete stored file
+    try:
+        storage = get_storage_provider()
+        storage.delete_file(document.storage_path)
+    except Exception as e:
+        logger.warning("file_delete_failed", error=str(e), path=document.storage_path)
+
+    doc_repo.delete(document)
+    db.commit()
+
+    return SuccessResponse(message=f"Document '{document_id}' deleted successfully")

docengine/app/api/v1/health.py

@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+from datetime import datetime
+
+import redis
+from fastapi import APIRouter, status
+
+from app.core.config import settings
+from app.core.database import check_database_connection
+from app.schemas.common import HealthResponse
+
+router = APIRouter(tags=["Health"])
+
+
+@router.get(
+    "/health",
+    response_model=HealthResponse,
+    status_code=status.HTTP_200_OK,
+    summary="Health Check",
+    description="Check the health status of the application and its dependencies.",
+)
+async def health_check() -> HealthResponse:
+    """Perform health check on all system components."""
+    # Check database
+    db_status = "healthy" if check_database_connection() else "unhealthy"
+
+    # Check Redis
+    redis_status = "healthy"
+    try:
+        r = redis.Redis(
+            host=settings.redis_host,
+            port=settings.redis_port,
+            db=settings.redis_db,
+            password=settings.redis_password or None,
+            socket_timeout=3,
+        )
+        r.ping()
+        r.close()
+    except Exception:
+        redis_status = "unhealthy"
+
+    overall_status = "healthy" if db_status == "healthy" and redis_status == "healthy" else "degraded"
+
+    return HealthResponse(
+        status=overall_status,
+        version=settings.app_version,
+        environment=settings.app_env,
+        database=db_status,
+        redis=redis_status,
+        timestamp=datetime.utcnow(),
+    )

docengine/app/api/v1/templates.py

@@ -0,0 +1,423 @@
+from __future__ import annotations
+
+import uuid
+
+from fastapi import APIRouter, Depends, HTTPException, Query, status
+from fastapi.responses import FileResponse
+from sqlalchemy.orm import Session
+
+from app.core.database import get_db
+from app.core.dependencies import CurrentUser
+from app.core.logging_config import get_logger
+from app.repositories.document_repository import DocumentRepository, TemplateMatchRepository
+from app.repositories.template_repository import (
+    TemplateRepository, 
+    DocumentRegionRepository, 
+    DocumentCellRepository,
+    TableFormatRepository,
+    TableColumnRepository
+)
+from app.schemas.common import PaginatedResponse, SuccessResponse
+from app.schemas.document import TemplateMatchRequest, TemplateMatchResponse
+from app.schemas.template import (
+    TemplateListResponse,
+    TemplateRenderRequest,
+    TemplateRenderResponse,
+    TemplateResponse,
+    TemplateCreateRequest,
+    TemplateMappingSaveRequest,
+)
+
+logger = get_logger(__name__)
+
+router = APIRouter(prefix="/templates", tags=["Templates"])
+
+
+@router.post(
+    "",
+    response_model=dict,
+    summary="Create Template",
+    description="Create a new template.",
+)
+def create_template(
+    payload: TemplateCreateRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> dict:
+    """Create a template."""
+    template_repo = TemplateRepository(db)
+    
+    template = template_repo.create_template(
+        name=payload.template_name,
+        page_width=1000.0,
+        page_height=1000.0,
+        source_document_id=uuid.UUID(payload.source_document_id) if payload.source_document_id else None
+    )
+    cell_repo = DocumentCellRepository(db)
+    saved_fields = []
+    
+    for index, field in enumerate(payload.fields):
+        cell = cell_repo.create_cell(
+            format_id=template.id,
+            page_number=1,
+            x=0.0,
+            y=0.0,
+            width=0.0,
+            height=0.0,
+            data_type=field.field_type,
+            field_name=field.field_label,
+            sequence=field.display_order or index,
+            is_dynamic=True
+        )
+        saved_fields.append({
+            "field_label": cell.field_name,
+            "field_type": cell.data_type
+        })
+        
+    db.commit()
+
+    return {
+        "pk_template_id": str(template.id),
+        "template_name": template.name,
+        "fields": saved_fields
+    }
+
+
+@router.put(
+    "/{template_id}",
+    response_model=dict,
+    summary="Update Template",
+    description="Update an existing template name and its fields.",
+)
+def update_template(
+    template_id: uuid.UUID,
+    payload: TemplateCreateRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> dict:
+    """Update a template."""
+    template_repo = TemplateRepository(db)
+    template = template_repo.get_by_id(template_id)
+    if not template:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Template '{template_id}' not found",
+        )
+
+    if template.name != payload.template_name:
+        template.name = payload.template_name
+
+    from app.models.template import DocumentCell
+    db.query(DocumentCell).filter(DocumentCell.format_id == template_id).delete()
+    
+    saved_fields = []
+    cell_repo = DocumentCellRepository(db)
+    for index, field in enumerate(payload.fields):
+        cell = cell_repo.create_cell(
+            format_id=template.id,
+            page_number=1,
+            row_no=0,
+            column_no=0,
+            field_name=field.field_label,
+            data_type=field.field_type,
+            x=0.0,
+            y=0.0,
+            width=0.0,
+            height=0.0,
+            is_dynamic=True
+        )
+        saved_fields.append({
+            "field_label": cell.field_name,
+            "field_type": cell.data_type
+        })
+        
+    db.commit()
+
+    return {
+        "pk_template_id": str(template.id),
+        "template_name": template.name,
+        "fields": saved_fields
+    }
+
+
+@router.post(
+    "/{template_id}/mappings/save",
+    response_model=SuccessResponse,
+    summary="Save Template Mappings",
+    description="Save the field mappings for a template.",
+)
+def save_mappings(
+    template_id: uuid.UUID,
+    payload: list[TemplateMappingSaveRequest],
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> SuccessResponse:
+    """Save template mappings."""
+    template_repo = TemplateRepository(db)
+    template = template_repo.get_by_id(template_id)
+    if not template:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Template '{template_id}' not found",
+        )
+
+    region_repo = DocumentRegionRepository(db)
+    cell_repo = DocumentCellRepository(db)
+    table_repo = TableFormatRepository(db)
+    table_col_repo = TableColumnRepository(db)
+    
+    # Delete existing field mappings and table formats for this template
+    from app.models.template import DocumentRegion, TableFormat
+    db.query(DocumentRegion).filter(
+        DocumentRegion.format_id == template_id, 
+        DocumentRegion.region_type == "field_mapping"
+    ).delete()
+    db.query(TableFormat).filter(TableFormat.format_id == template_id).delete()
+    db.flush()
+
+    cells = cell_repo.get_template_cells(template_id)
+    cell_map = {c.field_name: c for c in cells}
+    table_format = None
+    
+    # Save each mapped node
+    for mapping in payload:
+        cell = cell_map.get(mapping.field_name)
+        is_table_column = cell and cell.data_type == 'TABLE_COLUMN'
+        
+        for node_idx, node in enumerate(mapping.mapped_nodes):
+            if is_table_column:
+                if not table_format:
+                    table_format = table_repo.create_table_format(
+                        format_id=template_id,
+                        page_number=node.page_no or 1,
+                        x=0.0, y=0.0, width=1000.0, height=1000.0,
+                        rows=1, columns=10
+                    )
+                table_col_repo.create_column(
+                    table_format_id=table_format.id,
+                    column_index=node_idx,
+                    width=node.width,
+                    header_text=mapping.field_name,
+                    data_type="text"
+                )
+
+            region_repo.create_region(
+                format_id=template_id,
+                page_number=node.page_no or 1,
+                region_type="field_mapping",
+                x=node.x_coordinate,
+                y=node.y_coordinate,
+                width=node.width,
+                height=node.height,
+                content={
+                    "field_name": mapping.field_name,
+                    "text_value": node.text_value
+                }
+            )
+            
+    db.commit()
+    
+    # Generate fingerprint now that mappings are populated
+    from app.services.fingerprint_service import FingerprintService
+    FingerprintService(db).generate_fingerprint(template)
+    db.commit()
+    
+    return SuccessResponse(message="Mappings saved successfully")
+
+
+@router.get(
+    "",
+    response_model=PaginatedResponse[TemplateListResponse],
+    summary="List Templates",
+    description="List all active templates with pagination.",
+)
+def list_templates(
+    current_user: CurrentUser,
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=20, ge=1, le=100),
+    db: Session = Depends(get_db),
+) -> PaginatedResponse[TemplateListResponse]:
+    """List all active templates."""
+    template_repo = TemplateRepository(db)
+    offset = (page - 1) * page_size
+
+    templates = template_repo.get_active_templates(offset=offset, limit=page_size)
+    total = template_repo.count_active()
+
+    items = [TemplateListResponse.model_validate(t) for t in templates]
+    return PaginatedResponse.create(
+        items=items,
+        total=total,
+        page=page,
+        page_size=page_size,
+    )
+
+
+@router.get(
+    "/{template_id}",
+    response_model=TemplateResponse,
+    summary="Get Template",
+    description="Retrieve a template by ID with all its components.",
+)
+def get_template(
+    template_id: uuid.UUID,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> TemplateResponse:
+    """Get a template by ID."""
+    template_repo = TemplateRepository(db)
+    template = template_repo.get_by_id(template_id)
+    if not template:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Template '{template_id}' not found",
+        )
+    return TemplateResponse.model_validate(template)
+
+
+@router.delete(
+    "/{template_id}",
+    response_model=SuccessResponse,
+    summary="Delete Template",
+    description="Soft-delete a template by deactivating it.",
+)
+def delete_template(
+    template_id: uuid.UUID,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> SuccessResponse:
+    """Soft-delete a template."""
+    template_repo = TemplateRepository(db)
+    template = template_repo.get_by_id(template_id)
+    if not template:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Template '{template_id}' not found",
+        )
+
+    template_repo.deactivate_template(template_id)
+    db.commit()
+
+    return SuccessResponse(message=f"Template '{template_id}' deactivated successfully")
+
+
+@router.post(
+    "/match",
+    response_model=list[TemplateMatchResponse],
+    summary="Match Document to Templates",
+    description="Match a document against existing templates and return ranked results.",
+)
+def match_template(
+    payload: TemplateMatchRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> list[TemplateMatchResponse]:
+    """Match a document against existing templates."""
+    doc_repo = DocumentRepository(db)
+    document = doc_repo.get_by_id(payload.document_id)
+    if not document:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Document '{payload.document_id}' not found",
+        )
+
+    if document.status != "completed":
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Document must be in 'completed' status. Current status: '{document.status}'",
+        )
+
+    # Perform template matching
+    from app.services.matching_service import MatchingService
+    matching_service = MatchingService(db)
+    matches = matching_service.match_document(
+        document_id=payload.document_id,
+        min_confidence=payload.min_confidence,
+        max_results=payload.max_results,
+    )
+    db.commit()
+
+    results = []
+    for match in matches:
+        resp = TemplateMatchResponse(
+            id=match.id,
+            document_id=match.document_id,
+            format_id=match.format_id,
+            confidence_score=match.confidence_score,
+            match_details=match.match_details,
+            selected=match.selected,
+            template_name=match.template.name if match.template else None,
+            created_at=match.created_at,
+        )
+        results.append(resp)
+
+    return results
+
+
+@router.post(
+    "/render",
+    response_model=TemplateRenderResponse,
+    summary="Render Template to PDF",
+    description="Generate a PDF from a stored template with supplied data.",
+)
+def render_template(
+    payload: TemplateRenderRequest,
+    current_user: CurrentUser,
+    db: Session = Depends(get_db),
+) -> TemplateRenderResponse:
+    """Render a template to PDF."""
+    template_repo = TemplateRepository(db)
+    template = template_repo.get_by_id(payload.template_id)
+    if not template:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Template '{payload.template_id}' not found",
+        )
+
+    if not template.is_active:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Template is deactivated",
+        )
+
+    from app.services.reconstruction_service import ReconstructionService
+    reconstruction_service = ReconstructionService(db)
+    result = reconstruction_service.render_template(
+        template=template,
+        data=payload.data,
+        output_filename=payload.output_filename,
+        images=payload.images,
+    )
+
+    return result
+
+
+@router.get(
+    "/{template_id}/download",
+    summary="Download Rendered PDF",
+    description="Download a previously rendered PDF.",
+)
+def download_rendered_pdf(
+    template_id: uuid.UUID,
+    current_user: CurrentUser,
+    filename: str = Query(..., description="Filename of the rendered PDF"),
+    db: Session = Depends(get_db),
+) -> FileResponse:
+    """Download a rendered PDF."""
+    from app.storage.provider import get_storage_provider
+
+    storage = get_storage_provider()
+    storage_path = f"rendered/{filename}"
+
+    if not storage.file_exists(storage_path):
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Rendered PDF '{filename}' not found",
+        )
+
+    absolute_path = storage.get_absolute_path(storage_path)
+    return FileResponse(
+        path=absolute_path,
+        media_type="application/pdf",
+        filename=filename,
+    )

docengine/app/core/aes.py

@@ -0,0 +1,38 @@
+import base64
+from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
+
+def decrypt(base64_input: str, secret: bytes) -> str | None:
+    try:
+        combined = base64.b64decode(base64_input)
+        
+        # IV length is 12 bytes in the Java implementation
+        iv_length = 12
+        if len(combined) <= iv_length + 16:
+            return None
+            
+        iv = combined[:iv_length]
+        
+        # Java AES/GCM appends a 16-byte authentication tag at the end of cipherBytes
+        # cryptography library expects it to be passed into the modes.GCM(iv, tag)
+        cipher_bytes_with_tag = combined[iv_length:]
+        actual_ciphertext = cipher_bytes_with_tag[:-16]
+        tag = cipher_bytes_with_tag[-16:]
+        
+        cipher = Cipher(algorithms.AES(secret), modes.GCM(iv, tag))
+        decryptor = cipher.decryptor()
+        
+        plain_bytes = decryptor.update(actual_ciphertext) + decryptor.finalize()
+        return plain_bytes.decode('utf-8')
+    except Exception:
+        return None
+
+def get_string_value(encrypted_id: str, secret: str, secret_key_internal: str) -> str | None:
+    if not secret or not secret_key_internal:
+        return None
+    try:
+        # Replicate Java's logic: (s + secretKeyInternal).substring(0, Math.min(..., 16))
+        combined_str = secret + secret_key_internal
+        combined_secret = combined_str[:16].encode('utf-8')
+        return decrypt(encrypted_id, combined_secret)
+    except Exception:
+        return None