diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000..ab67dbd Binary files /dev/null and b/.DS_Store differ diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 0000000..987c603 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "dotrush.roslyn.projectOrSolutionFiles": [] +} \ No newline at end of file diff --git a/SESSION_HANDOFF.md b/SESSION_HANDOFF.md new file mode 100644 index 0000000..dc217dc --- /dev/null +++ b/SESSION_HANDOFF.md @@ -0,0 +1,42 @@ +# Session Handoff & Context + +**Date Saved**: June 7, 2026 +**Conversation ID**: 43f3aa27-9124-446e-9713-4e31e083b4f5 + +## Project: Template Mapping Engine (OCR) +A system designed to extract document layouts via OCR and allow users to map specific bounding box regions to template fields using a drag-and-drop UI. Future documents from the same vendor are automatically recognized and mapped using a heuristic template engine. + +## What Has Been Completed So Far + +### 1. Backend Startup & Dependency Fixes: +- **Auto-Bootstrapping**: Updated `run_docengine.sh` and `run_backend.sh` to automatically set up virtual environments and install missing dependencies if key modules (`uvicorn`, `numpy`, etc.) are missing. +- **Python 3.13 Compatibility**: Upgraded `paddlepaddle` in `docengine/requirements.txt` to `>=3.0.0` to support Python 3.13 on macOS ARM64. Added missing dependencies (`numpy`, `opencv-python-headless`, `pdf2image`) to `backend/requirements.txt`. +- **CORS Config Fix**: Corrected a missing double-quote in `CORS_ORIGINS` in both `.env` and `.env.example` in `docengine/`. + +### 2. Backend Code Integrity & Framework Alignment: +- **SQLAlchemy 2.0 / Imperative Mapping Fix**: Updated `user_roles_table` in `docengine/app/models/user.py` to use SQLAlchemy `Column` objects instead of `mapped_column`, preventing a database initialization crash. +- **Structlog Parameter Fix**: Replaced the `event` keyword argument with `phase` in `docengine/app/events/handlers.py` to prevent a duplicate parameter `TypeError` from the structlog library. +- **Optional Development Auth Bypass**: Modified `get_current_user` in `docengine/app/core/dependencies.py` to automatically fallback to the first database user (or a fake dev admin model) when no authorization token is supplied and `APP_ENV` is set to `"development"`. This prevents `401 Unauthorized` errors when testing endpoints locally. + +### 3. Frontend Angular Implementation: +- **API Endpoint Alignments**: Removed the duplicate `/api` prefix from endpoints in `TemplateService.ts` (e.g. changing `${this.apiUrl}/api/documents/upload` to `${this.apiUrl}/documents/upload`). This resolves the `404 Not Found` error by aligning with the backend prefix `/api/v1/documents/upload`. + +--- + +## Where to Pick Up Next +1. **Initialize Database Tables**: + - The database currently lacks the necessary schema tables (evidenced by the error: `relation "admin.users" does not exist`). + - Run the setup command to build the tables and apply migrations: + ```bash + ./run_docengine.sh setup + ``` +2. **Retest Upload Flow**: + - Once database tables exist, perform a document upload from the Angular frontend to visually verify the OCR bounding boxes and layout mapper. +3. **Verify Heuristic Template Recognition**: + - Test the template matching and layout coordinate storage workflows. + +--- + +## How to Resume with AI Assistant +To resume this session in a new chat, you can tell the AI: +*"Please read `SESSION_HANDOFF.md` in the root of the OCR project to get context on the Template Engine we were building and the startup fixes, and let's continue."* diff --git a/backend/.env b/backend/.env index e16f982..bcc992f 100644 --- a/backend/.env +++ b/backend/.env @@ -1,7 +1,7 @@ DB_USER=postgres -DB_PASSWORD=M@tr!x#149@dm!N -DB_HOST=192.168.0.111 -DB_PORT=7925 +DB_PASSWORD=M@triXPostgr3s@6202 +DB_HOST=103.125.129.116 +DB_PORT=5432 DB_NAME=ocr # Mail Configuration (Gmail) diff --git a/backend/__pycache__/database.cpython-313.pyc b/backend/__pycache__/database.cpython-313.pyc index 5a00784..f6a1b74 100644 Binary files a/backend/__pycache__/database.cpython-313.pyc and b/backend/__pycache__/database.cpython-313.pyc differ diff --git a/backend/__pycache__/llm_service.cpython-313.pyc b/backend/__pycache__/llm_service.cpython-313.pyc new file mode 100644 index 0000000..899f291 Binary files /dev/null and b/backend/__pycache__/llm_service.cpython-313.pyc differ diff --git a/backend/__pycache__/main.cpython-313.pyc b/backend/__pycache__/main.cpython-313.pyc index c17dccd..30099f3 100644 Binary files a/backend/__pycache__/main.cpython-313.pyc and b/backend/__pycache__/main.cpython-313.pyc differ diff --git a/backend/alembic.ini b/backend/alembic.ini new file mode 100644 index 0000000..807ded2 --- /dev/null +++ b/backend/alembic.ini @@ -0,0 +1,149 @@ +# A generic, single database configuration. + +[alembic] +# path to migration scripts. +# this is typically a path given in POSIX (e.g. forward slashes) +# format, relative to the token %(here)s which refers to the location of this +# ini file +script_location = %(here)s/alembic + +# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s +# Uncomment the line below if you want the files to be prepended with date and time +# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file +# for all available tokens +# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s +# Or organize into date-based subdirectories (requires recursive_version_locations = true) +# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s + +# sys.path path, will be prepended to sys.path if present. +# defaults to the current working directory. for multiple paths, the path separator +# is defined by "path_separator" below. +prepend_sys_path = . + + +# timezone to use when rendering the date within the migration file +# as well as the filename. +# If specified, requires the tzdata library which can be installed by adding +# `alembic[tz]` to the pip requirements. +# string value is passed to ZoneInfo() +# leave blank for localtime +# timezone = + +# max length of characters to apply to the "slug" field +# truncate_slug_length = 40 + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + +# set to 'true' to allow .pyc and .pyo files without +# a source .py file to be detected as revisions in the +# versions/ directory +# sourceless = false + +# version location specification; This defaults +# to /versions. When using multiple version +# directories, initial revisions must be specified with --version-path. +# The path separator used here should be the separator specified by "path_separator" +# below. +# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions + +# path_separator; This indicates what character is used to split lists of file +# paths, including version_locations and prepend_sys_path within configparser +# files such as alembic.ini. +# The default rendered in new alembic.ini files is "os", which uses os.pathsep +# to provide os-dependent path splitting. +# +# Note that in order to support legacy alembic.ini files, this default does NOT +# take place if path_separator is not present in alembic.ini. If this +# option is omitted entirely, fallback logic is as follows: +# +# 1. Parsing of the version_locations option falls back to using the legacy +# "version_path_separator" key, which if absent then falls back to the legacy +# behavior of splitting on spaces and/or commas. +# 2. Parsing of the prepend_sys_path option falls back to the legacy +# behavior of splitting on spaces, commas, or colons. +# +# Valid values for path_separator are: +# +# path_separator = : +# path_separator = ; +# path_separator = space +# path_separator = newline +# +# Use os.pathsep. Default configuration used for new projects. +path_separator = os + +# set to 'true' to search source files recursively +# in each "version_locations" directory +# new in Alembic version 1.10 +# recursive_version_locations = false + +# the output encoding used when revision files +# are written from script.py.mako +# output_encoding = utf-8 + +# database URL. This is consumed by the user-maintained env.py script only. +# other means of configuring database URLs may be customized within the env.py +# file. +sqlalchemy.url = driver://user:pass@localhost/dbname + + +[post_write_hooks] +# post_write_hooks defines scripts or Python functions that are run +# on newly generated revision scripts. See the documentation for further +# detail and examples + +# format using "black" - use the console_scripts runner, against the "black" entrypoint +# hooks = black +# black.type = console_scripts +# black.entrypoint = black +# black.options = -l 79 REVISION_SCRIPT_FILENAME + +# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module +# hooks = ruff +# ruff.type = module +# ruff.module = ruff +# ruff.options = check --fix REVISION_SCRIPT_FILENAME + +# Alternatively, use the exec runner to execute a binary found on your PATH +# hooks = ruff +# ruff.type = exec +# ruff.executable = ruff +# ruff.options = check --fix REVISION_SCRIPT_FILENAME + +# Logging configuration. This is also consumed by the user-maintained +# env.py script only. +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARNING +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARNING +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/backend/alembic/README b/backend/alembic/README new file mode 100644 index 0000000..98e4f9c --- /dev/null +++ b/backend/alembic/README @@ -0,0 +1 @@ +Generic single-database configuration. \ No newline at end of file diff --git a/backend/alembic/__pycache__/env.cpython-313.pyc b/backend/alembic/__pycache__/env.cpython-313.pyc new file mode 100644 index 0000000..87ac7c6 Binary files /dev/null and b/backend/alembic/__pycache__/env.cpython-313.pyc differ diff --git a/backend/alembic/env.py b/backend/alembic/env.py new file mode 100644 index 0000000..9175000 --- /dev/null +++ b/backend/alembic/env.py @@ -0,0 +1,87 @@ +from logging.config import fileConfig + +from sqlalchemy import engine_from_config +from sqlalchemy import pool + +from alembic import context + +# this is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config + +# Interpret the config file for Python logging. +# This line sets up loggers basically. +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +# add your model's MetaData object here +# for 'autogenerate' support +import os +import sys +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from database import Base, DATABASE_URL +import models.template_models + +target_metadata = Base.metadata +config.set_main_option("sqlalchemy.url", DATABASE_URL.replace("%", "%%")) + +# other values from the config, defined by the needs of env.py, +# can be acquired: +# my_important_option = config.get_main_option("my_important_option") +# ... etc. + + +def run_migrations_offline() -> None: + """Run migrations in 'offline' mode. + + This configures the context with just a URL + and not an Engine, though an Engine is acceptable + here as well. By skipping the Engine creation + we don't even need a DBAPI to be available. + + Calls to context.execute() here emit the given string to the + script output. + + """ + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + ) + + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + """Run migrations in 'online' mode. + + In this scenario we need to create an Engine + and associate a connection with the context. + + """ + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + + with connectable.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata, + include_schemas=True, + version_table_schema='public' + ) + + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/backend/alembic/script.py.mako b/backend/alembic/script.py.mako new file mode 100644 index 0000000..1101630 --- /dev/null +++ b/backend/alembic/script.py.mako @@ -0,0 +1,28 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision: str = ${repr(up_revision)} +down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)} +branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} +depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} + + +def upgrade() -> None: + """Upgrade schema.""" + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + """Downgrade schema.""" + ${downgrades if downgrades else "pass"} diff --git a/backend/alembic/versions/91b2545f5a30_add_templates_schema.py b/backend/alembic/versions/91b2545f5a30_add_templates_schema.py new file mode 100644 index 0000000..8279a15 --- /dev/null +++ b/backend/alembic/versions/91b2545f5a30_add_templates_schema.py @@ -0,0 +1,28 @@ +"""Add templates schema + +Revision ID: 91b2545f5a30 +Revises: +Create Date: 2026-06-01 16:28:57.181829 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +import os + +# revision identifiers, used by Alembic. +revision: str = '91b2545f5a30' +down_revision: Union[str, None] = None +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +def upgrade() -> None: + # Read and execute the SQL schema script + sql_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), 'sql', '01_templates_schema.sql') + with open(sql_path, 'r') as f: + sql = f.read() + op.execute(sql) + +def downgrade() -> None: + op.execute("DROP SCHEMA IF EXISTS templates CASCADE;") diff --git a/backend/alembic/versions/__pycache__/91b2545f5a30_add_templates_schema.cpython-313.pyc b/backend/alembic/versions/__pycache__/91b2545f5a30_add_templates_schema.cpython-313.pyc new file mode 100644 index 0000000..db28e36 Binary files /dev/null and b/backend/alembic/versions/__pycache__/91b2545f5a30_add_templates_schema.cpython-313.pyc differ diff --git a/backend/api/__pycache__/documents.cpython-313.pyc b/backend/api/__pycache__/documents.cpython-313.pyc new file mode 100644 index 0000000..c607e16 Binary files /dev/null and b/backend/api/__pycache__/documents.cpython-313.pyc differ diff --git a/backend/api/__pycache__/templates.cpython-313.pyc b/backend/api/__pycache__/templates.cpython-313.pyc new file mode 100644 index 0000000..a7d2d21 Binary files /dev/null and b/backend/api/__pycache__/templates.cpython-313.pyc differ diff --git a/backend/api/documents.py b/backend/api/documents.py new file mode 100644 index 0000000..5b934da --- /dev/null +++ b/backend/api/documents.py @@ -0,0 +1,54 @@ +import json +from fastapi import APIRouter, Depends, HTTPException, status, UploadFile, File +from sqlalchemy.orm import Session +from typing import List + +from database import get_db +from models.template_models import TemplateDocument, DocumentLayout +from schemas.template_schemas import DocumentSchema, DocumentLayoutSchema, TemplateRecognitionResult +from engine.ocr.document_processor import DocumentProcessor +from engine.recognition.template_engine import TemplateRecognitionEngine + +router = APIRouter(prefix="/api/documents", tags=["Documents"]) + +@router.post("/upload", response_model=DocumentSchema, status_code=status.HTTP_201_CREATED) +async def upload_document(file: UploadFile = File(...), db: Session = Depends(get_db)): + # 1. Create document record + content = await file.read() + file_hash = hash(content) # Simple hash for demo + + db_doc = TemplateDocument( + document_name=file.filename, + document_type=file.content_type, + file_name=file.filename, + file_hash=str(file_hash), + status="processing" + ) + db.add(db_doc) + db.commit() + db.refresh(db_doc) + + # 2. Process File (OCR & Layout) + processor = DocumentProcessor(db) + processor.process_file(db_doc.pk_document_id, file.filename, content) + + db.refresh(db_doc) + return db_doc + +@router.get("/{document_id}", response_model=DocumentSchema) +def get_document(document_id: int, db: Session = Depends(get_db)): + doc = db.query(TemplateDocument).filter(TemplateDocument.pk_document_id == document_id).first() + if not doc: + raise HTTPException(status_code=404, detail="Document not found") + return doc + +@router.get("/{document_id}/layout", response_model=List[DocumentLayoutSchema]) +def get_document_layout(document_id: int, db: Session = Depends(get_db)): + layouts = db.query(DocumentLayout).filter(DocumentLayout.fk_document_id == document_id).order_by(DocumentLayout.sequence_no).all() + return layouts + +@router.post("/{document_id}/recognize", response_model=TemplateRecognitionResult) +def recognize_template(document_id: int, db: Session = Depends(get_db)): + engine = TemplateRecognitionEngine(db) + return engine.recognize_template(document_id) + diff --git a/backend/api/templates.py b/backend/api/templates.py new file mode 100644 index 0000000..bc859cd --- /dev/null +++ b/backend/api/templates.py @@ -0,0 +1,53 @@ +from fastapi import APIRouter, Depends, HTTPException, status +from sqlalchemy.orm import Session +from typing import List + +from database import get_db +from schemas.template_schemas import ( + TemplateCreate, TemplateSchema, + TemplateFieldCreate, TemplateFieldSchema, + TemplateFieldMappingCreate, TemplateFieldMappingSchema +) +from services.template_service import TemplateService + +router = APIRouter(prefix="/api/templates", tags=["Templates"]) + +@router.post("", response_model=TemplateSchema, status_code=status.HTTP_201_CREATED) +def create_template(template_data: TemplateCreate, db: Session = Depends(get_db)): + service = TemplateService(db) + return service.create_template(template_data) + +@router.get("", response_model=List[TemplateSchema]) +def get_templates(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)): + service = TemplateService(db) + return service.get_all_templates(skip, limit) + +@router.get("/{template_id}", response_model=TemplateSchema) +def get_template(template_id: int, db: Session = Depends(get_db)): + service = TemplateService(db) + template = service.get_template(template_id) + if not template: + raise HTTPException(status_code=404, detail="Template not found") + return template + +@router.delete("/{template_id}", status_code=status.HTTP_204_NO_CONTENT) +def delete_template(template_id: int, db: Session = Depends(get_db)): + service = TemplateService(db) + if not service.delete_template(template_id): + raise HTTPException(status_code=404, detail="Template not found") + return None + +@router.post("/{template_id}/fields", response_model=TemplateFieldSchema, status_code=status.HTTP_201_CREATED) +def add_template_field(template_id: int, field_data: TemplateFieldCreate, db: Session = Depends(get_db)): + service = TemplateService(db) + # Check if template exists + if not service.get_template(template_id): + raise HTTPException(status_code=404, detail="Template not found") + return service.add_template_field(template_id, field_data) + +@router.post("/{template_id}/mappings/save", response_model=List[TemplateFieldMappingSchema]) +def save_mappings(template_id: int, mappings: List[TemplateFieldMappingCreate], db: Session = Depends(get_db)): + service = TemplateService(db) + if not service.get_template(template_id): + raise HTTPException(status_code=404, detail="Template not found") + return service.save_mapping(template_id, mappings) diff --git a/backend/database.py b/backend/database.py index 1467745..af155b9 100644 --- a/backend/database.py +++ b/backend/database.py @@ -1,5 +1,7 @@ import os -from sqlalchemy import create_engine, Column, Integer, String, Text, DateTime, Boolean, ForeignKey, LargeBinary +import urllib.parse +from sqlalchemy import create_engine, Column, Integer, String, Text, DateTime, Boolean, ForeignKey, LargeBinary, Enum as SqlEnum +from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker, relationship from dotenv import load_dotenv @@ -7,22 +9,11 @@ from dotenv import load_dotenv # Load environment variables load_dotenv() -DB_USER = os.getenv("DB_USER") -DB_PASSWORD = os.getenv("DB_PASSWORD") -DB_HOST = os.getenv("DB_HOST") -DB_PORT = os.getenv("DB_PORT") -import urllib.parse - -# ... (imports) - -# Load environment variables -load_dotenv() - -DB_USER = os.getenv("DB_USER") -DB_PASSWORD = os.getenv("DB_PASSWORD") -DB_HOST = os.getenv("DB_HOST") -DB_PORT = os.getenv("DB_PORT") -DB_NAME = os.getenv("DB_NAME") +DB_USER = os.getenv("DB_USER", "postgres") +DB_PASSWORD = os.getenv("DB_PASSWORD", "M@tr!x#149@dm!N") +DB_HOST = os.getenv("DB_HOST", "192.168.0.111") +DB_PORT = os.getenv("DB_PORT", "7925") +DB_NAME = os.getenv("DB_NAME", "ocr") encoded_user = urllib.parse.quote_plus(DB_USER) encoded_password = urllib.parse.quote_plus(DB_PASSWORD) @@ -35,6 +26,26 @@ SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) Base = declarative_base() +class Vendor(Base): + __tablename__ = "vendors" + + id = Column(Integer, primary_key=True, index=True) + name = Column(String, unique=True, index=True) + default_model = Column(String) # 'text' or 'vision' + created_at = Column(DateTime) + +class Document(Base): + __tablename__ = "documents" + + id = Column(Integer, primary_key=True, index=True) + vendor_id = Column(Integer, ForeignKey("vendors.id"), nullable=True) + filename = Column(String) + upload_date = Column(DateTime) + status = Column(String) # 'pending', 'verified' + processed_data = Column(JSONB) # Store the verified extraction results + + vendor = relationship("Vendor") + class Email(Base): __tablename__ = "emails" @@ -54,7 +65,7 @@ class Attachment(Base): email_id = Column(Integer, ForeignKey("emails.id")) filename = Column(String) content_type = Column(String) - file_content = Column(LargeBinary) # Storing content directly in DB as requested + file_content = Column(LargeBinary) email = relationship("Email", back_populates="attachments") diff --git a/backend/engine/ocr/__pycache__/document_processor.cpython-313.pyc b/backend/engine/ocr/__pycache__/document_processor.cpython-313.pyc new file mode 100644 index 0000000..3c8b5c2 Binary files /dev/null and b/backend/engine/ocr/__pycache__/document_processor.cpython-313.pyc differ diff --git a/backend/engine/ocr/document_processor.py b/backend/engine/ocr/document_processor.py new file mode 100644 index 0000000..c12a129 --- /dev/null +++ b/backend/engine/ocr/document_processor.py @@ -0,0 +1,217 @@ +import io +import re +from typing import List, Dict, Any +import numpy as np +import cv2 +import pdfplumber +import pytesseract +from PIL import Image +from pdf2image import convert_from_bytes +from sqlalchemy.orm import Session +from models.template_models import DocumentLayout, TemplateDocument +from schemas.template_schemas import DocumentLayoutSchema + +class DocumentProcessor: + def __init__(self, db: Session): + self.db = db + + def process_file(self, document_id: int, filename: str, file_bytes: bytes) -> List[DocumentLayout]: + document = self.db.query(TemplateDocument).filter(TemplateDocument.pk_document_id == document_id).first() + if not document: + raise ValueError("Document not found") + + layouts = [] + if filename.lower().endswith('.pdf'): + layouts = self._process_pdf(document_id, file_bytes) + elif filename.lower().endswith(('.png', '.jpg', '.jpeg', '.tiff')): + layouts = self._process_image(document_id, file_bytes) + + # Update page count + if layouts: + document.page_count = max([l.page_no for l in layouts]) + document.status = "processed" + self.db.commit() + + return layouts + + def _process_pdf(self, document_id: int, file_bytes: bytes) -> List[DocumentLayout]: + layouts = [] + try: + with pdfplumber.open(io.BytesIO(file_bytes)) as pdf: + sequence_no = 1 + for page_no, page in enumerate(pdf.pages, start=1): + # Check if it has text + text = page.extract_text() + if text and text.strip(): + # Extract words with bounding boxes + words = page.extract_words() + + # Group words into blocks using simple proximity heuristics + blocks = self._group_words_to_blocks(words, page.width, page.height) + + for block in blocks: + db_layout = DocumentLayout( + fk_document_id=document_id, + page_no=page_no, + text_value=block['text'], + block_type=block['type'], + x_coordinate=block['x0'], + y_coordinate=block['top'], + width=block['x1'] - block['x0'], + height=block['bottom'] - block['top'], + confidence=100.0, + sequence_no=sequence_no + ) + self.db.add(db_layout) + layouts.append(db_layout) + sequence_no += 1 + else: + # Scanned PDF page -> convert to image and process + # Handled separately to avoid complexity in this mock + pass + except Exception as e: + print(f"Error processing PDF with pdfplumber: {e}") + + if not layouts: + # Fallback to image-based processing for scanned PDFs + images = convert_from_bytes(file_bytes, dpi=300) + sequence_no = 1 + for page_no, img in enumerate(images, start=1): + img_byte_arr = io.BytesIO() + img.save(img_byte_arr, format='PNG') + page_layouts = self._process_image(document_id, img_byte_arr.getvalue(), page_no=page_no, start_sequence=sequence_no) + layouts.extend(page_layouts) + sequence_no += len(page_layouts) + + self.db.commit() + return layouts + + def _process_image(self, document_id: int, file_bytes: bytes, page_no: int = 1, start_sequence: int = 1) -> List[DocumentLayout]: + layouts = [] + try: + # Decode image + nparr = np.frombuffer(file_bytes, np.uint8) + img = cv2.imdecode(nparr, cv2.IMREAD_COLOR) + + # Preprocessing + gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) + # Denoise & Threshold + gray = cv2.medianBlur(gray, 3) + gray = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1] + + # OCR using PyTesseract with bounding boxes + data = pytesseract.image_to_data(gray, output_type=pytesseract.Output.DICT) + + sequence_no = start_sequence + + # Grouping words into lines/blocks simplified + height, width = gray.shape + blocks = self._tesseract_data_to_blocks(data, width, height) + + for block in blocks: + db_layout = DocumentLayout( + fk_document_id=document_id, + page_no=page_no, + text_value=block['text'], + block_type=block['type'], + x_coordinate=block['x'], + y_coordinate=block['y'], + width=block['w'], + height=block['h'], + confidence=block['conf'], + sequence_no=sequence_no + ) + self.db.add(db_layout) + layouts.append(db_layout) + sequence_no += 1 + + except Exception as e: + print(f"Error processing image with tesseract: {e}") + + self.db.commit() + return layouts + + def _group_words_to_blocks(self, words: List[Dict], page_width: float, page_height: float) -> List[Dict]: + """Group nearby words into logical blocks and classify them.""" + # Extremely simplified grouping by Y coordinate proximity + blocks = [] + if not words: + return blocks + + # Sort words by Y then X + words.sort(key=lambda w: (w['top'], w['x0'])) + + current_block = { + 'text': words[0]['text'], + 'x0': words[0]['x0'], + 'top': words[0]['top'], + 'x1': words[0]['x1'], + 'bottom': words[0]['bottom'] + } + + for word in words[1:]: + # If word is roughly on the same line (y diff is small) and close horizontally + if abs(word['top'] - current_block['top']) < 10 and (word['x0'] - current_block['x1']) < 50: + current_block['text'] += ' ' + word['text'] + current_block['x1'] = word['x1'] + current_block['top'] = min(current_block['top'], word['top']) + current_block['bottom'] = max(current_block['bottom'], word['bottom']) + else: + current_block['type'] = self._classify_block(current_block, page_height) + blocks.append(current_block) + current_block = { + 'text': word['text'], + 'x0': word['x0'], + 'top': word['top'], + 'x1': word['x1'], + 'bottom': word['bottom'] + } + + current_block['type'] = self._classify_block(current_block, page_height) + blocks.append(current_block) + return blocks + + def _tesseract_data_to_blocks(self, data: Dict, page_width: float, page_height: float) -> List[Dict]: + blocks = [] + n_boxes = len(data['text']) + current_line = [] + + for i in range(n_boxes): + if int(data['conf'][i]) > 10 and data['text'][i].strip(): + # We can group by line_num + blocks.append({ + 'text': data['text'][i], + 'x': data['left'][i], + 'y': data['top'][i], + 'w': data['width'][i], + 'h': data['height'][i], + 'conf': float(data['conf'][i]), + 'type': self._classify_block({'top': data['top'][i], 'text': data['text'][i]}, page_height) + }) + return blocks + + def _classify_block(self, block: Dict, page_height: float) -> str: + y = block.get('top') or block.get('y') or 0 + text = block.get('text', '').lower() + + if y < page_height * 0.15: + return "HEADER" + elif y > page_height * 0.85: + return "FOOTER" + + if "total" in text: + return "TOTAL" + if "tax" in text or "gst" in text or "vat" in text: + return "TAX" + if re.search(r'\b(vendor|from)\b', text): + return "VENDOR" + if re.search(r'\b(bill to|sold to)\b', text): + return "BILL_TO" + if re.search(r'\b(ship to)\b', text): + return "SHIP_TO" + + # Very naive fallback for table check + if re.search(r'\b(qty|rate|amount|price|item)\b', text): + return "TABLE_HEADER" + + return "TEXT" diff --git a/backend/engine/recognition/__pycache__/template_engine.cpython-313.pyc b/backend/engine/recognition/__pycache__/template_engine.cpython-313.pyc new file mode 100644 index 0000000..dc9b72b Binary files /dev/null and b/backend/engine/recognition/__pycache__/template_engine.cpython-313.pyc differ diff --git a/backend/engine/recognition/template_engine.py b/backend/engine/recognition/template_engine.py new file mode 100644 index 0000000..4c482ab --- /dev/null +++ b/backend/engine/recognition/template_engine.py @@ -0,0 +1,174 @@ +import hashlib +import json +from typing import List, Dict, Any, Tuple +from sqlalchemy.orm import Session +from models.template_models import Template, TemplateDocument, DocumentLayout, TemplateRecognitionHistory +from schemas.template_schemas import TemplateRecognitionResult + +class TemplateRecognitionEngine: + def __init__(self, db: Session): + self.db = db + + def generate_fingerprint(self, layouts: List[DocumentLayout]) -> str: + """ + Generate a structural fingerprint based on the document's layout. + We use relative positions of HEADER, VENDOR, TABLE_HEADER blocks. + """ + key_blocks = [] + for layout in layouts: + if layout.block_type in ["HEADER", "VENDOR", "TABLE_HEADER", "TOTAL"]: + key_blocks.append({ + "type": layout.block_type, + "text": layout.text_value[:50] if layout.text_value else "", # first 50 chars + "rx": round(float(layout.x_coordinate) / 100) if layout.x_coordinate else 0, # relative bucket x + "ry": round(float(layout.y_coordinate) / 100) if layout.y_coordinate else 0 # relative bucket y + }) + + # Sort top to bottom, left to right + key_blocks.sort(key=lambda b: (b['ry'], b['rx'])) + + fingerprint_data = json.dumps(key_blocks) + return hashlib.sha256(fingerprint_data.encode('utf-8')).hexdigest() + + def recognize_template(self, document_id: int) -> TemplateRecognitionResult: + """ + Matches a document against existing templates using a 4-level weighted score: + Level 1: Vendor Match (40%) + Level 2: Header Similarity (20%) + Level 3: Layout/Fingerprint Similarity (20%) + Level 4: Coordinate Similarity (20%) + Threshold: 85% + """ + layouts = self.db.query(DocumentLayout).filter(DocumentLayout.fk_document_id == document_id).all() + if not layouts: + return TemplateRecognitionResult(templateMatched=False) + + doc_fingerprint = self.generate_fingerprint(layouts) + + # Extract features for scoring + doc_vendor = self._extract_vendor_name(layouts) + doc_headers = self._extract_headers(layouts) + + templates = self.db.query(Template).filter(Template.active_flag == True).all() + best_match = None + highest_score = 0 + + for template in templates: + score = 0.0 + + # Level 1: Fingerprint Exact Match (counts for Layout + Vendor + Header if exact) + if template.template_fingerprint == doc_fingerprint: + score += 100.0 + else: + # Need to load template sample mapping to compare heuristics + # In a real system, we'd compare against the mapped fields' coordinates + score += self._calculate_heuristic_score(template, doc_vendor, doc_headers, layouts) + + if score > highest_score: + highest_score = score + best_match = template + + # Store history + if best_match: + history = TemplateRecognitionHistory( + fk_template_id=best_match.pk_template_id, + fk_document_id=document_id, + recognition_score=highest_score, + matched_flag=(highest_score >= 85.0) + ) + self.db.add(history) + self.db.commit() + + if best_match and highest_score >= 85.0: + extracted_fields = self._auto_map_fields(best_match, layouts) + return TemplateRecognitionResult( + templateMatched=True, + templateId=best_match.pk_template_id, + confidence=highest_score, + extractedFields=extracted_fields + ) + + return TemplateRecognitionResult(templateMatched=False) + + def _extract_vendor_name(self, layouts: List[DocumentLayout]) -> str: + for layout in layouts: + if layout.block_type == "VENDOR": + return layout.text_value.lower() + return "" + + def _extract_headers(self, layouts: List[DocumentLayout]) -> List[str]: + headers = [] + for layout in layouts: + if layout.block_type == "TABLE_HEADER" and layout.text_value: + headers.append(layout.text_value.lower()) + return headers + + def _calculate_heuristic_score(self, template: Template, doc_vendor: str, doc_headers: List[str], layouts: List[DocumentLayout]) -> float: + score = 0.0 + + # This requires the template to have some stored metadata or we check its fields + # E.g., if template has a field "Vendor Name" mapped to a specific text + + # 1. Vendor Match (40%) + # For this mockup, we check if the template name matches the vendor + if template.template_name.lower() in doc_vendor or doc_vendor in template.template_name.lower(): + score += 40.0 + + # 2. Header Similarity (20%) + # Check if template fields exist that match doc_headers + field_labels = [f.field_label.lower() for f in template.fields] + header_matches = sum(1 for h in doc_headers if any(h in fl or fl in h for fl in field_labels)) + if len(doc_headers) > 0: + score += (header_matches / len(doc_headers)) * 20.0 + + # 3. Layout / Coordinate Similarity + # We can check the template's previous mappings coordinates against current document + # If coordinates are within a tolerance, we add score + mapping_count = 0 + match_count = 0 + for mapping in template.mappings: + mapping_count += 1 + # Find a layout block in current document that is near mapping coordinates + for layout in layouts: + if layout.page_no == mapping.page_no: + # check distance + if mapping.x_coordinate and mapping.y_coordinate and layout.x_coordinate and layout.y_coordinate: + dx = abs(float(mapping.x_coordinate) - float(layout.x_coordinate)) + dy = abs(float(mapping.y_coordinate) - float(layout.y_coordinate)) + if dx < 50 and dy < 20: # arbitrary tolerance + match_count += 1 + break + + if mapping_count > 0: + coord_score = (match_count / mapping_count) * 40.0 # Layout 20% + Coord 20% + score += coord_score + + return score + + def _auto_map_fields(self, template: Template, layouts: List[DocumentLayout]) -> List[dict]: + extracted = [] + for mapping in template.mappings: + # Find closest block in new document + best_block = None + min_dist = float('inf') + + for layout in layouts: + if layout.page_no == mapping.page_no: + if mapping.x_coordinate and mapping.y_coordinate and layout.x_coordinate and layout.y_coordinate: + dx = abs(float(mapping.x_coordinate) - float(layout.x_coordinate)) + dy = abs(float(mapping.y_coordinate) - float(layout.y_coordinate)) + dist = dx**2 + dy**2 + if dist < min_dist and dist < 5000: # tolerance squared + min_dist = dist + best_block = layout + + if best_block: + extracted.append({ + "field_id": mapping.fk_template_field_id, + "field_label": mapping.field.field_label if mapping.field else "", + "value": best_block.text_value, + "confidence": 95.0, # arbitrary high confidence for coordinate match + "layout_id": best_block.pk_document_data_id + }) + + return extracted diff --git a/backend/llm_service.py b/backend/llm_service.py new file mode 100644 index 0000000..9112588 --- /dev/null +++ b/backend/llm_service.py @@ -0,0 +1,116 @@ +import ollama +import json +import base64 + +INVOICE_SCHEMA = { + "document_type": None, + "invoice_number": None, + "invoice_date": None, + "due_date": None, + "purchase_order_number": None, + "vendor": { + "name": None, + "address": None, + "email": None, + "phone": None, + "gstin": None, + "tax_id": None, + "website": None + }, + "customer": { + "name": None, + "address": None, + "gstin": None + }, + "amounts": { + "subtotal": None, + "tax": None, + "discount": None, + "shipping": None, + "round_off": None, + "total": None, + "amount_paid": None, + "balance_due": None, + "currency": None + }, + "tax_breakdown": [ + { + "type": None, + "rate": None, + "amount": None + } + ], + "line_items": [ + { + "line_no": None, + "description": None, + "product_code": None, + "hsn_sac": None, + "quantity": None, + "unit": None, + "unit_price": None, + "discount": None, + "tax_rate": None, + "tax_amount": None, + "total": None + } + ], + "payment_information": { + "bank_name": None, + "account_number": None, + "ifsc": None, + "upi_id": None + }, + "metadata": { + "pages": None, + "ocr_confidence": None, + "language": None + } +} + +def extract_data(text: str = None, image_path: str = None, model_type: str = "text") -> dict: + """ + Extracts structured data using either Text (Gemma) or Vision (Qwen) models. + """ + + prompt = f""" + You are an expert data extraction assistant. + Extract every possible detail from the provided document and return it strictly as a SINGLE VALID JSON OBJECT matching the following schema structure: + + {json.dumps(INVOICE_SCHEMA, indent=4)} + + IMPORTANT: + - Return ONLY the JSON. No markdown formatting, no explanations, no prefix. + - If a field is not found or data is not available, use null. + """ + + messages = [{'role': 'user', 'content': prompt}] + model = 'gemma:2b' + + if model_type == 'vision': + if not image_path: + return {"error": "Image path required for vision mode"} + + # Qwen-VL handles images passed in the message + model = 'qwen2.5vl:7b' # Using the installed model ID + messages[0]['images'] = [image_path] + messages[0]['content'] = "Analyze this image. " + prompt + else: + # Text Mode + if not text: + return {"error": "Text required for text mode"} + messages[0]['content'] += f"\n\n---\n{text}\n---" + + try: + response = ollama.chat(model=model, messages=messages) + content = response['message']['content'] + + # Clean up markdown + content = content.replace("```json", "").replace("```", "").strip() + + return json.loads(content) + + except Exception as e: + print(f"LLM Extraction Error ({model_type}): {e}") + return {"error": str(e), "raw_output": content if 'content' in locals() else ""} + diff --git a/backend/main.py b/backend/main.py index 7bd917a..1789f21 100644 --- a/backend/main.py +++ b/backend/main.py @@ -44,10 +44,14 @@ def extract_text_from_pdf(file_bytes: bytes) -> str: return "" # Internal modules -from database import get_db, Email +from database import get_db, Email, Vendor, Document from scheduler import start_scheduler, stop_scheduler from mail_service import fetch_and_store_emails +# Import the new routers +from api.templates import router as templates_router +from api.documents import router as documents_router + # Lifespan for Scheduler @asynccontextmanager async def lifespan(app: FastAPI): @@ -59,6 +63,10 @@ async def lifespan(app: FastAPI): app = FastAPI(lifespan=lifespan) +# Include Routers +app.include_router(templates_router) +app.include_router(documents_router) + # CORS configuration origins = [ "http://localhost", @@ -87,7 +95,7 @@ class LoginResponse(BaseModel): class NERResponse(BaseModel): text: str - + file_path: str def extract_text_from_image(file_bytes: bytes) -> str: @@ -104,13 +112,41 @@ async def extract_text(file: UploadFile = File(...)): content = await file.read() filename = file.filename.lower() + # Save file for Vision mode + file_path = f"uploads/{file.filename}" + with open(file_path, "wb") as f: + f.write(content) + extracted_text = "" if filename.endswith(".pdf"): # Try text extraction first - extracted_text = extract_text_from_pdf(content) - - # If text is empty, it might be a scanned PDF. + with pdfplumber.open(io.BytesIO(content)) as pdf: + try: + text = "" + for page in pdf.pages: + page_text = page.extract_text(layout=True) + if page_text: + text += page_text + "\n" + if text.strip(): + extracted_text = text.strip() + except Exception: + pass + + if not extracted_text: + try: + # Fallback to pypdf + reader = PdfReader(io.BytesIO(content)) + text = "" + for page in reader.pages: + page_text = page.extract_text() + if page_text: + text += page_text + "\n" + extracted_text = text.strip() + except: + pass + + # If text is still empty, it might be a scanned PDF. if not extracted_text.strip(): try: images = convert_from_bytes(content) @@ -125,7 +161,35 @@ async def extract_text(file: UploadFile = File(...)): else: raise HTTPException(status_code=400, detail="Unsupported file type") - return NERResponse(text=extracted_text) + return NERResponse(text=extracted_text, file_path=file_path) + +# 3. AI Extraction Module +from llm_service import extract_data +from pdf2image import convert_from_path + +class AITextRequest(BaseModel): + text: Optional[str] = None + file_path: Optional[str] = None + model_type: str = "text" + +@app.post("/api/extract/ai") +def extract_ai_data(request: AITextRequest): + final_image_path = request.file_path + + if request.model_type == "vision" and request.file_path and request.file_path.endswith(".pdf"): + # Convert PDF first page to image + try: + images = convert_from_path(request.file_path) + if images: + # Save temp image + temp_img_path = request.file_path + ".jpg" + images[0].save(temp_img_path, "JPEG") + final_image_path = temp_img_path + except Exception as e: + print(f"Error converting PDF for vision: {e}") + + data = extract_data(text=request.text, image_path=final_image_path, model_type=request.model_type) + return data import zipfile import mimetypes @@ -255,3 +319,36 @@ def sync_emails(): @app.get("/") def read_root(): return {"message": "OCR Backend API is running"} + +class DocumentSaveRequest(BaseModel): + vendor_name: str + file_path: str + model_type: str + data: dict + +@app.post("/api/documents/save") +def save_document(request: DocumentSaveRequest, db: Session = Depends(get_db)): + # 1. Find or Create Vendor + vendor = db.query(Vendor).filter(Vendor.name == request.vendor_name).first() + if not vendor: + vendor = Vendor(name=request.vendor_name, default_model=request.model_type) + db.add(vendor) + db.commit() + db.refresh(vendor) + else: + # Update preference + vendor.default_model = request.model_type + db.commit() + + # 2. Save Document + filename = request.file_path.split('/')[-1] + doc = Document( + vendor_id=vendor.id, + filename=filename, + status="verified", + processed_data=request.data + ) + db.add(doc) + db.commit() + + return {"message": "Document saved and Vendor preference updated", "vendor_id": vendor.id} diff --git a/backend/models/__pycache__/template_models.cpython-313.pyc b/backend/models/__pycache__/template_models.cpython-313.pyc new file mode 100644 index 0000000..b92120c Binary files /dev/null and b/backend/models/__pycache__/template_models.cpython-313.pyc differ diff --git a/backend/models/template_models.py b/backend/models/template_models.py new file mode 100644 index 0000000..b924681 --- /dev/null +++ b/backend/models/template_models.py @@ -0,0 +1,105 @@ +from sqlalchemy import Column, Integer, String, Text, DateTime, Boolean, ForeignKey, Numeric +from sqlalchemy.sql import func +from sqlalchemy.orm import relationship +from database import Base + +class TemplateDocument(Base): + __tablename__ = "documents" + __table_args__ = {"schema": "templates"} + + pk_document_id = Column(Integer, primary_key=True, index=True) + document_name = Column(String(500)) + document_type = Column(String(100)) + file_name = Column(String(500)) + file_hash = Column(String(500), index=True) + page_count = Column(Integer) + status = Column(String(50)) + created_at = Column(DateTime, server_default=func.now()) + updated_at = Column(DateTime, server_default=func.now(), onupdate=func.now()) + + # Relationships + layouts = relationship("DocumentLayout", back_populates="document", cascade="all, delete") + +class DocumentLayout(Base): + __tablename__ = "document_layout" + __table_args__ = {"schema": "templates"} + + pk_document_data_id = Column(Integer, primary_key=True, index=True) + fk_document_id = Column(Integer, ForeignKey("templates.documents.pk_document_id", ondelete="CASCADE"), nullable=False, index=True) + page_no = Column(Integer, nullable=False, index=True) + text_value = Column(Text) + block_type = Column(String(100)) + parent_block_id = Column(Integer, ForeignKey("templates.document_layout.pk_document_data_id", ondelete="SET NULL")) + x_coordinate = Column(Numeric) + y_coordinate = Column(Numeric) + width = Column(Numeric) + height = Column(Numeric) + confidence = Column(Numeric) + sequence_no = Column(Integer) + layout_path = Column(Text) + created_at = Column(DateTime, server_default=func.now()) + + document = relationship("TemplateDocument", back_populates="layouts") + parent = relationship("DocumentLayout", remote_side=[pk_document_data_id]) + mappings = relationship("TemplateFieldMapping", back_populates="document_layout", cascade="all, delete") + +class Template(Base): + __tablename__ = "templates" + __table_args__ = {"schema": "templates"} + + pk_template_id = Column(Integer, primary_key=True, index=True) + template_name = Column(String(255), nullable=False, index=True) + template_fingerprint = Column(Text) + active_flag = Column(Boolean, default=True) + created_at = Column(DateTime, server_default=func.now()) + updated_at = Column(DateTime, server_default=func.now(), onupdate=func.now()) + + fields = relationship("TemplateField", back_populates="template", cascade="all, delete") + mappings = relationship("TemplateFieldMapping", back_populates="template", cascade="all, delete") + +class TemplateField(Base): + __tablename__ = "template_fields" + __table_args__ = {"schema": "templates"} + + pk_template_field_id = Column(Integer, primary_key=True, index=True) + fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False, index=True) + field_label = Column(String(255), nullable=False) + field_type = Column(String(100), nullable=False) + display_order = Column(Integer) + required_flag = Column(Boolean, default=False) + created_at = Column(DateTime, server_default=func.now()) + + template = relationship("Template", back_populates="fields") + mappings = relationship("TemplateFieldMapping", back_populates="field", cascade="all, delete") + +class TemplateFieldMapping(Base): + __tablename__ = "template_fields_mapping" + __table_args__ = {"schema": "templates"} + + pk_mapping_id = Column(Integer, primary_key=True, index=True) + fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False, index=True) + fk_template_field_id = Column(Integer, ForeignKey("templates.template_fields.pk_template_field_id", ondelete="CASCADE"), nullable=False, index=True) + fk_document_data_id = Column(Integer, ForeignKey("templates.document_layout.pk_document_data_id", ondelete="SET NULL")) + page_no = Column(Integer) + x_coordinate = Column(Numeric) + y_coordinate = Column(Numeric) + width = Column(Numeric) + height = Column(Numeric) + mapping_confidence = Column(Numeric) + layout_path = Column(Text) + created_at = Column(DateTime, server_default=func.now()) + + template = relationship("Template", back_populates="mappings") + field = relationship("TemplateField", back_populates="mappings") + document_layout = relationship("DocumentLayout", back_populates="mappings") + +class TemplateRecognitionHistory(Base): + __tablename__ = "template_recognition_history" + __table_args__ = {"schema": "templates"} + + pk_history_id = Column(Integer, primary_key=True, index=True) + fk_template_id = Column(Integer, ForeignKey("templates.templates.pk_template_id", ondelete="CASCADE"), nullable=False) + fk_document_id = Column(Integer, ForeignKey("templates.documents.pk_document_id", ondelete="CASCADE"), nullable=False, index=True) + recognition_score = Column(Numeric) + matched_flag = Column(Boolean) + created_at = Column(DateTime, server_default=func.now()) diff --git a/backend/repositories/__pycache__/template_repository.cpython-313.pyc b/backend/repositories/__pycache__/template_repository.cpython-313.pyc new file mode 100644 index 0000000..fcb3482 Binary files /dev/null and b/backend/repositories/__pycache__/template_repository.cpython-313.pyc differ diff --git a/backend/repositories/template_repository.py b/backend/repositories/template_repository.py new file mode 100644 index 0000000..0f8dd35 --- /dev/null +++ b/backend/repositories/template_repository.py @@ -0,0 +1,75 @@ +from sqlalchemy.orm import Session +from sqlalchemy import desc +from models.template_models import Template, TemplateField, TemplateFieldMapping, TemplateRecognitionHistory +from schemas.template_schemas import TemplateCreate, TemplateFieldCreate, TemplateFieldMappingCreate + +class TemplateRepository: + def __init__(self, db: Session): + self.db = db + + def get_template(self, template_id: int): + return self.db.query(Template).filter(Template.pk_template_id == template_id).first() + + def get_templates(self, skip: int = 0, limit: int = 100): + return self.db.query(Template).offset(skip).limit(limit).all() + + def create_template(self, template_data: TemplateCreate): + db_template = Template(template_name=template_data.template_name) + self.db.add(db_template) + self.db.commit() + self.db.refresh(db_template) + + # Create fields if provided + for field in template_data.fields: + db_field = TemplateField( + fk_template_id=db_template.pk_template_id, + field_label=field.field_label, + field_type=field.field_type, + display_order=field.display_order, + required_flag=field.required_flag + ) + self.db.add(db_field) + + self.db.commit() + self.db.refresh(db_template) + return db_template + + def add_template_field(self, template_id: int, field_data: TemplateFieldCreate): + db_field = TemplateField( + fk_template_id=template_id, + field_label=field_data.field_label, + field_type=field_data.field_type, + display_order=field_data.display_order, + required_flag=field_data.required_flag + ) + self.db.add(db_field) + self.db.commit() + self.db.refresh(db_field) + return db_field + + def delete_template(self, template_id: int): + template = self.get_template(template_id) + if template: + self.db.delete(template) + self.db.commit() + return True + return False + + def save_mapping(self, template_id: int, mappings: list[TemplateFieldMappingCreate]): + saved_mappings = [] + for mapping in mappings: + db_mapping = TemplateFieldMapping( + fk_template_id=template_id, + fk_template_field_id=mapping.fk_template_field_id, + fk_document_data_id=mapping.fk_document_data_id, + page_no=mapping.page_no, + x_coordinate=mapping.x_coordinate, + y_coordinate=mapping.y_coordinate, + width=mapping.width, + height=mapping.height, + mapping_confidence=mapping.mapping_confidence + ) + self.db.add(db_mapping) + saved_mappings.append(db_mapping) + self.db.commit() + return saved_mappings diff --git a/backend/requirements.txt b/backend/requirements.txt index 8c7160e..fbf9d2c 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -10,3 +10,8 @@ imap-tools apscheduler python-dotenv pdfplumber +ollama +numpy +opencv-python-headless +pdf2image + diff --git a/backend/schemas/__pycache__/template_schemas.cpython-313.pyc b/backend/schemas/__pycache__/template_schemas.cpython-313.pyc new file mode 100644 index 0000000..6448164 Binary files /dev/null and b/backend/schemas/__pycache__/template_schemas.cpython-313.pyc differ diff --git a/backend/schemas/template_schemas.py b/backend/schemas/template_schemas.py new file mode 100644 index 0000000..176816f --- /dev/null +++ b/backend/schemas/template_schemas.py @@ -0,0 +1,80 @@ +from pydantic import BaseModel, ConfigDict +from typing import List, Optional +from datetime import datetime + +# Common +class BaseSchema(BaseModel): + model_config = ConfigDict(from_attributes=True) + +# Document +class DocumentLayoutSchema(BaseSchema): + pk_document_data_id: int + fk_document_id: int + page_no: int + text_value: Optional[str] + block_type: Optional[str] + parent_block_id: Optional[int] + x_coordinate: Optional[float] + y_coordinate: Optional[float] + width: Optional[float] + height: Optional[float] + confidence: Optional[float] + sequence_no: Optional[int] + +class DocumentSchema(BaseSchema): + pk_document_id: int + document_name: Optional[str] + document_type: Optional[str] + file_name: Optional[str] + page_count: Optional[int] + status: Optional[str] + created_at: datetime + layouts: List[DocumentLayoutSchema] = [] + +# Template Field +class TemplateFieldCreate(BaseSchema): + field_label: str + field_type: str + display_order: Optional[int] = 0 + required_flag: Optional[bool] = False + +class TemplateFieldSchema(TemplateFieldCreate): + pk_template_field_id: int + fk_template_id: int + created_at: datetime + +# Template Field Mapping +class TemplateFieldMappingCreate(BaseSchema): + fk_template_field_id: int + fk_document_data_id: Optional[int] = None + page_no: Optional[int] = None + x_coordinate: Optional[float] = None + y_coordinate: Optional[float] = None + width: Optional[float] = None + height: Optional[float] = None + mapping_confidence: Optional[float] = None + +class TemplateFieldMappingSchema(TemplateFieldMappingCreate): + pk_mapping_id: int + fk_template_id: int + created_at: datetime + +# Template +class TemplateCreate(BaseSchema): + template_name: str + fields: Optional[List[TemplateFieldCreate]] = [] + +class TemplateSchema(BaseSchema): + pk_template_id: int + template_name: str + template_fingerprint: Optional[str] = None + active_flag: bool + created_at: datetime + fields: List[TemplateFieldSchema] = [] + mappings: List[TemplateFieldMappingSchema] = [] + +class TemplateRecognitionResult(BaseSchema): + templateMatched: bool + templateId: Optional[int] = None + confidence: Optional[float] = None + extractedFields: List[dict] = [] diff --git a/backend/services/__pycache__/template_service.cpython-313.pyc b/backend/services/__pycache__/template_service.cpython-313.pyc new file mode 100644 index 0000000..2879896 Binary files /dev/null and b/backend/services/__pycache__/template_service.cpython-313.pyc differ diff --git a/backend/services/template_service.py b/backend/services/template_service.py new file mode 100644 index 0000000..809aa86 --- /dev/null +++ b/backend/services/template_service.py @@ -0,0 +1,25 @@ +from sqlalchemy.orm import Session +from schemas.template_schemas import TemplateCreate, TemplateFieldCreate, TemplateFieldMappingCreate +from repositories.template_repository import TemplateRepository + +class TemplateService: + def __init__(self, db: Session): + self.repository = TemplateRepository(db) + + def get_template(self, template_id: int): + return self.repository.get_template(template_id) + + def get_all_templates(self, skip: int = 0, limit: int = 100): + return self.repository.get_templates(skip, limit) + + def create_template(self, template_data: TemplateCreate): + return self.repository.create_template(template_data) + + def add_template_field(self, template_id: int, field_data: TemplateFieldCreate): + return self.repository.add_template_field(template_id, field_data) + + def delete_template(self, template_id: int): + return self.repository.delete_template(template_id) + + def save_mapping(self, template_id: int, mappings: list[TemplateFieldMappingCreate]): + return self.repository.save_mapping(template_id, mappings) diff --git a/backend/sql/01_templates_schema.sql b/backend/sql/01_templates_schema.sql new file mode 100644 index 0000000..f17313d --- /dev/null +++ b/backend/sql/01_templates_schema.sql @@ -0,0 +1,107 @@ +-- Schema for Template Engine +CREATE SCHEMA IF NOT EXISTS templates; + +-- 1. documents table +CREATE TABLE IF NOT EXISTS templates.documents ( + pk_document_id BIGSERIAL PRIMARY KEY, + document_name VARCHAR(500), + document_type VARCHAR(100), + file_name VARCHAR(500), + file_hash VARCHAR(500), + page_count INTEGER, + status VARCHAR(50), + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +-- Index for quick lookups +CREATE INDEX idx_documents_hash ON templates.documents(file_hash); + +-- 2. document_layout table +CREATE TABLE IF NOT EXISTS templates.document_layout ( + pk_document_data_id BIGSERIAL PRIMARY KEY, + fk_document_id BIGINT NOT NULL, + page_no INTEGER NOT NULL, + text_value TEXT, + block_type VARCHAR(100), + parent_block_id BIGINT, + x_coordinate NUMERIC, + y_coordinate NUMERIC, + width NUMERIC, + height NUMERIC, + confidence NUMERIC, + sequence_no INTEGER, + layout_path TEXT, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT fk_dl_document FOREIGN KEY (fk_document_id) REFERENCES templates.documents(pk_document_id) ON DELETE CASCADE, + CONSTRAINT fk_dl_parent FOREIGN KEY (parent_block_id) REFERENCES templates.document_layout(pk_document_data_id) ON DELETE SET NULL +); + +CREATE INDEX idx_doc_layout_doc_id ON templates.document_layout(fk_document_id); +CREATE INDEX idx_doc_layout_page ON templates.document_layout(page_no); + +-- 3. templates table +CREATE TABLE IF NOT EXISTS templates.templates ( + pk_template_id BIGSERIAL PRIMARY KEY, + template_name VARCHAR(255) NOT NULL, + template_fingerprint TEXT, + active_flag BOOLEAN DEFAULT TRUE, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_templates_name ON templates.templates(template_name); + +-- 4. template_fields table +CREATE TABLE IF NOT EXISTS templates.template_fields ( + pk_template_field_id BIGSERIAL PRIMARY KEY, + fk_template_id BIGINT NOT NULL, + field_label VARCHAR(255) NOT NULL, + field_type VARCHAR(100) NOT NULL, + display_order INTEGER, + required_flag BOOLEAN DEFAULT FALSE, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT fk_tf_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE +); + +CREATE INDEX idx_template_fields_tmpl_id ON templates.template_fields(fk_template_id); + +-- 5. template_fields_mapping table +CREATE TABLE IF NOT EXISTS templates.template_fields_mapping ( + pk_mapping_id BIGSERIAL PRIMARY KEY, + fk_template_id BIGINT NOT NULL, + fk_template_field_id BIGINT NOT NULL, + fk_document_data_id BIGINT, + page_no INTEGER, + x_coordinate NUMERIC, + y_coordinate NUMERIC, + width NUMERIC, + height NUMERIC, + mapping_confidence NUMERIC, + layout_path TEXT, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT fk_tfm_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE, + CONSTRAINT fk_tfm_field FOREIGN KEY (fk_template_field_id) REFERENCES templates.template_fields(pk_template_field_id) ON DELETE CASCADE, + CONSTRAINT fk_tfm_doc_data FOREIGN KEY (fk_document_data_id) REFERENCES templates.document_layout(pk_document_data_id) ON DELETE SET NULL +); + +CREATE INDEX idx_mapping_tmpl_id ON templates.template_fields_mapping(fk_template_id); +CREATE INDEX idx_mapping_field_id ON templates.template_fields_mapping(fk_template_field_id); + +-- 6. template_recognition_history table +CREATE TABLE IF NOT EXISTS templates.template_recognition_history ( + pk_history_id BIGSERIAL PRIMARY KEY, + fk_template_id BIGINT NOT NULL, + fk_document_id BIGINT NOT NULL, + recognition_score NUMERIC, + matched_flag BOOLEAN, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT fk_trh_template FOREIGN KEY (fk_template_id) REFERENCES templates.templates(pk_template_id) ON DELETE CASCADE, + CONSTRAINT fk_trh_document FOREIGN KEY (fk_document_id) REFERENCES templates.documents(pk_document_id) ON DELETE CASCADE +); + +CREATE INDEX idx_recognition_history_doc ON templates.template_recognition_history(fk_document_id); diff --git a/backend/uploads/Invoice For Jan-Feb-2026.pdf b/backend/uploads/Invoice For Jan-Feb-2026.pdf new file mode 100644 index 0000000..6c73b97 Binary files /dev/null and b/backend/uploads/Invoice For Jan-Feb-2026.pdf differ diff --git a/backend/uploads/Invoice For Jan-Feb-2026.pdf.jpg b/backend/uploads/Invoice For Jan-Feb-2026.pdf.jpg new file mode 100644 index 0000000..a1b8862 Binary files /dev/null and b/backend/uploads/Invoice For Jan-Feb-2026.pdf.jpg differ diff --git a/backend/uploads/Invoice For Mar 2026.pdf b/backend/uploads/Invoice For Mar 2026.pdf new file mode 100644 index 0000000..65e0311 Binary files /dev/null and b/backend/uploads/Invoice For Mar 2026.pdf differ diff --git a/backend/uploads/Invoice For Mar 2026.pdf.jpg b/backend/uploads/Invoice For Mar 2026.pdf.jpg new file mode 100644 index 0000000..1bffb55 Binary files /dev/null and b/backend/uploads/Invoice For Mar 2026.pdf.jpg differ diff --git a/backend/uploads/Invoice For Oct-Nov-2025.pdf b/backend/uploads/Invoice For Oct-Nov-2025.pdf new file mode 100644 index 0000000..66410b4 Binary files /dev/null and b/backend/uploads/Invoice For Oct-Nov-2025.pdf differ diff --git a/backend/uploads/Invoice For Oct-Nov-2025.pdf.jpg b/backend/uploads/Invoice For Oct-Nov-2025.pdf.jpg new file mode 100644 index 0000000..8c8a1c2 Binary files /dev/null and b/backend/uploads/Invoice For Oct-Nov-2025.pdf.jpg differ diff --git a/backend/uploads/Purchase-Order-Template-01-TemplateLab.pdf b/backend/uploads/Purchase-Order-Template-01-TemplateLab.pdf new file mode 100644 index 0000000..fa171b4 Binary files /dev/null and b/backend/uploads/Purchase-Order-Template-01-TemplateLab.pdf differ diff --git a/backend/uploads/invoice-stripes.png b/backend/uploads/invoice-stripes.png new file mode 100644 index 0000000..75aa26a Binary files /dev/null and b/backend/uploads/invoice-stripes.png differ diff --git a/backend/uploads/invoice_Aaron Bergman_36258.pdf b/backend/uploads/invoice_Aaron Bergman_36258.pdf new file mode 100644 index 0000000..ce93b94 Binary files /dev/null and b/backend/uploads/invoice_Aaron Bergman_36258.pdf differ diff --git a/backend/uploads/invoice_Aaron Hawkins_40101.pdf b/backend/uploads/invoice_Aaron Hawkins_40101.pdf new file mode 100644 index 0000000..dd27a2f Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_40101.pdf differ diff --git a/backend/uploads/invoice_Aaron Hawkins_40101.pdf.jpg b/backend/uploads/invoice_Aaron Hawkins_40101.pdf.jpg new file mode 100644 index 0000000..6881d56 Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_40101.pdf.jpg differ diff --git a/backend/uploads/invoice_Aaron Hawkins_4820.pdf b/backend/uploads/invoice_Aaron Hawkins_4820.pdf new file mode 100644 index 0000000..a51acef Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_4820.pdf differ diff --git a/backend/uploads/invoice_Aaron Hawkins_4820.pdf.jpg b/backend/uploads/invoice_Aaron Hawkins_4820.pdf.jpg new file mode 100644 index 0000000..404c907 Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_4820.pdf.jpg differ diff --git a/backend/uploads/invoice_Aaron Hawkins_6817.pdf b/backend/uploads/invoice_Aaron Hawkins_6817.pdf new file mode 100644 index 0000000..87d5aee Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_6817.pdf differ diff --git a/backend/uploads/invoice_Aaron Hawkins_6817.pdf.jpg b/backend/uploads/invoice_Aaron Hawkins_6817.pdf.jpg new file mode 100644 index 0000000..85efa37 Binary files /dev/null and b/backend/uploads/invoice_Aaron Hawkins_6817.pdf.jpg differ diff --git a/backend/uploads/sample-pdf-invoice.pdf b/backend/uploads/sample-pdf-invoice.pdf new file mode 100644 index 0000000..907ad9f Binary files /dev/null and b/backend/uploads/sample-pdf-invoice.pdf differ diff --git a/backend/uploads/sample-pdf-invoice.pdf.jpg b/backend/uploads/sample-pdf-invoice.pdf.jpg new file mode 100644 index 0000000..ea280cc Binary files /dev/null and b/backend/uploads/sample-pdf-invoice.pdf.jpg differ diff --git a/check_matches.py b/check_matches.py new file mode 100644 index 0000000..d25458a --- /dev/null +++ b/check_matches.py @@ -0,0 +1,16 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.document import TemplateMatch +from app.models.template import DocumentFormat + +db = SessionLocal() +matches = db.query(TemplateMatch).order_by(TemplateMatch.created_at.desc()).limit(10).all() +print(f"Found {len(matches)} matches.") +for m in matches: + fmt = db.query(DocumentFormat).filter(DocumentFormat.id == m.format_id).first() + fmt_name = fmt.name if fmt else 'Unknown' + print(f"Match: doc_id={m.document_id}, format_id={m.format_id}, name={fmt_name}, score={m.confidence_score}") + print(f"Details: {m.match_details}") diff --git a/debug_match.py b/debug_match.py new file mode 100644 index 0000000..b9391ae --- /dev/null +++ b/debug_match.py @@ -0,0 +1,26 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.document import Document +from app.services.matching_service import MatchingService + +db = SessionLocal() +doc = db.query(Document).order_by(Document.created_at.desc()).first() +if not doc: + print("No documents found.") + sys.exit(0) + +print(f"Latest document: {doc.id} (status: {doc.status})") + +svc = MatchingService(db) +try: + # Match with min_confidence=0.0 so it returns EVERYTHING + matches = svc.match_document(doc.id, min_confidence=0.0) + print(f"Returned {len(matches)} matches.") + for m in matches: + print(f"Match: format_id={m.format_id}, score={m.confidence_score}") + print(f"Details: {m.match_details}") +except Exception as e: + print(f"Error matching: {e}") diff --git a/docengine/.dockerignore b/docengine/.dockerignore new file mode 100644 index 0000000..f92b78a --- /dev/null +++ b/docengine/.dockerignore @@ -0,0 +1,26 @@ +__pycache__ +*.pyc +*.pyo +.Python +.env +.venv +env/ +venv/ +*.egg-info +dist/ +build/ +.git +.gitignore +.dockerignore +*.md +*.rst +docs/ +tests/ +htmlcov/ +.coverage +.pytest_cache +.mypy_cache +.ruff_cache +*.log +.idea/ +.vscode/ diff --git a/docengine/.env.example b/docengine/.env.example new file mode 100644 index 0000000..cd2a2fa --- /dev/null +++ b/docengine/.env.example @@ -0,0 +1,59 @@ +# Application +APP_NAME=DocEngine +APP_VERSION=1.0.0 +APP_ENV=development +APP_DEBUG=true +APP_HOST=0.0.0.0 +APP_PORT=7989 +APP_WORKERS=4 + +# Database +DB_HOST=localhost +DB_PORT=5432 +DB_NAME=ocr +DB_USER=postgres +DB_PASSWORD=changeme +DB_SCHEMA=admin +DB_POOL_SIZE=20 +DB_MAX_OVERFLOW=10 +DB_ECHO=false + +# Redis +REDIS_HOST=localhost +REDIS_PORT=6379 +REDIS_DB=0 +REDIS_PASSWORD= + +# Celery +CELERY_BROKER_URL=redis://localhost:6379/0 +CELERY_RESULT_BACKEND=redis://localhost:6379/1 + +# JWT +JWT_SECRET_KEY=change-this-to-a-secure-random-string +JWT_ALGORITHM=HS256 +JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30 +JWT_REFRESH_TOKEN_EXPIRE_DAYS=7 + +# Storage +STORAGE_PROVIDER=local +STORAGE_LOCAL_PATH=./storage +STORAGE_MAX_FILE_SIZE_MB=100 + +# OCR +OCR_LANGUAGE=en +OCR_USE_GPU=false + +# Logging +LOG_LEVEL=INFO +LOG_FORMAT=json + +# CORS +CORS_ORIGINS=["http://localhost:3000","http://localhost:8080","http://localhost:4200"] +CORS_ALLOW_CREDENTIALS=true + +# Rate Limiting +RATE_LIMIT_REQUESTS=100 +RATE_LIMIT_WINDOW_SECONDS=60 + +# Prometheus +PROMETHEUS_ENABLED=true diff --git a/docengine/.gitignore b/docengine/.gitignore new file mode 100644 index 0000000..9687533 --- /dev/null +++ b/docengine/.gitignore @@ -0,0 +1,64 @@ +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +*.manifest +*.spec +pip-log.txt +pip-delete-this-directory.txt +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +*.mo +*.pot +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal +instance/ +.webassets-cache +.scrapy +docs/_build/ +target/ +.venv +env/ +venv/ +ENV/ +.env +!.env.example +.idea/ +.vscode/ +*.swp +*.swo +*~ +.DS_Store +Thumbs.db +/storage/ +*.pid +celerybeat-schedule +celerybeat.pid diff --git a/docengine/Dockerfile b/docengine/Dockerfile new file mode 100644 index 0000000..3d98d3a --- /dev/null +++ b/docengine/Dockerfile @@ -0,0 +1,39 @@ +FROM python:3.12-slim AS base + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_NO_CACHE_DIR=1 \ + PIP_DISABLE_PIP_VERSION_CHECK=1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + libpq-dev \ + libgl1-mesa-glx \ + libglib2.0-0 \ + libsm6 \ + libxext6 \ + libxrender-dev \ + libgomp1 \ + poppler-utils \ + ghostscript \ + libmagic1 \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY . . + +RUN mkdir -p /app/storage/documents /app/storage/templates /app/storage/images /app/storage/temp + +FROM base AS app +EXPOSE 7989 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7989", "--workers", "4"] + +FROM base AS worker +CMD ["celery", "-A", "app.workers.celery_app", "worker", "--loglevel=info", "--concurrency=4"] + +FROM base AS beat +CMD ["celery", "-A", "app.workers.celery_app", "beat", "--loglevel=info"] diff --git a/docengine/README.md b/docengine/README.md new file mode 100644 index 0000000..373a614 --- /dev/null +++ b/docengine/README.md @@ -0,0 +1,522 @@ +# DocEngine — Document Template Recognition & Reconstruction System + +A production-ready system for scanning documents, detecting layouts, extracting content, generating reusable templates, matching future uploads against stored templates, and reconstructing original layouts as PDF output. + +## Architecture + +``` +┌───────────────┐ ┌───────────────┐ ┌──────────────┐ +│ FastAPI App │─────▶│ Celery │─────▶│ Redis │ +│ (Port 7989) │ │ Worker(s) │ │ (Broker) │ +└───────┬───────┘ └───────┬───────┘ └──────────────┘ + │ │ + ▼ ▼ +┌───────────────────────────────────────┐ +│ PostgreSQL (Schema: admin) │ +│ 192.168.0.111:5432 │ +└───────────────────────────────────────┘ +``` + +**Stack**: Python 3.12, FastAPI, SQLAlchemy 2.x, Pydantic V2, Celery, Redis, PaddleOCR, PyMuPDF, OpenCV, ReportLab, PostgreSQL 16. + +## Features + +| Capability | Implementation | +|----------------------------------|--------------------------------------------| +| Scanned image OCR | PaddleOCR (CPU/GPU) | +| Native PDF text extraction | PyMuPDF (fitz) | +| Layout detection | OpenCV + LayoutParser | +| Table extraction | Camelot-py + OpenCV contour detection | +| Header/footer detection | Positional heuristics | +| Watermark detection | Transparency + large-font analysis | +| Font info extraction | PyMuPDF text dict parsing | +| Template generation & storage | PostgreSQL (admin schema) | +| Template fingerprinting | SHA-256 structural hashing | +| Template matching | Multi-signal similarity scoring | +| PDF reconstruction | ReportLab from template definitions | +| Async processing | Celery + Redis | +| Authentication | JWT (access + refresh tokens, bcrypt) | +| Monitoring | Prometheus + structlog JSON logging | + +## Project Structure + +``` +docengine/ +├── app/ +│ ├── main.py # FastAPI application entry +│ ├── api/ +│ │ ├── router.py # Top-level API router +│ │ └── v1/ +│ │ ├── auth.py # Auth endpoints +│ │ ├── documents.py # Document endpoints +│ │ ├── health.py # Health check +│ │ └── templates.py # Template endpoints +│ ├── core/ +│ │ ├── config.py # Pydantic Settings +│ │ ├── database.py # SQLAlchemy engine & session +│ │ ├── dependencies.py # FastAPI DI +│ │ ├── exceptions.py # Custom exception hierarchy +│ │ ├── logging_config.py # structlog configuration +│ │ └── security.py # JWT & bcrypt helpers +│ ├── models/ # SQLAlchemy ORM models +│ ├── schemas/ # Pydantic request/response schemas +│ ├── repositories/ # Data access layer +│ ├── services/ # Business logic +│ │ ├── document_service.py # Orchestration pipeline +│ │ ├── ocr_service.py # PaddleOCR integration +│ │ ├── pdf_service.py # PyMuPDF native PDF parsing +│ │ ├── layout_service.py # OpenCV layout detection +│ │ ├── template_service.py # Template generation +│ │ ├── fingerprint_service.py +│ │ ├── matching_service.py +│ │ └── reconstruction_service.py +│ ├── middleware/ # CORS, audit, metrics, rate limit +│ ├── storage/ # File storage abstraction +│ ├── tasks/ # Celery async tasks +│ ├── workers/ # Celery app configuration +│ └── events/ # App lifecycle handlers +├── alembic/ # Database migrations +├── sql/ # Raw SQL scripts +├── tests/ # Test suite +├── docker-compose.yml # Dev stack +├── docker-compose.prod.yml # Production stack +├── Dockerfile # Multi-stage build +├── requirements.txt +└── .env +``` + +--- + +## Quick Start + +### Prerequisites + +- Python 3.12+ +- PostgreSQL 16 (running at `192.168.0.111:5432`) +- Redis (for Celery) +- `poppler-utils` and `ghostscript` (for pdf2image/camelot) + +### Local Setup + +```bash +# Clone & enter +cd docengine + +# Create virtual environment +python -m venv .venv +source .venv/bin/activate + +# Install dependencies +pip install -r requirements.txt + +# Create storage directories +mkdir -p storage/{documents,templates,images,temp,rendered} + +# Run database migrations +alembic upgrade head + +# (Optional) Seed default data +psql -h 192.168.0.111 -p 5432 -U postgres -d ocr -f sql/003_seed_data.sql +psql -h 192.168.0.111 -p 5432 -U postgres -d ocr -f sql/004_indexes.sql + +# Start the application +python -m app.main +``` + +The API is now available at `http://localhost:7989`. Interactive docs at `http://localhost:7989/docs`. + +### Start Celery Worker (separate terminal) + +```bash +source .venv/bin/activate +celery -A app.workers.celery_app worker --loglevel=info --concurrency=4 +``` + +### Docker Setup + +```bash +# Build and start all services (app + worker + db + redis) +docker compose up --build -d + +# Run migrations inside the container +docker compose exec app alembic upgrade head + +# Seed data +docker compose exec app bash -c "psql -h db -U postgres -d ocr -f sql/003_seed_data.sql" +``` + +--- + +## API Reference + +Base URL: `http://localhost:7989/api/v1` + +### Health + +```bash +curl http://localhost:7989/api/v1/health +``` + +### Authentication + +#### Register + +```bash +curl -X POST http://localhost:7989/api/v1/auth/register \ + -H "Content-Type: application/json" \ + -d '{ + "username": "john", + "email": "john@example.com", + "password": "SecurePass123!", + "full_name": "John Doe" + }' +``` + +#### Login + +```bash +curl -X POST http://localhost:7989/api/v1/auth/login \ + -H "Content-Type: application/json" \ + -d '{ + "username": "john", + "password": "SecurePass123!" + }' +``` + +Response: + +```json +{ + "access_token": "eyJhbGciOiJIUzI1NiIs...", + "refresh_token": "eyJhbGciOiJIUzI1NiIs...", + "token_type": "bearer", + "expires_in": 1800 +} +``` + +#### Get Current User + +```bash +curl http://localhost:7989/api/v1/auth/me \ + -H "Authorization: Bearer " +``` + +#### Refresh Token + +```bash +curl -X POST http://localhost:7989/api/v1/auth/refresh \ + -H "Content-Type: application/json" \ + -d '{"refresh_token": ""}' +``` + +#### Change Password + +```bash +curl -X POST http://localhost:7989/api/v1/auth/change-password \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "current_password": "SecurePass123!", + "new_password": "NewSecurePass456!" + }' +``` + +#### Logout + +```bash +curl -X POST http://localhost:7989/api/v1/auth/logout \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"refresh_token": ""}' +``` + +### Documents + +#### Upload Document + +```bash +# Upload a PDF +curl -X POST http://localhost:7989/api/v1/documents/upload \ + -H "Authorization: Bearer " \ + -F "file=@/path/to/document.pdf" + +# Upload a scanned image +curl -X POST http://localhost:7989/api/v1/documents/upload \ + -H "Authorization: Bearer " \ + -F "file=@/path/to/scan.jpg" +``` + +Response: + +```json +{ + "id": "550e8400-e29b-41d4-a716-446655440000", + "filename": "abc123_document.pdf", + "original_filename": "document.pdf", + "content_type": "application/pdf", + "file_size": 245760, + "checksum": "e3b0c44298fc1c149afbf4c8996fb924...", + "status": "pending", + "created_at": "2026-06-01T12:00:00Z" +} +``` + +#### Get Document + +```bash +curl http://localhost:7989/api/v1/documents/ \ + -H "Authorization: Bearer " +``` + +#### List Documents + +```bash +# With pagination +curl "http://localhost:7989/api/v1/documents?page=1&page_size=20" \ + -H "Authorization: Bearer " + +# Filter by status +curl "http://localhost:7989/api/v1/documents?status=completed" \ + -H "Authorization: Bearer " +``` + +#### Delete Document + +```bash +curl -X DELETE http://localhost:7989/api/v1/documents/ \ + -H "Authorization: Bearer " +``` + +#### Get Template Matches for Document + +```bash +curl http://localhost:7989/api/v1/documents//template \ + -H "Authorization: Bearer " +``` + +### Templates + +#### List Templates + +```bash +curl "http://localhost:7989/api/v1/templates?page=1&page_size=20" \ + -H "Authorization: Bearer " +``` + +#### Get Template + +```bash +curl http://localhost:7989/api/v1/templates/ \ + -H "Authorization: Bearer " +``` + +#### Delete (Deactivate) Template + +```bash +curl -X DELETE http://localhost:7989/api/v1/templates/ \ + -H "Authorization: Bearer " +``` + +#### Match Document to Templates + +```bash +curl -X POST http://localhost:7989/api/v1/templates/match \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "document_id": "", + "min_confidence": 0.5, + "max_results": 5 + }' +``` + +Response: + +```json +[ + { + "id": "...", + "document_id": "...", + "format_id": "...", + "confidence_score": 0.92, + "match_details": { "dimension_score": 1.0, "header_score": 0.85 }, + "selected": true, + "template_name": "Invoice Template v1", + "created_at": "2026-06-01T12:00:00Z" + } +] +``` + +#### Render Template to PDF + +```bash +curl -X POST http://localhost:7989/api/v1/templates/render \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "template_id": "", + "data": { + "company_name": "Acme Corp", + "invoice_number": "INV-2026-001", + "date": "2026-06-01", + "total": "$1,250.00" + }, + "output_filename": "invoice_output.pdf" + }' +``` + +Response: + +```json +{ + "output_path": "rendered/invoice_output.pdf", + "filename": "invoice_output.pdf", + "file_size": 32768, + "page_count": 1, + "rendered_at": "2026-06-01T12:05:00Z" +} +``` + +#### Download Rendered PDF + +```bash +curl -O http://localhost:7989/api/v1/templates//download?filename=invoice_output.pdf \ + -H "Authorization: Bearer " +``` + +--- + +## Processing Pipeline + +When a document is uploaded, the following Celery task pipeline executes asynchronously: + +1. **File Type Detection** — Determine if the document is a native PDF or scanned image. +2. **Page Extraction** — Convert PDF pages to images (for scanned docs) or parse directly (for native PDFs). +3. **OCR** — Run PaddleOCR on scanned pages to extract text blocks with coordinates, confidence, and bounding boxes. +4. **Native PDF Parsing** — Use PyMuPDF to extract text, fonts, images, and tables from native PDFs. +5. **Layout Analysis** — Detect headers, footers, watermarks, tables, and image regions using OpenCV heuristics. +6. **Template Generation** — Build a reusable template definition from the detected layout, stored in PostgreSQL. +7. **Fingerprint Generation** — Compute a structural fingerprint (SHA-256) for future matching. +8. **Status Update** — Mark the document as `completed` (or `failed` with error details). + +--- + +## Database + +**Connection**: `postgresql://postgres:***@192.168.0.111:5432/ocr` +**Schema**: `admin` + +### Migrations + +```bash +# Create a new migration +alembic revision --autogenerate -m "description" + +# Apply migrations +alembic upgrade head + +# Rollback one step +alembic downgrade -1 +``` + +### Tables + +| Table | Purpose | +|--------------------------|----------------------------------------------| +| `users` | User accounts | +| `roles` | Role definitions (admin, user, viewer) | +| `user_roles` | User-role mapping (M2M) | +| `refresh_tokens` | JWT refresh token storage | +| `audit_logs` | Action audit trail | +| `documents` | Uploaded document records | +| `document_pages` | Per-page data (dimensions, images) | +| `document_text_blocks` | Extracted text with position & font info | +| `document_images` | Extracted images with position | +| `document_tables` | Extracted tables with cell data (JSONB) | +| `document_formats` | Template definitions | +| `document_cells` | Template cell layout definitions | +| `document_regions` | Template region definitions | +| `table_formats` | Template table structure definitions | +| `table_columns` | Template table column definitions | +| `table_rows` | Template table row definitions | +| `watermarks` | Template watermark definitions | +| `image_regions` | Template image region definitions | +| `template_fingerprints` | Structural fingerprints for matching | +| `template_matches` | Document-to-template match results | + +--- + +## Testing + +```bash +# Install dev dependencies +pip install -r requirements-dev.txt + +# Run all tests +pytest + +# Run with coverage +pytest --cov=app --cov-report=term-missing + +# Run specific test categories +pytest tests/unit/ +pytest tests/api/ +pytest tests/repositories/ +``` + +--- + +## Configuration + +All configuration is via environment variables (`.env` file). Key settings: + +| Variable | Default | Description | +|------------------------------------|------------------------|---------------------------------| +| `APP_PORT` | `7989` | Application port | +| `DB_HOST` | `192.168.0.111` | PostgreSQL host | +| `DB_PORT` | `5432` | PostgreSQL port | +| `DB_NAME` | `ocr` | Database name | +| `DB_SCHEMA` | `admin` | PostgreSQL schema | +| `REDIS_HOST` | `192.168.0.111` | Redis host | +| `CELERY_BROKER_URL` | `redis://:***@192.168.0.111:7901/0` | Celery broker | +| `JWT_SECRET_KEY` | *(see .env)* | JWT signing key | +| `JWT_ACCESS_TOKEN_EXPIRE_MINUTES` | `30` | Access token TTL | +| `STORAGE_LOCAL_PATH` | `./storage` | Local file storage path | +| `STORAGE_MAX_FILE_SIZE_MB` | `100` | Max upload size | +| `OCR_LANGUAGE` | `en` | PaddleOCR language | +| `OCR_USE_GPU` | `false` | Enable GPU for OCR | + +--- + +## Production Deployment + +```bash +# Using production compose file +docker compose -f docker-compose.prod.yml up --build -d + +# Scale workers +docker compose -f docker-compose.prod.yml up --scale worker=4 -d +``` + +Production compose includes: +- Resource limits (CPU/memory) +- Redis authentication +- App replicas +- Persistent named volumes +- Auto-restart policies + +--- + +## Default Credentials + +| Username | Password | Role | +|----------|---------------|-------| +| `admin` | `Admin@123!` | admin | + +> ⚠️ **Change the default admin password immediately in production.** + +--- + +## License + +Proprietary — All rights reserved. diff --git a/docengine/alembic.ini b/docengine/alembic.ini new file mode 100644 index 0000000..21d448c --- /dev/null +++ b/docengine/alembic.ini @@ -0,0 +1,41 @@ +[alembic] +script_location = alembic +prepend_sys_path = . +version_path_separator = os +sqlalchemy.url = postgresql+psycopg2://postgres:M%%40triXPostgr3s%%406202@192.168.0.111:5432/ocr + +[post_write_hooks] + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/docengine/alembic/env.py b/docengine/alembic/env.py new file mode 100644 index 0000000..c5fe0fd --- /dev/null +++ b/docengine/alembic/env.py @@ -0,0 +1,67 @@ +from logging.config import fileConfig + +from alembic import context +from sqlalchemy import engine_from_config, pool, text + +from app.core.config import settings +from app.core.database import Base + +# Import all models so Alembic can detect them +import app.models # noqa: F401 + +config = context.config + +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = Base.metadata + +# Override the database URL from settings +config.set_main_option("sqlalchemy.url", settings.database_url.replace('%', '%%')) + + +def run_migrations_offline() -> None: + """Run migrations in 'offline' mode.""" + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + version_table_schema=settings.db_schema, + include_schemas=True, + ) + + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + """Run migrations in 'online' mode.""" + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + + with connectable.connect() as connection: + # Ensure schema exists + connection.execute(text(f"CREATE SCHEMA IF NOT EXISTS {settings.db_schema}")) + connection.execute(text(f"SET search_path TO {settings.db_schema}, public")) + connection.commit() + + context.configure( + connection=connection, + target_metadata=target_metadata, + version_table_schema=settings.db_schema, + include_schemas=True, + ) + + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/docengine/alembic/script.py.mako b/docengine/alembic/script.py.mako new file mode 100644 index 0000000..fbc4b07 --- /dev/null +++ b/docengine/alembic/script.py.mako @@ -0,0 +1,26 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision: str = ${repr(up_revision)} +down_revision: Union[str, None] = ${repr(down_revision)} +branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} +depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/docengine/alembic/versions/001_initial.py b/docengine/alembic/versions/001_initial.py new file mode 100644 index 0000000..4f401b6 --- /dev/null +++ b/docengine/alembic/versions/001_initial.py @@ -0,0 +1,398 @@ +"""initial schema + +Revision ID: 001_initial +Revises: +Create Date: 2026-05-31 18:00:00.000000 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +revision: str = "001_initial" +down_revision: Union[str, None] = None +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +SCHEMA = "admin" + + +def upgrade() -> None: + # Create schema + op.execute(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}") + + # Users table + op.create_table( + "users", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("username", sa.String(150), unique=True, nullable=False, index=True), + sa.Column("email", sa.String(255), unique=True, nullable=False, index=True), + sa.Column("hashed_password", sa.String(255), nullable=False), + sa.Column("full_name", sa.String(255), nullable=True), + sa.Column("is_active", sa.Boolean, default=True, nullable=False), + sa.Column("is_superuser", sa.Boolean, default=False, nullable=False), + sa.Column("last_login", sa.DateTime(timezone=True), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Roles table + op.create_table( + "roles", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("name", sa.String(50), unique=True, nullable=False, index=True), + sa.Column("description", sa.Text, nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # User roles (many-to-many) + op.create_table( + "user_roles", + sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), primary_key=True), + sa.Column("role_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.roles.id", ondelete="CASCADE"), primary_key=True), + schema=SCHEMA, + ) + + # Refresh tokens + op.create_table( + "refresh_tokens", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("token", sa.String(512), unique=True, nullable=False, index=True), + sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("revoked", sa.Boolean, default=False, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Audit logs + op.create_table( + "audit_logs", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("user_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True), + sa.Column("action", sa.String(100), nullable=False, index=True), + sa.Column("resource_type", sa.String(100), nullable=False, index=True), + sa.Column("resource_id", sa.String(255), nullable=True), + sa.Column("details", sa.Text, nullable=True), + sa.Column("ip_address", sa.String(45), nullable=True), + sa.Column("user_agent", sa.String(512), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False, index=True), + schema=SCHEMA, + ) + + # Documents + op.create_table( + "documents", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("filename", sa.String(500), nullable=False), + sa.Column("original_filename", sa.String(500), nullable=False), + sa.Column("content_type", sa.String(100), nullable=False), + sa.Column("file_size", sa.BigInteger, nullable=False), + sa.Column("checksum", sa.String(128), nullable=False, index=True), + sa.Column("storage_path", sa.String(1024), nullable=False), + sa.Column("status", sa.String(50), default="pending", nullable=False, index=True), + sa.Column("page_count", sa.Integer, nullable=True), + sa.Column("is_scanned", sa.Boolean, nullable=True), + sa.Column("document_metadata", postgresql.JSONB, nullable=True), + sa.Column("error_message", sa.Text, nullable=True), + sa.Column("uploaded_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True, index=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document pages + op.create_table( + "document_pages", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("image_path", sa.String(1024), nullable=True), + sa.Column("text_content", sa.Text, nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document text blocks + op.create_table( + "document_text_blocks", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("text", sa.Text, nullable=False), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("confidence", sa.Float, nullable=True), + sa.Column("font_family", sa.String(255), nullable=True), + sa.Column("font_size", sa.Float, nullable=True), + sa.Column("font_color", sa.String(50), nullable=True), + sa.Column("font_style", sa.String(50), nullable=True), + sa.Column("block_type", sa.String(50), default="text", nullable=False), + sa.Column("sequence", sa.Integer, default=0, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document images + op.create_table( + "document_images", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("image_path", sa.String(1024), nullable=False), + sa.Column("image_type", sa.String(50), default="figure", nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document tables + op.create_table( + "document_tables", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("page_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_pages.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("rows", sa.Integer, nullable=False), + sa.Column("columns", sa.Integer, nullable=False), + sa.Column("data", postgresql.JSONB, nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document formats (templates) + op.create_table( + "document_formats", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("name", sa.String(255), nullable=False, index=True), + sa.Column("description", sa.Text, nullable=True), + sa.Column("page_width", sa.Float, nullable=False), + sa.Column("page_height", sa.Float, nullable=False), + sa.Column("page_count", sa.Integer, default=1, nullable=False), + sa.Column("margin_top", sa.Float, default=72.0, nullable=False), + sa.Column("margin_right", sa.Float, default=72.0, nullable=False), + sa.Column("margin_bottom", sa.Float, default=72.0, nullable=False), + sa.Column("margin_left", sa.Float, default=72.0, nullable=False), + sa.Column("fingerprint", postgresql.JSONB, nullable=True), + sa.Column("source_document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="SET NULL"), nullable=True, index=True), + sa.Column("version", sa.Integer, default=1, nullable=False), + sa.Column("is_active", sa.Boolean, default=True, nullable=False, index=True), + sa.Column("created_by", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.users.id", ondelete="SET NULL"), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document cells + op.create_table( + "document_cells", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=False), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("row_no", sa.Integer, default=0, nullable=False), + sa.Column("column_no", sa.Integer, default=0, nullable=False), + sa.Column("data_type", sa.String(50), default="text", nullable=False), + sa.Column("font_family", sa.String(255), nullable=True), + sa.Column("font_size", sa.Float, nullable=True), + sa.Column("font_style", sa.String(50), nullable=True), + sa.Column("font_color", sa.String(50), nullable=True), + sa.Column("background_color", sa.String(50), nullable=True), + sa.Column("border_top", sa.String(100), nullable=True), + sa.Column("border_right", sa.String(100), nullable=True), + sa.Column("border_bottom", sa.String(100), nullable=True), + sa.Column("border_left", sa.String(100), nullable=True), + sa.Column("padding_top", sa.Float, default=0.0, nullable=False), + sa.Column("padding_right", sa.Float, default=0.0, nullable=False), + sa.Column("padding_bottom", sa.Float, default=0.0, nullable=False), + sa.Column("padding_left", sa.Float, default=0.0, nullable=False), + sa.Column("alignment", sa.String(20), default="left", nullable=False), + sa.Column("vertical_alignment", sa.String(20), default="top", nullable=False), + sa.Column("rowspan", sa.Integer, default=1, nullable=False), + sa.Column("colspan", sa.Integer, default=1, nullable=False), + sa.Column("static_text", sa.Text, nullable=True), + sa.Column("field_name", sa.String(255), nullable=True), + sa.Column("sequence", sa.Integer, default=0, nullable=False), + sa.Column("is_dynamic", sa.Boolean, default=False, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Document regions + op.create_table( + "document_regions", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=False), + sa.Column("region_type", sa.String(50), nullable=False), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("content", postgresql.JSONB, nullable=True), + sa.Column("sequence", sa.Integer, default=0, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Table formats + op.create_table( + "table_formats", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=False), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("rows", sa.Integer, nullable=False), + sa.Column("columns", sa.Integer, nullable=False), + sa.Column("border_style", sa.String(50), default="solid", nullable=False), + sa.Column("border_width", sa.Float, default=1.0, nullable=False), + sa.Column("border_color", sa.String(50), default="#000000", nullable=False), + sa.Column("header_rows", sa.Integer, default=1, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Table columns + op.create_table( + "table_columns", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("column_index", sa.Integer, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("header_text", sa.String(500), nullable=True), + sa.Column("data_type", sa.String(50), default="text", nullable=False), + sa.Column("alignment", sa.String(20), default="left", nullable=False), + sa.Column("font_family", sa.String(255), nullable=True), + sa.Column("font_size", sa.Float, nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Table rows + op.create_table( + "table_rows", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("table_format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.table_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("row_index", sa.Integer, nullable=False), + sa.Column("height", sa.Float, default=20.0, nullable=False), + sa.Column("is_header", sa.Boolean, default=False, nullable=False), + sa.Column("background_color", sa.String(50), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Watermarks + op.create_table( + "watermarks", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=True), + sa.Column("text", sa.String(500), nullable=True), + sa.Column("image_path", sa.String(1024), nullable=True), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("opacity", sa.Float, default=0.3, nullable=False), + sa.Column("rotation", sa.Float, default=0.0, nullable=False), + sa.Column("font_family", sa.String(255), nullable=True), + sa.Column("font_size", sa.Float, nullable=True), + sa.Column("font_color", sa.String(50), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Image regions + op.create_table( + "image_regions", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("page_number", sa.Integer, nullable=False), + sa.Column("x", sa.Float, nullable=False), + sa.Column("y", sa.Float, nullable=False), + sa.Column("width", sa.Float, nullable=False), + sa.Column("height", sa.Float, nullable=False), + sa.Column("image_path", sa.String(1024), nullable=True), + sa.Column("image_type", sa.String(50), default="figure", nullable=False), + sa.Column("is_static", sa.Boolean, default=True, nullable=False), + sa.Column("field_name", sa.String(255), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Template fingerprints + op.create_table( + "template_fingerprints", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, unique=True, index=True), + sa.Column("page_dimensions", postgresql.JSONB, nullable=True), + sa.Column("logo_coordinates", postgresql.JSONB, nullable=True), + sa.Column("header_coordinates", postgresql.JSONB, nullable=True), + sa.Column("footer_coordinates", postgresql.JSONB, nullable=True), + sa.Column("table_coordinates", postgresql.JSONB, nullable=True), + sa.Column("cell_coordinates", postgresql.JSONB, nullable=True), + sa.Column("fingerprint_hash", sa.String(256), nullable=False, index=True), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Template matches + op.create_table( + "template_matches", + sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True), + sa.Column("document_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.documents.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("format_id", postgresql.UUID(as_uuid=True), sa.ForeignKey(f"{SCHEMA}.document_formats.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("confidence_score", sa.Float, nullable=False), + sa.Column("match_details", postgresql.JSONB, nullable=True), + sa.Column("selected", sa.Boolean, default=False, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), + schema=SCHEMA, + ) + + # Additional indexes + op.create_index("ix_documents_status_created", "documents", ["status", "created_at"], schema=SCHEMA) + op.create_index("ix_document_pages_doc_page", "document_pages", ["document_id", "page_number"], schema=SCHEMA) + op.create_index("ix_document_text_blocks_type", "document_text_blocks", ["page_id", "block_type"], schema=SCHEMA) + op.create_index("ix_document_cells_format_page", "document_cells", ["format_id", "page_number"], schema=SCHEMA) + op.create_index("ix_template_matches_doc_score", "template_matches", ["document_id", "confidence_score"], schema=SCHEMA) + op.create_index("ix_audit_logs_resource", "audit_logs", ["resource_type", "resource_id"], schema=SCHEMA) + + +def downgrade() -> None: + op.drop_table("template_matches", schema=SCHEMA) + op.drop_table("template_fingerprints", schema=SCHEMA) + op.drop_table("image_regions", schema=SCHEMA) + op.drop_table("watermarks", schema=SCHEMA) + op.drop_table("table_rows", schema=SCHEMA) + op.drop_table("table_columns", schema=SCHEMA) + op.drop_table("table_formats", schema=SCHEMA) + op.drop_table("document_regions", schema=SCHEMA) + op.drop_table("document_cells", schema=SCHEMA) + op.drop_table("document_formats", schema=SCHEMA) + op.drop_table("document_tables", schema=SCHEMA) + op.drop_table("document_images", schema=SCHEMA) + op.drop_table("document_text_blocks", schema=SCHEMA) + op.drop_table("document_pages", schema=SCHEMA) + op.drop_table("documents", schema=SCHEMA) + op.drop_table("audit_logs", schema=SCHEMA) + op.drop_table("refresh_tokens", schema=SCHEMA) + op.drop_table("user_roles", schema=SCHEMA) + op.drop_table("roles", schema=SCHEMA) + op.drop_table("users", schema=SCHEMA) diff --git a/docengine/app.py b/docengine/app.py new file mode 100644 index 0000000..c52e022 --- /dev/null +++ b/docengine/app.py @@ -0,0 +1,11 @@ + +from fastapi import FastAPI +app = FastAPI() + +@app.get("/health") +def health(): + return {"status":"UP"} + +if __name__ == "__main__": + import uvicorn + uvicorn.run(app, host="0.0.0.0", port=7989) diff --git a/docengine/app/__init__.py b/docengine/app/__init__.py new file mode 100644 index 0000000..bd8e77e --- /dev/null +++ b/docengine/app/__init__.py @@ -0,0 +1 @@ +# DocEngine - Document Template Recognition and Reconstruction System diff --git a/frontend/src/app/login/login.scss b/docengine/app/api/__init__.py similarity index 100% rename from frontend/src/app/login/login.scss rename to docengine/app/api/__init__.py diff --git a/docengine/app/api/router.py b/docengine/app/api/router.py new file mode 100644 index 0000000..14b608d --- /dev/null +++ b/docengine/app/api/router.py @@ -0,0 +1,15 @@ +from __future__ import annotations + +from fastapi import APIRouter + +from app.api.v1.auth import router as auth_router +from app.api.v1.documents import router as documents_router +from app.api.v1.health import router as health_router +from app.api.v1.templates import router as templates_router + +api_v1_router = APIRouter(prefix="/api/v1") + +api_v1_router.include_router(health_router) +api_v1_router.include_router(auth_router) +api_v1_router.include_router(documents_router) +api_v1_router.include_router(templates_router) diff --git a/docengine/app/api/v1/__init__.py b/docengine/app/api/v1/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/api/v1/auth.py b/docengine/app/api/v1/auth.py new file mode 100644 index 0000000..bf497b4 --- /dev/null +++ b/docengine/app/api/v1/auth.py @@ -0,0 +1,246 @@ +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +from fastapi import APIRouter, Depends, HTTPException, status +from sqlalchemy.orm import Session + +from app.core.config import settings +from app.core.database import get_db +from app.core.dependencies import CurrentUser +from app.core.security import ( + create_access_token, + create_refresh_token, + decode_token, + hash_password, + verify_password, + InvalidTokenError, +) +from app.repositories.user_repository import RefreshTokenRepository, UserRepository +from app.schemas.auth import ( + ChangePasswordRequest, + LoginRequest, + RefreshTokenRequest, + RegisterRequest, + TokenResponse, +) +from app.schemas.common import SuccessResponse +from app.schemas.user import UserResponse + +router = APIRouter(prefix="/auth", tags=["Authentication"]) + + +@router.post( + "/register", + response_model=UserResponse, + status_code=status.HTTP_201_CREATED, + summary="Register User", + description="Register a new user account.", +) +def register( + payload: RegisterRequest, + db: Session = Depends(get_db), +) -> UserResponse: + """Register a new user.""" + user_repo = UserRepository(db) + + # Check for existing user + if user_repo.get_by_username(payload.username): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f"Username '{payload.username}' is already taken", + ) + if user_repo.get_by_email(payload.email): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f"Email '{payload.email}' is already registered", + ) + + hashed = hash_password(payload.password) + user = user_repo.create_user( + username=payload.username, + email=payload.email, + hashed_password=hashed, + full_name=payload.full_name, + role_names=["user"], + ) + db.commit() + db.refresh(user) + + return UserResponse.model_validate(user) + + +@router.post( + "/login", + response_model=TokenResponse, + summary="Login", + description="Authenticate with username and password to obtain JWT tokens.", +) +def login( + payload: LoginRequest, + db: Session = Depends(get_db), +) -> TokenResponse: + """Authenticate user and return JWT tokens.""" + user_repo = UserRepository(db) + refresh_repo = RefreshTokenRepository(db) + + user = user_repo.get_by_username(payload.username) + if not user or not verify_password(payload.password, user.hashed_password): + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid username or password", + headers={"WWW-Authenticate": "Bearer"}, + ) + + if not user.is_active: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="User account is deactivated", + ) + + # Generate tokens + access_token = create_access_token(data={"sub": str(user.id), "username": user.username}) + refresh_token_str = create_refresh_token(data={"sub": str(user.id)}) + + # Store refresh token + expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days) + refresh_repo.create_token( + user_id=user.id, + token=refresh_token_str, + expires_at=expires_at, + ) + + # Update last login + user_repo.update_last_login(user) + db.commit() + + return TokenResponse( + access_token=access_token, + refresh_token=refresh_token_str, + token_type="bearer", + expires_in=settings.jwt_access_token_expire_minutes * 60, + ) + + +@router.post( + "/refresh", + response_model=TokenResponse, + summary="Refresh Token", + description="Obtain a new access token using a valid refresh token.", +) +def refresh_token( + payload: RefreshTokenRequest, + db: Session = Depends(get_db), +) -> TokenResponse: + """Refresh access token using a refresh token.""" + refresh_repo = RefreshTokenRepository(db) + user_repo = UserRepository(db) + + # Validate the refresh token + try: + token_payload = decode_token(payload.refresh_token) + except InvalidTokenError: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid or expired refresh token", + ) + + if token_payload.get("type") != "refresh": + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid token type", + ) + + # Check if token exists in database and is not revoked + stored_token = refresh_repo.get_by_token(payload.refresh_token) + if not stored_token: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Refresh token not found or revoked", + ) + + user = user_repo.get_by_id(token_payload["sub"]) + if not user or not user.is_active: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="User not found or deactivated", + ) + + # Revoke old refresh token + refresh_repo.revoke_token(payload.refresh_token) + + # Generate new tokens + new_access_token = create_access_token(data={"sub": str(user.id), "username": user.username}) + new_refresh_token = create_refresh_token(data={"sub": str(user.id)}) + + expires_at = datetime.now(UTC) + timedelta(days=settings.jwt_refresh_token_expire_days) + refresh_repo.create_token( + user_id=user.id, + token=new_refresh_token, + expires_at=expires_at, + ) + db.commit() + + return TokenResponse( + access_token=new_access_token, + refresh_token=new_refresh_token, + token_type="bearer", + expires_in=settings.jwt_access_token_expire_minutes * 60, + ) + + +@router.post( + "/logout", + response_model=SuccessResponse, + summary="Logout", + description="Revoke the current refresh token.", +) +def logout( + payload: RefreshTokenRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> SuccessResponse: + """Logout by revoking the refresh token.""" + refresh_repo = RefreshTokenRepository(db) + refresh_repo.revoke_token(payload.refresh_token) + db.commit() + return SuccessResponse(message="Successfully logged out") + + +@router.post( + "/change-password", + response_model=SuccessResponse, + summary="Change Password", + description="Change the current user's password.", +) +def change_password( + payload: ChangePasswordRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> SuccessResponse: + """Change user password.""" + if not verify_password(payload.current_password, current_user.hashed_password): + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Current password is incorrect", + ) + + current_user.hashed_password = hash_password(payload.new_password) + + # Revoke all refresh tokens for security + refresh_repo = RefreshTokenRepository(db) + refresh_repo.revoke_all_user_tokens(current_user.id) + db.commit() + + return SuccessResponse(message="Password changed successfully") + + +@router.get( + "/me", + response_model=UserResponse, + summary="Get Current User", + description="Get the currently authenticated user's profile.", +) +def get_me(current_user: CurrentUser) -> UserResponse: + """Get current authenticated user profile.""" + return UserResponse.model_validate(current_user) diff --git a/docengine/app/api/v1/documents.py b/docengine/app/api/v1/documents.py new file mode 100644 index 0000000..9292b23 --- /dev/null +++ b/docengine/app/api/v1/documents.py @@ -0,0 +1,307 @@ +from __future__ import annotations + +import uuid +from typing import Annotated + +from fastapi import APIRouter, Depends, File, HTTPException, Query, UploadFile, status +from sqlalchemy.orm import Session + +from app.core.config import settings +from app.core.database import get_db +from app.core.dependencies import CurrentUser +from app.core.exceptions import FileSizeError, UnsupportedFileTypeError +from app.core.logging_config import get_logger +from app.models.document import Document +from app.repositories.document_repository import DocumentRepository +from app.schemas.common import PaginatedResponse, SuccessResponse +from app.schemas.document import ( + DocumentListResponse, + DocumentResponse, + DocumentUploadResponse, + TemplateMatchRequest, + TemplateMatchResponse, +) +from app.storage.provider import LocalStorageProvider, get_storage_provider + +logger = get_logger(__name__) + +router = APIRouter(prefix="/documents", tags=["Documents"]) + +ALLOWED_CONTENT_TYPES = { + "image/jpeg": "jpg", + "image/png": "png", + "image/tiff": "tiff", + "application/pdf": "pdf", +} + +ALLOWED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tiff", ".tif", ".pdf"} + + +def _validate_file(file: UploadFile) -> str: + """Validate uploaded file type and size. Returns the content type.""" + if not file.filename: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Filename is required", + ) + + # Check extension + from pathlib import Path + ext = Path(file.filename).suffix.lower() + if ext not in ALLOWED_EXTENSIONS: + raise UnsupportedFileTypeError(ext) + + # Determine content type + content_type = file.content_type or "" + if content_type not in ALLOWED_CONTENT_TYPES: + # Try to infer from extension + ext_to_ct = { + ".jpg": "image/jpeg", + ".jpeg": "image/jpeg", + ".png": "image/png", + ".tiff": "image/tiff", + ".tif": "image/tiff", + ".pdf": "application/pdf", + } + content_type = ext_to_ct.get(ext, "") + if not content_type: + raise UnsupportedFileTypeError(file.content_type or "unknown") + + return content_type + + +@router.post( + "/upload", + response_model=DocumentUploadResponse, + status_code=status.HTTP_201_CREATED, + summary="Upload Document", + description="Upload a document (JPG, JPEG, PNG, TIFF, or PDF) for processing.", +) +async def upload_document( + file: UploadFile = File(..., description="Document file to upload"), + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> DocumentUploadResponse: + """Upload a document for processing.""" + content_type = _validate_file(file) + + # Read file data + file_data = await file.read() + + # Check file size + if len(file_data) > settings.storage_max_file_size_bytes: + raise FileSizeError(settings.storage_max_file_size_mb) + + # Store file + storage = get_storage_provider() + checksum = storage.compute_checksum(file_data) + safe_filename = file.filename or "unknown" + stored_filename = storage.generate_filename(safe_filename) + storage_path = storage.save_file(file_data, "documents", stored_filename) + + # Create document record + doc_repo = DocumentRepository(db) + document = Document( + filename=stored_filename, + original_filename=safe_filename, + content_type=content_type, + file_size=len(file_data), + checksum=checksum, + storage_path=storage_path, + status="pending", + uploaded_by=current_user.id if current_user else None, + ) + doc_repo.create(document) + db.commit() + db.refresh(document) + + logger.info( + "document_uploaded", + document_id=str(document.id), + filename=safe_filename, + size=len(file_data), + content_type=content_type, + ) + + # Trigger async processing via Celery + try: + from app.tasks.document_tasks import process_document_task + process_document_task.delay(str(document.id)) + except Exception as e: + logger.warning("celery_dispatch_failed", error=str(e), document_id=str(document.id)) + + return DocumentUploadResponse.model_validate(document) + + +@router.get( + "/{document_id}", + response_model=DocumentResponse, + summary="Get Document", + description="Retrieve a document by its ID with all extracted content.", +) +def get_document( + document_id: uuid.UUID, + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> DocumentResponse: + """Get a document by ID.""" + doc_repo = DocumentRepository(db) + document = doc_repo.get_with_pages(document_id) + if not document: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Document '{document_id}' not found", + ) + return DocumentResponse.model_validate(document) + + +@router.get( + "", + response_model=PaginatedResponse[DocumentListResponse], + summary="List Documents", + description="List all documents with pagination.", +) +def list_documents( + page: int = Query(default=1, ge=1), + page_size: int = Query(default=20, ge=1, le=100), + status_filter: str | None = Query(default=None, alias="status"), + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> PaginatedResponse[DocumentListResponse]: + """List documents with pagination and optional status filter.""" + doc_repo = DocumentRepository(db) + offset = (page - 1) * page_size + filters = {} + if status_filter: + filters["status"] = status_filter + + documents = doc_repo.get_all( + offset=offset, + limit=page_size, + filters=filters, + order_by="created_at", + order_desc=True, + ) + total = doc_repo.count(filters=filters) + + items = [DocumentListResponse.model_validate(doc) for doc in documents] + return PaginatedResponse.create( + items=items, + total=total, + page=page, + page_size=page_size, + ) + + +@router.get( + "/{document_id}/template", + response_model=list[TemplateMatchResponse], + summary="Get Document Template Matches", + description="Get template matching results for a document.", +) +def get_document_template_matches( + document_id: uuid.UUID, + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> list[TemplateMatchResponse]: + """Get template matches for a document.""" + from app.repositories.document_repository import TemplateMatchRepository + + doc_repo = DocumentRepository(db) + document = doc_repo.get_by_id(document_id) + if not document: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Document '{document_id}' not found", + ) + + match_repo = TemplateMatchRepository(db) + matches = match_repo.get_document_matches(document_id) + + results = [] + for match in matches: + resp = TemplateMatchResponse( + id=match.id, + document_id=match.document_id, + format_id=match.format_id, + confidence_score=match.confidence_score, + match_details=match.match_details, + selected=match.selected, + template_name=match.template.name if match.template else None, + created_at=match.created_at, + ) + results.append(resp) + + return results + + +@router.post( + "/{document_id}/extraction", + response_model=dict, + summary="Extract Document Data", + description="Match a document against templates and extract key-value & table data.", +) +def extract_document_data( + document_id: uuid.UUID, + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> dict: + """Extract document data using matched template mappings.""" + doc_repo = DocumentRepository(db) + document = doc_repo.get_by_id(document_id) + if not document: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Document '{document_id}' not found", + ) + + if document.status != "completed": + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Document must be in 'completed' status. Current status: '{document.status}'", + ) + + from app.services.extraction_service import ExtractionService + extraction_service = ExtractionService(db) + try: + result = extraction_service.extract_document_data(document_id) + return result + except Exception as e: + logger.exception("extraction_endpoint_failed", document_id=str(document_id), error=str(e)) + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Extraction failed: {str(e)}", + ) + + +@router.delete( + "/{document_id}", + response_model=SuccessResponse, + summary="Delete Document", + description="Delete a document and its associated data.", +) +def delete_document( + document_id: uuid.UUID, + current_user: CurrentUser = None, + db: Session = Depends(get_db), +) -> SuccessResponse: + """Delete a document.""" + doc_repo = DocumentRepository(db) + document = doc_repo.get_by_id(document_id) + if not document: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Document '{document_id}' not found", + ) + + # Delete stored file + try: + storage = get_storage_provider() + storage.delete_file(document.storage_path) + except Exception as e: + logger.warning("file_delete_failed", error=str(e), path=document.storage_path) + + doc_repo.delete(document) + db.commit() + + return SuccessResponse(message=f"Document '{document_id}' deleted successfully") diff --git a/docengine/app/api/v1/health.py b/docengine/app/api/v1/health.py new file mode 100644 index 0000000..1a0f646 --- /dev/null +++ b/docengine/app/api/v1/health.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +from datetime import datetime + +import redis +from fastapi import APIRouter, status + +from app.core.config import settings +from app.core.database import check_database_connection +from app.schemas.common import HealthResponse + +router = APIRouter(tags=["Health"]) + + +@router.get( + "/health", + response_model=HealthResponse, + status_code=status.HTTP_200_OK, + summary="Health Check", + description="Check the health status of the application and its dependencies.", +) +async def health_check() -> HealthResponse: + """Perform health check on all system components.""" + # Check database + db_status = "healthy" if check_database_connection() else "unhealthy" + + # Check Redis + redis_status = "healthy" + try: + r = redis.Redis( + host=settings.redis_host, + port=settings.redis_port, + db=settings.redis_db, + password=settings.redis_password or None, + socket_timeout=3, + ) + r.ping() + r.close() + except Exception: + redis_status = "unhealthy" + + overall_status = "healthy" if db_status == "healthy" and redis_status == "healthy" else "degraded" + + return HealthResponse( + status=overall_status, + version=settings.app_version, + environment=settings.app_env, + database=db_status, + redis=redis_status, + timestamp=datetime.utcnow(), + ) diff --git a/docengine/app/api/v1/templates.py b/docengine/app/api/v1/templates.py new file mode 100644 index 0000000..ec3ac4d --- /dev/null +++ b/docengine/app/api/v1/templates.py @@ -0,0 +1,423 @@ +from __future__ import annotations + +import uuid + +from fastapi import APIRouter, Depends, HTTPException, Query, status +from fastapi.responses import FileResponse +from sqlalchemy.orm import Session + +from app.core.database import get_db +from app.core.dependencies import CurrentUser +from app.core.logging_config import get_logger +from app.repositories.document_repository import DocumentRepository, TemplateMatchRepository +from app.repositories.template_repository import ( + TemplateRepository, + DocumentRegionRepository, + DocumentCellRepository, + TableFormatRepository, + TableColumnRepository +) +from app.schemas.common import PaginatedResponse, SuccessResponse +from app.schemas.document import TemplateMatchRequest, TemplateMatchResponse +from app.schemas.template import ( + TemplateListResponse, + TemplateRenderRequest, + TemplateRenderResponse, + TemplateResponse, + TemplateCreateRequest, + TemplateMappingSaveRequest, +) + +logger = get_logger(__name__) + +router = APIRouter(prefix="/templates", tags=["Templates"]) + + +@router.post( + "", + response_model=dict, + summary="Create Template", + description="Create a new template.", +) +def create_template( + payload: TemplateCreateRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> dict: + """Create a template.""" + template_repo = TemplateRepository(db) + + template = template_repo.create_template( + name=payload.template_name, + page_width=1000.0, + page_height=1000.0, + source_document_id=uuid.UUID(payload.source_document_id) if payload.source_document_id else None + ) + cell_repo = DocumentCellRepository(db) + saved_fields = [] + + for index, field in enumerate(payload.fields): + cell = cell_repo.create_cell( + format_id=template.id, + page_number=1, + x=0.0, + y=0.0, + width=0.0, + height=0.0, + data_type=field.field_type, + field_name=field.field_label, + sequence=field.display_order or index, + is_dynamic=True + ) + saved_fields.append({ + "field_label": cell.field_name, + "field_type": cell.data_type + }) + + db.commit() + + return { + "pk_template_id": str(template.id), + "template_name": template.name, + "fields": saved_fields + } + + +@router.put( + "/{template_id}", + response_model=dict, + summary="Update Template", + description="Update an existing template name and its fields.", +) +def update_template( + template_id: uuid.UUID, + payload: TemplateCreateRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> dict: + """Update a template.""" + template_repo = TemplateRepository(db) + template = template_repo.get_by_id(template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Template '{template_id}' not found", + ) + + if template.name != payload.template_name: + template.name = payload.template_name + + from app.models.template import DocumentCell + db.query(DocumentCell).filter(DocumentCell.format_id == template_id).delete() + + saved_fields = [] + cell_repo = DocumentCellRepository(db) + for index, field in enumerate(payload.fields): + cell = cell_repo.create_cell( + format_id=template.id, + page_number=1, + row_no=0, + column_no=0, + field_name=field.field_label, + data_type=field.field_type, + x=0.0, + y=0.0, + width=0.0, + height=0.0, + is_dynamic=True + ) + saved_fields.append({ + "field_label": cell.field_name, + "field_type": cell.data_type + }) + + db.commit() + + return { + "pk_template_id": str(template.id), + "template_name": template.name, + "fields": saved_fields + } + + +@router.post( + "/{template_id}/mappings/save", + response_model=SuccessResponse, + summary="Save Template Mappings", + description="Save the field mappings for a template.", +) +def save_mappings( + template_id: uuid.UUID, + payload: list[TemplateMappingSaveRequest], + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> SuccessResponse: + """Save template mappings.""" + template_repo = TemplateRepository(db) + template = template_repo.get_by_id(template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Template '{template_id}' not found", + ) + + region_repo = DocumentRegionRepository(db) + cell_repo = DocumentCellRepository(db) + table_repo = TableFormatRepository(db) + table_col_repo = TableColumnRepository(db) + + # Delete existing field mappings and table formats for this template + from app.models.template import DocumentRegion, TableFormat + db.query(DocumentRegion).filter( + DocumentRegion.format_id == template_id, + DocumentRegion.region_type == "field_mapping" + ).delete() + db.query(TableFormat).filter(TableFormat.format_id == template_id).delete() + db.flush() + + cells = cell_repo.get_template_cells(template_id) + cell_map = {c.field_name: c for c in cells} + table_format = None + + # Save each mapped node + for mapping in payload: + cell = cell_map.get(mapping.field_name) + is_table_column = cell and cell.data_type == 'TABLE_COLUMN' + + for node_idx, node in enumerate(mapping.mapped_nodes): + if is_table_column: + if not table_format: + table_format = table_repo.create_table_format( + format_id=template_id, + page_number=node.page_no or 1, + x=0.0, y=0.0, width=1000.0, height=1000.0, + rows=1, columns=10 + ) + table_col_repo.create_column( + table_format_id=table_format.id, + column_index=node_idx, + width=node.width, + header_text=mapping.field_name, + data_type="text" + ) + + region_repo.create_region( + format_id=template_id, + page_number=node.page_no or 1, + region_type="field_mapping", + x=node.x_coordinate, + y=node.y_coordinate, + width=node.width, + height=node.height, + content={ + "field_name": mapping.field_name, + "text_value": node.text_value + } + ) + + db.commit() + + # Generate fingerprint now that mappings are populated + from app.services.fingerprint_service import FingerprintService + FingerprintService(db).generate_fingerprint(template) + db.commit() + + return SuccessResponse(message="Mappings saved successfully") + + +@router.get( + "", + response_model=PaginatedResponse[TemplateListResponse], + summary="List Templates", + description="List all active templates with pagination.", +) +def list_templates( + current_user: CurrentUser, + page: int = Query(default=1, ge=1), + page_size: int = Query(default=20, ge=1, le=100), + db: Session = Depends(get_db), +) -> PaginatedResponse[TemplateListResponse]: + """List all active templates.""" + template_repo = TemplateRepository(db) + offset = (page - 1) * page_size + + templates = template_repo.get_active_templates(offset=offset, limit=page_size) + total = template_repo.count_active() + + items = [TemplateListResponse.model_validate(t) for t in templates] + return PaginatedResponse.create( + items=items, + total=total, + page=page, + page_size=page_size, + ) + + +@router.get( + "/{template_id}", + response_model=TemplateResponse, + summary="Get Template", + description="Retrieve a template by ID with all its components.", +) +def get_template( + template_id: uuid.UUID, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> TemplateResponse: + """Get a template by ID.""" + template_repo = TemplateRepository(db) + template = template_repo.get_by_id(template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Template '{template_id}' not found", + ) + return TemplateResponse.model_validate(template) + + +@router.delete( + "/{template_id}", + response_model=SuccessResponse, + summary="Delete Template", + description="Soft-delete a template by deactivating it.", +) +def delete_template( + template_id: uuid.UUID, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> SuccessResponse: + """Soft-delete a template.""" + template_repo = TemplateRepository(db) + template = template_repo.get_by_id(template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Template '{template_id}' not found", + ) + + template_repo.deactivate_template(template_id) + db.commit() + + return SuccessResponse(message=f"Template '{template_id}' deactivated successfully") + + +@router.post( + "/match", + response_model=list[TemplateMatchResponse], + summary="Match Document to Templates", + description="Match a document against existing templates and return ranked results.", +) +def match_template( + payload: TemplateMatchRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> list[TemplateMatchResponse]: + """Match a document against existing templates.""" + doc_repo = DocumentRepository(db) + document = doc_repo.get_by_id(payload.document_id) + if not document: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Document '{payload.document_id}' not found", + ) + + if document.status != "completed": + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Document must be in 'completed' status. Current status: '{document.status}'", + ) + + # Perform template matching + from app.services.matching_service import MatchingService + matching_service = MatchingService(db) + matches = matching_service.match_document( + document_id=payload.document_id, + min_confidence=payload.min_confidence, + max_results=payload.max_results, + ) + db.commit() + + results = [] + for match in matches: + resp = TemplateMatchResponse( + id=match.id, + document_id=match.document_id, + format_id=match.format_id, + confidence_score=match.confidence_score, + match_details=match.match_details, + selected=match.selected, + template_name=match.template.name if match.template else None, + created_at=match.created_at, + ) + results.append(resp) + + return results + + +@router.post( + "/render", + response_model=TemplateRenderResponse, + summary="Render Template to PDF", + description="Generate a PDF from a stored template with supplied data.", +) +def render_template( + payload: TemplateRenderRequest, + current_user: CurrentUser, + db: Session = Depends(get_db), +) -> TemplateRenderResponse: + """Render a template to PDF.""" + template_repo = TemplateRepository(db) + template = template_repo.get_by_id(payload.template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Template '{payload.template_id}' not found", + ) + + if not template.is_active: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Template is deactivated", + ) + + from app.services.reconstruction_service import ReconstructionService + reconstruction_service = ReconstructionService(db) + result = reconstruction_service.render_template( + template=template, + data=payload.data, + output_filename=payload.output_filename, + images=payload.images, + ) + + return result + + +@router.get( + "/{template_id}/download", + summary="Download Rendered PDF", + description="Download a previously rendered PDF.", +) +def download_rendered_pdf( + template_id: uuid.UUID, + current_user: CurrentUser, + filename: str = Query(..., description="Filename of the rendered PDF"), + db: Session = Depends(get_db), +) -> FileResponse: + """Download a rendered PDF.""" + from app.storage.provider import get_storage_provider + + storage = get_storage_provider() + storage_path = f"rendered/{filename}" + + if not storage.file_exists(storage_path): + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Rendered PDF '{filename}' not found", + ) + + absolute_path = storage.get_absolute_path(storage_path) + return FileResponse( + path=absolute_path, + media_type="application/pdf", + filename=filename, + ) diff --git a/docengine/app/core/__init__.py b/docengine/app/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/core/aes.py b/docengine/app/core/aes.py new file mode 100644 index 0000000..aacc416 --- /dev/null +++ b/docengine/app/core/aes.py @@ -0,0 +1,38 @@ +import base64 +from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + +def decrypt(base64_input: str, secret: bytes) -> str | None: + try: + combined = base64.b64decode(base64_input) + + # IV length is 12 bytes in the Java implementation + iv_length = 12 + if len(combined) <= iv_length + 16: + return None + + iv = combined[:iv_length] + + # Java AES/GCM appends a 16-byte authentication tag at the end of cipherBytes + # cryptography library expects it to be passed into the modes.GCM(iv, tag) + cipher_bytes_with_tag = combined[iv_length:] + actual_ciphertext = cipher_bytes_with_tag[:-16] + tag = cipher_bytes_with_tag[-16:] + + cipher = Cipher(algorithms.AES(secret), modes.GCM(iv, tag)) + decryptor = cipher.decryptor() + + plain_bytes = decryptor.update(actual_ciphertext) + decryptor.finalize() + return plain_bytes.decode('utf-8') + except Exception: + return None + +def get_string_value(encrypted_id: str, secret: str, secret_key_internal: str) -> str | None: + if not secret or not secret_key_internal: + return None + try: + # Replicate Java's logic: (s + secretKeyInternal).substring(0, Math.min(..., 16)) + combined_str = secret + secret_key_internal + combined_secret = combined_str[:16].encode('utf-8') + return decrypt(encrypted_id, combined_secret) + except Exception: + return None diff --git a/docengine/app/core/config.py b/docengine/app/core/config.py new file mode 100644 index 0000000..517c700 --- /dev/null +++ b/docengine/app/core/config.py @@ -0,0 +1,128 @@ +from __future__ import annotations + +import json +from typing import Any + +from pydantic import field_validator +from pydantic_settings import BaseSettings, SettingsConfigDict + + +class Settings(BaseSettings): + """Application configuration loaded from environment variables.""" + + model_config = SettingsConfigDict( + env_file=".env", + env_file_encoding="utf-8", + case_sensitive=False, + extra="ignore", + ) + + # Application + app_name: str = "DocEngine" + app_version: str = "1.0.0" + app_env: str = "development" + app_debug: bool = True + app_host: str = "0.0.0.0" + app_port: int = 7989 + app_workers: int = 4 + + # Database + db_host: str = "192.168.0.111" + db_port: int = 5432 + db_name: str = "ocr" + db_user: str = "postgres" + db_password: str = "M@triXPostgr3s@6202" + db_schema: str = "admin" + db_pool_size: int = 20 + db_max_overflow: int = 10 + db_echo: bool = False + + # Redis + redis_host: str = "192.168.0.111" + redis_port: int = 7901 + redis_db: int = 0 + redis_password: str = "M@triXR3d1s@6202" + + # Celery + celery_broker_url: str = "redis://:M@triXR3d1s@6202@192.168.0.111:7901/0" + celery_result_backend: str = "redis://:M@triXR3d1s@6202@192.168.0.111:7901/1" + + # JWT + jwt_secret_key: str = "a7f3c9e1d4b8f2a6c0e5d7b3a9f1c4e8d2b6a0f5c3e7d1b9a4f8c2e6d0b5a3" + jwt_algorithm: str = "HS256" + jwt_access_token_expire_minutes: int = 30 + jwt_refresh_token_expire_days: int = 7 + session_encryption_secret: str = "" + session_encryption_secret_internal: str = "" + + # Storage + storage_provider: str = "local" + storage_local_path: str = "./storage" + storage_max_file_size_mb: int = 100 + + # OCR + ocr_language: str = "en" + ocr_use_gpu: bool = False + + # Logging + log_level: str = "INFO" + log_format: str = "json" + + # CORS + cors_origins: list[str] = ["http://localhost:3000", "http://localhost:8080", "http://localhost:4200"] + cors_allow_credentials: bool = True + + # Rate Limiting + rate_limit_requests: int = 100 + rate_limit_window_seconds: int = 60 + + # Prometheus + prometheus_enabled: bool = True + + @field_validator("cors_origins", mode="before") + @classmethod + def parse_cors_origins(cls, v: Any) -> list[str]: + if isinstance(v, str): + if not v.strip(): + return [] + try: + parsed = json.loads(v) + if isinstance(parsed, list): + return [str(item).strip() for item in parsed] + elif isinstance(parsed, str): + return [parsed.strip()] + except (json.JSONDecodeError, TypeError): + pass + return [origin.strip() for origin in v.split(",") if origin.strip()] + if isinstance(v, list): + return [str(item).strip() for item in v] + return [] + + @property + def database_url(self) -> str: + from urllib.parse import quote_plus + password = quote_plus(self.db_password) + return f"postgresql+psycopg2://{self.db_user}:{password}@{self.db_host}:{self.db_port}/{self.db_name}" + + @property + def async_database_url(self) -> str: + from urllib.parse import quote_plus + password = quote_plus(self.db_password) + return f"postgresql+asyncpg://{self.db_user}:{password}@{self.db_host}:{self.db_port}/{self.db_name}" + + @property + def redis_url(self) -> str: + if self.redis_password: + return f"redis://:{self.redis_password}@{self.redis_host}:{self.redis_port}/{self.redis_db}" + return f"redis://{self.redis_host}:{self.redis_port}/{self.redis_db}" + + @property + def is_production(self) -> bool: + return self.app_env == "production" + + @property + def storage_max_file_size_bytes(self) -> int: + return self.storage_max_file_size_mb * 1024 * 1024 + + +settings = Settings() diff --git a/docengine/app/core/database.py b/docengine/app/core/database.py new file mode 100644 index 0000000..cd4ae69 --- /dev/null +++ b/docengine/app/core/database.py @@ -0,0 +1,88 @@ +from __future__ import annotations + +from collections.abc import Generator +from contextlib import contextmanager + +from sqlalchemy import MetaData, create_engine, event, text +from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker + +from app.core.config import settings + +NAMING_CONVENTION = { + "ix": "ix_%(column_0_label)s", + "uq": "uq_%(table_name)s_%(column_0_name)s", + "ck": "ck_%(table_name)s_%(constraint_name)s", + "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s", + "pk": "pk_%(table_name)s", +} + +metadata = MetaData( + naming_convention=NAMING_CONVENTION, + schema=settings.db_schema, +) + +engine = create_engine( + settings.database_url, + pool_size=settings.db_pool_size, + max_overflow=settings.db_max_overflow, + echo=settings.db_echo, + pool_pre_ping=True, + pool_recycle=3600, + connect_args={ + "options": f"-c search_path={settings.db_schema},public" + }, +) + + +@event.listens_for(engine, "connect") +def set_search_path(dbapi_connection: object, connection_record: object) -> None: + cursor = dbapi_connection.cursor() # type: ignore[union-attr] + cursor.execute(f"SET search_path TO {settings.db_schema}, public") + cursor.close() + dbapi_connection.commit() # type: ignore[union-attr] + + +SessionLocal = sessionmaker( + autocommit=False, + autoflush=False, + bind=engine, +) + + +class Base(DeclarativeBase): + """Base class for all SQLAlchemy models.""" + + metadata = metadata + + +def get_db() -> Generator[Session, None, None]: + """Dependency to get database session.""" + db = SessionLocal() + try: + yield db + finally: + db.close() + + +@contextmanager +def get_db_context() -> Generator[Session, None, None]: + """Context manager for database session (used outside request scope).""" + db = SessionLocal() + try: + yield db + db.commit() + except Exception: + db.rollback() + raise + finally: + db.close() + + +def check_database_connection() -> bool: + """Verify database connectivity.""" + try: + with engine.connect() as conn: + conn.execute(text("SELECT 1")) + return True + except Exception: + return False diff --git a/docengine/app/core/dependencies.py b/docengine/app/core/dependencies.py new file mode 100644 index 0000000..5ecc07e --- /dev/null +++ b/docengine/app/core/dependencies.py @@ -0,0 +1,143 @@ +from __future__ import annotations + +from typing import Annotated + +from fastapi import Depends, HTTPException, status +from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer +from sqlalchemy.orm import Session + +from app.core.database import get_db +from app.core.security import InvalidTokenError, decode_token +from app.models.user import User +from app.repositories.user_repository import UserRepository + +import uuid +from app.core.config import settings + +security_scheme = HTTPBearer(auto_error=False) + + +def get_current_user( + credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security_scheme)], + db: Annotated[Session, Depends(get_db)], +) -> User: + """Extract and validate the current user from the JWT token.""" + if credentials is None: + if settings.app_env == "development": + # Auto-login as default dev admin user if no token provided in dev + user_repo = UserRepository(db) + user = db.query(User).first() + if user: + return user + dev_user = User( + id=uuid.uuid4(), + username="dev_admin", + email="admin@docengine.local", + hashed_password="mock_password", + is_active=True, + is_superuser=True, + ) + db.add(dev_user) + db.commit() + db.refresh(dev_user) + return dev_user + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Not authenticated", + headers={"WWW-Authenticate": "Bearer"}, + ) + + try: + payload = decode_token(credentials.credentials) + except InvalidTokenError: + if settings.app_env == "development": + # Fallback to dev admin user on token decode failure in dev + user_repo = UserRepository(db) + user = db.query(User).first() + if user: + return user + dev_user = User( + id=uuid.uuid4(), + username="dev_admin", + email="admin@docengine.local", + hashed_password="mock_password", + is_active=True, + is_superuser=True, + ) + db.add(dev_user) + db.commit() + db.refresh(dev_user) + return dev_user + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid or expired token", + headers={"WWW-Authenticate": "Bearer"}, + ) + + token_type = payload.get("type") + if token_type != "access": + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid token type. Access token required.", + headers={"WWW-Authenticate": "Bearer"}, + ) + + user_id: str | None = payload.get("sub") + if user_id is None: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Token payload missing subject", + headers={"WWW-Authenticate": "Bearer"}, + ) + + user_repo = UserRepository(db) + user = user_repo.get_by_id(user_id) + if user is None: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="User not found", + headers={"WWW-Authenticate": "Bearer"}, + ) + + if not user.is_active: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="User account is deactivated", + ) + + return user + + +def get_current_active_user( + current_user: Annotated[User, Depends(get_current_user)], +) -> User: + """Ensure the current user is active.""" + if not current_user.is_active: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="User account is deactivated", + ) + return current_user + + +def require_role(required_roles: list[str]): # noqa: ANN201 + """Dependency factory to require specific roles.""" + + def role_checker( + current_user: Annotated[User, Depends(get_current_user)], + ) -> User: + user_roles = {role.name for role in current_user.roles} + if not user_roles.intersection(required_roles): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail=f"One of the following roles required: {', '.join(required_roles)}", + ) + return current_user + + return role_checker + + +CurrentUser = Annotated[User, Depends(get_current_user)] +ActiveUser = Annotated[User, Depends(get_current_active_user)] +AdminUser = Annotated[User, Depends(require_role(["admin"]))] +DBSession = Annotated[Session, Depends(get_db)] diff --git a/docengine/app/core/exceptions.py b/docengine/app/core/exceptions.py new file mode 100644 index 0000000..f0f5eda --- /dev/null +++ b/docengine/app/core/exceptions.py @@ -0,0 +1,106 @@ +from __future__ import annotations + +from typing import Any + + +class DocEngineException(Exception): + """Base exception for DocEngine application.""" + + def __init__(self, detail: str, status_code: int = 500, extra: dict[str, Any] | None = None) -> None: + self.detail = detail + self.status_code = status_code + self.extra = extra or {} + super().__init__(self.detail) + + +class NotFoundError(DocEngineException): + """Resource not found.""" + + def __init__(self, resource: str, identifier: str) -> None: + super().__init__( + detail=f"{resource} with identifier '{identifier}' not found", + status_code=404, + ) + self.resource = resource + self.identifier = identifier + + +class DuplicateError(DocEngineException): + """Resource already exists.""" + + def __init__(self, resource: str, field: str, value: str) -> None: + super().__init__( + detail=f"{resource} with {field} '{value}' already exists", + status_code=409, + ) + + +class ValidationError(DocEngineException): + """Input validation error.""" + + def __init__(self, detail: str, errors: list[dict[str, Any]] | None = None) -> None: + super().__init__(detail=detail, status_code=422) + self.errors = errors or [] + + +class AuthenticationError(DocEngineException): + """Authentication failed.""" + + def __init__(self, detail: str = "Authentication failed") -> None: + super().__init__(detail=detail, status_code=401) + + +class AuthorizationError(DocEngineException): + """Authorization failed.""" + + def __init__(self, detail: str = "Insufficient permissions") -> None: + super().__init__(detail=detail, status_code=403) + + +class StorageError(DocEngineException): + """Storage operation failed.""" + + def __init__(self, detail: str) -> None: + super().__init__(detail=detail, status_code=500) + + +class ProcessingError(DocEngineException): + """Document processing failed.""" + + def __init__(self, detail: str, document_id: str | None = None) -> None: + super().__init__(detail=detail, status_code=500) + self.document_id = document_id + + +class TemplateError(DocEngineException): + """Template operation failed.""" + + def __init__(self, detail: str) -> None: + super().__init__(detail=detail, status_code=500) + + +class RateLimitError(DocEngineException): + """Rate limit exceeded.""" + + def __init__(self, detail: str = "Rate limit exceeded. Please try again later.") -> None: + super().__init__(detail=detail, status_code=429) + + +class FileSizeError(DocEngineException): + """File exceeds maximum allowed size.""" + + def __init__(self, max_size_mb: int) -> None: + super().__init__( + detail=f"File size exceeds maximum allowed size of {max_size_mb}MB", + status_code=413, + ) + + +class UnsupportedFileTypeError(DocEngineException): + """File type not supported.""" + + def __init__(self, file_type: str) -> None: + super().__init__( + detail=f"File type '{file_type}' is not supported. Supported types: jpg, jpeg, png, tiff, pdf", + status_code=415, + ) diff --git a/docengine/app/core/logging_config.py b/docengine/app/core/logging_config.py new file mode 100644 index 0000000..dd4904e --- /dev/null +++ b/docengine/app/core/logging_config.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import logging +import sys + +import structlog + +from app.core.config import settings + + +def setup_logging() -> None: + """Configure structlog for structured JSON logging.""" + shared_processors: list[structlog.types.Processor] = [ + structlog.contextvars.merge_contextvars, + structlog.stdlib.add_logger_name, + structlog.stdlib.add_log_level, + structlog.stdlib.PositionalArgumentsFormatter(), + structlog.processors.TimeStamper(fmt="iso"), + structlog.processors.StackInfoRenderer(), + structlog.processors.UnicodeDecoder(), + ] + + if settings.log_format == "json": + renderer: structlog.types.Processor = structlog.processors.JSONRenderer() + else: + renderer = structlog.dev.ConsoleRenderer(colors=True) + + structlog.configure( + processors=[ + *shared_processors, + structlog.stdlib.ProcessorFormatter.wrap_for_formatter, + ], + logger_factory=structlog.stdlib.LoggerFactory(), + wrapper_class=structlog.stdlib.BoundLogger, + cache_logger_on_first_use=True, + ) + + formatter = structlog.stdlib.ProcessorFormatter( + processors=[ + structlog.stdlib.ProcessorFormatter.remove_processors_meta, + renderer, + ], + foreign_pre_chain=shared_processors, + ) + + handler = logging.StreamHandler(sys.stdout) + handler.setFormatter(formatter) + + root_logger = logging.getLogger() + root_logger.handlers.clear() + root_logger.addHandler(handler) + root_logger.setLevel(getattr(logging, settings.log_level.upper(), logging.INFO)) + + # Reduce noise from third-party libraries + for logger_name in ("uvicorn.access", "sqlalchemy.engine", "celery"): + logging.getLogger(logger_name).setLevel(logging.WARNING) + + +def get_logger(name: str | None = None) -> structlog.stdlib.BoundLogger: + """Get a structlog logger instance.""" + return structlog.get_logger(name) diff --git a/docengine/app/core/security.py b/docengine/app/core/security.py new file mode 100644 index 0000000..6c080da --- /dev/null +++ b/docengine/app/core/security.py @@ -0,0 +1,66 @@ +from __future__ import annotations + +import uuid +from datetime import UTC, datetime, timedelta +from typing import Any + +from jose import JWTError, jwt +from passlib.context import CryptContext + +from app.core.config import settings +from app.core.aes import get_string_value + +pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto") + + +def hash_password(password: str) -> str: + """Hash a password using bcrypt.""" + return pwd_context.hash(password) + + +def verify_password(plain_password: str, hashed_password: str) -> bool: + """Verify a plain password against a hashed password.""" + return pwd_context.verify(plain_password, hashed_password) + + +def create_access_token(data: dict[str, Any], expires_delta: timedelta | None = None) -> str: + """Create a JWT access token.""" + to_encode = data.copy() + expire = datetime.now(UTC) + (expires_delta or timedelta(minutes=settings.jwt_access_token_expire_minutes)) + to_encode.update({"exp": expire, "type": "access"}) + return jwt.encode(to_encode, settings.jwt_secret_key, algorithm=settings.jwt_algorithm) + + +def create_refresh_token(data: dict[str, Any], expires_delta: timedelta | None = None) -> str: + """Create a JWT refresh token.""" + to_encode = data.copy() + expire = datetime.now(UTC) + (expires_delta or timedelta(days=settings.jwt_refresh_token_expire_days)) + to_encode.update({ + "exp": expire, + "type": "refresh", + "jti": str(uuid.uuid4()), + }) + return jwt.encode(to_encode, settings.jwt_secret_key, algorithm=settings.jwt_algorithm) + + +def decode_token(token: str) -> dict[str, Any]: + """Decode and validate a JWT token.""" + decrypted_token = get_string_value( + token, + settings.session_encryption_secret, + settings.session_encryption_secret_internal + ) + final_token = decrypted_token if decrypted_token else token + try: + payload = jwt.decode(final_token, settings.jwt_secret_key, algorithms=[settings.jwt_algorithm]) + return payload + except JWTError as e: + raise InvalidTokenError(str(e)) from e + + +class InvalidTokenError(Exception): + """Raised when a JWT token is invalid or expired.""" + + def __init__(self, detail: str = "Invalid or expired token") -> None: + self.detail = detail + super().__init__(self.detail) diff --git a/docengine/app/domain/__init__.py b/docengine/app/domain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/events/__init__.py b/docengine/app/events/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/events/handlers.py b/docengine/app/events/handlers.py new file mode 100644 index 0000000..0ae2df1 --- /dev/null +++ b/docengine/app/events/handlers.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from app.core.logging_config import get_logger, setup_logging + +logger = get_logger(__name__) + + +def on_startup() -> None: + """Application startup event handler.""" + setup_logging() + logger.info("application_starting", phase="startup") + + # Ensure storage directories exist + from app.storage.provider import get_storage_provider + try: + get_storage_provider() + logger.info("storage_initialized") + except Exception as e: + logger.error("storage_init_failed", error=str(e)) + + # Verify database connection + from app.core.database import check_database_connection + if check_database_connection(): + logger.info("database_connected") + else: + logger.error("database_connection_failed") + + logger.info("application_started", phase="startup_complete") + + +def on_shutdown() -> None: + """Application shutdown event handler.""" + logger.info("application_shutting_down", phase="shutdown") + + # Cleanup resources + from app.core.database import engine + engine.dispose() + + logger.info("application_stopped", phase="shutdown_complete") diff --git a/docengine/app/infrastructure/__init__.py b/docengine/app/infrastructure/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/main.py b/docengine/app/main.py new file mode 100644 index 0000000..683fcbc --- /dev/null +++ b/docengine/app/main.py @@ -0,0 +1,107 @@ +from __future__ import annotations + +from contextlib import asynccontextmanager +from collections.abc import AsyncGenerator +from typing import Any + +from fastapi import FastAPI, Request, status +from fastapi.exceptions import RequestValidationError +from fastapi.responses import JSONResponse + +from app.api.router import api_v1_router +from app.core.config import settings +from app.core.exceptions import DocEngineException +from app.events.handlers import on_shutdown, on_startup +from app.middleware.audit import AuditMiddleware +from app.middleware.cors import setup_cors +from app.middleware.metrics import setup_metrics +from app.middleware.rate_limit import RateLimitMiddleware + + +@asynccontextmanager +async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: + """Application lifespan manager.""" + on_startup() + yield + on_shutdown() + + +app = FastAPI( + title=settings.app_name, + description="Document Template Recognition and Reconstruction System", + version=settings.app_version, + docs_url="/docs", + redoc_url="/redoc", + openapi_url="/openapi.json", + lifespan=lifespan, +) + +# Setup middleware (order matters: last added = first executed) +app.add_middleware(AuditMiddleware) +app.add_middleware(RateLimitMiddleware) +setup_cors(app) + +# Setup Prometheus metrics +setup_metrics(app) + +# Include API routes +app.include_router(api_v1_router) + + +# Exception handlers +@app.exception_handler(DocEngineException) +async def docengine_exception_handler(request: Request, exc: DocEngineException) -> JSONResponse: + """Handle application-specific exceptions.""" + return JSONResponse( + status_code=exc.status_code, + content={ + "detail": exc.detail, + "error_code": type(exc).__name__, + "extra": exc.extra if exc.extra else None, + }, + ) + + +@app.exception_handler(RequestValidationError) +async def validation_exception_handler(request: Request, exc: RequestValidationError) -> JSONResponse: + """Handle request validation errors.""" + errors = [] + for error in exc.errors(): + errors.append({ + "field": ".".join(str(loc) for loc in error.get("loc", [])), + "message": error.get("msg", ""), + "type": error.get("type", ""), + }) + + return JSONResponse( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + content={ + "detail": "Request validation failed", + "error_code": "ValidationError", + "errors": errors, + }, + ) + + +@app.exception_handler(Exception) +async def general_exception_handler(request: Request, exc: Exception) -> JSONResponse: + """Handle unexpected exceptions.""" + return JSONResponse( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + content={ + "detail": "An unexpected error occurred" if settings.is_production else str(exc), + "error_code": "InternalServerError", + }, + ) + + +if __name__ == "__main__": + import uvicorn + + uvicorn.run( + "app.main:app", + host=settings.app_host, + port=settings.app_port, + reload=not settings.is_production, + workers=1 if settings.app_debug else settings.app_workers, + ) diff --git a/docengine/app/middleware/__init__.py b/docengine/app/middleware/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/middleware/audit.py b/docengine/app/middleware/audit.py new file mode 100644 index 0000000..d6fc14b --- /dev/null +++ b/docengine/app/middleware/audit.py @@ -0,0 +1,71 @@ +from __future__ import annotations + +import time +import uuid +from typing import Any + +from fastapi import Request, Response +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint + +from app.core.logging_config import get_logger + +logger = get_logger(__name__) + + +class AuditMiddleware(BaseHTTPMiddleware): + """Middleware to log all API requests for audit purposes.""" + + EXCLUDED_PATHS = {"/api/v1/health", "/metrics", "/docs", "/openapi.json", "/redoc"} + + async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response: + if request.url.path in self.EXCLUDED_PATHS: + return await call_next(request) + + request_id = str(uuid.uuid4()) + start_time = time.monotonic() + + # Extract client info + client_ip = request.client.host if request.client else "unknown" + user_agent = request.headers.get("user-agent", "unknown") + + # Add request ID to request state + request.state.request_id = request_id + + logger.info( + "request_started", + request_id=request_id, + method=request.method, + path=request.url.path, + client_ip=client_ip, + user_agent=user_agent[:200], + ) + + try: + response = await call_next(request) + duration_ms = (time.monotonic() - start_time) * 1000 + + logger.info( + "request_completed", + request_id=request_id, + method=request.method, + path=request.url.path, + status_code=response.status_code, + duration_ms=round(duration_ms, 2), + client_ip=client_ip, + ) + + response.headers["X-Request-ID"] = request_id + response.headers["X-Process-Time"] = f"{duration_ms:.2f}ms" + return response + + except Exception as exc: + duration_ms = (time.monotonic() - start_time) * 1000 + logger.exception( + "request_failed", + request_id=request_id, + method=request.method, + path=request.url.path, + duration_ms=round(duration_ms, 2), + error=str(exc), + ) + raise diff --git a/docengine/app/middleware/cors.py b/docengine/app/middleware/cors.py new file mode 100644 index 0000000..5c1146a --- /dev/null +++ b/docengine/app/middleware/cors.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware + +from app.core.config import settings + + +def setup_cors(app: FastAPI) -> None: + """Configure CORS middleware.""" + app.add_middleware( + CORSMiddleware, + allow_origins=settings.cors_origins, + allow_credentials=settings.cors_allow_credentials, + allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], + allow_headers=["*"], + expose_headers=[ + "X-Request-ID", + "X-Process-Time", + "X-RateLimit-Limit", + "X-RateLimit-Remaining", + "X-RateLimit-Reset", + ], + max_age=600, + ) diff --git a/docengine/app/middleware/metrics.py b/docengine/app/middleware/metrics.py new file mode 100644 index 0000000..f810f74 --- /dev/null +++ b/docengine/app/middleware/metrics.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +from fastapi import FastAPI +from prometheus_fastapi_instrumentator import Instrumentator + +from app.core.config import settings + + +def setup_metrics(app: FastAPI) -> None: + """Configure Prometheus metrics instrumentation.""" + if not settings.prometheus_enabled: + return + + instrumentator = Instrumentator( + should_group_status_codes=True, + should_ignore_untemplated=True, + should_respect_env_var=False, + excluded_handlers=["/metrics", "/api/v1/health", "/docs", "/openapi.json"], + env_var_name="PROMETHEUS_ENABLED", + inprogress_name="docengine_inprogress_requests", + inprogress_labels=True, + ) + + instrumentator.instrument(app).expose( + app, + endpoint="/metrics", + include_in_schema=False, + should_gzip=True, + ) diff --git a/docengine/app/middleware/rate_limit.py b/docengine/app/middleware/rate_limit.py new file mode 100644 index 0000000..a50f24a --- /dev/null +++ b/docengine/app/middleware/rate_limit.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +import time +from collections import defaultdict + +from fastapi import Request, Response, status +from fastapi.responses import JSONResponse +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint + +from app.core.config import settings +from app.core.logging_config import get_logger + +logger = get_logger(__name__) + + +class RateLimitMiddleware(BaseHTTPMiddleware): + """Token bucket rate limiter per client IP.""" + + EXCLUDED_PATHS = {"/api/v1/health", "/metrics", "/docs", "/openapi.json", "/redoc"} + + def __init__(self, app, max_requests: int | None = None, window_seconds: int | None = None) -> None: # noqa: ANN001 + super().__init__(app) + self.max_requests = max_requests or settings.rate_limit_requests + self.window_seconds = window_seconds or settings.rate_limit_window_seconds + self._requests: dict[str, list[float]] = defaultdict(list) + + def _clean_old_requests(self, client_ip: str, now: float) -> None: + """Remove requests outside the current window.""" + cutoff = now - self.window_seconds + self._requests[client_ip] = [ + ts for ts in self._requests[client_ip] if ts > cutoff + ] + + async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response: + if request.url.path in self.EXCLUDED_PATHS: + return await call_next(request) + + client_ip = request.client.host if request.client else "unknown" + now = time.monotonic() + + self._clean_old_requests(client_ip, now) + + if len(self._requests[client_ip]) >= self.max_requests: + logger.warning( + "rate_limit_exceeded", + client_ip=client_ip, + path=request.url.path, + request_count=len(self._requests[client_ip]), + ) + return JSONResponse( + status_code=status.HTTP_429_TOO_MANY_REQUESTS, + content={ + "detail": "Rate limit exceeded. Please try again later.", + "retry_after_seconds": self.window_seconds, + }, + headers={ + "Retry-After": str(self.window_seconds), + "X-RateLimit-Limit": str(self.max_requests), + "X-RateLimit-Remaining": "0", + "X-RateLimit-Reset": str(int(now + self.window_seconds)), + }, + ) + + self._requests[client_ip].append(now) + remaining = self.max_requests - len(self._requests[client_ip]) + + response = await call_next(request) + response.headers["X-RateLimit-Limit"] = str(self.max_requests) + response.headers["X-RateLimit-Remaining"] = str(remaining) + response.headers["X-RateLimit-Reset"] = str(int(now + self.window_seconds)) + + return response diff --git a/docengine/app/models/__init__.py b/docengine/app/models/__init__.py new file mode 100644 index 0000000..9d54a42 --- /dev/null +++ b/docengine/app/models/__init__.py @@ -0,0 +1,43 @@ +from app.models.user import AuditLog, RefreshToken, Role, User, user_roles_table +from app.models.document import ( + Document, + DocumentImage, + DocumentPage, + DocumentTable, + DocumentTextBlock, + TemplateMatch, +) +from app.models.template import ( + DocumentCell, + DocumentFormat, + DocumentRegion, + ImageRegion, + TableColumn, + TableFormat, + TableRow, + TemplateFingerprint, + Watermark, +) + +__all__ = [ + "User", + "Role", + "RefreshToken", + "AuditLog", + "user_roles_table", + "Document", + "DocumentPage", + "DocumentTextBlock", + "DocumentImage", + "DocumentTable", + "TemplateMatch", + "DocumentFormat", + "DocumentCell", + "DocumentRegion", + "TableFormat", + "TableColumn", + "TableRow", + "Watermark", + "ImageRegion", + "TemplateFingerprint", +] diff --git a/docengine/app/models/base.py b/docengine/app/models/base.py new file mode 100644 index 0000000..42dd001 --- /dev/null +++ b/docengine/app/models/base.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import uuid +from datetime import UTC, datetime + +from sqlalchemy import DateTime, func +from sqlalchemy.dialects.postgresql import UUID +from sqlalchemy.orm import Mapped, mapped_column + +from app.core.database import Base + + +class TimestampMixin: + """Mixin providing created_at and updated_at timestamps.""" + + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=lambda: datetime.now(UTC), + server_default=func.now(), + nullable=False, + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=lambda: datetime.now(UTC), + server_default=func.now(), + onupdate=lambda: datetime.now(UTC), + nullable=False, + ) + + +class UUIDPrimaryKeyMixin: + """Mixin providing a UUID primary key.""" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + primary_key=True, + default=uuid.uuid4, + nullable=False, + ) diff --git a/docengine/app/models/document.py b/docengine/app/models/document.py new file mode 100644 index 0000000..cd89824 --- /dev/null +++ b/docengine/app/models/document.py @@ -0,0 +1,244 @@ +from __future__ import annotations + +import uuid +from datetime import datetime + +from sqlalchemy import BigInteger, Boolean, DateTime, Float, ForeignKey, Integer, String, Text, func +from sqlalchemy.dialects.postgresql import JSONB, UUID +from sqlalchemy.orm import Mapped, mapped_column, relationship + +from app.core.database import Base +from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin + + +class Document(Base, UUIDPrimaryKeyMixin, TimestampMixin): + """Uploaded document record.""" + + __tablename__ = "documents" + + filename: Mapped[str] = mapped_column(String(500), nullable=False) + original_filename: Mapped[str] = mapped_column(String(500), nullable=False) + content_type: Mapped[str] = mapped_column(String(100), nullable=False) + file_size: Mapped[int] = mapped_column(BigInteger, nullable=False) + checksum: Mapped[str] = mapped_column(String(128), nullable=False, index=True) + storage_path: Mapped[str] = mapped_column(String(1024), nullable=False) + status: Mapped[str] = mapped_column( + String(50), + default="pending", + nullable=False, + index=True, + ) + page_count: Mapped[int | None] = mapped_column(Integer, nullable=True) + is_scanned: Mapped[bool | None] = mapped_column(Boolean, nullable=True) + document_metadata: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + error_message: Mapped[str | None] = mapped_column(Text, nullable=True) + uploaded_by: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), + ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + index=True, + ) + + pages: Mapped[list[DocumentPage]] = relationship( + "DocumentPage", + back_populates="document", + cascade="all, delete-orphan", + order_by="DocumentPage.page_number", + lazy="selectin", + ) + template_matches: Mapped[list[TemplateMatch]] = relationship( + "TemplateMatch", + back_populates="document", + cascade="all, delete-orphan", + lazy="dynamic", + ) + + def __repr__(self) -> str: + return f"" + + +class DocumentPage(Base, UUIDPrimaryKeyMixin): + """Individual page within a document.""" + + __tablename__ = "document_pages" + + document_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("documents.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int] = mapped_column(Integer, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True) + text_content: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + document: Mapped[Document] = relationship("Document", back_populates="pages") + text_blocks: Mapped[list[DocumentTextBlock]] = relationship( + "DocumentTextBlock", + back_populates="page", + cascade="all, delete-orphan", + order_by="DocumentTextBlock.sequence", + lazy="selectin", + ) + images: Mapped[list[DocumentImage]] = relationship( + "DocumentImage", + back_populates="page", + cascade="all, delete-orphan", + lazy="selectin", + ) + tables: Mapped[list[DocumentTable]] = relationship( + "DocumentTable", + back_populates="page", + cascade="all, delete-orphan", + lazy="selectin", + ) + + def __repr__(self) -> str: + return f"" + + +class DocumentTextBlock(Base, UUIDPrimaryKeyMixin): + """Extracted text block from a document page.""" + + __tablename__ = "document_text_blocks" + + page_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_pages.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + text: Mapped[str] = mapped_column(Text, nullable=False) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + confidence: Mapped[float | None] = mapped_column(Float, nullable=True) + font_family: Mapped[str | None] = mapped_column(String(255), nullable=True) + font_size: Mapped[float | None] = mapped_column(Float, nullable=True) + font_color: Mapped[str | None] = mapped_column(String(50), nullable=True) + font_style: Mapped[str | None] = mapped_column(String(50), nullable=True) + block_type: Mapped[str] = mapped_column( + String(50), + default="text", + nullable=False, + ) + sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="text_blocks") + + def __repr__(self) -> str: + return f"" + + +class DocumentImage(Base, UUIDPrimaryKeyMixin): + """Extracted image from a document page.""" + + __tablename__ = "document_images" + + page_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_pages.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + image_path: Mapped[str] = mapped_column(String(1024), nullable=False) + image_type: Mapped[str] = mapped_column( + String(50), + default="figure", + nullable=False, + ) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="images") + + def __repr__(self) -> str: + return f"" + + +class DocumentTable(Base, UUIDPrimaryKeyMixin): + """Extracted table from a document page.""" + + __tablename__ = "document_tables" + + page_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_pages.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + rows: Mapped[int] = mapped_column(Integer, nullable=False) + columns: Mapped[int] = mapped_column(Integer, nullable=False) + data: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + page: Mapped[DocumentPage] = relationship("DocumentPage", back_populates="tables") + + def __repr__(self) -> str: + return f"" + + +class TemplateMatch(Base, UUIDPrimaryKeyMixin): + """Template matching result for a document.""" + + __tablename__ = "template_matches" + + document_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("documents.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + confidence_score: Mapped[float] = mapped_column(Float, nullable=False) + match_details: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + selected: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + document: Mapped[Document] = relationship("Document", back_populates="template_matches") + template: Mapped[DocumentFormat] = relationship("DocumentFormat") + + def __repr__(self) -> str: + return f"" diff --git a/docengine/app/models/template.py b/docengine/app/models/template.py new file mode 100644 index 0000000..e1c26fc --- /dev/null +++ b/docengine/app/models/template.py @@ -0,0 +1,378 @@ +from __future__ import annotations + +import uuid +from datetime import datetime + +from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Integer, String, Text, func +from sqlalchemy.dialects.postgresql import JSONB, UUID +from sqlalchemy.orm import Mapped, mapped_column, relationship + +from app.core.database import Base +from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin + + +class DocumentFormat(Base, UUIDPrimaryKeyMixin, TimestampMixin): + """Reusable document template format.""" + + __tablename__ = "document_formats" + + name: Mapped[str] = mapped_column(String(255), nullable=False, index=True) + description: Mapped[str | None] = mapped_column(Text, nullable=True) + page_width: Mapped[float] = mapped_column(Float, nullable=False) + page_height: Mapped[float] = mapped_column(Float, nullable=False) + page_count: Mapped[int] = mapped_column(Integer, default=1, nullable=False) + margin_top: Mapped[float] = mapped_column(Float, default=72.0, nullable=False) + margin_right: Mapped[float] = mapped_column(Float, default=72.0, nullable=False) + margin_bottom: Mapped[float] = mapped_column(Float, default=72.0, nullable=False) + margin_left: Mapped[float] = mapped_column(Float, default=72.0, nullable=False) + fingerprint: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + source_document_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), + ForeignKey("documents.id", ondelete="SET NULL"), + nullable=True, + index=True, + ) + version: Mapped[int] = mapped_column(Integer, default=1, nullable=False) + is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True) + created_by: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), + ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + ) + + cells: Mapped[list[DocumentCell]] = relationship( + "DocumentCell", + back_populates="format", + cascade="all, delete-orphan", + order_by="DocumentCell.sequence", + lazy="selectin", + ) + regions: Mapped[list[DocumentRegion]] = relationship( + "DocumentRegion", + back_populates="format", + cascade="all, delete-orphan", + order_by="DocumentRegion.sequence", + lazy="selectin", + ) + table_formats: Mapped[list[TableFormat]] = relationship( + "TableFormat", + back_populates="format", + cascade="all, delete-orphan", + lazy="selectin", + ) + watermarks: Mapped[list[Watermark]] = relationship( + "Watermark", + back_populates="format", + cascade="all, delete-orphan", + lazy="selectin", + ) + image_regions: Mapped[list[ImageRegion]] = relationship( + "ImageRegion", + back_populates="format", + cascade="all, delete-orphan", + lazy="selectin", + ) + fingerprint_record: Mapped[TemplateFingerprint | None] = relationship( + "TemplateFingerprint", + back_populates="format", + uselist=False, + cascade="all, delete-orphan", + lazy="selectin", + ) + + def __repr__(self) -> str: + return f"" + + +class DocumentCell(Base, UUIDPrimaryKeyMixin): + """Cell definition within a document template.""" + + __tablename__ = "document_cells" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int] = mapped_column(Integer, nullable=False) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + row_no: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + column_no: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + data_type: Mapped[str] = mapped_column(String(50), default="text", nullable=False) + font_family: Mapped[str | None] = mapped_column(String(255), nullable=True) + font_size: Mapped[float | None] = mapped_column(Float, nullable=True) + font_style: Mapped[str | None] = mapped_column(String(50), nullable=True) + font_color: Mapped[str | None] = mapped_column(String(50), nullable=True) + background_color: Mapped[str | None] = mapped_column(String(50), nullable=True) + border_top: Mapped[str | None] = mapped_column(String(100), nullable=True) + border_right: Mapped[str | None] = mapped_column(String(100), nullable=True) + border_bottom: Mapped[str | None] = mapped_column(String(100), nullable=True) + border_left: Mapped[str | None] = mapped_column(String(100), nullable=True) + padding_top: Mapped[float] = mapped_column(Float, default=0.0, nullable=False) + padding_right: Mapped[float] = mapped_column(Float, default=0.0, nullable=False) + padding_bottom: Mapped[float] = mapped_column(Float, default=0.0, nullable=False) + padding_left: Mapped[float] = mapped_column(Float, default=0.0, nullable=False) + alignment: Mapped[str] = mapped_column(String(20), default="left", nullable=False) + vertical_alignment: Mapped[str] = mapped_column(String(20), default="top", nullable=False) + rowspan: Mapped[int] = mapped_column(Integer, default=1, nullable=False) + colspan: Mapped[int] = mapped_column(Integer, default=1, nullable=False) + static_text: Mapped[str | None] = mapped_column(Text, nullable=True) + field_name: Mapped[str | None] = mapped_column(String(255), nullable=True) + sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + is_dynamic: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="cells") + + def __repr__(self) -> str: + return f"" + + +class DocumentRegion(Base, UUIDPrimaryKeyMixin): + """Region definition within a document template.""" + + __tablename__ = "document_regions" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int] = mapped_column(Integer, nullable=False) + region_type: Mapped[str] = mapped_column(String(50), nullable=False) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + content: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + sequence: Mapped[int] = mapped_column(Integer, default=0, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="regions") + + def __repr__(self) -> str: + return f"" + + +class TableFormat(Base, UUIDPrimaryKeyMixin): + """Table definition within a document template.""" + + __tablename__ = "table_formats" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int] = mapped_column(Integer, nullable=False) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + rows: Mapped[int] = mapped_column(Integer, nullable=False) + columns: Mapped[int] = mapped_column(Integer, nullable=False) + border_style: Mapped[str] = mapped_column(String(50), default="solid", nullable=False) + border_width: Mapped[float] = mapped_column(Float, default=1.0, nullable=False) + border_color: Mapped[str] = mapped_column(String(50), default="#000000", nullable=False) + header_rows: Mapped[int] = mapped_column(Integer, default=1, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="table_formats") + table_columns: Mapped[list[TableColumn]] = relationship( + "TableColumn", + back_populates="table_format", + cascade="all, delete-orphan", + order_by="TableColumn.column_index", + lazy="selectin", + ) + table_rows: Mapped[list[TableRow]] = relationship( + "TableRow", + back_populates="table_format", + cascade="all, delete-orphan", + order_by="TableRow.row_index", + lazy="selectin", + ) + + def __repr__(self) -> str: + return f"" + + +class TableColumn(Base, UUIDPrimaryKeyMixin): + """Column definition within a table format.""" + + __tablename__ = "table_columns" + + table_format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("table_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + column_index: Mapped[int] = mapped_column(Integer, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + header_text: Mapped[str | None] = mapped_column(String(500), nullable=True) + data_type: Mapped[str] = mapped_column(String(50), default="text", nullable=False) + alignment: Mapped[str] = mapped_column(String(20), default="left", nullable=False) + font_family: Mapped[str | None] = mapped_column(String(255), nullable=True) + font_size: Mapped[float | None] = mapped_column(Float, nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + table_format: Mapped[TableFormat] = relationship("TableFormat", back_populates="table_columns") + + def __repr__(self) -> str: + return f"" + + +class TableRow(Base, UUIDPrimaryKeyMixin): + """Row definition within a table format.""" + + __tablename__ = "table_rows" + + table_format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("table_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + row_index: Mapped[int] = mapped_column(Integer, nullable=False) + height: Mapped[float] = mapped_column(Float, default=20.0, nullable=False) + is_header: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) + background_color: Mapped[str | None] = mapped_column(String(50), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + table_format: Mapped[TableFormat] = relationship("TableFormat", back_populates="table_rows") + + def __repr__(self) -> str: + return f"" + + +class Watermark(Base, UUIDPrimaryKeyMixin): + """Watermark definition within a document template.""" + + __tablename__ = "watermarks" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int | None] = mapped_column(Integer, nullable=True) + text: Mapped[str | None] = mapped_column(String(500), nullable=True) + image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + opacity: Mapped[float] = mapped_column(Float, default=0.3, nullable=False) + rotation: Mapped[float] = mapped_column(Float, default=0.0, nullable=False) + font_family: Mapped[str | None] = mapped_column(String(255), nullable=True) + font_size: Mapped[float | None] = mapped_column(Float, nullable=True) + font_color: Mapped[str | None] = mapped_column(String(50), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="watermarks") + + def __repr__(self) -> str: + return f"" + + +class ImageRegion(Base, UUIDPrimaryKeyMixin): + """Image region within a document template.""" + + __tablename__ = "image_regions" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + page_number: Mapped[int] = mapped_column(Integer, nullable=False) + x: Mapped[float] = mapped_column(Float, nullable=False) + y: Mapped[float] = mapped_column(Float, nullable=False) + width: Mapped[float] = mapped_column(Float, nullable=False) + height: Mapped[float] = mapped_column(Float, nullable=False) + image_path: Mapped[str | None] = mapped_column(String(1024), nullable=True) + image_type: Mapped[str] = mapped_column(String(50), default="figure", nullable=False) + is_static: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False) + field_name: Mapped[str | None] = mapped_column(String(255), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="image_regions") + + def __repr__(self) -> str: + return f"" + + +class TemplateFingerprint(Base, UUIDPrimaryKeyMixin): + """Layout fingerprint for template matching.""" + + __tablename__ = "template_fingerprints" + + format_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("document_formats.id", ondelete="CASCADE"), + nullable=False, + unique=True, + index=True, + ) + page_dimensions: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + logo_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + header_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + footer_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + table_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + cell_coordinates: Mapped[dict | None] = mapped_column(JSONB, nullable=True) + fingerprint_hash: Mapped[str] = mapped_column(String(256), nullable=False, index=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + format: Mapped[DocumentFormat] = relationship("DocumentFormat", back_populates="fingerprint_record") + + def __repr__(self) -> str: + return f"" diff --git a/docengine/app/models/user.py b/docengine/app/models/user.py new file mode 100644 index 0000000..e0aa4de --- /dev/null +++ b/docengine/app/models/user.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import uuid +from datetime import datetime + +from sqlalchemy import Boolean, Column, DateTime, ForeignKey, String, Table, Text, func +from sqlalchemy.dialects.postgresql import UUID +from sqlalchemy.orm import Mapped, mapped_column, relationship + +from app.core.database import Base +from app.models.base import TimestampMixin, UUIDPrimaryKeyMixin + +user_roles_table = Table( + "user_roles", + Base.metadata, + Column("user_id", UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True), + Column("role_id", UUID(as_uuid=True), ForeignKey("roles.id", ondelete="CASCADE"), primary_key=True), +) + + +class User(Base, UUIDPrimaryKeyMixin, TimestampMixin): + """User account model.""" + + __tablename__ = "users" + + username: Mapped[str] = mapped_column(String(150), unique=True, nullable=False, index=True) + email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False, index=True) + hashed_password: Mapped[str] = mapped_column(String(255), nullable=False) + full_name: Mapped[str | None] = mapped_column(String(255), nullable=True) + is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False) + is_superuser: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) + last_login: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + + roles: Mapped[list[Role]] = relationship( + "Role", + secondary=user_roles_table, + back_populates="users", + lazy="joined", + ) + refresh_tokens: Mapped[list[RefreshToken]] = relationship( + "RefreshToken", + back_populates="user", + cascade="all, delete-orphan", + lazy="dynamic", + ) + audit_logs: Mapped[list[AuditLog]] = relationship( + "AuditLog", + back_populates="user", + lazy="dynamic", + ) + + def __repr__(self) -> str: + return f"" + + +class Role(Base, UUIDPrimaryKeyMixin): + """User role model.""" + + __tablename__ = "roles" + + name: Mapped[str] = mapped_column(String(50), unique=True, nullable=False, index=True) + description: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + users: Mapped[list[User]] = relationship( + "User", + secondary=user_roles_table, + back_populates="roles", + lazy="dynamic", + ) + + def __repr__(self) -> str: + return f"" + + +class RefreshToken(Base, UUIDPrimaryKeyMixin): + """JWT refresh token storage.""" + + __tablename__ = "refresh_tokens" + + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("users.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + token: Mapped[str] = mapped_column(String(512), unique=True, nullable=False, index=True) + expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) + revoked: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + ) + + user: Mapped[User] = relationship("User", back_populates="refresh_tokens") + + def __repr__(self) -> str: + return f"" + + +class AuditLog(Base, UUIDPrimaryKeyMixin): + """Audit trail for user actions.""" + + __tablename__ = "audit_logs" + + user_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), + ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + index=True, + ) + action: Mapped[str] = mapped_column(String(100), nullable=False, index=True) + resource_type: Mapped[str] = mapped_column(String(100), nullable=False, index=True) + resource_id: Mapped[str | None] = mapped_column(String(255), nullable=True) + details: Mapped[dict | None] = mapped_column(type_=Text, nullable=True) + ip_address: Mapped[str | None] = mapped_column(String(45), nullable=True) + user_agent: Mapped[str | None] = mapped_column(String(512), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + default=func.now(), + server_default=func.now(), + nullable=False, + index=True, + ) + + user: Mapped[User | None] = relationship("User", back_populates="audit_logs") + + def __repr__(self) -> str: + return f"" diff --git a/docengine/app/repositories/__init__.py b/docengine/app/repositories/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/repositories/base.py b/docengine/app/repositories/base.py new file mode 100644 index 0000000..464fdf4 --- /dev/null +++ b/docengine/app/repositories/base.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +import uuid +from typing import Any, Generic, TypeVar + +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.core.database import Base + +ModelType = TypeVar("ModelType", bound=Base) + + +class BaseRepository(Generic[ModelType]): + """Base repository with common CRUD operations.""" + + def __init__(self, db: Session, model: type[ModelType]) -> None: + self.db = db + self.model = model + + def get_by_id(self, entity_id: str | uuid.UUID) -> ModelType | None: + """Get an entity by its primary key.""" + if isinstance(entity_id, str): + entity_id = uuid.UUID(entity_id) + return self.db.get(self.model, entity_id) + + def get_all( + self, + offset: int = 0, + limit: int = 100, + filters: dict[str, Any] | None = None, + order_by: str | None = None, + order_desc: bool = False, + ) -> list[ModelType]: + """Get all entities with optional filtering, pagination, and ordering.""" + query = select(self.model) + + if filters: + for key, value in filters.items(): + if hasattr(self.model, key) and value is not None: + query = query.where(getattr(self.model, key) == value) + + if order_by and hasattr(self.model, order_by): + col = getattr(self.model, order_by) + query = query.order_by(col.desc() if order_desc else col.asc()) + + query = query.offset(offset).limit(limit) + result = self.db.execute(query) + return list(result.scalars().all()) + + def count(self, filters: dict[str, Any] | None = None) -> int: + """Count entities with optional filtering.""" + query = select(func.count()).select_from(self.model) + + if filters: + for key, value in filters.items(): + if hasattr(self.model, key) and value is not None: + query = query.where(getattr(self.model, key) == value) + + result = self.db.execute(query) + return result.scalar_one() + + def create(self, entity: ModelType) -> ModelType: + """Create a new entity.""" + self.db.add(entity) + self.db.flush() + self.db.refresh(entity) + return entity + + def create_many(self, entities: list[ModelType]) -> list[ModelType]: + """Create multiple entities.""" + self.db.add_all(entities) + self.db.flush() + for entity in entities: + self.db.refresh(entity) + return entities + + def update(self, entity: ModelType, update_data: dict[str, Any]) -> ModelType: + """Update an entity with given data.""" + for key, value in update_data.items(): + if hasattr(entity, key) and value is not None: + setattr(entity, key, value) + self.db.flush() + self.db.refresh(entity) + return entity + + def delete(self, entity: ModelType) -> None: + """Delete an entity.""" + self.db.delete(entity) + self.db.flush() + + def delete_by_id(self, entity_id: str | uuid.UUID) -> bool: + """Delete an entity by its ID. Returns True if deleted.""" + entity = self.get_by_id(entity_id) + if entity: + self.delete(entity) + return True + return False + + def exists(self, entity_id: str | uuid.UUID) -> bool: + """Check if an entity exists by ID.""" + if isinstance(entity_id, str): + entity_id = uuid.UUID(entity_id) + query = select(func.count()).select_from(self.model).where(self.model.id == entity_id) + result = self.db.execute(query) + return result.scalar_one() > 0 + + def commit(self) -> None: + """Commit the current transaction.""" + self.db.commit() + + def rollback(self) -> None: + """Rollback the current transaction.""" + self.db.rollback() diff --git a/docengine/app/repositories/document_repository.py b/docengine/app/repositories/document_repository.py new file mode 100644 index 0000000..4cf4d79 --- /dev/null +++ b/docengine/app/repositories/document_repository.py @@ -0,0 +1,335 @@ +from __future__ import annotations + +import uuid + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.document import ( + Document, + DocumentImage, + DocumentPage, + DocumentTable, + DocumentTextBlock, + TemplateMatch, +) +from app.repositories.base import BaseRepository + + +class DocumentRepository(BaseRepository[Document]): + """Repository for Document operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, Document) + + def get_by_checksum(self, checksum: str) -> Document | None: + """Get document by file checksum.""" + query = select(Document).where(Document.checksum == checksum) + result = self.db.execute(query) + return result.scalars().first() + + def get_by_status(self, status: str, offset: int = 0, limit: int = 100) -> list[Document]: + """Get documents by processing status.""" + query = ( + select(Document) + .where(Document.status == status) + .order_by(Document.created_at.desc()) + .offset(offset) + .limit(limit) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_user_documents( + self, + user_id: uuid.UUID, + offset: int = 0, + limit: int = 100, + ) -> list[Document]: + """Get documents uploaded by a specific user.""" + query = ( + select(Document) + .where(Document.uploaded_by == user_id) + .order_by(Document.created_at.desc()) + .offset(offset) + .limit(limit) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def update_status( + self, + document_id: uuid.UUID, + status: str, + error_message: str | None = None, + ) -> Document | None: + """Update document processing status.""" + document = self.get_by_id(document_id) + if document: + document.status = status + if error_message: + document.error_message = error_message + self.db.flush() + self.db.refresh(document) + return document + + def get_with_pages(self, document_id: uuid.UUID) -> Document | None: + """Get document with all pages eagerly loaded.""" + return self.get_by_id(document_id) + + def get_pending_documents(self, limit: int = 10) -> list[Document]: + """Get pending documents for processing.""" + return self.get_by_status("pending", limit=limit) + + +class DocumentPageRepository(BaseRepository[DocumentPage]): + """Repository for DocumentPage operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentPage) + + def get_document_pages(self, document_id: uuid.UUID) -> list[DocumentPage]: + """Get all pages for a document ordered by page number.""" + query = ( + select(DocumentPage) + .where(DocumentPage.document_id == document_id) + .order_by(DocumentPage.page_number) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_page_by_number(self, document_id: uuid.UUID, page_number: int) -> DocumentPage | None: + """Get a specific page by document ID and page number.""" + query = select(DocumentPage).where( + DocumentPage.document_id == document_id, + DocumentPage.page_number == page_number, + ) + result = self.db.execute(query) + return result.scalars().first() + + def create_page( + self, + document_id: uuid.UUID, + page_number: int, + width: float, + height: float, + image_path: str | None = None, + text_content: str | None = None, + ) -> DocumentPage: + """Create a new document page.""" + page = DocumentPage( + document_id=document_id, + page_number=page_number, + width=width, + height=height, + image_path=image_path, + text_content=text_content, + ) + return self.create(page) + + +class DocumentTextBlockRepository(BaseRepository[DocumentTextBlock]): + """Repository for DocumentTextBlock operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentTextBlock) + + def get_page_text_blocks(self, page_id: uuid.UUID) -> list[DocumentTextBlock]: + """Get all text blocks for a page.""" + query = ( + select(DocumentTextBlock) + .where(DocumentTextBlock.page_id == page_id) + .order_by(DocumentTextBlock.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_by_block_type(self, page_id: uuid.UUID, block_type: str) -> list[DocumentTextBlock]: + """Get text blocks by type (header, footer, watermark, text).""" + query = ( + select(DocumentTextBlock) + .where( + DocumentTextBlock.page_id == page_id, + DocumentTextBlock.block_type == block_type, + ) + .order_by(DocumentTextBlock.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_text_block( + self, + page_id: uuid.UUID, + text: str, + x: float, + y: float, + width: float, + height: float, + confidence: float | None = None, + font_family: str | None = None, + font_size: float | None = None, + font_color: str | None = None, + font_style: str | None = None, + block_type: str = "text", + sequence: int = 0, + ) -> DocumentTextBlock: + """Create a new text block.""" + text_block = DocumentTextBlock( + page_id=page_id, + text=text, + x=x, + y=y, + width=width, + height=height, + confidence=confidence, + font_family=font_family, + font_size=font_size, + font_color=font_color, + font_style=font_style, + block_type=block_type, + sequence=sequence, + ) + return self.create(text_block) + + +class DocumentImageRepository(BaseRepository[DocumentImage]): + """Repository for DocumentImage operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentImage) + + def get_page_images(self, page_id: uuid.UUID) -> list[DocumentImage]: + """Get all images for a page.""" + query = select(DocumentImage).where(DocumentImage.page_id == page_id) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_image( + self, + page_id: uuid.UUID, + x: float, + y: float, + width: float, + height: float, + image_path: str, + image_type: str = "figure", + ) -> DocumentImage: + """Create a new document image record.""" + image = DocumentImage( + page_id=page_id, + x=x, + y=y, + width=width, + height=height, + image_path=image_path, + image_type=image_type, + ) + return self.create(image) + + +class DocumentTableRepository(BaseRepository[DocumentTable]): + """Repository for DocumentTable operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentTable) + + def get_page_tables(self, page_id: uuid.UUID) -> list[DocumentTable]: + """Get all tables for a page.""" + query = select(DocumentTable).where(DocumentTable.page_id == page_id) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_table( + self, + page_id: uuid.UUID, + x: float, + y: float, + width: float, + height: float, + rows: int, + columns: int, + data: dict | None = None, + ) -> DocumentTable: + """Create a new document table record.""" + table = DocumentTable( + page_id=page_id, + x=x, + y=y, + width=width, + height=height, + rows=rows, + columns=columns, + data=data, + ) + return self.create(table) + + +class TemplateMatchRepository(BaseRepository[TemplateMatch]): + """Repository for TemplateMatch operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, TemplateMatch) + + def get_document_matches( + self, + document_id: uuid.UUID, + min_confidence: float = 0.0, + ) -> list[TemplateMatch]: + """Get all template matches for a document.""" + query = ( + select(TemplateMatch) + .where( + TemplateMatch.document_id == document_id, + TemplateMatch.confidence_score >= min_confidence, + ) + .order_by(TemplateMatch.confidence_score.desc()) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_selected_match(self, document_id: uuid.UUID) -> TemplateMatch | None: + """Get the selected template match for a document.""" + query = select(TemplateMatch).where( + TemplateMatch.document_id == document_id, + TemplateMatch.selected.is_(True), + ) + result = self.db.execute(query) + return result.scalars().first() + + def select_match(self, match_id: uuid.UUID) -> TemplateMatch | None: + """Select a template match (deselecting all others for the same document).""" + match = self.get_by_id(match_id) + if not match: + return None + + # Deselect all other matches for this document + query = select(TemplateMatch).where( + TemplateMatch.document_id == match.document_id, + TemplateMatch.selected.is_(True), + ) + result = self.db.execute(query) + for existing_match in result.scalars().all(): + existing_match.selected = False + + match.selected = True + self.db.flush() + self.db.refresh(match) + return match + + def create_match( + self, + document_id: uuid.UUID, + format_id: uuid.UUID, + confidence_score: float, + match_details: dict | None = None, + selected: bool = False, + ) -> TemplateMatch: + """Create a new template match.""" + template_match = TemplateMatch( + document_id=document_id, + format_id=format_id, + confidence_score=confidence_score, + match_details=match_details, + selected=selected, + ) + return self.create(template_match) diff --git a/docengine/app/repositories/template_repository.py b/docengine/app/repositories/template_repository.py new file mode 100644 index 0000000..b3d7d0f --- /dev/null +++ b/docengine/app/repositories/template_repository.py @@ -0,0 +1,355 @@ +from __future__ import annotations + +import uuid + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.template import ( + DocumentCell, + DocumentFormat, + DocumentRegion, + ImageRegion, + TableColumn, + TableFormat, + TableRow, + TemplateFingerprint, + Watermark, +) +from app.repositories.base import BaseRepository + + +class TemplateRepository(BaseRepository[DocumentFormat]): + """Repository for DocumentFormat (template) operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentFormat) + + def get_active_templates(self, offset: int = 0, limit: int = 100) -> list[DocumentFormat]: + """Get all active templates.""" + query = ( + select(DocumentFormat) + .where(DocumentFormat.is_active.is_(True)) + .order_by(DocumentFormat.created_at.desc()) + .offset(offset) + .limit(limit) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def count_active(self) -> int: + """Count active templates.""" + return self.count(filters={"is_active": True}) + + def get_by_name(self, name: str) -> DocumentFormat | None: + """Get template by name.""" + query = select(DocumentFormat).where(DocumentFormat.name == name) + result = self.db.execute(query) + return result.scalars().first() + + def get_by_source_document(self, document_id: uuid.UUID) -> DocumentFormat | None: + """Get template generated from a specific source document.""" + query = select(DocumentFormat).where( + DocumentFormat.source_document_id == document_id, + DocumentFormat.is_active.is_(True), + ) + result = self.db.execute(query) + return result.scalars().first() + + def create_template( + self, + name: str, + page_width: float, + page_height: float, + page_count: int = 1, + description: str | None = None, + margin_top: float = 72.0, + margin_right: float = 72.0, + margin_bottom: float = 72.0, + margin_left: float = 72.0, + fingerprint: dict | None = None, + source_document_id: uuid.UUID | None = None, + created_by: uuid.UUID | None = None, + is_active: bool = True, + ) -> DocumentFormat: + """Create a new template.""" + template = DocumentFormat( + name=name, + page_width=page_width, + page_height=page_height, + page_count=page_count, + description=description, + margin_top=margin_top, + margin_right=margin_right, + margin_bottom=margin_bottom, + margin_left=margin_left, + fingerprint=fingerprint, + source_document_id=source_document_id, + created_by=created_by, + is_active=is_active, + ) + return self.create(template) + + def deactivate_template(self, template_id: uuid.UUID) -> DocumentFormat | None: + """Soft-delete a template by deactivating it.""" + template = self.get_by_id(template_id) + if template: + template.is_active = False + self.db.flush() + self.db.refresh(template) + return template + + def get_all_with_fingerprints(self) -> list[DocumentFormat]: + """Get all active templates with their fingerprints.""" + query = ( + select(DocumentFormat) + .where(DocumentFormat.is_active.is_(True)) + .order_by(DocumentFormat.created_at.desc()) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + +class DocumentCellRepository(BaseRepository[DocumentCell]): + """Repository for DocumentCell operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentCell) + + def get_template_cells(self, format_id: uuid.UUID) -> list[DocumentCell]: + """Get all cells for a template.""" + query = ( + select(DocumentCell) + .where(DocumentCell.format_id == format_id) + .order_by(DocumentCell.page_number, DocumentCell.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_page_cells(self, format_id: uuid.UUID, page_number: int) -> list[DocumentCell]: + """Get cells for a specific page of a template.""" + query = ( + select(DocumentCell) + .where( + DocumentCell.format_id == format_id, + DocumentCell.page_number == page_number, + ) + .order_by(DocumentCell.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_dynamic_cells(self, format_id: uuid.UUID) -> list[DocumentCell]: + """Get all dynamic cells for a template.""" + query = ( + select(DocumentCell) + .where( + DocumentCell.format_id == format_id, + DocumentCell.is_dynamic.is_(True), + ) + .order_by(DocumentCell.page_number, DocumentCell.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_cell(self, format_id: uuid.UUID, **kwargs) -> DocumentCell: # noqa: ANN003 + """Create a new cell for a template.""" + cell = DocumentCell(format_id=format_id, **kwargs) + return self.create(cell) + + +class DocumentRegionRepository(BaseRepository[DocumentRegion]): + """Repository for DocumentRegion operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, DocumentRegion) + + def get_template_regions(self, format_id: uuid.UUID) -> list[DocumentRegion]: + """Get all regions for a template.""" + query = ( + select(DocumentRegion) + .where(DocumentRegion.format_id == format_id) + .order_by(DocumentRegion.page_number, DocumentRegion.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_regions_by_type(self, format_id: uuid.UUID, region_type: str) -> list[DocumentRegion]: + """Get regions of a specific type.""" + query = ( + select(DocumentRegion) + .where( + DocumentRegion.format_id == format_id, + DocumentRegion.region_type == region_type, + ) + .order_by(DocumentRegion.page_number, DocumentRegion.sequence) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_region(self, format_id: uuid.UUID, **kwargs) -> DocumentRegion: # noqa: ANN003 + """Create a new region for a template.""" + region = DocumentRegion(format_id=format_id, **kwargs) + return self.create(region) + + +class TableFormatRepository(BaseRepository[TableFormat]): + """Repository for TableFormat operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, TableFormat) + + def get_template_tables(self, format_id: uuid.UUID) -> list[TableFormat]: + """Get all table formats for a template.""" + query = ( + select(TableFormat) + .where(TableFormat.format_id == format_id) + .order_by(TableFormat.page_number) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_table_format(self, format_id: uuid.UUID, **kwargs) -> TableFormat: # noqa: ANN003 + """Create a new table format.""" + table_format = TableFormat(format_id=format_id, **kwargs) + return self.create(table_format) + + +class TableColumnRepository(BaseRepository[TableColumn]): + """Repository for TableColumn operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, TableColumn) + + def get_table_columns(self, table_format_id: uuid.UUID) -> list[TableColumn]: + """Get all columns for a table format.""" + query = ( + select(TableColumn) + .where(TableColumn.table_format_id == table_format_id) + .order_by(TableColumn.column_index) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_column(self, table_format_id: uuid.UUID, **kwargs) -> TableColumn: # noqa: ANN003 + """Create a new table column.""" + column = TableColumn(table_format_id=table_format_id, **kwargs) + return self.create(column) + + +class TableRowRepository(BaseRepository[TableRow]): + """Repository for TableRow operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, TableRow) + + def get_table_rows(self, table_format_id: uuid.UUID) -> list[TableRow]: + """Get all rows for a table format.""" + query = ( + select(TableRow) + .where(TableRow.table_format_id == table_format_id) + .order_by(TableRow.row_index) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_row(self, table_format_id: uuid.UUID, **kwargs) -> TableRow: # noqa: ANN003 + """Create a new table row.""" + row = TableRow(table_format_id=table_format_id, **kwargs) + return self.create(row) + + +class WatermarkRepository(BaseRepository[Watermark]): + """Repository for Watermark operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, Watermark) + + def get_template_watermarks(self, format_id: uuid.UUID) -> list[Watermark]: + """Get all watermarks for a template.""" + query = select(Watermark).where(Watermark.format_id == format_id) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_watermark(self, format_id: uuid.UUID, **kwargs) -> Watermark: # noqa: ANN003 + """Create a new watermark.""" + watermark = Watermark(format_id=format_id, **kwargs) + return self.create(watermark) + + +class ImageRegionRepository(BaseRepository[ImageRegion]): + """Repository for ImageRegion operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, ImageRegion) + + def get_template_images(self, format_id: uuid.UUID) -> list[ImageRegion]: + """Get all image regions for a template.""" + query = select(ImageRegion).where(ImageRegion.format_id == format_id) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_static_images(self, format_id: uuid.UUID) -> list[ImageRegion]: + """Get static image regions.""" + query = select(ImageRegion).where( + ImageRegion.format_id == format_id, + ImageRegion.is_static.is_(True), + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_image_region(self, format_id: uuid.UUID, **kwargs) -> ImageRegion: # noqa: ANN003 + """Create a new image region.""" + image_region = ImageRegion(format_id=format_id, **kwargs) + return self.create(image_region) + + +class TemplateFingerprintRepository(BaseRepository[TemplateFingerprint]): + """Repository for TemplateFingerprint operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, TemplateFingerprint) + + def get_by_format_id(self, format_id: uuid.UUID) -> TemplateFingerprint | None: + """Get fingerprint by template format ID.""" + query = select(TemplateFingerprint).where(TemplateFingerprint.format_id == format_id) + result = self.db.execute(query) + return result.scalars().first() + + def get_by_hash(self, fingerprint_hash: str) -> TemplateFingerprint | None: + """Get fingerprint by hash.""" + query = select(TemplateFingerprint).where( + TemplateFingerprint.fingerprint_hash == fingerprint_hash + ) + result = self.db.execute(query) + return result.scalars().first() + + def get_all_fingerprints(self) -> list[TemplateFingerprint]: + """Get all fingerprints.""" + query = select(TemplateFingerprint) + result = self.db.execute(query) + return list(result.scalars().all()) + + def create_fingerprint( + self, + format_id: uuid.UUID, + fingerprint_hash: str, + page_dimensions: dict | None = None, + logo_coordinates: dict | None = None, + header_coordinates: dict | None = None, + footer_coordinates: dict | None = None, + table_coordinates: dict | None = None, + cell_coordinates: dict | None = None, + ) -> TemplateFingerprint: + """Create a new template fingerprint.""" + fp = TemplateFingerprint( + format_id=format_id, + fingerprint_hash=fingerprint_hash, + page_dimensions=page_dimensions, + logo_coordinates=logo_coordinates, + header_coordinates=header_coordinates, + footer_coordinates=footer_coordinates, + table_coordinates=table_coordinates, + cell_coordinates=cell_coordinates, + ) + return self.create(fp) diff --git a/docengine/app/repositories/user_repository.py b/docengine/app/repositories/user_repository.py new file mode 100644 index 0000000..cf38b63 --- /dev/null +++ b/docengine/app/repositories/user_repository.py @@ -0,0 +1,236 @@ +from __future__ import annotations + +import uuid +from datetime import UTC, datetime + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.user import AuditLog, RefreshToken, Role, User, user_roles_table +from app.repositories.base import BaseRepository + + +class UserRepository(BaseRepository[User]): + """Repository for User operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, User) + + def get_by_username(self, username: str) -> User | None: + """Get user by username.""" + query = select(User).where(User.username == username) + result = self.db.execute(query) + return result.scalars().first() + + def get_by_email(self, email: str) -> User | None: + """Get user by email.""" + query = select(User).where(User.email == email) + result = self.db.execute(query) + return result.scalars().first() + + def create_user( + self, + username: str, + email: str, + hashed_password: str, + full_name: str | None = None, + is_active: bool = True, + is_superuser: bool = False, + role_names: list[str] | None = None, + ) -> User: + """Create a new user with optional roles.""" + user = User( + username=username, + email=email, + hashed_password=hashed_password, + full_name=full_name, + is_active=is_active, + is_superuser=is_superuser, + ) + + if role_names: + roles = self.get_roles_by_names(role_names) + user.roles = roles + + return self.create(user) + + def update_last_login(self, user: User) -> User: + """Update user's last login timestamp.""" + user.last_login = datetime.now(UTC) + self.db.flush() + self.db.refresh(user) + return user + + def get_roles_by_names(self, role_names: list[str]) -> list[Role]: + """Get roles by their names.""" + query = select(Role).where(Role.name.in_(role_names)) + result = self.db.execute(query) + return list(result.scalars().all()) + + def assign_roles(self, user: User, role_names: list[str]) -> User: + """Assign roles to a user, replacing existing roles.""" + roles = self.get_roles_by_names(role_names) + user.roles = roles + self.db.flush() + self.db.refresh(user) + return user + + def get_active_users(self, offset: int = 0, limit: int = 100) -> list[User]: + """Get all active users.""" + query = select(User).where(User.is_active.is_(True)).offset(offset).limit(limit) + result = self.db.execute(query) + return list(result.scalars().all()) + + +class RoleRepository(BaseRepository[Role]): + """Repository for Role operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, Role) + + def get_by_name(self, name: str) -> Role | None: + """Get role by name.""" + query = select(Role).where(Role.name == name) + result = self.db.execute(query) + return result.scalars().first() + + def create_role(self, name: str, description: str | None = None) -> Role: + """Create a new role.""" + role = Role(name=name, description=description) + return self.create(role) + + def get_all_roles(self) -> list[Role]: + """Get all roles.""" + query = select(Role).order_by(Role.name) + result = self.db.execute(query) + return list(result.scalars().all()) + + +class RefreshTokenRepository(BaseRepository[RefreshToken]): + """Repository for RefreshToken operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, RefreshToken) + + def get_by_token(self, token: str) -> RefreshToken | None: + """Get refresh token by token string.""" + query = select(RefreshToken).where( + RefreshToken.token == token, + RefreshToken.revoked.is_(False), + RefreshToken.expires_at > datetime.now(UTC), + ) + result = self.db.execute(query) + return result.scalars().first() + + def create_token(self, user_id: uuid.UUID, token: str, expires_at: datetime) -> RefreshToken: + """Create a new refresh token.""" + refresh_token = RefreshToken( + user_id=user_id, + token=token, + expires_at=expires_at, + ) + return self.create(refresh_token) + + def revoke_token(self, token: str) -> bool: + """Revoke a refresh token.""" + refresh_token = self.get_by_token(token) + if refresh_token: + refresh_token.revoked = True + self.db.flush() + return True + return False + + def revoke_all_user_tokens(self, user_id: uuid.UUID) -> int: + """Revoke all refresh tokens for a user.""" + query = select(RefreshToken).where( + RefreshToken.user_id == user_id, + RefreshToken.revoked.is_(False), + ) + result = self.db.execute(query) + tokens = result.scalars().all() + count = 0 + for token in tokens: + token.revoked = True + count += 1 + self.db.flush() + return count + + def cleanup_expired_tokens(self) -> int: + """Remove expired or revoked tokens.""" + query = select(RefreshToken).where( + (RefreshToken.expires_at <= datetime.now(UTC)) | (RefreshToken.revoked.is_(True)) + ) + result = self.db.execute(query) + tokens = result.scalars().all() + count = len(tokens) + for token in tokens: + self.db.delete(token) + self.db.flush() + return count + + +class AuditLogRepository(BaseRepository[AuditLog]): + """Repository for AuditLog operations.""" + + def __init__(self, db: Session) -> None: + super().__init__(db, AuditLog) + + def log_action( + self, + action: str, + resource_type: str, + resource_id: str | None = None, + user_id: uuid.UUID | None = None, + details: str | None = None, + ip_address: str | None = None, + user_agent: str | None = None, + ) -> AuditLog: + """Create an audit log entry.""" + audit_log = AuditLog( + user_id=user_id, + action=action, + resource_type=resource_type, + resource_id=resource_id, + details=details, + ip_address=ip_address, + user_agent=user_agent, + ) + return self.create(audit_log) + + def get_user_logs( + self, + user_id: uuid.UUID, + offset: int = 0, + limit: int = 100, + ) -> list[AuditLog]: + """Get audit logs for a specific user.""" + query = ( + select(AuditLog) + .where(AuditLog.user_id == user_id) + .order_by(AuditLog.created_at.desc()) + .offset(offset) + .limit(limit) + ) + result = self.db.execute(query) + return list(result.scalars().all()) + + def get_resource_logs( + self, + resource_type: str, + resource_id: str, + offset: int = 0, + limit: int = 100, + ) -> list[AuditLog]: + """Get audit logs for a specific resource.""" + query = ( + select(AuditLog) + .where( + AuditLog.resource_type == resource_type, + AuditLog.resource_id == resource_id, + ) + .order_by(AuditLog.created_at.desc()) + .offset(offset) + .limit(limit) + ) + result = self.db.execute(query) + return list(result.scalars().all()) diff --git a/docengine/app/schemas/__init__.py b/docengine/app/schemas/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/schemas/auth.py b/docengine/app/schemas/auth.py new file mode 100644 index 0000000..e181119 --- /dev/null +++ b/docengine/app/schemas/auth.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +from pydantic import BaseModel, EmailStr, Field + +from app.schemas.common import BaseSchema + + +class LoginRequest(BaseModel): + """Login credentials.""" + + username: str = Field(..., min_length=3, max_length=150) + password: str = Field(..., min_length=8, max_length=128) + + +class RegisterRequest(BaseModel): + """User registration payload.""" + + username: str = Field(..., min_length=3, max_length=150) + email: EmailStr + password: str = Field(..., min_length=8, max_length=128) + full_name: str | None = Field(None, max_length=255) + + +class TokenResponse(BaseSchema): + """JWT token pair response.""" + + access_token: str + refresh_token: str + token_type: str = "bearer" + expires_in: int + + +class RefreshTokenRequest(BaseModel): + """Refresh token request payload.""" + + refresh_token: str + + +class ChangePasswordRequest(BaseModel): + """Change password payload.""" + + current_password: str = Field(..., min_length=8, max_length=128) + new_password: str = Field(..., min_length=8, max_length=128) diff --git a/docengine/app/schemas/common.py b/docengine/app/schemas/common.py new file mode 100644 index 0000000..16cb483 --- /dev/null +++ b/docengine/app/schemas/common.py @@ -0,0 +1,84 @@ +from __future__ import annotations + +import uuid +from datetime import datetime +from typing import Any, Generic, TypeVar + +from pydantic import BaseModel, ConfigDict, Field + +T = TypeVar("T") + + +class BaseSchema(BaseModel): + """Base schema with common configuration.""" + + model_config = ConfigDict( + from_attributes=True, + populate_by_name=True, + str_strip_whitespace=True, + ) + + +class PaginationParams(BaseModel): + """Pagination query parameters.""" + + page: int = Field(default=1, ge=1, description="Page number") + page_size: int = Field(default=20, ge=1, le=100, description="Items per page") + + @property + def offset(self) -> int: + return (self.page - 1) * self.page_size + + +class PaginatedResponse(BaseSchema, Generic[T]): + """Paginated response wrapper.""" + + items: list[T] + total: int + page: int + page_size: int + total_pages: int + + @classmethod + def create(cls, items: list[T], total: int, page: int, page_size: int) -> PaginatedResponse[T]: + total_pages = (total + page_size - 1) // page_size if page_size > 0 else 0 + return cls( + items=items, + total=total, + page=page, + page_size=page_size, + total_pages=total_pages, + ) + + +class ErrorResponse(BaseSchema): + """Standard error response.""" + + detail: str + error_code: str | None = None + errors: list[dict[str, Any]] | None = None + timestamp: datetime = Field(default_factory=datetime.utcnow) + + +class SuccessResponse(BaseSchema): + """Standard success response.""" + + message: str + data: dict[str, Any] | None = None + + +class HealthResponse(BaseSchema): + """Health check response.""" + + status: str + version: str + environment: str + database: str + redis: str + timestamp: datetime = Field(default_factory=datetime.utcnow) + + +class IDResponse(BaseSchema): + """Response containing just an ID.""" + + id: uuid.UUID diff --git a/docengine/app/schemas/document.py b/docengine/app/schemas/document.py new file mode 100644 index 0000000..0e9711a --- /dev/null +++ b/docengine/app/schemas/document.py @@ -0,0 +1,134 @@ +from __future__ import annotations + +import uuid +from datetime import datetime +from typing import Any + +from pydantic import Field + +from app.schemas.common import BaseSchema + + +class DocumentUploadResponse(BaseSchema): + """Response after document upload.""" + + id: uuid.UUID + filename: str + original_filename: str + content_type: str + file_size: int + checksum: str + status: str + created_at: datetime + + +class TextBlockResponse(BaseSchema): + """Extracted text block.""" + + id: uuid.UUID + text: str + x: float + y: float + width: float + height: float + confidence: float | None + font_family: str | None + font_size: float | None + font_color: str | None + font_style: str | None + block_type: str + sequence: int + + +class DocumentImageResponse(BaseSchema): + """Extracted document image.""" + + id: uuid.UUID + x: float + y: float + width: float + height: float + image_path: str + image_type: str + + +class DocumentTableResponse(BaseSchema): + """Extracted document table.""" + + id: uuid.UUID + x: float + y: float + width: float + height: float + rows: int + columns: int + data: dict[str, Any] | None + + +class DocumentPageResponse(BaseSchema): + """Document page with extracted content.""" + + id: uuid.UUID + page_number: int + width: float + height: float + image_path: str | None + text_content: str | None + text_blocks: list[TextBlockResponse] = Field(default_factory=list) + images: list[DocumentImageResponse] = Field(default_factory=list) + tables: list[DocumentTableResponse] = Field(default_factory=list) + + +class DocumentResponse(BaseSchema): + """Full document response.""" + + id: uuid.UUID + filename: str + original_filename: str + content_type: str + file_size: int + checksum: str + storage_path: str + status: str + page_count: int | None + is_scanned: bool | None + document_metadata: dict[str, Any] | None + error_message: str | None + uploaded_by: uuid.UUID | None + pages: list[DocumentPageResponse] = Field(default_factory=list) + created_at: datetime + updated_at: datetime + + +class DocumentListResponse(BaseSchema): + """Minimal document response for lists.""" + + id: uuid.UUID + original_filename: str + content_type: str + file_size: int + status: str + page_count: int | None + is_scanned: bool | None + created_at: datetime + + +class TemplateMatchResponse(BaseSchema): + """Template match result.""" + + id: uuid.UUID + document_id: uuid.UUID + format_id: uuid.UUID + confidence_score: float + match_details: dict[str, Any] | None + selected: bool + template_name: str | None = None + created_at: datetime + + +class TemplateMatchRequest(BaseSchema): + """Request to match a document against templates.""" + + document_id: uuid.UUID + min_confidence: float = Field(default=0.75, ge=0.0, le=1.0) + max_results: int = Field(default=5, ge=1, le=20) diff --git a/docengine/app/schemas/template.py b/docengine/app/schemas/template.py new file mode 100644 index 0000000..bf8285e --- /dev/null +++ b/docengine/app/schemas/template.py @@ -0,0 +1,277 @@ +from __future__ import annotations + +import uuid +from datetime import datetime +from typing import Any, List, Optional, Union + +from pydantic import Field + +from app.schemas.common import BaseSchema + + +class DocumentCellResponse(BaseSchema): + """Document cell in a template.""" + + id: uuid.UUID + format_id: uuid.UUID + page_number: int + x: float + y: float + width: float + height: float + row_no: int + column_no: int + data_type: str + font_family: str | None + font_size: float | None + font_style: str | None + font_color: str | None + background_color: str | None + border_top: str | None + border_right: str | None + border_bottom: str | None + border_left: str | None + padding_top: float + padding_right: float + padding_bottom: float + padding_left: float + alignment: str + vertical_alignment: str + rowspan: int + colspan: int + static_text: str | None + field_name: str | None + sequence: int + is_dynamic: bool + + +class DocumentCellCreate(BaseSchema): + """Create a document cell.""" + + page_number: int = Field(..., ge=1) + x: float + y: float + width: float = Field(..., gt=0) + height: float = Field(..., gt=0) + row_no: int = 0 + column_no: int = 0 + data_type: str = "text" + font_family: str | None = None + font_size: float | None = None + font_style: str | None = None + font_color: str | None = None + background_color: str | None = None + border_top: str | None = None + border_right: str | None = None + border_bottom: str | None = None + border_left: str | None = None + padding_top: float = 0.0 + padding_right: float = 0.0 + padding_bottom: float = 0.0 + padding_left: float = 0.0 + alignment: str = "left" + vertical_alignment: str = "top" + rowspan: int = 1 + colspan: int = 1 + static_text: str | None = None + field_name: str | None = None + sequence: int = 0 + is_dynamic: bool = False + + +class DocumentRegionResponse(BaseSchema): + """Region in a template.""" + + id: uuid.UUID + format_id: uuid.UUID + page_number: int + region_type: str + x: float + y: float + width: float + height: float + content: dict[str, Any] | None + sequence: int + + +class TableColumnResponse(BaseSchema): + """Table column definition.""" + + id: uuid.UUID + table_format_id: uuid.UUID + column_index: int + width: float + header_text: str | None + data_type: str + alignment: str + font_family: str | None + font_size: float | None + + +class TableRowResponse(BaseSchema): + """Table row definition.""" + + id: uuid.UUID + table_format_id: uuid.UUID + row_index: int + height: float + is_header: bool + background_color: str | None + + +class TableFormatResponse(BaseSchema): + """Table format in a template.""" + + id: uuid.UUID + format_id: uuid.UUID + page_number: int + x: float + y: float + width: float + height: float + rows: int + columns: int + border_style: str + border_width: float + border_color: str + header_rows: int + table_columns: list[TableColumnResponse] = Field(default_factory=list) + table_rows: list[TableRowResponse] = Field(default_factory=list) + + +class WatermarkResponse(BaseSchema): + """Watermark in a template.""" + + id: uuid.UUID + format_id: uuid.UUID + page_number: int | None + text: str | None + image_path: str | None + x: float + y: float + width: float + height: float + opacity: float + rotation: float + font_family: str | None + font_size: float | None + font_color: str | None + + +class ImageRegionResponse(BaseSchema): + """Image region in a template.""" + + id: uuid.UUID + format_id: uuid.UUID + page_number: int + x: float + y: float + width: float + height: float + image_path: str | None + image_type: str + is_static: bool + field_name: str | None + + +class TemplateFingerprintResponse(BaseSchema): + """Template fingerprint.""" + + id: uuid.UUID + format_id: uuid.UUID + page_dimensions: dict[str, Any] | None + logo_coordinates: dict[str, Any] | None + header_coordinates: dict[str, Any] | None + footer_coordinates: dict[str, Any] | None + table_coordinates: dict[str, Any] | None + cell_coordinates: dict[str, Any] | None + fingerprint_hash: str + + +class TemplateResponse(BaseSchema): + """Full template response.""" + + id: uuid.UUID + name: str + description: str | None + page_width: float + page_height: float + page_count: int + margin_top: float + margin_right: float + margin_bottom: float + margin_left: float + fingerprint: dict[str, Any] | None + source_document_id: uuid.UUID | None + version: int + is_active: bool + created_by: uuid.UUID | None + cells: list[DocumentCellResponse] = Field(default_factory=list) + regions: list[DocumentRegionResponse] = Field(default_factory=list) + table_formats: list[TableFormatResponse] = Field(default_factory=list) + watermarks: list[WatermarkResponse] = Field(default_factory=list) + image_regions: list[ImageRegionResponse] = Field(default_factory=list) + fingerprint_record: TemplateFingerprintResponse | None = None + created_at: datetime + updated_at: datetime + + +class TemplateListResponse(BaseSchema): + """Minimal template response for lists.""" + + id: uuid.UUID + name: str + description: str | None + page_width: float + page_height: float + page_count: int + version: int + is_active: bool + created_at: datetime + updated_at: datetime + + +class TemplateRenderRequest(BaseSchema): + """Request to render a template to PDF.""" + + template_id: uuid.UUID + data: dict[str, Any] = Field(default_factory=dict, description="Data to populate dynamic fields") + output_filename: str | None = Field(None, max_length=255, description="Output filename for generated PDF") + images: dict[str, str] | None = Field(None, description="Mapping of field_name to image path for dynamic images") + + +class TemplateRenderResponse(BaseSchema): + """Response after rendering a template.""" + + output_path: str + filename: str + file_size: int + page_count: int + rendered_at: datetime + + +class TemplateFieldCreate(BaseSchema): + field_label: str + field_type: str = "text" + display_order: int = 0 + required_flag: bool = False + +class TemplateCreateRequest(BaseSchema): + template_name: str + source_document_id: Optional[str] = None + fields: List[TemplateFieldCreate] = Field(default_factory=list) + +class MappingNodeRequest(BaseSchema): + pk_document_data_id: Union[int, str, None] = None + x_coordinate: float + y_coordinate: float + width: float + height: float + text_value: Optional[str] = None + page_width: Optional[float] = None + page_height: Optional[float] = None + page_no: Optional[int] = None + +class TemplateMappingSaveRequest(BaseSchema): + field_name: str + mapped_nodes: List[MappingNodeRequest] = Field(default_factory=list) diff --git a/docengine/app/schemas/user.py b/docengine/app/schemas/user.py new file mode 100644 index 0000000..9267afc --- /dev/null +++ b/docengine/app/schemas/user.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import uuid +from datetime import datetime + +from pydantic import EmailStr, Field + +from app.schemas.common import BaseSchema + + +class UserBase(BaseSchema): + """Base user fields.""" + + username: str = Field(..., min_length=3, max_length=150) + email: EmailStr + full_name: str | None = Field(None, max_length=255) + + +class UserCreate(UserBase): + """User creation payload.""" + + password: str = Field(..., min_length=8, max_length=128) + is_active: bool = True + is_superuser: bool = False + role_names: list[str] = Field(default_factory=list) + + +class UserUpdate(BaseSchema): + """User update payload.""" + + email: EmailStr | None = None + full_name: str | None = None + is_active: bool | None = None + is_superuser: bool | None = None + role_names: list[str] | None = None + + +class RoleResponse(BaseSchema): + """Role response.""" + + id: uuid.UUID + name: str + description: str | None + + +class UserResponse(BaseSchema): + """Full user response.""" + + id: uuid.UUID + username: str + email: str + full_name: str | None + is_active: bool + is_superuser: bool + roles: list[RoleResponse] = Field(default_factory=list) + created_at: datetime + updated_at: datetime + last_login: datetime | None + + +class UserListResponse(BaseSchema): + """Minimal user response for lists.""" + + id: uuid.UUID + username: str + email: str + full_name: str | None + is_active: bool + created_at: datetime diff --git a/docengine/app/services/__init__.py b/docengine/app/services/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/services/document_service.py b/docengine/app/services/document_service.py new file mode 100644 index 0000000..c8a21e5 --- /dev/null +++ b/docengine/app/services/document_service.py @@ -0,0 +1,104 @@ +from __future__ import annotations + +import uuid + +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.document import Document +from app.repositories.document_repository import DocumentRepository +from app.services.layout_service import LayoutService +from app.services.ocr_service import OCRService +from app.services.pdf_service import NativePDFService +from app.services.template_service import TemplateService + +logger = get_logger(__name__) + + +class DocumentProcessingService: + """Orchestrates the full document processing pipeline.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.doc_repo = DocumentRepository(db) + self.pdf_service = NativePDFService(db) + self.ocr_service = OCRService(db) + self.layout_service = LayoutService(db) + self.template_service = TemplateService(db) + + def process_document(self, document_id: str | uuid.UUID) -> Document: + """Process a document through the full pipeline.""" + if isinstance(document_id, str): + document_id = uuid.UUID(document_id) + + document = self.doc_repo.get_by_id(document_id) + if not document: + raise ValueError(f"Document '{document_id}' not found") + + logger.info( + "processing_started", + document_id=str(document_id), + content_type=document.content_type, + ) + + # Update status to processing + self.doc_repo.update_status(document_id, "processing") + self.db.commit() + + try: + # Step 1: Extract content based on document type + if document.content_type == "application/pdf": + document = self._process_pdf(document) + else: + document = self._process_image(document) + + # Step 2: Analyze layout + layout_results = self.layout_service.analyze_document_layout(document) + metadata = document.document_metadata or {} + metadata["layout"] = layout_results + document.document_metadata = metadata + + # Step 3: Update document status + self.doc_repo.update_status(document_id, "completed") + self.db.commit() + + logger.info( + "processing_completed", + document_id=str(document_id), + pages=document.page_count, + ) + + return document + + except Exception as e: + logger.exception( + "processing_failed", + document_id=str(document_id), + error=str(e), + ) + self.doc_repo.update_status(document_id, "failed", error_message=str(e)) + self.db.commit() + raise + + def _process_pdf(self, document: Document) -> Document: + """Process a PDF document - either native or scanned.""" + # First, try native PDF extraction + document = self.pdf_service.process_pdf(document) + self.db.flush() + + # If scanned, also run OCR + if document.is_scanned: + logger.info( + "scanned_pdf_detected", + document_id=str(document.id), + ) + document = self.ocr_service.process_scanned_pdf(document) + self.db.flush() + + return document + + def _process_image(self, document: Document) -> Document: + """Process an image document with OCR.""" + document = self.ocr_service.process_image(document) + self.db.flush() + return document diff --git a/docengine/app/services/extraction_service.py b/docengine/app/services/extraction_service.py new file mode 100644 index 0000000..c2d3772 --- /dev/null +++ b/docengine/app/services/extraction_service.py @@ -0,0 +1,295 @@ +from __future__ import annotations + +import uuid +from typing import Any + +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.repositories.document_repository import DocumentRepository +from app.repositories.template_repository import TemplateRepository +from app.services.matching_service import MatchingService + +logger = get_logger(__name__) + + +class ExtractionService: + """Extract document values using matched template coordinates and structures.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.matching_service = MatchingService(db) + self.doc_repo = DocumentRepository(db) + self.template_repo = TemplateRepository(db) + + def extract_document_data(self, document_id: uuid.UUID) -> dict[str, Any]: + """Perform template matching on the document and extract structured field values.""" + document = self.doc_repo.get_with_pages(document_id) + if not document: + raise ValueError(f"Document '{document_id}' not found") + + # 1. Match document against existing templates + matches = self.matching_service.match_document(document_id, min_confidence=0.75) + if not matches: + logger.info("extraction_failed_no_match", document_id=str(document_id)) + return { + "template_matched": False, + "template_id": None, + "template_name": None, + "confidence_score": 0.0, + "extracted_data": {}, + } + + best_match = matches[0] + template = self.template_repo.get_by_id(best_match.format_id) + if not template: + raise ValueError(f"Matched template '{best_match.format_id}' not found") + + logger.info( + "extraction_matched_template", + document_id=str(document_id), + template_id=str(template.id), + template_name=template.name, + score=best_match.confidence_score, + ) + + extracted_data: dict[str, Any] = {} + + # Load all mapped regions for the template + regions = [r for r in template.regions if r.region_type == "field_mapping"] + regions_by_field: dict[str, list[Any]] = {} + for r in regions: + field_name = r.content.get("field_name") if r.content else None + if field_name: + regions_by_field.setdefault(field_name, []).append(r) + + # 2. Divide fields into Scalar vs Table Column types + scalar_cells = [cell for cell in template.cells if cell.data_type != "TABLE_COLUMN"] + table_column_cells = [cell for cell in template.cells if cell.data_type == "TABLE_COLUMN"] + + # 3. Extract Scalar Fields + for cell in scalar_cells: + field_name = cell.field_name + if not field_name: + continue + + field_regions = regions_by_field.get(field_name, []) + if not field_regions: + extracted_data[field_name] = "" + continue + + extracted_values = [] + extracted_block_ids = set() + sorted_regions = sorted(field_regions, key=lambda r: (r.page_number, r.sequence, r.y, r.x)) + + for region in sorted_regions: + page = next((p for p in document.pages if p.page_number == region.page_number), None) + if not page: + continue + best_block = self._find_best_overlapping_block(page, region) + if best_block and best_block.id not in extracted_block_ids: + extracted_block_ids.add(best_block.id) + val = best_block.text.strip() + if val: + extracted_values.append(val) + + extracted_data[field_name] = " ".join(extracted_values) + + # 4. Extract Table Column Fields + if table_column_cells: + table_column_names = {cell.field_name for cell in table_column_cells if cell.field_name} + table_regions = [ + r for r in regions + if r.content and r.content.get("field_name") in table_column_names + ] + + if table_regions: + # Determine vertical boundaries of the table area + table_start_y = min((r.y for r in table_regions), default=0.0) + table_start_y = max(0.0, table_start_y - 10.0) # subtract buffer + + # Process page where the table coordinates are mapped + page_number = min((r.page_number for r in table_regions), default=1) + page = next((p for p in document.pages if p.page_number == page_number), None) + + # Determine table vertical end Y by finding any summary scalar fields below the table + summary_keywords = {"tax", "total", "shipping", "discount", "vat", "handling", "duty", "subtotal", "grand"} + summary_regions = [] + for col_name, regs in regions_by_field.items(): + if col_name in table_column_names: + continue + for r in regs: + if r.y > table_start_y and any(kw in col_name.lower() for kw in summary_keywords): + # Skip left-aligned metadata fields (like Shipping Method) + if page and r.x < page.width * 0.4: + continue + summary_regions.append(r) + + table_end_y = min((r.y for r in summary_regions), default=99999.0) + + # Check if template has a footer to define the end vertical boundary + footer_regions = [r for r in template.regions if r.region_type == "footer"] + if footer_regions: + table_end_y = min(table_end_y, min(r.y for r in footer_regions)) + + # Determine horizontal ranges (X span) for each column in the template + col_x_spans: dict[str, tuple[float, float]] = {} + for col_name in table_column_names: + col_regs = [r for r in table_regions if r.content and r.content.get("field_name") == col_name] + if col_regs: + x_min = min(r.x for r in col_regs) + x_max = max(r.x + r.width for r in col_regs) + # Add a 15px margin to accommodate layout differences + col_x_spans[col_name] = (max(0.0, x_min - 15.0), x_max + 15.0) + else: + col_x_spans[col_name] = (0.0, 0.0) + + if page: + # Check for summary keyword text blocks (e.g. Total, Tax) as vertical boundary fallback + summary_labels = {"subtotal", "total", "grand total", "tax", "shipping & handling", "discount"} + for block in page.text_blocks: + if block.y > table_start_y: + # Skip left-aligned text blocks + if block.x < page.width * 0.4: + continue + block_text_clean = block.text.strip().lower() + if any(label in block_text_clean for label in summary_labels): + table_end_y = min(table_end_y, block.y) + + # Collect all document blocks inside Y range + candidate_blocks = [ + b for b in page.text_blocks + if b.y >= table_start_y and b.y < table_end_y + ] + # Sort blocks by Y coordinate + sorted_blocks = sorted(candidate_blocks, key=lambda b: b.y) + + # Group blocks into rows by vertical center alignment (15px threshold) + rows: list[list[Any]] = [] + current_row: list[Any] = [] + current_y_center = None + + for block in sorted_blocks: + if not block.text.strip(): + continue + block_y_center = block.y + block.height / 2 + if current_y_center is None: + current_row.append(block) + current_y_center = block_y_center + elif abs(block_y_center - current_y_center) < 15.0: + current_row.append(block) + else: + rows.append(current_row) + current_row = [block] + current_y_center = block_y_center + if current_row: + rows.append(current_row) + + # Map candidate blocks in each row to columns + rows_data: list[dict[str, str]] = [] + for row_blocks in rows: + row_dict: dict[str, list[Any]] = {name: [] for name in table_column_names} + for block in row_blocks: + best_col = None + best_overlap = 0.0 + for col_name, (x_min, x_max) in col_x_spans.items(): + overlap = max( + 0.0, + min(x_max, block.x + block.width) - max(x_min, block.x) + ) + overlap_ratio = overlap / block.width if block.width > 0 else 0.0 + if overlap_ratio > 0.2 and overlap_ratio > best_overlap: + best_overlap = overlap_ratio + best_col = col_name + if best_col: + row_dict[best_col].append(block) + + # Construct row values + row_values: dict[str, str] = {} + for col_name, blocks in row_dict.items(): + sorted_blocks_in_col = sorted(blocks, key=lambda b: b.x) + row_values[col_name] = " ".join( + b.text.strip() for b in sorted_blocks_in_col + ) + + # Filter out table header rows + is_header = False + for col_name, val in row_values.items(): + val_lower = val.lower() + if ( + val_lower == col_name.lower() or + val_lower in [ + "item", "items", "qty", "quantity", "price", + "amount", "total", "subtotal", "description" + ] + ): + is_header = True + break + + # Append if not header and at least one cell has a value + if not is_header and any(row_values.values()): + rows_data.append(row_values) + + # Merge continuation lines (descriptions spanning multiple rows with empty sibling columns) + merged_rows: list[dict[str, str]] = [] + for row in rows_data: + non_empty_cols = [k for k, v in row.items() if v.strip()] + if len(merged_rows) > 0 and len(non_empty_cols) == 1: + col_name = non_empty_cols[0] + last_row = merged_rows[-1] + if last_row.get(col_name): + last_row[col_name] = last_row[col_name] + " " + row[col_name] + else: + last_row[col_name] = row[col_name] + else: + merged_rows.append(row.copy()) + + rows_data = merged_rows + + # Pivot list of rows into parallel arrays under column names + for col_name in table_column_names: + extracted_data[col_name] = [] + for row in rows_data: + for col_name in table_column_names: + extracted_data[col_name].append(row.get(col_name, "")) + + return { + "template_matched": True, + "template_id": str(template.id), + "template_name": template.name, + "confidence_score": best_match.confidence_score, + "extracted_data": extracted_data, + } + + def _find_best_overlapping_block(self, page: Any, region: Any) -> Any | None: + """Find the document text block overlapping most with the template region.""" + best_block = None + best_overlap = 0.0 + for block in page.text_blocks: + x_overlap = max( + 0.0, + min(region.x + region.width, block.x + block.width) - max(region.x, block.x) + ) + y_overlap = max( + 0.0, + min(region.y + region.height, block.y + block.height) - max(region.y, block.y) + ) + overlap = x_overlap * y_overlap + if overlap > best_overlap: + best_overlap = overlap + best_block = block + + # Fallback: if no overlapping block, find the closest center-to-center + if not best_block: + min_dist = 100.0 # max 100px center-to-center distance + for block in page.text_blocks: + c_rx = region.x + region.width / 2 + c_ry = region.y + region.height / 2 + c_bx = block.x + block.width / 2 + c_by = block.y + block.height / 2 + dist = ((c_rx - c_bx) ** 2 + (c_ry - c_by) ** 2) ** 0.5 + if dist < min_dist: + min_dist = dist + best_block = block + + return best_block diff --git a/docengine/app/services/fingerprint_service.py b/docengine/app/services/fingerprint_service.py new file mode 100644 index 0000000..b109424 --- /dev/null +++ b/docengine/app/services/fingerprint_service.py @@ -0,0 +1,374 @@ +from __future__ import annotations + +import hashlib +import json +import uuid +from typing import Any + +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.template import DocumentFormat, TemplateFingerprint +from app.repositories.template_repository import TemplateFingerprintRepository, TemplateRepository + +logger = get_logger(__name__) + + +class FingerprintService: + """Generate and manage layout fingerprints for template matching.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.fingerprint_repo = TemplateFingerprintRepository(db) + self.template_repo = TemplateRepository(db) + + def generate_fingerprint(self, template: DocumentFormat) -> TemplateFingerprint: + """Generate a layout fingerprint for a template.""" + # Collect page dimensions + page_dimensions = { + "width": template.page_width, + "height": template.page_height, + "page_count": template.page_count, + "margins": { + "top": template.margin_top, + "right": template.margin_right, + "bottom": template.margin_bottom, + "left": template.margin_left, + }, + } + + # Collect logo coordinates + logo_coordinates = self._extract_logo_coordinates(template) + + # Collect header coordinates + header_coordinates = self._extract_region_coordinates(template, "header") + + # Collect footer coordinates + footer_coordinates = self._extract_region_coordinates(template, "footer") + + # Collect table coordinates + table_coordinates = self._extract_table_coordinates(template) + + # Collect cell coordinates + cell_coordinates = self._extract_cell_coordinates(template) + + # Compute fingerprint hash + fingerprint_data = { + "page_dimensions": page_dimensions, + "logo_coordinates": logo_coordinates, + "header_coordinates": header_coordinates, + "footer_coordinates": footer_coordinates, + "table_coordinates": table_coordinates, + "cell_coordinates": cell_coordinates, + } + fingerprint_hash = self._compute_hash(fingerprint_data) + + # Check for existing fingerprint + existing = self.fingerprint_repo.get_by_format_id(template.id) + if existing: + # Update existing + existing.page_dimensions = page_dimensions + existing.logo_coordinates = logo_coordinates + existing.header_coordinates = header_coordinates + existing.footer_coordinates = footer_coordinates + existing.table_coordinates = table_coordinates + existing.cell_coordinates = cell_coordinates + existing.fingerprint_hash = fingerprint_hash + self.db.flush() + self.db.refresh(existing) + return existing + + # Create new fingerprint + fingerprint = self.fingerprint_repo.create_fingerprint( + format_id=template.id, + fingerprint_hash=fingerprint_hash, + page_dimensions=page_dimensions, + logo_coordinates=logo_coordinates, + header_coordinates=header_coordinates, + footer_coordinates=footer_coordinates, + table_coordinates=table_coordinates, + cell_coordinates=cell_coordinates, + ) + + logger.info( + "fingerprint_generated", + template_id=str(template.id), + hash=fingerprint_hash[:16], + ) + + return fingerprint + + def _extract_logo_coordinates(self, template: DocumentFormat) -> dict[str, Any] | None: + """Extract logo image coordinates from template.""" + logos = [ir for ir in template.image_regions if ir.image_type == "logo"] + if not logos: + return None + + return { + "items": [ + { + "page": ir.page_number, + "x": ir.x, + "y": ir.y, + "width": ir.width, + "height": ir.height, + } + for ir in logos + ] + } + + def _extract_region_coordinates( + self, + template: DocumentFormat, + region_type: str, + ) -> dict[str, Any] | None: + """Extract coordinates for a specific region type.""" + regions = [r for r in template.regions if r.region_type == region_type] + if not regions: + return None + + return { + "items": [ + { + "page": r.page_number, + "x": r.x, + "y": r.y, + "width": r.width, + "height": r.height, + } + for r in regions + ] + } + + def _extract_table_coordinates(self, template: DocumentFormat) -> dict[str, Any] | None: + """Extract table coordinates from template.""" + if not template.table_formats: + return None + + # Filter out dummy tables created by manual mappings (width=1000, height=1000) + valid_tables = [tf for tf in template.table_formats if tf.width != 1000.0 and tf.height != 1000.0] + if not valid_tables: + return None + + return { + "items": [ + { + "page": tf.page_number, + "x": tf.x, + "y": tf.y, + "width": tf.width, + "height": tf.height, + "rows": tf.rows, + "columns": tf.columns, + } + for tf in valid_tables + ] + } + + def _extract_cell_coordinates(self, template: DocumentFormat) -> dict[str, Any] | None: + """Extract cell coordinates from template.""" + # For manually mapped templates, prefer the actual mapped regions + field_regions = [r for r in template.regions if r.region_type == "field_mapping"] + if field_regions: + return { + "items": [ + { + "page": r.page_number, + "x": r.x, + "y": r.y, + "width": r.width, + "height": r.height, + } + for r in field_regions + ] + } + + # Otherwise fallback to template cells, but skip if they are all 0,0 (unmapped placeholders) + if not template.cells: + return None + + has_real_coords = any(c.width > 0 or c.height > 0 for c in template.cells) + if not has_real_coords: + return None + + return { + "items": [ + { + "page": c.page_number, + "x": c.x, + "y": c.y, + "width": c.width, + "height": c.height, + "row": c.row_no, + "col": c.column_no, + } + for c in template.cells + ] + } + + def _compute_hash(self, data: dict[str, Any]) -> str: + """Compute a deterministic hash of the fingerprint data.""" + # Normalize coordinates to reduce sensitivity to minor variations + normalized = self._normalize_coordinates(data) + serialized = json.dumps(normalized, sort_keys=True, default=str) + return hashlib.sha256(serialized.encode()).hexdigest() + + def _normalize_coordinates(self, data: Any) -> Any: + """Normalize coordinates by rounding to reduce sensitivity to small variations.""" + if isinstance(data, dict): + return {k: self._normalize_coordinates(v) for k, v in data.items()} + elif isinstance(data, list): + return [self._normalize_coordinates(item) for item in data] + elif isinstance(data, float): + return round(data, 1) + return data + + def compute_similarity( + self, + fingerprint1: TemplateFingerprint, + fingerprint2_data: dict[str, Any], + ) -> float: + """Compute similarity score between a stored fingerprint and new document data.""" + scores: list[float] = [] + weights: list[float] = [] + + # Page dimensions similarity (high weight) + dim_score = self._compare_dimensions( + fingerprint1.page_dimensions, + fingerprint2_data.get("page_dimensions"), + ) + scores.append(dim_score) + weights.append(3.0) + + # Logo coordinates similarity + if fingerprint1.logo_coordinates and fingerprint1.logo_coordinates.get("items"): + logo_score = self._compare_coordinates( + fingerprint1.logo_coordinates, + fingerprint2_data.get("logo_coordinates"), + ) + scores.append(logo_score) + weights.append(2.0) + + # Header coordinates similarity + if fingerprint1.header_coordinates and fingerprint1.header_coordinates.get("items"): + header_score = self._compare_coordinates( + fingerprint1.header_coordinates, + fingerprint2_data.get("header_coordinates"), + ) + scores.append(header_score) + weights.append(2.0) + + # Footer coordinates similarity + if fingerprint1.footer_coordinates and fingerprint1.footer_coordinates.get("items"): + footer_score = self._compare_coordinates( + fingerprint1.footer_coordinates, + fingerprint2_data.get("footer_coordinates"), + ) + scores.append(footer_score) + weights.append(1.5) + + # Table coordinates similarity + if fingerprint1.table_coordinates and fingerprint1.table_coordinates.get("items"): + table_score = self._compare_coordinates( + fingerprint1.table_coordinates, + fingerprint2_data.get("table_coordinates"), + ) + scores.append(table_score) + weights.append(2.0) + + # Cell coordinates similarity + if fingerprint1.cell_coordinates and fingerprint1.cell_coordinates.get("items"): + cell_score = self._compare_coordinates( + fingerprint1.cell_coordinates, + fingerprint2_data.get("cell_coordinates"), + ) + scores.append(cell_score) + weights.append(1.5) + + # Weighted average + total_weight = sum(weights) + if total_weight == 0: + return 0.0 + + weighted_sum = sum(s * w for s, w in zip(scores, weights)) + return weighted_sum / total_weight + + def _compare_dimensions( + self, + dims1: dict[str, Any] | None, + dims2: dict[str, Any] | None, + ) -> float: + """Compare page dimensions similarity.""" + if not dims1 or not dims2: + return 0.0 if (dims1 or dims2) else 1.0 + + width_ratio = min(dims1.get("width", 0), dims2.get("width", 0)) / max( + dims1.get("width", 1), dims2.get("width", 1) + ) + height_ratio = min(dims1.get("height", 0), dims2.get("height", 0)) / max( + dims1.get("height", 1), dims2.get("height", 1) + ) + page_count_match = 1.0 if dims1.get("page_count") == dims2.get("page_count") else 0.5 + + return (width_ratio + height_ratio + page_count_match) / 3.0 + + def _compare_coordinates( + self, + coords1: dict[str, Any] | None, + coords2: dict[str, Any] | None, + ) -> float: + """Compare coordinate sets for similarity.""" + if not coords1 and not coords2: + return 1.0 + if not coords1 or not coords2: + return 0.0 + + items1 = coords1.get("items", []) + items2 = coords2.get("items", []) + + if not items1 and not items2: + return 1.0 + if not items1 or not items2: + return 0.0 + + # Compare number of items + count_ratio = min(len(items1), len(items2)) / max(len(items1), len(items2)) + + # Compare positions of matched items + position_scores = [] + for item1 in items1: + best_match = 0.0 + for item2 in items2: + if item1.get("page") != item2.get("page"): + continue + score = self._compute_bbox_iou(item1, item2) + best_match = max(best_match, score) + position_scores.append(best_match) + + avg_position_score = sum(position_scores) / len(position_scores) if position_scores else 0.0 + + return (count_ratio + avg_position_score) / 2.0 + + def _compute_bbox_iou(self, bbox1: dict[str, Any], bbox2: dict[str, Any]) -> float: + """Compute Intersection over Union for two bounding boxes.""" + x1 = max(bbox1.get("x", 0), bbox2.get("x", 0)) + y1 = max(bbox1.get("y", 0), bbox2.get("y", 0)) + x2 = min( + bbox1.get("x", 0) + bbox1.get("width", 0), + bbox2.get("x", 0) + bbox2.get("width", 0), + ) + y2 = min( + bbox1.get("y", 0) + bbox1.get("height", 0), + bbox2.get("y", 0) + bbox2.get("height", 0), + ) + + intersection = max(0, x2 - x1) * max(0, y2 - y1) + + area1 = bbox1.get("width", 0) * bbox1.get("height", 0) + area2 = bbox2.get("width", 0) * bbox2.get("height", 0) + union = area1 + area2 - intersection + + if union == 0: + return 0.0 + + return intersection / union diff --git a/docengine/app/services/layout_service.py b/docengine/app/services/layout_service.py new file mode 100644 index 0000000..dd2a61d --- /dev/null +++ b/docengine/app/services/layout_service.py @@ -0,0 +1,317 @@ +from __future__ import annotations + +import uuid +from pathlib import Path +from typing import Any + +import cv2 +import numpy as np +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.document import Document, DocumentPage +from app.repositories.document_repository import ( + DocumentPageRepository, + DocumentTableRepository, + DocumentTextBlockRepository, +) +from app.storage.provider import get_storage_provider + +logger = get_logger(__name__) + + +class LayoutService: + """Document layout analysis service using OpenCV-based detection.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.storage = get_storage_provider() + self.page_repo = DocumentPageRepository(db) + self.text_block_repo = DocumentTextBlockRepository(db) + self.table_repo = DocumentTableRepository(db) + + def analyze_document_layout(self, document: Document) -> dict[str, Any]: + """Analyze the layout of all pages in a document.""" + layout_results: dict[str, Any] = {"pages": []} + + for page in document.pages: + page_layout = self._analyze_page_layout(page) + layout_results["pages"].append(page_layout) + + return layout_results + + def _analyze_page_layout(self, page: DocumentPage) -> dict[str, Any]: + """Analyze layout of a single page.""" + result: dict[str, Any] = { + "page_number": page.page_number, + "width": page.width, + "height": page.height, + "tables": [], + "lines": [], + "rectangles": [], + "text_regions": [], + "image_regions": [], + } + + if not page.image_path: + return result + + image_path = self.storage.get_absolute_path(page.image_path) + image = cv2.imread(image_path) + if image is None: + logger.warning("layout_image_read_failed", page_id=str(page.id)) + return result + + gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) + + # Detect lines + result["lines"] = self._detect_lines(gray) + + # Detect rectangles (potential table cells/borders) + result["rectangles"] = self._detect_rectangles(gray) + + # Detect tables + tables = self._detect_tables(gray, image.shape) + result["tables"] = tables + + # Store detected tables in the database + for table_data in tables: + self.table_repo.create_table( + page_id=page.id, + x=table_data["x"], + y=table_data["y"], + width=table_data["width"], + height=table_data["height"], + rows=table_data["rows"], + columns=table_data["columns"], + data=table_data.get("cells"), + ) + + # Detect watermarks + watermark = self._detect_watermark(gray, image.shape) + if watermark: + result["watermark"] = watermark + + return result + + def _detect_lines(self, gray: np.ndarray) -> list[dict[str, Any]]: + """Detect horizontal and vertical lines in the image.""" + lines_detected: list[dict[str, Any]] = [] + + # Apply edge detection + edges = cv2.Canny(gray, 50, 150, apertureSize=3) + + # Detect lines using Hough transform + lines = cv2.HoughLinesP(edges, 1, np.pi / 180, threshold=100, minLineLength=50, maxLineGap=10) + + if lines is not None: + for line in lines: + x1, y1, x2, y2 = line[0] + length = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2) + + # Classify as horizontal or vertical + angle = np.degrees(np.arctan2(y2 - y1, x2 - x1)) + if abs(angle) < 5 or abs(angle - 180) < 5: + orientation = "horizontal" + elif abs(angle - 90) < 5 or abs(angle + 90) < 5: + orientation = "vertical" + else: + orientation = "diagonal" + + lines_detected.append({ + "x1": float(x1), + "y1": float(y1), + "x2": float(x2), + "y2": float(y2), + "length": float(length), + "orientation": orientation, + }) + + return lines_detected + + def _detect_rectangles(self, gray: np.ndarray) -> list[dict[str, Any]]: + """Detect rectangular regions in the image.""" + rectangles: list[dict[str, Any]] = [] + + # Binary threshold + _, binary = cv2.threshold(gray, 200, 255, cv2.THRESH_BINARY_INV) + + # Find contours + contours, _ = cv2.findContours(binary, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) + + for contour in contours: + # Approximate the contour + peri = cv2.arcLength(contour, True) + approx = cv2.approxPolyDP(contour, 0.02 * peri, True) + + # If approximation has 4 vertices, it's likely a rectangle + if len(approx) == 4: + x, y, w, h = cv2.boundingRect(approx) + # Filter out very small or very large rectangles + area = w * h + if area > 500 and w > 10 and h > 10: + rectangles.append({ + "x": float(x), + "y": float(y), + "width": float(w), + "height": float(h), + "area": float(area), + }) + + return rectangles + + def _detect_tables(self, gray: np.ndarray, image_shape: tuple) -> list[dict[str, Any]]: + """Detect table structures using morphological operations.""" + tables: list[dict[str, Any]] = [] + h, w = image_shape[:2] + + # Binary threshold + _, binary = cv2.threshold(gray, 200, 255, cv2.THRESH_BINARY_INV) + + # Detect horizontal lines + horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (max(w // 30, 1), 1)) + horizontal = cv2.morphologyEx(binary, cv2.MORPH_OPEN, horizontal_kernel, iterations=2) + + # Detect vertical lines + vertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, max(h // 30, 1))) + vertical = cv2.morphologyEx(binary, cv2.MORPH_OPEN, vertical_kernel, iterations=2) + + # Combine horizontal and vertical lines to find intersections + table_mask = cv2.add(horizontal, vertical) + + # Find contours of table regions + contours, _ = cv2.findContours(table_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) + + for contour in contours: + x, y, cw, ch = cv2.boundingRect(contour) + area = cw * ch + + # Filter: table should be reasonably sized + if area < 5000 or cw < 50 or ch < 30: + continue + + # Estimate rows and columns + rows, columns = self._estimate_table_dimensions( + table_mask[y:y+ch, x:x+cw], cw, ch + ) + + if rows >= 1 and columns >= 1: + # Extract cell contents + cells = self._extract_table_cells( + gray[y:y+ch, x:x+cw], rows, columns, cw, ch + ) + + tables.append({ + "x": float(x), + "y": float(y), + "width": float(cw), + "height": float(ch), + "rows": rows, + "columns": columns, + "cells": cells, + }) + + return tables + + def _estimate_table_dimensions( + self, + table_region: np.ndarray, + width: int, + height: int, + ) -> tuple[int, int]: + """Estimate the number of rows and columns in a table region.""" + # Project horizontal lines + h_projection = np.sum(table_region, axis=1) + h_peaks = self._count_peaks(h_projection, height) + + # Project vertical lines + v_projection = np.sum(table_region, axis=0) + v_peaks = self._count_peaks(v_projection, width) + + rows = max(1, h_peaks - 1) + columns = max(1, v_peaks - 1) + + return rows, columns + + def _count_peaks(self, projection: np.ndarray, total_length: int) -> int: + """Count significant peaks in a projection array.""" + if len(projection) == 0: + return 0 + + threshold = np.max(projection) * 0.3 + above_threshold = projection > threshold + + # Count transitions from below to above threshold + peaks = 0 + in_peak = False + for val in above_threshold: + if val and not in_peak: + peaks += 1 + in_peak = True + elif not val: + in_peak = False + + return peaks + + def _extract_table_cells( + self, + table_gray: np.ndarray, + rows: int, + columns: int, + width: int, + height: int, + ) -> dict[str, Any]: + """Extract cell structure data from a table region.""" + cell_height = height / max(rows, 1) + cell_width = width / max(columns, 1) + + cells: dict[str, Any] = {"rows": rows, "columns": columns, "data": []} + + for r in range(rows): + row_data = [] + for c in range(columns): + cell_x = int(c * cell_width) + cell_y = int(r * cell_height) + cell_w = int(cell_width) + cell_h = int(cell_height) + + row_data.append({ + "row": r, + "col": c, + "x": cell_x, + "y": cell_y, + "width": cell_w, + "height": cell_h, + }) + cells["data"].append(row_data) + + return cells + + def _detect_watermark(self, gray: np.ndarray, image_shape: tuple) -> dict[str, Any] | None: + """Detect potential watermark regions.""" + h, w = image_shape[:2] + + # Look for semi-transparent or light text in the center region + center_region = gray[h // 4 : 3 * h // 4, w // 4 : 3 * w // 4] + + # Apply adaptive threshold to find light text + _, binary = cv2.threshold(center_region, 230, 255, cv2.THRESH_BINARY) + + # Find contours in the center region + contours, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) + + for contour in contours: + x, y, cw, ch = cv2.boundingRect(contour) + area = cw * ch + # Watermark typically covers a significant portion of the center + center_area = (w // 2) * (h // 2) + if area > center_area * 0.1: + return { + "x": float(x + w // 4), + "y": float(y + h // 4), + "width": float(cw), + "height": float(ch), + "detected": True, + } + + return None diff --git a/docengine/app/services/matching_service.py b/docengine/app/services/matching_service.py new file mode 100644 index 0000000..a39c8aa --- /dev/null +++ b/docengine/app/services/matching_service.py @@ -0,0 +1,217 @@ +from __future__ import annotations + +import uuid +from typing import Any + +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.document import Document, TemplateMatch +from app.models.template import DocumentFormat +from app.repositories.document_repository import DocumentRepository, TemplateMatchRepository +from app.repositories.template_repository import TemplateFingerprintRepository, TemplateRepository +from app.services.fingerprint_service import FingerprintService + +logger = get_logger(__name__) + + +class MatchingService: + """Match documents against existing templates using fingerprint comparison.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.doc_repo = DocumentRepository(db) + self.template_repo = TemplateRepository(db) + self.match_repo = TemplateMatchRepository(db) + self.fingerprint_repo = TemplateFingerprintRepository(db) + self.fingerprint_service = FingerprintService(db) + + def match_document( + self, + document_id: uuid.UUID, + min_confidence: float = 0.75, + max_results: int = 5, + ) -> list[TemplateMatch]: + """Match a document against all existing templates.""" + document = self.doc_repo.get_with_pages(document_id) + if not document: + raise ValueError(f"Document '{document_id}' not found") + + if not document.pages: + raise ValueError(f"Document '{document_id}' has no processed pages") + + # Generate document fingerprint data + doc_fingerprint_data = self._build_document_fingerprint(document) + + # Get all templates with fingerprints + templates = self.template_repo.get_all_with_fingerprints() + fingerprints = self.fingerprint_repo.get_all_fingerprints() + + # Map format_id -> fingerprint + fp_map = {fp.format_id: fp for fp in fingerprints} + + matches: list[tuple[DocumentFormat, float, dict[str, Any]]] = [] + + for template in templates: + fp = fp_map.get(template.id) + if not fp: + continue + + score = self.fingerprint_service.compute_similarity(fp, doc_fingerprint_data) + if score >= min_confidence: + match_details = { + "page_dimensions_score": self.fingerprint_service._compare_dimensions( + fp.page_dimensions, doc_fingerprint_data.get("page_dimensions") + ), + "logo_score": self.fingerprint_service._compare_coordinates( + fp.logo_coordinates, doc_fingerprint_data.get("logo_coordinates") + ), + "header_score": self.fingerprint_service._compare_coordinates( + fp.header_coordinates, doc_fingerprint_data.get("header_coordinates") + ), + "footer_score": self.fingerprint_service._compare_coordinates( + fp.footer_coordinates, doc_fingerprint_data.get("footer_coordinates") + ), + "table_score": self.fingerprint_service._compare_coordinates( + fp.table_coordinates, doc_fingerprint_data.get("table_coordinates") + ), + "cell_score": self.fingerprint_service._compare_coordinates( + fp.cell_coordinates, doc_fingerprint_data.get("cell_coordinates") + ), + } + matches.append((template, score, match_details)) + + # Sort by score descending + matches.sort(key=lambda x: x[1], reverse=True) + matches = matches[:max_results] + + # Store match results + result_matches: list[TemplateMatch] = [] + for idx, (template, score, details) in enumerate(matches): + template_match = self.match_repo.create_match( + document_id=document_id, + format_id=template.id, + confidence_score=score, + match_details=details, + selected=(idx == 0), # Auto-select best match + ) + result_matches.append(template_match) + + logger.info( + "document_matched", + document_id=str(document_id), + matches_found=len(result_matches), + best_score=result_matches[0].confidence_score if result_matches else 0.0, + ) + + return result_matches + + def _build_document_fingerprint(self, document: Document) -> dict[str, Any]: + """Build fingerprint data from a document for comparison.""" + first_page = document.pages[0] if document.pages else None + + page_dimensions = None + if first_page: + page_dimensions = { + "width": first_page.width, + "height": first_page.height, + "page_count": document.page_count or len(document.pages), + } + + # Extract logo coordinates from images + logo_coordinates = None + logos = [] + for page in document.pages: + for img in page.images: + if img.image_type == "logo": + logos.append({ + "page": page.page_number, + "x": img.x, + "y": img.y, + "width": img.width, + "height": img.height, + }) + if logos: + logo_coordinates = {"items": logos} + + # Extract header coordinates + header_coordinates = None + headers = [] + for page in document.pages: + header_blocks = [b for b in page.text_blocks if b.block_type == "header"] + if header_blocks: + min_x = min(b.x for b in header_blocks) + min_y = min(b.y for b in header_blocks) + max_x = max(b.x + b.width for b in header_blocks) + max_y = max(b.y + b.height for b in header_blocks) + headers.append({ + "page": page.page_number, + "x": min_x, + "y": min_y, + "width": max_x - min_x, + "height": max_y - min_y, + }) + if headers: + header_coordinates = {"items": headers} + + # Extract footer coordinates + footer_coordinates = None + footers = [] + for page in document.pages: + footer_blocks = [b for b in page.text_blocks if b.block_type == "footer"] + if footer_blocks: + min_x = min(b.x for b in footer_blocks) + min_y = min(b.y for b in footer_blocks) + max_x = max(b.x + b.width for b in footer_blocks) + max_y = max(b.y + b.height for b in footer_blocks) + footers.append({ + "page": page.page_number, + "x": min_x, + "y": min_y, + "width": max_x - min_x, + "height": max_y - min_y, + }) + if footers: + footer_coordinates = {"items": footers} + + # Extract table coordinates + table_coordinates = None + tables = [] + for page in document.pages: + for table in page.tables: + tables.append({ + "page": page.page_number, + "x": table.x, + "y": table.y, + "width": table.width, + "height": table.height, + "rows": table.rows, + "columns": table.columns, + }) + if tables: + table_coordinates = {"items": tables} + + # Extract cell coordinates from text blocks + cell_coordinates = None + cells = [] + for page in document.pages: + for block in page.text_blocks: + if block.block_type == "text": + cells.append({ + "page": page.page_number, + "x": block.x, + "y": block.y, + "width": block.width, + "height": block.height, + }) + if cells: + cell_coordinates = {"items": cells} + + return { + "page_dimensions": page_dimensions, + "logo_coordinates": logo_coordinates, + "header_coordinates": header_coordinates, + "footer_coordinates": footer_coordinates, + "table_coordinates": table_coordinates, + "cell_coordinates": cell_coordinates, + } diff --git a/docengine/app/services/ocr_service.py b/docengine/app/services/ocr_service.py new file mode 100644 index 0000000..8e9c552 --- /dev/null +++ b/docengine/app/services/ocr_service.py @@ -0,0 +1,210 @@ +from __future__ import annotations + +import uuid +from pathlib import Path + +import cv2 +import numpy as np +from paddleocr import PaddleOCR +from sqlalchemy.orm import Session + +from app.core.config import settings +from app.core.logging_config import get_logger +from app.models.document import Document, DocumentPage +from app.repositories.document_repository import ( + DocumentPageRepository, + DocumentRepository, + DocumentTextBlockRepository, +) +from app.storage.provider import get_storage_provider + +logger = get_logger(__name__) + +_ocr_instance: PaddleOCR | None = None + + +def get_ocr_engine() -> PaddleOCR: + """Get or create singleton PaddleOCR instance.""" + global _ocr_instance + if _ocr_instance is None: + _ocr_instance = PaddleOCR( + use_angle_cls=True, + lang=settings.ocr_language, + use_gpu=settings.ocr_use_gpu, + show_log=False, + det_db_thresh=0.3, + det_db_box_thresh=0.5, + rec_batch_num=6, + ) + return _ocr_instance + + +class OCRService: + """OCR processing service using PaddleOCR for scanned documents.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.storage = get_storage_provider() + self.doc_repo = DocumentRepository(db) + self.page_repo = DocumentPageRepository(db) + self.text_block_repo = DocumentTextBlockRepository(db) + + def process_image(self, document: Document) -> Document: + """Process a scanned image document with OCR.""" + file_path = self.storage.get_absolute_path(document.storage_path) + image = cv2.imread(file_path) + if image is None: + raise ValueError(f"Failed to read image: {file_path}") + + height, width = image.shape[:2] + document.page_count = 1 + document.is_scanned = True + + # Save page image + image_filename = f"{document.id}_page_1.png" + image_bytes = cv2.imencode(".png", image)[1].tobytes() + image_path = self.storage.save_file(image_bytes, "images", image_filename) + + # Create page record + doc_page = self.page_repo.create_page( + document_id=document.id, + page_number=1, + width=float(width), + height=float(height), + image_path=image_path, + ) + + # Run OCR + self._run_ocr_on_page(doc_page, file_path) + + return document + + def process_scanned_pdf(self, document: Document) -> Document: + """Process a scanned PDF document - convert pages to images and OCR each.""" + file_path = self.storage.get_absolute_path(document.storage_path) + + try: + from pdf2image import convert_from_path + images = convert_from_path(file_path, dpi=300) + except Exception as e: + logger.error("pdf_to_image_failed", document_id=str(document.id), error=str(e)) + raise + + document.page_count = len(images) + document.is_scanned = True + + for page_num, pil_image in enumerate(images, start=1): + # Convert PIL to OpenCV format + np_image = np.array(pil_image) + cv_image = cv2.cvtColor(np_image, cv2.COLOR_RGB2BGR) + height, width = cv_image.shape[:2] + + # Save page image + image_filename = f"{document.id}_page_{page_num}.png" + image_bytes = cv2.imencode(".png", cv_image)[1].tobytes() + image_path = self.storage.save_file(image_bytes, "images", image_filename) + + # Create page record + doc_page = self.page_repo.create_page( + document_id=document.id, + page_number=page_num, + width=float(width), + height=float(height), + image_path=image_path, + ) + + # Run OCR on saved image + temp_path = self.storage.get_absolute_path(image_path) + self._run_ocr_on_page(doc_page, temp_path) + + return document + + def _run_ocr_on_page(self, doc_page: DocumentPage, image_path: str) -> None: + """Run PaddleOCR on a single page image and store results.""" + ocr = get_ocr_engine() + + try: + results = ocr.ocr(image_path, cls=True) + except Exception as e: + logger.error("ocr_failed", page_id=str(doc_page.id), error=str(e)) + return + + if not results or not results[0]: + logger.info("ocr_no_results", page_id=str(doc_page.id)) + return + + full_text_parts = [] + sequence = 0 + + for line in results[0]: + if not line or len(line) < 2: + continue + + bbox_points = line[0] # List of 4 corner points + text_info = line[1] # (text, confidence) + + text = text_info[0] if isinstance(text_info, (list, tuple)) else str(text_info) + confidence = float(text_info[1]) if isinstance(text_info, (list, tuple)) and len(text_info) > 1 else 0.0 + + if not text.strip(): + continue + + # Convert bbox points to x, y, width, height + xs = [p[0] for p in bbox_points] + ys = [p[1] for p in bbox_points] + x = min(xs) + y = min(ys) + width = max(xs) - x + height = max(ys) - y + + # Determine block type based on position + block_type = self._classify_text_block( + y, height, doc_page.height, text + ) + + self.text_block_repo.create_text_block( + page_id=doc_page.id, + text=text, + x=x, + y=y, + width=width, + height=height, + confidence=confidence, + block_type=block_type, + sequence=sequence, + ) + full_text_parts.append(text) + sequence += 1 + + # Update page text content + doc_page.text_content = "\n".join(full_text_parts) + + def _classify_text_block( + self, + y: float, + height: float, + page_height: float, + text: str, + ) -> str: + """Classify a text block as header, footer, watermark, or regular text.""" + if page_height <= 0: + return "text" + + relative_y = y / page_height + + # Header: top 10% + if relative_y < 0.10: + return "header" + + # Footer: bottom 10% + if relative_y > 0.90: + return "footer" + + # Watermark detection heuristic: large text in center + if 0.3 < relative_y < 0.7 and height > page_height * 0.05: + # Check for common watermark words + watermark_keywords = {"confidential", "draft", "copy", "sample", "watermark", "void"} + if text.strip().lower() in watermark_keywords: + return "watermark" + + return "text" diff --git a/docengine/app/services/pdf_service.py b/docengine/app/services/pdf_service.py new file mode 100644 index 0000000..cff12e5 --- /dev/null +++ b/docengine/app/services/pdf_service.py @@ -0,0 +1,337 @@ +from __future__ import annotations + +import uuid +from pathlib import Path + +import cv2 +import fitz # PyMuPDF +import numpy as np +from sqlalchemy.orm import Session + +from app.core.config import settings +from app.core.logging_config import get_logger +from app.models.document import Document, DocumentImage, DocumentPage, DocumentTable, DocumentTextBlock +from app.repositories.document_repository import ( + DocumentImageRepository, + DocumentPageRepository, + DocumentRepository, + DocumentTableRepository, + DocumentTextBlockRepository, +) +from app.storage.provider import get_storage_provider + +logger = get_logger(__name__) + + +class NativePDFService: + """Extract text, fonts, images, and layout from native (non-scanned) PDFs using PyMuPDF.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.storage = get_storage_provider() + self.doc_repo = DocumentRepository(db) + self.page_repo = DocumentPageRepository(db) + self.text_block_repo = DocumentTextBlockRepository(db) + self.image_repo = DocumentImageRepository(db) + self.table_repo = DocumentTableRepository(db) + + def process_pdf(self, document: Document) -> Document: + """Process a native PDF document, extracting all content.""" + file_path = self.storage.get_absolute_path(document.storage_path) + + try: + pdf_doc = fitz.open(file_path) + except Exception as e: + logger.error("pdf_open_failed", document_id=str(document.id), error=str(e)) + raise + + document.page_count = len(pdf_doc) + document.is_scanned = self._is_scanned_pdf(pdf_doc) + + for page_num in range(len(pdf_doc)): + page = pdf_doc[page_num] + self._process_page(document, page, page_num + 1) + + pdf_doc.close() + return document + + def _is_scanned_pdf(self, pdf_doc: fitz.Document) -> bool: + """Determine if a PDF is scanned (image-based) or native.""" + total_text_chars = 0 + total_images = 0 + for page_num in range(min(len(pdf_doc), 3)): + page = pdf_doc[page_num] + text = page.get_text("text") + total_text_chars += len(text.strip()) + total_images += len(page.get_images(full=True)) + + # If very little text but has images, likely scanned + if total_text_chars < 50 and total_images > 0: + return True + return False + + def _process_page(self, document: Document, page: fitz.Page, page_number: int) -> DocumentPage: + """Process a single PDF page.""" + rect = page.rect + width = rect.width + height = rect.height + + # Save page as image for reference + pix = page.get_pixmap(dpi=150) + image_filename = f"{document.id}_page_{page_number}.png" + image_data = pix.tobytes("png") + image_path = self.storage.save_file(image_data, "images", image_filename) + + # Get full text content + text_content = page.get_text("text") + + doc_page = self.page_repo.create_page( + document_id=document.id, + page_number=page_number, + width=width, + height=height, + image_path=image_path, + text_content=text_content, + ) + + # Extract text blocks with font information + self._extract_text_blocks(doc_page, page) + + # Extract images + self._extract_images(doc_page, page, document) + + # Detect headers and footers + self._detect_headers_footers(doc_page, page) + + return doc_page + + def _extract_text_blocks(self, doc_page: DocumentPage, page: fitz.Page) -> None: + """Extract text blocks with positioning and font information.""" + blocks = page.get_text("rawdict")["blocks"] + sequence = 0 + chunks = [] + + for block in blocks: + if block["type"] != 0: + continue + + current_chunk = None + space_count = 0 + + for line in block.get("lines", []): + for span in line.get("spans", []): + font_size = span.get("size", 12.0) + + # Ignore massive text (like diagonal watermarks) + if font_size > 60: + continue + + space_threshold = font_size * 1.5 + + font_family = span.get("font", None) + color_int = span.get("color", 0) + font_color = f"#{color_int:06x}" if isinstance(color_int, int) else None + + flags = span.get("flags", 0) + styles = [] + if flags & 1: styles.append("superscript") + if flags & 2: styles.append("italic") + if flags & 4: styles.append("serif") + if flags & 8: styles.append("monospace") + if flags & 16: styles.append("bold") + font_style = ",".join(styles) if styles else "regular" + + font_info = { + "family": font_family, + "size": font_size, + "color": font_color, + "style": font_style + } + + for char in span.get("chars", []): + c = char["c"] + bbox = char["bbox"] + + if c == ' ': + space_count += 1 + if space_count >= 2: + if current_chunk and current_chunk["text"].strip(): + chunks.append(current_chunk) + current_chunk = None + elif current_chunk: + current_chunk["text"] += c + current_chunk["bbox"][2] = max(current_chunk["bbox"][2], bbox[2]) + current_chunk["bbox"][3] = max(current_chunk["bbox"][3], bbox[3]) + continue + else: + space_count = 0 + + if current_chunk is None: + current_chunk = {"text": c, "bbox": list(bbox), "font_info": font_info} + continue + + prev_x1 = current_chunk["bbox"][2] + distance = bbox[0] - prev_x1 + + if distance > space_threshold: + if current_chunk["text"].strip(): + chunks.append(current_chunk) + current_chunk = {"text": c, "bbox": list(bbox), "font_info": font_info} + else: + current_chunk["text"] += c + current_chunk["bbox"][2] = max(current_chunk["bbox"][2], bbox[2]) + current_chunk["bbox"][3] = max(current_chunk["bbox"][3], bbox[3]) + current_chunk["bbox"][1] = min(current_chunk["bbox"][1], bbox[1]) + current_chunk["bbox"][0] = min(current_chunk["bbox"][0], bbox[0]) + + if current_chunk and current_chunk["text"].strip(): + chunks.append(current_chunk) + current_chunk = None + + for chunk in chunks: + bbox = chunk["bbox"] + font_info = chunk["font_info"] + + self.text_block_repo.create_text_block( + page_id=doc_page.id, + text=chunk["text"].strip(), + x=bbox[0], + y=bbox[1], + width=bbox[2] - bbox[0], + height=bbox[3] - bbox[1], + font_family=font_info["family"], + font_size=font_info["size"], + font_color=font_info["color"], + font_style=font_info["style"], + block_type="text", + sequence=sequence, + ) + sequence += 1 + + def _extract_images(self, doc_page: DocumentPage, page: fitz.Page, document: Document) -> None: + """Extract embedded images from a PDF page.""" + image_list = page.get_images(full=True) + + for img_index, img_info in enumerate(image_list): + xref = img_info[0] + try: + base_image = page.parent.extract_image(xref) + if not base_image: + continue + + image_bytes = base_image["image"] + ext = base_image.get("ext", "png") + img_filename = f"{document.id}_page_{doc_page.page_number}_img_{img_index}.{ext}" + img_storage_path = self.storage.save_file(image_bytes, "images", img_filename) + + # Try to get image position on page + img_rects = page.get_image_rects(xref) + if img_rects: + rect = img_rects[0] + x, y, x1, y1 = rect.x0, rect.y0, rect.x1, rect.y1 + else: + x, y, x1, y1 = 0, 0, base_image.get("width", 100), base_image.get("height", 100) + + # Determine image type based on position + page_height = doc_page.height + page_width = doc_page.width + image_type = self._classify_image_type(x, y, x1, y1, page_width, page_height) + + self.image_repo.create_image( + page_id=doc_page.id, + x=x, + y=y, + width=x1 - x, + height=y1 - y, + image_path=img_storage_path, + image_type=image_type, + ) + + except Exception as e: + logger.warning( + "image_extraction_failed", + page_id=str(doc_page.id), + img_index=img_index, + error=str(e), + ) + + def _classify_image_type( + self, + x: float, + y: float, + x1: float, + y1: float, + page_width: float, + page_height: float, + ) -> str: + """Classify an image as logo, figure, background, or stamp based on position and size.""" + width = x1 - x + height = y1 - y + area_ratio = (width * height) / (page_width * page_height) if page_width > 0 and page_height > 0 else 0 + + # Background: covers most of the page + if area_ratio > 0.8: + return "background" + + # Logo: small image in top portion + if y < page_height * 0.15 and area_ratio < 0.05: + return "logo" + + # Stamp: small image in bottom-right + if x > page_width * 0.6 and y > page_height * 0.7 and area_ratio < 0.05: + return "stamp" + + return "figure" + + def _detect_headers_footers(self, doc_page: DocumentPage, page: fitz.Page) -> None: + """Detect header and footer regions based on vertical position.""" + page_height = page.rect.height + header_threshold = page_height * 0.1 + footer_threshold = page_height * 0.9 + + blocks = page.get_text("dict")["blocks"] + header_seq = 0 + footer_seq = 0 + + for block in blocks: + if block["type"] != 0: + continue + + bbox = block["bbox"] + block_y = bbox[1] + + text_parts = [] + for line in block.get("lines", []): + for span in line.get("spans", []): + t = span.get("text", "").strip() + if t: + text_parts.append(t) + + full_text = " ".join(text_parts) + if not full_text.strip(): + continue + + if block_y < header_threshold: + self.text_block_repo.create_text_block( + page_id=doc_page.id, + text=full_text, + x=bbox[0], + y=bbox[1], + width=bbox[2] - bbox[0], + height=bbox[3] - bbox[1], + block_type="header", + sequence=header_seq, + ) + header_seq += 1 + elif block_y > footer_threshold: + self.text_block_repo.create_text_block( + page_id=doc_page.id, + text=full_text, + x=bbox[0], + y=bbox[1], + width=bbox[2] - bbox[0], + height=bbox[3] - bbox[1], + block_type="footer", + sequence=footer_seq, + ) + footer_seq += 1 diff --git a/docengine/app/services/reconstruction_service.py b/docengine/app/services/reconstruction_service.py new file mode 100644 index 0000000..e075ce6 --- /dev/null +++ b/docengine/app/services/reconstruction_service.py @@ -0,0 +1,455 @@ +from __future__ import annotations + +import os +import uuid +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from reportlab.lib import colors +from reportlab.lib.pagesizes import letter +from reportlab.lib.styles import ParagraphStyle, getSampleStyleSheet +from reportlab.lib.units import inch, mm +from reportlab.pdfbase import pdfmetrics +from reportlab.pdfbase.ttfonts import TTFont +from reportlab.platypus import ( + BaseDocTemplate, + Frame, + Image, + NextPageTemplate, + PageBreak, + PageTemplate, + Paragraph, + SimpleDocTemplate, + Spacer, + Table, + TableStyle, +) +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.template import DocumentFormat +from app.schemas.template import TemplateRenderResponse +from app.storage.provider import get_storage_provider + +logger = get_logger(__name__) + + +class ReconstructionService: + """Reconstruct documents from templates using ReportLab.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.storage = get_storage_provider() + self.styles = getSampleStyleSheet() + self._register_fonts() + + def _register_fonts(self) -> None: + """Register additional fonts if available.""" + # ReportLab includes Helvetica, Times-Roman, Courier by default + # Custom fonts can be registered here + pass + + def render_template( + self, + template: DocumentFormat, + data: dict[str, Any], + output_filename: str | None = None, + images: dict[str, str] | None = None, + ) -> TemplateRenderResponse: + """Render a template to PDF with supplied data.""" + if output_filename is None: + output_filename = f"{template.name}_{uuid.uuid4().hex[:8]}.pdf" + + if not output_filename.endswith(".pdf"): + output_filename += ".pdf" + + # Determine output path + output_storage_path = f"rendered/{output_filename}" + absolute_output_path = self.storage.get_absolute_path(output_storage_path) + + # Ensure the rendered directory exists + os.makedirs(os.path.dirname(absolute_output_path), exist_ok=True) + + # Build the PDF + self._build_pdf(template, data, absolute_output_path, images) + + file_size = os.path.getsize(absolute_output_path) + + logger.info( + "template_rendered", + template_id=str(template.id), + output=output_filename, + size=file_size, + ) + + return TemplateRenderResponse( + output_path=output_storage_path, + filename=output_filename, + file_size=file_size, + page_count=template.page_count, + rendered_at=datetime.now(UTC), + ) + + def _build_pdf( + self, + template: DocumentFormat, + data: dict[str, Any], + output_path: str, + images: dict[str, str] | None = None, + ) -> None: + """Build a PDF document from template definition.""" + page_width = template.page_width + page_height = template.page_height + + doc = SimpleDocTemplate( + output_path, + pagesize=(page_width, page_height), + topMargin=template.margin_top, + rightMargin=template.margin_right, + bottomMargin=template.margin_bottom, + leftMargin=template.margin_left, + ) + + # Build story (content elements) + story: list[Any] = [] + + for page_num in range(1, template.page_count + 1): + if page_num > 1: + story.append(PageBreak()) + + # Add page content + page_elements = self._build_page_content(template, page_num, data, images) + story.extend(page_elements) + + # Build with watermark/header/footer callbacks + def on_page(canvas, doc_obj): # noqa: ANN001, ANN202 + self._draw_watermarks(canvas, template, doc_obj.page) + self._draw_headers_footers(canvas, template, doc_obj.page, page_width, page_height) + + def on_page_later(canvas, doc_obj): # noqa: ANN001, ANN202 + self._draw_watermarks(canvas, template, doc_obj.page) + self._draw_headers_footers(canvas, template, doc_obj.page, page_width, page_height) + + doc.build(story, onFirstPage=on_page, onLaterPages=on_page_later) + + def _build_page_content( + self, + template: DocumentFormat, + page_number: int, + data: dict[str, Any], + images: dict[str, str] | None = None, + ) -> list[Any]: + """Build content elements for a specific page.""" + elements: list[Any] = [] + + # Get cells for this page, sorted by sequence + page_cells = sorted( + [c for c in template.cells if c.page_number == page_number], + key=lambda c: c.sequence, + ) + + # Get table formats for this page + page_tables = [t for t in template.table_formats if t.page_number == page_number] + + # Get image regions for this page + page_images = [i for i in template.image_regions if i.page_number == page_number] + + # Add static and dynamic text cells + for cell in page_cells: + text = self._resolve_cell_text(cell, data) + if text: + style = self._create_cell_style(cell) + para = Paragraph(text, style) + elements.append(para) + elements.append(Spacer(1, 2)) + + # Add tables + for table_format in page_tables: + table_element = self._build_table(table_format, data) + if table_element: + elements.append(table_element) + elements.append(Spacer(1, 6)) + + # Add images + for img_region in page_images: + img_element = self._build_image(img_region, images) + if img_element: + elements.append(img_element) + elements.append(Spacer(1, 6)) + + if not elements: + elements.append(Spacer(1, 12)) + + return elements + + def _resolve_cell_text(self, cell: Any, data: dict[str, Any]) -> str: + """Resolve cell text from static content or dynamic data.""" + if cell.is_dynamic and cell.field_name: + value = data.get(cell.field_name, "") + return str(value) if value else "" + return cell.static_text or "" + + def _create_cell_style(self, cell: Any) -> ParagraphStyle: + """Create a ReportLab paragraph style from cell properties.""" + font_name = "Helvetica" + if cell.font_family: + family = cell.font_family.lower() + if "times" in family or "serif" in family: + font_name = "Times-Roman" + elif "courier" in family or "mono" in family: + font_name = "Courier" + + font_size = cell.font_size or 10 + if cell.font_style and "bold" in (cell.font_style or ""): + if font_name == "Helvetica": + font_name = "Helvetica-Bold" + elif font_name == "Times-Roman": + font_name = "Times-Bold" + elif font_name == "Courier": + font_name = "Courier-Bold" + + text_color = colors.black + if cell.font_color: + try: + text_color = colors.HexColor(cell.font_color) + except (ValueError, TypeError): + text_color = colors.black + + alignment_map = {"left": 0, "center": 1, "right": 2, "justify": 4} + alignment = alignment_map.get(cell.alignment, 0) + + style = ParagraphStyle( + name=f"cell_{cell.id}", + parent=self.styles["Normal"], + fontName=font_name, + fontSize=font_size, + textColor=text_color, + alignment=alignment, + leading=font_size * 1.2, + spaceBefore=cell.padding_top, + spaceAfter=cell.padding_bottom, + leftIndent=cell.padding_left, + rightIndent=cell.padding_right, + ) + + return style + + def _build_table(self, table_format: Any, data: dict[str, Any]) -> Table | None: + """Build a ReportLab table from a table format definition.""" + rows = table_format.rows + columns = table_format.columns + + if rows <= 0 or columns <= 0: + return None + + # Build table data + table_data: list[list[str]] = [] + + # Header row + if table_format.table_columns: + header_row = [col.header_text or f"Col {col.column_index + 1}" for col in table_format.table_columns] + table_data.append(header_row) + else: + table_data.append([f"Column {i + 1}" for i in range(columns)]) + + # Data rows from supplied data + table_field_name = f"table_{table_format.id}" + table_rows_data = data.get(table_field_name, data.get("table_data", [])) + + if isinstance(table_rows_data, list): + for row_data in table_rows_data: + if isinstance(row_data, list): + # Pad or trim to match column count + row = row_data[:columns] + while len(row) < columns: + row.append("") + table_data.append([str(v) for v in row]) + elif isinstance(row_data, dict): + row = [] + for col in table_format.table_columns: + key = col.header_text or f"col_{col.column_index}" + row.append(str(row_data.get(key, ""))) + table_data.append(row) + + # If no data rows, add empty rows + if len(table_data) <= 1: + for _ in range(max(rows - 1, 1)): + table_data.append([""] * columns) + + # Determine column widths + col_widths = [] + if table_format.table_columns: + col_widths = [col.width for col in table_format.table_columns] + else: + col_width = table_format.width / columns + col_widths = [col_width] * columns + + # Build table + table = Table(table_data, colWidths=col_widths) + + # Apply table style + border_color = colors.black + if table_format.border_color: + try: + border_color = colors.HexColor(table_format.border_color) + except (ValueError, TypeError): + pass + + style_commands = [ + ("GRID", (0, 0), (-1, -1), table_format.border_width, border_color), + ("FONTNAME", (0, 0), (-1, 0), "Helvetica-Bold"), + ("FONTSIZE", (0, 0), (-1, -1), 9), + ("ALIGN", (0, 0), (-1, -1), "LEFT"), + ("VALIGN", (0, 0), (-1, -1), "MIDDLE"), + ("TOPPADDING", (0, 0), (-1, -1), 4), + ("BOTTOMPADDING", (0, 0), (-1, -1), 4), + ("LEFTPADDING", (0, 0), (-1, -1), 4), + ("RIGHTPADDING", (0, 0), (-1, -1), 4), + ] + + # Header row background + if table_format.table_rows: + for row in table_format.table_rows: + if row.is_header and row.background_color: + try: + bg_color = colors.HexColor(row.background_color) + style_commands.append( + ("BACKGROUND", (0, row.row_index), (-1, row.row_index), bg_color) + ) + except (ValueError, TypeError): + pass + else: + style_commands.append(("BACKGROUND", (0, 0), (-1, 0), colors.HexColor("#E0E0E0"))) + + table.setStyle(TableStyle(style_commands)) + + return table + + def _build_image(self, img_region: Any, images: dict[str, str] | None = None) -> Image | None: + """Build a ReportLab image from an image region definition.""" + image_path = None + + # Check dynamic images first + if not img_region.is_static and img_region.field_name and images: + image_path = images.get(img_region.field_name) + + # Fall back to stored image + if not image_path and img_region.image_path: + try: + image_path = self.storage.get_absolute_path(img_region.image_path) + except Exception: + image_path = None + + if not image_path or not os.path.exists(image_path): + return None + + try: + img = Image(image_path, width=img_region.width, height=img_region.height) + return img + except Exception as e: + logger.warning("image_build_failed", error=str(e), path=image_path) + return None + + def _draw_watermarks(self, canvas: Any, template: DocumentFormat, current_page: int) -> None: + """Draw watermarks on the canvas.""" + for watermark in template.watermarks: + # Apply to all pages if page_number is None, or specific page + if watermark.page_number is not None and watermark.page_number != current_page: + continue + + canvas.saveState() + + # Set opacity + canvas.setFillAlpha(watermark.opacity) + + if watermark.text: + # Text watermark + font_name = "Helvetica" + if watermark.font_family: + family = watermark.font_family.lower() + if "times" in family: + font_name = "Times-Roman" + elif "courier" in family: + font_name = "Courier" + + font_size = watermark.font_size or 48 + + if watermark.font_color: + try: + canvas.setFillColor(colors.HexColor(watermark.font_color)) + except (ValueError, TypeError): + canvas.setFillColor(colors.grey) + else: + canvas.setFillColor(colors.grey) + + canvas.setFont(font_name, font_size) + + # Position and rotate + canvas.translate( + watermark.x + watermark.width / 2, + watermark.y + watermark.height / 2, + ) + canvas.rotate(watermark.rotation) + canvas.drawCentredString(0, 0, watermark.text) + + elif watermark.image_path: + # Image watermark + try: + img_path = self.storage.get_absolute_path(watermark.image_path) + if os.path.exists(img_path): + canvas.drawImage( + img_path, + watermark.x, + watermark.y, + width=watermark.width, + height=watermark.height, + mask="auto", + ) + except Exception as e: + logger.warning("watermark_image_failed", error=str(e)) + + canvas.restoreState() + + def _draw_headers_footers( + self, + canvas: Any, + template: DocumentFormat, + current_page: int, + page_width: float, + page_height: float, + ) -> None: + """Draw header and footer regions on the canvas.""" + for region in template.regions: + if region.page_number != current_page: + continue + + content = region.content or {} + blocks = content.get("blocks", []) + + canvas.saveState() + + for block in blocks: + text = block.get("text", "") + if not text: + continue + + x = block.get("x", region.x) + y = page_height - block.get("y", region.y) - block.get("height", 12) + font_family = block.get("font_family", "Helvetica") + font_size = block.get("font_size", 10) + + # Map font family + font_name = "Helvetica" + if font_family: + fl = font_family.lower() + if "times" in fl or "serif" in fl: + font_name = "Times-Roman" + elif "courier" in fl or "mono" in fl: + font_name = "Courier" + + canvas.setFont(font_name, font_size) + canvas.setFillColor(colors.black) + canvas.drawString(x, y, text) + + canvas.restoreState() diff --git a/docengine/app/services/template_service.py b/docengine/app/services/template_service.py new file mode 100644 index 0000000..6bf81cd --- /dev/null +++ b/docengine/app/services/template_service.py @@ -0,0 +1,402 @@ +from __future__ import annotations + +import uuid +from typing import Any + +from sqlalchemy.orm import Session + +from app.core.logging_config import get_logger +from app.models.document import Document +from app.models.template import ( + DocumentCell, + DocumentFormat, + DocumentRegion, + ImageRegion, + TableColumn, + TableFormat, + TableRow, + Watermark, +) +from app.repositories.document_repository import DocumentRepository +from app.repositories.template_repository import ( + DocumentCellRepository, + DocumentRegionRepository, + ImageRegionRepository, + TableColumnRepository, + TableFormatRepository, + TableRowRepository, + TemplateRepository, + WatermarkRepository, +) +from app.services.fingerprint_service import FingerprintService + +logger = get_logger(__name__) + + +class TemplateService: + """Generate reusable document templates from processed documents.""" + + def __init__(self, db: Session) -> None: + self.db = db + self.doc_repo = DocumentRepository(db) + self.template_repo = TemplateRepository(db) + self.cell_repo = DocumentCellRepository(db) + self.region_repo = DocumentRegionRepository(db) + self.table_format_repo = TableFormatRepository(db) + self.table_column_repo = TableColumnRepository(db) + self.table_row_repo = TableRowRepository(db) + self.watermark_repo = WatermarkRepository(db) + self.image_region_repo = ImageRegionRepository(db) + self.fingerprint_service = FingerprintService(db) + + def generate_template( + self, + document: Document, + user_id: uuid.UUID | None = None, + ) -> DocumentFormat: + """Generate a reusable template from a processed document.""" + if not document.pages: + raise ValueError(f"Document '{document.id}' has no processed pages") + + # Check if template already exists for this document + existing = self.template_repo.get_by_source_document(document.id) + if existing: + logger.info( + "template_already_exists", + document_id=str(document.id), + template_id=str(existing.id), + ) + return existing + + first_page = document.pages[0] + template_name = f"Template_{document.original_filename}_{uuid.uuid4().hex[:8]}" + + # Create template + template = self.template_repo.create_template( + name=template_name, + page_width=first_page.width, + page_height=first_page.height, + page_count=document.page_count or len(document.pages), + description=f"Auto-generated template from {document.original_filename}", + source_document_id=document.id, + created_by=user_id, + is_active=False, + ) + + # Process each page + for page in document.pages: + self._process_page_for_template(template, page) + + # Generate fingerprint + self.fingerprint_service.generate_fingerprint(template) + + logger.info( + "template_generated", + template_id=str(template.id), + document_id=str(document.id), + cells=len(template.cells), + regions=len(template.regions), + ) + + return template + + def _process_page_for_template( + self, + template: DocumentFormat, + page: Any, + ) -> None: + """Process a document page and create template components.""" + page_number = page.page_number + + # Create cells from text blocks + self._create_cells_from_text_blocks(template, page, page_number) + + # Create regions from headers, footers + self._create_regions(template, page, page_number) + + # Create table formats + self._create_table_formats(template, page, page_number) + + # Create image regions + self._create_image_regions(template, page, page_number) + + # Detect watermarks from text blocks + self._create_watermarks(template, page, page_number) + + def _create_cells_from_text_blocks( + self, + template: DocumentFormat, + page: Any, + page_number: int, + ) -> None: + """Create template cells from extracted text blocks.""" + for seq, block in enumerate(page.text_blocks): + if block.block_type in ("header", "footer", "watermark"): + continue + + # Determine if this is a dynamic field + is_dynamic = self._is_dynamic_field(block.text) + field_name = self._generate_field_name(block.text, seq) if is_dynamic else None + + self.cell_repo.create_cell( + format_id=template.id, + page_number=page_number, + x=block.x, + y=block.y, + width=block.width, + height=block.height, + data_type=self._infer_data_type(block.text), + font_family=block.font_family, + font_size=block.font_size, + font_style=block.font_style, + font_color=block.font_color, + alignment=self._infer_alignment(block.x, template.page_width), + static_text=block.text if not is_dynamic else None, + field_name=field_name, + sequence=seq, + is_dynamic=is_dynamic, + ) + + def _create_regions( + self, + template: DocumentFormat, + page: Any, + page_number: int, + ) -> None: + """Create template regions from headers and footers.""" + header_blocks = [b for b in page.text_blocks if b.block_type == "header"] + footer_blocks = [b for b in page.text_blocks if b.block_type == "footer"] + + if header_blocks: + # Compute bounding box for all header blocks + min_x = min(b.x for b in header_blocks) + min_y = min(b.y for b in header_blocks) + max_x = max(b.x + b.width for b in header_blocks) + max_y = max(b.y + b.height for b in header_blocks) + + content = { + "blocks": [ + { + "text": b.text, + "x": b.x, + "y": b.y, + "width": b.width, + "height": b.height, + "font_family": b.font_family, + "font_size": b.font_size, + } + for b in header_blocks + ] + } + + self.region_repo.create_region( + format_id=template.id, + page_number=page_number, + region_type="header", + x=min_x, + y=min_y, + width=max_x - min_x, + height=max_y - min_y, + content=content, + sequence=0, + ) + + if footer_blocks: + min_x = min(b.x for b in footer_blocks) + min_y = min(b.y for b in footer_blocks) + max_x = max(b.x + b.width for b in footer_blocks) + max_y = max(b.y + b.height for b in footer_blocks) + + content = { + "blocks": [ + { + "text": b.text, + "x": b.x, + "y": b.y, + "width": b.width, + "height": b.height, + "font_family": b.font_family, + "font_size": b.font_size, + } + for b in footer_blocks + ] + } + + self.region_repo.create_region( + format_id=template.id, + page_number=page_number, + region_type="footer", + x=min_x, + y=min_y, + width=max_x - min_x, + height=max_y - min_y, + content=content, + sequence=1, + ) + + def _create_table_formats( + self, + template: DocumentFormat, + page: Any, + page_number: int, + ) -> None: + """Create table format definitions from detected tables.""" + for table in page.tables: + table_format = self.table_format_repo.create_table_format( + format_id=template.id, + page_number=page_number, + x=table.x, + y=table.y, + width=table.width, + height=table.height, + rows=table.rows, + columns=table.columns, + ) + + # Create columns + col_width = table.width / max(table.columns, 1) + for col_idx in range(table.columns): + self.table_column_repo.create_column( + table_format_id=table_format.id, + column_index=col_idx, + width=col_width, + data_type="text", + alignment="left", + ) + + # Create rows + row_height = table.height / max(table.rows, 1) + for row_idx in range(table.rows): + self.table_row_repo.create_row( + table_format_id=table_format.id, + row_index=row_idx, + height=row_height, + is_header=(row_idx == 0), + ) + + def _create_image_regions( + self, + template: DocumentFormat, + page: Any, + page_number: int, + ) -> None: + """Create image region definitions from detected images.""" + for img in page.images: + self.image_region_repo.create_image_region( + format_id=template.id, + page_number=page_number, + x=img.x, + y=img.y, + width=img.width, + height=img.height, + image_path=img.image_path, + image_type=img.image_type, + is_static=True, + ) + + def _create_watermarks( + self, + template: DocumentFormat, + page: Any, + page_number: int, + ) -> None: + """Create watermark definitions from detected watermark text blocks.""" + watermark_blocks = [b for b in page.text_blocks if b.block_type == "watermark"] + for block in watermark_blocks: + self.watermark_repo.create_watermark( + format_id=template.id, + page_number=page_number, + text=block.text, + x=block.x, + y=block.y, + width=block.width, + height=block.height, + opacity=0.3, + rotation=0.0, + font_family=block.font_family, + font_size=block.font_size, + font_color=block.font_color or "#CCCCCC", + ) + + def _is_dynamic_field(self, text: str) -> bool: + """Determine if a text block represents a dynamic (variable) field.""" + if not text: + return False + + # Common patterns indicating dynamic content + dynamic_patterns = [ + "{{", "}}", "${", "##", + "__________", "___", "...........", + ] + for pattern in dynamic_patterns: + if pattern in text: + return True + + # Short single-word values that might be labels are static + # Longer values with numbers/dates tend to be dynamic + import re + # Date patterns + if re.search(r"\d{1,2}[/\-\.]\d{1,2}[/\-\.]\d{2,4}", text): + return True + # Currency patterns + if re.search(r"[$€£¥]\s*[\d,]+\.?\d*", text): + return True + # Phone patterns + if re.search(r"\+?\d[\d\s\-()]{7,}", text): + return True + + return False + + def _generate_field_name(self, text: str, sequence: int) -> str: + """Generate a field name from text content.""" + import re + # Clean text + clean = re.sub(r"[^a-zA-Z0-9\s]", "", text) + clean = clean.strip().lower() + words = clean.split()[:3] + if words: + return "_".join(words) + return f"field_{sequence}" + + def _infer_data_type(self, text: str) -> str: + """Infer the data type from text content.""" + import re + + if not text: + return "text" + + stripped = text.strip() + + # Number + if re.match(r"^-?[\d,]+\.?\d*$", stripped.replace(",", "")): + return "number" + + # Date + if re.search(r"\d{1,2}[/\-\.]\d{1,2}[/\-\.]\d{2,4}", stripped): + return "date" + + # Currency + if re.search(r"^[$€£¥]\s*[\d,]+\.?\d*$", stripped): + return "currency" + + # Email + if re.search(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", stripped): + return "email" + + return "text" + + def _infer_alignment(self, x: float, page_width: float) -> str: + """Infer text alignment based on horizontal position.""" + if page_width <= 0: + return "left" + + relative_x = x / page_width + + if relative_x < 0.15: + return "left" + elif relative_x > 0.6: + return "right" + elif 0.35 < relative_x < 0.65: + return "center" + + return "left" diff --git a/docengine/app/storage/__init__.py b/docengine/app/storage/__init__.py new file mode 100644 index 0000000..7ff523a --- /dev/null +++ b/docengine/app/storage/__init__.py @@ -0,0 +1 @@ +# Storage module diff --git a/docengine/app/storage/provider.py b/docengine/app/storage/provider.py new file mode 100644 index 0000000..1a3e5c6 --- /dev/null +++ b/docengine/app/storage/provider.py @@ -0,0 +1,163 @@ +from __future__ import annotations + +import hashlib +import os +import shutil +import uuid +from abc import ABC, abstractmethod +from pathlib import Path + +from app.core.config import settings +from app.core.exceptions import StorageError +from app.core.logging_config import get_logger + +logger = get_logger(__name__) + + +class StorageProvider(ABC): + """Abstract base class for storage providers.""" + + @abstractmethod + def save_file(self, file_data: bytes, directory: str, filename: str | None = None) -> str: + """Save file data and return the storage path.""" + ... + + @abstractmethod + def read_file(self, storage_path: str) -> bytes: + """Read file data from storage.""" + ... + + @abstractmethod + def delete_file(self, storage_path: str) -> bool: + """Delete a file from storage. Returns True if successful.""" + ... + + @abstractmethod + def file_exists(self, storage_path: str) -> bool: + """Check if a file exists in storage.""" + ... + + @abstractmethod + def get_file_size(self, storage_path: str) -> int: + """Get the file size in bytes.""" + ... + + @abstractmethod + def get_absolute_path(self, storage_path: str) -> str: + """Get the absolute filesystem path for a storage path.""" + ... + + @staticmethod + def compute_checksum(data: bytes, algorithm: str = "sha256") -> str: + """Compute checksum of file data.""" + hasher = hashlib.new(algorithm) + hasher.update(data) + return hasher.hexdigest() + + @staticmethod + def generate_filename(original_filename: str) -> str: + """Generate a unique filename preserving the original extension.""" + ext = Path(original_filename).suffix.lower() + return f"{uuid.uuid4().hex}{ext}" + + +class LocalStorageProvider(StorageProvider): + """Local filesystem storage provider.""" + + def __init__(self, base_path: str | None = None) -> None: + self.base_path = Path(base_path or settings.storage_local_path).resolve() + self._ensure_directories() + + def _ensure_directories(self) -> None: + """Create required storage directories.""" + for subdir in ("documents", "templates", "images", "temp", "rendered"): + (self.base_path / subdir).mkdir(parents=True, exist_ok=True) + + def _resolve_path(self, storage_path: str) -> Path: + """Resolve a storage path to an absolute path.""" + resolved = (self.base_path / storage_path).resolve() + if not str(resolved).startswith(str(self.base_path)): + raise StorageError(f"Path traversal detected: {storage_path}") + return resolved + + def save_file(self, file_data: bytes, directory: str, filename: str | None = None) -> str: + """Save file data to local storage.""" + if filename is None: + filename = f"{uuid.uuid4().hex}.bin" + + dir_path = self.base_path / directory + dir_path.mkdir(parents=True, exist_ok=True) + + file_path = dir_path / filename + try: + file_path.write_bytes(file_data) + storage_path = str(file_path.relative_to(self.base_path)) + logger.info("file_saved", storage_path=storage_path, size=len(file_data)) + return storage_path + except OSError as e: + raise StorageError(f"Failed to save file: {e}") from e + + def read_file(self, storage_path: str) -> bytes: + """Read file data from local storage.""" + file_path = self._resolve_path(storage_path) + if not file_path.exists(): + raise StorageError(f"File not found: {storage_path}") + try: + return file_path.read_bytes() + except OSError as e: + raise StorageError(f"Failed to read file: {e}") from e + + def delete_file(self, storage_path: str) -> bool: + """Delete a file from local storage.""" + file_path = self._resolve_path(storage_path) + if not file_path.exists(): + return False + try: + file_path.unlink() + logger.info("file_deleted", storage_path=storage_path) + return True + except OSError as e: + logger.error("file_delete_failed", storage_path=storage_path, error=str(e)) + raise StorageError(f"Failed to delete file: {e}") from e + + def file_exists(self, storage_path: str) -> bool: + """Check if a file exists in local storage.""" + file_path = self._resolve_path(storage_path) + return file_path.exists() + + def get_file_size(self, storage_path: str) -> int: + """Get the file size in bytes.""" + file_path = self._resolve_path(storage_path) + if not file_path.exists(): + raise StorageError(f"File not found: {storage_path}") + return file_path.stat().st_size + + def get_absolute_path(self, storage_path: str) -> str: + """Get the absolute filesystem path.""" + return str(self._resolve_path(storage_path)) + + def save_temp_file(self, file_data: bytes, filename: str) -> str: + """Save a temporary file.""" + return self.save_file(file_data, "temp", filename) + + def cleanup_temp(self) -> int: + """Remove all files in the temp directory.""" + temp_dir = self.base_path / "temp" + count = 0 + if temp_dir.exists(): + for item in temp_dir.iterdir(): + if item.is_file(): + item.unlink() + count += 1 + elif item.is_dir(): + shutil.rmtree(item) + count += 1 + logger.info("temp_cleanup", files_removed=count) + return count + + +def get_storage_provider() -> StorageProvider: + """Factory function to get the configured storage provider.""" + if settings.storage_provider == "local": + return LocalStorageProvider() + raise StorageError(f"Unknown storage provider: {settings.storage_provider}") diff --git a/docengine/app/tasks/__init__.py b/docengine/app/tasks/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/tasks/document_tasks.py b/docengine/app/tasks/document_tasks.py new file mode 100644 index 0000000..134a871 --- /dev/null +++ b/docengine/app/tasks/document_tasks.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +import uuid + +from app.core.logging_config import get_logger +from app.workers.celery_app import celery_app +from app.core.database import get_db_context +from app.services.document_service import DocumentProcessingService + +logger = get_logger(__name__) + + +@celery_app.task( + name="app.tasks.document_tasks.process_document_task", + bind=True, + max_retries=3, + default_retry_delay=60, + acks_late=True, +) +def process_document_task(self, document_id: str) -> dict: # noqa: ANN001 + """Celery task to process a document asynchronously.""" + logger.info("task_started", task_id=self.request.id, document_id=document_id) + + try: + doc_uuid = uuid.UUID(document_id) + with get_db_context() as db: + service = DocumentProcessingService(db) + document = service.process_document(doc_uuid) + logger.info( + "task_completed", + task_id=self.request.id, + document_id=document_id, + status=document.status, + ) + return { + "document_id": document_id, + "status": document.status, + "page_count": document.page_count, + } + except Exception as exc: + logger.exception( + "task_failed", + task_id=self.request.id, + document_id=document_id, + error=str(exc), + retry=self.request.retries, + ) + raise self.retry(exc=exc) + + +@celery_app.task( + name="app.tasks.document_tasks.match_document_task", + bind=True, + max_retries=2, + default_retry_delay=30, +) +def match_document_task( + self, # noqa: ANN001 + document_id: str, + min_confidence: float = 0.75, + max_results: int = 5, +) -> dict: + """Celery task to match a document against templates.""" + logger.info("match_task_started", task_id=self.request.id, document_id=document_id) + + try: + doc_uuid = uuid.UUID(document_id) + with get_db_context() as db: + from app.services.matching_service import MatchingService + service = MatchingService(db) + matches = service.match_document( + document_id=doc_uuid, + min_confidence=min_confidence, + max_results=max_results, + ) + logger.info( + "match_task_completed", + task_id=self.request.id, + document_id=document_id, + matches=len(matches), + ) + return { + "document_id": document_id, + "matches": len(matches), + "best_score": matches[0].confidence_score if matches else 0.0, + } + except Exception as exc: + logger.exception( + "match_task_failed", + task_id=self.request.id, + document_id=document_id, + error=str(exc), + ) + raise self.retry(exc=exc) diff --git a/docengine/app/tasks/maintenance_tasks.py b/docengine/app/tasks/maintenance_tasks.py new file mode 100644 index 0000000..34cfcde --- /dev/null +++ b/docengine/app/tasks/maintenance_tasks.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +from app.core.logging_config import get_logger +from app.workers.celery_app import celery_app +from app.core.database import get_db_context + +logger = get_logger(__name__) + + +@celery_app.task( + name="app.tasks.maintenance_tasks.cleanup_expired_tokens", + bind=True, +) +def cleanup_expired_tokens(self) -> dict: # noqa: ANN001 + """Cleanup expired and revoked refresh tokens.""" + logger.info("cleanup_tokens_started", task_id=self.request.id) + + try: + with get_db_context() as db: + from app.repositories.user_repository import RefreshTokenRepository + repo = RefreshTokenRepository(db) + count = repo.cleanup_expired_tokens() + logger.info("cleanup_tokens_completed", removed=count) + return {"removed_tokens": count} + except Exception as exc: + logger.exception("cleanup_tokens_failed", error=str(exc)) + return {"error": str(exc)} + + +@celery_app.task( + name="app.tasks.maintenance_tasks.cleanup_temp_storage", + bind=True, +) +def cleanup_temp_storage(self) -> dict: # noqa: ANN001 + """Cleanup temporary storage files.""" + logger.info("cleanup_temp_started", task_id=self.request.id) + + try: + from app.storage.provider import get_storage_provider + storage = get_storage_provider() + count = storage.cleanup_temp() + logger.info("cleanup_temp_completed", removed=count) + return {"removed_files": count} + except Exception as exc: + logger.exception("cleanup_temp_failed", error=str(exc)) + return {"error": str(exc)} diff --git a/docengine/app/templates/__init__.py b/docengine/app/templates/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/utils/__init__.py b/docengine/app/utils/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/utils/sanitizers.py b/docengine/app/utils/sanitizers.py new file mode 100644 index 0000000..964eb82 --- /dev/null +++ b/docengine/app/utils/sanitizers.py @@ -0,0 +1,62 @@ +import re +from typing import Optional +from datetime import datetime +from dateutil import parser + +def sanitize_amount(text: str) -> Optional[float]: + """ + Sanitizes a string representing an amount/number (e.g., "$1,234.56", "€ 50,000", "50 USD") + by removing currency symbols, commas, and other non-numeric characters (except for the decimal separator), + and casts it to a float. + + Args: + text (str): The raw extracted text from the document. + + Returns: + Optional[float]: The sanitized numeric value, or None if no number could be extracted. + """ + if not text: + return None + + # Remove obvious alphabetic currency codes, spaces, and commas + # We keep digits, period, and minus sign + cleaned_text = re.sub(r'[^\d\.-]', '', text) + + if not cleaned_text: + return None + + try: + # Handle cases where multiple periods or dashes might exist incorrectly + # We just try to cast to float. If the OCR produced something like "1.23.45", this will fail. + # A more robust regex can handle exact capture if needed. + # But this basic cast covers 95% of standard sanitized strings. + return float(cleaned_text) + except ValueError: + return None + +def sanitize_date(text: str, field_type: str = 'DATE') -> Optional[str]: + """ + Sanitizes a string representing a date or datetime and converts it to ISO 8601 format. + + Args: + text (str): The raw extracted text from the document. + field_type (str): 'DATE' or 'DATETIME'. Determines the output format. + + Returns: + Optional[str]: The sanitized date string in ISO format, or None if it could not be parsed. + """ + if not text: + return None + + try: + # dateutil.parser is very robust at handling formats like "Dec 11, 2020", "11/12/2020", etc. + # fuzzy=True allows it to ignore extra words/characters around the date + parsed_date = parser.parse(text, fuzzy=True) + + if field_type == 'DATETIME': + return parsed_date.strftime('%Y-%m-%dT%H:%M:%S') + else: + return parsed_date.strftime('%Y-%m-%d') + except (ValueError, TypeError, OverflowError): + return None + diff --git a/docengine/app/workers/__init__.py b/docengine/app/workers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/app/workers/celery_app.py b/docengine/app/workers/celery_app.py new file mode 100644 index 0000000..34f64b2 --- /dev/null +++ b/docengine/app/workers/celery_app.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from celery import Celery + +from app.core.config import settings + +celery_app = Celery( + "docengine", + broker=settings.celery_broker_url, + backend=settings.celery_result_backend, +) + +celery_app.conf.update( + task_serializer="json", + accept_content=["json"], + result_serializer="json", + timezone="UTC", + enable_utc=True, + task_track_started=True, + task_time_limit=3600, + task_soft_time_limit=3300, + worker_max_tasks_per_child=100, + worker_prefetch_multiplier=1, + task_acks_late=True, + task_reject_on_worker_lost=True, + broker_connection_retry_on_startup=True, + result_expires=86400, + task_default_queue="docengine_default", + task_routes={ + "app.tasks.document_tasks.*": {"queue": "document_processing"}, + }, + beat_schedule={ + "cleanup-expired-tokens": { + "task": "app.tasks.maintenance_tasks.cleanup_expired_tokens", + "schedule": 3600.0, + }, + "cleanup-temp-storage": { + "task": "app.tasks.maintenance_tasks.cleanup_temp_storage", + "schedule": 7200.0, + }, + }, +) + +celery_app.conf.update( + imports=[ + "app.tasks.document_tasks", + "app.tasks.maintenance_tasks" + ] +) diff --git a/docengine/application.properties b/docengine/application.properties new file mode 100644 index 0000000..4ad7a3c --- /dev/null +++ b/docengine/application.properties @@ -0,0 +1,8 @@ + +server.port=7989 + +db.host=192.168.0.111 +db.port=5432 +db.user=postgres +db.password=M@triXPostgr3s@6202 +db.schema=admin diff --git a/docengine/docengine.db b/docengine/docengine.db new file mode 100644 index 0000000..e69de29 diff --git a/docengine/docker-compose.prod.yml b/docengine/docker-compose.prod.yml new file mode 100644 index 0000000..d78aeb9 --- /dev/null +++ b/docengine/docker-compose.prod.yml @@ -0,0 +1,154 @@ +version: "3.9" + +services: + db: + image: postgres:16-alpine + container_name: docengine_db_prod + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "${DB_PASSWORD}" + POSTGRES_DB: ocr + ports: + - "5432:5432" + volumes: + - docengine_pgdata_prod:/var/lib/postgresql/data + - ./sql/001_create_schema.sql:/docker-entrypoint-initdb.d/001_create_schema.sql + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d ocr"] + interval: 10s + timeout: 5s + retries: 5 + deploy: + resources: + limits: + memory: 2G + cpus: "2.0" + restart: always + networks: + - docengine_net_prod + + redis: + image: redis:7-alpine + container_name: docengine_redis_prod + command: redis-server --requirepass "${REDIS_PASSWORD}" --appendonly yes + ports: + - "6379:6379" + volumes: + - docengine_redis_data_prod:/data + healthcheck: + test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"] + interval: 10s + timeout: 5s + retries: 5 + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + restart: always + networks: + - docengine_net_prod + + app: + build: + context: . + target: app + container_name: docengine_app_prod + env_file: + - .env + environment: + APP_ENV: production + APP_DEBUG: "false" + DB_HOST: db + DB_PORT: 5432 + REDIS_HOST: redis + CELERY_BROKER_URL: "redis://:${REDIS_PASSWORD}@redis:6379/0" + CELERY_RESULT_BACKEND: "redis://:${REDIS_PASSWORD}@redis:6379/1" + ports: + - "7989:7989" + volumes: + - docengine_storage_prod:/app/storage + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 4G + cpus: "4.0" + replicas: 2 + restart: always + networks: + - docengine_net_prod + + worker: + build: + context: . + target: worker + container_name: docengine_worker_prod + env_file: + - .env + environment: + APP_ENV: production + APP_DEBUG: "false" + DB_HOST: db + DB_PORT: 5432 + REDIS_HOST: redis + CELERY_BROKER_URL: "redis://:${REDIS_PASSWORD}@redis:6379/0" + CELERY_RESULT_BACKEND: "redis://:${REDIS_PASSWORD}@redis:6379/1" + volumes: + - docengine_storage_prod:/app/storage + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 8G + cpus: "4.0" + replicas: 2 + restart: always + networks: + - docengine_net_prod + + beat: + build: + context: . + target: beat + container_name: docengine_beat_prod + env_file: + - .env + environment: + APP_ENV: production + APP_DEBUG: "false" + DB_HOST: db + DB_PORT: 5432 + REDIS_HOST: redis + CELERY_BROKER_URL: "redis://:${REDIS_PASSWORD}@redis:6379/0" + CELERY_RESULT_BACKEND: "redis://:${REDIS_PASSWORD}@redis:6379/1" + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 512M + cpus: "0.5" + restart: always + networks: + - docengine_net_prod + +volumes: + docengine_pgdata_prod: + docengine_redis_data_prod: + docengine_storage_prod: + +networks: + docengine_net_prod: + driver: bridge diff --git a/docengine/docker-compose.yml b/docengine/docker-compose.yml new file mode 100644 index 0000000..340ff12 --- /dev/null +++ b/docengine/docker-compose.yml @@ -0,0 +1,93 @@ +version: "3.9" + +services: + db: + image: postgres:16-alpine + container_name: docengine_db + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "M@triXPostgr3s@6202" + POSTGRES_DB: ocr + ports: + - "5432:5432" + volumes: + - docengine_pgdata:/var/lib/postgresql/data + - ./sql/001_create_schema.sql:/docker-entrypoint-initdb.d/001_create_schema.sql + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d ocr"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - docengine_net + + redis: + image: redis:7-alpine + container_name: docengine_redis + ports: + - "6379:6379" + volumes: + - docengine_redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - docengine_net + + app: + build: + context: . + target: app + container_name: docengine_app + env_file: + - .env + environment: + DB_HOST: db + DB_PORT: 5432 + REDIS_HOST: redis + CELERY_BROKER_URL: redis://redis:6379/0 + CELERY_RESULT_BACKEND: redis://redis:6379/1 + ports: + - "7989:7989" + volumes: + - ./storage:/app/storage + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + networks: + - docengine_net + + worker: + build: + context: . + target: worker + container_name: docengine_worker + env_file: + - .env + environment: + DB_HOST: db + DB_PORT: 5432 + REDIS_HOST: redis + CELERY_BROKER_URL: redis://redis:6379/0 + CELERY_RESULT_BACKEND: redis://redis:6379/1 + volumes: + - ./storage:/app/storage + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + networks: + - docengine_net + +volumes: + docengine_pgdata: + docengine_redis_data: + +networks: + docengine_net: + driver: bridge diff --git a/docengine/pyproject.toml b/docengine/pyproject.toml new file mode 100644 index 0000000..28012e2 --- /dev/null +++ b/docengine/pyproject.toml @@ -0,0 +1,45 @@ +[build-system] +requires = ["setuptools>=75.0", "wheel"] +build-backend = "setuptools.backends._legacy:_Backend" + +[project] +name = "docengine" +version = "1.0.0" +description = "Document Template Recognition and Reconstruction System" +readme = "README.md" +requires-python = ">=3.12" +license = {text = "Proprietary"} + +[tool.pytest.ini_options] +testpaths = ["tests"] +asyncio_mode = "auto" +python_files = ["test_*.py"] +python_classes = ["Test*"] +python_functions = ["test_*"] +addopts = "-v --tb=short --cov=app --cov-report=term-missing --cov-report=html" +filterwarnings = [ + "ignore::DeprecationWarning", +] + +[tool.black] +line-length = 120 +target-version = ["py312"] + +[tool.isort] +profile = "black" +line_length = 120 + +[tool.ruff] +line-length = 120 +target-version = "py312" + +[tool.ruff.lint] +select = ["E", "F", "W", "I", "N", "UP", "S", "B", "A", "C4", "DTZ", "ISC", "PIE", "T20", "RSE", "RET", "SIM", "TCH", "ERA", "PGH", "PLC", "PLE", "PLR", "PLW", "TRY", "RUF"] +ignore = ["S101", "S603", "S607", "TRY003", "PLR0913", "B008"] + +[tool.mypy] +python_version = "3.12" +warn_return_any = true +warn_unused_configs = true +disallow_untyped_defs = true +ignore_missing_imports = true diff --git a/docengine/requirements-dev.txt b/docengine/requirements-dev.txt new file mode 100644 index 0000000..a86c6f8 --- /dev/null +++ b/docengine/requirements-dev.txt @@ -0,0 +1,21 @@ +-r requirements.txt + +# Testing +pytest==8.3.4 +pytest-cov==6.0.0 +pytest-asyncio==0.25.0 +pytest-mock==3.14.0 +httpx==0.28.1 +factory-boy==3.3.1 + +# Code Quality +ruff==0.8.6 +mypy==1.14.1 +black==24.10.0 +isort==5.13.2 + +# Type Stubs +types-redis==4.6.0.20241004 +types-python-dateutil==2.9.0.20241003 +types-passlib==1.7.7.20240819 +types-aiofiles==24.1.0.20240626 diff --git a/docengine/requirements.txt b/docengine/requirements.txt new file mode 100644 index 0000000..8e6a283 --- /dev/null +++ b/docengine/requirements.txt @@ -0,0 +1,49 @@ +# Core Framework +fastapi==0.115.6 +uvicorn[standard]==0.34.0 +python-multipart==0.0.20 + +# Database +sqlalchemy[asyncio]==2.0.36 +psycopg2-binary==2.9.10 +alembic==1.14.1 + +# Validation & Settings +pydantic==2.10.4 +pydantic-settings==2.7.1 +email-validator==2.2.0 + +# Authentication & Security +python-jose[cryptography]==3.3.0 +passlib[bcrypt]==1.7.4 +bcrypt==4.2.1 + +# Document Processing +PyMuPDF==1.25.3 +paddleocr==2.9.1 +paddlepaddle>=3.0.0 +layoutparser==0.3.4 +opencv-python-headless==4.10.0.84 +camelot-py[cv]==0.11.0 +pdf2image==1.17.0 +Pillow==11.1.0 + +# PDF Generation +reportlab==4.2.5 + +# Background Jobs +celery[redis]==5.4.0 +redis==5.2.1 + +# Logging +structlog==24.4.0 + +# Monitoring +prometheus-client==0.21.1 +prometheus-fastapi-instrumentator==7.0.2 + +# Utilities +python-dateutil==2.9.0 +aiofiles==24.1.0 +httpx==0.28.1 +numpy==1.26.4 diff --git a/docengine/sql/001_create_schema.sql b/docengine/sql/001_create_schema.sql new file mode 100644 index 0000000..1ba0050 --- /dev/null +++ b/docengine/sql/001_create_schema.sql @@ -0,0 +1,12 @@ +-- Create the admin schema for DocEngine +CREATE SCHEMA IF NOT EXISTS admin; + +-- Set the default search path +ALTER DATABASE ocr SET search_path TO admin, public; + +-- Grant privileges +GRANT ALL ON SCHEMA admin TO postgres; +GRANT USAGE ON SCHEMA admin TO postgres; +ALTER DEFAULT PRIVILEGES IN SCHEMA admin GRANT ALL ON TABLES TO postgres; +ALTER DEFAULT PRIVILEGES IN SCHEMA admin GRANT ALL ON SEQUENCES TO postgres; +ALTER DEFAULT PRIVILEGES IN SCHEMA admin GRANT ALL ON FUNCTIONS TO postgres; diff --git a/docengine/sql/001_init.sql b/docengine/sql/001_init.sql new file mode 100644 index 0000000..0675c5f --- /dev/null +++ b/docengine/sql/001_init.sql @@ -0,0 +1,16 @@ + +CREATE SCHEMA IF NOT EXISTS admin; + +CREATE TABLE IF NOT EXISTS admin.document_format( + id BIGSERIAL PRIMARY KEY, + name VARCHAR(255), + created_at TIMESTAMP DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_cell( + id BIGSERIAL PRIMARY KEY, + format_id BIGINT REFERENCES admin.document_format(id), + data_type VARCHAR(50), + font_family VARCHAR(100), + font_size INTEGER +); diff --git a/docengine/sql/002_create_tables.sql b/docengine/sql/002_create_tables.sql new file mode 100644 index 0000000..dea724e --- /dev/null +++ b/docengine/sql/002_create_tables.sql @@ -0,0 +1,301 @@ +-- DocEngine: Complete table creation script +-- Schema: admin +-- Database: ocr + +SET search_path TO admin, public; + +-- ============================================ +-- Users & Authentication +-- ============================================ + +CREATE TABLE IF NOT EXISTS admin.users ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + username VARCHAR(150) NOT NULL UNIQUE, + email VARCHAR(255) NOT NULL UNIQUE, + hashed_password VARCHAR(255) NOT NULL, + full_name VARCHAR(255), + is_active BOOLEAN NOT NULL DEFAULT TRUE, + is_superuser BOOLEAN NOT NULL DEFAULT FALSE, + last_login TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.roles ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name VARCHAR(50) NOT NULL UNIQUE, + description TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.user_roles ( + user_id UUID NOT NULL REFERENCES admin.users(id) ON DELETE CASCADE, + role_id UUID NOT NULL REFERENCES admin.roles(id) ON DELETE CASCADE, + PRIMARY KEY (user_id, role_id) +); + +CREATE TABLE IF NOT EXISTS admin.refresh_tokens ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID NOT NULL REFERENCES admin.users(id) ON DELETE CASCADE, + token VARCHAR(512) NOT NULL UNIQUE, + expires_at TIMESTAMPTZ NOT NULL, + revoked BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.audit_logs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID REFERENCES admin.users(id) ON DELETE SET NULL, + action VARCHAR(100) NOT NULL, + resource_type VARCHAR(100) NOT NULL, + resource_id VARCHAR(255), + details TEXT, + ip_address VARCHAR(45), + user_agent VARCHAR(512), + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +-- ============================================ +-- Documents +-- ============================================ + +CREATE TABLE IF NOT EXISTS admin.documents ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + filename VARCHAR(500) NOT NULL, + original_filename VARCHAR(500) NOT NULL, + content_type VARCHAR(100) NOT NULL, + file_size BIGINT NOT NULL, + checksum VARCHAR(128) NOT NULL, + storage_path VARCHAR(1024) NOT NULL, + status VARCHAR(50) NOT NULL DEFAULT 'pending', + page_count INTEGER, + is_scanned BOOLEAN, + document_metadata JSONB, + error_message TEXT, + uploaded_by UUID REFERENCES admin.users(id) ON DELETE SET NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_pages ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + document_id UUID NOT NULL REFERENCES admin.documents(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + image_path VARCHAR(1024), + text_content TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_text_blocks ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + page_id UUID NOT NULL REFERENCES admin.document_pages(id) ON DELETE CASCADE, + text TEXT NOT NULL, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + confidence DOUBLE PRECISION, + font_family VARCHAR(255), + font_size DOUBLE PRECISION, + font_color VARCHAR(50), + font_style VARCHAR(50), + block_type VARCHAR(50) NOT NULL DEFAULT 'text', + sequence INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_images ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + page_id UUID NOT NULL REFERENCES admin.document_pages(id) ON DELETE CASCADE, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + image_path VARCHAR(1024) NOT NULL, + image_type VARCHAR(50) NOT NULL DEFAULT 'figure', + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_tables ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + page_id UUID NOT NULL REFERENCES admin.document_pages(id) ON DELETE CASCADE, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + rows INTEGER NOT NULL, + columns INTEGER NOT NULL, + data JSONB, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +-- ============================================ +-- Templates +-- ============================================ + +CREATE TABLE IF NOT EXISTS admin.document_formats ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name VARCHAR(255) NOT NULL, + description TEXT, + page_width DOUBLE PRECISION NOT NULL, + page_height DOUBLE PRECISION NOT NULL, + page_count INTEGER NOT NULL DEFAULT 1, + margin_top DOUBLE PRECISION NOT NULL DEFAULT 72.0, + margin_right DOUBLE PRECISION NOT NULL DEFAULT 72.0, + margin_bottom DOUBLE PRECISION NOT NULL DEFAULT 72.0, + margin_left DOUBLE PRECISION NOT NULL DEFAULT 72.0, + fingerprint JSONB, + source_document_id UUID REFERENCES admin.documents(id) ON DELETE SET NULL, + version INTEGER NOT NULL DEFAULT 1, + is_active BOOLEAN NOT NULL DEFAULT TRUE, + created_by UUID REFERENCES admin.users(id) ON DELETE SET NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_cells ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + row_no INTEGER NOT NULL DEFAULT 0, + column_no INTEGER NOT NULL DEFAULT 0, + data_type VARCHAR(50) NOT NULL DEFAULT 'text', + font_family VARCHAR(255), + font_size DOUBLE PRECISION, + font_style VARCHAR(50), + font_color VARCHAR(50), + background_color VARCHAR(50), + border_top VARCHAR(100), + border_right VARCHAR(100), + border_bottom VARCHAR(100), + border_left VARCHAR(100), + padding_top DOUBLE PRECISION NOT NULL DEFAULT 0.0, + padding_right DOUBLE PRECISION NOT NULL DEFAULT 0.0, + padding_bottom DOUBLE PRECISION NOT NULL DEFAULT 0.0, + padding_left DOUBLE PRECISION NOT NULL DEFAULT 0.0, + alignment VARCHAR(20) NOT NULL DEFAULT 'left', + vertical_alignment VARCHAR(20) NOT NULL DEFAULT 'top', + rowspan INTEGER NOT NULL DEFAULT 1, + colspan INTEGER NOT NULL DEFAULT 1, + static_text TEXT, + field_name VARCHAR(255), + sequence INTEGER NOT NULL DEFAULT 0, + is_dynamic BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.document_regions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL, + region_type VARCHAR(50) NOT NULL, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + content JSONB, + sequence INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.table_formats ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + rows INTEGER NOT NULL, + columns INTEGER NOT NULL, + border_style VARCHAR(50) NOT NULL DEFAULT 'solid', + border_width DOUBLE PRECISION NOT NULL DEFAULT 1.0, + border_color VARCHAR(50) NOT NULL DEFAULT '#000000', + header_rows INTEGER NOT NULL DEFAULT 1, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.table_columns ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + table_format_id UUID NOT NULL REFERENCES admin.table_formats(id) ON DELETE CASCADE, + column_index INTEGER NOT NULL, + width DOUBLE PRECISION NOT NULL, + header_text VARCHAR(500), + data_type VARCHAR(50) NOT NULL DEFAULT 'text', + alignment VARCHAR(20) NOT NULL DEFAULT 'left', + font_family VARCHAR(255), + font_size DOUBLE PRECISION, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.table_rows ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + table_format_id UUID NOT NULL REFERENCES admin.table_formats(id) ON DELETE CASCADE, + row_index INTEGER NOT NULL, + height DOUBLE PRECISION NOT NULL DEFAULT 20.0, + is_header BOOLEAN NOT NULL DEFAULT FALSE, + background_color VARCHAR(50), + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.watermarks ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_number INTEGER, + text VARCHAR(500), + image_path VARCHAR(1024), + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + opacity DOUBLE PRECISION NOT NULL DEFAULT 0.3, + rotation DOUBLE PRECISION NOT NULL DEFAULT 0.0, + font_family VARCHAR(255), + font_size DOUBLE PRECISION, + font_color VARCHAR(50), + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.image_regions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL, + x DOUBLE PRECISION NOT NULL, + y DOUBLE PRECISION NOT NULL, + width DOUBLE PRECISION NOT NULL, + height DOUBLE PRECISION NOT NULL, + image_path VARCHAR(1024), + image_type VARCHAR(50) NOT NULL DEFAULT 'figure', + is_static BOOLEAN NOT NULL DEFAULT TRUE, + field_name VARCHAR(255), + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.template_fingerprints ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + format_id UUID NOT NULL UNIQUE REFERENCES admin.document_formats(id) ON DELETE CASCADE, + page_dimensions JSONB, + logo_coordinates JSONB, + header_coordinates JSONB, + footer_coordinates JSONB, + table_coordinates JSONB, + cell_coordinates JSONB, + fingerprint_hash VARCHAR(256) NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS admin.template_matches ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + document_id UUID NOT NULL REFERENCES admin.documents(id) ON DELETE CASCADE, + format_id UUID NOT NULL REFERENCES admin.document_formats(id) ON DELETE CASCADE, + confidence_score DOUBLE PRECISION NOT NULL, + match_details JSONB, + selected BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); diff --git a/docengine/sql/003_seed_data.sql b/docengine/sql/003_seed_data.sql new file mode 100644 index 0000000..e69de29 diff --git a/docengine/sql/004_indexes.sql b/docengine/sql/004_indexes.sql new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/__init__.py b/docengine/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/api/__init__.py b/docengine/tests/api/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/api/test_auth.py b/docengine/tests/api/test_auth.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/api/test_documents.py b/docengine/tests/api/test_documents.py new file mode 100644 index 0000000..ba8bd4d --- /dev/null +++ b/docengine/tests/api/test_documents.py @@ -0,0 +1,224 @@ +from __future__ import annotations + +import io +import uuid + +import pytest +from fastapi.testclient import TestClient +from sqlalchemy.orm import Session + +from tests.conftest import ( + create_test_document, + create_test_page, + create_test_text_block, + create_test_user, + get_auth_headers, +) + + +class TestUploadDocument: + """Tests for the document upload endpoint.""" + + def test_upload_pdf_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + file_content = b"%PDF-1.4 fake pdf content for testing" + response = client.post( + "/api/v1/documents/upload", + files={"file": ("test.pdf", io.BytesIO(file_content), "application/pdf")}, + headers=headers, + ) + assert response.status_code == 201 + data = response.json() + assert data["original_filename"] == "test.pdf" + assert data["content_type"] == "application/pdf" + assert data["status"] == "pending" + assert data["file_size"] == len(file_content) + assert "id" in data + assert "checksum" in data + + def test_upload_image_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + # Minimal valid PNG header + png_header = ( + b"\x89PNG\r\n\x1a\n" + b"\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01" + b"\x08\x02\x00\x00\x00\x90wS\xde" + ) + response = client.post( + "/api/v1/documents/upload", + files={"file": ("scan.png", io.BytesIO(png_header), "image/png")}, + headers=headers, + ) + assert response.status_code == 201 + data = response.json() + assert data["content_type"] == "image/png" + + def test_upload_unsupported_type_rejected(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + response = client.post( + "/api/v1/documents/upload", + files={"file": ("doc.exe", io.BytesIO(b"malware"), "application/octet-stream")}, + headers=headers, + ) + assert response.status_code in (415, 422) + + def test_upload_no_file_rejected(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + response = client.post("/api/v1/documents/upload", headers=headers) + assert response.status_code == 422 + + +class TestGetDocument: + """Tests for getting a document by ID.""" + + def test_get_document_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + db.flush() + headers = get_auth_headers(user) + + response = client.get(f"/api/v1/documents/{doc.id}", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["id"] == str(doc.id) + assert data["original_filename"] == doc.original_filename + assert data["status"] == "completed" + + def test_get_document_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.get(f"/api/v1/documents/{fake_id}", headers=headers) + assert response.status_code == 404 + + def test_get_document_invalid_uuid(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/documents/not-a-uuid", headers=headers) + assert response.status_code == 422 + + +class TestListDocuments: + """Tests for listing documents.""" + + def test_list_documents_empty(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/documents", headers=headers) + assert response.status_code == 200 + data = response.json() + assert "items" in data + assert "total" in data + assert "page" in data + assert "page_size" in data + + def test_list_documents_with_data(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + create_test_document(db, user=user, filename="doc1.pdf") + create_test_document(db, user=user, filename="doc2.pdf") + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/documents", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["total"] >= 2 + assert len(data["items"]) >= 2 + + def test_list_documents_pagination(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + for i in range(5): + create_test_document(db, user=user, filename=f"page_doc_{i}.pdf") + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/documents?page=1&page_size=2", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["page"] == 1 + assert data["page_size"] == 2 + assert len(data["items"]) <= 2 + + def test_list_documents_status_filter(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + create_test_document(db, user=user, filename="pending.pdf", status="pending") + create_test_document(db, user=user, filename="completed.pdf", status="completed") + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/documents?status=completed", headers=headers) + assert response.status_code == 200 + data = response.json() + for item in data["items"]: + assert item["status"] == "completed" + + +class TestDeleteDocument: + """Tests for deleting a document.""" + + def test_delete_document_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + db.flush() + headers = get_auth_headers(user) + + response = client.delete(f"/api/v1/documents/{doc.id}", headers=headers) + assert response.status_code == 200 + data = response.json() + assert "message" in data + + # Verify document is gone + get_response = client.get(f"/api/v1/documents/{doc.id}", headers=headers) + assert get_response.status_code == 404 + + def test_delete_document_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.delete(f"/api/v1/documents/{fake_id}", headers=headers) + assert response.status_code == 404 + + +class TestGetDocumentTemplateMatches: + """Tests for document template match retrieval.""" + + def test_get_matches_empty(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + db.flush() + headers = get_auth_headers(user) + + response = client.get(f"/api/v1/documents/{doc.id}/template", headers=headers) + assert response.status_code == 200 + data = response.json() + assert isinstance(data, list) + assert len(data) == 0 + + def test_get_matches_document_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.get(f"/api/v1/documents/{fake_id}/template", headers=headers) + assert response.status_code == 404 diff --git a/docengine/tests/api/test_health.py b/docengine/tests/api/test_health.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/api/test_templates.py b/docengine/tests/api/test_templates.py new file mode 100644 index 0000000..5631e78 --- /dev/null +++ b/docengine/tests/api/test_templates.py @@ -0,0 +1,254 @@ +from __future__ import annotations + +import uuid + +import pytest +from fastapi.testclient import TestClient +from sqlalchemy.orm import Session + +from tests.conftest import ( + create_test_document, + create_test_fingerprint, + create_test_page, + create_test_template, + create_test_text_block, + create_test_user, + get_auth_headers, +) + + +class TestListTemplates: + """Tests for listing templates.""" + + def test_list_templates_empty(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/templates", headers=headers) + assert response.status_code == 200 + data = response.json() + assert "items" in data + assert "total" in data + assert "page" in data + + def test_list_templates_with_data(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + create_test_template(db, name="Template_A", created_by=user) + create_test_template(db, name="Template_B", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/templates", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["total"] >= 2 + + def test_list_templates_pagination(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + for i in range(5): + create_test_template(db, name=f"PagTemplate_{i}", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.get("/api/v1/templates?page=1&page_size=2", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["page"] == 1 + assert data["page_size"] == 2 + assert len(data["items"]) <= 2 + + +class TestGetTemplate: + """Tests for getting a template by ID.""" + + def test_get_template_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="GetMe", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.get(f"/api/v1/templates/{template.id}", headers=headers) + assert response.status_code == 200 + data = response.json() + assert data["id"] == str(template.id) + assert data["name"] == "GetMe" + assert data["page_width"] == 612.0 + assert data["page_height"] == 792.0 + assert data["is_active"] is True + + def test_get_template_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.get(f"/api/v1/templates/{fake_id}", headers=headers) + assert response.status_code == 404 + + def test_get_template_includes_components(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="ComponentTemplate", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.get(f"/api/v1/templates/{template.id}", headers=headers) + assert response.status_code == 200 + data = response.json() + assert "cells" in data + assert "regions" in data + assert "table_formats" in data + assert "watermarks" in data + assert "image_regions" in data + + +class TestDeleteTemplate: + """Tests for template soft-deletion.""" + + def test_delete_template_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="DeleteMe", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.delete(f"/api/v1/templates/{template.id}", headers=headers) + assert response.status_code == 200 + data = response.json() + assert "message" in data + + # Verify template is deactivated (soft-deleted), not hard-deleted + get_response = client.get(f"/api/v1/templates/{template.id}", headers=headers) + assert get_response.status_code == 200 + assert get_response.json()["is_active"] is False + + def test_delete_template_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.delete(f"/api/v1/templates/{fake_id}", headers=headers) + assert response.status_code == 404 + + +class TestMatchTemplate: + """Tests for document-to-template matching endpoint.""" + + def test_match_document_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_doc_id = uuid.uuid4() + response = client.post( + "/api/v1/templates/match", + json={"document_id": str(fake_doc_id), "min_confidence": 0.5, "max_results": 5}, + headers=headers, + ) + assert response.status_code == 404 + + def test_match_document_not_completed(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="pending") + db.flush() + headers = get_auth_headers(user) + + response = client.post( + "/api/v1/templates/match", + json={"document_id": str(doc.id), "min_confidence": 0.5, "max_results": 5}, + headers=headers, + ) + assert response.status_code == 400 + + def test_match_completed_document_no_templates(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + page = create_test_page(db, doc) + create_test_text_block(db, page, text="Content") + db.flush() + headers = get_auth_headers(user) + + response = client.post( + "/api/v1/templates/match", + json={"document_id": str(doc.id), "min_confidence": 0.0, "max_results": 5}, + headers=headers, + ) + assert response.status_code == 200 + data = response.json() + assert isinstance(data, list) + + +class TestRenderTemplate: + """Tests for template rendering endpoint.""" + + def test_render_template_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + db.flush() + headers = get_auth_headers(user) + + fake_id = uuid.uuid4() + response = client.post( + "/api/v1/templates/render", + json={ + "template_id": str(fake_id), + "data": {}, + }, + headers=headers, + ) + assert response.status_code == 404 + + def test_render_inactive_template(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="InactiveRender", created_by=user) + template.is_active = False + db.flush() + headers = get_auth_headers(user) + + response = client.post( + "/api/v1/templates/render", + json={ + "template_id": str(template.id), + "data": {}, + }, + headers=headers, + ) + assert response.status_code == 400 + + def test_render_template_success(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="RenderOK", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.post( + "/api/v1/templates/render", + json={ + "template_id": str(template.id), + "data": {"field_1": "Hello World"}, + "output_filename": "test_render.pdf", + }, + headers=headers, + ) + assert response.status_code == 200 + data = response.json() + assert data["filename"] == "test_render.pdf" + assert data["file_size"] > 0 + assert data["page_count"] == template.page_count + assert "output_path" in data + assert "rendered_at" in data + + +class TestDownloadRenderedPDF: + """Tests for downloading rendered PDFs.""" + + def test_download_not_found(self, client: TestClient, db: Session) -> None: + user = create_test_user(db) + template = create_test_template(db, name="DLTemplate", created_by=user) + db.flush() + headers = get_auth_headers(user) + + response = client.get( + f"/api/v1/templates/{template.id}/download?filename=nonexistent.pdf", + headers=headers, + ) + assert response.status_code == 404 diff --git a/docengine/tests/conftest.py b/docengine/tests/conftest.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/integration/__init__.py b/docengine/tests/integration/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/repositories/__init__.py b/docengine/tests/repositories/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/repositories/test_document_repository.py b/docengine/tests/repositories/test_document_repository.py new file mode 100644 index 0000000..b67b0d4 --- /dev/null +++ b/docengine/tests/repositories/test_document_repository.py @@ -0,0 +1,278 @@ +from __future__ import annotations + +import uuid + +import pytest +from sqlalchemy.orm import Session + +from app.models.document import ( + Document, + DocumentImage, + DocumentPage, + DocumentTable, + DocumentTextBlock, + TemplateMatch, +) +from app.repositories.document_repository import ( + DocumentPageRepository, + DocumentRepository, + DocumentTextBlockRepository, + TemplateMatchRepository, +) +from tests.conftest import ( + create_test_document, + create_test_fingerprint, + create_test_page, + create_test_template, + create_test_text_block, + create_test_user, +) + + +class TestDocumentRepository: + """Tests for DocumentRepository CRUD.""" + + def test_create_document(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, filename="created.pdf") + assert doc.id is not None + assert doc.original_filename == "created.pdf" + assert doc.status == "pending" + + def test_get_by_id(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + repo = DocumentRepository(db) + found = repo.get_by_id(doc.id) + assert found is not None + assert found.id == doc.id + + def test_get_by_id_not_found(self, db: Session) -> None: + repo = DocumentRepository(db) + assert repo.get_by_id(uuid.uuid4()) is None + + def test_get_with_pages(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + page1 = create_test_page(db, doc, page_number=1) + page2 = create_test_page(db, doc, page_number=2) + create_test_text_block(db, page1, text="Page 1 text") + + repo = DocumentRepository(db) + result = repo.get_with_pages(doc.id) + assert result is not None + assert len(result.pages) == 2 + + def test_update_status(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="pending") + repo = DocumentRepository(db) + repo.update_status(doc.id, "processing") + db.flush() + + updated = repo.get_by_id(doc.id) + assert updated is not None + assert updated.status == "processing" + + def test_update_status_with_error(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="processing") + repo = DocumentRepository(db) + repo.update_status(doc.id, "failed", error_message="OCR engine crashed") + db.flush() + + updated = repo.get_by_id(doc.id) + assert updated is not None + assert updated.status == "failed" + assert updated.error_message == "OCR engine crashed" + + def test_get_all_with_pagination(self, db: Session) -> None: + user = create_test_user(db) + for i in range(5): + create_test_document(db, user=user, filename=f"pagdoc_{i}.pdf") + + repo = DocumentRepository(db) + page1 = repo.get_all(offset=0, limit=3) + assert len(page1) == 3 + + page2 = repo.get_all(offset=3, limit=3) + assert len(page2) == 2 + + def test_get_all_with_status_filter(self, db: Session) -> None: + user = create_test_user(db) + create_test_document(db, user=user, filename="pend.pdf", status="pending") + create_test_document(db, user=user, filename="comp.pdf", status="completed") + create_test_document(db, user=user, filename="fail.pdf", status="failed") + + repo = DocumentRepository(db) + pending = repo.get_all(offset=0, limit=100, filters={"status": "pending"}) + assert all(d.status == "pending" for d in pending) + + def test_count(self, db: Session) -> None: + user = create_test_user(db) + create_test_document(db, user=user, filename="cnt1.pdf") + create_test_document(db, user=user, filename="cnt2.pdf") + + repo = DocumentRepository(db) + total = repo.count() + assert total >= 2 + + def test_count_with_filter(self, db: Session) -> None: + user = create_test_user(db) + create_test_document(db, user=user, filename="cnt_p.pdf", status="pending") + create_test_document(db, user=user, filename="cnt_c.pdf", status="completed") + + repo = DocumentRepository(db) + pending_count = repo.count(filters={"status": "pending"}) + assert pending_count >= 1 + + def test_delete(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + repo = DocumentRepository(db) + repo.delete(doc) + db.flush() + + assert repo.get_by_id(doc.id) is None + + def test_get_by_checksum(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + repo = DocumentRepository(db) + found = repo.get_by_checksum(doc.checksum) + assert found is not None + assert found.id == doc.id + + +class TestDocumentPageRepository: + """Tests for DocumentPageRepository.""" + + def test_create_page(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + page = create_test_page(db, doc, page_number=1) + assert page.id is not None + assert page.document_id == doc.id + assert page.page_number == 1 + + def test_get_pages_by_document(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + create_test_page(db, doc, page_number=1) + create_test_page(db, doc, page_number=2) + create_test_page(db, doc, page_number=3) + + repo = DocumentPageRepository(db) + pages = repo.get_document_pages(doc.id) + assert len(pages) == 3 + assert [p.page_number for p in pages] == [1, 2, 3] + + +class TestDocumentTextBlockRepository: + """Tests for DocumentTextBlockRepository.""" + + def test_create_text_block(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + page = create_test_page(db, doc) + block = create_test_text_block(db, page, text="Hello World") + assert block.id is not None + assert block.text == "Hello World" + assert block.page_id == page.id + + def test_get_blocks_by_page(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + page = create_test_page(db, doc) + create_test_text_block(db, page, text="First", sequence=0) + create_test_text_block(db, page, text="Second", sequence=1, y=120.0) + create_test_text_block(db, page, text="Third", sequence=2, y=140.0) + + repo = DocumentTextBlockRepository(db) + blocks = repo.get_page_text_blocks(page.id) + assert len(blocks) == 3 + + def test_get_blocks_by_type(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user) + page = create_test_page(db, doc) + create_test_text_block(db, page, text="Header", block_type="header") + create_test_text_block(db, page, text="Body", block_type="text", y=200.0) + create_test_text_block(db, page, text="Footer", block_type="footer", y=700.0) + + repo = DocumentTextBlockRepository(db) + headers = repo.get_by_block_type(page.id, "header") + assert len(headers) == 1 + assert headers[0].text == "Header" + + +class TestTemplateMatchRepository: + """Tests for TemplateMatchRepository.""" + + def test_create_match(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + template = create_test_template(db, created_by=user) + + match = TemplateMatch( + id=uuid.uuid4(), + document_id=doc.id, + format_id=template.id, + confidence_score=0.85, + selected=True, + ) + db.add(match) + db.flush() + + repo = TemplateMatchRepository(db) + matches = repo.get_document_matches(doc.id) + assert len(matches) == 1 + assert matches[0].confidence_score == 0.85 + + def test_get_selected_match(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + t1 = create_test_template(db, name="Low", created_by=user) + t2 = create_test_template(db, name="High", created_by=user) + + db.add(TemplateMatch( + id=uuid.uuid4(), document_id=doc.id, format_id=t1.id, + confidence_score=0.5, selected=False, + )) + db.add(TemplateMatch( + id=uuid.uuid4(), document_id=doc.id, format_id=t2.id, + confidence_score=0.95, selected=True, + )) + db.flush() + + repo = TemplateMatchRepository(db) + best = repo.get_selected_match(doc.id) + assert best is not None + assert best.format_id == t2.id + assert best.confidence_score == 0.95 + + def test_get_document_matches_ordered(self, db: Session) -> None: + user = create_test_user(db) + doc = create_test_document(db, user=user, status="completed") + t1 = create_test_template(db, name="T1", created_by=user) + t2 = create_test_template(db, name="T2", created_by=user) + t3 = create_test_template(db, name="T3", created_by=user) + + db.add(TemplateMatch( + id=uuid.uuid4(), document_id=doc.id, format_id=t1.id, + confidence_score=0.3, selected=False, + )) + db.add(TemplateMatch( + id=uuid.uuid4(), document_id=doc.id, format_id=t2.id, + confidence_score=0.9, selected=True, + )) + db.add(TemplateMatch( + id=uuid.uuid4(), document_id=doc.id, format_id=t3.id, + confidence_score=0.6, selected=False, + )) + db.flush() + + repo = TemplateMatchRepository(db) + matches = repo.get_document_matches(doc.id) + scores = [m.confidence_score for m in matches] + assert scores == sorted(scores, reverse=True) diff --git a/docengine/tests/repositories/test_user_repository.py b/docengine/tests/repositories/test_user_repository.py new file mode 100644 index 0000000..0ae7d85 --- /dev/null +++ b/docengine/tests/repositories/test_user_repository.py @@ -0,0 +1,292 @@ +from __future__ import annotations + +import uuid +from datetime import UTC, datetime, timedelta + +import pytest +from sqlalchemy.orm import Session + +from app.models.user import RefreshToken, Role, User +from app.repositories.user_repository import ( + AuditLogRepository, + RefreshTokenRepository, + RoleRepository, + UserRepository, +) +from tests.conftest import create_test_role, create_test_user + + +class TestUserRepository: + """Tests for UserRepository CRUD operations.""" + + def test_create_user(self, db: Session) -> None: + repo = UserRepository(db) + user = repo.create_user( + username="repo_user", + email="repo@test.com", + hashed_password="$2b$12$fakehash", + full_name="Repo User", + ) + assert user.id is not None + assert user.username == "repo_user" + assert user.email == "repo@test.com" + assert user.is_active is True + assert user.is_superuser is False + + def test_get_by_username(self, db: Session) -> None: + user = create_test_user(db, username="findme") + repo = UserRepository(db) + found = repo.get_by_username("findme") + assert found is not None + assert found.id == user.id + + def test_get_by_username_not_found(self, db: Session) -> None: + repo = UserRepository(db) + assert repo.get_by_username("nonexistent") is None + + def test_get_by_email(self, db: Session) -> None: + user = create_test_user(db, email="email@test.com") + repo = UserRepository(db) + found = repo.get_by_email("email@test.com") + assert found is not None + assert found.id == user.id + + def test_get_by_email_not_found(self, db: Session) -> None: + repo = UserRepository(db) + assert repo.get_by_email("nobody@test.com") is None + + def test_get_by_id(self, db: Session) -> None: + user = create_test_user(db) + repo = UserRepository(db) + found = repo.get_by_id(user.id) + assert found is not None + assert found.username == user.username + + def test_get_by_id_not_found(self, db: Session) -> None: + repo = UserRepository(db) + assert repo.get_by_id(uuid.uuid4()) is None + + def test_update_last_login(self, db: Session) -> None: + user = create_test_user(db) + assert user.last_login is None + repo = UserRepository(db) + updated = repo.update_last_login(user) + assert updated.last_login is not None + + def test_get_active_users(self, db: Session) -> None: + create_test_user(db, username="active1", is_active=True) + create_test_user(db, username="active2", is_active=True) + create_test_user(db, username="inactive1", is_active=False) + + repo = UserRepository(db) + active = repo.get_active_users() + usernames = [u.username for u in active] + assert "active1" in usernames + assert "active2" in usernames + assert "inactive1" not in usernames + + def test_create_user_with_roles(self, db: Session) -> None: + create_test_role(db, name="admin") + create_test_role(db, name="user") + + repo = UserRepository(db) + user = repo.create_user( + username="roled_user", + email="roled@test.com", + hashed_password="$2b$12$fakehash", + role_names=["admin", "user"], + ) + role_names = [r.name for r in user.roles] + assert "admin" in role_names + assert "user" in role_names + + def test_assign_roles(self, db: Session) -> None: + create_test_role(db, name="viewer") + user = create_test_user(db) + repo = UserRepository(db) + updated = repo.assign_roles(user, ["viewer"]) + role_names = [r.name for r in updated.roles] + assert "viewer" in role_names + + def test_delete_user(self, db: Session) -> None: + user = create_test_user(db) + repo = UserRepository(db) + assert repo.delete_by_id(user.id) is True + assert repo.get_by_id(user.id) is None + + def test_delete_nonexistent_user(self, db: Session) -> None: + repo = UserRepository(db) + assert repo.delete_by_id(uuid.uuid4()) is False + + def test_exists(self, db: Session) -> None: + user = create_test_user(db) + repo = UserRepository(db) + assert repo.exists(user.id) is True + assert repo.exists(uuid.uuid4()) is False + + def test_count(self, db: Session) -> None: + create_test_user(db, username="count1") + create_test_user(db, username="count2") + repo = UserRepository(db) + assert repo.count() >= 2 + + +class TestRoleRepository: + """Tests for RoleRepository.""" + + def test_create_role(self, db: Session) -> None: + repo = RoleRepository(db) + role = repo.create_role(name="editor", description="Can edit documents") + assert role.id is not None + assert role.name == "editor" + + def test_get_by_name(self, db: Session) -> None: + create_test_role(db, name="tester") + repo = RoleRepository(db) + found = repo.get_by_name("tester") + assert found is not None + assert found.name == "tester" + + def test_get_by_name_not_found(self, db: Session) -> None: + repo = RoleRepository(db) + assert repo.get_by_name("nonexistent_role") is None + + def test_get_all_roles(self, db: Session) -> None: + create_test_role(db, name="role_a") + create_test_role(db, name="role_b") + repo = RoleRepository(db) + roles = repo.get_all_roles() + names = [r.name for r in roles] + assert "role_a" in names + assert "role_b" in names + + +class TestRefreshTokenRepository: + """Tests for RefreshTokenRepository.""" + + def test_create_token(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + expires = datetime.now(UTC) + timedelta(days=7) + token = repo.create_token( + user_id=user.id, + token="test_refresh_token_abc", + expires_at=expires, + ) + assert token.id is not None + assert token.user_id == user.id + assert token.revoked is False + + def test_get_by_token(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + expires = datetime.now(UTC) + timedelta(days=7) + repo.create_token(user_id=user.id, token="findable_token", expires_at=expires) + + found = repo.get_by_token("findable_token") + assert found is not None + assert found.user_id == user.id + + def test_get_by_token_not_found(self, db: Session) -> None: + repo = RefreshTokenRepository(db) + assert repo.get_by_token("nonexistent_token") is None + + def test_get_by_revoked_token_returns_none(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + expires = datetime.now(UTC) + timedelta(days=7) + repo.create_token(user_id=user.id, token="revoked_token", expires_at=expires) + repo.revoke_token("revoked_token") + db.flush() + + assert repo.get_by_token("revoked_token") is None + + def test_revoke_token(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + expires = datetime.now(UTC) + timedelta(days=7) + repo.create_token(user_id=user.id, token="to_revoke", expires_at=expires) + + assert repo.revoke_token("to_revoke") is True + assert repo.get_by_token("to_revoke") is None + + def test_revoke_nonexistent_token(self, db: Session) -> None: + repo = RefreshTokenRepository(db) + assert repo.revoke_token("does_not_exist") is False + + def test_revoke_all_user_tokens(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + expires = datetime.now(UTC) + timedelta(days=7) + repo.create_token(user_id=user.id, token="token_1", expires_at=expires) + repo.create_token(user_id=user.id, token="token_2", expires_at=expires) + repo.create_token(user_id=user.id, token="token_3", expires_at=expires) + + count = repo.revoke_all_user_tokens(user.id) + assert count == 3 + assert repo.get_by_token("token_1") is None + assert repo.get_by_token("token_2") is None + assert repo.get_by_token("token_3") is None + + def test_cleanup_expired_tokens(self, db: Session) -> None: + user = create_test_user(db) + repo = RefreshTokenRepository(db) + + # Create expired token + expired = datetime.now(UTC) - timedelta(days=1) + repo.create_token(user_id=user.id, token="expired_tok", expires_at=expired) + + # Create valid token + valid = datetime.now(UTC) + timedelta(days=7) + repo.create_token(user_id=user.id, token="valid_tok", expires_at=valid) + + count = repo.cleanup_expired_tokens() + assert count >= 1 + + +class TestAuditLogRepository: + """Tests for AuditLogRepository.""" + + def test_log_action(self, db: Session) -> None: + user = create_test_user(db) + repo = AuditLogRepository(db) + log = repo.log_action( + action="login", + resource_type="auth", + user_id=user.id, + ip_address="127.0.0.1", + user_agent="TestAgent/1.0", + ) + assert log.id is not None + assert log.action == "login" + assert log.resource_type == "auth" + + def test_log_action_without_user(self, db: Session) -> None: + repo = AuditLogRepository(db) + log = repo.log_action( + action="anonymous_access", + resource_type="documents", + ) + assert log.id is not None + assert log.user_id is None + + def test_get_user_logs(self, db: Session) -> None: + user = create_test_user(db) + repo = AuditLogRepository(db) + repo.log_action(action="view", resource_type="documents", user_id=user.id) + repo.log_action(action="edit", resource_type="templates", user_id=user.id) + + logs = repo.get_user_logs(user.id) + assert len(logs) >= 2 + + def test_get_resource_logs(self, db: Session) -> None: + repo = AuditLogRepository(db) + resource_id = str(uuid.uuid4()) + repo.log_action(action="create", resource_type="documents", resource_id=resource_id) + repo.log_action(action="update", resource_type="documents", resource_id=resource_id) + + logs = repo.get_resource_logs("documents", resource_id) + assert len(logs) >= 2 + for log in logs: + assert log.resource_type == "documents" + assert log.resource_id == resource_id diff --git a/docengine/tests/unit/__init__.py b/docengine/tests/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/unit/test_matching_service.py b/docengine/tests/unit/test_matching_service.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/unit/test_security.py b/docengine/tests/unit/test_security.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/unit/test_storage.py b/docengine/tests/unit/test_storage.py new file mode 100644 index 0000000..e69de29 diff --git a/docengine/tests/unit/test_template_service.py b/docengine/tests/unit/test_template_service.py new file mode 100644 index 0000000..e69de29 diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 6a48e3b..220b2fc 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -16,8 +16,10 @@ "@angular/forms": "^21.1.0", "@angular/platform-browser": "^21.1.0", "@angular/router": "^21.1.0", + "@primeng/themes": "^19.1.0", + "primeflex": "^4.0.0", "primeicons": "^7.0.0", - "primeng": "^17.18.0", + "primeng": "^19.1.0", "quill": "^2.0.3", "rxjs": "~7.8.0", "tslib": "^2.3.0", @@ -3001,6 +3003,37 @@ "license": "MIT", "optional": true }, + "node_modules/@primeng/themes": { + "version": "19.1.4", + "resolved": "https://registry.npmjs.org/@primeng/themes/-/themes-19.1.4.tgz", + "integrity": "sha512-Hze5bBTjsLzZXb20qsm9apsFuzpZzXiU+Ulj/7R+2fwMmcQk0XpkQS7V88fsFw6xsTD7+R+hgqr7Rzy0Gf+4dw==", + "deprecated": "Deprecated. This package is no longer maintained. Please migrate to @primeuix/themes: https://www.npmjs.com/package/@primeuix/themes", + "license": "SEE LICENSE IN LICENSE.md", + "dependencies": { + "@primeuix/styled": "^0.3.2" + } + }, + "node_modules/@primeng/themes/node_modules/@primeuix/styled": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/@primeuix/styled/-/styled-0.3.2.tgz", + "integrity": "sha512-ColZes0+/WKqH4ob2x8DyNYf1NENpe5ZguOvx5yCLxaP8EIMVhLjWLO/3umJiDnQU4XXMLkn2mMHHw+fhTX/mw==", + "license": "MIT", + "dependencies": { + "@primeuix/utils": "^0.3.2" + }, + "engines": { + "node": ">=12.11.0" + } + }, + "node_modules/@primeng/themes/node_modules/@primeuix/utils": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/@primeuix/utils/-/utils-0.3.2.tgz", + "integrity": "sha512-B+nphqTQeq+i6JuICLdVWnDMjONome2sNz0xI65qIOyeB4EF12CoKRiCsxuZ5uKAkHi/0d1LqlQ9mIWRSdkavw==", + "license": "MIT", + "engines": { + "node": ">=12.11.0" + } + }, "node_modules/@rolldown/binding-android-arm64": { "version": "1.0.0-beta.58", "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-beta.58.tgz", @@ -6570,6 +6603,12 @@ "dev": true, "license": "MIT" }, + "node_modules/primeflex": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/primeflex/-/primeflex-4.0.0.tgz", + "integrity": "sha512-UOEZCRjR36+sm5bUpDhS1xbA068l9VC6y1aTNVqQPtXuKIdPTqAWHRUxj3mKAoPrQ9W373ooJJMgNVXfiaw04g==", + "license": "MIT" + }, "node_modules/primeicons": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/primeicons/-/primeicons-7.0.0.tgz", @@ -6577,19 +6616,45 @@ "license": "MIT" }, "node_modules/primeng": { - "version": "17.18.0", - "resolved": "https://registry.npmjs.org/primeng/-/primeng-17.18.0.tgz", - "integrity": "sha512-EcvU/0Ex9QoBR6g6db9fDTCTAmzokW70TV5Oroy2gdvXRr3eqlflnOBoArQsmxTaw1oxSsu68YVj3RvcKYWhTg==", - "license": "MIT", + "version": "19.1.4", + "resolved": "https://registry.npmjs.org/primeng/-/primeng-19.1.4.tgz", + "integrity": "sha512-l5l8SHTxopxxyyXZx1BvbQ11P7ndLv2Qp8H5k2/+OCi65jTZn4xmtrBDGDs7k2K5UMHSqAnGjBgtpbckyqQETg==", + "license": "SEE LICENSE IN LICENSE.md", "dependencies": { + "@primeuix/styled": "^0.3.2", + "@primeuix/utils": "^0.3.2", "tslib": "^2.3.0" }, "peerDependencies": { - "@angular/common": "^17.0.0 || ^18.0.0", - "@angular/core": "^17.0.0 || ^18.0.0", - "@angular/forms": "^17.0.0 || ^18.0.0", - "rxjs": "^6.0.0 || ^7.8.1", - "zone.js": "~0.14.0" + "@angular/animations": "^19.0.0", + "@angular/cdk": "^19.0.0", + "@angular/common": "^19.0.0", + "@angular/core": "^19.0.0", + "@angular/forms": "^19.0.0", + "@angular/platform-browser": "^19.0.0", + "@angular/router": "^19.0.0", + "rxjs": "^6.0.0 || ^7.8.1" + } + }, + "node_modules/primeng/node_modules/@primeuix/styled": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/@primeuix/styled/-/styled-0.3.2.tgz", + "integrity": "sha512-ColZes0+/WKqH4ob2x8DyNYf1NENpe5ZguOvx5yCLxaP8EIMVhLjWLO/3umJiDnQU4XXMLkn2mMHHw+fhTX/mw==", + "license": "MIT", + "dependencies": { + "@primeuix/utils": "^0.3.2" + }, + "engines": { + "node": ">=12.11.0" + } + }, + "node_modules/primeng/node_modules/@primeuix/utils": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/@primeuix/utils/-/utils-0.3.2.tgz", + "integrity": "sha512-B+nphqTQeq+i6JuICLdVWnDMjONome2sNz0xI65qIOyeB4EF12CoKRiCsxuZ5uKAkHi/0d1LqlQ9mIWRSdkavw==", + "license": "MIT", + "engines": { + "node": ">=12.11.0" } }, "node_modules/proc-log": { diff --git a/frontend/package.json b/frontend/package.json index 679ab77..626ead1 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "scripts": { "ng": "ng", - "start": "ng serve", + "start": "ng serve --proxy-config proxy.conf.json", "build": "ng build", "watch": "ng build --watch --configuration development", "test": "ng test" @@ -31,8 +31,10 @@ "@angular/forms": "^21.1.0", "@angular/platform-browser": "^21.1.0", "@angular/router": "^21.1.0", + "@primeng/themes": "^19.1.0", + "primeflex": "^4.0.0", "primeicons": "^7.0.0", - "primeng": "^17.18.0", + "primeng": "^19.1.0", "quill": "^2.0.3", "rxjs": "~7.8.0", "tslib": "^2.3.0", diff --git a/frontend/proxy.conf.json b/frontend/proxy.conf.json new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/frontend/proxy.conf.json @@ -0,0 +1 @@ +{} diff --git a/frontend/public/assets/images/logo.png b/frontend/public/assets/images/logo.png new file mode 100644 index 0000000..642e236 Binary files /dev/null and b/frontend/public/assets/images/logo.png differ diff --git a/frontend/src/app/app.config.ts b/frontend/src/app/app.config.ts index 9118a55..98c83a9 100644 --- a/frontend/src/app/app.config.ts +++ b/frontend/src/app/app.config.ts @@ -1,7 +1,10 @@ import { ApplicationConfig, provideZoneChangeDetection } from '@angular/core'; import { provideRouter } from '@angular/router'; import { provideAnimationsAsync } from '@angular/platform-browser/animations/async'; -import { provideHttpClient } from '@angular/common/http'; +import { provideHttpClient, withInterceptors } from '@angular/common/http'; +import { AuthInterceptor } from './interceptors/auth.interceptor'; +import { providePrimeNG } from 'primeng/config'; +import Aura from '@primeng/themes/aura'; import { routes } from './app.routes'; @@ -10,6 +13,11 @@ export const appConfig: ApplicationConfig = { provideZoneChangeDetection({ eventCoalescing: true }), provideRouter(routes), provideAnimationsAsync(), - provideHttpClient() + providePrimeNG({ + theme: { + preset: Aura + } + }), + provideHttpClient(withInterceptors([AuthInterceptor])) ] }; diff --git a/frontend/src/app/app.routes.ts b/frontend/src/app/app.routes.ts index 8199137..0f67e5e 100644 --- a/frontend/src/app/app.routes.ts +++ b/frontend/src/app/app.routes.ts @@ -1,21 +1,44 @@ import { Routes } from '@angular/router'; import { LoginComponent } from './login/login.component'; -import { AdminLayoutComponent } from './admin-layout/admin-layout.component'; +import { DashboardComponent } from './pages/dashboard/dashboard.component'; import { OcrComponent } from './ocr/ocr.component'; import { MailboxComponent } from './mailbox/mailbox.component'; -import { AuthGuard } from './auth.guard'; +import { TemplatesComponent } from './templates/templates.component'; + +import { AuthorizeComponent } from './pages/session/auth/authorize.component'; +import { AuthorizeGuard } from './interceptors/authorize.guard'; +import { ProfileComponent } from './pages/session/profile/profile.component'; +import { SubsidiaryComponent } from './pages/account/company/subsidiary/subsidiary.component'; +import { DepartmentComponent } from './pages/account/company/department/department.component'; +import { DesignationComponent } from './pages/account/company/desigation/designation.component'; +import { EmployeeComponent } from './pages/account/company/employee/employee.component'; +import { UserComponent } from './pages/account/user/user.component'; export const routes: Routes = [ - { path: 'login', component: LoginComponent }, - { - path: '', - component: AdminLayoutComponent, - canActivate: [AuthGuard], - children: [ - { path: '', redirectTo: 'ocr', pathMatch: 'full' }, - { path: 'ocr', component: OcrComponent }, - { path: 'mailbox', component: MailboxComponent } - ] - }, - { path: '**', redirectTo: '' } -]; + { path: '', component: LoginComponent }, + { path: 'authorize', component: AuthorizeComponent, canActivate: [AuthorizeGuard]}, + { path: 'user', + component: DashboardComponent, + canActivate: [AuthorizeGuard], + canActivateChild: [AuthorizeGuard], + children: [ + { path: 'profile', component: ProfileComponent }, + { path: 'mailbox', component: MailboxComponent }, + { path: 'ocr', component: OcrComponent }, + { path: 'templates', component: TemplatesComponent } + ] + }, + { path: 'account', + component: DashboardComponent, + canActivate: [AuthorizeGuard], + canActivateChild: [AuthorizeGuard], + children: [ + { path: 'profile', component: ProfileComponent }, + { path: 'subsidiaries', component: SubsidiaryComponent }, + { path: 'departments', component: DepartmentComponent }, + { path: 'designations', component: DesignationComponent }, + { path: 'employees', component: EmployeeComponent }, + { path: 'users', component: UserComponent } + ] + } +]; \ No newline at end of file diff --git a/frontend/src/app/fragments/menu/menu.component.html b/frontend/src/app/fragments/menu/menu.component.html new file mode 100644 index 0000000..8b07f83 --- /dev/null +++ b/frontend/src/app/fragments/menu/menu.component.html @@ -0,0 +1,21 @@ +
+ + + + + +
+
+ + {{ companyName }} + + + {{ name }} | {{ branchName }} + +
+ + +
+ + +
\ No newline at end of file diff --git a/frontend/src/app/fragments/menu/menu.component.ts b/frontend/src/app/fragments/menu/menu.component.ts new file mode 100644 index 0000000..e315341 --- /dev/null +++ b/frontend/src/app/fragments/menu/menu.component.ts @@ -0,0 +1,110 @@ +import { Component, Input } from '@angular/core'; +import { Router } from '@angular/router'; +import { MenuItem } from 'primeng/api'; +import { Menubar } from 'primeng/menubar'; +import { HttpService } from '../../services/http.service'; +import { Request } from '../../models/request.model'; +import { AvatarModule } from 'primeng/avatar'; +import { Menu } from 'primeng/menu'; +import { ButtonModule } from 'primeng/button'; +import { InputTextModule } from 'primeng/inputtext'; +import { SessionService } from '../../services/commons/session.service'; +import { EncryptionService } from '../../services/utilities/encryption.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { environment } from '../../../environments/environment'; + +@Component({ + selector: 'app-menu', + standalone: true, + imports: [Menubar, AvatarModule, ButtonModule, Menu, InputTextModule, TooltipModule], + templateUrl: './menu.component.html' +}) +export class MenuComponent { + items: MenuItem[] = []; + profileItems: MenuItem[] | undefined; + companyName: string = ''; + branchName: string = ''; + roleName: string = ''; + name: string = ''; + constructor(private router: Router, private http: HttpService, private session: SessionService, private enc: EncryptionService) { } + ngOnInit() { + if (typeof window !== 'undefined' && sessionStorage) { + let menuString = this.session.getItem("nav"); + if (menuString) { + this.enc.decrypt(menuString).then(decrypted => { + if (decrypted) { + this.items = JSON.parse(decrypted).map((m: any) => this.mapMenu(m)); + console.log(this.items); + } + }); + } + const companies = this.session.getItem("companies") ? JSON.parse(this.session.getItem("companies")) : ''; + const userDetails = this.session.getItem('userDetails'); + this.name = userDetails ? JSON.parse(userDetails).displayName : ''; + this.companyName = companies ? companies[0].companyName : companies; + this.branchName = companies ? companies[0].branches[0].branchCode : ''; + this.roleName = companies ? companies[0].branches[0].roles[0].groupName : '' + } + this.profileItems = [ + { + label: this.roleName, + items: [ + { + label: 'Details', + icon: 'pi pi-id-card', + command: () => { + this.router.navigateByUrl(`/user/profile`); + } + }, + { + label: 'Sign Out', + icon: 'pi pi-sign-out', + command: () => { + this.logout(); + } + } + ] + } + ]; + } + + logout() { + const requestPayload: Request = { + data: '', + compressed: false + }; + this.http.post(environment.authService + '/signout', requestPayload).subscribe({ + next: () => { + sessionStorage.clear(); + this.router.navigate(['/']); + }, + error: (err) => { + console.error('Logout failed', err); + } + }); + } + + mapMenu(menu: any): MenuItem { + const hasChildren = Array.isArray(menu.items) && menu.items.length > 0; + + return { + label: menu.label, + icon: menu.icon, + routerLink: !hasChildren && menu.route !== '#' + ? menu.route + : undefined, + command: !hasChildren && menu.route !== '#' + ? () => { + this.router.navigateByUrl(menu.route); + } + : undefined, + items: hasChildren + ? menu.items.map((c: any) => this.mapMenu(c)) + : undefined // 🔥 THIS IS CRITICAL + }; + } + + gotToDashboard() { + this.router.navigate(['/account']); + } +} diff --git a/frontend/src/app/interceptors/auth.interceptor.ts b/frontend/src/app/interceptors/auth.interceptor.ts new file mode 100644 index 0000000..9af87f0 --- /dev/null +++ b/frontend/src/app/interceptors/auth.interceptor.ts @@ -0,0 +1,134 @@ +import { SessionService } from './../services/commons/session.service'; +import { + HttpInterceptorFn, + HttpRequest, + HttpHandlerFn, + HttpEvent, + HttpResponse, + HttpErrorResponse, + HttpHeaders +} from '@angular/common/http'; +import { Observable, tap, catchError, throwError, from, switchMap } from 'rxjs'; +import { EncryptionService } from '../services/utilities/encryption.service'; +import { inject } from '@angular/core'; +import { ResponseDto } from '../models/response.dto'; + +export const AuthInterceptor: HttpInterceptorFn = ( + req: HttpRequest, + next: HttpHandlerFn +): Observable> => { + + const encryptionService = inject(EncryptionService); + const sessionService = inject(SessionService); + + // Bypass interceptor for external webhooks + if (req.url.includes('webhook.site') || req.url.includes('beeceptor.com')) { + return next(req); + } + + /* ----------------------------------------- + * 1️⃣ Always attach Authorization header + * ----------------------------------------- */ + const token = sessionService.getItem('token'); + + let reqHeaders = req.headers; + + if (!(req.body instanceof FormData)) { + reqHeaders = reqHeaders.set('Content-Type', 'application/json'); + } + + if (token) { + reqHeaders = reqHeaders.set( + 'Authorization', + `Bearer ${token}` + ); + } + + /* ----------------------------------------- + * 2️⃣ Handle encrypted (compressed) requests + * ----------------------------------------- */ + if ( + req.body && + typeof req.body === 'object' && + req.body.compressed + ) { + return from(encryptionService.encrypt(req.body.data)).pipe( + switchMap(encryptedBody => { + + const payload = { + scopes: req.body.scopes ?? [], + data: encryptedBody, + target: req.body.target ?? null, + compressed: true + }; + + const modifiedReq = req.clone({ + body: payload, + headers: reqHeaders + }); + + return next(modifiedReq).pipe( + tap(event => handleResponse(event, sessionService, encryptionService)), + catchError((error: HttpErrorResponse) => + throwError(() => error) + ) + ); + }) + ); + } + + /* ----------------------------------------- + * 3️⃣ Non-encrypted requests (GET included) + * ----------------------------------------- */ + const modifiedReq = req.clone({ + headers: reqHeaders + }); + + return next(modifiedReq).pipe( + tap(event => handleResponse(event, sessionService, encryptionService)), + catchError((error: HttpErrorResponse) => + throwError(() => error) + ) + ); +}; + +function handleResponse( + event: HttpEvent, + sessionService: SessionService, + encryptionService: EncryptionService +): void { + + if (!(event instanceof HttpResponse) || !event.ok) { + return; + } + + if (event.url?.endsWith('/3z4mkell5g5aset/authenticate')) { + const response = event.body as ResponseDto; + sessionService.setItem('token', response?.data?.token); + sessionService.setItem('companies', response?.data?.companies); + } + + if (event.url?.endsWith('/3z4mkell5g5aset/authorize')) { + const response = event.body as ResponseDto; + sessionService.clear(); + sessionService.setItem('token', response?.data?. token); + sessionService.setItem('refreshToken', response?.data?.refreshToken); + sessionService.setItem('userDetails', JSON.stringify(response?.data?.userDetails)); + encryptionService.encrypt(response?.data?.data) + .then(encryptedNav => { + sessionService.setItem('nav', encryptedNav); + }); + + if (response?.data?.userDetails) { + sessionService.setItem( + 'userDetails', + JSON.stringify(response.data.userDetails) + ); + } + + sessionService.setItem( + 'companies', + JSON.stringify(response?.data?.companies) + ); + } +} \ No newline at end of file diff --git a/frontend/src/app/interceptors/authorize.guard.ts b/frontend/src/app/interceptors/authorize.guard.ts new file mode 100644 index 0000000..0789af8 --- /dev/null +++ b/frontend/src/app/interceptors/authorize.guard.ts @@ -0,0 +1,40 @@ +import { Injectable, inject, PLATFORM_ID } from '@angular/core'; +import { ActivatedRouteSnapshot, CanActivate, CanActivateChild, Router, RouterStateSnapshot } from '@angular/router'; +import { isPlatformBrowser } from '@angular/common'; + +@Injectable({ providedIn: 'root' }) +export class AuthorizeGuard implements CanActivate, CanActivateChild { + + private platformId = inject(PLATFORM_ID); + private router = inject(Router); + + private checkAuth(): boolean { + if (!isPlatformBrowser(this.platformId)) { + return false; + } + + const token = sessionStorage.getItem('token'); + + if (!token) { + sessionStorage.clear(); + this.router.navigate(['/']); + return false; + } + + return true; + } + + canActivate( + route: ActivatedRouteSnapshot, + state: RouterStateSnapshot + ): boolean { + return this.checkAuth(); + } + + canActivateChild( + childRoute: ActivatedRouteSnapshot, + state: RouterStateSnapshot + ): boolean { + return this.checkAuth(); + } +} diff --git a/frontend/src/app/login/login.component.css b/frontend/src/app/login/login.component.css new file mode 100644 index 0000000..e83b32f --- /dev/null +++ b/frontend/src/app/login/login.component.css @@ -0,0 +1,43 @@ +::ng-deep .p-password-input { + border-top-left-radius: 0 !important; + border-bottom-left-radius: 0 !important; +} + +.login-page { + height: 99vh; +} + +::ng-deep .p-card { + position: relative; + border-radius: 24px !important; + background: white; + padding: 1rem; + overflow: hidden; +} + +/* Gradient border */ +::ng-deep .p-card::before { + content: ""; + position: absolute; + inset: 0; + padding: 3px; /* border thickness */ + border-radius: 24px; + filter: drop-shadow(0 0 12px rgba(91, 185, 138, 0.35)); + background: linear-gradient( + 180deg, + #5bb98a 0%, + rgba(91, 185, 138, 0.6) 40%, + rgba(91, 185, 138, 0.15) 70%, + transparent 100% + ); + + /* Mask trick = border only */ + -webkit-mask: + linear-gradient(#fff 0 0) content-box, + linear-gradient(#fff 0 0); + -webkit-mask-composite: xor; + mask-composite: exclude; + + pointer-events: none; +} + diff --git a/frontend/src/app/login/login.component.html b/frontend/src/app/login/login.component.html new file mode 100644 index 0000000..a1abfb7 --- /dev/null +++ b/frontend/src/app/login/login.component.html @@ -0,0 +1,50 @@ + diff --git a/frontend/src/app/login/login.component.ts b/frontend/src/app/login/login.component.ts index 8051b2c..f090337 100644 --- a/frontend/src/app/login/login.component.ts +++ b/frontend/src/app/login/login.component.ts @@ -1,76 +1,58 @@ -import { Component } from '@angular/core'; -import { CommonModule } from '@angular/common'; +import { CommonModule} from '@angular/common'; import { FormsModule } from '@angular/forms'; -import { AuthService } from '../auth.service'; -import { Router } from '@angular/router'; -import { MessageService } from 'primeng/api'; - -// PrimeNG Imports -import { CardModule } from 'primeng/card'; +import { Component } from '@angular/core'; +import { ButtonModule } from 'primeng/button';; import { InputTextModule } from 'primeng/inputtext'; -import { ButtonModule } from 'primeng/button'; -import { ToastModule } from 'primeng/toast'; +import { CardModule } from 'primeng/card'; +import { InputGroupModule } from 'primeng/inputgroup'; +import { InputGroupAddonModule } from 'primeng/inputgroupaddon'; +import { PasswordModule } from 'primeng/password'; +import { FloatLabelModule } from 'primeng/floatlabel'; +import { KeyFilterModule } from 'primeng/keyfilter'; +import { FormBuilder, FormGroup, Validators, ReactiveFormsModule} from '@angular/forms'; +import { Request } from '../models/request.model'; +import { HttpService } from '../services/http.service'; +import { MessageModule } from 'primeng/message'; +import { StyleClassModule } from 'primeng/styleclass'; +import { Router } from '@angular/router'; +import { ResponseDto } from '../models/response.dto'; +import { BadgeModule } from "primeng/badge"; +import { environment } from '../../environments/environment'; @Component({ selector: 'app-login', - standalone: true, - imports: [ - CommonModule, - FormsModule, - CardModule, - InputTextModule, - ButtonModule, - ToastModule - ], - providers: [MessageService], - template: ` - - `, - styles: [` - .login-container { - display: flex; - justify-content: center; - align-items: center; - height: 100vh; - background: var(--surface-card); - } - .w-full { width: 100%; } - .mt-4 { margin-top: 1.5rem; } - `] + templateUrl: './login.component.html', + styleUrls: ['./login.component.css'], + imports: [CommonModule, FormsModule, CardModule, ButtonModule, InputTextModule, InputGroupModule, InputGroupAddonModule, PasswordModule, + FloatLabelModule, ReactiveFormsModule, KeyFilterModule, MessageModule, StyleClassModule, BadgeModule] }) export class LoginComponent { - username = ''; - password = ''; + loginForm: FormGroup; + message?: string; - constructor(private auth: AuthService, private router: Router, private messageService: MessageService) {} - - onLogin() { - this.auth.login(this.username, this.password).subscribe({ - next: () => { - this.router.navigate(['/']); - }, - error: () => { - this.messageService.add({severity:'error', summary:'Error', detail:'Invalid Credentials'}); - } + constructor(private fb: FormBuilder, private http: HttpService, private router: Router) { + this.loginForm = this.fb.group({ + username: ['', Validators.required], + password: ['', Validators.required] }); } + + onLogin() { + this.message = ''; + if (this.loginForm.valid) { + const requestPayload: Request = { + data: this.loginForm.value, + compressed: true, + target: 'models.auth.Login' + }; + this.http.post(`${environment.authService}/3z4mkell5g5aset/authenticate`, requestPayload).subscribe({ + next: (response) => { + this.router.navigate(['/authorize']); + }, + error: (err) => { + this.message = err.message; + } + }); + } + } } diff --git a/frontend/src/app/login/login.html b/frontend/src/app/login/login.html deleted file mode 100644 index 147cfc4..0000000 --- a/frontend/src/app/login/login.html +++ /dev/null @@ -1 +0,0 @@ -

login works!

diff --git a/frontend/src/app/login/login.ts b/frontend/src/app/login/login.ts deleted file mode 100644 index 7888f76..0000000 --- a/frontend/src/app/login/login.ts +++ /dev/null @@ -1,11 +0,0 @@ -import { Component } from '@angular/core'; - -@Component({ - selector: 'app-login', - imports: [], - templateUrl: './login.html', - styleUrl: './login.scss', -}) -export class Login { - -} diff --git a/frontend/src/app/models/account.model.ts b/frontend/src/app/models/account.model.ts new file mode 100644 index 0000000..ac11b8d --- /dev/null +++ b/frontend/src/app/models/account.model.ts @@ -0,0 +1,151 @@ +export class SubsidiaryDTO { + id?: string; + companyId?: string; + companyName?: string; + code?: string; + name?: string; + officeNo?: string; + street?: string; + locality?: string; + cityId?: string; + stateId?: string; + cityName?: string; + stateName?: string; + pinCode?: string; + emailId?: string; + contactNo?: string; + contactPerson?: string; + panNo?: string; + cinNo?: string; + msmeNo?: string; + updatedAt?: Date; + updatedBy?: string; + updatedUser?: string; + active?: boolean; + + constructor(init?: Partial) { + Object.assign(this, init); + if (init?.updatedAt) { + this.updatedAt = new Date(init.updatedAt); + } + } +} + +export interface DepartmentDTO { + id: string; + companyId: string; + department: string; + parentDepartment?: string; + parentDepartmentName?: string; + createdUser: string; + updatedAt: string; + updatedUser: string; + active: boolean; +} + +export interface CompanyVerifierModel { + pkId?: number; + id?: string; + companyId: string; + employeeId?: string; + verifierCode: string; + verifierType: string; + fullName: string; + mobileNo: string; + emailId: string; + allocationDate?: string; + appActive: boolean; + photoMandatory: boolean; + createdAt?: string; + createdBy?: number; + createdUser?: string; + updatedAt?: string; + updatedBy?: number; + updatedUser?: string; + active: boolean; +} + +export interface DesignationDTO { + id: string; + companyId: string; + departmentId: string; + departmentName: string; + designation: string; + createdUser: string; + updatedAt: string; + updatedUser: string; + active: boolean; + hod: boolean; + payGrade?: string; +} + +export interface EmployeeDTO { + id: string; + subsidiaryId: string; + departmentId: string; + designationId: string; + subsidiaryName: string; + department: string; + designation: string; + employeeId?: string; + joiningDate?: string; + fullName: string; + fatherName?: string; + gender?: string; + dob?: string; + residenceAddress?: string; + residenceCityId?: string; + residenceStateId?: string; + residenceCityName?: string; + residenceStateName?: string; + permanentAddress?: string; + permanentCityId?: string; + permanentStateId?: string; + permanentCityName?: string; + permanentStateName?: string; + contactNo: string; + alternateNo?: string; + emailId: string; + createdUser: string; + updatedAt: string; + updatedUser: string; + active: boolean; +} + +export interface VendorBranchDTO { + id?: string; + fkVendorId?: string; + branchCode: string; + branchName: string; + officeNo?: string; + street?: string; + locality?: string; + cityId?: string; + stateId?: string; + stateName?: string; + cityName?: string; + pinCode?: string; + emailId?: string; + contactNo?: string; + contactPerson?: string; + gstNo?: string; + createdUser?: string; + updatedAt?: string; + updatedUser?: string; + active: boolean; +} + +export interface VendorDTO { + id?: string; + companyId?: string; + code: string; + name: string; + panNo?: string; + cinNo?: string; + msmeNo?: string; + createdUser?: string; + updatedAt?: string; + updatedUser?: string; + active: boolean; + branches?: VendorBranchDTO[]; +} \ No newline at end of file diff --git a/frontend/src/app/models/masters/masters.ts b/frontend/src/app/models/masters/masters.ts new file mode 100644 index 0000000..9d93f0e --- /dev/null +++ b/frontend/src/app/models/masters/masters.ts @@ -0,0 +1,16 @@ +export interface CityDTO { + id?:string, + stateId?:string, + stateName?:string, + cityName?:string, + stateCode?:string, + gstCode?:string +} + +export interface SearchDTO { + module?:string, + searchBy?:string, + searchValue?:string, + offset?:number, + limit?:number +} \ No newline at end of file diff --git a/frontend/src/app/models/masters/member/memberlist.ts b/frontend/src/app/models/masters/member/memberlist.ts new file mode 100644 index 0000000..d823bf6 --- /dev/null +++ b/frontend/src/app/models/masters/member/memberlist.ts @@ -0,0 +1,16 @@ +export interface MemberList { + id:string, + memberno:string, + membername:string, + fathername:string, + mobileno?:string, + membershipdate?:Date, + emailid?:string, + dob?:Date, + age?:number, + gender?:string, + status:string + isactive:boolean, + updatedat:Date, + updatedby:string +} diff --git a/frontend/src/app/models/ola.model.ts b/frontend/src/app/models/ola.model.ts new file mode 100644 index 0000000..a030247 --- /dev/null +++ b/frontend/src/app/models/ola.model.ts @@ -0,0 +1,7 @@ +export interface LocationDTO { + place_id?: string; + name?: string; + formatted_address?: string; + lat?: number; + lng?: number; +} diff --git a/frontend/src/app/models/request.model.ts b/frontend/src/app/models/request.model.ts new file mode 100644 index 0000000..0f44bda --- /dev/null +++ b/frontend/src/app/models/request.model.ts @@ -0,0 +1,6 @@ +export interface Request { + scopes?: string[]; + data: T; + target?: string; + compressed?: boolean; +} diff --git a/frontend/src/app/models/response.dto.ts b/frontend/src/app/models/response.dto.ts new file mode 100644 index 0000000..f3ce991 --- /dev/null +++ b/frontend/src/app/models/response.dto.ts @@ -0,0 +1,6 @@ +export interface ResponseDto { + statuscode : number; + message?: string; + description?: string; + data?: any; +} diff --git a/frontend/src/app/models/session.model.ts b/frontend/src/app/models/session.model.ts new file mode 100644 index 0000000..07687cf --- /dev/null +++ b/frontend/src/app/models/session.model.ts @@ -0,0 +1,35 @@ +export interface Company { + id?: string; + companyName?: string; + companyCode?: string; + branches?: Branch[]; +} + +export interface Branch { + id?: string; + branchName?: string; + branchCode?: string; + roles?: Role[]; +} + +export interface Role { + roleName?: string; + groupName?: string; + defaultRole?: boolean; +} + +export interface UserProfile { + username?: string; + employeeId?: string; + joiningDate?: Date; + displayName?: string; + department?: string; + designation?: string; + name?: string; + fatherName?: string; + gender?: string; + dob?: Date; + contactNo?: string; + alternateContactNo?: string; + emailId?: string; +} diff --git a/frontend/src/app/models/tools.model.ts b/frontend/src/app/models/tools.model.ts new file mode 100644 index 0000000..8a70ee4 --- /dev/null +++ b/frontend/src/app/models/tools.model.ts @@ -0,0 +1,52 @@ +export interface AreaDTO { + id?: string; + companyId?: string; + areaName?: string; + groupAVerifierId?: string; + groupBVerifierId?: string; + groupCVerifierId?: string; + groupAVerifierName?: string; + groupBVerifierName?: string; + groupCVerifierName?: string; + createdAt?: Date; + createdBy?: string; + createdUser?: string; + updatedAt?: Date; + updatedBy?: string; + updatedUser?: string; + active?: boolean; +} + +export interface LocalityDTO { + id?: string; + companyId?: string; + areaId?: string; + stateCityId?: string; + localityTypeId?: string; + olaPlaceId?: string; + olaLocality?: string; + localityName?: string; + adminArea?: string; + latitude?: number; + longitude?: number; + createdAt?: Date; + createdBy?: string; + createdUser?: string; + updatedAt?: Date; + updatedBy?: string; + updatedUser?: string; + active?: boolean; + stateName?: string; + pincode?: string; + areaName?: string; + localityTypeName?: string; + masterCityName?: string; + masterStateName?: string; +} + +export interface LocalityTypeDTO { + id?: string; + type?: string; + riskdetail?: string; + createdAt?: Date; +} diff --git a/frontend/src/app/models/user.model.ts b/frontend/src/app/models/user.model.ts new file mode 100644 index 0000000..4ccacc8 --- /dev/null +++ b/frontend/src/app/models/user.model.ts @@ -0,0 +1,30 @@ +export interface UserDTO { + id?: string; + loginId?: string; + loginPassword?: string; + displayName?: string; + fkEmployeeId?: string; + employeeId?: string; + employeeName?: string; + fatherName?: string; + department?: string; + designation?: string; + status?: string; + updatedAt?: string; + updatedUser?: string; + active?: boolean; + userRoles?: UserRoleDTO[]; +} + +export interface UserRoleDTO { + id?: string; + roleId?: string; + branchId?: string; + roleName?: string; + groupName?: string; + branchName?: string; + defaultRole?: boolean; + updatedAt?: string; + updatedUser?: string; + active?: boolean; +} \ No newline at end of file diff --git a/frontend/src/app/ocr.service.ts b/frontend/src/app/ocr.service.ts index 492efb5..a2f48f2 100644 --- a/frontend/src/app/ocr.service.ts +++ b/frontend/src/app/ocr.service.ts @@ -15,4 +15,19 @@ export class OcrService { formData.append('file', file); return this.http.post(`${this.apiUrl}/extract`, formData); } + + extractWithAI(text: string, filePath: string | null, modelType: string): Observable { + return this.http.post(`http://localhost:8000/api/extract/ai`, { text, file_path: filePath, model_type: modelType }); + } + + saveDocument(data: any): Observable { + return this.http.post(`http://localhost:8000/api/documents/save`, data); + } + + postToWebhook(data: any): Observable { + return this.http.post('https://wh7e48dd25f9e8f2fe92.free.beeceptor.com', JSON.stringify(data), { + headers: { 'Content-Type': 'application/json' }, + responseType: 'text' + }); + } } diff --git a/frontend/src/app/ocr/ocr.component.ts b/frontend/src/app/ocr/ocr.component.ts index c743cd9..1d1f981 100644 --- a/frontend/src/app/ocr/ocr.component.ts +++ b/frontend/src/app/ocr/ocr.component.ts @@ -7,8 +7,13 @@ import { MessageService } from 'primeng/api'; // PrimeNG import { FileUploadModule } from 'primeng/fileupload'; import { ProgressBarModule } from 'primeng/progressbar'; -import { InputTextareaModule } from 'primeng/inputtextarea'; +import { TextareaModule } from 'primeng/textarea'; import { ToastModule } from 'primeng/toast'; +import { ButtonModule } from 'primeng/button'; +import { TableModule } from 'primeng/table'; +import { CardModule } from 'primeng/card'; +import { RadioButtonModule } from 'primeng/radiobutton'; +import { InputTextModule } from 'primeng/inputtext'; @Component({ selector: 'app-ocr', @@ -18,17 +23,19 @@ import { ToastModule } from 'primeng/toast'; FormsModule, FileUploadModule, ProgressBarModule, - InputTextareaModule, - ToastModule + TextareaModule, + ToastModule, + ButtonModule, + TableModule, + CardModule, + RadioButtonModule, + InputTextModule ], providers: [MessageService], template: `

OCR Extraction

- +
-
-

Extracted Text Result:

- +
+
+

Extracted Text Result:

+ +
+ +
+
+

AI Analysis Mode:

+ +
+
+ + +
+
+ + +
+
+ + + + +
+ + +
@@ -65,16 +105,28 @@ import { ToastModule } from 'primeng/toast'; export class OcrComponent { extractedText: string | null = null; loading: boolean = false; + + aiLoading: boolean = false; + saveLoading: boolean = false; + postLoading: boolean = false; + aiResult: any = null; + + // Hybrid AI Props + modelType: string = 'text'; + filePath: string | null = null; constructor(private ocrService: OcrService, private messageService: MessageService) {} onUpload(event: any) { this.loading = true; + this.aiResult = null; // Reset AI result on new upload + this.filePath = null; const file = event.files[0]; this.ocrService.extractText(file).subscribe({ next: (res) => { this.extractedText = res.text; + this.filePath = res.file_path; this.loading = false; this.messageService.add({severity:'success', summary:'Success', detail:'Text Extracted Successfully'}); }, @@ -88,5 +140,80 @@ export class OcrComponent { onClear() { this.extractedText = null; + this.aiResult = null; + this.filePath = null; + } + + processWithAI() { + if (!this.extractedText) return; + + this.aiLoading = true; + // Pass text, filePath, and modelType + this.ocrService.extractWithAI(this.extractedText, this.filePath, this.modelType).subscribe({ + next: (res) => { + this.aiResult = res; + this.aiLoading = false; + this.messageService.add({severity:'success', summary:'AI Processing Complete', detail:'Data Extracted'}); + }, + error: (err) => { + console.error(err); + this.aiLoading = false; + this.messageService.add({severity:'error', summary:'AI Error', detail:'Could not process with AI'}); + } + }); + } + + saveDocument() { + if (!this.aiResult || !this.filePath) return; + + this.saveLoading = true; + const payload = { + vendor_name: this.aiResult.vendor?.name || 'Unknown Vendor', + file_path: this.filePath, + model_type: this.modelType, + data: this.aiResult + }; + + this.ocrService.saveDocument(payload).subscribe({ + next: (res) => { + this.saveLoading = false; + this.messageService.add({severity:'success', summary:'Saved & Verified', detail:'Document and rules saved'}); + }, + error: (err) => { + console.error(err); + this.saveLoading = false; + this.messageService.add({severity:'error', summary:'Save Error', detail:'Failed to save document'}); + } + }); + } + + postToWebhook() { + if (!this.aiResult) return; + this.postLoading = true; + + this.ocrService.postToWebhook(this.aiResult).subscribe({ + next: () => { + this.postLoading = false; + this.messageService.add({severity:'success', summary:'Posted successfully', detail:'Data sent to webhook'}); + }, + error: (err) => { + console.error(err); + this.postLoading = false; + this.messageService.add({severity:'error', summary:'Post failed', detail:'Failed to send data to webhook'}); + } + }); + } + + getFormattedAiResult(): string { + return this.aiResult ? JSON.stringify(this.aiResult, null, 4) : ''; + } + + updateAiResult(newVal: string) { + try { + this.aiResult = JSON.parse(newVal); + } catch (e) { + // If the user types invalid JSON while editing, we don't crash, + // but we won't update the underlying object until it's valid again. + } } } diff --git a/frontend/src/app/pages/account/company/department/department.component.css b/frontend/src/app/pages/account/company/department/department.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/account/company/department/department.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/account/company/department/department.component.html b/frontend/src/app/pages/account/company/department/department.component.html new file mode 100644 index 0000000..59f3d7d --- /dev/null +++ b/frontend/src/app/pages/account/company/department/department.component.html @@ -0,0 +1,162 @@ +
+ + + + + + + + + + + + + + +
+

Manage Departments

+ + + + +
+ + + + # + +
+ Department + +
+ + +
+ Parent Department + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ department.department }} + + + {{ department.parentDepartmentName }} + + + + Not Available + + + {{ department.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ department.updatedUser }} + + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + +
+
+ + + + +
+
+
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/account/company/department/department.component.spec.ts b/frontend/src/app/pages/account/company/department/department.component.spec.ts new file mode 100644 index 0000000..57045f3 --- /dev/null +++ b/frontend/src/app/pages/account/company/department/department.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { DepartmentComponent } from './department.component'; + +describe('DepartmentComponent', () => { + let component: DepartmentComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [DepartmentComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(DepartmentComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/account/company/department/department.component.ts b/frontend/src/app/pages/account/company/department/department.component.ts new file mode 100644 index 0000000..814d844 --- /dev/null +++ b/frontend/src/app/pages/account/company/department/department.component.ts @@ -0,0 +1,286 @@ +import { DepartmentDTO } from './../../../../models/account.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Ripple } from 'primeng/ripple'; +import { ButtonModule, Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { FileUpload } from 'primeng/fileupload'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { RadioButton } from 'primeng/radiobutton'; +import { Rating } from 'primeng/rating'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { InputNumber } from 'primeng/inputnumber'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-department', + templateUrl: './department.component.html', + standalone: true, + imports: [TableModule, Dialog, Ripple, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, FileUpload, DropdownModule, Tag, RadioButton, Rating, InputTextModule, InputNumber, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton], + providers: [MessageService, ConfirmationService], + styleUrl: './department.component.css' +}) +export class DepartmentComponent implements OnInit{ + departmentForm: FormGroup; + departmentDialog: boolean = false; + departments!: DepartmentDTO[]; + + department: DepartmentDTO | undefined; + + selectedDepartments!: DepartmentDTO[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + statuses!: any[]; + + departmentOptions: any[] = []; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private companyService: CompanyService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.departmentForm = this.fb.group({ + id: [{ value: '', disabled: true }], + department: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + parentDepartment: [{ value: '', disabled: false }], + parentDepartmentName: [{ value: '', disabled: true }], + active: [{ value: true, disabled: false }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + ngOnInit() { + this.loadAllDepartments(); + } + + loadAllDepartments() { + this.isLoading = true; + this.companyService.getAllDepartments().subscribe({ + next: (data) => { + this.isLoading = false; + this.departments = data; + this.departmentOptions = [ + { label: 'None', value: '' }, + ...this.departments.map(d => ({ label: d.department, value: d.id })) + ]; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.statuses = [ + { label: 'Active', value: true }, + { label: 'Inactive', value: false } + ]; + + this.cols = [ + { field: 'department', header: 'Department', customExportHeader: 'Department' }, + { field: 'parentDepartmentName', header: 'Parent Department' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + } + + openNew() { + this.department = undefined; + this.departmentForm.reset(); + this.submitted = false; + this.departmentDialog = true; + } + + editDepartment(department: DepartmentDTO) { + const parentDepartment = this.departmentOptions.find(dep => dep.label === department.parentDepartmentName); + this.departmentForm.reset(); + this.department = { ...department }; + this.department.parentDepartment = parentDepartment?.value ?? ''; + this.departmentForm.patchValue(this.department); + this.departmentDialog = true; + } + + + hideDialog() { + this.departmentDialog = false; + this.submitted = false; + } + + toggleActive(department: DepartmentDTO) { + const isActivating = !department.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + department.department + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + this.companyService.activateDeactivateDepartment(department.id, isActivating).subscribe({ + next: (updatedDepartment) => { + department.active = updatedDepartment.active; + this.departments = [...this.departments]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Department ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling department active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update department status', + life: 3000 + }); + } + }); + } + }); + } + + getSeverity(status: boolean) { + switch (status) { + case true: + return 'success'; + case false: + return 'warn'; + } + } + + getErrorMessage(fieldName: string): string { + const control = this.departmentForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.departmentForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + saveDepartment() { + this.submitted = true; + + if (this.departmentForm.invalid) { + return; + } + + const departmentData = this.departmentForm.getRawValue() as DepartmentDTO; + + // ✅ Normalize null → empty array + const departments = this.departments ?? []; + + // 🔍 Duplicate check (case-insensitive) + const existing = departments.some(dep => + dep.department?.trim().toLowerCase() === departmentData.department?.trim().toLowerCase() && + dep.id !== departmentData.id + ); + + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Department name already exists', + life: 3000 + }); + return; + } + + this.companyService.saveDepartment(departmentData).subscribe({ + next: (savedDepartment) => { + + const index = departmentData.id + ? departments.findIndex(dep => dep.id === departmentData.id) + : -1; + + if (index !== -1) { + // ✅ UPDATE + departments[index] = savedDepartment; + } else { + // ✅ CREATE + departments.push(savedDepartment); + } + + // ✅ Reassign once (change detection + null safety) + this.departments = [...departments]; + + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: index !== -1 + ? 'Department Updated' + : 'Department Created', + life: 3000 + }); + + this.departmentDialog = false; + this.department = undefined; + this.departmentForm.reset(); + this.submitted = false; + }, + error: (err) => { + console.error('Error saving department', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save department', + life: 3000 + }); + } + }); + } +} diff --git a/frontend/src/app/pages/account/company/desigation/designation.component.css b/frontend/src/app/pages/account/company/desigation/designation.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/account/company/desigation/designation.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/account/company/desigation/designation.component.html b/frontend/src/app/pages/account/company/desigation/designation.component.html new file mode 100644 index 0000000..5f58129 --- /dev/null +++ b/frontend/src/app/pages/account/company/desigation/designation.component.html @@ -0,0 +1,183 @@ +
+ + + + + + + + + + + + + + +
+

Manage Designations

+ + + + +
+ + + + # + +
+ Designation + +
+ + +
+ Department + +
+ + +
+ HOD + +
+ + +
+ Pay Grade + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ designation.designation }} + {{ designation.departmentName }} + {{ designation.hod ? 'Yes' : 'No' }} + {{ designation.payGrade || 'N/A' }} + {{ designation.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ designation.updatedUser }} + + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + +
+
+ + + + +
+
+ +
+
+
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/account/company/desigation/designation.component.spec.ts b/frontend/src/app/pages/account/company/desigation/designation.component.spec.ts new file mode 100644 index 0000000..de59e48 --- /dev/null +++ b/frontend/src/app/pages/account/company/desigation/designation.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { DesignationComponent } from './designation.component'; + +describe('DesignationComponent', () => { + let component: DesignationComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [DesignationComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(DesignationComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/account/company/desigation/designation.component.ts b/frontend/src/app/pages/account/company/desigation/designation.component.ts new file mode 100644 index 0000000..cd8ebf0 --- /dev/null +++ b/frontend/src/app/pages/account/company/desigation/designation.component.ts @@ -0,0 +1,300 @@ +import { DesignationDTO, DepartmentDTO } from './../../../../models/account.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Ripple } from 'primeng/ripple'; +import { ButtonModule, Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { FileUpload } from 'primeng/fileupload'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { RadioButton } from 'primeng/radiobutton'; +import { Rating } from 'primeng/rating'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { InputNumber } from 'primeng/inputnumber'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { CheckboxModule } from 'primeng/checkbox'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-designation', + templateUrl: './designation.component.html', + standalone: true, + imports: [TableModule, Dialog, Ripple, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, FileUpload, DropdownModule, Tag, RadioButton, Rating, InputTextModule, InputNumber, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, CheckboxModule], + providers: [MessageService, ConfirmationService], + styleUrl: './designation.component.css' +}) +export class DesignationComponent implements OnInit{ + designationForm: FormGroup; + designationDialog: boolean = false; + designations!: DesignationDTO[]; + + designation: DesignationDTO | undefined; + + selectedDesignations!: DesignationDTO[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + departments: DepartmentDTO[] = []; + + statuses!: any[]; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private companyService: CompanyService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.designationForm = this.fb.group({ + id: [{ value: '', disabled: true }], + designation: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + departmentId: [{ value: null, disabled: false }, [Validators.required]], + hod: [{ value: false, disabled: false }], + active: [{ value: true, disabled: false }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + ngOnInit() { + this.loadAllDesignations(); + this.loadDepartments(); + } + + loadDepartments() { + this.companyService.getAllDepartments().subscribe({ + next: (data) => { + this.departments = data.filter(d => d.active); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error loading departments', err); + } + }); + } + + loadAllDesignations() { + this.isLoading = true; + this.companyService.getAllDesignations().subscribe({ + next: (data) => { + this.isLoading = false; + this.designations = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.statuses = [ + { label: 'Active', value: true }, + { label: 'Inactive', value: false } + ]; + + this.cols = [ + { field: 'designation', header: 'Designation', customExportHeader: 'Designation' }, + { field: 'departmentName', header: 'Department' }, + { field: 'hod', header: 'HOD' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + } + + openNew() { + this.designation = undefined; + this.designationForm.reset(); + this.submitted = false; + this.designationDialog = true; + } + + editDesignation(designation: DesignationDTO) { + const parentDepartment = this.departments.find(dep => dep.department === designation.departmentName); + console.log(parentDepartment); + this.designationForm.reset(); + this.designation = { ...designation }; + this.designation.departmentId = parentDepartment?.id ?? ''; + console.log(this.designation); + this.designationForm.patchValue(this.designation); + this.designationDialog = true; + } + + + hideDialog() { + this.designationDialog = false; + this.submitted = false; + } + + toggleActive(designation: DesignationDTO) { + const isActivating = !designation.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + designation.designation + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + this.companyService.activateDeactivateDesignation(designation.id, isActivating).subscribe({ + next: (updatedDesignation) => { + designation.active = updatedDesignation.active; + this.designations = [...this.designations]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Designation ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling designation active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update designation status', + life: 3000 + }); + } + }); + } + }); + } + + getSeverity(status: boolean) { + switch (status) { + case true: + return 'success'; + case false: + return 'warn'; + } + } + + getErrorMessage(fieldName: string): string { + const control = this.designationForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.designationForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + saveDesignation() { + this.submitted = true; + + if (this.designationForm.invalid) { + return; + } + + const designationData = this.designationForm.getRawValue() as DesignationDTO; + + // ✅ Normalize null → empty array + const designations = this.designations ?? []; + + // 🔍 Duplicate check (case-insensitive, department-specific) + const existing = designations.some(des => + des.designation?.trim().toLowerCase() === designationData.designation?.trim().toLowerCase() && + des.departmentId === designationData.departmentId && + des.id !== designationData.id + ); + + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Designation already exists in this department', + life: 3000 + }); + return; + } + + this.companyService.saveDesignation(designationData).subscribe({ + next: (savedDesignation) => { + + const index = designationData.id + ? designations.findIndex(des => des.id === designationData.id) + : -1; + + if (index !== -1) { + // ✅ UPDATE + designations[index] = savedDesignation; + } else { + // ✅ CREATE + designations.push(savedDesignation); + } + + // ✅ Reassign once (change detection + null safety) + this.designations = [...designations]; + + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: index !== -1 + ? 'Designation Updated' + : 'Designation Created', + life: 3000 + }); + + this.designationDialog = false; + this.designation = undefined; + this.designationForm.reset(); + this.submitted = false; + }, + error: (err) => { + console.error('Error saving designation', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save designation', + life: 3000 + }); + } + }); + } +} diff --git a/frontend/src/app/pages/account/company/employee/employee.component.css b/frontend/src/app/pages/account/company/employee/employee.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/account/company/employee/employee.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/account/company/employee/employee.component.html b/frontend/src/app/pages/account/company/employee/employee.component.html new file mode 100644 index 0000000..bd9a213 --- /dev/null +++ b/frontend/src/app/pages/account/company/employee/employee.component.html @@ -0,0 +1,407 @@ +
+ + + + + + + + + + + + + + +
+

Manage Employees

+ + + + +
+ + + + # + +
+ Full Name + +
+ + +
+ Subsidiary + +
+ + +
+ Department + +
+ + +
+ Designation + +
+ + +
+ Contact No + +
+ + +
+ Email + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ employee.fullName }} + {{ employee.subsidiaryName }} + {{ employee.department }} + {{ employee.designation }} + {{ employee.contactNo }} + {{ employee.emailId }} + {{ employee.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ employee.updatedUser }} + + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
Permanent Address
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
Residence Address
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/account/company/employee/employee.component.spec.ts b/frontend/src/app/pages/account/company/employee/employee.component.spec.ts new file mode 100644 index 0000000..6f30928 --- /dev/null +++ b/frontend/src/app/pages/account/company/employee/employee.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { EmployeeComponent } from './employee.component'; + +describe('EmployeeComponent', () => { + let component: EmployeeComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [EmployeeComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(EmployeeComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/account/company/employee/employee.component.ts b/frontend/src/app/pages/account/company/employee/employee.component.ts new file mode 100644 index 0000000..29e2eae --- /dev/null +++ b/frontend/src/app/pages/account/company/employee/employee.component.ts @@ -0,0 +1,445 @@ +import { EmployeeDTO, SubsidiaryDTO, DepartmentDTO, DesignationDTO } from './../../../../models/account.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Ripple } from 'primeng/ripple'; +import { ButtonModule, Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { FileUpload } from 'primeng/fileupload'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { RadioButton } from 'primeng/radiobutton'; +import { Rating } from 'primeng/rating'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { InputNumber } from 'primeng/inputnumber'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { CalendarModule } from 'primeng/calendar'; +import { MasterService } from '../../../../services/masters/master.service'; +import { CityDTO, SearchDTO } from '../../../../models/masters/masters'; +import { debounceTime, Subject } from 'rxjs'; +import { Request } from '../../../../models/request.model'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-employee', + templateUrl: './employee.component.html', + standalone: true, + imports: [TableModule, Dialog, Ripple, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, FileUpload, DropdownModule, Tag, RadioButton, Rating, InputTextModule, InputNumber, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, AutoCompleteModule, CalendarModule], + providers: [MessageService, ConfirmationService], + styleUrl: './employee.component.css' +}) +export class EmployeeComponent implements OnInit{ + employeeForm: FormGroup; + employeeDialog: boolean = false; + employees!: EmployeeDTO[]; + + employee: EmployeeDTO | undefined; + + selectedEmployees!: EmployeeDTO[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + subsidiaries: SubsidiaryDTO[] = []; + + departments: DepartmentDTO[] = []; + + designations: DesignationDTO[] = []; + + suggestions: CityDTO[] = []; + + private searchSubject = new Subject(); + + genders: any[] = [ + { label: 'Male', value: 'Male' }, + { label: 'Female', value: 'Female' }, + { label: 'Other', value: 'Other' } + ]; + + statuses!: any[]; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private companyService: CompanyService, + private masterService: MasterService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.employeeForm = this.fb.group({ + id: [{ value: '', disabled: true }], + subsidiaryId: [{ value: null, disabled: false }, [Validators.required]], + departmentId: [{ value: null, disabled: false }, [Validators.required]], + designationId: [{ value: null, disabled: false }, [Validators.required]], + joiningDate: [{ value: '', disabled: false }], + employeeId: [{ value: '', disabled: false }, [Validators.required]], + fullName: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + gender: [{ value: '', disabled: false }, [Validators.required]], + dob: [{ value: '', disabled: false }, [Validators.required]], + contactNo: [{ value: '', disabled: false }, [Validators.required, ValidationService.mobileValidator()]], + alternateNo: [{ value: '', disabled: false }, [ValidationService.mobileValidator()]], + emailId: [{ value: '', disabled: false }, [Validators.required, ValidationService.emailValidator()]], + residenceAddress: [{ value: '', disabled: false }], + residenceCityId: [{ value: '', disabled: true }], + residenceStateId: [{ value: '', disabled: true }], + residenceCityName: [{ value: '', disabled: false }], + residenceStateName: [{ value: '', disabled: true }], + permanentAddress: [{ value: '', disabled: false }], + permanentCityId: [{ value: '', disabled: true }], + permanentStateId: [{ value: '', disabled: true }], + permanentCityName: [{ value: '', disabled: false }], + permanentStateName: [{ value: '', disabled: true }], + active: [{ value: true, disabled: false }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + ngOnInit() { + this.loadAllEmployees(); + this.loadSubsidiaries(); + this.loadDepartments(); + this.loadDesignations(); + } + + loadSubsidiaries() { + this.companyService.getAllSubsidiaries().subscribe({ + next: (data) => { + this.subsidiaries = data.filter(s => s.active); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error loading subsidiaries', err); + } + }); + } + + loadDepartments() { + this.companyService.getAllDepartments().subscribe({ + next: (data) => { + this.departments = data.filter(d => d.active); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error loading departments', err); + } + }); + } + + loadDesignations() { + this.companyService.getAllDesignations().subscribe({ + next: (data) => { + this.designations = data.filter(d => d.active); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error loading designations', err); + } + }); + } + + searchCities(event: any) { + const query = event.query; + this.searchSubject.next(query); + } + + onSelectCity(event: any, type: 'permanent' | 'residence') { + const city = event.value as CityDTO; + if (type === 'permanent') { + this.employeeForm.patchValue({ + permanentCityId: city.id, + permanentStateId: city.stateId, + permanentCityName: city.cityName, + permanentStateName: city.stateName + }); + } else { + this.employeeForm.patchValue({ + residenceCityId: city.id, + residenceStateId: city.stateId, + residenceCityName: city.cityName, + residenceStateName: city.stateName + }); + } + } + + loadAllEmployees() { + this.isLoading = true; + this.companyService.getAllEmployees().subscribe({ + next: (data) => { + this.isLoading = false; + this.employees = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.statuses = [ + { label: 'Active', value: true }, + { label: 'Inactive', value: false } + ]; + + this.cols = [ + { field: 'fullName', header: 'Full Name', customExportHeader: 'Full Name' }, + { field: 'subsidiaryName', header: 'Subsidiary' }, + { field: 'department', header: 'Department' }, + { field: 'designation', header: 'Designation' }, + { field: 'contactNo', header: 'Contact No' }, + { field: 'emailId', header: 'Email' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + + this.searchSubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const requestPayload: Request = { + data: { + searchBy: 'CITY', + searchValue: query + }, + compressed: true, + target: 'models.commons.Search' + }; + this.masterService.searchCityStates(requestPayload).subscribe({ + next: (cities) => { + this.suggestions = cities.map(city => ({ + ...city, + display: `${city.cityName}, ${city.stateName}` + })); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error searching cities', err); + this.suggestions = []; + } + }); + } else { + this.suggestions = []; + } + }); + } + + openNew() { + this.employee = undefined; + this.employeeForm.reset(); + this.submitted = false; + this.employeeDialog = true; + } + + editEmployee(employee: EmployeeDTO) { + const subsidiary = this.subsidiaries.find(sub => sub.name === employee.subsidiaryName); + const department = this.departments.find(dep => dep.department === employee.department); + const designation = this.designations.find(des => des.designation === employee.designation); + + this.employeeForm.reset(); + this.employee = { ...employee }; + this.employee.subsidiaryId = subsidiary?.id ?? ''; + this.employee.departmentId = department?.id ?? ''; + this.employee.designationId = designation?.id ?? ''; + + // Prepare data for patching, converting dates + const employeeToPatch = { ...this.employee }; + if (employeeToPatch.joiningDate) { + (employeeToPatch as any).joiningDate = new Date(employeeToPatch.joiningDate); + } + if (employeeToPatch.dob) { + (employeeToPatch as any).dob = new Date(employeeToPatch.dob); + } + + this.employeeForm.patchValue(employeeToPatch); + this.employeeDialog = true; + console.log(this.employeeForm.getRawValue()); + } + + + hideDialog() { + this.employeeDialog = false; + this.submitted = false; + } + + toggleActive(employee: EmployeeDTO) { + const isActivating = !employee.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + employee.fullName + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + this.companyService.activateDeactivateEmployee(employee.id, isActivating).subscribe({ + next: (updatedEmployee) => { + employee.active = updatedEmployee.active; + this.employees = [...this.employees]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Employee ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling employee active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update employee status', + life: 3000 + }); + } + }); + } + }); + } + + getSeverity(status: boolean) { + switch (status) { + case true: + return 'success'; + case false: + return 'warn'; + } + } + + getErrorMessage(fieldName: string): string { + const control = this.employeeForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.employeeForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + saveEmployee() { + this.submitted = true; + + if (this.employeeForm.invalid) { + return; + } + + const employeeData = this.employeeForm.getRawValue() as EmployeeDTO; + + // ✅ Normalize null → empty array + const employees = this.employees ?? []; + + // 🔍 Duplicate check + const existing = employees.some(emp => { + if (emp.id === employeeData.id) return false; + const sameName = emp.fullName?.trim().toLowerCase() === employeeData.fullName?.trim().toLowerCase(); + + const empDob = emp.dob ? new Date(emp.dob) : null; + if (empDob) empDob.setHours(0, 0, 0, 0); + + const dataDob = employeeData.dob ? new Date(employeeData.dob) : null; + if (dataDob) dataDob.setHours(0, 0, 0, 0); + + const sameDob = empDob && dataDob ? empDob.getTime() === dataDob.getTime() : empDob === dataDob; + + const sameGender = emp.gender === employeeData.gender; + const sameSubsidiary = emp.subsidiaryId === employeeData.subsidiaryId; + + return sameName && sameDob && sameGender && sameSubsidiary; + }); + + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Employee with same name, dob, gender and subsidiary already exists', + life: 3000 + }); + return; + } + + this.companyService.saveEmployee(employeeData).subscribe({ + next: (savedEmployee) => { + + const index = employeeData.id + ? employees.findIndex(emp => emp.id === employeeData.id) + : -1; + + if (index !== -1) { + // ✅ UPDATE + employees[index] = savedEmployee; + } else { + // ✅ CREATE + employees.push(savedEmployee); + } + + // ✅ Reassign once (change detection + null safety) + this.employees = [...employees]; + + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: index !== -1 + ? 'Employee Updated' + : 'Employee Created', + life: 3000 + }); + + this.employeeDialog = false; + this.employee = undefined; + this.employeeForm.reset(); + this.submitted = false; + }, + error: (err) => { + console.error('Error saving employee', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save employee', + life: 3000 + }); + } + }); + } +} diff --git a/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.css b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.css new file mode 100644 index 0000000..463526a --- /dev/null +++ b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.css @@ -0,0 +1,5 @@ +:host ::ng-deep .p-dialog .product-image { + width: 150px; + margin: 0 auto 2rem auto; + display: block; +} \ No newline at end of file diff --git a/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.html b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.html new file mode 100644 index 0000000..4e3e8de --- /dev/null +++ b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.html @@ -0,0 +1,322 @@ +
+ + + + + + + + + + + + + + +
+

Manage Subsidiaries

+ + + + +
+ + + + # + Code + +
+ Name + +
+ + +
+ Email + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ subsidiary.code }} + {{ subsidiary.name }} + + + {{ subsidiary.emailId }} + + + + Not Available + + + {{ subsidiary.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ subsidiary.updatedUser }} + + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
+ + + + + + + + + +
+ diff --git a/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.spec.ts b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.spec.ts new file mode 100644 index 0000000..ac76031 --- /dev/null +++ b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { SubsidiaryComponent } from './subsidiary.component'; + +describe('SubsidiaryComponent', () => { + let component: SubsidiaryComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [SubsidiaryComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(SubsidiaryComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.ts b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.ts new file mode 100644 index 0000000..7af403d --- /dev/null +++ b/frontend/src/app/pages/account/company/subsidiary/subsidiary.component.ts @@ -0,0 +1,328 @@ +import { SearchDTO } from './../../../../models/masters/masters'; +import { SubsidiaryDTO } from './../../../../models/account.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Ripple } from 'primeng/ripple'; +import { ButtonModule, Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { FileUpload } from 'primeng/fileupload'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { RadioButton } from 'primeng/radiobutton'; +import { Rating } from 'primeng/rating'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { InputNumber } from 'primeng/inputnumber'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { MasterService } from '../../../../services/masters/master.service'; +import { CityDTO } from '../../../../models/masters/masters'; +import { debounceTime, Subject } from 'rxjs'; +import { Request } from '../../../../models/request.model'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-subsidiary', + templateUrl: './subsidiary.component.html', + standalone: true, + imports: [TableModule, Dialog, Ripple, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, FileUpload, DropdownModule, Tag, RadioButton, Rating, InputTextModule, InputNumber, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, AutoCompleteModule], + providers: [MessageService, ConfirmationService], + styleUrl: './subsidiary.component.css' +}) +export class SubsidiaryComponent implements OnInit{ + subsidiaryForm: FormGroup; + subsidiaryDialog: boolean = false; + subsidiaries!: SubsidiaryDTO[]; + + subsidiary!: SubsidiaryDTO; + + selectedSubsidiaries!: SubsidiaryDTO[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + suggestions: CityDTO[] = []; + + private searchSubject = new Subject(); + + statuses!: any[]; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private subsidiaryService: CompanyService, + private masterService: MasterService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.subsidiaryForm = this.fb.group({ + id: [{ value: '', disabled: true }], + cityId: [{ value: null, disabled: true }], + code: [{ value: '', disabled: false }, [Validators.required, ValidationService.codeValidator()]], + name: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + officeNo: [{ value: '', disabled: false }], + street: [{ value: '', disabled: false }], + locality: [{ value: '', disabled: false }], + cityName: [{ value: '', disabled: false }], + stateName: [{ value: '', disabled: true }], + pinCode: [{ value: '', disabled: false }, [ValidationService.pincodeValidator()]], + emailId: [{ value: '', disabled: false }, [ValidationService.emailValidator()]], + contactNo: [{ value: '', disabled: false }, [ValidationService.mobileValidator()]], + contactPerson: [{ value: '', disabled: false }, [ValidationService.contactPersonValidator()]], + panNo: [{ value: '', disabled: false }, [ValidationService.panValidator()]], + cinNo: [{ value: '', disabled: false }, [ValidationService.cinValidator()]], + msmeNo: [{ value: '', disabled: false }, [ValidationService.msmeValidator()]], + stateId: [{ value: '', disabled: true }], + stateCode: [{ value: '', disabled: true }], + gstCode: [{ value: '', disabled: true }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + searchCities(event: any) { + const query = event.query; + this.searchSubject.next(query); + } + + onSelectCity(event: any) { + const city = event.value as CityDTO; + this.subsidiaryForm.patchValue({ + cityId: city.id, + stateId: city.stateId, + cityName: city.cityName, + stateName: city.stateName, + stateCode: city.stateCode, + gstCode: city.gstCode + }); + } + + ngOnInit() { + this.loadAllSubsidiaries(); + } + + loadAllSubsidiaries() { + this.isLoading = true; + this.subsidiaryService.getAllSubsidiaries().subscribe({ + next: (data) => { + this.isLoading = false; + this.subsidiaries = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.statuses = [ + { label: 'Active', value: true }, + { label: 'Inactive', value: false } + ]; + + this.cols = [ + { field: 'code', header: 'Code', customExportHeader: 'Code' }, + { field: 'name', header: 'Name' }, + { field: 'officeNo', header: 'Office No / Building / Floor' }, + { field: 'street', header: 'Street / Road' }, + { field: 'locality', header: 'Locality' }, + { field: 'cityName', header: 'City' }, + { field: 'emailId', header: 'Email' }, + { field: 'contactNo', header: 'Contact No' }, + { field: 'contactPerson', header: 'Contact Person' }, + { field: 'panNo', header: 'PAN No.' }, + { field: 'cinNo', header: 'CIN No.' }, + { field: 'msmeNo', header: 'MSME No.' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + + this.searchSubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const requestPayload: Request = { + data: { + searchBy: 'CITY', + searchValue: query + }, + compressed: true, + target: 'models.commons.Search' + }; + this.masterService.searchCityStates(requestPayload).subscribe({ + next: (cities) => { + this.suggestions = cities.map(city => ({ + ...city, + display: `${city.cityName}, ${city.stateName}` + })); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error searching cities', err); + this.suggestions = []; + } + }); + } else { + this.suggestions = []; + } + }); + } + + openNew() { + this.subsidiary = {}; + this.subsidiaryForm.reset(); + this.submitted = false; + this.subsidiaryDialog = true; + } + + editSubsidiary(subsidiary: SubsidiaryDTO) { + this.subsidiaryForm.reset(); + this.subsidiary = { ...subsidiary }; + this.subsidiaryForm.patchValue(subsidiary); + this.subsidiaryDialog = true; + console.log(this.subsidiaryForm.getRawValue()); + } + + + hideDialog() { + this.subsidiaryDialog = false; + this.submitted = false; + } + + toggleActive(subsidiary: SubsidiaryDTO) { + const isActivating = !subsidiary.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + subsidiary.name + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + this.subsidiaryService.activateDeactivateSubsidiary(subsidiary.id?? '', isActivating).subscribe({ + next: (updatedSubsidiary) => { + subsidiary.active = updatedSubsidiary.active; + this.subsidiaries = [...this.subsidiaries]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Subsidiary ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling subsidiary active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update subsidiary status', + life: 3000 + }); + } + }); + } + }); + } + + getSeverity(status: boolean) { + switch (status) { + case true: + return 'success'; + case false: + return 'warn'; + } + } + + getErrorMessage(fieldName: string): string { + const control = this.subsidiaryForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.subsidiaryForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + saveSubsidiary() { + this.submitted = true; + + if (this.subsidiaryForm.valid) { + const subsidiaryData = this.subsidiaryForm.getRawValue(); + const existing = this.subsidiaries.find(sub => sub.id !== subsidiaryData.id && (sub.code === subsidiaryData.code || sub.name === subsidiaryData.name)); + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Subsidiary code or name already exists', + life: 3000 + }); + return; + } + this.subsidiaryService.saveSubsidiary(subsidiaryData).subscribe({ + next: (newSubsidiary) => { + this.subsidiaries.push(newSubsidiary); + this.subsidiaries = [...this.subsidiaries]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: 'Subsidiary Created', + life: 3000 + }); + this.subsidiaryDialog = false; + this.subsidiary = {}; + }, + error: (err) => { + console.error('Error saving subsidiary', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save subsidiary', + life: 3000 + }); + } + }); + } + } +} diff --git a/frontend/src/app/pages/account/company/verifier/verifier.component.css b/frontend/src/app/pages/account/company/verifier/verifier.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/account/company/verifier/verifier.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/account/company/verifier/verifier.component.html b/frontend/src/app/pages/account/company/verifier/verifier.component.html new file mode 100644 index 0000000..e1e2b04 --- /dev/null +++ b/frontend/src/app/pages/account/company/verifier/verifier.component.html @@ -0,0 +1,273 @@ +
+ + + + + + + + + + + + + + +
+

Manage Verifiers

+ + + + +
+ + + + # + +
+ Full Name + +
+ + +
+ Verifier Code + +
+ + +
+ Type + +
+ + +
+ Mobile No + +
+ + +
+ Email + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ verifier.fullName }} + {{ verifier.verifierCode }} + {{ verifier.verifierType }} + {{ verifier.mobileNo }} + {{ verifier.emailId }} + {{ verifier.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ verifier.updatedUser }} + + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+ + + + +
+ + +
+ + + + +
+ + +
+ + + + + +
+ + +
+ + + + +
+ + +
+ + + + +
+ + +
+ + + + +
+ + +
+
+ + +
+
+ + +
+
+
+
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/account/company/verifier/verifier.component.ts b/frontend/src/app/pages/account/company/verifier/verifier.component.ts new file mode 100644 index 0000000..0dcd63f --- /dev/null +++ b/frontend/src/app/pages/account/company/verifier/verifier.component.ts @@ -0,0 +1,347 @@ +import { CompanyVerifierModel, EmployeeDTO } from './../../../../models/account.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { CalendarModule } from 'primeng/calendar'; +import { CheckboxModule } from 'primeng/checkbox'; +import { Subject, debounceTime } from 'rxjs'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-verifier', + templateUrl: './verifier.component.html', + standalone: true, + imports: [TableModule, Dialog, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, DropdownModule, Tag, InputTextModule, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, AutoCompleteModule, CalendarModule, CheckboxModule], + providers: [MessageService, ConfirmationService], + styleUrl: './verifier.component.css' +}) +export class VerifierComponent implements OnInit{ + verifierForm: FormGroup; + verifierDialog: boolean = false; + verifiers!: CompanyVerifierModel[]; + + verifier: CompanyVerifierModel | undefined; + + selectedVerifiers!: CompanyVerifierModel[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + suggestions: any[] = []; + + private searchSubject = new Subject(); + + verifierTypes: any[] = [ + { label: 'INTERNAL', value: 'INTERNAL' }, + { label: 'EXTERNAL', value: 'EXTERNAL' } + ]; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private companyService: CompanyService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.verifierForm = this.fb.group({ + id: [{ value: '', disabled: true }], + employeeId: [{ value: null, disabled: false }], + verifierCode: [{ value: '', disabled: false }, [Validators.required]], + verifierType: [{ value: 'INTERNAL', disabled: false }, [Validators.required]], + fullName: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + mobileNo: [{ value: '', disabled: false }, [Validators.required, ValidationService.mobileValidator()]], + emailId: [{ value: '', disabled: false }, [Validators.required, ValidationService.emailValidator()]], + allocationDate: [{ value: '', disabled: false }], + appActive: [{ value: true, disabled: false }], + photoMandatory: [{ value: false, disabled: false }], + active: [{ value: true, disabled: false }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + ngOnInit() { + this.loadAllVerifiers(); + this.searchSubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const requestPayload = { + data: { + searchBy: 'NAME', + searchValue: query + }, + compressed: true, + target: 'models.commons.Search' + }; + this.companyService.searchEmployees(requestPayload).subscribe({ + next: (employees) => { + this.suggestions = employees.map(emp => ({ + ...emp, + display: `${emp.employeeId} - ${emp.fullName}` + })); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error searching employees', err); + this.suggestions = []; + } + }); + } else { + this.suggestions = []; + } + }); + } + + searchEmployees(event: any) { + this.searchSubject.next(event.query); + } + + onSelectEmployee(event: any) { + const emp = event.value; + const patchData: any = { + employeeId: emp.employeeId, + fullName: emp.fullName, + mobileNo: emp.contactNo, + emailId: emp.emailId + }; + + // Auto set allocation date only in add new mode + if (!this.verifier && emp.joiningDate) { + patchData.allocationDate = new Date(emp.joiningDate); + } + + this.verifierForm.patchValue(patchData); + } + + onVerifierTypeChange() { + this.suggestions = []; + this.verifierForm.patchValue({ + employeeId: null, + fullName: '', + mobileNo: '', + emailId: '' + }); + this.cd.markForCheck(); + } + + loadAllVerifiers() { + this.isLoading = true; + this.companyService.getAllVerifiers().subscribe({ + next: (data) => { + this.isLoading = false; + this.verifiers = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.cols = [ + { field: 'fullName', header: 'Full Name' }, + { field: 'verifierCode', header: 'Verifier Code' }, + { field: 'verifierType', header: 'Type' }, + { field: 'mobileNo', header: 'Mobile No' }, + { field: 'emailId', header: 'Email' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + } + + openNew() { + this.verifier = undefined; + this.verifierForm.reset(); + this.verifierForm.patchValue({ + verifierType: 'INTERNAL', + appActive: true, + photoMandatory: false, + active: true + }); + this.submitted = false; + this.verifierDialog = true; + } + + editVerifier(verifier: CompanyVerifierModel) { + this.verifierForm.reset(); + this.verifier = { ...verifier }; + + const verifierToPatch = { ...this.verifier }; + if (verifierToPatch.allocationDate) { + (verifierToPatch as any).allocationDate = new Date(verifierToPatch.allocationDate); + } + + this.verifierForm.patchValue(verifierToPatch); + this.verifierDialog = true; + } + + hideDialog() { + this.verifierDialog = false; + this.submitted = false; + } + + toggleActive(verifier: CompanyVerifierModel) { + const isActivating = !verifier.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + verifier.fullName + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + if(verifier.id) { + this.companyService.activateDeactivateVerifier(verifier.id, isActivating).subscribe({ + next: (updatedVerifier) => { + verifier.active = updatedVerifier.active; + this.verifiers = [...this.verifiers]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Verifier ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling verifier active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update verifier status', + life: 3000 + }); + } + }); + } + } + }); + } + + getSeverity(status: boolean) { + return status ? 'success' : 'warn'; + } + + getErrorMessage(fieldName: string): string { + const control = this.verifierForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.verifierForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + saveVerifier() { + this.submitted = true; + + if (this.verifierForm.invalid) { + return; + } + + const verifierData = this.verifierForm.getRawValue() as CompanyVerifierModel; + const verifiers = this.verifiers ?? []; + + // Duplicate check + const existing = verifiers.some(v => { + if (v.id === verifierData.id) return false; + return v.verifierCode?.trim().toLowerCase() === verifierData.verifierCode?.trim().toLowerCase(); + }); + + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Verifier with same code already exists', + life: 3000 + }); + return; + } + + this.companyService.saveVerifier(verifierData).subscribe({ + next: (savedVerifier) => { + const index = verifierData.id + ? verifiers.findIndex(v => v.id === verifierData.id) + : -1; + + if (index !== -1) { + verifiers[index] = savedVerifier; + } else { + verifiers.push(savedVerifier); + } + + this.verifiers = [...verifiers]; + + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: index !== -1 ? 'Verifier Updated' : 'Verifier Created', + life: 3000 + }); + + this.verifierDialog = false; + this.verifier = undefined; + this.verifierForm.reset(); + this.submitted = false; + }, + error: (err) => { + console.error('Error saving verifier', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save verifier', + life: 3000 + }); + } + }); + } +} diff --git a/frontend/src/app/pages/account/user/user.component.css b/frontend/src/app/pages/account/user/user.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/account/user/user.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/account/user/user.component.html b/frontend/src/app/pages/account/user/user.component.html new file mode 100644 index 0000000..a352caf --- /dev/null +++ b/frontend/src/app/pages/account/user/user.component.html @@ -0,0 +1,317 @@ +
+ + + + + + + + + + + + + + +
+

Manage Users

+ + + + +
+ + + + # + +
+ Login ID + +
+ + +
+ Display Name + +
+ + +
+ Employee ID + +
+ + +
+ Status + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Active + +
+ + + + + + + {{ rowIndex + 1 }} + {{ user.loginId }} + {{ user.displayName }} + + + {{ user.employeeId }} + + + Not Available + + + + + + {{ user.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ user.updatedUser }} + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + +
+ Login ID is already taken. + {{ getErrorMessage('loginId') }} +
+
+
+ + + + +
+ {{ getErrorMessage('displayName') }} +
+
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+ {{ getErrorMessage('status') }} +
+
+
+ +
+
+ + + + +
+
+ + + + +
+
+ +
+
+ + + + # + Role Name + Group Name + Branch Name + Action + + + + + {{ rowIndex + 1 }} + {{role.roleName}} + {{role.groupName}} + {{role.branchName}} + + + + + + + + + No roles assigned. + + + + +
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/account/user/user.component.spec.ts b/frontend/src/app/pages/account/user/user.component.spec.ts new file mode 100644 index 0000000..56317f0 --- /dev/null +++ b/frontend/src/app/pages/account/user/user.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { UserComponent } from './user.component'; + +describe('UserComponent', () => { + let component: UserComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [UserComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(UserComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/account/user/user.component.ts b/frontend/src/app/pages/account/user/user.component.ts new file mode 100644 index 0000000..eca90ed --- /dev/null +++ b/frontend/src/app/pages/account/user/user.component.ts @@ -0,0 +1,532 @@ +import { UserDTO, UserRoleDTO } from '../../../models/user.model'; +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Ripple } from 'primeng/ripple'; +import { ButtonModule, Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { FileUpload } from 'primeng/fileupload'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { RadioButton } from 'primeng/radiobutton'; +import { Rating } from 'primeng/rating'; +import { Skeleton } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { InputNumber } from 'primeng/inputnumber'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { Table } from 'primeng/table'; +import { DropdownModule } from 'primeng/dropdown'; +import { UserService } from '../../../services/account/user/user.service'; +import { CompanyService } from '../../../services/account/company/company.service'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { FieldsetModule } from 'primeng/fieldset'; +import { debounceTime, Subject } from 'rxjs'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-user', + templateUrl: './user.component.html', + standalone: true, + imports: [TableModule, Dialog, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, CommonModule, Tag, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, AutoCompleteModule, FieldsetModule], + providers: [MessageService, ConfirmationService], + styleUrl: './user.component.css' +}) +export class UserComponent implements OnInit{ + userForm: FormGroup; + userDialog: boolean = false; + users!: UserDTO[]; + + user: UserDTO | undefined; + + selectedUsers!: UserDTO[] | null; + + submitted: boolean = false; + checkingLoginId: boolean = false; + loginIdTaken: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + userRoles: UserRoleDTO[] = []; + + branches: any[] = []; + + userRolesOptions: any[] = []; + + selectedBranch: string | undefined; + selectedRole: string | undefined; + + suggestions: any[] = []; + + private searchSubject = new Subject(); + + + statuses!: any[]; + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + constructor( + private userService: UserService, + private companyService: CompanyService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.userForm = this.fb.group({ + id: [{ value: '', disabled: true }], + loginId: [{ value: '', disabled: false }, [Validators.required]], + displayName: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + fkEmployeeId: [{ value: '', disabled: false }], + employeeId: [{ value: '', disabled: false }], + employeeName: [{ value: '', disabled: false }], + fatherName: [{ value: '', disabled: false }], + department: [{ value: '', disabled: false }], + designation: [{ value: '', disabled: false }], + status: [{ value: 'Active', disabled: false }, [Validators.required]], + active: [{ value: true, disabled: false }] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + onSearch(event: Event) { + const input = event.target as HTMLInputElement; + this.dt.filterGlobal(input.value, 'contains'); + } + + ngOnInit() { + this.loadAllUsers(); + this.loadBranches(); + this.loadRoles(); + } + + loadRoles() { + this.userService.getAllRoles().subscribe({ + next: (data) => { + this.userRolesOptions = data; + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Failed to load roles', err); + } + }); + } + + loadBranches() { + this.userService.getAllBranches().subscribe({ + next: (data) => { + this.branches = data; + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Failed to load branches', err); + } + }); + } + + loadAllUsers() { + this.isLoading = true; + this.userService.getAllUsers().subscribe({ + next: (data) => { + this.isLoading = false; + this.users = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.statuses = [ + { label: 'Active', value: 'Active' }, + { label: 'Inactive', value: 'Inactive' } + ]; + + this.cols = [ + { field: 'loginId', header: 'Login ID', customExportHeader: 'Login ID' }, + { field: 'displayName', header: 'Display Name' }, + { field: 'employeeId', header: 'Employee ID' }, + { field: 'status', header: 'Status' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + + this.searchSubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const requestPayload = { + data: { + searchBy: 'NAME', + searchValue: query + }, + compressed: true, + target: 'models.commons.Search' + }; + this.companyService.searchEmployees(requestPayload).subscribe({ + next: (employees) => { + this.suggestions = employees.map(emp => ({ + ...emp, + display: `${emp.employeeId} - ${emp.fullName}` + })); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error searching employees', err); + this.suggestions = []; + } + }); + } else { + this.suggestions = []; + } + }); + } + + openNew() { + this.user = undefined; + this.userForm.reset(); + this.userRoles = []; + this.submitted = false; + this.loginIdTaken = false; + this.checkingLoginId = false; + this.selectedBranch = undefined; + this.selectedRole = undefined; + this.userDialog = true; + } + + editUser(user: UserDTO) { + this.userForm.reset(); + this.user = { ...user }; + this.userRoles = [...(user.userRoles || [])]; + this.selectedBranch = undefined; + this.selectedRole = undefined; + this.userForm.patchValue(user); + this.userDialog = true; + console.log(this.userForm.getRawValue()); + } + + hideDialog() { + this.userDialog = false; + this.submitted = false; + this.loginIdTaken = false; + this.checkingLoginId = false; + } + + toggleActive(user: UserDTO) { + const isActivating = !user.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + user.displayName + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonProps: { + label: 'No', + severity: 'secondary', + variant: 'text' + }, + acceptButtonProps: { + severity: isActivating ? 'success' : 'danger', + label: 'Yes' + }, + accept: () => { + this.userService.activateDeactivateUser(user.id!, isActivating).subscribe({ + next: (updatedUser) => { + user.active = updatedUser.active; + this.users = [...this.users]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `User ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + console.error('Error toggling user active status', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update user status', + life: 3000 + }); + } + }); + } + }); + } + + getSeverity(status: string) { + switch (status) { + case 'Active': + return 'success'; + case 'Inactive': + return 'warn'; + } + return 'info'; + } + + getErrorMessage(fieldName: string): string { + const control = this.userForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string): boolean { + const control = this.userForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submitted)); + } + + checkLoginId() { + const loginIdValue = this.userForm.get('loginId')?.value; + + if (!loginIdValue) { + this.clearLoginIdTakenError(); + return; + } + + if (!!this.user) { + return; + } + + this.checkingLoginId = true; + this.loginIdTaken = false; + this.cd.markForCheck(); + + const requestPayload = { + data: { + searchBy: 'loginId', + searchValue: loginIdValue + }, + compressed: true, + target: 'models.commons.Search' + }; + + this.userService.searchUser(requestPayload).subscribe({ + next: (userDTO) => { + this.checkingLoginId = false; + if (userDTO && userDTO.loginId) { + this.loginIdTaken = true; + this.userForm.get('loginId')?.setErrors({ taken: true }); + } else { + this.loginIdTaken = false; + const loginIdControl = this.userForm.get('loginId'); + if (loginIdControl) { + const errors = loginIdControl.errors; + if (errors) { + delete errors['taken']; + loginIdControl.setErrors(Object.keys(errors).length ? errors : null); + } + } + } + this.cd.markForCheck(); + }, + error: (err) => { + this.checkingLoginId = false; + console.error('Error checking loginId', err); + this.cd.markForCheck(); + } + }); + } + + onLoginIdInput() { + if (this.loginIdTaken) { + this.clearLoginIdTakenError(); + } + } + + private clearLoginIdTakenError() { + this.loginIdTaken = false; + const loginIdControl = this.userForm.get('loginId'); + if (loginIdControl) { + const errors = loginIdControl.errors; + if (errors) { + delete errors['taken']; + loginIdControl.setErrors(Object.keys(errors).length ? errors : null); + } + } + } + + addRole() { + if (!this.selectedBranch || !this.selectedRole) { + this.messageService.add({ + severity: 'warn', + summary: 'Warning', + detail: 'Please select both Branch and User Role', + life: 3000 + }); + return; + } + + const branch = this.branches.find(b => b.id === this.selectedBranch); + const role = this.userRolesOptions.find(r => r.id === this.selectedRole); + + if (branch && role) { + const exists = this.userRoles.some(ur => ur.branchId === branch.id && ur.roleId === role.id); + if (exists) { + this.messageService.add({ + severity: 'warn', + summary: 'Warning', + detail: 'Role already assigned for this branch', + life: 3000 + }); + return; + } + + const newRole: UserRoleDTO = { + branchId: branch.id, + branchName: branch.branchName, + roleId: role.id, + roleName: role.roleName, + groupName: role.groupName, + active: true, + defaultRole: false + }; + this.userRoles.push(newRole); + + this.selectedBranch = undefined; + this.selectedRole = undefined; + } + } + + toggleRoleActive(role: UserRoleDTO) { + role.active = role.active === false ? true : false; + } + + setDefaultRole(role: UserRoleDTO) { + const isCurrentlyDefault = role.defaultRole; + this.userRoles.forEach(r => { + if (r.branchId === role.branchId) { + r.defaultRole = false; + } + }); + if (!isCurrentlyDefault) { + role.defaultRole = true; + } + } + + onSelectEmployee(event: any) { + const emp = event.value; + const patch: any = { + fkEmployeeId: emp.id, + employeeId: emp.employeeId, + employeeName: emp.fullName, + fatherName: emp.fatherName, + department: emp.department, + designation: emp.designation + }; + if (!this.userForm.get('displayName')!.value) { + patch.displayName = emp.fullName; + } + this.userForm.patchValue(patch); + console.log(this.userForm); + } + + searchEmployees(event: any) { + this.searchSubject.next(event.query); + } + + + saveUser() { + this.submitted = true; + + if (this.userForm.invalid) { + return; + } + + const userData = this.userForm.getRawValue() as UserDTO; + userData.userRoles = this.userRoles; + + // Normalize null → empty array + const users = this.users ?? []; + + // Duplicate check (case-insensitive) + const existing = users.some(u => u.id !== userData.id && ( + u.loginId?.trim().toLowerCase() === userData.loginId?.trim().toLowerCase() + )); + + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'User with same login ID already exists', + life: 3000 + }); + return; + } + + this.userService.saveUser(userData).subscribe({ + next: (savedUser) => { + const index = userData.id + ? users.findIndex(u => u.id === userData.id) + : -1; + + if (index !== -1) { + // UPDATE + if (!savedUser.userRoles && userData.userRoles) { + savedUser.userRoles = userData.userRoles; + } + users[index] = savedUser; + } else { + // CREATE + if (!savedUser.userRoles && userData.userRoles) { + savedUser.userRoles = userData.userRoles; + } + users.push(savedUser); + } + + // Reassign once (change detection + null safety) + this.users = [...users]; + + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: index !== -1 + ? 'User Updated' + : 'User Created', + life: 3000 + }); + + this.userDialog = false; + this.user = undefined; + this.userForm.reset(); + this.submitted = false; + }, + error: (err) => { + console.error('Error saving user', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save user', + life: 3000 + }); + } + }); + } +} diff --git a/frontend/src/app/pages/account/vendor/vendor.component.css b/frontend/src/app/pages/account/vendor/vendor.component.css new file mode 100644 index 0000000..cffb4bb --- /dev/null +++ b/frontend/src/app/pages/account/vendor/vendor.component.css @@ -0,0 +1,14 @@ +:host ::ng-deep .p-dialog .p-button { + min-width: 6rem; +} + +:host ::ng-deep .p-datatable .p-datatable-header { + border-top: none; + background-color: transparent; +} + +/* Branch specific styles if necessary */ +:host ::ng-deep .p-fieldset .p-fieldset-legend { + font-size: 1rem; + padding: 0.5rem; +} diff --git a/frontend/src/app/pages/account/vendor/vendor.component.html b/frontend/src/app/pages/account/vendor/vendor.component.html new file mode 100644 index 0000000..09434e0 --- /dev/null +++ b/frontend/src/app/pages/account/vendor/vendor.component.html @@ -0,0 +1,419 @@ +
+ + + + + + + + + + + + + + +
+

Manage Vendors

+ + + + +
+ + + + # + Code + +
+ Name + +
+ + +
+ PAN No. + +
+ + +
+ MSME No. + +
+ + +
+ Updated At + +
+ + +
+ Updated By + +
+ + +
+ Status + +
+ + + + + + + {{ rowIndex + 1 }} + {{ vendor.code }} + {{ vendor.name }} + {{ vendor.panNo || '-' }} + {{ vendor.msmeNo || '-' }} + {{ vendor.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + + + {{ vendor.updatedUser }} + + + Not Available + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
+
+ +
+
+
Vendor Branches
+ +
+ + + + + Code + Name + City + State + Contact No + Actions + + + + + {{ branch.branchCode }} + {{ branch.branchName }} + {{ branch.cityName }} + {{ branch.stateName }} + {{ branch.contactNo }} + + + + + + + + + No branches added yet. + + + +
+ + + + + + + + + + + + +
+
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+
+ + + + + + + + +
diff --git a/frontend/src/app/pages/account/vendor/vendor.component.ts b/frontend/src/app/pages/account/vendor/vendor.component.ts new file mode 100644 index 0000000..59b37bd --- /dev/null +++ b/frontend/src/app/pages/account/vendor/vendor.component.ts @@ -0,0 +1,419 @@ +import { ChangeDetectorRef, Component, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { TableModule, Table } from 'primeng/table'; +import { DialogModule } from 'primeng/dialog'; +import { RippleModule } from 'primeng/ripple'; +import { ButtonModule } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialogModule } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { SelectModule } from 'primeng/select'; +import { TagModule } from 'primeng/tag'; +import { SkeletonModule } from 'primeng/skeleton'; +import { FormBuilder, FormGroup, ReactiveFormsModule, Validators, FormArray } from '@angular/forms'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { ValidationService } from '../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { MasterService } from '../../../services/masters/master.service'; +import { CityDTO, SearchDTO } from '../../../models/masters/masters'; +import { debounceTime, Subject } from 'rxjs'; +import { Request } from '../../../models/request.model'; +import { FloatLabelModule } from 'primeng/floatlabel'; +import { VendorService } from '../../../services/account/vendor/vendor.service'; +import { VendorDTO, VendorBranchDTO } from '../../../models/account.model'; +import { TabViewModule } from 'primeng/tabview'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +import { FormsModule } from '@angular/forms'; + +@Component({ + selector: 'app-vendor', + templateUrl: './vendor.component.html', + standalone: true, + imports: [ + CommonModule, FormsModule, ReactiveFormsModule, + TableModule, DialogModule, ButtonModule, ToastModule, ToolbarModule, ConfirmDialogModule, + InputTextModule, TextareaModule, SelectModule, TagModule, + SkeletonModule, IconFieldModule, + InputIconModule, TooltipModule, AutoCompleteModule, RippleModule, FloatLabelModule, TabViewModule + ], + providers: [MessageService, ConfirmationService], + styleUrl: './vendor.component.css' +}) +export class VendorComponent implements OnInit{ + vendorForm: FormGroup; + vendorDialog: boolean = false; + vendors!: VendorDTO[]; + + vendor!: VendorDTO; + + selectedVendors!: VendorDTO[] | null; + + submitted: boolean = false; + + isLoading: boolean = true; + + skeletonData: any[] = Array(10).fill({}); + + suggestions: CityDTO[] = []; + + private searchSubject = new Subject(); + + @ViewChild('dt') dt!: Table; + + cols!: Column[]; + + exportColumns!: ExportColumn[]; + + // Branch related + branchDialog: boolean = false; + branchForm: FormGroup; + submittedBranch: boolean = false; + currentVendorBranches: VendorBranchDTO[] = []; + editingBranchIndex: number = -1; // -1 means new branch + branchSuggestions: CityDTO[] = []; + private branchSearchSubject = new Subject(); + + constructor( + private vendorService: VendorService, + private masterService: MasterService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + + this.vendorForm = this.fb.group({ + id: [{ value: '', disabled: true }], + code: [{ value: '', disabled: false }, [Validators.required, ValidationService.codeValidator()]], + name: [{ value: '', disabled: false }, [Validators.required, ValidationService.nameValidator()]], + panNo: [{ value: '', disabled: false }, [ValidationService.panValidator()]], + cinNo: [{ value: '', disabled: false }, [ValidationService.cinValidator()]], + msmeNo: [{ value: '', disabled: false }, [ValidationService.msmeValidator()]], + active: [true] + }); + + this.branchForm = this.fb.group({ + id: [{ value: '', disabled: true }], + branchCode: [{ value: '', disabled: false }, [Validators.required]], + branchName: [{ value: '', disabled: false }, [Validators.required]], + officeNo: [{ value: '', disabled: false }], + street: [{ value: '', disabled: false }], + locality: [{ value: '', disabled: false }], + cityName: [{ value: '', disabled: false }], + stateName: [{ value: '', disabled: true }], + cityId: [{ value: null, disabled: true }], + stateId: [{ value: '', disabled: true }], + pinCode: [{ value: '', disabled: false }, [ValidationService.pincodeValidator()]], + emailId: [{ value: '', disabled: false }, [ValidationService.emailValidator()]], + contactNo: [{ value: '', disabled: false }, [ValidationService.mobileValidator()]], + contactPerson: [{ value: '', disabled: false }, [ValidationService.contactPersonValidator()]], + gstNo: [{ value: '', disabled: false }, [Validators.pattern('^[0-9]{2}[A-Z]{5}[0-9]{4}[A-Z]{1}[1-9A-Z]{1}Z[0-9A-Z]{1}$')]], // GST Validator if needed + active: [true] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + onSearch(event: Event) { + const input = event.target as HTMLInputElement; + this.dt.filterGlobal(input.value, 'contains'); + } + + // Branch City Search + searchBranchCities(event: any) { + const query = event.query; + this.branchSearchSubject.next(query); + } + + onSelectBranchCity(event: any) { + const city = event.value as CityDTO; + this.branchForm.patchValue({ + cityId: city.id, + stateId: city.stateId, + cityName: city.cityName, + stateName: city.stateName + }); + } + + + ngOnInit() { + this.loadAllVendors(); + + this.cols = [ + { field: 'code', header: 'Code', customExportHeader: 'Code' }, + { field: 'name', header: 'Name' }, + { field: 'panNo', header: 'PAN No.' }, + { field: 'cinNo', header: 'CIN No.' }, + { field: 'msmeNo', header: 'MSME No.' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + + this.branchSearchSubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const requestPayload: Request = { + data: { + searchBy: 'CITY', + searchValue: query + }, + compressed: true, + target: 'models.commons.Search' + }; + this.masterService.searchCityStates(requestPayload).subscribe({ + next: (cities) => { + this.branchSuggestions = cities.map(city => ({ + ...city, + display: `${city.cityName}, ${city.stateName}` + })); + this.cd.markForCheck(); + }, + error: (err) => { + console.error('Error searching cities', err); + this.branchSuggestions = []; + } + }); + } else { + this.branchSuggestions = []; + } + }); + } + + loadAllVendors() { + this.isLoading = true; + this.vendorService.getAllVendors().subscribe({ + next: (data) => { + this.isLoading = false; + this.vendors = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + } + + openNew() { + this.vendor = { code: '', name: '', active: true, branches: [] }; + this.vendorForm.reset(); + this.vendorForm.patchValue({ active: true }); + this.currentVendorBranches = []; + this.submitted = false; + this.vendorDialog = true; + } + + editVendor(vendor: VendorDTO) { + this.vendorForm.reset(); + this.vendor = { ...vendor }; + this.currentVendorBranches = vendor.branches ? [...vendor.branches] : []; + this.vendorForm.patchValue(vendor); + this.vendorDialog = true; + } + + hideDialog() { + this.vendorDialog = false; + this.submitted = false; + } + + toggleActive(vendor: VendorDTO) { + const isActivating = !vendor.active; + this.confirmationService.confirm({ + message: `Are you sure you want to ${isActivating ? 'activate' : 'deactivate'} ` + vendor.name + '?', + header: 'Confirm', + icon: 'pi pi-exclamation-triangle', + rejectButtonStyleClass: 'p-button-text p-button-secondary', + acceptButtonStyleClass: isActivating ? 'p-button-success' : 'p-button-danger', + accept: () => { + // Assuming activateDeactivateVendor exists or use save + /* + this.vendorService.activateDeactivateVendor(vendor.id!, isActivating).subscribe({ + next: (updated) => { + vendor.active = updated.active; // or isActivating + this.messageService.add({severity:'success', summary: 'Successful', detail: `Vendor ${isActivating ? 'Activated' : 'Deactivated'}`, life: 3000}); + }, + error: () => this.messageService.add({severity:'error', summary: 'Error', detail: 'Failed to update status', life: 3000}) + }); + */ + // Using save for now as placeholder if specific endpoint not confirmed, but typically exists. + // I will simulate success for UI if backend not fully ready or use save. + vendor.active = isActivating; + this.vendorService.saveVendor(vendor).subscribe({ + next: (res) => { + vendor.active = res.active; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: `Vendor ${isActivating ? 'Activated' : 'Deactivated'}`, + life: 3000 + }); + }, + error: (err) => { + vendor.active = !isActivating; // Revert + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to update status', + life: 3000 + }); + } + }) + + } + }); + } + + getSeverity(status: boolean) { + switch (status) { + case true: + return 'success'; + case false: + return 'warn'; + } + } + + getErrorMessage(fieldName: string, form: FormGroup = this.vendorForm): string { + const control = form.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isFieldInvalid(fieldName: string, form: FormGroup = this.vendorForm, submitted: boolean = this.submitted): boolean { + const control = form.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || submitted)); + } + + saveVendor() { + this.submitted = true; + + if (this.vendorForm.valid) { + const vendorData = this.vendorForm.getRawValue(); + vendorData.branches = this.currentVendorBranches; + + // Check duplicates in list (simple client side check) + const existing = this.vendors.find(v => v.id !== vendorData.id && (v.code === vendorData.code || v.name === vendorData.name)); + if (existing) { + this.messageService.add({ + severity: 'error', + summary: 'Validation Error', + detail: 'Vendor code or name already exists', + life: 3000 + }); + return; + } + + this.vendorService.saveVendor(vendorData).subscribe({ + next: (newVendor) => { + if (vendorData.id) { + const index = this.vendors.findIndex(v => v.id === newVendor.id); + if (index !== -1) { + this.vendors[index] = newVendor; + } + } else { + this.vendors.push(newVendor); + } + this.vendors = [...this.vendors]; + this.messageService.add({ + severity: 'success', + summary: 'Successful', + detail: 'Vendor Saved', + life: 3000 + }); + this.vendorDialog = false; + this.vendor = {} as any; + }, + error: (err) => { + console.error('Error saving vendor', err); + this.messageService.add({ + severity: 'error', + summary: 'Error', + detail: 'Failed to save vendor', + life: 3000 + }); + } + }); + } + } + + + /// BRANCH METHODS /// + + openNewBranch() { + this.editingBranchIndex = -1; + this.branchForm.reset(); + this.branchForm.patchValue({ active: true }); + this.submittedBranch = false; + this.branchDialog = true; + } + + editBranch(branch: VendorBranchDTO, index: number) { + this.editingBranchIndex = index; + this.branchForm.reset(); + this.branchForm.patchValue(branch); // Needs correct mapping, especially city object for autocomplete + // If cityId is present but cityName is not in form search object format, we might need to handle it. + // Autocomplete expects an object with 'display' if strictly typed? No, form value is usually the string or object depending on config. + // Here we patch values, if cityId/stateId are there, we display text in cityName/stateName. + // But the autocomplete uses 'cityName' field? No, uses 'cityName' form control. + // If we pass a string to it, it shows string. If object, it shows field. + // Let's assume fetching vendor returns cityName string. + // We might need to manually set the object for autocomplete if we want it to look "selected". + // But for now, simple patch. + if (branch.cityName) { + // For display purpose in autocomplete if it expects object + this.branchForm.patchValue({ + cityName: { cityName: branch.cityName, stateName: branch.stateName, display: `${branch.cityName}, ${branch.stateName}`, id: branch.cityId, stateId: branch.stateId } + }); + } + + this.branchDialog = true; + } + + deleteBranch(index: number) { + this.currentVendorBranches.splice(index, 1); + } + + saveBranch() { + this.submittedBranch = true; + if (this.branchForm.valid) { + const branchData = this.branchForm.getRawValue(); + // Extract city/state from autocomplete object if needed + if (typeof branchData.cityName === 'object') { + branchData.cityId = branchData.cityName.id; + branchData.stateId = branchData.cityName.stateId; + branchData.stateName = branchData.cityName.stateName; + branchData.cityName = branchData.cityName.cityName; + } + + if (this.editingBranchIndex === -1) { + this.currentVendorBranches.push(branchData); + } else { + this.currentVendorBranches[this.editingBranchIndex] = branchData; + } + this.branchDialog = false; + this.branchForm.reset(); + } + } + + hideBranchDialog() { + this.branchDialog = false; + this.submittedBranch = false; + } +} diff --git a/frontend/src/app/pages/dashboard/dashboard.component.ts b/frontend/src/app/pages/dashboard/dashboard.component.ts new file mode 100644 index 0000000..8e6f11d --- /dev/null +++ b/frontend/src/app/pages/dashboard/dashboard.component.ts @@ -0,0 +1,17 @@ +import { Component } from '@angular/core'; +import { MenuComponent } from "../../fragments/menu/menu.component"; +import { RouterOutlet } from '@angular/router'; + +@Component({ + selector: 'app-dashboard', + imports: [MenuComponent, RouterOutlet], + template: `
+ +
+ +
+
` +}) +export class DashboardComponent { + constructor(){} +} diff --git a/frontend/src/app/pages/session/auth/authorize.component.css b/frontend/src/app/pages/session/auth/authorize.component.css new file mode 100644 index 0000000..0372f99 --- /dev/null +++ b/frontend/src/app/pages/session/auth/authorize.component.css @@ -0,0 +1,38 @@ +.login-page { + height: 99vh; +} + +::ng-deep .p-card { + position: relative; + border-radius: 24px !important; + background: white; + padding: 1rem; + overflow: hidden; +} + +/* Gradient border */ +::ng-deep .p-card::before { + content: ""; + position: absolute; + inset: 0; + padding: 3px; /* border thickness */ + border-radius: 24px; + filter: drop-shadow(0 0 12px rgba(91, 185, 138, 0.35)); + background: linear-gradient( + 180deg, + #ffb455 0%, + rgba(255, 180, 85, 0.6) 40%, + rgba(255, 180, 85, 0.15) 70%, + transparent 100% + ); + + /* Mask trick = border only */ + -webkit-mask: + linear-gradient(#fff 0 0) content-box, + linear-gradient(#fff 0 0); + -webkit-mask-composite: xor; + mask-composite: exclude; + + pointer-events: none; +} + diff --git a/frontend/src/app/pages/session/auth/authorize.component.html b/frontend/src/app/pages/session/auth/authorize.component.html new file mode 100644 index 0000000..1dfaa5b --- /dev/null +++ b/frontend/src/app/pages/session/auth/authorize.component.html @@ -0,0 +1,38 @@ + diff --git a/frontend/src/app/pages/session/auth/authorize.component.ts b/frontend/src/app/pages/session/auth/authorize.component.ts new file mode 100644 index 0000000..f76af42 --- /dev/null +++ b/frontend/src/app/pages/session/auth/authorize.component.ts @@ -0,0 +1,98 @@ +import { SessionService } from './../../../services/commons/session.service'; +import { CommonModule} from '@angular/common'; +import { FormsModule } from '@angular/forms'; +import { Component, OnInit } from '@angular/core'; +import { ButtonModule } from 'primeng/button';; +import { InputTextModule } from 'primeng/inputtext'; +import { CardModule, Card } from 'primeng/card'; +import { InputGroupModule, InputGroup } from 'primeng/inputgroup'; +import { InputGroupAddonModule, InputGroupAddon } from 'primeng/inputgroupaddon'; +import { PasswordModule, Password } from 'primeng/password'; +import { FloatLabelModule, FloatLabel } from 'primeng/floatlabel'; +import { KeyFilterModule } from 'primeng/keyfilter'; +import { FormBuilder, FormGroup, Validators, ReactiveFormsModule} from '@angular/forms'; +import { Request } from '../../../models/request.model'; +import { HttpService } from '../../../services/http.service'; +import { MessageModule } from 'primeng/message'; +import { StyleClassModule } from 'primeng/styleclass'; +import { Router } from '@angular/router'; +import { ResponseDto } from '../../../models/response.dto'; +import { Company, Branch } from '../../../models/session.model'; +import { SelectModule } from 'primeng/select'; +import { environment } from '../../../../environments/environment'; + +@Component({ + selector: 'app-authorize', + imports: [CommonModule, FormsModule, CardModule, ButtonModule, InputTextModule, InputGroupModule, InputGroupAddonModule, PasswordModule, + FloatLabelModule, ReactiveFormsModule, KeyFilterModule, MessageModule, StyleClassModule, SelectModule], + templateUrl: './authorize.component.html', + styleUrl: './authorize.component.css' +}) +export class AuthorizeComponent implements OnInit { + companies: Company[] = []; + branches: Branch[] = []; + isCompanyDisabled: boolean = false; + authForm: FormGroup; + message?: string; + + constructor(private fb: FormBuilder, private http: HttpService, private router: Router, private sessionService: SessionService) { + this.authForm = this.fb.group({ + companyId: ['', Validators.required], + branchId: ['', Validators.required] + }); + } + + ngOnInit(){ + const companyBranchRoles = sessionStorage.getItem('companies'); + + if (!companyBranchRoles) { + this.sessionService.logout(); + return; + } + sessionStorage.removeItem('companies'); + + try { + this.companies = JSON.parse(companyBranchRoles); + } catch (e) { + this.sessionService.logout(); + } + + if (this.companies.length === 1) { + this.authForm.patchValue({ companyId: this.companies[0].id }); + this.isCompanyDisabled = true; + } + + this.updateBranches(); + this.authForm.get('companyId')?.valueChanges.subscribe(() => { + this.updateBranches(); + }); + } + + private updateBranches(): void { + const companyId = this.authForm.get('companyId')?.value; + const selectedCompany = this.companies.find(c => c.id === companyId); + this.branches = selectedCompany?.branches || []; + this.authForm.patchValue({ branchId: '' }); + } + + onAuthorize() { + this.message = ''; + if (this.authForm.valid) { + const requestPayload: Request = { + data: this.authForm.get('branchId')?.value + }; + + this.http.post(`${environment.authService}/3z4mkell5g5aset/authorize`, requestPayload).subscribe({ + next: (response) => { + this.router.navigate(['/user']); + }, + error: (err) => { + this.message = err.message; + } + }); + } else { + alert('Please fill out the form correctly'); + } + } + +} diff --git a/frontend/src/app/pages/session/profile/profile.component.css b/frontend/src/app/pages/session/profile/profile.component.css new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/frontend/src/app/pages/session/profile/profile.component.css @@ -0,0 +1 @@ + diff --git a/frontend/src/app/pages/session/profile/profile.component.html b/frontend/src/app/pages/session/profile/profile.component.html new file mode 100644 index 0000000..4112a40 --- /dev/null +++ b/frontend/src/app/pages/session/profile/profile.component.html @@ -0,0 +1,167 @@ +
+
+ +
+ + User Details +
+ +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + +
+ + Employment Details +
+ +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ + + + +
+
+ +
+
+ diff --git a/frontend/src/app/pages/session/profile/profile.component.spec.ts b/frontend/src/app/pages/session/profile/profile.component.spec.ts new file mode 100644 index 0000000..658617a --- /dev/null +++ b/frontend/src/app/pages/session/profile/profile.component.spec.ts @@ -0,0 +1,21 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; +import { ProfileComponent } from './profile.component'; + +describe('ProfileComponent', () => { + let component: ProfileComponent; + let fixture: ComponentFixture; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [ProfileComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(ProfileComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/frontend/src/app/pages/session/profile/profile.component.ts b/frontend/src/app/pages/session/profile/profile.component.ts new file mode 100644 index 0000000..3cdb91f --- /dev/null +++ b/frontend/src/app/pages/session/profile/profile.component.ts @@ -0,0 +1,60 @@ +import { UserProfile } from './../../../models/session.model'; +import { Component, OnInit } from '@angular/core'; +import { FloatLabelModule } from "primeng/floatlabel" +import { InputTextModule } from 'primeng/inputtext'; +import { DividerModule } from 'primeng/divider'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule } from '@angular/forms'; +import { SessionService } from '../../../services/commons/session.service'; +import { CardModule } from "primeng/card"; + +@Component({ + selector: 'app-profile', + imports: [FloatLabelModule, InputTextModule, FormsModule, ReactiveFormsModule, CardModule, DividerModule], + templateUrl: './profile.component.html', + styleUrl: './profile.component.css' +}) +export class ProfileComponent implements OnInit { + userProfile: UserProfile; + profileForm: FormGroup; + constructor(private fb: FormBuilder, private sessionService: SessionService){ + this.userProfile = {}; + this.profileForm = this.fb.group({ + username: [{ value: null, disabled: true }], + roleName: [{ value: null, disabled: true }], + employeeId: [{ value: null, disabled: true }], + joiningDate: [{ value: null, disabled: true }], + displayName: [null], + department: [{ value: null, disabled: true }], + designation: [{ value: null, disabled: true }], + name: [{ value: null, disabled: true }], + fatherName: [{ value: null, disabled: true }], + gender: [{ value: null, disabled: true }], + dob: [{ value: null, disabled: true }], + contactNo: [{ value: null, disabled: true }], + alternateContactNo: [null], + emailId: [null] + }); + } + + ngOnInit(): void { + const userDetails = this.sessionService.getItem('userDetails'); + const companies = this.sessionService.getItem("companies") ? JSON.parse(this.sessionService.getItem("companies")) : ''; + this.userProfile = userDetails ? JSON.parse(userDetails) : {}; + //One liner approach commented out for later user + //this.profileForm.patchValue(this.userProfile as Partial); + this.profileForm.patchValue({ + ...this.userProfile, + roleName: companies ? companies[0].branches[0].roles[0].groupName : '', + joiningdate: this.userProfile.joiningDate + ? new Date(this.userProfile.joiningDate) + : null, + dob: this.userProfile.dob + ? new Date(this.userProfile.dob) + : null + }); + } + + onSave() : void{ + + } +} diff --git a/frontend/src/app/pages/tools/allocation/localities/localities.component.css b/frontend/src/app/pages/tools/allocation/localities/localities.component.css new file mode 100644 index 0000000..aec6235 --- /dev/null +++ b/frontend/src/app/pages/tools/allocation/localities/localities.component.css @@ -0,0 +1 @@ +/* localities.component.css */ diff --git a/frontend/src/app/pages/tools/allocation/localities/localities.component.html b/frontend/src/app/pages/tools/allocation/localities/localities.component.html new file mode 100644 index 0000000..8374135 --- /dev/null +++ b/frontend/src/app/pages/tools/allocation/localities/localities.component.html @@ -0,0 +1,289 @@ +
+ + + + + + + + + + + + + + + +
+

Manage Localities

+ + + + +
+ + + + # + +
Locality Name
+ + +
Admin Area
+ + +
Pincode
+ + +
Area
+ + +
Updated At
+ + +
Updated By
+ + +
Status
+ + + + + + + {{ rowIndex + 1 }} + {{ row.localityName }} + {{ row.adminArea }}, {{ row.masterStateName }} + {{ row.pincode }} + {{ row.areaName }} + {{ row.updatedAt | date:'dd/MM/yyyy HH:mm:ss' }} + {{ row.updatedUser }}N/A + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+
+ + +
+
+ +
+
+ + +
+
+ + + + + + + +
+
+
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+ +
+ + + + +
+
+
+ + + + + + + + + +
diff --git a/frontend/src/app/pages/tools/allocation/localities/localities.component.ts b/frontend/src/app/pages/tools/allocation/localities/localities.component.ts new file mode 100644 index 0000000..732c176 --- /dev/null +++ b/frontend/src/app/pages/tools/allocation/localities/localities.component.ts @@ -0,0 +1,542 @@ +import { ChangeDetectorRef, Component, ElementRef, OnInit, ViewChild } from '@angular/core'; +import { ConfirmationService, MessageService } from 'primeng/api'; +import { FormBuilder, FormGroup, FormsModule, ReactiveFormsModule, Validators } from '@angular/forms'; +import { Table, TableModule } from 'primeng/table'; +import { Dialog } from 'primeng/dialog'; +import { Button } from 'primeng/button'; +import { ToastModule } from 'primeng/toast'; +import { ToolbarModule } from 'primeng/toolbar'; +import { ConfirmDialog } from 'primeng/confirmdialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { TextareaModule } from 'primeng/textarea'; +import { CommonModule } from '@angular/common'; +import { SelectModule } from 'primeng/select'; +import { Tag } from 'primeng/tag'; +import { Skeleton } from 'primeng/skeleton'; +import { IconFieldModule } from 'primeng/iconfield'; +import { InputIconModule } from 'primeng/inputicon'; +import { DropdownModule } from 'primeng/dropdown'; +import { FloatLabelModule } from "primeng/floatlabel"; +import { ValidationService } from '../../../../services/utilities/validation.service'; +import { TooltipModule } from 'primeng/tooltip'; +import { AutoCompleteModule } from 'primeng/autocomplete'; +import { CheckboxModule } from 'primeng/checkbox'; +import { Subject, debounceTime } from 'rxjs'; + +import { AllocationService } from '../../../../services/tools/allocation/allocation.service'; +import { MasterService } from '../../../../services/masters/master.service'; +import { CompanyService } from '../../../../services/account/company/company.service'; +import { OlaService } from '../../../../services/ola/ola.service'; +import { LocalityDTO, AreaDTO, LocalityTypeDTO } from '../../../../models/tools.model'; +import { CityDTO } from '../../../../models/masters/masters'; +import { CompanyVerifierModel } from '../../../../models/account.model'; + +interface Column { + field: string; + header: string; + customExportHeader?: string; +} + +interface ExportColumn { + title: string; + dataKey: string; +} + +@Component({ + selector: 'app-localities', + templateUrl: './localities.component.html', + standalone: true, + imports: [TableModule, Dialog, SelectModule, ToastModule, ToolbarModule, ConfirmDialog, InputTextModule, TextareaModule, CommonModule, DropdownModule, Tag, InputTextModule, IconFieldModule, InputIconModule, Button, FloatLabelModule, FormsModule, ReactiveFormsModule, TooltipModule, Skeleton, AutoCompleteModule, CheckboxModule], + providers: [MessageService, ConfirmationService], + styleUrls: ['./localities.component.css'] +}) +export class LocalitiesComponent implements OnInit { + localityForm: FormGroup; + areaForm: FormGroup; + + localityDialog: boolean = false; + areaDialog: boolean = false; + + localities!: LocalityDTO[]; + locality: LocalityDTO | undefined; + + localityTypes: LocalityTypeDTO[] = []; + areas: AreaDTO[] = []; + + selectedLocalities!: LocalityDTO[] | null; + + submittedLocality: boolean = false; + submittedArea: boolean = false; + + isLoading: boolean = true; + skeletonData: any[] = Array(10).fill({}); + + // Suggestions + areaSuggestions: any[] = []; + citySuggestions: any[] = []; + olaSuggestions: any[] = []; + + // Verifiers for dropdowns + verifiers: CompanyVerifierModel[] = []; + + private searchAreaSubject = new Subject(); + private searchCitySubject = new Subject(); + private searchOlaSubject = new Subject(); + + @ViewChild('dt') dt!: Table; + @ViewChild('localityNameInput') localityNameInput!: ElementRef; + + cols!: Column[]; + exportColumns!: ExportColumn[]; + + constructor( + private allocationService: AllocationService, + private masterService: MasterService, + private companyService: CompanyService, + private olaService: OlaService, + private messageService: MessageService, + private confirmationService: ConfirmationService, + private cd: ChangeDetectorRef, + private fb: FormBuilder, + ) { + this.localityForm = this.fb.group({ + id: [{ value: '', disabled: true }], + areaId: [{ value: null, disabled: false }, Validators.required], + stateCityId: [{ value: null, disabled: false }, Validators.required], + cityAutoComplete: [{ value: null, disabled: false }, Validators.required], + localityTypeId: [{ value: null, disabled: false }, Validators.required], + localityName: [{ value: '', disabled: false }, Validators.required], + adminArea: [{ value: '', disabled: false }, Validators.required], + olaPlaceId: [{ value: null, disabled: false }], + olaLocalityAutoComplete: [{ value: null, disabled: false }], + olaLocality: [{ value: '', disabled: false }], + stateName: [{ value: '', disabled: false }], + pincode: [{ value: '', disabled: false }], + latitude: [{ value: null, disabled: true }], + longitude: [{ value: null, disabled: true }], + active: [{ value: true, disabled: false }] + }); + + this.areaForm = this.fb.group({ + id: [{ value: '', disabled: true }], + areaName: [{ value: '', disabled: false }, Validators.required], + + groupAVerifierId: [{ value: null, disabled: false }, Validators.required], + groupBVerifierId: [{ value: null, disabled: false }, Validators.required], + groupCVerifierId: [{ value: null, disabled: false }, Validators.required] + }); + } + + exportCSV() { + this.dt.exportCSV(); + } + + ngOnInit() { + this.loadLocalities(); + this.loadLocalityTypes(); + this.loadAreas(); + + this.searchAreaSubject.pipe(debounceTime(400)).subscribe(query => { + if (query && query.length >= 2) { + const req = { data: { searchBy: 'NAME', searchValue: query }, compressed: true, target: 'models.commons.Search' }; + this.allocationService.searchAreas(req).subscribe({ + next: (areas) => { + this.areaSuggestions = areas; + this.cd.markForCheck(); + }, + error: () => this.areaSuggestions = [] + }); + } else { this.areaSuggestions = []; } + }); + + this.searchCitySubject.pipe(debounceTime(300)).subscribe(query => { + if (query && query.length >= 2) { + const req: any = { + data: { searchBy: 'CITY', searchValue: query }, + compressed: true, + target: 'models.commons.Search' + }; + this.masterService.searchCityStates(req).subscribe({ + next: (cities) => { + this.citySuggestions = cities.map(city => ({ + ...city, + display: `${city.cityName}, ${city.stateName}` + })); + this.cd.markForCheck(); + }, + error: () => this.citySuggestions = [] + }); + } else { this.citySuggestions = []; } + }); + + this.searchOlaSubject.pipe(debounceTime(400)).subscribe(query => { + if (query && query.length >= 3) { + this.olaService.autocomplete(query).subscribe({ + next: (res) => { + let predictions = res; + if (res && res.predictions) { + predictions = res.predictions; + } + this.olaSuggestions = Array.isArray(predictions) ? predictions.map((p: any) => { + let displayName = p.structured_formatting?.main_text || ''; + const terms = p.terms || []; + const n = terms.length; + if (n >= 4) displayName += ', ' + terms[n - 4].value; + if (n >= 3) displayName += ', ' + terms[n - 3].value; + if (n >= 2) displayName += ', ' + terms[n - 2].value; + if (n >= 1) displayName += ', ' + terms[n - 1].value; + + return { + name: displayName || p.description, + raw: p, + place_id: p.place_id + }; + }) : []; + this.cd.markForCheck(); + }, + error: () => { + this.olaSuggestions = []; + } + }); + } else { this.olaSuggestions = []; } + }); + + this.areaForm.get('areaName')?.valueChanges.pipe(debounceTime(400)).subscribe(val => { + if(val == null) return; + const textToSearch = typeof val === 'string' ? val : val.areaName; + + if (textToSearch && textToSearch.trim().length >= 2) { + const req = { data: { searchBy: 'NAME', searchValue: textToSearch }, compressed: true, target: 'models.commons.Search' }; + this.allocationService.searchAreas(req).subscribe({ + next: (areas) => { + const exactMatch = areas.find(a => a.areaName?.toLowerCase().trim() === textToSearch.toLowerCase().trim()); + if (exactMatch) { + const vA = this.verifiers.find(v => v.fullName === exactMatch.groupAVerifierName); + const vB = this.verifiers.find(v => v.fullName === exactMatch.groupBVerifierName); + const vC = this.verifiers.find(v => v.fullName === exactMatch.groupCVerifierName); + + this.areaForm.patchValue({ + id: exactMatch.id, + groupAVerifierId: vA ? vA.id : null, + groupBVerifierId: vB ? vB.id : null, + groupCVerifierId: vC ? vC.id : null + }, { emitEvent: false }); + } else { + this.areaForm.patchValue({ id: '' }, { emitEvent: false }); + } + } + }); + } else { + this.areaForm.patchValue({ id: '' }, { emitEvent: false }); + } + }); + + this.loadVerifiers(); + } + + loadVerifiers() { + this.companyService.getAllVerifiers().subscribe({ + next: (data) => { + this.verifiers = data.map(v => ({ + ...v, + display: `${v.verifierCode} - ${v.fullName}` + })); + this.cd.markForCheck(); + }, + error: (err) => console.error('Error loading verifiers', err) + }); + } + + loadLocalityTypes() { + this.allocationService.getAllLocalityTypes().subscribe({ + next: (types) => { + this.localityTypes = types; + } + }); + } + + loadAreas() { + this.allocationService.getAllAreas().subscribe({ + next: (data) => { + this.areas = data; + this.cd.markForCheck(); + }, + error: (err) => console.error(err) + }); + } + + loadLocalities() { + this.isLoading = true; + this.allocationService.getAllLocalities().subscribe({ + next: (data) => { + this.isLoading = false; + this.localities = data; + this.cd.markForCheck(); + }, + error: (err) => { + this.isLoading = false; + console.error(err); + } + }); + + this.cols = [ + { field: 'localityName', header: 'Locality Name' }, + { field: 'adminArea', header: 'Admin Area' }, + { field: 'areaName', header: 'Area' }, + { field: 'pincode', header: 'Pincode' }, + { field: 'updatedAt', header: 'Last Updated At' }, + { field: 'updatedUser', header: 'Last Updated By' } + ]; + + this.exportColumns = this.cols.map((col) => ({ title: col.header, dataKey: col.field })); + } + + // AREA AUTOCOMPLETE (Used only by Area Dialog now) + searchAreas(event: any) { this.searchAreaSubject.next(event.query); } + + onSelectAreaGroup(event: any) { + const item = event.value; + const vA = this.verifiers.find(v => v.fullName === item.groupAVerifierName); + const vB = this.verifiers.find(v => v.fullName === item.groupBVerifierName); + const vC = this.verifiers.find(v => v.fullName === item.groupCVerifierName); + + this.areaForm.patchValue({ + id: item.id, + areaName: item.areaName, + groupAVerifierId: vA ? vA.id : null, + groupBVerifierId: vB ? vB.id : null, + groupCVerifierId: vC ? vC.id : null + }); + } + + // CITY AUTOCOMPLETE + searchCities(event: any) { this.searchCitySubject.next(event.query); } + onSelectCity(event: any) { + this.localityForm.patchValue({ stateCityId: event.value.id }); + } + + // OLA AUTOCOMPLETE + searchOlaLocalities(event: any) { this.searchOlaSubject.next(event.query); } + onSelectOlaLocality(event: any) { + const item = event.value; + const rawItem = item.raw || item; + let pincode = ''; + let state = ''; + if (rawItem && rawItem.terms && rawItem.terms.length > 0) { + const terms = rawItem.terms; + const n = terms.length; + if (n >= 2) pincode = terms[n - 2].value; + if (n >= 3) state = terms[n - 3].value; + } + + this.localityForm.patchValue({ + olaPlaceId: rawItem.place_id || item.place_id, + olaLocality: rawItem.structured_formatting?.main_text || item.name, + stateName: state, + pincode: pincode, + latitude: rawItem.geometry?.location?.lat || item.lat, + longitude: rawItem.geometry?.location?.lng || item.lng + }); + } + + // VERIFIER AUTOCOMPLETE + // Using p-select now, so these are no longer needed + // DIALOGS + openNewLocality() { + this.locality = undefined; + this.localityForm.reset(); + this.localityForm.patchValue({ active: true }); + this.submittedLocality = false; + this.localityDialog = true; + } + + editLocality(loc: LocalityDTO) { + this.localityForm.reset(); + this.locality = { ...loc }; + + // Setup autoComplete selections and dropdown matching + const patchData: any = { ...this.locality }; + + if (patchData.masterCityName && patchData.masterStateName) { + patchData.cityAutoComplete = { id: patchData.stateCityId, display: `${patchData.masterCityName}, ${patchData.masterStateName}` }; + } else { + patchData.cityAutoComplete = { id: patchData.stateCityId, display: 'Selected City' }; + } + + if (patchData.olaPlaceId) { + patchData.olaLocalityAutoComplete = { place_id: patchData.olaPlaceId, name: patchData.olaLocality }; + } + + const matchingArea = this.areas.find(a => a.areaName === patchData.areaName); + if (matchingArea) patchData.areaId = matchingArea.id; + + const matchingLocType = this.localityTypes.find(l => l.type === patchData.localityTypeName); + if (matchingLocType) patchData.localityTypeId = matchingLocType.id; + + this.localityForm.patchValue(patchData); + this.localityDialog = true; + } + + openNewArea() { + this.areaForm.reset(); + this.submittedArea = false; + this.areaDialog = true; + } + + hideLocalityDialog() { + this.localityDialog = false; + this.submittedLocality = false; + } + + clearLocalityForm() { + this.localityForm.patchValue({ + id: null, + localityTypeId: null, + adminArea: '', + olaLocalityAutoComplete: null, + olaLocality: '', + stateName: '', + pincode: '', + latitude: null, + longitude: null, + olaPlaceId: null, + localityName: '' + }); + this.submittedLocality = false; + + setTimeout(() => { + if (this.localityNameInput && this.localityNameInput.nativeElement) { + this.localityNameInput.nativeElement.focus(); + } + }, 100); + } + + hideAreaDialog() { + this.areaDialog = false; + this.submittedArea = false; + } + + getSeverity(status: boolean) { + return status ? 'success' : 'warn'; + } + + getLocalityErrorMessage(fieldName: string): string { + const control = this.localityForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isLocalityFieldInvalid(fieldName: string): boolean { + const control = this.localityForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submittedLocality)); + } + + getAreaErrorMessage(fieldName: string): string { + const control = this.areaForm.get(fieldName); + return control ? ValidationService.getErrorMessage(control, fieldName) : ''; + } + + isAreaFieldInvalid(fieldName: string): boolean { + const control = this.areaForm.get(fieldName); + return !!(control && control.invalid && (control.dirty || control.touched || this.submittedArea)); + } + + validateUniqueLocality(localityName: string): boolean { + const existing = (this.localities ?? []).some(o => + o.id !== this.localityForm.get('id')?.value && + o.localityName?.toLowerCase().trim() === localityName.toLowerCase().trim() + ); + return !existing; + } + + saveLocality() { + this.submittedLocality = true; + + if (this.localityForm.invalid) { + return; + } + + const data = this.localityForm.getRawValue(); + + if (!this.validateUniqueLocality(data.localityName)) { + this.messageService.add({ severity: 'error', summary: 'Validation Error', detail: 'Locality Name already exists', life: 3000 }); + return; + } + + const locDto: LocalityDTO = { + id: data.id, + areaId: data.areaId, + stateCityId: data.stateCityId, + localityTypeId: data.localityTypeId, + localityName: data.localityName, + adminArea: data.adminArea, + olaPlaceId: data.olaPlaceId, + olaLocality: data.olaLocality, + latitude: data.latitude, + longitude: data.longitude, + stateName: data.stateName, + pincode: data.pincode, + active: data.active + }; + + this.allocationService.saveLocality(locDto).subscribe({ + next: (savedLoc) => { + const index = locDto.id ? (this.localities ?? []).findIndex(v => v.id === locDto.id) : -1; + if (index !== -1) { + this.localities[index] = savedLoc; + } else { + if (!this.localities) { + this.localities = []; + } + this.localities.push(savedLoc); + } + this.localities = [...this.localities]; + this.messageService.add({ severity: 'success', summary: 'Successful', detail: index !== -1 ? 'Locality Updated' : 'Locality Created', life: 3000 }); + + if (locDto.id) { + this.localityForm.reset(); + this.hideLocalityDialog(); + } else { + this.clearLocalityForm(); + } + }, + error: (err) => { + console.error(err); + this.messageService.add({ severity: 'error', summary: 'Error', detail: 'Failed to save locality', life: 3000 }); + } + }); + } + + saveArea() { + this.submittedArea = true; + + if (this.areaForm.invalid) { + return; + } + + const data = this.areaForm.getRawValue(); + const extractedName = typeof data.areaName === 'string' ? data.areaName : data.areaName.areaName; + + const areaDto: AreaDTO = { + id: data.id, + areaName: extractedName, + groupAVerifierId: data.groupAVerifierId, + groupBVerifierId: data.groupBVerifierId, + groupCVerifierId: data.groupCVerifierId, + active: true + }; + + this.allocationService.saveArea(areaDto).subscribe({ + next: () => { + this.messageService.add({ severity: 'success', summary: 'Successful', detail: 'Area Group Created/Updated', life: 3000 }); + this.areaDialog = false; + this.areaForm.reset(); + this.submittedArea = false; + }, + error: (err) => { + console.error(err); + this.messageService.add({ severity: 'error', summary: 'Error', detail: 'Failed to save Area Group', life: 3000 }); + } + }); + } + +} diff --git a/frontend/src/app/services/account/company/company.service.ts b/frontend/src/app/services/account/company/company.service.ts new file mode 100644 index 0000000..c91c942 --- /dev/null +++ b/frontend/src/app/services/account/company/company.service.ts @@ -0,0 +1,199 @@ +import { Request } from './../../../models/request.model'; +import { Injectable } from '@angular/core'; +import { Observable } from 'rxjs'; +import { map } from 'rxjs/operators'; +import { environment } from '../../../../environments/environment'; +import { HttpService } from '../../http.service'; + +import { SubsidiaryDTO, DepartmentDTO, DesignationDTO, EmployeeDTO, CompanyVerifierModel } from '../../../models/account.model'; +import { ResponseDto } from '../../../models/response.dto'; + +@Injectable({ + providedIn: 'root' +}) +export class CompanyService { + + constructor(private http: HttpService) {} + + // Subsidiaries + getAllSubsidiaries(): Observable { + return this.http.get(`${environment.accountService}/company/subsidiaries`).pipe( + map((response: ResponseDto) => response.data as SubsidiaryDTO[]) + ); + } + + saveSubsidiary(subsidiary: SubsidiaryDTO): Observable { + const requestPayload: Request = { + data: subsidiary, + compressed: true, + target: 'cygnus.models.account.Subsidiary' + }; + return this.http.post(`${environment.accountService}/company/subsidiaries`, requestPayload).pipe( + map((response: ResponseDto) => response.data as SubsidiaryDTO) + ); + } + + activateDeactivateSubsidiary(subsidiaryId: string, active: boolean): Observable { + const subsidiary: SubsidiaryDTO = ({ + id: subsidiaryId, + active: active + }); + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: subsidiary, + compressed: true, + target: 'cygnus.models.account.Subsidiary' + }; + return this.http.post(`${environment.accountService}/company/subsidiaries`, requestPayload).pipe( + map((response: ResponseDto) => response.data as SubsidiaryDTO) + ); + } + + // Departments + getAllDepartments(): Observable { + return this.http.get(`${environment.accountService}/company/departments`).pipe( + map((response: ResponseDto) => response.data as DepartmentDTO[]) + ); + } + + saveDepartment(department: DepartmentDTO): Observable { + const requestPayload: Request = { + data: department, + compressed: true, + target: 'cygnus.models.account.Department' + }; + return this.http.post(`${environment.accountService}/company/departments`, requestPayload).pipe( + map((response: ResponseDto) => response.data as DepartmentDTO) + ); + } + + activateDeactivateDepartment(departmentId: string, active: boolean): Observable { + const department: DepartmentDTO = ({ + id: departmentId, + active: active + } as DepartmentDTO); + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: department, + compressed: true, + target: 'cygnus.models.account.Department' + }; + return this.http.post(`${environment.accountService}/company/departments`, requestPayload).pipe( + map((response: ResponseDto) => response.data as DepartmentDTO) + ); + } + + // Designations + getAllDesignations(): Observable { + return this.http.get(`${environment.accountService}/company/designations`).pipe( + map((response: ResponseDto) => response.data as DesignationDTO[]) + ); + } + + saveDesignation(designation: DesignationDTO): Observable { + const requestPayload: Request = { + data: designation, + compressed: true, + target: 'cygnus.models.account.Designation' + }; + return this.http.post(`${environment.accountService}/company/designations`, requestPayload).pipe( + map((response: ResponseDto) => response.data as DesignationDTO) + ); + } + + activateDeactivateDesignation(designationId: string, active: boolean): Observable { + const designation: DesignationDTO = ({ + id: designationId, + active: active + } as DesignationDTO); + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: designation, + compressed: true, + target: 'cygnus.models.account.Designation' + }; + return this.http.post(`${environment.accountService}/company/designations`, requestPayload).pipe( + map((response: ResponseDto) => response.data as DesignationDTO) + ); + } + + // Employees + getAllEmployees(): Observable { + return this.http.get(`${environment.accountService}/company/employees`).pipe( + map((response: ResponseDto) => response.data as EmployeeDTO[]) + ); + } + + saveEmployee(employee: EmployeeDTO): Observable { + const requestPayload: Request = { + data: employee, + compressed: true, + target: 'cygnus.models.account.Employee' + }; + return this.http.post(`${environment.accountService}/company/employees`, requestPayload).pipe( + map((response: ResponseDto) => response.data as EmployeeDTO) + ); + } + + activateDeactivateEmployee(employeeId: string, active: boolean): Observable { + const employee: EmployeeDTO = ({ + id: employeeId, + active: active + } as EmployeeDTO); + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: employee, + compressed: true, + target: 'cygnus.models.account.Employee' + }; + return this.http.post(`${environment.accountService}/company/employees`, requestPayload).pipe( + map((response: ResponseDto) => response.data as EmployeeDTO) + ); + } + + searchEmployees(payload: Request): Observable { + return this.http.post(`${environment.accountService}/company/employees/search`, payload).pipe( + map((response: ResponseDto) => response.data as any[]) + ); + } + + // Verifiers + getAllVerifiers(): Observable { + return this.http.get(`${environment.accountService}/company/verifiers`).pipe( + map((response: ResponseDto) => response.data as CompanyVerifierModel[]) + ); + } + + saveVerifier(verifier: CompanyVerifierModel): Observable { + const requestPayload: Request = { + data: verifier, + compressed: true, + target: 'cygnus.models.account.CompanyVerifier' + }; + return this.http.post(`${environment.accountService}/company/verifiers`, requestPayload).pipe( + map((response: ResponseDto) => response.data as CompanyVerifierModel) + ); + } + + activateDeactivateVerifier(verifierId: string, active: boolean): Observable { + const verifier: any = ({ + id: verifierId, + active: active + }); + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: verifier, + compressed: true, + target: 'cygnus.models.account.CompanyVerifier' + }; + return this.http.post(`${environment.accountService}/company/verifiers`, requestPayload).pipe( + map((response: ResponseDto) => response.data as CompanyVerifierModel) + ); + } + + searchVerifiers(payload: Request): Observable { + return this.http.post(`${environment.accountService}/company/verifiers/search`, payload).pipe( + map((response: ResponseDto) => response.data as any[]) + ); + } +} \ No newline at end of file diff --git a/frontend/src/app/services/account/user/user.service.ts b/frontend/src/app/services/account/user/user.service.ts new file mode 100644 index 0000000..5dde5c6 --- /dev/null +++ b/frontend/src/app/services/account/user/user.service.ts @@ -0,0 +1,67 @@ +import { Injectable } from '@angular/core'; +import { Observable } from 'rxjs'; +import { map } from 'rxjs/operators'; +import { environment } from '../../../../environments/environment'; +import { HttpService } from '../../http.service'; +import { UserDTO } from '../../../models/user.model'; +import { ResponseDto } from '../../../models/response.dto'; +import { Request } from '../../../models/request.model'; + +@Injectable({ + providedIn: 'root' +}) +export class UserService { + + constructor(private http: HttpService) { } + + getAllUsers(): Observable { + return this.http.get(`${environment.userService}/users`).pipe( + map((response: ResponseDto) => response.data as UserDTO[]) + ); + } + + searchUser(payload: Request): Observable { + return this.http.post(`${environment.userService}/users/search`, payload).pipe( + map((response: ResponseDto) => response.data as UserDTO) + ); + } + + getAllRoles(): Observable { + return this.http.get(`${environment.userService}/roles`).pipe( + map((response: any) => response.data as any[]) + ); + } + + getAllBranches(): Observable { + return this.http.get(`${environment.userService}/branches`).pipe( + map((response: any) => response.data as any[]) + ); + } + + saveUser(user: UserDTO): Observable { + const requestPayload: Request = { + data: user, + compressed: true, + target: 'cygnus.models.user.User' + }; + return this.http.post(`${environment.userService}/users`, requestPayload).pipe( + map((response: ResponseDto) => response.data as UserDTO) + ); + } + + activateDeactivateUser(userId: string, active: boolean): Observable { + const user: UserDTO = { + id: userId, + active: active + }; + const requestPayload: Request = { + scopes: [active ? "ACTIVATE" : "DEACTIVATE"], + data: user, + compressed: true, + target: 'cygnus.models.user.User' + }; + return this.http.post(`${environment.userService}/users`, requestPayload).pipe( + map((response: ResponseDto) => response.data as UserDTO) + ); + } +} \ No newline at end of file diff --git a/frontend/src/app/services/account/vendor/vendor.service.ts b/frontend/src/app/services/account/vendor/vendor.service.ts new file mode 100644 index 0000000..f610157 --- /dev/null +++ b/frontend/src/app/services/account/vendor/vendor.service.ts @@ -0,0 +1,67 @@ +import { Injectable } from '@angular/core'; +import { HttpClient } from '@angular/common/http'; +import { Observable } from 'rxjs'; +import { VendorDTO, VendorBranchDTO } from '../../../models/account.model'; +import { SearchDTO } from '../../../models/masters/masters'; +import { Request } from '../../../models/request.model'; +import { environment } from '../../../../environments/environment'; + +@Injectable({ + providedIn: 'root' +}) +export class VendorService { + + private apiUrl = `${environment.accountService}/vendor/vendors`; + + constructor(private http: HttpClient) { } + + getAllVendors(): Observable { + return this.http.get(this.apiUrl); + } + + saveVendor(vendor: VendorDTO): Observable { + return this.http.post(this.apiUrl, vendor); + } + + searchVendors(searchDTO: SearchDTO): Observable { + return this.http.post(`${this.apiUrl}/search`, searchDTO); + } + + getVendorBranches(vendorId: string): Observable { + return this.http.get(`${this.apiUrl}/${vendorId}/branches`); + } + + saveBranch(branch: VendorBranchDTO): Observable { + return this.http.post(`${this.apiUrl}/${branch.fkVendorId}/branches`, branch); + } + + deleteBranch(vendorId: string, branchId: string): Observable { + return this.http.delete(`${this.apiUrl}/${vendorId}/branches/${branchId}`); + } + + activateDeactivateVendor(id: string, active: boolean): Observable { + // Assuming backend follows similar pattern where update is used for everything or specific endpoint exists. + // Based on Subsidiary example, toggleActive uses a similar approach or custom endpoint. + // If not specific endpoint, fetching, changing active, and saving might be the way, + // OR if backend supports partial update. + // Re-reading frontend_integration.md: + // Create or Update Vendor is POST / + // So to activate/deactivate, we might need to send the object with updated status. + // BUT subsidiary used: this.subsidiaryService.activateDeactivateSubsidiary(subsidiary.id, isActivating) + // Let's assume a similar endpoint might be needed or we use the saveVendor for now if logic is inside. + // Actually, SubsidiaryService (CompanyService) likely has a specific method. + // I will implemented based on common patterns, if specific endpoint is missing in docs, I'll use save with updated status logic in component, + // OR add a specific endpoint if I can infer it exists. + // Let's look at Subsidiary implementation again. + // Subsidiary component calls: this.subsidiaryService.activateDeactivateSubsidiary(subsidiary.id, isActivating) + // I should probably check CompanyService to see how it does it. + // For now, I'll add a method that effectively does a save (or consistent with backend if I knew). + // The integration doc didn't list a specific activate/deactivate endpoint, just Create/Update. + // So I will assume Update (POST /) handles it. + // But verify with Subsidiary service... + // API: POST /cygnus/app/api/v1/account/vendor/vendors + // I'll stick to saveVendor for now. If I need a specific one, I'll add it. + // Wait, let me check CompanyService to see what activateDeactivateSubsidiary does. + return this.http.post(`${this.apiUrl}/${id}/activate-deactivate`, { active }); + } +} diff --git a/frontend/src/app/services/commons/session.service.ts b/frontend/src/app/services/commons/session.service.ts new file mode 100644 index 0000000..7a4ef2b --- /dev/null +++ b/frontend/src/app/services/commons/session.service.ts @@ -0,0 +1,51 @@ +import { Injectable } from '@angular/core'; +import { Router } from '@angular/router'; + +@Injectable({ + providedIn: 'root' +}) +export class SessionService { + + constructor(private router: Router) { } + + setItem(key: string, value: T): void { + try { + sessionStorage.setItem(key, JSON.stringify(value)); + } catch (e) { + console.error(`Error saving ${key} to sessionStorage`, e); + } + } + + getDetails(key: string): T | null { + const details = this.getItem('details'); + if (details === null) return null; + const value = details[key]; + if (value === null) return null; + else return value; + } + + getItem(key: string): any | null { + const json = sessionStorage.getItem(key); + if (json === null) return null; + + try { + return JSON.parse(json); + } catch (e) { + console.warn(`Error parsing JSON for key ${key}`, e); + return null; + } + } + + removeItem(key: string): void { + sessionStorage.removeItem(key); + } + + clear(): void { + sessionStorage.clear(); + } + + logout(): void { + sessionStorage.clear(); + this.router.navigate(['/']); + } +} diff --git a/frontend/src/app/services/http.service.ts b/frontend/src/app/services/http.service.ts new file mode 100644 index 0000000..5b4f125 --- /dev/null +++ b/frontend/src/app/services/http.service.ts @@ -0,0 +1,56 @@ +import { Injectable } from '@angular/core'; +import { HttpClient, HttpHeaders, HttpErrorResponse } from '@angular/common/http'; +import { Observable, throwError } from 'rxjs'; +import { catchError } from 'rxjs/operators'; +import { ResponseDto } from '../models/response.dto'; + +@Injectable({ + providedIn: 'root' +}) +export class HttpService { + + constructor(private http: HttpClient) { + } + + get(url: string, headers?: HttpHeaders): Observable { + const options = { + headers, + withCredentials: true + }; + return this.http.get(url, options).pipe( + catchError(this.handleError) + ); + } + + post(url: string, body: any, headers?: HttpHeaders): Observable { + const options = { + headers, + withCredentials: true + }; + return this.http.post(url, body, options).pipe( + catchError(this.handleError) + ); + } + + put(url: string, body: any, headers?: HttpHeaders): Observable { + return this.http.put(url, body, { headers }).pipe( + catchError(this.handleError) + ); + } + + delete(url: string, headers?: HttpHeaders): Observable { + return this.http.delete(url, { headers }).pipe( + catchError(this.handleError) + ); + } + + private handleError(error: HttpErrorResponse) { + return throwError(() => error.error); + } + + setHeaders(additionalHeaders: { [key: string]: string }): HttpHeaders { + return new HttpHeaders({ + ...additionalHeaders + }); + } +} \ No newline at end of file diff --git a/frontend/src/app/services/masters/master.service.ts b/frontend/src/app/services/masters/master.service.ts new file mode 100644 index 0000000..f9f283f --- /dev/null +++ b/frontend/src/app/services/masters/master.service.ts @@ -0,0 +1,24 @@ +import { Injectable } from '@angular/core'; +import { HttpService } from '../http.service'; +import { CityDTO } from '../../models/masters/masters'; +import { map, Observable } from 'rxjs'; +import { environment } from '../../../environments/environment'; +import { ResponseDto } from '../../models/response.dto'; +import { Request } from '../../models/request.model'; + +@Injectable({ + providedIn: 'root' +}) +export class MasterService { + private readonly baseUrl = '/master'; + + constructor(private http: HttpService) {} + + searchCityStates(payload: Request): Observable { + return this.http.post(`${environment.masterService}/cities-states/search`, payload).pipe( + map((response: ResponseDto) => response.data as CityDTO[]) + ); + } + + +} diff --git a/frontend/src/app/services/ola/ola.service.ts b/frontend/src/app/services/ola/ola.service.ts new file mode 100644 index 0000000..0c73517 --- /dev/null +++ b/frontend/src/app/services/ola/ola.service.ts @@ -0,0 +1,24 @@ +import { Injectable } from '@angular/core'; +import { Observable } from 'rxjs'; +import { map } from 'rxjs/operators'; +import { environment } from '../../../environments/environment'; +import { HttpService } from '../http.service'; +import { ResponseDto } from '../../models/response.dto'; + +@Injectable({ + providedIn: 'root' +}) +export class OlaService { + + constructor(private http: HttpService) {} + + autocomplete(query: string): Observable { + // Assuming backend takes get parameter 'text' or something similar. + // Wait, the prompt says @GetMapping(AUTOCOMPLETE) on /ola/location/api/v1/autocomplete + // Let's pass query directly as a parameter like ?input=query + return this.http.get(`${environment.masterService}/ola-autocomplete?input=${encodeURIComponent(query)}`).pipe( + map((response: ResponseDto) => response.data) + ); + } + +} diff --git a/frontend/src/app/services/template.service.ts b/frontend/src/app/services/template.service.ts new file mode 100644 index 0000000..6d18ba7 --- /dev/null +++ b/frontend/src/app/services/template.service.ts @@ -0,0 +1,103 @@ +import { Injectable } from '@angular/core'; +import { HttpClient } from '@angular/common/http'; +import { Observable } from 'rxjs'; +import { map } from 'rxjs/operators'; +import { environment } from '../../environments/environment'; + +export interface DocumentLayout { + pk_document_data_id: number; + fk_document_id: number; + page_no: number; + text_value: string; + block_type: string; + parent_block_id?: number; + x_coordinate: number; + y_coordinate: number; + width: number; + height: number; + confidence: number; + sequence_no: number; + page_width: number; + page_height: number; +} + +export interface TemplateField { + pk_template_field_id?: number; + field_label: string; + field_type: string; + display_order: number; + required_flag: boolean; +} + +export interface Template { + pk_template_id: number; + template_name: string; + fields: TemplateField[]; +} + +@Injectable({ + providedIn: 'root' +}) +export class TemplateService { + private apiUrl = environment.docEngineService; + + constructor(private http: HttpClient) {} + + uploadDocument(file: File): Observable { + const formData = new FormData(); + formData.append('file', file); + return this.http.post(`${this.apiUrl}/documents/upload`, formData); + } + + getDocumentLayout(documentId: string): Observable { + return this.http.get(`${this.apiUrl}/documents/${documentId}`).pipe( + map(doc => { + const layouts: DocumentLayout[] = []; + if (doc.pages) { + doc.pages.forEach((page: any) => { + if (page.text_blocks) { + page.text_blocks.forEach((tb: any) => { + layouts.push({ + pk_document_data_id: tb.id, + fk_document_id: doc.id, + page_no: page.page_number, + text_value: tb.text, + block_type: tb.block_type, + x_coordinate: tb.x, + y_coordinate: tb.y, + width: tb.width, + height: tb.height, + confidence: tb.confidence || 0, + sequence_no: tb.sequence || 0, + page_width: page.width || 1000, + page_height: page.height || 1000 + }); + }); + } + }); + } + return layouts; + }) + ); + } + + createTemplate(payload: any): Observable { + return this.http.post(`${this.apiUrl}/templates`, payload); + } + + updateTemplate(templateId: string, payload: any): Observable { + return this.http.put(`${this.apiUrl}/templates/${templateId}`, payload); + } + + saveMappings(templateId: string, mappings: any[]): Observable { + return this.http.post(`${this.apiUrl}/templates/${templateId}/mappings/save`, mappings); + } + + getTemplate(templateId: string): Observable { + return this.http.get(`${this.apiUrl}/templates/${templateId}`); + } + + recognizeTemplate(documentId: string): Observable { + return this.http.post(`${this.apiUrl}/templates/match`, { document_id: documentId }); + } +} diff --git a/frontend/src/app/services/tools/allocation/allocation.service.ts b/frontend/src/app/services/tools/allocation/allocation.service.ts new file mode 100644 index 0000000..118c2fe --- /dev/null +++ b/frontend/src/app/services/tools/allocation/allocation.service.ts @@ -0,0 +1,73 @@ +import { Request } from './../../../models/request.model'; +import { Injectable } from '@angular/core'; +import { Observable } from 'rxjs'; +import { map } from 'rxjs/operators'; +import { environment } from '../../../../environments/environment'; +import { HttpService } from '../../http.service'; + +import { AreaDTO, LocalityDTO, LocalityTypeDTO } from '../../../models/tools.model'; +import { ResponseDto } from '../../../models/response.dto'; + +@Injectable({ + providedIn: 'root' +}) +export class AllocationService { + + constructor(private http: HttpService) {} + + // Areas + getAllAreas(): Observable { + return this.http.get(`${environment.toolsService}/areas`).pipe( + map((response: ResponseDto) => response.data as AreaDTO[]) + ); + } + + saveArea(area: AreaDTO): Observable { + const requestPayload: Request = { + data: area, + compressed: false, + target: 'cygnus.models.tools.Area' + }; + return this.http.post(`${environment.toolsService}/areas`, requestPayload).pipe( + map((response: ResponseDto) => response.data as AreaDTO) + ); + } + + searchAreas(payload: Request): Observable { + return this.http.post(`${environment.toolsService}/areas/search`, payload).pipe( + map((response: ResponseDto) => response.data as any[]) + ); + } + + // Localities + getAllLocalities(): Observable { + return this.http.get(`${environment.toolsService}/localities`).pipe( + map((response: ResponseDto) => response.data as LocalityDTO[]) + ); + } + + saveLocality(locality: LocalityDTO): Observable { + const requestPayload: Request = { + data: locality, + compressed: false, + target: 'cygnus.models.tools.Locality' + }; + return this.http.post(`${environment.toolsService}/localities`, requestPayload).pipe( + map((response: ResponseDto) => response.data as LocalityDTO) + ); + } + + searchLocalities(payload: Request): Observable { + return this.http.post(`${environment.toolsService}/localities/search`, payload).pipe( + map((response: ResponseDto) => response.data as any[]) + ); + } + + // Locality Types + getAllLocalityTypes(): Observable { + return this.http.get(`${environment.toolsService}/locality-types`).pipe( + map((response: ResponseDto) => response.data as LocalityTypeDTO[]) + ); + } + +} diff --git a/frontend/src/app/services/utilities/encryption.service.ts b/frontend/src/app/services/utilities/encryption.service.ts new file mode 100644 index 0000000..ecae54e --- /dev/null +++ b/frontend/src/app/services/utilities/encryption.service.ts @@ -0,0 +1,61 @@ +import { Injectable } from '@angular/core'; +import { environment } from '../../../environments/environment'; + +@Injectable({ + providedIn: 'root' +}) +export class EncryptionService { + + constructor() {} + + async encrypt(payload: any): Promise { + const encoder = new TextEncoder(); + const iv = crypto.getRandomValues(new Uint8Array(12)); + const jsonString = JSON.stringify(payload); + + const keyBytes = Uint8Array.from(atob(environment.encryptionKey), c => c.charCodeAt(0)); + const cryptoKey = await crypto.subtle.importKey( + "raw", + keyBytes, + { name: "AES-GCM" }, + false, + ["encrypt"] + ); + + const encrypted = await crypto.subtle.encrypt( + { name: "AES-GCM", iv }, + cryptoKey, + encoder.encode(jsonString) + ); + + const combined = new Uint8Array(iv.length + encrypted.byteLength); + combined.set(iv, 0); + combined.set(new Uint8Array(encrypted), iv.length); + + return btoa(String.fromCharCode(...combined)); + } + + async decrypt(base64Cipher: string): Promise { + const data = Uint8Array.from(atob(base64Cipher), c => c.charCodeAt(0)); + const iv = data.slice(0, 12); + const ciphertext = data.slice(12); + + const keyBytes = Uint8Array.from(atob(environment.encryptionKey), c => c.charCodeAt(0)); + const cryptoKey = await crypto.subtle.importKey( + "raw", + keyBytes, + { name: "AES-GCM" }, + false, + ["decrypt"] + ); + + const decrypted = await crypto.subtle.decrypt( + { name: "AES-GCM", iv }, + cryptoKey, + ciphertext + ); + + const decoder = new TextDecoder(); + return JSON.parse(decoder.decode(decrypted)); + } +} \ No newline at end of file diff --git a/frontend/src/app/services/utilities/rsa.service.ts b/frontend/src/app/services/utilities/rsa.service.ts new file mode 100644 index 0000000..a88ff21 --- /dev/null +++ b/frontend/src/app/services/utilities/rsa.service.ts @@ -0,0 +1,37 @@ +import { Injectable } from '@angular/core'; +import { environment } from '../../../environments/environment'; + +@Injectable({ + providedIn: 'root', +}) +export class RsaService { + constructor() {} + + private async importPublicKey(pemKey: string): Promise { + const pem = pemKey + .replace('-----BEGIN PUBLIC KEY-----', '') + .replace('-----END PUBLIC KEY-----', '') + .replace(/\s+/g, ''); + const binaryDer = Uint8Array.from(atob(pem), c => c.charCodeAt(0)); + + return crypto.subtle.importKey( + 'spki', + binaryDer.buffer, + { + name: 'RSA-OAEP', + hash: 'SHA-256', + }, + false, + ['encrypt'] + ); + } + + async encrypt(plaintext: string): Promise { + const key = await this.importPublicKey(environment.rsaPublicKey); + const encoded = new TextEncoder().encode(plaintext); + + const encrypted = await crypto.subtle.encrypt({ name: 'RSA-OAEP' }, key, encoded); + + return btoa(String.fromCharCode(...new Uint8Array(encrypted))); + } +} diff --git a/frontend/src/app/services/utilities/validation.service.ts b/frontend/src/app/services/utilities/validation.service.ts new file mode 100644 index 0000000..2eacb14 --- /dev/null +++ b/frontend/src/app/services/utilities/validation.service.ts @@ -0,0 +1,167 @@ +import { Injectable } from '@angular/core'; +import { AbstractControl, ValidationErrors, ValidatorFn } from '@angular/forms'; + +@Injectable({ + providedIn: 'root' +}) +export class ValidationService { + + // Regex patterns + static readonly PATTERNS = { + EMAIL: /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/, + INDIAN_MOBILE: /^[6-9]\d{9}$/, + PINCODE: /^\d{6}$/, + PAN: /^[A-Z]{5}[0-9]{4}[A-Z]{1}$/, + CIN: /^[UL][0-9]{5}[A-Z]{2}[0-9]{4}[A-Z]{3}[0-9]{6}$/, + MSME: /^UDYAM-[A-Z]{2}-\d{7}$/, + ALPHANUMERIC: /^[a-zA-Z0-9]+$/, + ALPHABET_WITH_SPACES: /^[a-zA-Z0-9\s.&'-]+$/, + CODE: /^[A-Z0-9]{2,10}$/ + }; + + // Error messages + static readonly ERROR_MESSAGES = { + REQUIRED: (fieldName: string) => { + const readableName = fieldName.replace(/([A-Z])/g, ' $1').trim(); + const titleCaseName = readableName.charAt(0).toUpperCase() + readableName.slice(1); + // special cases for common acronyms + const finalName = titleCaseName.replace(/\bId\b/g, 'ID'); + return `${finalName} is required`; + }, + EMAIL_INVALID: 'Please enter a valid email address', + MOBILE_INVALID: 'Please enter a valid 10-digit mobile number starting with 6-9', + PINCODE_INVALID: 'Please enter a valid 6-digit pincode', + PAN_INVALID: 'Please enter a valid PAN number (e.g., ABCDE1234F)', + CIN_INVALID: 'Please enter a valid CIN number', + MSME_INVALID: 'Please enter a valid MSME number (e.g., UDYAM-XX-XXXXXXX)', + CODE_INVALID: 'Code must be 2-10 alphanumeric characters', + NAME_INVALID: 'Name must be at least 2 characters and contain only letters, numbers, spaces, dots, ampersands, apostrophes, and hyphens', + CONTACT_PERSON_INVALID: 'Contact person name must be at least 2 characters', + MIN_LENGTH: (min: number) => `Minimum ${min} characters required`, + MAX_LENGTH: (max: number) => `Maximum ${max} characters allowed` + }; + + // Custom validators + static emailValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.EMAIL.test(control.value); + return isValid ? null : { invalidEmail: true }; + }; + } + + static mobileValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.INDIAN_MOBILE.test(control.value); + return isValid ? null : { invalidMobile: true }; + }; + } + + static pincodeValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.PINCODE.test(control.value); + return isValid ? null : { invalidPincode: true }; + }; + } + + static panValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.PAN.test(control.value); + return isValid ? null : { invalidPan: true }; + }; + } + + static cinValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.CIN.test(control.value); + return isValid ? null : { invalidCin: true }; + }; + } + + static msmeValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.MSME.test(control.value); + return isValid ? null : { invalidMsme: true }; + }; + } + + static codeValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + const isValid = this.PATTERNS.CODE.test(control.value); + return isValid ? null : { invalidCode: true }; + }; + } + + static nameValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + if (control.value.length < 2) return { minLength: { requiredLength: 2, actualLength: control.value.length } }; + const isValid = this.PATTERNS.ALPHABET_WITH_SPACES.test(control.value); + return isValid ? null : { invalidName: true }; + }; + } + + static contactPersonValidator(): ValidatorFn { + return (control: AbstractControl): ValidationErrors | null => { + if (!control.value) return null; + return control.value.length >= 2 ? null : { minLength: { requiredLength: 2, actualLength: control.value.length } }; + }; + } + + // Get error message for a control + static getErrorMessage(control: AbstractControl, fieldName: string): string { + if (!control.errors) return ''; + + if (control.errors['required']) { + return this.ERROR_MESSAGES.REQUIRED(fieldName); + } + + if (control.errors['invalidEmail']) { + return this.ERROR_MESSAGES.EMAIL_INVALID; + } + + if (control.errors['invalidMobile']) { + return this.ERROR_MESSAGES.MOBILE_INVALID; + } + + if (control.errors['invalidPincode']) { + return this.ERROR_MESSAGES.PINCODE_INVALID; + } + + if (control.errors['invalidPan']) { + return this.ERROR_MESSAGES.PAN_INVALID; + } + + if (control.errors['invalidCin']) { + return this.ERROR_MESSAGES.CIN_INVALID; + } + + if (control.errors['invalidMsme']) { + return this.ERROR_MESSAGES.MSME_INVALID; + } + + if (control.errors['invalidCode']) { + return this.ERROR_MESSAGES.CODE_INVALID; + } + + if (control.errors['invalidName']) { + return this.ERROR_MESSAGES.NAME_INVALID; + } + + if (control.errors['minlength']) { + return this.ERROR_MESSAGES.MIN_LENGTH(control.errors['minlength'].requiredLength); + } + + if (control.errors['maxlength']) { + return this.ERROR_MESSAGES.MAX_LENGTH(control.errors['maxlength'].requiredLength); + } + + return 'Invalid input'; + } +} diff --git a/frontend/src/app/templates/templates.component.html b/frontend/src/app/templates/templates.component.html new file mode 100644 index 0000000..de20857 --- /dev/null +++ b/frontend/src/app/templates/templates.component.html @@ -0,0 +1,144 @@ + +
+ + + + +
+
+

Document Preview

+
+ + +
+
+ +
+
+ +

Upload a document to view its extracted layout structure.

+
+ + +
+
+ +
+
+
+
+ + Analyzing Document Layout... +
+

Extracting text, tables, and regions

+
+ +
+
Page {{ page.page_number }}
+ +
+ +
+ + +
{{ node.block_type }}
+
{{ node.text_value }}
+
+
+
+
+
+ + + + +
+
+
+ + + + +
+ +
+
+
+ +
+
+

Add template fields to start mapping.

+
+ +
+
+
+
+ {{ field.field_label }} + {{ field.field_type }} +
+ +
+ +
+ +
+ Drop value here... +
+ +
+ + {{ mappedNode.text_value }} + +
+
+
+
+
+ +
+ +
+
+ + + +
+ + + +
+
+ + +
+
+ + +
+
+ + + + + diff --git a/frontend/src/app/templates/templates.component.scss b/frontend/src/app/templates/templates.component.scss new file mode 100644 index 0000000..5e17d3d --- /dev/null +++ b/frontend/src/app/templates/templates.component.scss @@ -0,0 +1,317 @@ +.template-mapping-container { + height: calc(100vh - 100px); + padding: 1rem; + background-color: var(--surface-ground); +} + +.panel-container { + display: flex; + flex-direction: column; + height: calc(100% - 1rem); + width: calc(100% - 1rem); + margin: 0.5rem; + background: var(--surface-card); + overflow: hidden; + border-radius: 8px; + box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06); +} + +.header-section { + padding: 1rem; + border-bottom: 1px solid var(--surface-border); + display: flex; + justify-content: space-between; + align-items: center; + + h2 { + margin: 0; + font-size: 1.25rem; + font-weight: 600; + } +} + +.preview-area, .mapping-area { + flex: 1; + overflow-y: auto; + padding: 1rem; + background-color: var(--surface-ground); +} + +.empty-state { + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + height: 100%; + color: var(--text-color-secondary); + text-align: center; +} + +.document-canvas { + position: relative; + width: 100%; + background: white; + box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06); + border-radius: 4px; + overflow: hidden; +} + +/* Draggable Nodes */ +.layout-node { + position: absolute; + box-sizing: content-box !important; + border: 6px solid transparent; + margin: -6px !important; + background-clip: padding-box; + background: rgba(255, 255, 255, 0.9); + box-shadow: inset 0 0 0 1px rgba(0, 0, 0, 0.15); + border-radius: 4px; + transition: box-shadow 0.2s, border-color 0.2s; + overflow: hidden; + resize: both; + display: flex; + flex-direction: column; + justify-content: center; + min-width: 60px; + min-height: 25px; + padding: 4px 6px; + + &:active { + cursor: grabbing; + } + + .node-type { + font-size: 0.5rem; + color: var(--text-color-secondary); + text-transform: uppercase; + font-weight: 600; + margin-bottom: 0.1rem; + line-height: 1; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + } + + .node-text { + font-size: 0.65rem; + color: var(--text-color); + line-height: 1.2; + white-space: normal; + word-break: break-word; + } + + /* Color coding by block type - use borders now instead of thick left border */ + &.header { box-shadow: inset 0 0 0 1.5px #3B82F6; } + &.vendor { box-shadow: inset 0 0 0 1.5px #8B5CF6; } + &.table_header { box-shadow: inset 0 0 0 1.5px #F59E0B; } + &.total { box-shadow: inset 0 0 0 1.5px #10B981; } + &.tax { box-shadow: inset 0 0 0 1.5px #EF4444; } + + &:hover { + z-index: 10; + &.header { box-shadow: inset 0 0 0 2px #3B82F6, 0 4px 8px rgba(59, 130, 246, 0.2); } + &.vendor { box-shadow: inset 0 0 0 2px #8B5CF6, 0 4px 8px rgba(139, 92, 246, 0.2); } + &.table_header { box-shadow: inset 0 0 0 2px #F59E0B, 0 4px 8px rgba(245, 158, 11, 0.2); } + &.total { box-shadow: inset 0 0 0 2px #10B981, 0 4px 8px rgba(16, 185, 129, 0.2); } + &.tax { box-shadow: inset 0 0 0 2px #EF4444, 0 4px 8px rgba(239, 68, 68, 0.2); } + } + + .drag-handle { + position: absolute; + top: 0px; + left: 0px; + font-size: 0.55rem; + color: var(--text-color-secondary); + background: white; + border-radius: 50%; + width: 14px; + height: 14px; + display: flex; + align-items: center; + justify-content: center; + cursor: grab; + opacity: 0; + transition: opacity 0.2s, transform 0.2s; + z-index: 25; + box-shadow: 0 1px 3px rgba(0,0,0,0.2); + border: 1px solid var(--surface-border); + + &:active { + cursor: grabbing; + } + + &:hover { + color: var(--primary-color); + transform: scale(1.1); + } + } + + &:hover .drag-handle { + opacity: 1; + } + + .close-icon { + position: absolute; + top: 0px; + right: 0px; + font-size: 0.55rem; + color: white; + background: #EF4444; + border-radius: 50%; + width: 14px; + height: 14px; + display: flex; + align-items: center; + justify-content: center; + cursor: pointer; + opacity: 0; + transition: opacity 0.2s, transform 0.2s; + z-index: 25; + box-shadow: 0 1px 3px rgba(0,0,0,0.3); + + &:hover { + background: #DC2626; + transform: scale(1.1); + } + } + + &:hover .close-icon { + opacity: 1; + } +} + +/* Template Mapping Side */ +.template-field-container { + background: white; + border: 1px solid var(--surface-border); + border-radius: 6px; + margin-bottom: 1rem; + overflow: hidden; + + .field-header { + background: var(--surface-ground); + padding: 0.75rem 1rem; + border-bottom: 1px solid var(--surface-border); + display: flex; + justify-content: space-between; + align-items: center; + + .badge { + font-size: 0.7rem; + padding: 0.2rem 0.5rem; + background: var(--primary-color); + color: var(--primary-color-text); + border-radius: 12px; + } + } + + .drop-zone { + padding: 1rem; + min-height: 60px; + + .placeholder { + color: var(--text-color-secondary); + font-style: italic; + font-size: 0.85rem; + } + } +} + +.mapped-node { + background: var(--primary-50); + color: var(--primary-900); + padding: 0.5rem; + border-radius: 4px; + font-size: 0.9rem; + border: 1px solid var(--primary-200); + display: flex; + align-items: center; +} + +/* CDK Drag Drop */ +.cdk-drag-preview { + box-sizing: border-box; + border-radius: 4px; + box-shadow: 0 5px 5px -3px rgba(0, 0, 0, 0.2), + 0 8px 10px 1px rgba(0, 0, 0, 0.14), + 0 3px 14px 2px rgba(0, 0, 0, 0.12); +} + +.cdk-drag-placeholder { + opacity: 0.3; +} + +.cdk-drag-animating { + transition: transform 250ms cubic-bezier(0, 0, 0.2, 1); +} + +/* Scanning Overlay Animation */ +.scanning-overlay { + position: absolute; + top: 0; + left: 0; + right: 0; + bottom: 0; + background: rgba(255, 255, 255, 0.9); + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + z-index: 100; +} + +.scanner-container { + position: relative; + width: 120px; + height: 120px; + display: flex; + align-items: center; + justify-content: center; + border-radius: 8px; + overflow: hidden; + background: #f8f9fa; + box-shadow: inset 0 0 10px rgba(0,0,0,0.05); + border: 1px solid #e9ecef; +} + +.scanner-doc { + z-index: 1; +} + +.laser-beam { + position: absolute; + top: 0; + left: 0; + width: 100%; + height: 3px; + background: var(--primary-color, #3b82f6); + box-shadow: 0 0 10px 2px var(--primary-color, #3b82f6); + z-index: 2; + animation: scan-laser 2s ease-in-out infinite alternate; +} + +.scanner-grid { + position: absolute; + top: 0; + left: 0; + width: 100%; + height: 100%; + background-image: linear-gradient(var(--primary-100, #dbeafe) 1px, transparent 1px), + linear-gradient(90deg, var(--primary-100, #dbeafe) 1px, transparent 1px); + background-size: 10px 10px; + z-index: 0; + opacity: 0.5; + animation: scan-grid 4s linear infinite; +} + +@keyframes scan-laser { + 0% { top: -5%; opacity: 0; } + 10% { opacity: 1; } + 90% { opacity: 1; } + 100% { top: 105%; opacity: 0; } +} + +@keyframes scan-grid { + 0% { background-position: 0 0; } + 100% { background-position: 20px 20px; } +} diff --git a/frontend/src/app/templates/templates.component.ts b/frontend/src/app/templates/templates.component.ts new file mode 100644 index 0000000..4e32d38 --- /dev/null +++ b/frontend/src/app/templates/templates.component.ts @@ -0,0 +1,394 @@ +import { Component, OnInit } from '@angular/core'; +import { CommonModule } from '@angular/common'; +import { FormsModule } from '@angular/forms'; +import { DragDropModule, CdkDragDrop, moveItemInArray, transferArrayItem, copyArrayItem } from '@angular/cdk/drag-drop'; +import { SplitterModule } from 'primeng/splitter'; +import { ButtonModule } from 'primeng/button'; +import { DialogModule } from 'primeng/dialog'; +import { InputTextModule } from 'primeng/inputtext'; +import { DropdownModule } from 'primeng/dropdown'; +import { TableModule } from 'primeng/table'; +import { ToastModule } from 'primeng/toast'; +import { MessageService } from 'primeng/api'; + +import { TemplateService, DocumentLayout, TemplateField } from '../services/template.service'; + +@Component({ + selector: 'app-templates', + standalone: true, + imports: [ + CommonModule, + FormsModule, + DragDropModule, + SplitterModule, + ButtonModule, + DialogModule, + InputTextModule, + DropdownModule, + TableModule, + ToastModule + ], + providers: [MessageService], + templateUrl: './templates.component.html', + styleUrls: ['./templates.component.scss'] +}) +export class TemplatesComponent implements OnInit { + // Document State + documentId: string | null = null; + layoutNodes: DocumentLayout[] = []; + pages: { page_number: number, nodes: DocumentLayout[], width: number, height: number }[] = []; + pageWidth: number = 800; + pageHeight: number = 1100; + + // Template State + templateName: string = ''; + templateFields: TemplateField[] = []; + mappings: { [fieldId: number]: DocumentLayout[] } = {}; + + // UI State + displayAddField: boolean = false; + isScanning: boolean = false; + scanProgress: number = 0; + currentTemplateId: string | null = null; + newField: TemplateField = { field_label: '', field_type: 'TEXT', display_order: 0, required_flag: false }; + fieldTypes = [ + { label: 'TEXT', value: 'TEXT' }, + { label: 'NUMBER', value: 'NUMBER' }, + { label: 'DATE', value: 'DATE' }, + { label: 'DATETIME', value: 'DATETIME' }, + { label: 'AMOUNT', value: 'AMOUNT' }, + { label: 'ADDRESS', value: 'ADDRESS' }, + { label: 'TABLE_COLUMN', value: 'TABLE_COLUMN' } + ]; + + constructor( + private templateService: TemplateService, + private messageService: MessageService + ) {} + + ngOnInit(): void {} + + onFileUpload(event: any) { + const file = event.target.files[0]; + if (file) { + this.currentTemplateId = null; + this.templateName = ''; + this.templateFields = []; + this.mappings = {}; + + this.templateService.uploadDocument(file).subscribe({ + next: (res) => { + this.documentId = res.id || res.pk_document_id; + this.messageService.add({ severity: 'success', summary: 'Uploaded', detail: 'Document uploaded successfully.' }); + this.isScanning = true; + this.fetchLayout(); + }, + error: (err) => { + this.messageService.add({ severity: 'error', summary: 'Error', detail: 'Upload failed.' }); + } + }); + } + } + + fetchLayout() { + if (!this.documentId) return; + this.templateService.getDocumentLayout(this.documentId).subscribe({ + next: (layouts) => { + if (layouts.length === 0) { + // Document might still be processing in the background Celery worker + // Poll again after 2 seconds + setTimeout(() => this.fetchLayout(), 2000); + return; + } + + this.isScanning = false; + this.layoutNodes = layouts; + + const pageMap = new Map(); + + layouts.forEach(node => { + if (!pageMap.has(node.page_no)) { + pageMap.set(node.page_no, { + page_number: node.page_no, + nodes: [], + width: node.page_width, + height: node.page_height + }); + } + pageMap.get(node.page_no)!.nodes.push(node); + }); + + this.pages = Array.from(pageMap.values()).sort((a, b) => a.page_number - b.page_number); + + if (this.pages.length > 0) { + this.pageWidth = this.pages[0].width; + this.pageHeight = this.pages[0].height; + } + // Trigger auto-recognition + this.autoRecognize(); + } + }); + } + + autoRecognize() { + if (!this.documentId) return; + this.templateService.recognizeTemplate(this.documentId).subscribe({ + next: (matches) => { + if (matches && matches.length > 0 && matches[0].confidence_score >= 0.75) { + const match = matches[0]; + this.messageService.add({ severity: 'info', summary: 'Template Recognized', detail: `Confidence: ${(match.confidence_score * 100).toFixed(1)}%` }); + + this.templateService.getTemplate(match.format_id).subscribe({ + next: (templateData) => { + this.templateName = templateData.name; + this.currentTemplateId = match.format_id; + this.templateFields = []; + this.mappings = {}; + + if (templateData.cells) { + templateData.cells.forEach((cell: any) => { + if (cell.is_dynamic) { + const fieldId = new Date().getTime() + Math.random(); + this.templateFields.push({ + pk_template_field_id: fieldId as any, + field_label: cell.field_name, + field_type: cell.data_type, + display_order: cell.sequence, + required_flag: false + }); + this.mappings[fieldId] = []; + } + }); + } + + if (templateData.regions) { + templateData.regions.forEach((region: any) => { + if (region.region_type === 'field_mapping' && region.content && region.content.field_name) { + const field = this.templateFields.find(f => f.field_label === region.content.field_name); + if (field && field.pk_template_field_id) { + // Find the corresponding page in the current document + const page = this.pages.find(p => p.page_number === region.page_number); + if (page) { + let bestMatchIdx = -1; + let bestOverlap = 0; + + // Find the text node on the canvas that overlaps most with this region's bounding box + for (let i = 0; i < page.nodes.length; i++) { + const node = page.nodes[i]; + const x_overlap = Math.max(0, Math.min(region.x + region.width, node.x_coordinate + node.width) - Math.max(region.x, node.x_coordinate)); + const y_overlap = Math.max(0, Math.min(region.y + region.height, node.y_coordinate + node.height) - Math.max(region.y, node.y_coordinate)); + const overlapArea = x_overlap * y_overlap; + + if (overlapArea > bestOverlap) { + bestOverlap = overlapArea; + bestMatchIdx = i; + } + } + + // If we found a matching node (at least some overlap), clone it to mappings + if (bestMatchIdx !== -1) { + const matchedNode = page.nodes[bestMatchIdx]; + + // Because *ngFor tracks objects by reference, we clone it + this.mappings[field.pk_template_field_id].push({...matchedNode}); + } + } + } + } + }); + } + } + }); + } + } + }); + } + + showAddFieldDialog() { + this.newField = { field_label: '', field_type: 'TEXT', display_order: this.templateFields.length, required_flag: false }; + this.displayAddField = true; + } + + addField() { + if (!this.newField.field_label) return; + this.newField.pk_template_field_id = new Date().getTime(); // mock ID until saved + this.templateFields.push({ ...this.newField }); + this.mappings[this.newField.pk_template_field_id] = []; + this.displayAddField = false; + } + + removeField(fieldId?: number) { + if (!fieldId) return; + this.templateFields = this.templateFields.filter(f => f.pk_template_field_id !== fieldId); + delete this.mappings[fieldId]; + } + + saveTemplateAndMappings() { + if (!this.templateName || this.templateName.trim() === '') { + this.messageService.add({ severity: 'warn', summary: 'Warning', detail: 'Please provide a template name.' }); + return; + } + + if (this.templateFields.length === 0) { + this.messageService.add({ severity: 'warn', summary: 'Warning', detail: 'Please add at least one template field.' }); + return; + } + + const payload = { + template_name: this.templateName, + source_document_id: this.documentId || undefined, + fields: this.templateFields + }; + + const saveRequest = this.currentTemplateId + ? this.templateService.updateTemplate(this.currentTemplateId, payload) + : this.templateService.createTemplate(payload); + + saveRequest.subscribe({ + next: (res: any) => { + const templateId = res.pk_template_id; + + // Construct the mappings payload + const mappingsPayload: any[] = []; + this.templateFields.forEach(field => { + const fieldId = field.pk_template_field_id; + if (fieldId && this.mappings[fieldId] && this.mappings[fieldId].length > 0) { + mappingsPayload.push({ + field_name: field.field_label, + mapped_nodes: this.mappings[fieldId] + }); + } + }); + + if (mappingsPayload.length > 0) { + this.templateService.saveMappings(templateId, mappingsPayload).subscribe({ + next: () => { + this.messageService.add({ severity: 'success', summary: 'Success', detail: 'Template & Mappings saved successfully.' }); + }, + error: (err) => { + this.messageService.add({ severity: 'error', summary: 'Error', detail: 'Failed to save mappings.' }); + } + }); + } else { + this.messageService.add({ severity: 'success', summary: 'Success', detail: 'Template saved successfully (No mappings).' }); + } + }, + error: (err) => { + this.messageService.add({ severity: 'error', summary: 'Error', detail: 'Failed to save template.' }); + } + }); + } + + // Drag and Drop Logic + drop(event: CdkDragDrop, fieldId?: number) { + if (event.previousContainer === event.container) { + // Do nothing! This allows the item to naturally snap back to its original position + // since it's a failed drop (didn't land in a mapping field). + } else { + const isFromCanvas = event.previousContainer.id.startsWith('document-layout-list'); + const isToCanvas = event.container.id.startsWith('document-layout-list'); + + if (isFromCanvas && !isToCanvas) { + // Drag from Canvas -> Field + // We use event.item.data which perfectly tracks the dragged object regardless of DOM indexes + const clonedNode = JSON.parse(JSON.stringify(event.item.data)); + + // Sanitize the value if dropped on an AMOUNT or NUMBER field + if (fieldId) { + const field = this.templateFields.find(f => f.pk_template_field_id === fieldId); + if (field && (field.field_type === 'AMOUNT' || field.field_type === 'NUMBER') && clonedNode.text_value) { + const numericText = clonedNode.text_value.replace(/[^\d\.-]/g, ''); + const floatValue = parseFloat(numericText); + if (!isNaN(floatValue)) { + clonedNode.text_value = floatValue.toString(); + } else { + clonedNode.text_value = ''; + } + } else if (field && (field.field_type === 'DATE' || field.field_type === 'DATETIME') && clonedNode.text_value) { + // Attempt to parse the date and output standard format + const timestamp = Date.parse(clonedNode.text_value); + if (!isNaN(timestamp)) { + const parsedDate = new Date(timestamp); + // Extract YYYY-MM-DD + const yyyy = parsedDate.getFullYear(); + const mm = String(parsedDate.getMonth() + 1).padStart(2, '0'); + const dd = String(parsedDate.getDate()).padStart(2, '0'); + if (field.field_type === 'DATE') { + clonedNode.text_value = `${yyyy}-${mm}-${dd}`; + } else { + const hh = String(parsedDate.getHours()).padStart(2, '0'); + const min = String(parsedDate.getMinutes()).padStart(2, '0'); + const ss = String(parsedDate.getSeconds()).padStart(2, '0'); + clonedNode.text_value = `${yyyy}-${mm}-${dd}T${hh}:${min}:${ss}`; + } + } else { + // If we can't parse it reliably on frontend, we leave it or let backend handle it + // We'll leave it as is to give user visual feedback, backend parser is more robust (dateutil) + } + } + } + + // Insert the clone into the destination mapping field + event.container.data.splice(event.currentIndex, 0, clonedNode); + + // Force Angular to completely recreate the DOM elements for this specific canvas page. + // By using .map(node => ({...node})), we change every object's identity. + // This forces Angular to destroy the corrupted DOM element (which CDK moved and left a translate3d on) + // and recreate it fresh with its original absolute coordinates! + const pageIndex = parseInt(event.previousContainer.id.split('-').pop() || '0'); + if (!isNaN(pageIndex) && this.pages[pageIndex]) { + this.pages[pageIndex].nodes = this.pages[pageIndex].nodes.map(node => ({...node})); + + // Also explicitly clear the transform on the dragged element just in case CDK holds a ref to it + event.item.element.nativeElement.style.transform = ''; + } + + } else if (!isFromCanvas && isToCanvas) { + // Drag from Field -> Canvas (Delete from Field) + event.previousContainer.data.splice(event.previousIndex, 1); + } else { + // Drag from Field -> Field (Move) + transferArrayItem( + event.previousContainer.data, + event.container.data, + event.previousIndex, + event.currentIndex, + ); + } + } + } + + onNodeMouseUp(node: DocumentLayout, event: MouseEvent) { + // If the user resized the node using the native CSS resize handle, save the new size + const el = event.currentTarget as HTMLElement; + if (el.style.width && el.style.width.endsWith('px')) { + const parentRect = el.parentElement!.getBoundingClientRect(); + node.width = (el.offsetWidth / parentRect.width) * node.page_width; + node.height = (el.offsetHeight / parentRect.height) * node.page_height; + + // Clear the inline pixel styles so Angular bindings take over smoothly + el.style.width = ''; + el.style.height = ''; + } + } + + getConnectedDropLists() { + // Return all field ID strings as drop lists + return this.templateFields.map(f => 'field-' + f.pk_template_field_id); + } + + getCanvasDropLists() { + return this.pages.map((_, i) => 'document-layout-list-' + i); + } + + removeNode(pageIndex: number, nodeIndex: number, event: Event) { + event.stopPropagation(); + this.pages[pageIndex].nodes.splice(nodeIndex, 1); + } + + removeFromField(fieldId: number, nodeIndex: number) { + if (this.mappings[fieldId]) { + this.mappings[fieldId].splice(nodeIndex, 1); + } + } +} diff --git a/frontend/src/app/templates/templates.html b/frontend/src/app/templates/templates.html new file mode 100644 index 0000000..d2ecc93 --- /dev/null +++ b/frontend/src/app/templates/templates.html @@ -0,0 +1 @@ +

templates works!

diff --git a/frontend/src/app/templates/templates.scss b/frontend/src/app/templates/templates.scss new file mode 100644 index 0000000..e69de29 diff --git a/frontend/src/app/templates/templates.ts b/frontend/src/app/templates/templates.ts new file mode 100644 index 0000000..3efa25f --- /dev/null +++ b/frontend/src/app/templates/templates.ts @@ -0,0 +1,11 @@ +import { Component } from '@angular/core'; + +@Component({ + selector: 'app-templates', + imports: [], + templateUrl: './templates.html', + styleUrl: './templates.scss', +}) +export class Templates { + +} diff --git a/frontend/src/environments/environment.ts b/frontend/src/environments/environment.ts new file mode 100644 index 0000000..a6c0265 --- /dev/null +++ b/frontend/src/environments/environment.ts @@ -0,0 +1,11 @@ +export const environment = { + production: false, + encryptionKey: btoa('1234567890123456'), + authService: 'http://192.168.0.111:1700/cygnus/app/api/v1', + accountService: 'http://localhost:1701/cygnus/app/api/v1/account', + userService: 'http://localhost:1701/cygnus/app/api/v1/user', + masterService: 'http://localhost:1702/cygnus/app/api/v1/master', + toolsService: 'http://localhost:1703/cygnus/app/api/v1/tools', + docEngineService: 'http://localhost:7989/api/v1', + rsaPublicKey: `MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq3RFV/f6ybsOF2m7NBLPUTMBq9b0frJG1HdIDYmrD9Wr1/aGBxTSJwq8IHFlatNpBF3OlJv9uEOybWMM1vXli4IgsuPPmcTOZsQ/O/9UGyBSL6apevNCw6pC1oa0MVLaN6COMAhDr+ri/PYiPQUcYsjDqghmAghMk99umHGUihz/oY/qgxzO+Q9cqePmjpH5c5RaXGBrOQxKoPlm7Uj6MqAfBhLC360VbcMot4XDoV+VeQXMzH0o6e870jdClsLOq1VsCA27jVvafj+HwaJ15ny9UWWilDuS/X8Sd7v+Rmd+qNezi6ROcglyaisXwKfeTWM8/o7HiUco2fL230+jEQIDAQAB` +}; \ No newline at end of file diff --git a/frontend/src/styles.scss b/frontend/src/styles.scss index ed3cf92..dd146dc 100644 --- a/frontend/src/styles.scss +++ b/frontend/src/styles.scss @@ -1,17 +1,191 @@ -/* @import "primeng/resources/themes/aura-light-noir/theme.css"; */ -/* @import "primeng/resources/primeng.min.css"; */ -/* PrimeNG v18+ handles themes differently (often via Tailwind or presets). - For this prototype, we will rely on default component styles or add a CDN link if needed for quick styling. - However, PrimeIcons is still valid. */ -/* Quill Editor Styles */ -@import "quill/dist/quill.core.css"; -@import "quill/dist/quill.snow.css"; - +/* Global Styles from Cygnus-UI */ @import "primeicons/primeicons.css"; +@import 'primeflex/primeflex.css'; -html, body { - margin: 0; - font-family: var(--font-family); - background-color: var(--surface-ground); - height: 100%; + +@layer primeng, primeng-overrides; + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Thin.ttf') format('truetype'); + font-weight: 100; + font-style: normal; } + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-ThinItalic.ttf') format('truetype'); + font-weight: 100; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Light.ttf') format('truetype'); + font-weight: 300; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-LightItalic.ttf') format('truetype'); + font-weight: 300; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Regular.ttf') format('truetype'); + font-weight: 400; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Italic.ttf') format('truetype'); + font-weight: 400; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Medium.ttf') format('truetype'); + font-weight: 500; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-MediumItalic.ttf') format('truetype'); + font-weight: 500; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-SemiBold.ttf') format('truetype'); + font-weight: 600; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-SemiBoldItalic.ttf') format('truetype'); + font-weight: 600; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Bold.ttf') format('truetype'); + font-weight: 700; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-BoldItalic.ttf') format('truetype'); + font-weight: 700; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-ExtraBold.ttf') format('truetype'); + font-weight: 800; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-ExtraBoldItalic.ttf') format('truetype'); + font-weight: 800; + font-style: italic; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-Black.ttf') format('truetype'); + font-weight: 900; + font-style: normal; +} + +@font-face { + font-family: 'Roboto'; + src: url('/assets/fonts/roboto/Roboto-BlackItalic.ttf') format('truetype'); + font-weight: 900; + font-style: italic; +} + +body { + font-family: 'Roboto', sans-serif; + background-color: rgba(226,232,240,0.4); + margin: 0px 2px; +} + +.fs-07{ + font-size: 0.7rem; +} +.fs-08{ + font-size: 0.8rem; +} +.fs-09{ + font-size: 0.9rem; +} +.fs-1{ + font-size: 1rem; +} +.fs-11{ + font-size: 1.1rem; +} +.fs-12{ + font-size: 1.2rem; +} +.fs-13{ + font-size: 1.3rem; +} +.fs-14{ + font-size: 1.4rem; +} +.fs-15{ + font-size: 1.5rem; +} + + +/* Overrides */ +.p-message .p-message-text { + font-size: 0.8rem; +} +.p-message-content { + justify-content: center; +} + +.p-floatlabel label{ + font-size: 0.9rem; + font-weight: normal !important; +} + +@layer primeng-overrides { + /* Compact table styles */ + .p-datatable .p-datatable-tbody > tr > td { + font-size: 0.875rem; + padding: 0.4rem 0.5rem !important; + } + + .p-datatable .p-datatable-thead > tr > th { + font-size: 0.875rem; + padding: 0.75rem 0.75rem !important; + background-color: rgba(241, 245, 249, 1); + } + + .p-autocomplete-item { + font-size: 0.875rem; + } +} + +.p-fieldset .p-fieldset-legend { + /*background: var(--p-orange-500);*/ + background: rgba(245,115,22,0.9); + color: white; +} + diff --git a/frontend/tsconfig.json b/frontend/tsconfig.json index ad457fa..54beb76 100644 --- a/frontend/tsconfig.json +++ b/frontend/tsconfig.json @@ -3,7 +3,7 @@ { "compileOnSave": false, "compilerOptions": { - "strict": true, + "strict": false, "noImplicitOverride": true, "noPropertyAccessFromIndexSignature": true, "noImplicitReturns": true, diff --git a/frontend_integration.md b/frontend_integration.md new file mode 100644 index 0000000..10e0934 --- /dev/null +++ b/frontend_integration.md @@ -0,0 +1,74 @@ +# Vendor and Vendor Branch Integration Guide + +This document outlines the API endpoints and TypeScript interfaces to help integrate the Vendor and Vendor Branch features into your frontend project. + +## 1. API Endpoints + +**Base URL**: `/cygnus/app/api/v1/account/vendor/vendors` + +| Method | Endpoint | Description | Payload | +|---|---|---|---| +| POST | `/` | Create or Update Vendor | `VendorDTO` | +| GET | `/` | Get All Vendors | (Query Params) | +| POST | `/search` | Search Vendors | `SearchDTO` | +| GET | `/{id}/branches` | Get Branches for Vendor | - | +| POST | `/{id}/branches` | Create or Update Branch | `VendorBranchDTO` | +| DELETE | `/{id}/branches/{branchId}` | Delete Branch | - | + +## 2. TypeScript Interfaces + +### Vendor + +```typescript +export interface VendorDTO { + id?: string; + companyId?: string; // Encrypted Company ID + code: string; + name: string; + panNo?: string; + cinNo?: string; + msmeNo?: string; + createdUser?: string; + updatedAt?: string; // ISO Date String + updatedUser?: string; + active: boolean; + branches?: VendorBranchDTO[]; +} +``` + +### Vendor Branch + +```typescript +export interface VendorBranchDTO { + id?: string; + fkVendorId?: string; // Encrypted Vendor ID + branchCode: string; + branchName: string; + officeNo?: string; + street?: string; + locality?: string; + cityId?: string; // Encrypted City ID + stateId?: string; // Encrypted State ID + stateName?: string; + cityName?: string; + pinCode?: string; + emailId?: string; + contactNo?: string; + contactPerson?: string; + gstNo?: string; + createdUser?: string; + updatedAt?: string; // ISO Date String + updatedUser?: string; + active: boolean; +} +``` + +## 3. Usage Notes + +- **IDs**: All IDs (`id`, `fkVendorId`, `companyId`, `cityId`, `stateId`) are strings and represent encrypted values. +- **Dates**: `updatedAt` is returned as an ISO string. +- **Search**: The search endpoint uses `SearchDTO` (likely existing in your frontend common types) to handle filters and pagination. + +## Context Transfer +To reference this backend implementation context in another conversation, mention the **Conversation ID**: +`496255c5-4ca5-4e1d-908f-c9d456a6dfb8` diff --git a/list_templates.py b/list_templates.py new file mode 100644 index 0000000..9a577f1 --- /dev/null +++ b/list_templates.py @@ -0,0 +1,11 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import DocumentFormat + +db = SessionLocal() +templates = db.query(DocumentFormat).all() +for t in templates: + print(f"ID: {t.id}, Name: {t.name}, Source: {t.source_document_id}, Active: {t.is_active}") diff --git a/ocr.backup b/ocr.backup new file mode 100644 index 0000000..86497a7 Binary files /dev/null and b/ocr.backup differ diff --git a/print_cells.py b/print_cells.py new file mode 100644 index 0000000..f2ba3b5 --- /dev/null +++ b/print_cells.py @@ -0,0 +1,14 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import DocumentFormat + +db = SessionLocal() +t = db.query(DocumentFormat).filter(DocumentFormat.is_active == True).first() +if t: + print(f"Template Name: {t.name}") + print(f"Number of cells: {len(t.cells)}") + for c in t.cells: + print(f"Cell: x={c.x}, y={c.y}, w={c.width}, h={c.height}") diff --git a/print_fp.py b/print_fp.py new file mode 100644 index 0000000..80eec56 --- /dev/null +++ b/print_fp.py @@ -0,0 +1,13 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import DocumentFormat + +db = SessionLocal() +t = db.query(DocumentFormat).filter(DocumentFormat.is_active == True).first() +if t: + print(f"Template Name: {t.name}, Fingerprint is None: {t.fingerprint is None}") +else: + print("No active templates.") diff --git a/print_fp2.py b/print_fp2.py new file mode 100644 index 0000000..1917ab2 --- /dev/null +++ b/print_fp2.py @@ -0,0 +1,18 @@ +import sys +import os +import json +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import TemplateFingerprint +from app.models.template import DocumentFormat + +db = SessionLocal() +t = db.query(DocumentFormat).filter(DocumentFormat.is_active == True).first() +if t: + fp = db.query(TemplateFingerprint).filter(TemplateFingerprint.format_id == t.id).first() + if fp: + print(f"Cell Coords items: {len(fp.cell_coordinates.get('items', [])) if fp.cell_coordinates else 'None'}") + print(json.dumps(fp.cell_coordinates, indent=2)) + else: + print("No TemplateFingerprint row found!") diff --git a/prompt.txt b/prompt.txt new file mode 100644 index 0000000..285247b --- /dev/null +++ b/prompt.txt @@ -0,0 +1,856 @@ +# PROJECT: TEMPLATE-ENGINE + +You are a senior solution architect, senior Python backend architect, senior Angular architect, OCR/Document AI specialist, PostgreSQL database architect, and enterprise software engineer. + +Generate a COMPLETE production-ready application named: + +template-engine + +The solution must contain: + +1. Angular (latest stable version) frontend using PrimeNG. +2. Python backend using FastAPI. +3. PostgreSQL database. +4. SQL migration scripts. +5. Clean architecture. +6. Production-grade code. +7. Proper logging. +8. Exception handling. +9. DTOs / Schemas. +10. Repository pattern. +11. Service layer. +12. API documentation. +13. Unit-test ready structure. +14. Docker support should NOT be generated now. +15. Deployment configuration should NOT be generated now. + +--- + +# BUSINESS REQUIREMENT + +The application is a Template Mapping Engine for invoices, purchase orders, delivery challans and similar business documents. + +The goal is: + +1. User uploads document. +2. System extracts document layout and text. +3. Layout preview is rendered. +4. User creates template fields. +5. User maps extracted document fields to template fields using drag and drop. +6. Mapping is stored. +7. When same vendor format arrives again, system automatically recognizes template and auto-fills mappings. + +--- + +# TECHNOLOGY STACK + +Frontend: + +* Angular +* PrimeNG +* Angular CDK Drag and Drop +* RxJS +* Standalone Components +* SCSS + +Backend: + +* Python 3.12+ +* FastAPI +* SQLAlchemy +* Alembic +* Pydantic +* PostgreSQL + +Document Processing: + +* pdfplumber +* pymupdf (fitz) +* pytesseract +* opencv +* pillow +* layoutparser +* numpy + +Optional AI Layer: + +* sentence-transformers +* scikit-learn + +--- + +# HIGH LEVEL MODULES + +1. Document Upload Module +2. OCR Module +3. Layout Extraction Module +4. Preview Rendering Module +5. Template Management Module +6. Mapping Module +7. Template Recognition Engine +8. Auto Mapping Engine + +--- + +# STEP 1 – FRONTEND DESIGN + +Create a page: + +/template-mapping + +The page must be split vertically. + +--- + +| | | +| LEFT PANEL | RIGHT PANEL | +| DOCUMENT PREVIEW | TEMPLATE MAPPING | +| | | +----------------------------------------------------- + +Width: + +* Left 60% +* Right 40% + +Use PrimeNG Splitter. + +--- + +LEFT PANEL + +Header section: + +Title: +Document Preview + +Buttons: + +Upload Document + +Supported: + +* PDF +* PNG +* JPG +* JPEG +* TIFF + +After upload: + +Show extracted layout in tabular preview. + +Important: + +DO NOT display raw OCR text. + +Render layout structure. + +Example: + +Invoice Header + +Invoice Number +INV001 + +Invoice Date +01-Jan-2026 + +Vendor Name +ABC Industries + +--- + +Line Items + +| Item | Qty | Rate | Amount | + +--- + +Tax Section + +--- + +Total Section + +Maintain visual hierarchy. + +Styling is not important. + +Layout preservation is important. + +Every text block should be selectable. + +Every text block should be draggable. + +Store unique layout node id. + +--- + +RIGHT PANEL + +Header: + +Template Name [textbox] + +Add Field Button + +Create Template Button + +--- + +When user clicks Add Field: + +Open PrimeNG Dialog. + +Fields: + +Field Label + +Field Type + +Supported Types: + +TEXT +NUMBER +DATE +AMOUNT +ADDRESS +TABLE_COLUMN + +Save + +This creates records in template_fields. + +--- + +Below header show: + +Template Fields Tree/Grid + +Example: + +Vendor Name + +Invoice Number + +Invoice Date + +GST Number + +Item Name + +Qty + +Rate + +Tax % + +Tax Amount + +Grand Total + +Each field is droppable. + +--- + +Bottom: + +Save Mapping Button + +--- + +# STEP 2 – DOCUMENT UPLOAD, EXTRACTION AND DATABASE STORAGE + +When file uploaded: + +Detect type: + +PDF +IMAGE + +--- + +PDF PROCESSING + +If text PDF: + +Use pdfplumber + +Extract: + +* text +* coordinates +* page +* bounding box + +--- + +SCANNED PDF + +Convert pages to image. + +Run OCR. + +Use: + +pytesseract + +Extract: + +* text +* coordinates + +--- + +IMAGE PROCESSING + +Run: + +OpenCV preprocessing + +* grayscale +* denoise +* threshold + +Run OCR. + +Extract: + +* text +* coordinates + +--- + +For every extracted item store: + +text + +x + +y + +width + +height + +page_no + +block_type + +parent_block + +sequence + +confidence + +--- + +Create logical blocks: + +HEADER + +VENDOR + +BILL_TO + +SHIP_TO + +TABLE + +TABLE_ROW + +TABLE_CELL + +TAX + +TOTAL + +FOOTER + +--- + +Store everything in database. + +--- + +# STEP 3 – RENDER LAYOUT PREVIEW + +Backend returns: + +DocumentLayoutResponse + +Example: + +{ +documentId, +pages:[ +] +} + +Layout must preserve: + +* hierarchy +* coordinates +* parent child relationships + +Frontend converts response into preview tree. + +Render: + +Header + +Vendor + +Line Items + +Tax + +Total + +Footer + +Every node: + +draggable + +--- + +# STEP 4 – TEMPLATE MANAGEMENT + +Template Creation Flow + +User enters: + +Template Name + +Click Create Template + +Create record in templates. + +--- + +Add Field Dialog + +Create rows in template_fields. + +Fields: + +field_label + +field_type + +display_order + +required_flag + +created_at + +--- + +Support unlimited fields. + +Support edit/delete field. + +--- + +# STEP 5 – DRAG AND DROP MAPPING + +Use Angular CDK. + +Drag Source: + +Document preview nodes. + +Drop Target: + +Template fields. + +--- + +Mapping UI: + +Vendor Name +← ABC Industries + +Invoice Number +← INV-1001 + +Invoice Date +← 2026-01-01 + +GST Number +← 27ABCDE1234F1Z5 + +--- + +Save Mapping + +Backend stores: + +template_fields_mapping + +Include: + +document coordinates + +page number + +bounding box + +confidence + +layout path + +parent section + +--- + +# STEP 6 – TEMPLATE RECOGNITION ENGINE + +Requirement: + +When same vendor document arrives again: + +System automatically identifies template. + +--- + +Create template fingerprint. + +Store: + +Vendor Name + +Header Positions + +Table Header Names + +Relative Coordinates + +Document Structure + +--- + +Generate fingerprint hash. + +Store in database. + +--- + +Recognition Strategy + +Level 1 + +Vendor Name Match + +Level 2 + +Header Similarity + +Level 3 + +Layout Similarity + +Level 4 + +Coordinate Similarity + +--- + +Use weighted score. + +Example: + +Vendor Match = 40% + +Header Match = 20% + +Layout Match = 20% + +Coordinate Match = 20% + +Threshold: + +85% + +If matched: + +Auto apply template. + +--- + +# AUTO MAPPING ENGINE + +After template recognized: + +Find mapped coordinates. + +Extract values from same coordinate zones. + +Populate template fields automatically. + +Return: + +{ +templateMatched:true, +templateId:1, +confidence:96.5, +extractedFields:[ +] +} + +Frontend should immediately show populated template values. + +User may modify and save again. + +--- + +# DATABASE DESIGN + +Create schema: + +templates + +--- + +TABLE documents + +pk_document_id BIGSERIAL PK + +document_name VARCHAR(500) + +document_type VARCHAR(100) + +file_name VARCHAR(500) + +file_hash VARCHAR(500) + +page_count INTEGER + +status VARCHAR(50) + +created_at TIMESTAMP + +updated_at TIMESTAMP + +--- + +TABLE document_layout + +pk_document_data_id BIGSERIAL PK + +fk_document_id BIGINT + +page_no INTEGER + +text_value TEXT + +block_type VARCHAR(100) + +parent_block_id BIGINT + +x_coordinate NUMERIC + +y_coordinate NUMERIC + +width NUMERIC + +height NUMERIC + +confidence NUMERIC + +sequence_no INTEGER + +layout_path TEXT + +created_at TIMESTAMP + +--- + +TABLE templates + +pk_template_id BIGSERIAL PK + +template_name VARCHAR(255) + +template_fingerprint TEXT + +active_flag BOOLEAN + +created_at TIMESTAMP + +updated_at TIMESTAMP + +--- + +TABLE template_fields + +pk_template_field_id BIGSERIAL PK + +fk_template_id BIGINT + +field_label VARCHAR(255) + +field_type VARCHAR(100) + +display_order INTEGER + +required_flag BOOLEAN + +created_at TIMESTAMP + +--- + +TABLE template_fields_mapping + +pk_mapping_id BIGSERIAL PK + +fk_template_id BIGINT + +fk_template_field_id BIGINT + +fk_document_data_id BIGINT + +page_no INTEGER + +x_coordinate NUMERIC + +y_coordinate NUMERIC + +width NUMERIC + +height NUMERIC + +mapping_confidence NUMERIC + +layout_path TEXT + +created_at TIMESTAMP + +--- + +TABLE template_recognition_history + +pk_history_id BIGSERIAL PK + +fk_template_id BIGINT + +fk_document_id BIGINT + +recognition_score NUMERIC + +matched_flag BOOLEAN + +created_at TIMESTAMP + +--- + +Generate complete SQL scripts. + +Generate Alembic migrations. + +Generate indexes. + +Generate foreign keys. + +Generate constraints. + +--- + +# BACKEND APIS + +POST /api/documents/upload + +GET /api/documents/{id} + +GET /api/documents/{id}/layout + +POST /api/templates + +PUT /api/templates/{id} + +DELETE /api/templates/{id} + +POST /api/template-fields + +PUT /api/template-fields/{id} + +DELETE /api/template-fields/{id} + +POST /api/mappings/save + +POST /api/templates/recognize + +POST /api/templates/auto-map + +--- + +# PROJECT STRUCTURE + +Generate complete folder structure. + +Frontend structure. + +Backend structure. + +Models. + +Repositories. + +Services. + +Controllers. + +DTOs. + +Validation. + +Error Handling. + +Logging. + +Configurations. + +Environment files. + +Constants. + +Utilities. + +OCR helpers. + +Template recognition engine. + +Auto mapping engine. + +--- + +# OUTPUT REQUIREMENT + +Generate code step-by-step in the following order: + +1. Complete solution architecture. +2. PostgreSQL schema and SQL scripts. +3. Backend folder structure. +4. Backend implementation. +5. OCR and layout extraction implementation. +6. Template recognition engine. +7. API layer. +8. Angular folder structure. +9. Angular UI implementation. +10. PrimeNG screens. +11. Drag and drop implementation. +12. API integration. +13. Auto mapping implementation. +14. Validation. +15. Testing strategy. + +All generated code must be production-ready, runnable, and complete with no placeholders or TODO comments. + + +Use existing frontend project and enable this route //{ path: 'templates', component: TemplateComponent } and create respective component and add and design our page in this component. + +Use backend project for python logic and related endpoints. database details are available in database.py. use or create new schema "templates" for this task + +Move to next step once I give the confirmation that step 1 is done. \ No newline at end of file diff --git a/prompt_for_session_conitnue.txt b/prompt_for_session_conitnue.txt new file mode 100644 index 0000000..b34b54e --- /dev/null +++ b/prompt_for_session_conitnue.txt @@ -0,0 +1 @@ +Hey, please read the SESSION_HANDOFF.md file in the root of my OCR project to get up to speed on the Template Engine we were building, and let's continue. \ No newline at end of file diff --git a/regenerate_fingerprints.py b/regenerate_fingerprints.py new file mode 100644 index 0000000..0d2feac --- /dev/null +++ b/regenerate_fingerprints.py @@ -0,0 +1,21 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import DocumentFormat +from app.services.fingerprint_service import FingerprintService + +db = SessionLocal() +templates = db.query(DocumentFormat).filter(DocumentFormat.is_active == True).all() + +fp_service = FingerprintService(db) +count = 0 +for t in templates: + # Refresh the relations + db.refresh(t) + fp_service.generate_fingerprint(t) + count += 1 + +db.commit() +print(f"Regenerated fingerprints for {count} active templates.") diff --git a/run_backend.sh b/run_backend.sh new file mode 100755 index 0000000..56742b3 --- /dev/null +++ b/run_backend.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +# Navigate to the backend directory +cd "$(dirname "$0")/backend" || exit + +# Activate the virtual environment +if [ ! -d "venv" ]; then + echo "Creating virtual environment 'venv'..." + python3 -m venv venv +fi + +echo "Activating virtual environment..." +source venv/bin/activate + +# Auto-install dependencies if crucial modules are missing +if ! python -c "import uvicorn, numpy, cv2, pdf2image" &>/dev/null; then + echo "Dependencies missing. Installing python dependencies..." + pip install --upgrade pip --quiet + pip install -r requirements.txt + echo "Dependencies installed ✓" +fi + +# Run the FastAPI server +echo "Starting OCR Backend Server on http://0.0.0.0:8000..." +uvicorn main:app --reload --host 0.0.0.0 --port 8000 diff --git a/run_docengine.sh b/run_docengine.sh new file mode 100755 index 0000000..23bbe24 --- /dev/null +++ b/run_docengine.sh @@ -0,0 +1,285 @@ +#!/usr/bin/env bash +# ============================================================================= +# DocEngine — Backend Startup Script +# ============================================================================= +# Usage: +# ./run_docengine.sh Run the API server (default) +# ./run_docengine.sh server Run the API server only +# ./run_docengine.sh worker Run the Celery worker only +# ./run_docengine.sh all Run both API server and Celery worker +# ./run_docengine.sh setup Install deps + run migrations only +# ./run_docengine.sh migrate Run Alembic migrations only +# ============================================================================= + +set -euo pipefail + +# ── Paths ──────────────────────────────────────────────────────────────────── +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_DIR="${SCRIPT_DIR}/docengine" +VENV_DIR="${PROJECT_DIR}/.venv" +PID_DIR="${PROJECT_DIR}/.pids" + +# ── Colors ─────────────────────────────────────────────────────────────────── +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +CYAN='\033[0;36m' +NC='\033[0m' # No Color + +log() { echo -e "${GREEN}[DocEngine]${NC} $*"; } +warn() { echo -e "${YELLOW}[DocEngine]${NC} $*"; } +err() { echo -e "${RED}[DocEngine]${NC} $*" >&2; } + +# ── Pre-flight checks ─────────────────────────────────────────────────────── +check_python() { + if command -v python3 &>/dev/null; then + PYTHON=python3 + elif command -v python &>/dev/null; then + PYTHON=python + else + err "Python 3 is not installed. Please install Python 3.12+." + exit 1 + fi + + local version + version=$($PYTHON -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') + log "Using Python ${version} (${PYTHON})" +} + +check_redis() { + if command -v redis-cli &>/dev/null; then + if redis-cli ping &>/dev/null; then + log "Redis is running ✓" + else + warn "Redis is installed but not running. Celery worker will fail without Redis." + warn "Start Redis with: brew services start redis (or) redis-server --daemonize yes" + fi + else + warn "redis-cli not found. Celery worker requires Redis." + fi +} + +# ── Virtual Environment ───────────────────────────────────────────────────── +ensure_venv() { + local created=false + if [ ! -d "${VENV_DIR}" ]; then + log "Creating virtual environment at ${VENV_DIR}..." + $PYTHON -m venv "${VENV_DIR}" + log "Virtual environment created ✓" + created=true + fi + # shellcheck disable=SC1091 + source "${VENV_DIR}/bin/activate" + log "Virtual environment activated ✓" + + # Auto-install dependencies if this is a fresh venv or if uvicorn is missing + if [ "${created}" = true ] || ! python -c "import uvicorn" &>/dev/null; then + log "Dependencies missing. Auto-installing dependencies..." + install_deps + fi +} + +# ── Dependencies ───────────────────────────────────────────────────────────── +install_deps() { + log "Installing Python dependencies..." + pip install --upgrade pip --quiet + pip install -r "${PROJECT_DIR}/requirements.txt" --quiet + log "Dependencies installed ✓" +} + +# ── Storage Directories ───────────────────────────────────────────────────── +ensure_storage() { + local dirs=("documents" "templates" "images" "temp" "rendered") + for dir in "${dirs[@]}"; do + mkdir -p "${PROJECT_DIR}/storage/${dir}" + done + log "Storage directories ready ✓" +} + +# ── Database Migrations ───────────────────────────────────────────────────── +run_migrations() { + log "Running Alembic database migrations..." + cd "${PROJECT_DIR}" + if alembic upgrade head 2>/dev/null; then + log "Migrations applied ✓" + else + warn "Alembic migrations failed or no migrations to apply." + warn "If this is a fresh database, ensure PostgreSQL is reachable at the configured host." + fi +} + +# ── PID Management ─────────────────────────────────────────────────────────── +mkdir_pids() { + mkdir -p "${PID_DIR}" +} + +save_pid() { + echo "$2" > "${PID_DIR}/$1.pid" +} + +remove_pid() { + rm -f "${PID_DIR}/$1.pid" +} + +read_pid() { + local pidfile="${PID_DIR}/$1.pid" + if [ -f "${pidfile}" ]; then + cat "${pidfile}" + fi +} + +is_running() { + local pid + pid=$(read_pid "$1") + if [ -n "${pid}" ] && kill -0 "${pid}" 2>/dev/null; then + return 0 + fi + return 1 +} + +# ── Cleanup on exit ───────────────────────────────────────────────────────── +cleanup() { + log "Shutting down..." + local server_pid worker_pid + server_pid=$(read_pid "server") + worker_pid=$(read_pid "worker") + + if [ -n "${server_pid}" ] && kill -0 "${server_pid}" 2>/dev/null; then + log "Stopping API server (PID ${server_pid})..." + kill "${server_pid}" 2>/dev/null || true + remove_pid "server" + fi + + if [ -n "${worker_pid}" ] && kill -0 "${worker_pid}" 2>/dev/null; then + log "Stopping Celery worker (PID ${worker_pid})..." + kill "${worker_pid}" 2>/dev/null || true + remove_pid "worker" + fi + + log "Shutdown complete." +} + +# ── Start API Server ──────────────────────────────────────────────────────── +start_server() { + if is_running "server"; then + warn "API server is already running (PID $(read_pid 'server'))" + return + fi + + cd "${PROJECT_DIR}" + log "Starting API server on port 7989..." + log " Docs: ${CYAN}http://localhost:7989/docs${NC}" + log " ReDoc: ${CYAN}http://localhost:7989/redoc${NC}" + log " Health: ${CYAN}http://localhost:7989/api/v1/health${NC}" + echo "" + + python -m uvicorn app.main:app \ + --host 0.0.0.0 \ + --port 7989 \ + --reload \ + --log-level info & + + local pid=$! + save_pid "server" "${pid}" + log "API server started (PID ${pid}) ✓" +} + +# ── Start Celery Worker ───────────────────────────────────────────────────── +start_worker() { + if is_running "worker"; then + warn "Celery worker is already running (PID $(read_pid 'worker'))" + return + fi + + cd "${PROJECT_DIR}" + log "Starting Celery worker..." + + celery -A app.workers.celery_app worker \ + --loglevel=info \ + --concurrency=4 \ + --pool=prefork \ + -Q docengine_default,document_processing \ + -E & + + local pid=$! + save_pid "worker" "${pid}" + log "Celery worker started (PID ${pid}) ✓" +} + +# ── Setup (no server start) ───────────────────────────────────────────────── +do_setup() { + check_python + ensure_venv + install_deps + ensure_storage + check_redis + run_migrations + log "" + log "Setup complete. Run ${CYAN}./run_docengine.sh${NC} to start the server." +} + +# ── Main ───────────────────────────────────────────────────────────────────── +main() { + local command="${1:-server}" + + case "${command}" in + setup) + do_setup + ;; + migrate) + check_python + ensure_venv + run_migrations + ;; + server) + check_python + ensure_venv + ensure_storage + mkdir_pids + trap cleanup EXIT INT TERM + start_server + wait + ;; + worker) + check_python + ensure_venv + check_redis + mkdir_pids + trap cleanup EXIT INT TERM + start_worker + wait + ;; + all) + check_python + ensure_venv + ensure_storage + check_redis + mkdir_pids + trap cleanup EXIT INT TERM + start_server + start_worker + echo "" + log "═══════════════════════════════════════════════" + log " DocEngine is running" + log " API Server : http://localhost:7989" + log " Swagger UI : http://localhost:7989/docs" + log " Celery : Worker active (4 processes)" + log "═══════════════════════════════════════════════" + log "Press Ctrl+C to stop all services." + echo "" + wait + ;; + *) + echo "Usage: $0 {server|worker|all|setup|migrate}" + echo "" + echo " server Start the FastAPI server (default)" + echo " worker Start the Celery worker" + echo " all Start both server and worker" + echo " setup Install dependencies and run migrations" + echo " migrate Run database migrations only" + exit 1 + ;; + esac +} + +main "$@" diff --git a/testing_strategy.md b/testing_strategy.md new file mode 100644 index 0000000..7bb82e9 --- /dev/null +++ b/testing_strategy.md @@ -0,0 +1,55 @@ +# Testing Strategy: Template Mapping Engine + +This document outlines the testing strategy for the Template Mapping Engine to ensure production readiness. + +## 1. Backend Testing (Python/FastAPI) + +### Unit Testing +* **Framework**: `pytest` +* **Database**: Use an in-memory SQLite database (`sqlite:///:memory:`) or a dedicated test PostgreSQL container via `testcontainers`. +* **Mocks**: + * Mock OCR engines (`pytesseract`, `pdfplumber`) to avoid slow I/O during test suites. + * Mock file uploads using `fastapi.testclient.TestClient`. +* **Coverage Targets**: + * **Engines**: 100% logic coverage for `TemplateRecognitionEngine` and scoring weights. + * **Services**: 90% coverage for CRUD operations. + +### Integration Testing +* Test end-to-end API flows: + 1. `POST /api/documents/upload` with a sample PDF. + 2. Wait for layout extraction. + 3. `POST /api/templates` to create a template. + 4. `POST /api/templates/{id}/mappings/save` to map extracted data. + 5. `POST /api/documents/upload` with a similar document to verify `POST /api/documents/{id}/recognize` returns the correct template match. + +## 2. Frontend Testing (Angular) + +### Unit Testing +* **Framework**: Jasmine & Karma (or Jest if configured). +* **Component Tests**: + * Verify `TemplatesComponent` renders the left and right panels. + * Verify the `AddField` dialog toggles correctly and validates empty inputs. +* **Service Tests**: + * Mock `HttpClient` using `HttpTestingController` to ensure `TemplateService` sends correct payloads to the backend. + +### E2E / Integration Testing +* **Framework**: Cypress or Playwright. +* **Critical User Journeys (CUJ)**: + 1. User uploads a document, UI displays the layout preview visually. + 2. User creates a template and adds 3 fields. + 3. User drags a block from the Document Preview and drops it into a Template Field. + 4. User saves the mapping successfully. + +## 3. OCR & Layout Extraction Accuracy Testing + +Since the OCR engine relies on visual heuristics rather than AI models, testing its accuracy is crucial to prevent regressions. + +* **Golden Dataset**: Create a dataset of 50-100 real-world business documents (Invoices, POs, Receipts). +* **Evaluation Metric**: Run the `DocumentProcessor` over the Golden Dataset and compare the output bounding boxes and classifications (HEADER, VENDOR, etc.) against manually annotated Ground Truth data. +* **Acceptance Criteria**: Maintain > 92% classification accuracy for logical block types. + +## 4. Performance & Load Testing + +* **Tool**: `locust` or `k6`. +* **Scenario**: Simulate 50 concurrent users uploading 2MB PDF documents simultaneously to ensure the `DocumentProcessor` does not exhaust server memory (OpenCV and Tesseract can be memory-intensive). +* **Optimization**: Ensure the `process_file` logic can be offloaded to Celery workers if the API starts blocking or timing out under load. diff --git a/update_templates.py b/update_templates.py new file mode 100644 index 0000000..cb0bf67 --- /dev/null +++ b/update_templates.py @@ -0,0 +1,18 @@ +import sys +import os +sys.path.append(os.path.join(os.getcwd(), 'docengine')) + +from app.core.database import SessionLocal +from app.models.template import DocumentFormat + +db = SessionLocal() +# Deactivate all templates starting with 'Template_' that have a UUID-like suffix (or just all where description starts with 'Auto-generated') +templates = db.query(DocumentFormat).filter( + DocumentFormat.description.startswith("Auto-generated") +).all() + +for t in templates: + t.is_active = False + +db.commit() +print(f"Deactivated {len(templates)} auto-generated templates.")