Text Extraction Done, UI Fixes Done, Extracted Template Layout Saved in DB

prompt.txt

@@ -0,0 +1,856 @@
+# PROJECT: TEMPLATE-ENGINE
+
+You are a senior solution architect, senior Python backend architect, senior Angular architect, OCR/Document AI specialist, PostgreSQL database architect, and enterprise software engineer.
+
+Generate a COMPLETE production-ready application named:
+
+template-engine
+
+The solution must contain:
+
+1. Angular (latest stable version) frontend using PrimeNG.
+2. Python backend using FastAPI.
+3. PostgreSQL database.
+4. SQL migration scripts.
+5. Clean architecture.
+6. Production-grade code.
+7. Proper logging.
+8. Exception handling.
+9. DTOs / Schemas.
+10. Repository pattern.
+11. Service layer.
+12. API documentation.
+13. Unit-test ready structure.
+14. Docker support should NOT be generated now.
+15. Deployment configuration should NOT be generated now.
+
+---
+
+# BUSINESS REQUIREMENT
+
+The application is a Template Mapping Engine for invoices, purchase orders, delivery challans and similar business documents.
+
+The goal is:
+
+1. User uploads document.
+2. System extracts document layout and text.
+3. Layout preview is rendered.
+4. User creates template fields.
+5. User maps extracted document fields to template fields using drag and drop.
+6. Mapping is stored.
+7. When same vendor format arrives again, system automatically recognizes template and auto-fills mappings.
+
+---
+
+# TECHNOLOGY STACK
+
+Frontend:
+
+* Angular
+* PrimeNG
+* Angular CDK Drag and Drop
+* RxJS
+* Standalone Components
+* SCSS
+
+Backend:
+
+* Python 3.12+
+* FastAPI
+* SQLAlchemy
+* Alembic
+* Pydantic
+* PostgreSQL
+
+Document Processing:
+
+* pdfplumber
+* pymupdf (fitz)
+* pytesseract
+* opencv
+* pillow
+* layoutparser
+* numpy
+
+Optional AI Layer:
+
+* sentence-transformers
+* scikit-learn
+
+---
+
+# HIGH LEVEL MODULES
+
+1. Document Upload Module
+2. OCR Module
+3. Layout Extraction Module
+4. Preview Rendering Module
+5. Template Management Module
+6. Mapping Module
+7. Template Recognition Engine
+8. Auto Mapping Engine
+
+---
+
+# STEP 1 – FRONTEND DESIGN
+
+Create a page:
+
+/template-mapping
+
+The page must be split vertically.
+
+---
+
+|                    |                              |
+| LEFT PANEL         | RIGHT PANEL                 |
+| DOCUMENT PREVIEW   | TEMPLATE MAPPING            |
+|                    |                              |
+-----------------------------------------------------
+
+Width:
+
+* Left 60%
+* Right 40%
+
+Use PrimeNG Splitter.
+
+---
+
+LEFT PANEL
+
+Header section:
+
+Title:
+Document Preview
+
+Buttons:
+
+Upload Document
+
+Supported:
+
+* PDF
+* PNG
+* JPG
+* JPEG
+* TIFF
+
+After upload:
+
+Show extracted layout in tabular preview.
+
+Important:
+
+DO NOT display raw OCR text.
+
+Render layout structure.
+
+Example:
+
+Invoice Header
+
+Invoice Number
+INV001
+
+Invoice Date
+01-Jan-2026
+
+Vendor Name
+ABC Industries
+
+---
+
+Line Items
+
+| Item | Qty | Rate | Amount |
+
+---
+
+Tax Section
+
+---
+
+Total Section
+
+Maintain visual hierarchy.
+
+Styling is not important.
+
+Layout preservation is important.
+
+Every text block should be selectable.
+
+Every text block should be draggable.
+
+Store unique layout node id.
+
+---
+
+RIGHT PANEL
+
+Header:
+
+Template Name [textbox]
+
+Add Field Button
+
+Create Template Button
+
+---
+
+When user clicks Add Field:
+
+Open PrimeNG Dialog.
+
+Fields:
+
+Field Label
+
+Field Type
+
+Supported Types:
+
+TEXT
+NUMBER
+DATE
+AMOUNT
+ADDRESS
+TABLE_COLUMN
+
+Save
+
+This creates records in template_fields.
+
+---
+
+Below header show:
+
+Template Fields Tree/Grid
+
+Example:
+
+Vendor Name
+
+Invoice Number
+
+Invoice Date
+
+GST Number
+
+Item Name
+
+Qty
+
+Rate
+
+Tax %
+
+Tax Amount
+
+Grand Total
+
+Each field is droppable.
+
+---
+
+Bottom:
+
+Save Mapping Button
+
+---
+
+# STEP 2 – DOCUMENT UPLOAD, EXTRACTION AND DATABASE STORAGE
+
+When file uploaded:
+
+Detect type:
+
+PDF
+IMAGE
+
+---
+
+PDF PROCESSING
+
+If text PDF:
+
+Use pdfplumber
+
+Extract:
+
+* text
+* coordinates
+* page
+* bounding box
+
+---
+
+SCANNED PDF
+
+Convert pages to image.
+
+Run OCR.
+
+Use:
+
+pytesseract
+
+Extract:
+
+* text
+* coordinates
+
+---
+
+IMAGE PROCESSING
+
+Run:
+
+OpenCV preprocessing
+
+* grayscale
+* denoise
+* threshold
+
+Run OCR.
+
+Extract:
+
+* text
+* coordinates
+
+---
+
+For every extracted item store:
+
+text
+
+x
+
+y
+
+width
+
+height
+
+page_no
+
+block_type
+
+parent_block
+
+sequence
+
+confidence
+
+---
+
+Create logical blocks:
+
+HEADER
+
+VENDOR
+
+BILL_TO
+
+SHIP_TO
+
+TABLE
+
+TABLE_ROW
+
+TABLE_CELL
+
+TAX
+
+TOTAL
+
+FOOTER
+
+---
+
+Store everything in database.
+
+---
+
+# STEP 3 – RENDER LAYOUT PREVIEW
+
+Backend returns:
+
+DocumentLayoutResponse
+
+Example:
+
+{
+documentId,
+pages:[
+]
+}
+
+Layout must preserve:
+
+* hierarchy
+* coordinates
+* parent child relationships
+
+Frontend converts response into preview tree.
+
+Render:
+
+Header
+
+Vendor
+
+Line Items
+
+Tax
+
+Total
+
+Footer
+
+Every node:
+
+draggable
+
+---
+
+# STEP 4 – TEMPLATE MANAGEMENT
+
+Template Creation Flow
+
+User enters:
+
+Template Name
+
+Click Create Template
+
+Create record in templates.
+
+---
+
+Add Field Dialog
+
+Create rows in template_fields.
+
+Fields:
+
+field_label
+
+field_type
+
+display_order
+
+required_flag
+
+created_at
+
+---
+
+Support unlimited fields.
+
+Support edit/delete field.
+
+---
+
+# STEP 5 – DRAG AND DROP MAPPING
+
+Use Angular CDK.
+
+Drag Source:
+
+Document preview nodes.
+
+Drop Target:
+
+Template fields.
+
+---
+
+Mapping UI:
+
+Vendor Name
+← ABC Industries
+
+Invoice Number
+← INV-1001
+
+Invoice Date
+← 2026-01-01
+
+GST Number
+← 27ABCDE1234F1Z5
+
+---
+
+Save Mapping
+
+Backend stores:
+
+template_fields_mapping
+
+Include:
+
+document coordinates
+
+page number
+
+bounding box
+
+confidence
+
+layout path
+
+parent section
+
+---
+
+# STEP 6 – TEMPLATE RECOGNITION ENGINE
+
+Requirement:
+
+When same vendor document arrives again:
+
+System automatically identifies template.
+
+---
+
+Create template fingerprint.
+
+Store:
+
+Vendor Name
+
+Header Positions
+
+Table Header Names
+
+Relative Coordinates
+
+Document Structure
+
+---
+
+Generate fingerprint hash.
+
+Store in database.
+
+---
+
+Recognition Strategy
+
+Level 1
+
+Vendor Name Match
+
+Level 2
+
+Header Similarity
+
+Level 3
+
+Layout Similarity
+
+Level 4
+
+Coordinate Similarity
+
+---
+
+Use weighted score.
+
+Example:
+
+Vendor Match = 40%
+
+Header Match = 20%
+
+Layout Match = 20%
+
+Coordinate Match = 20%
+
+Threshold:
+
+85%
+
+If matched:
+
+Auto apply template.
+
+---
+
+# AUTO MAPPING ENGINE
+
+After template recognized:
+
+Find mapped coordinates.
+
+Extract values from same coordinate zones.
+
+Populate template fields automatically.
+
+Return:
+
+{
+templateMatched:true,
+templateId:1,
+confidence:96.5,
+extractedFields:[
+]
+}
+
+Frontend should immediately show populated template values.
+
+User may modify and save again.
+
+---
+
+# DATABASE DESIGN
+
+Create schema:
+
+templates
+
+---
+
+TABLE documents
+
+pk_document_id BIGSERIAL PK
+
+document_name VARCHAR(500)
+
+document_type VARCHAR(100)
+
+file_name VARCHAR(500)
+
+file_hash VARCHAR(500)
+
+page_count INTEGER
+
+status VARCHAR(50)
+
+created_at TIMESTAMP
+
+updated_at TIMESTAMP
+
+---
+
+TABLE document_layout
+
+pk_document_data_id BIGSERIAL PK
+
+fk_document_id BIGINT
+
+page_no INTEGER
+
+text_value TEXT
+
+block_type VARCHAR(100)
+
+parent_block_id BIGINT
+
+x_coordinate NUMERIC
+
+y_coordinate NUMERIC
+
+width NUMERIC
+
+height NUMERIC
+
+confidence NUMERIC
+
+sequence_no INTEGER
+
+layout_path TEXT
+
+created_at TIMESTAMP
+
+---
+
+TABLE templates
+
+pk_template_id BIGSERIAL PK
+
+template_name VARCHAR(255)
+
+template_fingerprint TEXT
+
+active_flag BOOLEAN
+
+created_at TIMESTAMP
+
+updated_at TIMESTAMP
+
+---
+
+TABLE template_fields
+
+pk_template_field_id BIGSERIAL PK
+
+fk_template_id BIGINT
+
+field_label VARCHAR(255)
+
+field_type VARCHAR(100)
+
+display_order INTEGER
+
+required_flag BOOLEAN
+
+created_at TIMESTAMP
+
+---
+
+TABLE template_fields_mapping
+
+pk_mapping_id BIGSERIAL PK
+
+fk_template_id BIGINT
+
+fk_template_field_id BIGINT
+
+fk_document_data_id BIGINT
+
+page_no INTEGER
+
+x_coordinate NUMERIC
+
+y_coordinate NUMERIC
+
+width NUMERIC
+
+height NUMERIC
+
+mapping_confidence NUMERIC
+
+layout_path TEXT
+
+created_at TIMESTAMP
+
+---
+
+TABLE template_recognition_history
+
+pk_history_id BIGSERIAL PK
+
+fk_template_id BIGINT
+
+fk_document_id BIGINT
+
+recognition_score NUMERIC
+
+matched_flag BOOLEAN
+
+created_at TIMESTAMP
+
+---
+
+Generate complete SQL scripts.
+
+Generate Alembic migrations.
+
+Generate indexes.
+
+Generate foreign keys.
+
+Generate constraints.
+
+---
+
+# BACKEND APIS
+
+POST /api/documents/upload
+
+GET /api/documents/{id}
+
+GET /api/documents/{id}/layout
+
+POST /api/templates
+
+PUT /api/templates/{id}
+
+DELETE /api/templates/{id}
+
+POST /api/template-fields
+
+PUT /api/template-fields/{id}
+
+DELETE /api/template-fields/{id}
+
+POST /api/mappings/save
+
+POST /api/templates/recognize
+
+POST /api/templates/auto-map
+
+---
+
+# PROJECT STRUCTURE
+
+Generate complete folder structure.
+
+Frontend structure.
+
+Backend structure.
+
+Models.
+
+Repositories.
+
+Services.
+
+Controllers.
+
+DTOs.
+
+Validation.
+
+Error Handling.
+
+Logging.
+
+Configurations.
+
+Environment files.
+
+Constants.
+
+Utilities.
+
+OCR helpers.
+
+Template recognition engine.
+
+Auto mapping engine.
+
+---
+
+# OUTPUT REQUIREMENT
+
+Generate code step-by-step in the following order:
+
+1. Complete solution architecture.
+2. PostgreSQL schema and SQL scripts.
+3. Backend folder structure.
+4. Backend implementation.
+5. OCR and layout extraction implementation.
+6. Template recognition engine.
+7. API layer.
+8. Angular folder structure.
+9. Angular UI implementation.
+10. PrimeNG screens.
+11. Drag and drop implementation.
+12. API integration.
+13. Auto mapping implementation.
+14. Validation.
+15. Testing strategy.
+
+All generated code must be production-ready, runnable, and complete with no placeholders or TODO comments.
+
+
+Use existing frontend project and enable this route //{ path: 'templates', component: TemplateComponent } and create respective component and add and design our page in this component. 
+
+Use backend project for python logic and related endpoints. database details are available in database.py. use or create new schema "templates" for this task
+
+Move to next step once I give the confirmation that step 1 is done.