diff --git a/docs/database-performance-audit.md b/docs/database-performance-audit.md new file mode 100644 index 0000000..ba9f616 --- /dev/null +++ b/docs/database-performance-audit.md @@ -0,0 +1,132 @@ +# Database performance audit + +## Scope and safety + +This audit covers the PostgreSQL schema, indexes, constraints, views, functions, and their relationship to `src/matrix/nimble/conf/nimble.qry`, with particular focus on Visit MIS and Dashboard. + +The catalog inspection is read-only. No indexes, views, functions, or PostgreSQL settings have been changed. The audited local database is only about 30 MB and its core business tables currently contain almost no live data, so local execution times cannot represent production. Final index selection must be validated with production statistics and `EXPLAIN (ANALYZE, BUFFERS)` captured for representative parameters. + +## Inventory + +- PostgreSQL: 18.4 +- User tables: 120 +- Normal views: 80 +- Materialized views: 2 +- User functions: 220 +- Indexes: 187 +- Function statistics: disabled (`track_functions = none`) +- Statement statistics: unavailable (`pg_stat_statements` is not preloaded) + +The raw catalog exports are under `target/database-audit/` and can be regenerated with `matrix.nimble.DatabaseAuditRunner`. + +The subsequent analysis of the complete `matrix.sql` schema dump confirmed that `portfolio_bank_branch` and `colony_allocation` are normal views rather than tables. Index recommendations therefore target their underlying tables (`portfolio_branch`, `bank_branch`, `portfolio`, `colony`, `belt_allocation`, and `verifier`). + +Generated index script: `docs/production-indexes.sql`. It contains preflight +checks, high-confidence Visit MIS/Dashboard indexes, foreign-key support indexes, +statistics refresh, postflight validation, and a commented rollback reference. + +## Application-to-database mapping + +| Feature | Application query | Database entry point | +|---|---|---| +| Visit MIS | Query306 | `monthvisitmis(...)` | +| Dashboard initial load | Query473 | `dashboard()` | +| Dashboard partial refresh | Query525 | `dashboardpartial(...)` | + +`nimble.qry` contains about 460 application query definitions. Its most frequently referenced core relation is `main`, followed by document/bank-report relations, `app_user`, `main_mainopr`, `verifier`, `temp_reporting`, and `portfolio_bank_branch`. This is source frequency, not runtime frequency; production statement statistics are required for workload ranking. + +## Visit MIS findings + +### Execution shape + +`monthvisitmis(...)` loops over each requested day and performs an aggregate against `visit_mis`. For every verifier/operator row returned, it runs a second query to find `max(doneon)`. This is an N+1 query pattern: cost grows with both date range and verifier count. + +`vervisitmis(...)` has the same per-verifier `max(doneon)` lookup and an additional per-verifier aggregate against `verifier_monthly_data`. It also dynamically switches between `current_visit_mis` and `visit_mis`. + +Both functions build complete HTML reports inside PL/pgSQL through repeated text concatenation. This increases database CPU and memory usage and prevents the Java/UI layer from receiving structured rows. It is not the first optimization target, but should ultimately be replaced with a set-returning/data function and JSP rendering. + +### View expansion + +`visit_mis` and `current_visit_mis` each expand into three `UNION ALL` branches for residence, office, and property visits. Every branch joins: + +- `main_mainopr`, itself a join of `main` and `main_operations` by `uuid` +- `portfolio_bank_branch` by `portbranch_id` +- `verifier` by the visit-specific verifier column +- `app_user` by verifier operator +- `riskalerts` by `(uuid, visit)` + +Every branch also performs a correlated zone aggregation against `colony_allocation` by `verifier_id`. + +`verifier_monthly_data` is another three-branch `UNION ALL` over `main`, filtered by `receivedate`, visit flag, verifier, and same-address flag. + +### Index gaps and mismatches + +These are candidates to validate on production, not DDL to run blindly: + +1. `riskalerts` has separate indexes on `uuid` and `visit`, although the views join using both. A composite index beginning with `(uuid, visit)` matches the access path better. +2. `colony_allocation` is a view. Its base table `belt_allocation` has an index on `verifier_id`, but lacks an index beginning with `belt_id` for its join to `colony`; the generated script adds `(belt_id, verifier_id)`. +3. `main_operations` has single-column indexes on `oprsendon`, `sdoneon`, `sdone`, and `fileclosed`, but the report predicates combine time/status data after joining by `uuid`. Production plans are needed to decide between composite and partial indexes. +4. `main` has individual verifier indexes and a `receivedate` index. `verifier_monthly_data` combines each verifier with `receivedate`, a visit flag, and same-address status, so one-column indexes may cause excessive filtering after index access. +5. `portfolio_bank_branch` is a view. Its base table `portfolio_branch` already has a primary key on `portbranch_id`; additional indexes are recommended for its portfolio/active and bank-branch access paths. +6. Several existing indexes duplicate or overlap: `main.bank_branch_id` has two btree indexes, and `main_operations.uuid` has both a primary btree index and a hash index. Production usage and index size/write cost should be checked before removing anything. + +### Primary query rewrite + +The per-verifier `max(doneon)` must be folded into the main grouped query as `max(doneon) FILTER (...)`. Monthly visit count/limit should also be pre-aggregated once and joined by verifier. This changes roughly `1 + N + N` queries per report/day into one grouped query per day, or preferably one query for the entire requested range. + +The date loop can be replaced by a single `generate_series` relation joined to the source data, or by grouping source rows by reporting date. The best variant depends on the required backlog semantics and will be verified against current output before replacement. + +## Dashboard findings + +`dashboard()` executes at least ten child functions serially: + +1. `dashvisits()` +2. `dashscannerhourly()` +3. `dashoperatorhourly('opr')` +4. `dashrihhourly()` +5. `dashoperatorhourly('edp')` +6. `dashhourlytobesolved()` +7. `dashriskandphoto()` +8. `dashageing()` +9. `dash3monthsfigures(1, 0)` +10. `dash3monthsfigures(0, 0)` + +The major repeated work is: + +- `dashvisits()` repeatedly counts the same expanded `current_visit_mis` and `scanning_grid` views for different categories. +- `dashageing()` scans `scanning_grid` about seven times to produce age buckets. +- `dashboardpartial()` calls `dash3monthsfigures()` twice for one widget refresh. +- Dashboard views expand through `main`, `main_operations`, `temp_reporting`, `riskalerts`, user/company/branch/portfolio lookups, and report tables. + +These should be consolidated using conditional aggregates (`count(*) FILTER (WHERE ...)`) so each source is scanned once per widget. The initial dashboard response should then call fewer functions or one data-returning function. UI polling should avoid starting a refresh while the previous request is still running. + +## Schema and observability findings + +- The database relies heavily on views and PL/pgSQL functions, including dynamic SQL. PostgreSQL dependency metadata cannot discover relations embedded only in dynamic SQL, so function definitions must be reviewed in addition to catalog dependencies. +- A number of foreign keys lack a leading index. These matter mainly for parent updates/deletes and child-side joins; each must be ranked by actual workload rather than indexed automatically. +- Core tables contain some redundant indexes, while some logical join keys have no visible supporting index. +- `track_functions = none` prevents attribution of time to the 220 functions. +- `pg_stat_statements` is unavailable, preventing reliable ranking by total time, mean time, calls, rows, and temporary I/O. + +## Required production evidence + +Before applying point 5, collect during a normal busy period: + +1. Row counts and relation/index sizes for the Visit MIS and Dashboard base tables. +2. `pg_stat_user_tables` and `pg_stat_user_indexes` snapshots before and after the period. +3. `EXPLAIN (ANALYZE, BUFFERS, WAL, SETTINGS, VERBOSE)` for Visit MIS source aggregates using representative branch, verifier, and date-range values. +4. Slow statement data from `pg_stat_statements` after enabling it through the normal PostgreSQL change process. +5. Function timing after setting `track_functions = pl` (or `all`) through the normal production change process. +6. PostgreSQL logs with `log_min_duration_statement` set to a suitable temporary threshold, if extensions/settings cannot be enabled immediately. + +Do not run `EXPLAIN ANALYZE` on production-changing functions. Extract and explain their underlying `SELECT` statements inside a read-only transaction. + +## Recommended implementation order for point 5 + +1. Enable observability and capture a baseline. +2. Add only production-plan-confirmed indexes, using `CREATE INDEX CONCURRENTLY` and checking duplicate/overlapping indexes first. +3. Rewrite Visit MIS to eliminate N+1 lookups and repeated date scans while preserving report totals exactly. +4. Consolidate Dashboard counts into single-pass aggregates. +5. Return structured data from new versioned functions and render HTML in the application; retain old functions during comparison. +6. Run old and new implementations side by side for representative dates/branches and compare every total before switching. +7. Load-test with at least 20 concurrent sessions and monitor database CPU, pool wait time, query latency, temporary I/O, and heap usage. diff --git a/docs/production-indexes.sql b/docs/production-indexes.sql new file mode 100644 index 0000000..10bfd53 --- /dev/null +++ b/docs/production-indexes.sql @@ -0,0 +1,245 @@ +-- Cygnus PostgreSQL index migration +-- Derived from matrix.sql, nimble.qry, Visit MIS, and Dashboard functions/views. +-- +-- IMPORTANT +-- 1. Run each CREATE INDEX as a separate statement. Do not wrap this file in BEGIN/COMMIT. +-- 2. CREATE INDEX CONCURRENTLY minimizes blocking but still consumes CPU, I/O and disk. +-- 3. Run during a lower-traffic window and monitor replication lag and free disk space. +-- 4. IF NOT EXISTS checks the name only. Run the preflight query first to inspect equivalent indexes. +-- 5. These statements do not change table columns, constraints, or stored data. + +-- --------------------------------------------------------------------------- +-- Preflight: current sizes and existing definitions +-- --------------------------------------------------------------------------- + +SELECT + n.nspname AS schema_name, + t.relname AS table_name, + i.relname AS index_name, + pg_size_pretty(pg_relation_size(i.oid)) AS index_size, + pg_get_indexdef(i.oid) AS definition +FROM pg_index x +JOIN pg_class t ON t.oid = x.indrelid +JOIN pg_class i ON i.oid = x.indexrelid +JOIN pg_namespace n ON n.oid = t.relnamespace +WHERE n.nspname = 'public' + AND t.relname IN ( + 'riskalerts', 'belt_allocation', 'portfolio_branch', 'main', + 'main_operations', 'temp_reporting' + ) +ORDER BY t.relname, i.relname; + +SELECT + relname, + n_live_tup, + n_dead_tup, + seq_scan, + seq_tup_read, + idx_scan, + pg_size_pretty(pg_total_relation_size(relid)) AS total_size +FROM pg_stat_user_tables +WHERE relname IN ( + 'riskalerts', 'belt_allocation', 'portfolio_branch', 'main', + 'main_operations', 'temp_reporting' +) +ORDER BY pg_total_relation_size(relid) DESC; + +-- --------------------------------------------------------------------------- +-- Phase 1: high-confidence join and filter indexes +-- --------------------------------------------------------------------------- + +-- visit_mis/current_visit_mis join riskalerts using both columns. The dump has +-- separate uuid and visit indexes, which do not provide this composite lookup. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_riskalerts_uuid_visit + ON public.riskalerts USING btree (uuid, visit); + +-- colony_allocation is a view joining colony to belt_allocation by belt_id. +-- Only belt_allocation(verifier_id) exists in the supplied schema. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_belt_allocation_belt_verifier + ON public.belt_allocation USING btree (belt_id, verifier_id); + +-- portfolio_bank_branch expands through portfolio_branch. These support its +-- joins and the common portfolio/isactive lookup used by nimble.qry. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_portfolio_branch_portfolio_active + ON public.portfolio_branch USING btree (portfolio_id, isactive, portbranch_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_portfolio_branch_bank_branch + ON public.portfolio_branch USING btree (bank_branch_id); + +-- Dashboard hourly_tobesolved_cases filters on tobesolvedon and then reads +-- sdone/sdoneon. The existing temp_reporting indexes cover uuid and isclosed. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_temp_reporting_tobesolvedon_open + ON public.temp_reporting USING btree (tobesolvedon) + INCLUDE (sdone, sdoneon, portfolio_id) + WHERE tobesolvedon IS NOT NULL; + +-- scan_mis and Dashboard read today's completed scans. This avoids scanning all +-- main_operations rows having the same low-cardinality sdone value. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_operations_scan_doneon + ON public.main_operations USING btree (sdoneon, sdoneby, uuid) + WHERE sdone = 1 AND sdoneon IS NOT NULL; + +-- Visit MIS expands residence/office/property separately. These partial indexes +-- remove deleted/non-applicable rows before branch/verifier matching. They also +-- retain uuid for the main_operations join. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_visit_res_branch_verifier + ON public.main USING btree (branch_id, resiverifier, uuid) + WHERE isdeleted = 0 AND rv = 1; + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_visit_off_branch_verifier + ON public.main USING btree (branch_id, offverifier, uuid) + WHERE isdeleted = 0 AND ov = 1; + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_visit_prop_branch_verifier + ON public.main USING btree (branch_id, propverifier, uuid) + WHERE isdeleted = 0 AND pv = 1; + +-- vervisitmis queries verifier_monthly_data by verifier and receivedate. These +-- are partial and therefore smaller than unrestricted composite indexes. +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_monthly_res_verifier_date + ON public.main USING btree (resiverifier, receivedate) + WHERE isdeleted = 0 AND rv = 1 AND sradd = 0; + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_monthly_off_verifier_date + ON public.main USING btree (offverifier, receivedate) + WHERE isdeleted = 0 AND ov = 1 AND soadd = 0; + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_main_monthly_prop_verifier_date + ON public.main USING btree (propverifier, receivedate) + WHERE isdeleted = 0 AND pv = 1 AND spadd = 0; + +-- --------------------------------------------------------------------------- +-- Phase 2: foreign-key support indexes +-- --------------------------------------------------------------------------- +-- These do not directly solve Visit MIS/Dashboard latency. They prevent child +-- table scans during parent updates/deletes and support related joins. + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_case_documents_uuid + ON public.case_documents USING btree (uuid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_contact_mapping_contact_id + ON public.contact_mapping USING btree (contact_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_contacts_history_contact_id + ON public.contacts_history USING btree (contact_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_document_data_uuid + ON public.document_data USING btree (uuid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_field_mapping_field_id + ON public.field_mapping USING btree (field_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_field_mapping_template_id + ON public.field_mapping USING btree (template_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_field_value_field_id + ON public.field_value USING btree (field_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_onlinecasedetails_uuid + ON public.onlinecasedetails USING btree (uuid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_options_mapping_opt_value_id + ON public.options_mapping USING btree (opt_value_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_permission_page_id + ON public.permission USING btree (page_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_phoneno_history_uuid + ON public.phoneno_history USING btree (uuid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tagdoc_history_docid + ON public.tagdoc_history USING btree (docid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tagged_docs_uuid + ON public.tagged_docs USING btree (uuid); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_template_mapping_portfolio_id + ON public.template_mapping USING btree (portfolio_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_template_mapping_template_id + ON public.template_mapping USING btree (template_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tracker_template_mapping_portfolio + ON public.tracker_template_mapping USING btree (portfolio_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tracker_template_mapping_tracker + ON public.tracker_template_mapping USING btree (tracker_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tracker_sheet_layout_sheet + ON public.tracker_template_sheet_layout USING btree (sheet_id); + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tracker_template_sheets_tracker + ON public.tracker_template_sheets USING btree (tracker_id); + +-- Refresh planner statistics after all builds. ANALYZE does not rewrite tables, +-- but it consumes resources, so run it after the index builds at low traffic. +ANALYZE public.riskalerts; +ANALYZE public.belt_allocation; +ANALYZE public.portfolio_branch; +ANALYZE public.temp_reporting; +ANALYZE public.main_operations; +ANALYZE public.main; + +-- --------------------------------------------------------------------------- +-- Postflight validation +-- --------------------------------------------------------------------------- + +SELECT + indexrelname, + idx_scan, + idx_tup_read, + idx_tup_fetch, + pg_size_pretty(pg_relation_size(indexrelid)) AS index_size +FROM pg_stat_user_indexes +WHERE indexrelname LIKE 'idx_%' + AND relname IN ( + 'riskalerts', 'belt_allocation', 'portfolio_branch', 'main', + 'main_operations', 'temp_reporting' + ) +ORDER BY relname, indexrelname; + +-- Check for invalid indexes left by an interrupted concurrent build. +SELECT n.nspname, c.relname AS index_name +FROM pg_index i +JOIN pg_class c ON c.oid = i.indexrelid +JOIN pg_namespace n ON n.oid = c.relnamespace +WHERE n.nspname = 'public' + AND NOT i.indisvalid; + +-- --------------------------------------------------------------------------- +-- Emergency rollback reference -- intentionally commented out +-- --------------------------------------------------------------------------- +-- Uncomment and run only the index that must be removed. Do not execute this +-- section after a successful migration unless a measured regression occurs. +-- +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_riskalerts_uuid_visit; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_belt_allocation_belt_verifier; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_portfolio_branch_portfolio_active; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_portfolio_branch_bank_branch; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_temp_reporting_tobesolvedon_open; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_operations_scan_doneon; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_visit_res_branch_verifier; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_visit_off_branch_verifier; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_visit_prop_branch_verifier; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_monthly_res_verifier_date; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_monthly_off_verifier_date; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_main_monthly_prop_verifier_date; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_case_documents_uuid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_contact_mapping_contact_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_contacts_history_contact_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_document_data_uuid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_field_mapping_field_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_field_mapping_template_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_field_value_field_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_onlinecasedetails_uuid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_options_mapping_opt_value_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_permission_page_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_phoneno_history_uuid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tagdoc_history_docid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tagged_docs_uuid; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_template_mapping_portfolio_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_template_mapping_template_id; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tracker_template_mapping_portfolio; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tracker_template_mapping_tracker; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tracker_sheet_layout_sheet; +-- DROP INDEX CONCURRENTLY IF EXISTS public.idx_tracker_template_sheets_tracker; diff --git a/src/test/java/matrix/nimble/DatabaseAuditRunner.java b/src/test/java/matrix/nimble/DatabaseAuditRunner.java new file mode 100644 index 0000000..b8e05b2 --- /dev/null +++ b/src/test/java/matrix/nimble/DatabaseAuditRunner.java @@ -0,0 +1,199 @@ +package matrix.nimble; + +import java.io.BufferedWriter; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.sql.Statement; + +import matrix.nimble.utilities.DatabaseConnectionPool; + +/** Read-only PostgreSQL catalog audit. Run explicitly; it is not a JUnit test. */ +public final class DatabaseAuditRunner { + private static final Path OUTPUT = Path.of("target", "database-audit"); + + private DatabaseAuditRunner() { + } + + public static void main(String[] args) throws Exception { + Files.createDirectories(OUTPUT); + try { + try (Connection connection = DatabaseConnectionPool.getConnection()) { + connection.setReadOnly(true); + connection.setAutoCommit(false); + try (Statement statement = connection.createStatement()) { + statement.execute("set transaction read only"); + statement.execute("set statement_timeout = '30s'"); + statement.execute("set lock_timeout = '2s'"); + } + + export(connection, "database.tsv", """ + select current_database() database_name, current_user database_user, + current_setting('server_version') server_version, + pg_size_pretty(pg_database_size(current_database())) database_size, + current_setting('track_functions') track_functions + """); + export(connection, "schemas.tsv", """ + select nspname schema_name + from pg_namespace + where nspname not like 'pg_%' and nspname <> 'information_schema' + order by nspname + """); + export(connection, "tables.tsv", """ + select s.schemaname schema_name, s.relname table_name, + s.n_live_tup estimated_live_rows, s.n_dead_tup estimated_dead_rows, + s.seq_scan, s.seq_tup_read, s.idx_scan, s.idx_tup_fetch, + s.last_analyze, s.last_autoanalyze, s.last_vacuum, s.last_autovacuum, + pg_total_relation_size(s.relid) total_bytes, + pg_relation_size(s.relid) table_bytes, + pg_indexes_size(s.relid) indexes_bytes + from pg_stat_user_tables s + order by pg_total_relation_size(s.relid) desc + """); + export(connection, "columns.tsv", """ + select table_schema, table_name, ordinal_position, column_name, data_type, + udt_name, is_nullable, column_default + from information_schema.columns + where table_schema not in ('pg_catalog', 'information_schema') + order by table_schema, table_name, ordinal_position + """); + export(connection, "constraints.tsv", """ + select n.nspname schema_name, c.relname table_name, con.conname constraint_name, + case con.contype when 'p' then 'PRIMARY KEY' when 'f' then 'FOREIGN KEY' + when 'u' then 'UNIQUE' when 'c' then 'CHECK' when 'x' then 'EXCLUSION' + else con.contype::text end constraint_type, + pg_get_constraintdef(con.oid, true) definition + from pg_constraint con + join pg_class c on c.oid = con.conrelid + join pg_namespace n on n.oid = c.relnamespace + where n.nspname not in ('pg_catalog', 'information_schema') + order by n.nspname, c.relname, con.contype, con.conname + """); + export(connection, "indexes.tsv", """ + select ui.schemaname schema_name, ui.relname table_name, ui.indexrelname index_name, + ui.idx_scan, ui.idx_tup_read, ui.idx_tup_fetch, + pg_relation_size(ui.indexrelid) index_bytes, + pi.indisprimary, pi.indisunique, pi.indisvalid, + pg_get_indexdef(ui.indexrelid) definition + from pg_stat_user_indexes ui + join pg_index pi on pi.indexrelid = ui.indexrelid + order by ui.schemaname, ui.relname, ui.indexrelname + """); + export(connection, "foreign-keys-without-leading-index.tsv", """ + select n.nspname schema_name, c.relname table_name, con.conname foreign_key, + pg_get_constraintdef(con.oid, true) definition + from pg_constraint con + join pg_class c on c.oid = con.conrelid + join pg_namespace n on n.oid = c.relnamespace + where con.contype = 'f' + and n.nspname not in ('pg_catalog', 'information_schema') + and not exists ( + select 1 from pg_index i + where i.indrelid = con.conrelid and i.indisvalid + and (i.indkey::smallint[])[0:cardinality(con.conkey)-1] = con.conkey + ) + order by n.nspname, c.relname, con.conname + """); + export(connection, "views.tsv", """ + select schemaname schema_name, viewname view_name, definition + from pg_views + where schemaname not in ('pg_catalog', 'information_schema') + order by schemaname, viewname + """); + export(connection, "materialized-views.tsv", """ + select schemaname schema_name, matviewname view_name, ispopulated, + pg_total_relation_size(format('%I.%I', schemaname, matviewname)::regclass) total_bytes, + definition + from pg_matviews + order by schemaname, matviewname + """); + export(connection, "functions.tsv", """ + select n.nspname schema_name, p.proname function_name, + pg_get_function_identity_arguments(p.oid) identity_arguments, + l.lanname language, p.provolatile volatility, p.proparallel parallel_safety, + p.prosecdef security_definer, coalesce(s.calls, 0) calls, + coalesce(s.total_time, 0) total_time_ms, coalesce(s.self_time, 0) self_time_ms, + pg_get_functiondef(p.oid) definition + from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + join pg_language l on l.oid = p.prolang + left join pg_stat_user_functions s on s.funcid = p.oid + where n.nspname not in ('pg_catalog', 'information_schema') + order by n.nspname, p.proname, pg_get_function_identity_arguments(p.oid) + """); + export(connection, "view-dependencies.tsv", """ + select distinct vn.nspname view_schema, v.relname view_name, + tn.nspname referenced_schema, t.relname referenced_object + from pg_rewrite r + join pg_class v on v.oid = r.ev_class + join pg_namespace vn on vn.oid = v.relnamespace + join pg_depend d on d.objid = r.oid + join pg_class t on t.oid = d.refobjid + join pg_namespace tn on tn.oid = t.relnamespace + where v.relkind in ('v', 'm') and t.oid <> v.oid + and vn.nspname not in ('pg_catalog', 'information_schema') + order by vn.nspname, v.relname, tn.nspname, t.relname + """); + export(connection, "mis-menu-mapping.tsv", """ + select p.page_id, p.menulabel, p.targeturl, + string_agg(distinct pe.requestval, ' | ' order by pe.requestval) request_values + from pages p + left join permission pe on pe.page_id = p.page_id + where lower(coalesce(p.targeturl, '')) like '%vddtmis%' + or lower(coalesce(p.menulabel, '')) like '%visit%mis%' + group by p.page_id, p.menulabel, p.targeturl + order by p.page_id + """); + exportOptional(connection, "pg-stat-statements.tsv", """ + select calls, total_exec_time, mean_exec_time, rows, + shared_blks_hit, shared_blks_read, temp_blks_read, temp_blks_written, query + from pg_stat_statements + where dbid = (select oid from pg_database where datname = current_database()) + order by total_exec_time desc + limit 200 + """); + + connection.rollback(); + } + } finally { + DatabaseConnectionPool.close(); + } + System.out.println("Database audit written to " + OUTPUT.toAbsolutePath()); + } + + private static void exportOptional(Connection connection, String fileName, String sql) throws IOException { + try { + export(connection, fileName, sql); + } catch (SQLException exception) { + Files.writeString(OUTPUT.resolve(fileName + ".error"), exception.getMessage(), StandardCharsets.UTF_8); + } + } + + private static void export(Connection connection, String fileName, String sql) throws SQLException, IOException { + try (PreparedStatement statement = connection.prepareStatement(sql); + ResultSet result = statement.executeQuery(); + BufferedWriter writer = Files.newBufferedWriter(OUTPUT.resolve(fileName), StandardCharsets.UTF_8)) { + ResultSetMetaData metadata = result.getMetaData(); + int columns = metadata.getColumnCount(); + for (int column = 1; column <= columns; column++) { + if (column > 1) writer.write('\t'); + writer.write(metadata.getColumnLabel(column)); + } + writer.newLine(); + while (result.next()) { + for (int column = 1; column <= columns; column++) { + if (column > 1) writer.write('\t'); + String value = result.getString(column); + writer.write(value == null ? "" : value.replace("\t", " ").replace("\r", "").replace("\n", "\\n")); + } + writer.newLine(); + } + } + } +}