Dashboard
Overview of GeoRef system health, cluster status, and data status.
Active Cluster
Target Cluster
MapServer
Last Job
Reference Data Row Counts
Side-by-side comparison of active and target clusters. Zero counts indicate data not yet imported. Import data to the target cluster, then rotate to make it active.
| Table | (active) | (target) | Description |
|---|---|---|---|
Application Data (DynamoDB)
Sources
Collections
Jobs
Collections
A collection defines what data to import. It groups data sources and locations together so you can run them as a single job.
Create Collection
| Name | Items | Auto-Rotate | Last Run | Actions |
|---|---|---|---|---|
No collections match your filter.
No collections yet. Create one to get started.
Data Source Items
Each item specifies a data source and which locations (states) to import. Click "Add Source" to include a data source in this collection.
| Data Type | Source | Country | Locations | Actions |
|---|---|---|---|---|
|
No data sources added yet. Add sources below.
Add Data Source
All available data sources have been added to this collection.
Recent Jobs
| ID | Status | Target | Started |
|---|---|---|---|
Data Sources
Data sources define where to download geospatial reference data from government servers. Edit URLs here if a government server changes its address.
| Type | Name | Country | Format | Rate Limit | Active | Actions |
|---|---|---|---|---|---|---|
|
Import Jobs
Import jobs download data from government sources and load it into the inactive database cluster. Each job runs multiple steps in parallel.
| ID | Collection | Status | Target | Progress | Started | Runtime | Actions |
|---|---|---|---|---|---|---|---|
No import jobs yet. Start one from the Collections page.
Import Schedules
Schedules automate import jobs. Individual collections can have their own cron schedule, or you can group multiple collections into a sequence with configurable delays between each.
Group Schedules
| Name | Schedule | Collections | Delay | Status | Last Run | Actions |
|---|---|---|---|---|---|---|
|
, |
No group schedules yet. Click "+ New Schedule" above to create one.
Individual Collection Schedules
Set a cron expression on a collection to run it automatically. Leave blank to disable.
| Collection | Schedule (cron) | Last Run | Actions |
|---|---|---|---|
|
Blue/Green Rotation
GeoRef uses two database clusters (blue and green). Imports load data into the inactive cluster. When ready, swap to make the new data live. Rollback reverts to the previous cluster.
Active Cluster
Inactive Cluster
Rotation History
| From | To | Type | By | Date |
|---|---|---|---|---|
Schema Patches
Schema patches modify the database structure (tables, columns, indexes). Patches are only applied to the inactive cluster to avoid disrupting live consumers.
Active Cluster
Inactive Cluster
Pending
Applied
Pending Patches
| Patch File | Status | Actions |
|---|---|---|
|
pending |
Applied Patches
| Patch File | Status | Applied |
|---|---|---|
|
applied |
No patch files found. Patches are committed to Schema/patches/ by developers.
GeoRef Admin Help
What is GeoRef?
GeoRef is a centralized geospatial reference database that stores SSURGO soil data, PLSS land survey boundaries, US Census state/county boundaries, and TIGER transportation/hydrology data. It replaces duplicate copies of this data across multiple systems (HighQ, Stratum, FieldReveal) with a single shared source.
Architecture: Blue/Green Clusters
GeoRef maintains two Aurora PostgreSQL database clusters called blue and green. At any time, one is active (serving live queries from all consumers) and one is inactive (available for data imports).
- All consumers (MapServer, Stratum, HighQ, FieldReveal) always read from the active cluster
- Import jobs always write to the inactive cluster
- When an import is validated, you swap to make the new data live
- If something is wrong, you rollback to the previous cluster
Typical Workflow
- Check Dashboard - verify system is healthy and review current data counts
- Review Data Sources - ensure government download URLs are current (they change occasionally)
- Create or select a Collection - a collection groups the data sources and locations you want to import
- Apply any pending Patches - schema changes must be applied to the inactive cluster before importing
- Run the Import - click "Run Import" on your collection. The job downloads from government servers, transforms the data, and loads it into the inactive cluster
- Monitor the Job - watch progress on the Jobs page. Each step (data type + location) runs in parallel
- Validate - check the Dashboard for row counts on the inactive cluster. Compare against expected values
- Swap Clusters - go to Rotation and click "Swap Clusters" to make the new data live
Pages
- Dashboard
- System health overview. Shows active cluster, PostGIS version, database connectivity, and row counts for all reference tables. Zero counts indicate data that hasn't been imported yet.
- Collections
- Manage import collections. A collection is a named group of data sources and locations. For example, "Kansas Full Import" might include SSURGO, PLSS, roads, and water for Kansas. Click "Run Import" to start a job.
- Data Sources
- View and edit data source configurations. Each source has a download URL pattern, file format, and rate limit. Edit URLs here when government servers change their addresses. The rate limit prevents overloading government servers.
- Jobs
- Monitor import jobs. Each job has multiple steps (one per data type + location). Steps run in parallel. If a step fails, you can "Resume" to retry only the failed steps. You can also "Cancel" a running job.
- Rotation
- Swap the active database cluster. "Swap Clusters" makes the inactive cluster active. "Rollback" reverts to the previous cluster. Every rotation is logged with who did it and when.
- Patches
- Apply database schema changes to the inactive cluster. Patches are SQL files committed by developers. They are never applied to the active cluster to avoid disrupting live consumers.
Data Sources Reference
| Type | Source | Content | Update Frequency |
|---|---|---|---|
census_states | US Census TIGER | State/territory boundaries with FIPS codes | Annual |
census_counties | US Census TIGER | County boundaries with FIPS codes | Annual |
tiger_roads | US Census TIGER | Road centerlines (per county) | Annual |
tiger_areawater | US Census TIGER | Lakes and ponds (per county) | Annual |
tiger_linearwater | US Census TIGER | Streams and rivers (per county) | Annual |
tiger_pointlm | US Census TIGER | Point landmarks (per state) | Annual |
plss_township | BLM ArcGIS | Township/Range boundaries | Ongoing |
plss_firstdivision | BLM ArcGIS | Section boundaries | Ongoing |
plss_seconddivision | BLM ArcGIS | Quarter section boundaries | Ongoing |
ssurgo | USDA NRCS | Soil survey polygons + properties | Annual |
ca_provinces | Statistics Canada | Canadian province/territory boundaries | Every 5 years |
ca_census_divisions | Statistics Canada | Canadian census division boundaries | Every 5 years |
ca_roads | NRCan NRN | Canadian road network (per province) | Ongoing |
ca_hydro | NRCan NHN | Canadian hydrology network | Ongoing |
ca_soil_dss | AAFC | Canadian detailed soil survey | Infrequent |
Troubleshooting
- Dashboard shows "error" or "degraded"
- The active cluster may be unreachable. Check Aurora status in the AWS console. If the cluster is at 0 ACU, it may need a moment to wake up.
- Import job step shows "failed"
- Click "Details" to see the error message. Common causes: government server changed URL (update in Data Sources), rate limiting (increase rate_limit_ms), or network timeout (retry with Resume).
- SSURGO import says "not found in S3 cache"
- SSURGO GeoPackage files must be manually downloaded from NRCS Box.com and uploaded to the S3 import cache bucket. The files are too large for automated download.
- All table counts show 0
- No data has been imported yet. Create a collection, add data sources, and run an import job.
- After swapping, consumers show old data
- Consumers cache the active cluster endpoint. MapServer Fargate tasks restart automatically. Lambda functions pick up the change on the next invocation. Allow up to 5 minutes for DNS propagation.
About
GeoRef v1.0 - Geospatial Reference Database
AgWorks, LLC
Repository