This page is for sharing LTS system development and maintenance plans for the Harvard Geospatial Library.
I. Problem/Value Statement
The Harvard academic and research community are increasingly using geospatial data analysis and visualizations to do research and to present visualizations of research results. Additionally, Harvard is a partner in the OpenGeoportal cross-institutional collaboration to share GIS data sets widely for reuse and create open source discovery and repository software. Metadata about Harvard GIS data sets, and previews of the data itself, has been made available through OpenGeoportal, and is directly searchable in library supported portals at Stanford, MIT, Tufts, UC Berkeley, and many others.
The Graduate School of Design teaches classes in geographic information systems (GIS) and requires students to learn and become fluent in geospatial analysis and mapping. The Center for Geographic Analysis includes links to library data sets in WorldMap. A centrally supported GIS data repository and search tool enables researchers to discover GIS datasets that are available on-line from Harvard and peer institutions, and to access and download datasets for use in a wide variety of research, teaching, and learning applications. Additionally, tools and GIS technical support is critical for library staff creating metadata for GIS data sets and to load metadata and GIS data into the repository.
HGL requires a major software upgrade because a key software component has reached end of life. This upgrade requires substantial software changes, as well as new installations of key software components and data migration. This work requires the collaboration of the HGL software developer, LTS operations, HUIT database analysts, and library experts from the Harvard Library Map Collection.
The Harvard Geospatial Library provides a centrally supported resource to faculty, students, GSD teaching and library staff, and HCL Maps and ITS Library staff. Without HGL, these staff could not offer services to meet their university missions. In addition, GIS partners around the world view Harvard's commitment to GIS data set discovery and reuse as exemplary and valuable.
II. Vision and Approach
Describe the solution:
HGL is a service provided centrally by the Harvard Library with critical support from LTS. The LTS portion is comprised of a constellation of software applications, servers, and technical support service to library staff. See this diagram of the HGL universe for an overview of the systems and services provided. HGL includes not only a web based discovery interface and repository for GIS data sets, but also back end servers accessed directly by GSD and Map Library and ITS staff to enrich and manage their GIS holdings.
Like any digital system, HGL requires ongoing maintenance, including software and server upgrades to supported and secure versions of software applications, operating systems, databases, and server hardware.
- Configure a new database server for HGL with the latest Oracle release and latest ESRI ArcSDE software (mercator)
- Migrate the database contents from the older HGL databases to the new database
- Develop new HGL software for map tile caching and interaction with a new ESRI component, ArcGIS Server.
- Update the user interface for HGL to use Open Geoportal 2.0 (pelham)
- Publish all layers as new web services for OGP 2.0
- Work closely with Map Library staff to test the migrated HGL data
- Shut down the old HGL application server (calvert) and database server (alcott)
See HGL FY18 Roadmap
Who is the work being done for? (Sponsor) Harvard Geospatial Working Group of the Research, Teaching, and Learning Standing Committee, Harvard Library
Who is funding the work? HL through the LTS operating budget
Who will accept the work? Bonnie Burns, chair, Harvard Geospatial Working Group
Who is the project manager? Randy Stern
Who will be involved in doing the work (service area, department, etc …)? [Include name, project role, and estimated percentage of time per month for the project duration]
|Resource Name||Role(s)||Project Estimate (through project Completion||Ongoing support||Work|
|Dave Siegel||developer and FY17 software support||100%||30%|
|Tim Elliott||Data publishing and tile caching script development||25% for 2 months.||0%||Productionize Bonnie Burns' prototype script|
|Bonnie Burns||HGL working group chair, publishing script requirements||small||Create prototype scripts, assess intermediate results|
|Danielle Brown||geospatial data technician, ITS Metadata Management||small|
|Marc McGee||metadata analyst, ITS Metadata Management||small|
|Venu Ramaswamy||HUIT DBA||as much as needed||Migrate the databases fro 3 ArcSDE instances to new versions of ArcSDE|
|Sharon||Dev/ops manager||small||review architecture for security, scalability, sustainability, cost|
|Tom Scorpa||Dev/ops engineer||5% for 1 year||5%?|
|Anthony||architect||review architecture for security, scalability, sustainability, cost|
|LTS Support (tbd)||Support lead||define and implement support policies, public support links, and internal documentation|
IV. Schedule and Cost
Schedule: [include project phases, known reporting dates, and delivery deadlines]
See HGL disk storage and costs for detail.
|Prioritization and other work||Prioritization by Harvard Library||Ann Whiteside||4/21/15||done||Bonnie and Janina wrote up a one page document describing the need for HGL technical work, the broad spectrum of users, the connection (and/or future connection) to CGA, the importance of working within the Open GeoPortal consortium, the work that is needed to bring HGL up to the right version and other needs. A working group will write a report for a TBD standing committee and LLT.|
|HOLLIS+ GIS integration and embedded maps||Dave||7/1/15||done||Support for HOLLIS team in adding Geospatial data from the Harvard Geospatial Library (HGL) to HOLLIS+.|
|HGL developer available after DRS/FITS work reassigned||Dave||8/1/15||done||HGL developer assigned|
|Migrate HGL to tomcat7/java7||Dave||12/22/15||done||HUIT required security update to HGL|
|HGL Working Group prioritizes OGP 2.0||HGL Working Group||1/4/16||done||The HGL Working Group performed usability testing of the HGL 2.0 interface. The group strongly feels that we should move ahead with this.|
|Hardware/software configuration||Migrate HGL QA environment from ArcSDE 9.3.1/Oracle to ArcSDE 10.2/Postgres w ArcGIS server with OpenGeoportal 2.0||Dave||2/24/16||done||Proof of concept development|
|Attempt to migrate new HGL servers to the AWS Cloud||2/1/16||done||LTS operations configures new HGL in the cloud. Decision is to revert to existing VM architecture.|
|QA testing complete on VM architecture, request for LTS OPS production machine setup||Dave||3/23/16||done||Initiate production deployment planning with LTS operations group|
|Setup base software on new machines||LTS OPS||5/6/16||done||Machines had been setup on by Feb 3, before AWS attempt|
|Configure machines for access as needed||LTS OPS||6/10/16||done|
|Database migration||HUIT DBA help requested in Service Now - INC01797979||Randy||6/29/16||done||Venu Ramaswamy assigned 7/13/16 to perform database migration to new ArcSDE schema|
|Develop and test upgrade path for data migration from old HGL to new HGL||HUIT DBA (Venu), Dave, LTS OPS||10/15/16||done||Many iterations with ESRI, HUOT DBA, and Dave to configure ESRI and Oracle database|
|Database migrated for first SDE instance||Venu, Dave||10/31/16||done||Took 3.5 months|
|Database migrated for second SDE instance||Venu, Dave||11/20/16||failed||Should have been to copy procedure from first migration. Has run into many issues with HUIT, LTS, ESRI.|
|Database migrated for PROD SDE instance||Dave, Tim, Bonnie||1/3117||done (2/3/17)||Latest proposal is to try to bypass the HUIT database migration.|
|Data publication||Data publication with Map Library||Dave||4/1/17?||"Publishing" the data makes it available to the HGL user interface. Sub projects: Script development, raster data publication, vector data copy and publication|
|Testing complete||Bonnie Burns||4/15/17?|
|Launch||Launch new HGL and shut down old HGL||Bonnie Burns, LTS OPS||5/1/17?|
Constraints: Coordinating LTS OPS, HUIT DBAs, ESRI tech support, and the HGL developer is a challenge.
Assumptions: Map library can create publish scripts and execute them in a timely manner. HUIT DBA can upgrade last 2 ArcSDE instances in a timely manner.
Dependencies: Dependent on HUIT DBA schedule and Map Library publishing script creation and execution
Risks (description, plan, impact, owner): There is a single knowledgeable HGL developer in LTS, so illness, vacation, etc can delay projects. HGL development is lower priority than production system issues both for LTS OPS and for HUIT DBAs.