PhaseSubProject or componentWhoStartCompleteStatusNotes
Hardware/software configuration

Complete setup of the new prod machines: pelham.lib.harvard.edu - HGL Frontend (HUIT hosted VM), mercator.lib.harvard.edu - HGL Oracle Database Backend (HUIT hosted VM)

Tom Scorpa11/9/2015 2/3/2016 done 
 

Complete setup of the base software on the new machines:  tomcat8, java8, ARCGIS server software, SDE and Oracle Database, GDAL (including java bindings)

Tom Scorpa3/23/16 5/3/2016 done

Service Now ticket  INC01661016 

Java 8, Tomcat 7, ArcGIS Server 10.4.1 ( available through my.esri.com. I can generate a new provisioning file. The ArcGIS Server web admin tool should be operational as it's the only entry point to the application. GDAL, with its Java bindings are on ARC.LTS now, and are needed on the new front end box (pelham). The specific command needed most is 'ogr2ogr' which convers GeoJson to ESRI shape files. Ports on SDE LDR, PRD & SHARE should be open to MAPs and GSD.

I think Benson did this already but the ST_RASTER and ST_GEOMETRY libs needs installation on mercator too.

**Do not need to install ArcGIS Engine **Hold off on installing ArcGIS Objects (may have a workaround)

Everything should now be in place.

 Configure machines (ACLs, DRS mounts, shared drive between the two machines, etc.) just like the dev machines (arc.lts.harvard.edu, arcdb.lts.harvard.edu) that were already set up (with prod configuration)Tom Scorpa5/2/2016 6/10/16 done

A shared drive is needed between arc and arcdb as well as one between the new front end server (pelham) and mercator. This is pretty much the same as /hgl_cache

The DRS mounts are needed on pelham too until all data from DRS is migrated to DRS2

For the PROD Cache, i200 GB is good to start with?.We can grow as needed  

/hgl_cache is now in place on pelham/mercator.

 Install new HGL front end and solr under tomcat, plus ancillary java projects on pelhamDave5/2/20166/10/16 doneI have some questions about Solr in a prod environment.
 BrandingDave  doneThe HL "banner" was added. The HGL logo was added. On DEV, not prd yet.
 PIN/AuthenticationDave/Grainne9/15/1610/1/16done add 2 access points
 Install SolrDave  doneInstalled on pelham.lib and arc.lts – Should duplicate exact setup though.
 Install the OGP HarvesterDave XdoneInstalled on Mac and working. Cannot get it operational on any of the VMs. Still looking into it but this is a minor headache,
 Import latest metadata records to SolrDave ongoingdone'Latest' is a bit relative until there's a definite close on any more data loading to 'old' HGL
 Correct UI bugs that prevent download of "Scanned Maps"Dave XdoneCorrected but code needs a pull request
 Python scripts for manipulating Solr recordsDave / Chris / Dani  ongoingSharing scripts with Tufts. Some issues with inconsistent XML codes between Harvard and general community - 'Scanned Map' vs 'Paper Map' in DataType.
 Migrate the HGL tomcat app.to new OGP environmentDave2 weeks doneExternal catalog support. Data download request handling for raster data types. Waiting for DRS2 data to test program.
 OGP 2.0 site configurationDave ongoingdoneMostly complete. Will have some URL issues for swapping pointer to hgl.harvard.edu
 Install GDAL and JNI on pelham and arcTom/Rei11/28/1612/10/16done 
DRS2 supportDRS2 test dataVitaly6/1/169/15/16doneWe need some data in DRS2 to test the HGL data download code. Work stopped, waiting for test data.
 Deploy DRS2 support to old architectureDave12/20/161/31/17will not do 
Database migrationHUIT DBA to migrate database from ArcSDE 9.32 to ArcSDE 10.4 in all three database instances.Venu, Dave7/13/162/3/17doneHGLLDR, and HGLTEST - Huit DBA, HGLPROD -  use alternate approach proposed by Dave and Bonnie
 Test configuration - Add a small amount of migrated data sets to the new ARCGIS/SDEDave2/1/17 donedepends on new HGLPROD DB
 Migrate and synch Oracle Meta TablesDave2/1/17 donedepends on new HGLPROD DB. Table synchronization to occur when data loader comes on board. Other 'LDI' Apps that connect directly to Oracle will need to check 2 databases once the data migration starts.
Data publicationDevelop python publication script and benchmark it on vector layers and raster layersBonnie, Tim, Dave11/1/161/31/17done

Requirements need review by Dave and Bonnie.

How # of tile cache layers should be determined per GIS data set? How to make projection decisions per layer into webmercator projection? How long per layer will this take (we have 14000 layers)? Could be 10 minutes/layer for raster layers, vector could be faster.

 Publish all of the data to the new architecture. using the scriptDave1/2/17?3/1/17 5/1/17doneWe do not have an estimate of how long this will take. At 10 minutes/layer, running 7x24, this would take 120 days.
 Test access to the new ARCGIS server/SDE from GIS desktops and that all HGL functionality is fully replaced.Map Collection3/1/17 5/2/174/1/17 5/19/17done 
 Develop python caching script and benchmark itBonnie, Tim, Dave2/1/173/1/17done 
 Cache hi priority layers using caching scriptDave3/1/174/1/17doneraster layers plus fraction of vector layers (polygons first) - We do not have an estimate of how long this will take.
 Re-format the tile cache disk storageLTS Ops4/11/175/1/17doneNeeded to install XFS
 Design and implement backup strategy for pelham  mounted application disk storageLTS Ops5/1/176/30/17 5/18/17donetile cache and application data will be backed up separately
TestFull deploy new architecture with full dataDave4/1/175/1/17done 
 Cache remaining layers in background using caching scriptDave  doneWe do not have an estimate of how long this will take.
 Beta test at Map Collection and fix hi pri bugsDave5/1/17

5/19/17

6/1/17

in progressBonnie and Marc McGee are testing
 Go LivePublicize syntax for pointing to ArcGIS Server REST based web map servicesDave / Ben TBDto doThis is for CGA WorldMap. Need to sit down with their developer to discuss.
 Turnover meeting at LTSRandy/Dave

5/22/17

6/13/17

5/22/17

6/13/17

to doWith LTS OPs and LTS Support
 Deploy to production at hgl.harvard.eduDave, LTS OPS

5/22/17

6/13/17

5/22/17

6/13/17

to doIts possible that the new HGL front end (OGP 2.0) will point to the new backend for the new data, but at the same time the new front end will point to the old HGL for most of the data, which has not yet been migrated to the new architecture
 Finally, we can shut down the old HGL.LTS OPS6/1/176/30/17to do 
 Post-release priorities     
 Design and implement backup strategy for pelham  mounted tile cache disk storageLTS Ops5/1/176/30/17in progresstile cache and application data will be backed up separately
 Support DRS2  6/30/17to doDevelopment is complete, needs final test and deploy to prod
 Set up change control for the new HGL  FY18 

Implement Stage, Publish, Mov2Prod, and Rollback scripts

 Bulk ingest tool for new architecture  FY18  
 Enhancements to the OGP 2.0 UI, plus OGP2 UI changes as identified in usability testingDave FY18 As requested by the Map Library. Trying to get Tufts to accept some of these changes so the project branches do not separate too far. Trying to locate the doc that lists all the changes.
 Dashboard and statistics  FY18  
 GeoBlacklight assessment  FY18  
 Migrate from FGDC to ISO metadata standard     
 Work with LTS to incorporate GIS formats in the DRS