Harvard Wiki has been integrated with Group Services.
Wiki administrators: visit IT Help for an overview of the changes to managing groups in your wikis.
Skip to end of metadata
Go to start of metadata

Deployments

Architecture

For reference see diagram at IDS 2017 New Architecture (DEV)

Functional Goals

  1. Continue to expose the IIIF image api and information api.
  2. Continue to expose the legacy IDS api
  3. Continue to provide image delivery service specialized functions (access management, image captions, image size restrictions, delivery of JP2, JPG, and TIF images from the DRS, delivery of pre-created thumbnails)
  4. Decompose the solution into modular components
  5. Move as much as possible of the architecture to the cloud for scalability
  6. Add a Varnish cache to improve performance and reduce image conversion server load
  7. Replace the IDS UI with Mirador or an equivalent IIIF compatible image viewer

Architecture

IDS Browser software

ServiceFunctionsKey technologiesCalls
Browser UI
  • Standard UI (includes metadata)
  • Embeddable UI
Javascript
  • IIIF Presentation API
  • IDS IIIF Image API

IDS Cloud deployed services

ServiceFunctionsKey technologiesCalls
UI Service
  • Delivers html/javascript UI to browser
  • At a later stage, will also migrate the PDS UI delivery to this serbvice
Apache, Mirador, ELBnone
Legacy API Service
  • Exposes legacy IDS API
  • Transforms legacy URL pattern into IIIF URL pattern
    • Initial release will proxy to the legacy IDS running on the old IDS machines
  • Redirects to IIIF Image API service
  • Will run the same machine as the IIIF Image API Service, but in a separate tomcat
Tomcat 8, Java 8IIIF Image API service
IIIF Image API Service

Django, python, AWS, ELB

 

DRS database

AMS

Varnish

Delivery file cache

Delivery file cache
  • Caches JPEG and GIF delivery files based on input URL
  • Calls Image Transformation Server if URL not found in cache
VarnishIDS Image Transformation Server
IIIF Presentation API
  • Existing IIIF presentation API service enhanced to generate manifests dynamically for single images
  • In the future may serve manifests with multiple images for 2 sided originals and/or VIA records
Apache, django, pythonDRS Solr Service

LTS hosted services

ServiceFunctionsKey technologiesCalls
IDS Image Transformation Server
  • Exposes IIIF image API
  • Adds captions as necessary to returned jpeg images
  • Returns TIF->JPG conversion via Loris
  • Returns DRS JPG images via Loris
  • In the future, we hope to add a DRS ingest step to create JP2 deliverable images for all JPG and TIF deliverable images, ans eliminate TIF->GIF
  • Handles color profile mapping for sRGB
  • Apache, Docker, Ubuntu linux, Loris, DJango, Python, Kakadu
  • Caption processing and TIF–>GIF may require tomcat/java subservice
Mounts DRS File System

Open implementation Issues

ServiceIssueOptions
Apache configurationsNeed to specify exactly the URLs, ports, and redirects required for ELB and each machineAnthony to do?
IIIF Image API Service?Assuming we initially retain the pre-cached thumbnails, which layer will be responsible for choosing and returning them?
  1. IIIF image API service - prior to even going to Varnish
  2. Varnish - would be logically preferable if this is doable with a plug in to Varnish
DRS Solr ServiceIIIF uses a DRS solr service to obtain metadata. That service will not work for IDS imagesAnthony to request DRS team to add a solr service call for IDS files, similar to PDS, but without the structmap
IIIF image API ServiceDecide on python or Java

Chip is creating a pros/cons list so we can decide.

python/django pros - one language/container/architecture for all of IDS, faster development cycle

java/tomcat - more java devs at LTS

Image Viewers

Ideas for inspiration

ExampleSample URLNotes
Stanford viewerhttps://searchworks.stanford.edu/view/br443jf2607
  • openseadragon viewer with zoom/pan
  • popup metadata panel, with
    • IIIF drag and drop link
    • persistent URL
    • rights statement
  • full screen mode
  • self documenting embed code
  • caption option
  • image download
Miradorhttps://iiif.lib.harvard.edu/manifests/view/drs:13182628$5i
  • openseadragon (OSD) viewer with zoom/pan/rotate
  • print to PDF
  • Cite
  • Related links
  • Help
Universal Viewerhttp://universalviewer.io/examples/ 
MooViewer

http://iipimage.sourceforge.net/documentation/iipmooviewer/

http://iipimage.sourceforge.net/demo

This viewer comes with IIPImage. Works with IIIF. looks clunkier than Stanford OSD viewer

 

Image Servers

Server Options

Image Server Architecture Consideration: https://github.com/loris-imageserver/loris/issues/217

ServerURLIIIF complianceNative viewerLicenseNotes
Loris

https://github.com/loris-imageserver/loris

http://libimages.princeton.edu/loris2/pudl0001%2F5138415%2F00000011.jp2/full/full/0/default.jpg

http://libimages.princeton.edu/osd-demo/

 

Image API 2.0 Level 2noneBSD-2
  • python 2.7
  • Kakadu jp2 decoder
  • Native IIIF support by one of the IIIF editors
IIPImagehttp://iipimage.sourceforge.net/Image API 2.0 Level1 (almost all Level 2 except PNG export)

IIPMooViewer

, with context thumbnail (http://merovingio.c2rmf.cnrs.fr/iipimage/openseadragon/ )
GPL V3
  • C++ code, redhat version available
Cantaloupehttps://medusa-project.github.io/cantaloupe/   

written in java

 

JP2 Decoder

Kakadu  - http://kakadusoftware.com/

Image Server Deployment

Prototype instances

Configuration

configuration filecontents (for Loris)
docker-compose.ymlversion: '2'
services:
  ids_loris:
    build:
      context: .
      dockerfile: Dockerfile
    image: lts/loris:0.1.5
    volumes:
      - ../Images:/usr/local/share/images
      - /drs2dev/drsfs:/usr/local/share/images/drsfs:ro
    ports:
      - "10888:80"
supervisor.conf

[supervisord]

nodaemon=true

[program:apache2]

command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"

default.conf

ExpiresActive On

ExpiresDefault "access plus 5184000 seconds"

AllowEncodedSlashes On

WSGIDaemonProcess loris2 user=loris group=loris processes=10 threads=5 maximum-requests=10000

WSGIScriptAlias /loris /var/www/loris2/loris2.wsgi

WSGIProcessGroup loris2

 

Docker instructions

You will need to keep in mind that a folder will need to exist at the same level as the git repositories called Images.  This folder should contain your test images.  I do not have the images in the git repository.

  1. Download and install Docker Community Edition: https://www.docker.com/community-edition#/download
  2. Start Docker Community Edition
  3. Checkout Loris and IIP Image docker containers from internal github:
  4. From a working folder such as ~/IDSDev/ run these commands
    1. mkdir Loris
    2. cd Loris
    3. git clone ssh://<YOURID>@github.hul.harvard.edu/home/git/IDS_Loris
    4. git checkout IDS_Loris
    5. docker-compose build
    6. docker-compose up –d
    7. cd ..
    8. mkdir IIPImage
    9. cd IIPImage
    10. git clone ssh://<YOURID>@github.hul.harvard.edu/home/git/IIPImage
    11. git checkout IIPImage
    12. docker-compose build
    13. docker-compose up –d

After you start the machines you can shut them down with docker-compose down from inside their folders, or you can stop them from the kitematic tool that comes with the engine.  There are other commands that make life easier:

  1. docker ps
  2. docker stop <imagename>
  3. docker exec –it <imagename> bash

Image Server Evaluation Criteria

Meets Luratech functionality benchmarks, and preferably meets or exceeds Luratech performance benchmarks

ClassCriteriaLorisIIPImageLuratech
Performance

Converts test images listed at https://docs.google.com/a/harvard.edu/spreadsheets/d/1exr7-CIAQXeyVIY8MNoRPXeHS-ecu5ZDF7_90EMEbaA/edit?usp=sharing with acceptable performance (more test images needed)  (See IDS Performance test results )

  yes
 

Ability to extract 1024x1024 tiles from the above images at Luratech speeds, eg

https://ids.lib.harvard.edu/ids/iiif/48309578/2048,1024,1024,1024/1024,/0/native.jpg

  yes
FunctionalityIIIF compliance level (http://iiif.io/api/image/2.0/compliance )  1.0 compliant, Level 1
 Ability to perform IIIF image operations on JP2s (crop, scale, rotate) for all test images  yes
 Ability to perform IIIF image operations on JPGs (crop, scale, rotate)  limited
 Ability to perform IIIF image operations on tiffs or gifs (crop, scale, rotate)  limited
 Ease of integration with OpenSeadragon UI  yes
 Embeds sRGB color profile in delivered JPEGs (Bill Comstock requirement)  no
 Can embed captions in the delivered JPEG  yes
 Able to implement the existing IDS URLs, documented at Linking to JPEG2000 Images in the DRS  yes
 Scales on a Luratech equivalent server to handle 300 simultaneous image display requests  unknown
ReliabilityShould be able to cycle through displaying a test set of 100 different images reliably up to 1,000,000 image displays without reduction in performance (question)  unknown
 Should be able to display full images and 1024 x 1024 tiles for all images in the DRS  yes

 

Note on sRGB profile embedding

it is important that image server applications generate JPEG files that maintain the color rendering directives encoded into their source JP2 files. This could be accomplished in two ways, and applications should provide both options:

1) When image servers generate JPEG format files, the application needs to also tag the file with the same ICC display profile as the source JP2, or

2) Image servers should generate JPEG files tagged with an sRGB ICC profile by performing an accurate profile-to-profile color-encoding transformation between the source JP2 file's ICC display profile and sRGB.

https://drs2.lib.harvard.edu:9400/drs2_webadmin/file?fileId=430942035#summary is a test image . If the image server operates as desired, the image will be rendered with the color names aligned correctly with the color wheel, WHEN rendered in a browser that respects ICC profiles.

The key is not to just embed sRGB in to any file. The file must be encoded to sRGB for that option to be valid. The delivery system should tag non-sRGB files with the appropriate ICC profile. For delivery, I still think convert to sRGB (if not already sRGB) and tag with sRGB is best. Ideally the delivery system supports both options and we can choose which we prefer and when it may be appropriate to switch the default.Con

  • No labels