Difference between revisions of "Re-implementing the MMS"

From Metadata-Registry
Jump to: navigation, search
(Phase IV)
(Re-Implementing the MMS for GEM)
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Re-Implementing the MMS for GEM=
+
=Re-Implementing the MMS=
  
 
==Phase I==
 
==Phase I==
Line 5: Line 5:
 
Current implementation has very tight ties to an Oracle database.  For new, generalized MMS, the functions need to be more abstract, in order to work with any database. The internal user management systems were handed off to an external service in the original implementation. In the re-implementation, roles and rights for users need to be abstracted to work either wholly within the MMS or linked to an external user authentication system.
 
Current implementation has very tight ties to an Oracle database.  For new, generalized MMS, the functions need to be more abstract, in order to work with any database. The internal user management systems were handed off to an external service in the original implementation. In the re-implementation, roles and rights for users need to be abstracted to work either wholly within the MMS or linked to an external user authentication system.
  
#New implementation of the Repository database
+
#New implementation of the main Repository database (JP estimate: 2 weeks)
#New implementation of the internal management database
+
#* Currently based on Oracle 9i. We need to decide whether to go with Oracle's new release (10g called 'Express Edition') which is free for limited use, or switch to MySQL. Regardless of which is chosen for the first re-implementation, the code will be developed to make that decision a local one, with the MMS working with any database.
#Abstracting user and service management and adding to internal management database (using registry model)
+
#* One advantage of sticking with Oracle is that they are coming out with a native RDF database. Another advantage is that some Oracle specific development done for the NSDL can be re-used.
#Setup of simple services structure (similar to Recommender)
+
#* Develop simple ingest process to bring data into Repository
#Implement new harvester
+
#New implementation of the internal management database (JP estimate: 3 days)
 +
#* The current database has been developed over time and is due for re-factoring, though the core ideas are the same.  It is currently based on Qualified DC and it's not clear that anything else would be necessary for this portion of the internal work.
 +
#Abstracting user and service management and adding to internal management database (using registry model) (JP estimate: 4 weeks)
 +
#* User management was based on an incomplete integration with SourceForge which made it impossible to manage users effectively. Service management was limited for some of the same reasons, although internally we set up a framework that was managed manually.
 +
#* A new user management module needs to be developed that can be easily integrated into other user management systems.  This will allow the building of a service management based on lessons learned in the NSDL implementation.
 +
#Setup of simple services infrastructure to support services similar to the current Recommender (JP estimate: 2 weeks)
 +
#* Current Recommender implementation too closely integrated into the MMS management database
 +
#* New implementation will be a separate service harvested into the Repository
 +
#* Other services should be set up on the same model: harvest, change, expose. This includes human mediated services such as editing and alignment services.  CWIS might be an effective start on this (though should be pre-configured).
 +
#Implement new harvester (JP estimate: 3 weeks)<br [[HarvestSteps]]
 +
#* New harvester should be based on existing harvester, modified to be more forgiving of invalid XML, that will report errors down to specific line, enabling effective notification.  Acceptance should cascade from full harvest to statement-by-statement, only rejecting the smallest possible portions of a potential harvest.
 +
#Re-implementing the MMS OAI server (JP estimate: 3 days)
  
 
==Phase II==
 
==Phase II==
  
#Re-implementing the MMS OAI server (exising server needs to have data backend made more abstract to deal with any data schema or source)
 
 
#Adjust the metadata editor to accommodate any metadata schema and work with the simple services structure
 
#Adjust the metadata editor to accommodate any metadata schema and work with the simple services structure
 +
#* Current metadata editor tightly tied to qualified Dublin Core (flat and non-hierarchical) and with the database.  Needs to stand alone as a separate service.
 +
#* Enabling editing for hierarchical schemas (such as LOM) will require additional work
 
#Implement services registry on top of existing components (user management and repository)
 
#Implement services registry on top of existing components (user management and repository)
 +
#* Needs to make a distinction between services and service providers, which doesn't currently exist.
 +
#* Current service setup is unidirectional (only works with services it controls). Also needs to register service interfaces that can be managed by the MMS (Lenny)
 +
#* Explicit round trip services
 
#Implement error handling and notifications
 
#Implement error handling and notifications
 +
#* Agents choose preferred notification type and methodology (email, RSS, etc.)
 +
#* Defaults are configured by agent role
 
#Setup safe transformation service, generalized to any repository
 
#Setup safe transformation service, generalized to any repository
 +
#* Another simple service separately configured
 +
#* Based on NSDL safe transforms
 +
#Upgrade the MMS OAI server
 +
#* Exising server needs to have data backend made more abstract to deal with any data schema or source
 +
#* Needs to be distributable for use with any system
  
 
==Phase III==
 
==Phase III==
  
#Implement harvest diagnostics and helpdesk functionality
+
#Implement comprehensive harvest diagnostics, validity checkers and helpdesk functionality
 +
#* New or changed harvest generate statistical summary and sample or full export file for evaluation tool (generated on the fly)
 +
#* May use separate third party software for helpdesk
 
#Implement distributed GEM community services (includes OAI server capable of working on many platforms, plugin data connections, etc.)
 
#Implement distributed GEM community services (includes OAI server capable of working on many platforms, plugin data connections, etc.)
#Implement comprehensive evaluation and test suite, validity checkers
+
#Implement collection-specific transforms
+
 
 
==Phase IV==
 
==Phase IV==
  

Latest revision as of 08:38, 16 October 2007

Re-Implementing the MMS

Phase I

Current implementation has very tight ties to an Oracle database. For new, generalized MMS, the functions need to be more abstract, in order to work with any database. The internal user management systems were handed off to an external service in the original implementation. In the re-implementation, roles and rights for users need to be abstracted to work either wholly within the MMS or linked to an external user authentication system.

  1. New implementation of the main Repository database (JP estimate: 2 weeks)
    • Currently based on Oracle 9i. We need to decide whether to go with Oracle's new release (10g called 'Express Edition') which is free for limited use, or switch to MySQL. Regardless of which is chosen for the first re-implementation, the code will be developed to make that decision a local one, with the MMS working with any database.
    • One advantage of sticking with Oracle is that they are coming out with a native RDF database. Another advantage is that some Oracle specific development done for the NSDL can be re-used.
    • Develop simple ingest process to bring data into Repository
  2. New implementation of the internal management database (JP estimate: 3 days)
    • The current database has been developed over time and is due for re-factoring, though the core ideas are the same. It is currently based on Qualified DC and it's not clear that anything else would be necessary for this portion of the internal work.
  3. Abstracting user and service management and adding to internal management database (using registry model) (JP estimate: 4 weeks)
    • User management was based on an incomplete integration with SourceForge which made it impossible to manage users effectively. Service management was limited for some of the same reasons, although internally we set up a framework that was managed manually.
    • A new user management module needs to be developed that can be easily integrated into other user management systems. This will allow the building of a service management based on lessons learned in the NSDL implementation.
  4. Setup of simple services infrastructure to support services similar to the current Recommender (JP estimate: 2 weeks)
    • Current Recommender implementation too closely integrated into the MMS management database
    • New implementation will be a separate service harvested into the Repository
    • Other services should be set up on the same model: harvest, change, expose. This includes human mediated services such as editing and alignment services. CWIS might be an effective start on this (though should be pre-configured).
  5. Implement new harvester (JP estimate: 3 weeks)<br HarvestSteps
    • New harvester should be based on existing harvester, modified to be more forgiving of invalid XML, that will report errors down to specific line, enabling effective notification. Acceptance should cascade from full harvest to statement-by-statement, only rejecting the smallest possible portions of a potential harvest.
  6. Re-implementing the MMS OAI server (JP estimate: 3 days)

Phase II

  1. Adjust the metadata editor to accommodate any metadata schema and work with the simple services structure
    • Current metadata editor tightly tied to qualified Dublin Core (flat and non-hierarchical) and with the database. Needs to stand alone as a separate service.
    • Enabling editing for hierarchical schemas (such as LOM) will require additional work
  2. Implement services registry on top of existing components (user management and repository)
    • Needs to make a distinction between services and service providers, which doesn't currently exist.
    • Current service setup is unidirectional (only works with services it controls). Also needs to register service interfaces that can be managed by the MMS (Lenny)
    • Explicit round trip services
  3. Implement error handling and notifications
    • Agents choose preferred notification type and methodology (email, RSS, etc.)
    • Defaults are configured by agent role
  4. Setup safe transformation service, generalized to any repository
    • Another simple service separately configured
    • Based on NSDL safe transforms
  5. Upgrade the MMS OAI server
    • Exising server needs to have data backend made more abstract to deal with any data schema or source
    • Needs to be distributable for use with any system

Phase III

  1. Implement comprehensive harvest diagnostics, validity checkers and helpdesk functionality
    • New or changed harvest generate statistical summary and sample or full export file for evaluation tool (generated on the fly)
    • May use separate third party software for helpdesk
  2. Implement distributed GEM community services (includes OAI server capable of working on many platforms, plugin data connections, etc.)
  3. Implement collection-specific transforms

Phase IV

  1. Implement metadata augmentation, improvement and rating services
  2. Non-OAI data services
  3. Secure OAI services