EaaSI Development Update - February 2023

Program Announcements
1

Development work in February largely focused on planned upgrades to Yale’s EaaSI node, using a copy of the data and configuration on the live/production instance on a test instance and bringing it from v2019.11 to the current codebase. In testing, attempts to migrate the relatively large amount of data and resources on that server to EaaS’ new S3-compatible image and object storage structure revealed a number of inefficiencies and bugs in the eaasi-installer. Fixes have been found and pushed and will be part of the planned v2023.[xx] release process.

  • Recent work accomplished (January 2023 - February 2023)

    • Completed fixes with low-resourced EaaSI deployments running out of socket descriptors (see last month’s update, no further regressions found
    • Reworked design for parallelizing running processes during eaasi-installer migrations
      • EaaS application runtime was not correctly picking up on the eaasi-installer efforts to run multiple processes in parallel, resulting in serial task processing and exceptionally long (multiple day) running times for multiple TBs of data during an upgrade run
      • with fixes, eaasi-installer upgrade runs dropped to ~4 hours per run on Yale’s test server with multiple TBs of legacy data
    • adjusted eaasi-installer migrations to account for recent versions of qemu-img dropping support for auto-detection of disk image file format when re-basing QCOW image chains (necessary for migrating image storage URLs from absolute to relative file paths)
    • improved error handling of re-basing environment QCOW image chains if one or more intermediary layers are not found/reachable at time of migration
    • troubleshot and prepped a work-around for out-of-memory (RAM) errors encountered when trying to migrate large amounts of data
      • sourced to upstream behavior in the Wildfly HTTP client, an EaaS dependency
      • Wildfly POST requests necessary for storage migrations were redundantly storing data on persistent storage, and on occasion were not properly cleaning up already-migrated data
      • resulting fixes were incorporated into both eaasi-installer migrations and more broadly across the EaaS back-end; should result in more stable large-file handling, including uploads via the EaaSI UI, in future tagged releases
    • investigated and fixed metadata migration issues with various subsets of Environment and Software resources on Yale’s test node
    • investigated auto-generating change logs based on GitLab commits to assist with providing release notes and dev updates
    • provided fixes to Emulation-as-a-Service’s environment-proposer API, the core of proposed future “UVI” services
      • environment-proposer API had broken at unknown point in tagged EaaSI releases between 2019.11 and 2021.10
      • adjusted API to work with new S3-compatible storage implementation
      • the environment-proposer API and related experimental UVI workflows will be added to the API docs EaaSI User Handbook following v2023.[xx] tagged release

  • Forum tickets/bugs resolved

  • Expected work (February - March 2023)

    • continue working with Dual Lab to implement UI elements and test “Advanced” mode of EaaSI UI Emulation Project interface
    • continue + complete debugging of back-end data migrations necessary to upgrade Yale’s test instance of EaaSI from v2019.11 to current codebase
    • test user-facing workflows on Yale’s test instance of EaaSI
      • depending on pace/results of QC on user-facing workflows: perform upgrade on Yale’s production instance of EaaSI
    • debug issues with legacy metadata encountered attempting to upgrade EaaSI’s deployed version of Keycloak to latest stable release
    • draft release notes for v2023.[xx] and distribute to community, particularly to sysadmins on self-hosted Network nodes to prep for the upgrade process