Duplicate DB name Updating to 2021.10 from 2020.03

Once you have built the full-chain cert, simply re-run the installer again.

Ah, of course! Thank you again.

Now the ansible script goes through without fail and I can get to the front end through the browser.
But there is no keycloak-users.csv and I cannot log in with the other usernames or passwords that we have on file.
I have this section uncommented in the yml file:

  # to update older installations,
  # uncomment the following lines...
  migrations:
  - name: "import-legacy-users"
    args:
     tenant_id: "eaasi"
     orgname: "EaaSI"

I don’t see anything in the tasks run that are related to users. Perhaps something was skipped in the ansible script? I don’t see anything related to users. I have attached that file.
eaasi-ansible-playbook-output.txt (100.3 KB)

In the server.log I see that six migrations were skipped but they don’t see relevant to users.

2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-image-index'
2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-emulator-index'
2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-image-archive-v1'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-local-emil-environments'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-emil-database-v1'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'create-absent-emil-environments'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Found 6 available migration(s)
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Running 6 configured migration(s)...
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-image-index'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-emulator-index'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-image-archive-v1'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-local-emil-environments'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-emil-database-v1'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'create-absent-emil-environments'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [SERVER-LIFECYCLE-HOOKS] Application started!
2022-03-04 15:27:31.489 |W| (default task-1) [Authentication] Authentication token is missing! Continue as anonymous user.
2022-03-04 15:27:31.493 |I| (default task-1) [EMIL] Preparing environment-repository...

OK, great! Now let’s try to find out why login fails.

That specific migration of legacy users is executed outside of our backend, so we need to check the following log instead:

$ sudo journalctl -u eaas | grep eaasi-web-api > eaasi-web-api.log

Please post the produced log file here.

ty
Most of it looks like this:

Mar 04 10:25:22 eaasi-prod docker-compose[2790775]: Stopping eaasi-web-api   ...
Mar 04 10:25:23 eaasi-prod docker-compose[2784958]: eaasi-web-api exited with code 143
Mar 04 10:25:26 eaasi-prod docker-compose[2790775]: Removing eaasi-web-api   ...
Mar 04 10:26:57 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Waiting for database at 'eaasi-database:5432'...
Mar 04 10:26:57 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Running database migrations...
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      |
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Sequelize CLI [Node: 16.13.1, CLI: 6.2.0, ORM: 6.7.0]
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      |
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Loaded configuration file "dist/data_access/config/config.js".
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Using environment "production".
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | No migrations were executed, database schema was already up to date.
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice New minor version of npm available! 8.1.2 -> 8.5.3
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.3>
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice Run `npm install -g npm@8.5.3` to update!
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Executing 'node ./dist/app.js'...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Initializing EaaSI Web-API (production)...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Preparing database...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Starting HTTP server...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Server is listening on port 8081
Mar 06 01:42:33 eaasi-prod docker-compose[2794082]: eaasi-web-api      | (node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
Mar 06 01:42:33 eaasi-prod docker-compose[2794082]: eaasi-web-api      | (Use `node --trace-warnings ...` to show where the warning was created)

eaasi-web-api.log (19.9 KB)

Yes, the log does not contain any interesting information… But, let’s check the database content directly then.

Please post the output of the following commands:

$ sudo docker exec -it eaasi-database psql eaasi webapi -c '\dt'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE "SequelizeMeta"'

sure thing.

kiritharan@eaasi-prod:~$ sudo docker exec -it eaasi-database psql eaasi webapi -c '\dt'
                     List of relations
 Schema |               Name               | Type  | Owner  
--------+----------------------------------+-------+--------
 public | SequelizeMeta                    | table | webapi
 public | application_log                  | table | webapi
 public | bookmark                         | table | webapi
 public | eaasi_role                       | table | webapi
 public | eaasi_task                       | table | webapi
 public | emulation_project                | table | webapi
 public | emulation_project_resource       | table | webapi
 public | emulation_project_task_successor | table | webapi
 public | user_imported_content            | table | webapi
 public | user_imported_environment        | table | webapi
 public | user_imported_image              | table | webapi
 public | user_imported_software           | table | webapi
(12 rows)

kiritharan@eaasi-prod:~$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE "SequelizeMeta"'
                           name                            
-----------------------------------------------------------
 000-create-eaasiRole.js
 003-create-bookmark.js
 004-create-eaasiTask.js
 008-create-applicationLog.js
 20200618161213-create-emulation-project.js
 20200618161225-create-emulation-project-resource.js
 20200618161227-create-emulation-project-task-successor.js
 20211020193035-insert-default-eaasi-roles.js
 20211020201524-delete-legacy-tables.js
 20211020213045-delete-emulators.js
 20211020213740-delete-temp-environments.js
 20211020213910-delete-imported-software-resources.js
 20211021141205-update-eaasi-tasks.js
 20211022152408-rename-tables-for-imported-resources.js
 20211022160402-rename-columns-for-imported-resources.js
 20211022164522-rename-columns-for-bookmarks.js
 20211109170504-create-keycloak-users.js
(17 rows)

I have the impression, that your database (which the installer attempted to migrate) was empty and did not contain any users. Do you have a backup available?

Let’s also check the content of some tables to verify this:

$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_content'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_image'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_software'

Do not post the output, just check that those tables are not empty!

Each of those tables are empty and return no rows.
The database should not be empty as we did have an active user from last March… But I had not been involved very much with this project since 2020.

I looked at this other ticket I had made earlier and assumed we would have some content available:

But when I look within the users folder of /eaasi/image-archive/meta-data/user I don’t see anything…
Is everything we had in here lost now?
I have a backup of the directory structure from October of 2020, that may suffice? When I look at the user folder there I see there is an xml file within the user folder.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:emulationEnvironment xmlns:ns2="http://bwfla.bwl.de/common/datatypes">
    <ns2:id>1392752e-ef8d-4cd6-a4f8-5d31ddb2e25d</ns2:id>
    <ns2:timestamp>2019-10-16T18:20:57.280Z</ns2:timestamp>
    <ns2:description>
        <ns2:title>Mac OS 7.53 + CAD tools</ns2:title>
...

which seems to contain a project. I have an sql dump as well from the same date in 2020.
Would I put this in place of the current eaasi home and run update again? I have a feeling this data may be two versions behind.
thanks again for your help.

Actually, image-archive’s data is not related to the missing users in that specific case. Most of your images and metadata should have been moved to /eaasi/minio/data/image-archive during the upgrade.

But, in version 2020.03 all users were stored in a separate PostgreSQL database and managed by the EaaSI-UI directly. In version 2021.10 we have introduced a specialized OAuth service for managing users. Migrating users from the old legacy database to the new service is required to make previously created user accounts available after the upgrade.

You should try to restore the legacy EaaSI-UI’s database content from your backup and then re-run the installer again. It should be enough to restore a single directory located at /eaasi/portal for this.

Thanks,
@oooleg How can I restore the legacy Eaasi-UI database content? If the version is no longer using postgresql for users, where would I run this sql file before updating?

If you have a directory located at /eaasi/portal in your backup, then just restore it as-is to your EaaSI installation directory. The installer should then migrate its content correctly.

alright~ I got the keycloak-users file to be generated!
After copying the ENTIRE backup folder, not just the eaasbackup/portal folder.

and so I logged in using the keycloak thing there. I noticed that the contents are a little strange

username,email,password
eaasi_group,eaasi@andrew.cmu.edu,##
zfurste,zfurste@andrew.cmu.edu,##
zfurste,zfurste@andrew.cmu.edu,##
zfurste,zfurste@andrew.cmu.edu,##
zfurste,zfurste@andrew.cmu.edu,##
zfurste,zfurste@andrew.cmu.edu,##

several users have the same exact name.

When I log in as the eaasi_group user and the bottom-most “zfurste” user I don’t see any images saved when I go to
image
and when i go to

I think there should be some resources because when I was looking at the backup folder I found these two iso files in the objects folder:

Binary file eaas-home-backup-Oct152020/objects/Eric_MicrosoftEncarta98/iso/Encarta98_2.iso matches
Binary file eaas-home-backup-Oct152020/objects/Eric_MicrosoftEncarta98/iso/Encarta98_1.iso matches

Do I just have to recreate these or should they have been imported when doing the update?

Thanks again for all your help!

Yeah, that list of users looks strange indeed. Could you also send me the corresponding log as described earlier?

That call did not return anything

$ sudo journalctl -u eaas | grep eaasi-web-api > eaasi-web-api.log
kiritharan@eaasi-prod:~$ 

Should it be grepping for eaasi-web-api or something else for the database?

Here are also the two database calls looking at the serialized data and below is the server log.

$ sudo docker exec -it eaasi-database psql eaasi webapi -c '\dt'
                     List of relations
 Schema |               Name               | Type  | Owner  
--------+----------------------------------+-------+--------
 public | SequelizeMeta                    | table | webapi
 public | application_log                  | table | webapi
 public | bookmark                         | table | webapi
 public | eaasi_role                       | table | webapi
 public | eaasi_task                       | table | webapi
 public | emulation_project                | table | webapi
 public | emulation_project_resource       | table | webapi
 public | emulation_project_task_successor | table | webapi
 public | user_imported_content            | table | webapi
 public | user_imported_environment        | table | webapi
 public | user_imported_image              | table | webapi
 public | user_imported_software           | table | webapi
(12 rows)


$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE "SequelizeMeta"'
                           name                            
-----------------------------------------------------------
 000-create-eaasiRole.js
 000-create-eaasiTask.js
 000-create-eventType.js
 000-create-language.js
 000-create-machineInterface.js
 000-create-mimeType.js
 000-create-operationType.js
 000-create-unitOfInformation.js
 000-create_country.js
 000-create_mediaType.js
 000-create_mimeType.js
 001-create-audioDevice.js
 001-create-eaasiUser.js
 002-create-colorDepth.js
 002-create-eaasiUserHash.js
 002-create-imported-software-resource.js
 003-create-bookmark.js
 003-create-chipset.js
 003-create-configNetwork.js
 003-create-frequencyUnit.js
 004-create-cpuArchitecture.js
 004-create-emulator.js
 005-create-contentObject.js
 006-create-displayDevice.js
 007-create-displayInterface.js
 007-create-displayRes.js
 008-create-fileExtension.js
 009-create-fileFormat.js
 010-create-fileSystem.js
 011-create-developer.js
 012-create-file.js
 012-create-fmtOperation.js
 013-create-gpuDevice.js
 014-create-kbLayout.js
 015-create-machineType.js
 017-create-networkDevice.js
 018-create-networkService.js
 019-create-objFileOp.js
 019-create-systemRequirements.js
 020-create-osVersion.js
 021-create-pointerDevType.js
 022-create-programLang.js
 023-create-readWriteStatus.js
 024-create-region.js
 025-create-softwareLicense.js
 026-create-softwareObject.js
 027-create-softwareProduct.js
 028-create-softwareType.js
 029-create-storageDeviceType.js
 030-create-timezone.js
 031-create-timezoneLabel.js
 032-create-userInformation.js
 034-create-softwareVersion.js
 035-create-configuredOS.js
 036-create-configSoftware.js
 036-create-processorDevice.js
 037-create-configMachine.js
 038-create_audioDevDrivSoft.js
 039-create-pointerDevice.js
 041-create-softwareEnv.js
 042-create-audioDevHasEquiv.js
 043-create-audioDevMachInterface.js
 044-create-computingEnv.js
 045-create-configAudioDev.js
 046-create-configGpuDev.js
 049-create-keyboardDevice.js
 050-create-cfgKeyboardDevice.js
 051-create-cfgGpuDevHasDispDev.js
 053-create-cfgNetworkEmulatesNetworkSvc.js
 054-create-cfgNetworkHasCfgMach.js
 056-create-cfgNetworkDevice.js
 057-create-cfgNetworkMachExpNwSvc.js
 059-create-fileExtension.js
 060-create-storageDevice.js
 061-create-fmtImplementation.js
 064-create-cfgOsHasUserInfo.js
 065-create-configuredOsLang.js
 066-create-cfgPointerDevice.js
 068-create-cfgSwHasUserInfo.js
 069-create-cfgSwUsesFmtImplementation.js
 070-create-cfgStorageDevice.js
 071-create-contentEnv.js
 072-create-osVersColorDepthSettings.js
 073-create-osVersKeyboardLangSettings.js
 073-create-osVersKeyboardLayoutSettings.js
 074-create-osVersDisplayResSettings.js
 075-create-osVersKbLayoutSettings.js
 075-create-osVersLangSettings.js
 076-create-osVersRegionSettings.js
 077-create-softwareVersLangSettings.js
 080-create-contentObjHasObjFile.js
 081-create-dispDevHasColorDepth.js
 082-create-dispDevHasDispInterface.js
 083-create-dispDevHasDriverSoftware.js
 084-create-dispDevHasDispResolution.js
 087-create-fileFmtHasFileExt.js
 089-create-gpuDevHasDispInterface.js
 090-create-gpuDevHasDriverSoftware.js
 091-create-gpuDevHasMachInterface.js
 092-create-gpuDevHasEquivalent.js
 093-create-KeybDevHasDriverSw.js
 094-create-KeybDevHasLang.js
 095-create-KeybDevHasMachInterface.js
 096-create-networkDevHasDriverSw.js
 097-create-networkDevHasMachInterface.js
 100-create-fileOperation.js
 100-create-osVerHasAltID.js
 102-create-osVerHasProgLang.js
 103-create-osVerHasSwLic.js
 104-create-osVerIncSwVer.js
 105-create-osVerIsCompatWCfgMach.js
 106-create-osVerHasTimezSet.js
 107-create-ptrDevHasDriverSw.js
 108-create-ptrDevHasMachInterface.js
 109-create-swEnvHasPartConfgSw.js
 110-create-swEnvHasDiskImage.js
 112-create-swFamVerHasPtSwVer.js
 113-create-swFamHasPtSwProd.js
 114-create-swObjIsManifestOsVer.js
 115-create-swObjHasAltId.js
 117-create-swObjHasObjFile.js
 119-create-swObjIsManifestSwVer.js
 120-create-swProdHasAltName.js
 121-create-swProdHasSwType.js
 122-create-swVerHasAltId.js
 122-create-swVerHasAltName.js
 123-create-swVerHasDev.js
 123-create-swVerHasFileOp.js
 123-create-swVerHasLangSettings.js
 123-create-swVerHasSysReq.js
 125-create-SwVerHasFmtImp.js
 126-create-SwVerHasProgLang.js
 127-create-SwVerHasSwLic.js
 127-create-SwVerIncSwVer.js
 128-create-swVerCompatCpuEnv.js
 129-create-storageDevHasDriverSw.js
 130-create-storageDevHasMachInterface.js
 131-create-sysReqIncAudioDev.js
 132-create-sysReqNetworkDev.js
 133-create-sysReqIncGpuDev.js
 134-create-sysReqProcessor.js
 136-create-sysReqInPtrDevType.js
 137-create-sysReqInSwVer.js
 138-create-sysReqIncStorageDevTyp.js
 139-create-sysReqKeyboardDev.js
 143-create-applicationLog.js
 145-create-fileOperationHasFileExt.js
 146-create-fileOperationNearestFileFmt.js
 148-create-processorDeviceHasDriverSw.js
 148-create-processorDeviceHasMachInter.js
 148-create-recommendationLevel.js
 149-create-configuredDisk.js
 150-create-configuredDiskHasPartition.js
 151-create-machineRecommendation.js
 156-create-fileFormatHasMimeType.js
 004-create-eaasiTask.js
 008-create-applicationLog.js
 20200618161213-create-emulation-project.js
 20200618161225-create-emulation-project-resource.js
 20200618161227-create-emulation-project-task-successor.js
 20211020193035-insert-default-eaasi-roles.js
 20211020201524-delete-legacy-tables.js
 20211020213045-delete-emulators.js
 20211020213740-delete-temp-environments.js
 20211020213910-delete-imported-software-resources.js
 20211021141205-update-eaasi-tasks.js
 20211022152408-rename-tables-for-imported-resources.js
 20211022160402-rename-columns-for-imported-resources.js
 20211022164522-rename-columns-for-bookmarks.js
 20211109170504-create-keycloak-users.js
(170 rows)

servermarch172022.log (552.1 KB)

@oooleg do you need any other documentation from me on this?
I am not sure where else to look to see that our old images can be accessible.

My apologies, @oooleg
There was output for that command I just was looking at this again and realized it was being written to a file, not to be printed out… My bad.
I see a couple of FetchError: invalid json response in the March 16 attempt.

eaasi-web-api.logmarch28 (96.8 KB)

Importing users has been successful according to the log, but no ownership information has been found in the database:

Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Imported 6 legacy user(s)
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Updating user-ids in table "bookmark"...
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | --> No records found! Nothing to update.
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Updating user-ids in table "user_imported_content"...
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | --> No records found! Nothing to update.
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Updating user-ids in table "user_imported_software"...
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | --> No records found! Nothing to update.
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Updating user-ids in table "user_imported_environment"...
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | --> No records found! Nothing to update.
Mar 16 00:02:54 eaasi-prod docker-compose[1737132]: eaasi-web-api      | Table "user_imported_image" not found! Skipping it.

I’m not sure, how that can be fixed now. Your backed up database seems to be incomplete or broken. Do you have any other copies?

Another problem appears to be, that the back-end failed to import legacy image-archive, because your server ran out of storage.

looking at the .sql file for the database dump i can confirm that the user imported environments are missing…

--
-- Data for Name: user_imported_environment; Type: TABLE DATA; Schema: public; Owner: eaasi_dev
--

COPY public.user_imported_environment (id, "userID", "eaasiID", "createdAt", "updatedAt") FROM stdin;
\.

there would normally be lines of data below that copy but there’s nothing.
I guess the dump is incomplete. I don’t have any other copies. I think we may just have to manually add the images that are within the objects folder here

thanks again for your help i think this issue can be closed unless you have any other suggestions :slight_smile:

Yes, the simplest solution would probably be to add specific ownership records for each resource into the user_imported_* tables manually. Just specify resource IDs as eaasiID and assign them to user IDs from Keycloak as reported by the migration under:

Mar 15 14:09:06 eaasi-prod docker-compose[1677783]: eaasi-web-api      | Importing legacy users into Keycloak...
Mar 15 14:09:07 eaasi-prod docker-compose[1677783]: eaasi-web-api      | --> User created: OLD-USER-ID -> NEW-USER-ID
...

The above snippet means, that a legacy user with ID OLD-USER-ID has been mapped to a user in Keycloak with ID NEW-USER-ID, so your tables should now use these newly allocated IDs as userID.

Also, ownership records for different resource kinds should be added to corresponding tables to work properly. Object resources are located at <eaasi-install-dir>/objects and records should be added to table user_imported_content. Images on the other hand are located at <eaasi-install-dir>/minio/data/image-archive/images and records should be added to table user_imported_image. Environments are located at <eaasi-install-dir>/minio/data/image-archive/environments and records should be added for each private environment to table user_imported_environment.