Duplicate DB name Updating to 2021.10 from 2020.03

Deployment:
a local node

EaaSI Version:
2020.03 to 2021.10

Browser:
chromium on linux

Description:
When trying to run the update following the documentation here
https://eaasi.gitlab.io/eaasi_user_handbook/overview/install/setup.html#migrating-from-2020-03-to-2021-10
and here
https://eaasi.gitlab.io/eaasi_user_handbook/overview/install/setup.html#updating-eaasi

I kept running in to an issue where it was saying that I had uncommitted changes within a submodule. I should not have changed any contents on my control/deployment machine that hosts the installer so I didn’t understand this. I ended up just performing a git checkout on the affected portion (the docker directory in eaasi-installer/eaas/ansible/docker):

error: Your local changes to the following files would be overwritten by checkout:
	docker/alpine/Dockerfile
	docker/ansible/data/installer.sh
	docker/ansible/data/py-deps.txt
Please commit your changes or stash them before you switch branches.
Aborting
Submodule path 'eaasi-ui/ansible': checked out 'b1de8a68a7ca76450bc5959f9e9a42eea3f8938c'
Unable to checkout '60885d444fce169e7e342963c74dc7b58452dc1c' in submodule path 'eaas/ansible'

After performing the git checkout updating the template to my configurations and uncommenting the update lines on the bottom, the gitpull.sh script worked. I ran the update.sh script and it worked until it got to the portion about spinning up the database and I received the error i’ve included as a txt file
eaasi-dockerdb-error.txt (5.7 KB)

After hanging for a while it said: “Error response from daemon: Container e8802f56c6b758dc6125f7869307c99d0aea7637416430beacf1f5dd7487fb08 is not running”
and then

ERROR: for eaasi-database  Cannot create container for service eaasi-database: Conflict. The container name "/eaasi-database" is already in use by container "e8802f56c6b758dc6125f7869307c99d0aea7637416430beacf1f5dd7487fb08". You have to remove (or rename) that container to be able to reuse that name.

There isn’t a place in the config/.eaasi.yaml file where i can update the database name, and I’m not sure if I should because perhaps my data wouldn’t get transferred?

There are only two containers running when I go on to the eaasi target server and run docker ps, nginx and eaas-custom.

I had a previous issue that I thought was ssl - related but the end suggestion was to wait until the next release.

not sure if there’s any other helpful info about my problem here^

Are you able to reproduce the issue or did it happen once? What steps can you take to repeat the issue? What did you expect to occur and what was the actual outcome?

i can keep trying to run the update script but the same thing happens.

Urgency: If possible, please give an indication of how urgently the issue needs to be addressed - is there a timeline or deadline (e.g. upcoming demo, researcher request, etc.) that EaaSI support staff should be aware of?

We have started talking with a CMU faculty member that would like to put a project done on Hypercards to an emulated instance that has the work of JSG Boggs and the Center for the Advancement of Applied Ethics. They’ve sent us the files, we just need to get them up there but this is what we’re using to re-kickoff our involvement in this project. So the sooner the better, but no real timeline or deadline currently.

Please let me know if there’s any additional information you’d like from me.

Thanks,

Jonathan

Hi @kirichacha! It looks like you have some garbage containers on your server left over from previous installation attempts, which still block allocation of container names.

You should try to remove the containers manually, by running the following commands on your target server:

$ sudo systemctl stop eaas
$ sudo docker ps -a

# remove each listed container with:
$ sudo docker rm <container-id-or-name>

Then simply re-run update.sh script again.

Thanks!
It moves past that task but now fails at:

TASK [wait for eaas-server to start up] **************************************************************************************
fatal: [eaas-gateway]: FAILED! => changed=false 
  attempts: 1
  content: ''
  elapsed: 0
  msg: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>'
  redirected: false
  status: -1
  url: https://eaasi-prod.library.cmu.edu/emil/environment-repository/actions/prepare

I see that nginx is running but the connection is refused on 80 and 443.
I can get to 8080 and in the source code see there’s an iframe but it’s hosting a broken site
image

I think it is some sort of cross site scripting error of trying to run an https webpage through an http hosted iframe.

CMU has had issues with its certificates in the past but this doesn’t seem to be the same issue in the log as it was running in to a clear java error.
https://groups.google.com/g/eaasi-tech-talk/c/5uydmrfCeJs/m/iiES6Z-sBAAJ
&
https://groups.google.com/g/eaasi-tech-talk/c/9vTtHP26A1s/m/Kggv3hlLDAAJ

I see that they do exist on the eaas container within this folder: /eaas/certificates/
but the browser doesn’t seem to load it.
image

sorry i keep coming to you with wild ssl errors

@oooleg Any further suggestions here? I tried doing the same method of updating the cacerts to accept the CMU certificates so that https would work but it did not solve the problem.

Sorry, @kirichacha! Your messages somehow went under.

Are all of the containers actually running? What does the following command report:

$ sudo docker ps -a

@oooleg
they seem to be…

$ sudo docker ps -a
CONTAINER ID   IMAGE                                                                 COMMAND                  CREATED       STATUS                          PORTS                                       NAMES
a628c180a274   nginx:stable                                                          "/docker-entrypoint.…"   2 weeks ago   Restarting (1) 57 seconds ago                                               eaasi-nginx
645b18aeeb01   eaas/eaas-appserver:v2021.10-eaasi                                    "/init"                  2 weeks ago   Up 2 weeks                                                                  eaas
ee25b2bf2318   registry.gitlab.com/eaasi/eaasi-client-pub/eaasi-web-api:v2021.10     "docker-entrypoint.s…"   2 weeks ago   Up 2 weeks                      0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   eaasi-web-api
0da8d466c4a7   jboss/keycloak:15.0.2                                                 "/opt/jboss/tools/do…"   2 weeks ago   Up 2 weeks                      8080/tcp, 8443/tcp                          keycloak
32e6f27a5c88   minio/minio:RELEASE.2021-11-03T03-36-36Z                              "/usr/bin/docker-ent…"   2 weeks ago   Up 2 weeks                      9000/tcp                                    minio
899325d1969a   nginx                                                                 "nginx -g 'daemon of…"   2 weeks ago   Up 2 weeks                      80/tcp                                      nginx
060d539145f8   registry.gitlab.com/eaasi/eaasi-client-pub/eaasi-database:v2021.10    "docker-entrypoint.s…"   2 weeks ago   Up 2 weeks                      0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   eaasi-database
666190b761d4   registry.gitlab.com/eaasi/eaasi-client-pub/eaasi-front-end:v2021.10   "/docker-entrypoint.…"   2 weeks ago   Up 2 weeks                      0.0.0.0:8080->80/tcp, :::8080->80/tcp       eaasi-front-end

Attached is the server log :slight_smile:
server.log (127.7 KB)

OK, the eaasi-nginx container seems to be restarted repeatedly. Because of that the communication on the backend fails too. What does its log contain?

$ sudo docker logs eaasi-nginx

This is the main error here:

/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/03/02 00:59:07 [emerg] 1#1: SSL_CTX_use_PrivateKey("/etc/nginx/certs/server.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/nginx/certs/server.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)

so I was able to update that issue by just making sure that I was using the right key and cert.
But I still get the same error as above

TASK [wait for eaas-server to start up] *****************************************************************************************************
fatal: [eaas-gateway]: FAILED! => changed=false 
  attempts: 1
  content: ''
  elapsed: 0
  msg: 'Status code was -1 and not [200]: Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>'
  redirected: false
  status: -1
  url: https://eaasi-prod.library.cmu.edu/emil/environment-repository/actions/prepare

However! The web interface actually loads. But I am unable to log in with the users and emails I had set up previously.
Is this because the deployment script did not fully update and move users over?
I do not see anything in
/eaasi/eaasi-ui/migrations/exports
as suggested in the update - there should be keycloak-users.csv

Hm, browsers appear to accept that certificate… but the installer doesn’t like it. Probably, our backend won’t accept it either.

Your cert seems to be incomplete:

$ openssl s_client -connect 'eaasi-prod.library.cmu.edu:443' -showcerts </dev/null | while openssl x509 -text; do :; done
depth=0 C = US, ST = Pennsylvania, L = Pittsburgh, O = Carnegie Mellon University, OU = Carnegie Mellon University, CN = eaasi-prod.library.cmu.edu
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = Pennsylvania, L = Pittsburgh, O = Carnegie Mellon University, OU = Carnegie Mellon University, CN = eaasi-prod.library.cmu.edu
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = US, ST = Pennsylvania, L = Pittsburgh, O = Carnegie Mellon University, OU = Carnegie Mellon University, CN = eaasi-prod.library.cmu.edu
verify return:1
DONE

You should try to follow these steps again to produce a full-chain cert: https://groups.google.com/g/eaasi-tech-talk/c/9vTtHP26A1s/m/cuLtkFcHDQAJ

Once you have built the full-chain cert, simply re-run the installer again.

Ah, of course! Thank you again.

Now the ansible script goes through without fail and I can get to the front end through the browser.
But there is no keycloak-users.csv and I cannot log in with the other usernames or passwords that we have on file.
I have this section uncommented in the yml file:

  # to update older installations,
  # uncomment the following lines...
  migrations:
  - name: "import-legacy-users"
    args:
     tenant_id: "eaasi"
     orgname: "EaaSI"

I don’t see anything in the tasks run that are related to users. Perhaps something was skipped in the ansible script? I don’t see anything related to users. I have attached that file.
eaasi-ansible-playbook-output.txt (100.3 KB)

In the server.log I see that six migrations were skipped but they don’t see relevant to users.

2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-image-index'
2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-emulator-index'
2022-03-04 15:27:28.767 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-image-archive-v1'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-local-emil-environments'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'import-legacy-emil-database-v1'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Registered migration 'create-absent-emil-environments'
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Found 6 available migration(s)
2022-03-04 15:27:28.768 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Running 6 configured migration(s)...
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-image-index'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-emulator-index'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-image-archive-v1'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-local-emil-environments'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'import-legacy-emil-database-v1'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [MIGRATIONS] Skipping executed migration 'create-absent-emil-environments'!
2022-03-04 15:27:28.769 |I| (EE-ManagedExecutorService-io-Thread-3) [SERVER-LIFECYCLE-HOOKS] Application started!
2022-03-04 15:27:31.489 |W| (default task-1) [Authentication] Authentication token is missing! Continue as anonymous user.
2022-03-04 15:27:31.493 |I| (default task-1) [EMIL] Preparing environment-repository...

OK, great! Now let’s try to find out why login fails.

That specific migration of legacy users is executed outside of our backend, so we need to check the following log instead:

$ sudo journalctl -u eaas | grep eaasi-web-api > eaasi-web-api.log

Please post the produced log file here.

ty
Most of it looks like this:

Mar 04 10:25:22 eaasi-prod docker-compose[2790775]: Stopping eaasi-web-api   ...
Mar 04 10:25:23 eaasi-prod docker-compose[2784958]: eaasi-web-api exited with code 143
Mar 04 10:25:26 eaasi-prod docker-compose[2790775]: Removing eaasi-web-api   ...
Mar 04 10:26:57 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Waiting for database at 'eaasi-database:5432'...
Mar 04 10:26:57 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Running database migrations...
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      |
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Sequelize CLI [Node: 16.13.1, CLI: 6.2.0, ORM: 6.7.0]
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      |
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Loaded configuration file "dist/data_access/config/config.js".
Mar 04 10:26:58 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Using environment "production".
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | No migrations were executed, database schema was already up to date.
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice New minor version of npm available! 8.1.2 -> 8.5.3
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.3>
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice Run `npm install -g npm@8.5.3` to update!
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | npm notice
Mar 04 10:26:59 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Executing 'node ./dist/app.js'...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Initializing EaaSI Web-API (production)...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Preparing database...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Starting HTTP server...
Mar 04 10:27:00 eaasi-prod docker-compose[2794082]: eaasi-web-api      | Server is listening on port 8081
Mar 06 01:42:33 eaasi-prod docker-compose[2794082]: eaasi-web-api      | (node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
Mar 06 01:42:33 eaasi-prod docker-compose[2794082]: eaasi-web-api      | (Use `node --trace-warnings ...` to show where the warning was created)

eaasi-web-api.log (19.9 KB)

Yes, the log does not contain any interesting information… But, let’s check the database content directly then.

Please post the output of the following commands:

$ sudo docker exec -it eaasi-database psql eaasi webapi -c '\dt'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE "SequelizeMeta"'

sure thing.

kiritharan@eaasi-prod:~$ sudo docker exec -it eaasi-database psql eaasi webapi -c '\dt'
                     List of relations
 Schema |               Name               | Type  | Owner  
--------+----------------------------------+-------+--------
 public | SequelizeMeta                    | table | webapi
 public | application_log                  | table | webapi
 public | bookmark                         | table | webapi
 public | eaasi_role                       | table | webapi
 public | eaasi_task                       | table | webapi
 public | emulation_project                | table | webapi
 public | emulation_project_resource       | table | webapi
 public | emulation_project_task_successor | table | webapi
 public | user_imported_content            | table | webapi
 public | user_imported_environment        | table | webapi
 public | user_imported_image              | table | webapi
 public | user_imported_software           | table | webapi
(12 rows)

kiritharan@eaasi-prod:~$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE "SequelizeMeta"'
                           name                            
-----------------------------------------------------------
 000-create-eaasiRole.js
 003-create-bookmark.js
 004-create-eaasiTask.js
 008-create-applicationLog.js
 20200618161213-create-emulation-project.js
 20200618161225-create-emulation-project-resource.js
 20200618161227-create-emulation-project-task-successor.js
 20211020193035-insert-default-eaasi-roles.js
 20211020201524-delete-legacy-tables.js
 20211020213045-delete-emulators.js
 20211020213740-delete-temp-environments.js
 20211020213910-delete-imported-software-resources.js
 20211021141205-update-eaasi-tasks.js
 20211022152408-rename-tables-for-imported-resources.js
 20211022160402-rename-columns-for-imported-resources.js
 20211022164522-rename-columns-for-bookmarks.js
 20211109170504-create-keycloak-users.js
(17 rows)

I have the impression, that your database (which the installer attempted to migrate) was empty and did not contain any users. Do you have a backup available?

Let’s also check the content of some tables to verify this:

$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_content'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_image'
$ sudo docker exec -it eaasi-database psql eaasi webapi -c 'TABLE user_imported_software'

Do not post the output, just check that those tables are not empty!

Each of those tables are empty and return no rows.
The database should not be empty as we did have an active user from last March… But I had not been involved very much with this project since 2020.

I looked at this other ticket I had made earlier and assumed we would have some content available:

But when I look within the users folder of /eaasi/image-archive/meta-data/user I don’t see anything…
Is everything we had in here lost now?
I have a backup of the directory structure from October of 2020, that may suffice? When I look at the user folder there I see there is an xml file within the user folder.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:emulationEnvironment xmlns:ns2="http://bwfla.bwl.de/common/datatypes">
    <ns2:id>1392752e-ef8d-4cd6-a4f8-5d31ddb2e25d</ns2:id>
    <ns2:timestamp>2019-10-16T18:20:57.280Z</ns2:timestamp>
    <ns2:description>
        <ns2:title>Mac OS 7.53 + CAD tools</ns2:title>
...

which seems to contain a project. I have an sql dump as well from the same date in 2020.
Would I put this in place of the current eaasi home and run update again? I have a feeling this data may be two versions behind.
thanks again for your help.

Actually, image-archive’s data is not related to the missing users in that specific case. Most of your images and metadata should have been moved to /eaasi/minio/data/image-archive during the upgrade.

But, in version 2020.03 all users were stored in a separate PostgreSQL database and managed by the EaaSI-UI directly. In version 2021.10 we have introduced a specialized OAuth service for managing users. Migrating users from the old legacy database to the new service is required to make previously created user accounts available after the upgrade.

You should try to restore the legacy EaaSI-UI’s database content from your backup and then re-run the installer again. It should be enough to restore a single directory located at /eaasi/portal for this.

Thanks,
@oooleg How can I restore the legacy Eaasi-UI database content? If the version is no longer using postgresql for users, where would I run this sql file before updating?