Unable to Run Mac OS 7 Emulation (or windows)

[kirichacha (jonathan kiritharan)]

Deployment: local
EaaSI Version: v2021.10
Browser: chromium and firefox for linux

Description:

We are working on a project where we must run some hypercard software from 1993. This would call for a Mac OS 7 base. I had synced with the Yale node and populated our resources tab.
Added the Apple Mac OS 7.01 - Mac IIci Base using BasiliskII
I hit Run in emulator and then get a

Error Message:

Starting environment session failed!

Trying to get the backend logs from the button gives a 500 page. I have included the server log
server.log (2.4 MB)

The main error seems to be something about sockets…

2022-04-13 02:20:26.957 |I| (default task-23) [BasiliskIIBean] (id:b2acba02-d92d-4f17-bba5-f3156b2d1780) Unmounting bindings...
2022-04-13 02:20:26.957 |I| (default task-23) [BasiliskIIBean] (id:b2acba02-d92d-4f17-bba5-f3156b2d1780) EmulatorBean destroyed.
2022-04-13 02:20:26.957 |I| (default task-23) [BasiliskIIBean] (id:b2acba02-d92d-4f17-bba5-f3156b2d1780) Running cleanup tasks...
2022-04-13 02:20:26.957 |I| (default task-23) [BasiliskIIBean] (id:b2acba02-d92d-4f17-bba5-f3156b2d1780) Working directory removed: /tmp-storage/bindings-8227539722520844785
2022-04-13 02:20:26.958 |I| (default task-23) [BasiliskIIBean] (id:b2acba02-d92d-4f17-bba5-f3156b2d1780) Working directory removed: /tmp/eaas-11638984468608914803
2022-04-13 02:20:27.084 |I| (default task-23) [PhaseInterceptorChain] Application {http://bwfla.bwl.de/api/emucomp}ComponentService#{http://bwfla.bwl.de/api/emucomp}initialize has thrown exception, unwinding now: de.bwl.bwfla.common.exceptions.BWFLAException: Constructing control sockets failed! Could not initialize class de.bwl.bwfla.emucomp.components.emulators.SocketAPI (BUILD: 0401EF6384, ID: B2ACBA02-D92D-4F17-BBA5-F3156B2D1780)
2022-04-13 02:20:27.089 |W| (default task-16) [EaasWS] Creating new session failed!
: de.bwl.bwfla.common.exceptions.BWFLAException: Constructing control sockets failed! Could not initialize class de.bwl.bwfla.emucomp.components.emulators.SocketAPI (BUILD: 0401EF6384, ID: B2ACBA02-D92D-4F17-BBA5-F3156B2D1780)

Just digging around to see what else could be at fault I see that the emulator being used is on a version from 2019 but that is the latest. Is that correct?

git+eaas-01032019 / docker://registry.gitlab.com/emulation-as-a-service/emulators/macemu-eaas-basilisk2 / latest

I also tried out a windows machine. That also failed to load. And I tried uploading an .iso but both of these efforts failed as well. These should be preserved in the server.log file.

I also see some things in the log file about other images that are not connected to anything. I think these can be ignored because of an issue with our database that was unresolved. We are not using these images but I would think that we should be able to add new images/emulation projects and can ignore these.

2022-04-13 00:27:23.172 |S| (default task-1) [EmilEnvironmentRepository] Software: 1e2616ba-448a-4796-892c-45ff716840cc not found. Probably software archive is not configured.
2022-04-13 00:27:23.187 |S| (default task-1) [EmilEnvironmentRepository] Software: 1a957f45-ed0f-4f4c-8447-1d8d38611e70 not found. Probably software archive is not configured.
2022-04-13 00:27:23.208 |S| (default task-1) [EmilEnvironmentRepository] Software: AdobePagemaker6-5-Plus_Mac not found. Probably software archive is not configured.
2022-04-13 00:27:23.228 |S| (default task-1) [EmilEnvironmentRepository] Software: software-e1ebe244-bd0d-11e9-864d-88b111586e02 not found. Probably software archive is not configured.
2022-04-13 00:27:23.257 |S| (default task-1) [EmilEnvironmentRepository] Software: Windows98SE not found. Probably software archive is not configured.

Urgency: The faculty member who asked us to put together this software for him requested it be done in early February and is not a little antsy as we had to rebuild and upgrade eaasi and work through some other issues. So this is a bit more urgent than the last ticket: Duplicate DB name Updating to 2021.10 from 2020.03 - #30 by oooleg

[oooleg (Oleg Stobbe)]

The main issue appears to be, that you are running out of storage space:

2022-04-12 18:49:37.062 |S| (default task-62) [BasiliskIIBean] (id:0583c417-f1f7-43e3-adbc-0d3288edf594) Constructing control sockets failed!: java.lang.UnsatisfiedLinkError: Failed to create temporary file for /com/sun/jna/linux-x86-64/libjnidispatch.so library: No space left on device
...
2022-04-12 18:49:44.985 |I| (default task-55) [eaas/error-report] Subprocess 5398 stderr:
cp: error writing './config/admin-ui.json': No space left on device
cp: error writing './config/eaas-config.d/02-clustermgr.yaml': No space left on device
cp: error writing './config/eaas-config.d/01-main.yaml': No space left on device

Please try to free or add enough storage space for EaaSI to work properly.

[kirichacha (jonathan kiritharan)]

thank you @oooleg

I have over 300GB free on /. How much should be available to run an emulation?

I cleared unused images (sudo docker image prune -a)
I removed everything within the docker tmpfs (sudo rm /var/lib/docker/volumes/eaas_tmpfs_vol/_data/*)

still getting

2022-04-19 16:28:40.881 |I| (default task-1) [PhaseInterceptorChain] Application {http://bwfla.bwl.de/api/eaas}EaasWSService#{http://bwfla.bwl.de/api/eaas}createSessionWithOptions has thrown exception, unwinding now: de.bwl.bwfla.common.exceptions.BWFLAException: Constructing control sockets failed! Failed to create temporary file for /com/sun/jna/linux-x86-64/libjnidispatch.so library: No space left on device (BUILD: 0401EF6384, ID: 14242A33-A7FC-4719-989A-BD8565B356C0)
2022-04-19 16:28:40.886 |S| (default task-3) [eaas/components] Components create machine failed: de.bwl.bwfla.common.exceptions.BWFLAException: Constructing control sockets failed! Failed to create temporary file for /com/sun/jna/linux-x86-64/libjnidispatch.so library: No space left on device (BUILD: 0401EF6384, ID: 14242A33-A7FC-4719-989A-BD8565B356C0)

here is my df

kiritharan@eaasi-prod:/eaas-home$ df -h
Filesystem                        Size  Used Avail Use% Mounted on
udev                              2.9G     0  2.9G   0% /dev
tmpfs                             592M  2.1M  590M   1% /run
/dev/mapper/eaasi--prod--vg-root  434G   98G  315G  24% /
tmpfs                             2.9G     0  2.9G   0% /dev/shm
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                             2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/sda1                         472M  306M  142M  69% /boot
/dev/loop3                         62M   62M     0 100% /snap/core20/1376
/dev/loop4                         44M   44M     0 100% /snap/snapd/15177
/dev/loop5                         68M   68M     0 100% /snap/lxd/22526
/dev/loop2                         68M   68M     0 100% /snap/lxd/22753
/dev/loop0                         62M   62M     0 100% /snap/core20/1405
/dev/loop1                         45M   45M     0 100% /snap/snapd/15314
tmpfs                             592M     0  592M   0% /run/user/1002
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/6d9e02ec2de3a730d61045284ee8e0cf54c5c25ce6e46cdad2da74d713069591/merged
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/5cc4cefab03ad832caa784db0f7fd88d6123fbe12520b0d1ab4d39616419bd6f/merged
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/196917056df6ea89f41023113f5a5123d3bff68c056529c6d19ceea0fe298ae2/merged
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/a980ae1d870466b1efb82e13dab702e271ac6254c5f8e03d9fc642234fd16dde/merged
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/2f0c6a1d55225a160d9b3f1607650b781fb0ec19f49a833a4becb2ce3ae1c7a7/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/053fa9a754e184ec675bfc1be182781521bab7427740ecbcbb370a3906fd7fa2/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/8f02579d7f98f85ebe291868cb74326478263935c1e1570f8aebd80b2959b1ba/mounts/shm
shm                                64M   16K   64M   1% /var/lib/docker/containers/de5b2ecf441c15183d8c4531d8ece3621cc6414e172b74bc9ed4d0bb209ea0da/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/292041ee8c2fdd45234c98232bcd12d7d0c55a7cc7aa03b4f70ae0dfb010cdbf/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/5e4fccf759508d5f65c14ffcda364a4ab2fdb17ccf59d4ace11f2dc8f8e3237e/mounts/shm
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/6c6e7e9e3005302d10f21bf3d7ffea2519e93c95bdc491cdf35402fb0c6b8865/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/d35a1e8e3d61047cb308dcf5c9f460e98ebc37c72fbda0fd5a789993078c7e72/mounts/shm
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/8d7984672c4879a61181933d1479bef2137208658eda34b762174a569c1975eb/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/92f32508802dcf9533af2e1665e1d344dd985c8bf8e9b014f096180eae3d0d86/mounts/shm
tmpfs                             2.9G  2.9G     0 100% /var/lib/docker/volumes/eaas_tmpfs_vol/_data
overlay                           434G   98G  315G  24% /var/lib/docker/overlay2/52367721490fd4e241932dd9d3c90060c9d9d99af687c386e22147ad58d3d83d/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/3f7206649877807b0a2ec0f66545ac225d1a3b4cffc722e8fec943e2491b4d59/mounts/shm

I went and increased the tmpfs at /var/lib/docker/volumes/eaas_tmpfs_vol/_data
by updating /etc/fstab and remounting ( sudo mount -o remount,size=6G /var/lib/docker/volumes/eaas_tmpfs_vol/_data)
and it read as 6gb on another df - h but there was still an error when trying to run the emulator.

So, I then tried to run the update.sh script with the increased tmp size.
The script failed:

TASK [wait for eaas-server to start up] *******************************************************************************************************************************************************************
FAILED - RETRYING: wait for eaas-server to start up (360 retries left).
fatal: [eaas-gateway]: FAILED! => changed=false 
  access_control_allow_headers: content-type, authorization
  access_control_allow_methods: POST, GET, DELETE, OPTIONS
  access_control_allow_origin: '*'
  attempts: 2
  connection: close
  content: '{"status":"2","message":"Internal error: de.bwl.bwfla.common.exceptions.BWFLAException: Importing legacy images failed! (BUILD: 0401EF6384)"}'
  content_length: '141'
  content_type: application/json
  date: Tue, 19 Apr 2022 19:04:36 GMT
  elapsed: 11
  json:
    message: 'Internal error: de.bwl.bwfla.common.exceptions.BWFLAException: Importing legacy images failed! (BUILD: 0401EF6384)'
    status: '2'
  msg: 'Status code was 500 and not [200]: HTTP Error 500: Internal Server Error'
  redirected: false
  server: nginx/1.20.2
  status: 500
  url: https://eaasi-prod.library.cmu.edu/emil/environment-repository/actions/prepare

that was accompanied by

2022-04-19 19:04:36.674 |W| (EE-ManagedExecutorService-io-Thread-1) [EMIL] Importing image '791ed319-aa98-4f5c-a8db-6060a1476596' failed!: javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.lang.RuntimeException: java.io.IOException: No space left on device

And this made the tmpfs mount go back to 2.9G. Should I be updating this in the eaasi installer?
How can I increase the size of this ? Or where am I running out of space?

Here is the server.log again
serverapril19.log (7.3 MB)

[ethan.gates]

possibly a tangent but just to check @kirichacha - when you tried to run an environment with the expanded 6GB tmpfs volume and still got an error (before running the update.sh script) which environment were you trying to run? the Apple Mac OS 7.01 - Mac IIci one that you started this thread with should only have a ~60 MB image so should not be an issue, but some of the Windows environments are larger, up to nearly 20GB on the Windows 98 (SE) - Base V2 environment, for instance. I’m not positive how/if image data may be getting passed between these Docker volumes when an environment gets run, but wondering if we can take that off the table for troubleshooting

(I’ll start a thread elsewhere in the Forum with published base image sizes for people’s reference in general, I can quickly share that info)

[kirichacha (jonathan kiritharan)]

Ah, yes it may be two different issues. However, I get an error for both the windows and the Mac instance with the expanded tmpfs. Sorry, I will focus on just the Mac instance for this thread.

Here’s what I just did and have a log from the start of my session included at the bottom:

• I expanded the tmpfs to 6G
• searched the list of emulators and found the Mac OS 7
• clicked that and hit run
• it thought for a while and then i saw this:
  
  

This is the status of the tmpfs
tmpfs 6.0G 2.9G 3.2G 49% /var/lib/docker/volumes/eaas_tmpfs_vol/_data
And here is the log
420bro.log (151 KB)

This is an error i’m seeing for the qemu…
_qemu-img: /tmp-storage/bindings-7176582052353885508/main_hdd.cow: CURL: Error opening file: The requested URL returned error: 502 Bad Gateway Could not open backing image.
a few missing environments… I’m not sure which thing to investigate more.

I don’t see anything of interest in the front end error logs.

[oooleg (Oleg Stobbe)]

Both issues are probably related to the tmpfs-mount running out of space. How does the container’s internal view look like:

$ sudo docker exec -it eaas df -h
$ sudo docker exec -it eaas apt install tree
$ sudo docker exec -it eaas tree /tmp

Failures to start sessions because of missing images might be related to incomplete import of images (and their layers).

[kirichacha (jonathan kiritharan)]

looks like there’s 3G available.

$ sudo docker exec -it eaas df -h
Filesystem                        Size  Used Avail Use% Mounted on
overlay                           434G   99G  314G  24% /
tmpfs                              64M     0   64M   0% /dev
tmpfs                             2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/mapper/eaasi--prod--vg-root  434G   99G  314G  24% /tmp-storage
tmpfs                             6.0G  2.9G  3.2G  49% /tmp
shm                                64M     0   64M   0% /dev/shm
/eaas/config                      434G   99G  314G  24% /home/bwfla/.bwFLA
/eaas/server-data                 434G   99G  314G  24% /home/bwfla/server-data
/eaas/image-archive               434G   99G  314G  24% /home/bwfla/image-archive
/eaas/objects                     434G   99G  314G  24% /home/bwfla/objects
/eaas/export                      434G   99G  314G  24% /home/bwfla/export
/eaas/import                      434G   99G  314G  24% /home/bwfla/import

...installed tree...

$ sudo docker exec -it eaas tree /tmp
/tmp
|-- 34569a771e733b310147422282541732494.tmp
|-- 34569a771e733b311235146236809993556.tmp
|-- 34569a771e733b312860505172816172675.tmp
|-- 34569a771e733b313096761740052709856.tmp
|-- 34569a771e733b313244185768508822517.tmp
|-- 34569a771e733b313547290337120023840.tmp
|-- 34569a771e733b314723431513559102409.tmp
|-- 34569a771e733b316221163763969129868.tmp
|-- 34569a771e733b317050549831433318510.tmp
|-- 34569a771e733b317063837752747817722.tmp
|-- 34569a771e733b318363536957440139315.tmp
|-- 34569a771e733b31907850958359963063.tmp
|-- 34569a771e733b32242929267483385884.tmp
|-- 34569a771e733b32376734461671087503.tmp
|-- 34569a771e733b3336629223712322440.tmp
|-- 34569a771e733b33758489223462817378.tmp
|-- 34569a771e733b35069089284167437329.tmp
|-- 34569a771e733b35405394577784059275.tmp
|-- 34569a771e733b35992988322322846295.tmp
|-- 34569a771e733b37232732821274278955.tmp
|-- 34569a771e733b37539781188026349330.tmp
|-- 34569a771e733b37927160252534713555.tmp
|-- 34569a771e733b38217759053633275244.tmp
|-- 34569a771e733b38254332684186970660.tmp
|-- 34569a771e733b38515790847485522202.tmp
|-- 34569a771e733b39985902330188882665.tmp
|-- hsperfdata_bwfla
|   `-- 345
|-- hsperfdata_root
|   `-- 62112
|-- jna-94151654
|-- mongodb-27017.sock
`-- process-runners

4 directories, 29 files

Do you have any other suggestions for troubleshooting this?
Is there a way to spin up the eaasi service and have a different default tmp storage directory size?
We would like to get this working as soon as we can and have a faculty member interested in using some software that he gave us back in January.

[oooleg (Oleg Stobbe)]

That’s strange, the mounts look right to me. For operations with big temporary files, we have added an explicit disk-based /tmp-storage mount, to avoid space problems with tmpfs in newer Linux sysytems.

How big are those *.tmp files in your case?

$ sudo docker exec -it eaas tree -h /tmp

Usually, the data in /tmp mount should consume only a few megabytes or less, e.g.:

ubuntu@eaasi-dev:~$ sudo docker exec -it eaas df -h
Filesystem           Size  Used Avail Use% Mounted on
overlay              194G  128G   67G  66% /
tmpfs                 64M     0   64M   0% /dev
tmpfs                7.4G     0  7.4G   0% /sys/fs/cgroup
/dev/root            194G  128G   67G  66% /tmp-storage
tmpfs                7.4G  264K  7.4G   1% /tmp
shm                   64M     0   64M   0% /dev/shm
/eaas/config         194G  128G   67G  66% /home/bwfla/.bwFLA
/eaas/server-data    194G  128G   67G  66% /home/bwfla/server-data
/eaas/image-archive  194G  128G   67G  66% /home/bwfla/image-archive
/eaas/objects        194G  128G   67G  66% /home/bwfla/objects
/eaas/export         194G  128G   67G  66% /home/bwfla/export
/eaas/import         194G  128G   67G  66% /home/bwfla/import

[oooleg (Oleg Stobbe)]

@kirichacha: as a workaround (until we find out what exactly is happening), you could try to explicitly disable tmpfs-based /tmp mount in your eaasi.yaml:

docker:
  disable_tmp_volume: true

[kirichacha (jonathan kiritharan)]

thank you

/tmp
|-- [   0]  34569a771e733b310147422282541732494.tmp
|-- [   0]  34569a771e733b311235146236809993556.tmp
|-- [   0]  34569a771e733b312860505172816172675.tmp
|-- [   0]  34569a771e733b313096761740052709856.tmp
|-- [   0]  34569a771e733b313244185768508822517.tmp
|-- [   0]  34569a771e733b313547290337120023840.tmp
|-- [   0]  34569a771e733b314723431513559102409.tmp
|-- [2.9G]  34569a771e733b316221163763969129868.tmp
|-- [   0]  34569a771e733b317050549831433318510.tmp
|-- [   0]  34569a771e733b317063837752747817722.tmp
|-- [   0]  34569a771e733b318363536957440139315.tmp
|-- [   0]  34569a771e733b31907850958359963063.tmp
|-- [   0]  34569a771e733b32242929267483385884.tmp
|-- [   0]  34569a771e733b32376734461671087503.tmp
|-- [   0]  34569a771e733b3336629223712322440.tmp
|-- [   0]  34569a771e733b33758489223462817378.tmp
|-- [   0]  34569a771e733b35069089284167437329.tmp
|-- [   0]  34569a771e733b35405394577784059275.tmp
|-- [   0]  34569a771e733b35992988322322846295.tmp
|-- [   0]  34569a771e733b37232732821274278955.tmp
|-- [   0]  34569a771e733b37539781188026349330.tmp
|-- [   0]  34569a771e733b37927160252534713555.tmp
|-- [   0]  34569a771e733b38217759053633275244.tmp
|-- [   0]  34569a771e733b38254332684186970660.tmp
|-- [   0]  34569a771e733b38515790847485522202.tmp
|-- [   0]  34569a771e733b39985902330188882665.tmp
|-- [  60]  eaas-er-vY7fo
|   `-- [  60]  report
|       `-- [ 200]  config
|           |-- [1.2K]  admin-ui.json
|           |-- [  80]  eaas-config.d
|           |   |-- [5.0K]  01-main.yaml
|           |   `-- [ 437]  02-clustermgr.yaml
|           |-- [  80]  keycloak
|           |   |-- [ 64K]  defaults.json
|           |   `-- [  40]  secrets
|           |-- [  80]  minio
|           |   |-- [  60]  policies
|           |   |   `-- [ 446]  image-archive.json
|           |   `-- [  40]  secrets
|           |-- [ 100]  nameindices
|           |   |-- [  40]  default
|           |   |-- [  40]  public
|           |   `-- [  40]  remote
|           |-- [  60]  nghttpx
|           |   `-- [ 873]  nghttpx.conf
|           |-- [  80]  nginx
|           |   |-- [2.7K]  nginx.conf
|           |   `-- [ 805]  proxy.js
|           `-- [  40]  secrets
|-- [  60]  hsperfdata_bwfla
|   `-- [ 32K]  345
|-- [  60]  hsperfdata_root
|   `-- [ 32K]  145692
|-- [  40]  jna-94151654
|-- [   0]  mongodb-27017.sock
`-- [  40]  process-runners

20 directories, 37 files

i tried deleting that 2.9GB file and running again before and after a restart and still no luck.

added the flag to the eaasi.yml file and ran the update script.
i tried to run the emulation again and it does not give an error but it also doesn’t start up.

here is the log starting from today.
newlog.log (1.4 MB)

seeing a lot of this
[Authentication] Validation of authentication token failed!: com.auth0.jwt.exceptions.TokenExpiredException: The Token has expired on Tue Apr 26 00:22:05 GMT 2022.

at this point, how feasible is it to become a hosted node? or has that ship sailed as we were not part of the program at the beginning of the year?

[oooleg (Oleg Stobbe)]

That 2.9GB file looks suspicious, we have to investigate the code to find out if and where these temporary files might be created. But, with disabled tmpfs mount it should not matter.

According to the log, some of your images are about 20GB in size and will take some time to migrate. But, it seems, that you have started your sessions before this process has finished. Hence emulation sessions failed because of missing images:

2022-04-28 21:04:38.682 |I| (default task-10) [BasiliskIIBean] (id:672b9d18-1908-4e5a-9542-a6e4e3350320) Subprocess 1988 started:  'qemu-img' 'info' '--output' 'json' 'http://nginx:82/672b9d18-1908-4e5a-9542-a6e4e3350320/Windows_98_SE-Base_V2.img'
2022-04-28 21:04:38.682 |I| (default task-10) [BasiliskIIBean] (id:672b9d18-1908-4e5a-9542-a6e4e3350320) Waiting for subprocess 1988 to finish...
2022-04-28 21:04:38.837 |I| (default task-10) [BasiliskIIBean] (id:672b9d18-1908-4e5a-9542-a6e4e3350320) Subprocess 1988 terminated with code 1
2022-04-28 21:04:38.838 |I| (default task-10) [BasiliskIIBean] (id:672b9d18-1908-4e5a-9542-a6e4e3350320) Subprocess 1988 stderr:
_qemu-img: Could not open 'http://nginx:82/672b9d18-1908-4e5a-9542-a6e4e3350320/Windows_98_SE-Base_V2.img': CURL: Error opening file: The requested URL returned error: 502 Bad Gateway

The installer checks and waits until all migrations complete, you should generally not attempt to use the server during that period. To verify manually, that your missing image has been successfully imported, you should find a similar line in your latest log (it is missing in provided log though):

YYYY-MM-DD HH:MM:SS |I| (EE-ManagedExecutorService-io-Thread-3) [EMIL] Imported image 'Windows_98_SE-Base_V2.img'

You should also check, that all of your images have been successfully imported as part of import-legacy-image-archive-v1 migration. The following directories should be empty:

• /eaasi/image-archive/images
• /eaasi/image-archive/public/images

[kirichacha (jonathan kiritharan)]

That 2.9GB file looks suspicious, we have to investigate the code to find out if and where these temporary files might be created. But, with disabled tmpfs mount it should not matter.

sure… after adding the flag you suggested in to the eaasi.yml file this is my df -h for the server

Filesystem                        Size  Used Avail Use% Mounted on
udev                              2.9G     0  2.9G   0% /dev
tmpfs                             592M  2.1M  590M   1% /run
/dev/mapper/eaasi--prod--vg-root  434G   80G  332G  20% /
tmpfs                             2.9G     0  2.9G   0% /dev/shm
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                             2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/loop0                         62M   62M     0 100% /snap/core20/1405
/dev/sda1                         472M  207M  241M  47% /boot
/dev/loop3                         68M   68M     0 100% /snap/lxd/22753
/dev/loop2                         68M   68M     0 100% /snap/lxd/22526
/dev/loop5                         44M   44M     0 100% /snap/snapd/15177
/dev/loop6                         45M   45M     0 100% /snap/snapd/15534
/dev/loop4                         62M   62M     0 100% /snap/core20/1434
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/dc19d6e6567f81a9e8a0303d3a4769025546b7cb745c3dba188515f33c4b5419/merged
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/79c28f7b8b543d781a74d59dec88e04210deed3ec9ceafee73cff373beae0805/merged
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/64627be2e06773bb3e2b1e6e4db6b96efbe7b63bc7ed0116e0d7898822d0c318/merged
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/dfb37d23d5e8f7f84df5eaebdc9cea0e6cef7e8716c5c650054cbb03fddb0974/merged
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/136da19c2575502ed80186f30f5efa4a5ed2055ef4d6ed3846a9517794f66bb8/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/cde1b84e23832bd8fa92749653c279522530ca51a339c86d1ac79c05b776722a/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/ecc91a8a535939853ae4a7223f6daa70c52167edd715ada6f04f280f9051b94f/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/b4aa2ee5fd02f6f499ff17346736e711e29acc587d76417ffcecc013ce6a892f/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/aab16f34088291245818e1dc60a98e66cada93c36b22da94d37d1659f13a6a52/mounts/shm
shm                                64M   16K   64M   1% /var/lib/docker/containers/f78bf44647ee7bb105063f8196bbb7fb49cc3ba58c86b626e406deaef4cd3bd7/mounts/shm
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/65790fc9b06ccd97eaeadf429abc84bf597a130a861c70f5bf899a4e9cbe304f/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/1d1f087e428b2fd0f2008bc7281834ae9f0441f3936056ab3a3d66ad7f7e669f/mounts/shm
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/2e9fce1fb6cd434b2d43131c3acee4c0fa4b5192d3a81de52f345641df207fdf/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/63e9c5d72a6b79783e187cc705038cd1a073c00a9266c21023c9c33084a17213/mounts/shm
overlay                           434G   80G  332G  20% /var/lib/docker/overlay2/5d9fb5ab252499ced92e4ce3315a51e0e23404707710fb47eb1b38749cad5fad/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/06f4f6dcd8f087660e91e3e6fbcea4189dbc1aea5c89fb3a7693055dfc048b14/mounts/shm
tmpfs                             592M     0  592M   0% /run/user/1002

and this is the result of the df command from docker

$ sudo docker exec -it eaas df -h
Filesystem                        Size  Used Avail Use% Mounted on
overlay                           434G   80G  332G  20% /
tmpfs                              64M     0   64M   0% /dev
tmpfs                             2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/mapper/eaasi--prod--vg-root  434G   80G  332G  20% /tmp-storage
shm                                64M     0   64M   0% /dev/shm
/eaas/config                      434G   80G  332G  20% /home/bwfla/.bwFLA
/eaas/server-data                 434G   80G  332G  20% /home/bwfla/server-data
/eaas/image-archive               434G   80G  332G  20% /home/bwfla/image-archive
/eaas/objects                     434G   80G  332G  20% /home/bwfla/objects
/eaas/export                      434G   80G  332G  20% /home/bwfla/export
/eaas/import                      434G   80G  332G  20% /home/bwfla/import

According to the log, some of your images are about 20GB in size and will take some time to migrate. But, it seems, that you have started your sessions before this process has finished. Hence emulation sessions failed because of missing images

I don’t understand this. Do you mean that I started my session to open the emulation too soon? Or that I even tried to use the server after performing the update too soon?

I ran the update script and then checked back in 5 hours
I keep checking the server.log but it still seems to be at this portion:

2022-05-02 14:40:54.849 |W| (default task-2) [Authentication] Authentication token is missing! Continue as anonymous user.
2022-05-02 14:40:54.855 |I| (default task-2) [EMIL] Preparing environment-repository...
2022-05-02 19:27:30.350 |I| (default task-1) [EMIL] Listing all software-packages...

^ the last line is when i looked again

I would like to point out that the update.sh script does not fully complete.

TASK [wait for eaas-server to start up] ****************************************
FAILED - RETRYING: wait for eaas-server to start up (360 retries left).
fatal: [eaas-gateway]: FAILED! => changed=false 
  access_control_allow_headers: content-type, authorization
  access_control_allow_methods: POST, GET, DELETE, OPTIONS
  access_control_allow_origin: '*'
  attempts: 2
  connection: close
  content: '{"status":"2","message":"Internal error: java.lang.RuntimeException: org.jboss.weld.contexts.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.RequestScoped"}'
  content_length: '207'
  content_type: application/json
  date: Mon, 02 May 2022 14:40:54 GMT
  elapsed: 0
  json:
    message: 'Internal error: java.lang.RuntimeException: org.jboss.weld.contexts.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.RequestScoped'
    status: '2'
  msg: 'Status code was 500 and not [200]: HTTP Error 500: Internal Server Error'
  redirected: false
  server: nginx/1.20.2
  status: 500
  url: https://eaasi-prod.library.cmu.edu/emil/environment-repository/actions/prepare

I went to the server and tried to run an emulation and kept getting errors again for multiple emulations.
Am I doing something wrong there?
i find an environment saved locally here
/resources/explore
select as so:

click “Run in Emulator”. wait about 10 seconds and then get the error

should i be waiting longer to click “run in emulator”?

You should also check, that all of your images have been successfully imported as part of import-legacy-image-archive-v1 migration. The following directories should be empty:

Here are those directories:

$ tree /eaas-home/image-archive/images/
/eaas-home/image-archive/images/
├── base
├── checkpoints
├── containers
├── derivate
├── object
├── patches
├── roms
├── runtime
├── sessions
├── system
├── template
├── tmp
└── user

13 directories, 0 files

no files in /images

$ tree /eaas-home/image-archive/public/images/
/eaas-home/image-archive/public/images/
├── base
├── checkpoints
├── containers
├── derivate
├── object
├── patches
├── roms
├── runtime
├── sessions
├── system
├── template
├── tmp
└── user

13 directories, 0 files

no files in /public/images

but directories within both. Is that alright?
I think we know that the images had not been a full success in the transfer from the last issue I had raised:

Duplicate DB name Updating to 2021.10 from 2020.03

Deployment: a local node EaaSI Version: 2020.03 to 2021.10 Browser: chromium on linux Description: When trying to run the update following the documentation here https://eaasi.gitlab.io/eaasi_user_handbook/overview/install/setup.html#migrating-from-2020-03-to-2021-10 and here https://eaasi.gitlab.io/eaasi_user_handbook/overview/install/setup.html#updating-eaasi I kept running in to an issue where it was saying that I had uncommitted changes within a submodule. I should not have change…

The last post I had put in there was saying we can just move on rather than trying to drag those other images back up. You responded giving some direction to update some records directly in the sql database so that the IDs can be associated (I did not do this because I talked to our group and we decided to just move on without our old images). Is this what is causing issues? I now see some of those images in the saved locally list after waiting longer. This one for example:

But I was originally trying to run the Mac7 emulation which was an entirely new setup.

Thank you again for your help with this. I would love to know what I’m doing wrong.

As we continue this thread I feel like it gets further in the weeds. I also wrote in my last post if we could just get on a hosted node because it’s been 3 months of trying to get this to work and would like to know if that’s possible because at this point it seems like an easier solution. Either that or would you or anyone on the eaasi team be able to have a zoom call for like 45 minutes to maybe just talk through this? That would probably be a bit more efficient than checking in twice a week to run a single command and write a response. I want to be conscious of your bandwidth.

Thanks

[ethan.gates]

@kirichacha I have created a node for CMU in the hosted service and will follow up with you one-on-one with access details - that makes sense to keep your users moving while you all continue to troubleshoot

@oooleg does a Zoom call where the two of you can screen share sound like a good option at this point to you? I’d be happy to facilitate the scheduling and get you all set up together

[oooleg (Oleg Stobbe)]

Yes, we can schedule a call and try to debug the issues live.