That 2.9GB file looks suspicious, we have to investigate the code to find out if and where these temporary files might be created. But, with disabled
tmpfsmount it should not matter.
sure… after adding the flag you suggested in to the eaasi.yml file this is my df -h for the server
Filesystem Size Used Avail Use% Mounted on
udev 2.9G 0 2.9G 0% /dev
tmpfs 592M 2.1M 590M 1% /run
/dev/mapper/eaasi--prod--vg-root 434G 80G 332G 20% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
/dev/loop0 62M 62M 0 100% /snap/core20/1405
/dev/sda1 472M 207M 241M 47% /boot
/dev/loop3 68M 68M 0 100% /snap/lxd/22753
/dev/loop2 68M 68M 0 100% /snap/lxd/22526
/dev/loop5 44M 44M 0 100% /snap/snapd/15177
/dev/loop6 45M 45M 0 100% /snap/snapd/15534
/dev/loop4 62M 62M 0 100% /snap/core20/1434
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/dc19d6e6567f81a9e8a0303d3a4769025546b7cb745c3dba188515f33c4b5419/merged
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/79c28f7b8b543d781a74d59dec88e04210deed3ec9ceafee73cff373beae0805/merged
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/64627be2e06773bb3e2b1e6e4db6b96efbe7b63bc7ed0116e0d7898822d0c318/merged
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/dfb37d23d5e8f7f84df5eaebdc9cea0e6cef7e8716c5c650054cbb03fddb0974/merged
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/136da19c2575502ed80186f30f5efa4a5ed2055ef4d6ed3846a9517794f66bb8/merged
shm 64M 0 64M 0% /var/lib/docker/containers/cde1b84e23832bd8fa92749653c279522530ca51a339c86d1ac79c05b776722a/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ecc91a8a535939853ae4a7223f6daa70c52167edd715ada6f04f280f9051b94f/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b4aa2ee5fd02f6f499ff17346736e711e29acc587d76417ffcecc013ce6a892f/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/aab16f34088291245818e1dc60a98e66cada93c36b22da94d37d1659f13a6a52/mounts/shm
shm 64M 16K 64M 1% /var/lib/docker/containers/f78bf44647ee7bb105063f8196bbb7fb49cc3ba58c86b626e406deaef4cd3bd7/mounts/shm
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/65790fc9b06ccd97eaeadf429abc84bf597a130a861c70f5bf899a4e9cbe304f/merged
shm 64M 0 64M 0% /var/lib/docker/containers/1d1f087e428b2fd0f2008bc7281834ae9f0441f3936056ab3a3d66ad7f7e669f/mounts/shm
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/2e9fce1fb6cd434b2d43131c3acee4c0fa4b5192d3a81de52f345641df207fdf/merged
shm 64M 0 64M 0% /var/lib/docker/containers/63e9c5d72a6b79783e187cc705038cd1a073c00a9266c21023c9c33084a17213/mounts/shm
overlay 434G 80G 332G 20% /var/lib/docker/overlay2/5d9fb5ab252499ced92e4ce3315a51e0e23404707710fb47eb1b38749cad5fad/merged
shm 64M 0 64M 0% /var/lib/docker/containers/06f4f6dcd8f087660e91e3e6fbcea4189dbc1aea5c89fb3a7693055dfc048b14/mounts/shm
tmpfs 592M 0 592M 0% /run/user/1002
and this is the result of the df command from docker
$ sudo docker exec -it eaas df -h
Filesystem Size Used Avail Use% Mounted on
overlay 434G 80G 332G 20% /
tmpfs 64M 0 64M 0% /dev
tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
/dev/mapper/eaasi--prod--vg-root 434G 80G 332G 20% /tmp-storage
shm 64M 0 64M 0% /dev/shm
/eaas/config 434G 80G 332G 20% /home/bwfla/.bwFLA
/eaas/server-data 434G 80G 332G 20% /home/bwfla/server-data
/eaas/image-archive 434G 80G 332G 20% /home/bwfla/image-archive
/eaas/objects 434G 80G 332G 20% /home/bwfla/objects
/eaas/export 434G 80G 332G 20% /home/bwfla/export
/eaas/import 434G 80G 332G 20% /home/bwfla/import
According to the log, some of your images are about 20GB in size and will take some time to migrate. But, it seems, that you have started your sessions before this process has finished. Hence emulation sessions failed because of missing images
I don’t understand this. Do you mean that I started my session to open the emulation too soon? Or that I even tried to use the server after performing the update too soon?
I ran the update script and then checked back in 5 hours
I keep checking the server.log but it still seems to be at this portion:
2022-05-02 14:40:54.849 |W| (default task-2) [Authentication] Authentication token is missing! Continue as anonymous user.
2022-05-02 14:40:54.855 |I| (default task-2) [EMIL] Preparing environment-repository...
2022-05-02 19:27:30.350 |I| (default task-1) [EMIL] Listing all software-packages...
^ the last line is when i looked again
I would like to point out that the update.sh script does not fully complete.
TASK [wait for eaas-server to start up] ****************************************
FAILED - RETRYING: wait for eaas-server to start up (360 retries left).
fatal: [eaas-gateway]: FAILED! => changed=false
access_control_allow_headers: content-type, authorization
access_control_allow_methods: POST, GET, DELETE, OPTIONS
access_control_allow_origin: '*'
attempts: 2
connection: close
content: '{"status":"2","message":"Internal error: java.lang.RuntimeException: org.jboss.weld.contexts.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.RequestScoped"}'
content_length: '207'
content_type: application/json
date: Mon, 02 May 2022 14:40:54 GMT
elapsed: 0
json:
message: 'Internal error: java.lang.RuntimeException: org.jboss.weld.contexts.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.RequestScoped'
status: '2'
msg: 'Status code was 500 and not [200]: HTTP Error 500: Internal Server Error'
redirected: false
server: nginx/1.20.2
status: 500
url: https://eaasi-prod.library.cmu.edu/emil/environment-repository/actions/prepare
I went to the server and tried to run an emulation and kept getting errors again for multiple emulations.
Am I doing something wrong there?
i find an environment saved locally here
/resources/explore
select as so:
click “Run in Emulator”. wait about 10 seconds and then get the error
should i be waiting longer to click “run in emulator”?
You should also check, that all of your images have been successfully imported as part of
import-legacy-image-archive-v1migration. The following directories should be empty:
Here are those directories:
$ tree /eaas-home/image-archive/images/
/eaas-home/image-archive/images/
├── base
├── checkpoints
├── containers
├── derivate
├── object
├── patches
├── roms
├── runtime
├── sessions
├── system
├── template
├── tmp
└── user
13 directories, 0 files
no files in /images
$ tree /eaas-home/image-archive/public/images/
/eaas-home/image-archive/public/images/
├── base
├── checkpoints
├── containers
├── derivate
├── object
├── patches
├── roms
├── runtime
├── sessions
├── system
├── template
├── tmp
└── user
13 directories, 0 files
no files in /public/images
but directories within both. Is that alright?
I think we know that the images had not been a full success in the transfer from the last issue I had raised:
The last post I had put in there was saying we can just move on rather than trying to drag those other images back up. You responded giving some direction to update some records directly in the sql database so that the IDs can be associated (I did not do this because I talked to our group and we decided to just move on without our old images). Is this what is causing issues? I now see some of those images in the saved locally list after waiting longer. This one for example:
But I was originally trying to run the Mac7 emulation which was an entirely new setup.
Thank you again for your help with this. I would love to know what I’m doing wrong.
As we continue this thread I feel like it gets further in the weeds. I also wrote in my last post if we could just get on a hosted node because it’s been 3 months of trying to get this to work and would like to know if that’s possible because at this point it seems like an easier solution. Either that or would you or anyone on the eaasi team be able to have a zoom call for like 45 minutes to maybe just talk through this? That would probably be a bit more efficient than checking in twice a week to run a single command and write a response. I want to be conscious of your bandwidth.
Thanks