Web UI login goes to blank screen; Internal error: de.bwl.bwfla.common.exceptions.BWFLAException: Connecting to legacy image-archive failed!

[mwhitmor (Michael_Whitmore)]

Deployment: Self hosted; VMSphere hypervisor; Ubuntu 20.04 virtual machines

EaaSI Version: 2021.10

Browser: Chromium v105.0.5160.0

Description: In trying to deploy EaaSI in a self-hosted environment using virtual machines, we have hit what appears to be a blocking issue: when running the deploy script, it is failing at the task wait for eaas-server to start up with a 502 Bad Gateway error. At the same time, we were seeing SSL-related errors in the logs for the eaas container. However running this command (gleaned from the setup page of the EaaSI handbook) fixed that issue and (we are assuming) addressed the cause of the 502 error the deploy script was encountering:

$ sudo docker exec eaas sv restart nghttpx

We are able to access the login page using a browser, however if I enter correct credentials I am then taken to a blank page, which eventually redirects back to the login page. Inspecting the source for this page reveals a body tag that contains only a ‘noscript’ tag with a message about EaaSI not working without JavaScript – our browser is running JavaScript, so we do not see this message on the page.

And when we rerun the request we believe the deploy script failed on:

curl -X POST https://eaasi-test-server/emil/environment-repository/actions/prepare

We receive the response:

{"status":"2","message":"Internal error: de.bwl.bwfla.common.exceptions.BWFLAException: Connecting to legacy image-archive failed! (BUILD: 0401EF6384)"}

In case it’s relevant, included below are a number of other problems we encountered and the mitigating steps we took:

Due to being behind a corporate firewall, we had to insert our proxy information (as an environment variable) into the Alpine Dockerfile (/eaasi-installer/eaas/ansible/docker/alpine/Dockerfile ) so that it was able to use its package manager:

[...]
ENV HTTP_PROXY="http://firewall_name.abc.uk:7070"
ENV HTTPS_PROXY="http://firewall_name.abc.uk:7070"
ENV NO_PROXY="localhost,127.0.0.1,192.168.1.1,192.168.1.2"
[...]

We also had to insert our firewall’s certificate into Alpine’s SSL store. In the same file as above:

[...]
COPY gateway.crt .
RUN cat gateway.crt >> /etc/ssl/cert.pem
[...]

Another problem we encountered during deployment was an 'unknown directive “js_import”' error being thrown by the ‘nginx’ container. It appears this directive has been removed from more recent versions of nginx. To fix this we had to pin the nginx image to an older version (1.18.0) here:

eaasi-installer/eaas/ansible/roles/eaas-server/templates/docker/docker-compose.yaml.j2:

[...]
services:
	nginx:
		image: nginx:1.18.0
		container_name: nginx
[...]

Any advice or suggestion would be warmly welcomed!

Are you able to reproduce the issue or did it happen once? The problems described above occur consistently. When what we expected was for the EaaSI web UI to be usable after login.

Urgency: Quite high: we are working on a proof of concept using EaaSI and would like to move it forward as quickly as possible.

[oooleg (Oleg Stobbe)]

Hi @mwhitmor! Sorry for the delayed reaction, we are busy preparing next release.

First of all, your workarounds make sense and some of these have already been implemented as part of the upcoming release too.

But, your problems running EaaSI seem to be rooted in the deployment’s configuration being incompatible with your specific corporate networking setup.

[oooleg (Oleg Stobbe)]

@mwhitmor, please provide additional information for further debugging:

1. Does your corporate firewall terminate SSL connections?
2. Could you send me the backend’s log located at <eaasi-install-dir>/log/server/server.log via a private message? If not, do you see any related errors about connection failures there?
3. What’s the output of the following command:

   $ sudo docker exec -it eaas curl -v -X POST https://eaasi-test-server/emil/environment-repository/actions/prepare
   

[mwhitmor (Michael_Whitmore)]

Hi, thanks for this!

So in answer to 1: i think yes. Our firewall decrypts and re-encrypts SSL traffic, using it’s own certificate. I believe that ‘termination’ is at least part of that process.

I sent you the log, but I will also post the output of the command you mentioned, in case someone else reading this has a similar issue:

*   Trying 192.168.001.[1...
* TCP_NODELAY set
* Connected to 192.168.001.01 (192.168.001.01) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate in certificate chain
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[mwhitmor (Michael_Whitmore)]

Note for anyone reading this thread:

It seems that whilst a change might have to be made to EaaSI to allow for SSL certificates that do not have a domain name (but instead use only an IP address), it transpired that after doing some adjacent work to use self-signed certificates that our own corporate firewall disallows them!

So two things made deployment tricky: trying to use SSL without a fully qualified domain name and trying to use a self signed certificate (with this being specific to our organisation).