How To Use Buildout To Create an HTML5 Application Server

This document will present and explain the improvements of an existing software release available on SlapOS: HTML5 Application Server(html5as). Readers can therefore have an understanding of how Buildout works on SlapOS and Webrunner. To follow the steps in this document, you are supposed to know the architecture of SlapOS and be familiar with Buildout. There is also a tutorial to Extend Software Release: helloworld can be referred to.

Table of Content

Table of Content

1. Introduction of Technologies 1.1 SlapOS 1.2 Webrunner 1.3 Buildout

2. Prerequisites 2.1 IPv6 Internet connection with standard compliant web browser 2.2 A SlapOS Webrunner

3. Deployment of Software Release: html5as

3.1 Connect to your Webrunner 3.2 Select the software to use 3.3 [Improvement] Add parameter 3.4 Access to your bookmark website 3.5 Running process 3.6 [Improvement] Change the default nginx directory 3.6.1 Manually 3.6.2 DevOps: Automate Actions With SlapOS 3.6.2.1 Set Directory 3.6.2.2 Restart 3.7 [Improvement] An Instance Requests Another Instance 3.7.1 Make a Content Delivery Network (CDN) 3.7.1.1 Replicated Instance 3.7.1.2 Request 2 instances from “replicate” instance 3.7.1.3 Resolve the Conflict of Ports 3.7.2 [Improvement] Add a Promise to Check the Site

1. Introduction of Technologies

1.1 SlapOS

SlapOS is a general purpose operating system usable on distributed POSIX infrastructures (Linux, xBSD) to provide and manage software services focussing on cloud orchestration and edge computing. It allows automating DevOps including deployment, monitoring and billing of software usage on arbitrary cloud infrastructures. SlapOS can be used to act as cloud provider selling software as a service (SaaS), to offer platform as a service (PaaS) solutions or also basic infrastructure as a service (IaaS). SlapOS can be run on private hardware and data centers or any public/cloud infrastructure. This document explains the concepts underlying the SlapOS architecture.

1.2 Webrunner

The Webrunner is part of the SlapOS toolbox. It is a platform-as-a-service (PaaS) and browser-based development IDE containing a sandbox for a SlapOS application. It can be used to develop this application itself or host another production environment (like ERP5) which this way also benefits from SlapOS resiliency.

1.3 Buildout

Buildout is an open-source software build tool. Buildout is created using the Python programming language. It implements a principle of separation of configuration from the scripts that do the setting up. Buildout is primarily used to download and set up dependencies in Python eggs format of the software being developed or deployed. Recipes for build tasks in any environment can be created, and many are already available.

Sample configuration

 [buildout]
 develop = .
 parts = test

 [test]
 recipe = zc.recipe.testrunner
 eggs = theegg

2. Prerequisites

2.1 IPv6 Internet connection with standard compliant web browser

If you don't have a IPv6 internet, you can follow How-to Install IPv6 On Your PC.

2.2 A SlapOS Webrunner

As long as have a validated account in Rapid.Space, you can access to Advanced SlapOS UI and apply for a SlapOS Webrunner (Services → Add → SlapOS Web Runner → SlapOS Web Runner → Input a title → Proceed). Wait for a while to have your url in Connection Parameter. It is the url for your Web Runner Frontend. The int-user and int-password are your login information.

3. Deployment of Software Release: html5as

3.1 Connect to your Webrunner

url from Connection Parameters with int-password and int-user(“admin” as usual). After logged in, you will see the platform of Webrunner:

3.2 Select the software to use

1. Click on the Home icon to display the home page
2. Click Open Software Release
3. Select html5as in software under slapos (slapos→software→html5as)
4. Click Open Software The profile of your software displays and you can have a glimpse of Buildout now:
5. Click Save then press the green Play button to build and run your software It takes a while to build your software. There is an error expected, which is the first improvement we are about to make.

3.3 [Improvement] Add parameter

1. Check the logs, there is an error, a parameter is missing Running state is Failed due to Error: Missing option: slap-parameter:download_url. As it implied, we are supposed to add the certain parameter somewhere.
2. Go on Services tab
3. In Services tab go in Parameters tab
4. Click [new]
5. Add the parameter: download_url = https://softinst51206.host.vifib.net/public/bookmark.tgz and click Update Values to have your instance parameters updated.
6. Run your instance by hitting the play button The software will be built immediately this time and please check there is no error left.

3.4 Access to your bookmark website

The URL of your website displayed on Connection Information of Services tab: Now you can access to your website via server_url/bookmark/ (e.g. http://[2001:67c:1254:e:c0::ac97]:8081/bookmark/): Note: IPv6 network required.

3.5 Running process

• Downloader process is in charge of downloading the provided file
• Launcher is a NGINX server web. If you click on RUNNING (the status becomes STOPPED), your service will be stopped and your website will be inaccessible. However, you always can start the service by clicking on STOPPED.

3.6 [Improvement] Change the default nginx directory

To access your website, you have to add /bookmark/ at the end of server_url which is because that the default directory is not properly set. To improve it, we are able to change the default nginx directory manually or automatically(DevOps). In this section, we are going to explain thoroughly the both way.

3.6.1 Manually

1. Go to Editor tab:
   More details on Webrunner Editor Interface explained here.
2. Use the switch from toolbar to be in Working Dir. You'll now see your full folder tree. More explanation about Folder Hierarchy here.
3. Go to instance/slappart0/etc and select nginx.conf. Add “/bookmark/” at the end of the root folder to automatically use the downloaded folder
4. Restart The change is not applied yet, you need to restart: Services → Process → Restart at slappart0:launcher
5. Test it works Ensure the status of slappart0:launcher is RUNNING, meaning it restarted successfully. Now you can go to your server_url directly without /bookmark/ to have your website accessed. The poorness of this manually way, however, requires to be recast each time you deploy this site.

3.6.2 DevOps: Automate Actions With SlapOS

As we did in manual, 2 main actions need to be automated: Set directory and Restart.
Note: Before you started the following part, please undo the actions you did manually by hitting the Play button to redepoly your software, which will remove your manually work as we expected.

3.6.2.1 Set Directory

Set the directory as a parameter of the instance (just like download_url)

1. Go to Editor tab and use the switch to be in This project, Your software is defined in software/html5as

• instance_html5as.cfg is used to configure the instance,
• templates/nginx_conf.in is used as a template to generate the configuration

2. Find where the root path is defined and add a parameter to it. To have default path available as a parameter, edit your instance as how it present here.
   • Add /$${slap-parameter:default-path} after docroot = $${basedirectory:data};
   • Add [slap-parameter] at the end of instance_html5as.cfg:

   # Define default values of parameters here
   [slap-parameter]
   default-path =
   

Available parameters should always be defined in slap-parameter. download_url is also defined as:

```
key download_url slap-parameter:download_url
```

3. Trigger build and run (Play)

   If you trigger it now, it probably will get a md5sum error like:

   [2020-10-20 15:17:39,715] INFO     Error: MD5 checksum mismatch for local resource at '/srv/slapgrid/slappart40/srv/runner/project/slapos/software/html5as/instance_html5as.cfg'.
   [2020-10-20 15:17:39,715] ERROR    Failed to run buildout profile in directory '/srv/slapgrid/slappart40/srv/runner/software/be63092f596f6c6b6cd66d7097f014dc'
   

   It is because instance_html5as.cfg is not properly updated. To update it, you are supposed to Get Or update md5sum: Editor -> instance_html5as.cfg (the file you want to get or update md5sum) -> Menu -> Get or Update md5sum:

   

   Then you will have a message like:

   Md5sum for file [instance_html5as.cfg]: 91d6e7fc3465d0eba1d7dfe9ebb03206
   

   At the bottom of the Editor Interface. Copy Md5sum into This project/software/html5_as/software.cfg. Replace the md5sum of [instance_html5as]:

   

   You are able to build your software successfully now by trigger the Play button.

4. Add the parameter in Services and trigger build and run

   As defined default-path, you are able to add the parameter just like you did for download_url: Services -> Parameters -> [new] -> Input default-path=/bookmark/->Update Values

   

5. Check Nginx configuration

   Go to instance/slappart0/etc and select nginx.conf:

   

   Notice that path for static files is changed from

   root /srv/slapgrid/slappart40/srv/runner/instance/slappart0/srv/html5as/bookmark/
   

3.6.2.2 Restart

Instead of restarting services manually, which generates downtime, we are supposed to use gracefully reload.

You can add graceful reload to your software release in a similar way as this commit. However, there is a trick inside the commit. As you have read some documentation about gracefully reload Nginx, you are expected to find the correct signal to gracefully restart Nginx: use HUP rather than USR1.

The master Nginx process can handle the following signals:

• TERM, INT : Quick shutdown
• QUIT : Graceful shutdown
• KILL : Halts a stubborn process
• HUP : Configuration reload. Start the new worker processes with a new configuration. Gracefully shutdown the old worker processes
• USR1 : Reopen the log files
• USR2 : Upgrade Executable on the fly
• WINCH : Gracefully shutdown the worker processes

Mark sure you have updated the md5sum before you trigger build and run. After that, you will have your nginx-graceful process in slappart0 (partition):

But if check your website, you will notice that you still need to add “/bookmark/” at the end of server_url. This is another improvement we are going to make.

Let's roll back to the top of instance_html5as.conf, you will find a partition tree:

# partition tree
# /
# |- etc/
# |    |- nginx.conf
# |    |- run/
# |         |- html5as (binary)
# |- var/
# |    |- run/
# |    |    |- nginx.pid
# |    |- log/
# |    |    |- nginx.log
# |    |    |- nginx.access.log
# |- srv/
# |    |- html5as/ (doc root)
# |    |        |- index.html
# |    |- backup/

It is pointed to the directory below: Working dir/instance/slappart0.

As the partition tree indicates, Working dir/instance/slappart0/srv/html5as/ is doc root, and the index.html page should be directly under doc root while look at our actual directory tree: the index.html is under Working dir/instance/slappart0/srv/html5as/bookmark/bookmark/. It seems like we extracted website tarball (from download_url = https://softinst51206.host.vifib.net/public/bookmark.tgz ) into a wrong directory. So where can we extract it properly? There is a file named downloader.in inside This project/software/html5as/templates/:

exec {{ param_html5as['path_curl'] }} -ks {{ download_url }} | {{ param_html5as['path_tar'] }} -z -x -C {{ param_html5as['docroot'] }}

To get familiar with extracting a tarball, you can try to type tar --help in your Terminal. Read thoroughly, you will find a command might help:

      --strip-components=NUMBER   strip NUMBER leading components from file names on extraction   

Hence we add it into downloader.in:

exec {{ param_html5as['path_curl'] }} -ks {{ download_url }} | {{ param_html5as['path_tar'] }} -z -x --strip-components=1 -C {{ param_html5as['docroot'] }}

**Note: ** Remember that we have default-path defined as “bookmark/", and our doc root is

docroot = $${basedirectory:data}/$${slap-parameter:default-path}

It can be translated into

 docroot = Working dir/instance/slappart0/srv/html5as/bookmark/

Thus, we are expected strip ONE leading components from file names on extraction.

That's it. Don't forget to get md5sum from downloader.in and update to software.cfg before you build and run. Files in templates will not initially occurs the md5sum error, thus you are supposed to ensure they are up-to-date.

Check index.html is properly placed as expected:

And go to your website via server_url directly. “403 Forbidden” has gone and you have your Bookmark Application eventually.

3.7 [Improvement] An Instance Requests Another Instance

3.7.1 Make a Content Delivery Network (CDN)

A content delivery network (CDN) refers to a geographically distributed group of servers which work together to provide fast delivery of Internet content. A CDN allows for the quick transfer of assets needed for loading Internet content including HTML pages, javascript files, stylesheets, images, and videos. A CDN on SlapOS will do the same thing: the “root” instances receives all the parameters and send them to other instances running the actual frontend. Our goal for this improvement is to have a replicated instance and have 2 instances requested.

3.7.1.1 Replicated Instance

Referring to instance_html5as.cfg, you are able to have an idea of creating a basic replicated instance. There are more information here to help you build one. As the commit indicates, you are supposed to complete the steps as following:

1. Add replicate = ${instance_replicate:output} to [switch-softwaretype] in instance.cfg. In this way, you can change Software Type of main instance from one to another on Parameters of Services tab
2. Get md5sum from instance.cfg and update to [template] in software.cfg
3. Create instance_replicate.cfg as it was defined in instance.cfg. There is only one instance defined in instance_replicate.cfg. We will have it to request two instances later.
4. Detail [instance_replicate] in software.cfg and don't forget to add instance_replicate in parts. You might need to get md5sum from instance_replicate.cfg then update it to [instance_replicate]
5. Trigger Play button to build and run and ensure it completes

At this moment, your instance and application changed nothing, because you are still in default software type. Applying to the replicated instance you just created, you are supposed to replace Software Type of main instance from “default” to “replicate”:

Update Values and trigger Play button again to have your “replicate” instance deployed. You can verify it by checking the partition of the process: there was slappart0 as a “default” software type instead of slappart1.

3.7.1.2 Request 2 instances from “replicate” instance

In instance_replicate.cfg, you already have one instance [instance-1]:

[instance-1]
<= slap-connection
recipe = slapos.cookbook:request
# Name of the instance
name = instance1
# It is the same software as teh current one
software-url = $${slap-connection:software-release-url}
# We want the default behaviour
software-type = default
# What parameter are neede to be retrieved
return = server_url
# Provided parameters
config-default-path = $${slap-parameter:default-path}
config-download_url = $${slap-parameter:download_url}

The second one is quiet same as [instance-1]. Add [instance-2] after [instance-1]:

[instance-2]
<= slap-connection
recipe = slapos.cookbook:request
# Name of the instance
name = instance2
# It is the same software as teh current one
software-url = $${slap-connection:software-release-url}
# We want the default behaviour
software-type = default
# What parameter are neede to be retrieved
return = server_url
# Provided parameters
config-default-path = $${slap-parameter:default-path}
config-download_url = $${slap-parameter:download_url}

And add server_url-2 for the second instance in [publish-connection-infomation] in the end of instance_replicate.cfg, although it doesn't work yet:

[publish-connection-information]
recipe = slapos.cookbook:publish
server_url_1 = $${instance-1:connection-server_url}
server_url_2 = $${instance-2:connection-server_url}

Get md5sum from instance_replicate.cfg and update it to [instance_replicate] on software.cfg then press build and run to have you instance running:

3.7.1.3 Resolve the Conflict of Ports

As the screenshot showed above, there is only one launcher running, meaning that there is only one instance running. The reason accounting for it is that both Nginx instances are using the same port. To solve it, we have to review some steps of adding parameters with DevOps.

We are clear that 2 instances from instance_replicate are based on “default” software type. Thus if we want to find some clues about the port, the answer is expected to be in instance_html5as.cfg, which defines the behaviour of default software type.

As you can tell, the port for the instance in default software type has the exact number of port: 8081. And if search for “port” in this file, we will find how the port applied:

[publish-connection-information]
recipe = slapos.cookbook:publish
server_url = http://[$${html5as:ip}]:$${html5as:port}

It will be written as a part of server_url and be retrieved in instance_replicate.cfg.

In order to customise the port and ideally to make it as a parameter, $${slap-parameter} is expected:

port = $${slap-parameter:default-port}

Get and update md5sum as always.

After default port in default software type defined, the ports for [instance-1] and [instance-2] can be defined in similar way:

[instance-1]
...
config-default-port = $${slap-parameter:default-port}

[instance-2]
...
config-default-port = $${slap-parameter:default-port_2}

[slap-parameter]
...
default-port =

Get and update your md5sum and add parameters as we did for download_url and default-path.

Then build and run your software. As long as it completes successfully, 2 instances wil be both running in slappart1 and slappart2 responsively:

In Connection Information, the server_url for each instance can be found. You can try to access to the website via the url, but a 404 error is waiting for you. A 404 error means that your website is accessible (urls and ports are fine), but no index.html found to display. As it presented above, we should have index.html in Working dir/instance/slappartX/srv/html5as/bookmark/. In fact, we haven't neither in slappart1 nor slappart2. It must be something wrong with downloader!

Fortunately, supervisord tail for “slappartX:downloader” is available by clicking *slappartX:downloader in Process of Services tab.

There is an error: No such file or directory. In slappart1 and slappart2, we just have a directory until an empty html5as (Working dir/instance/slappartX/srv/html5as/). There is no directory named bookmark for downloader.

Take a look at downloader.in:

# Remove exsistant website
rm -r {{ param_html5as['docroot'] }}/*

It is not properly written. Considered the current situation, we can modify it as

# Remove exsistant website
if [ -d "{{ param_html5as['docroot'] }}" ]; then
  rm -r {{ param_html5as['docroot'] }}/*
else
  mkdir {{ param_html5as['docroot'] }}
fi

Don't forget to get and update md5sum of downloader.in before hit build and run.

By now, two instance should be running well with different ports.

3.7.2 [Improvement] Add a Promise to Check the Site

To understand how promises work on SlapOS, the documentation can be referred to.

To add a promise, the README can be an example.

1. Add [promise-check-site] into instance_html5as.cfg

   [promise-check-site]
   <= monitor-promise-base
   module = check_port_listening
   name = check_nginx_port.py
   config-hostname = $${html5as:ip}
   config-port = $${html5as:port}
   

2. Necessary implementation

   In instance_html5as.cfg, extend monitor template and a monitor-base to parts:

   [buildout]
   extends = 
     ${monitor-template:rendered}
   parts = 
     ...
     monitor-base
     promise-check-site
   

   

   In software.cfg, extends the stack and slapis-toolbox in parts:

   [buildout]
   extends =
     ...
     ../../stack/monitor/buildout.cfg
       
    parts = 
    ...
    slapos-toolbox
    ...
   

   

3. Get md5sum from instance_html5as.cfg and update it to software.cfg before trigger Play button to build and run. This time the process will be long because it's installing.

4. Verify the result

   Normally, there are already some informations on Logs to indicate: port connection OK. You can also have a look at Services tab, more monitor processes are listed:

   

   Slapgrid will run each promise every time a partition is processed (every minutes in theory), if the partition is up to date, slapgrid will only run promises anomaly check and save the result in a json file. Here is an exemple of promise result:

   {"result": {"date": "2018-03-22T15:35:07", "failed": false, "message": "buildout is OK", "type": "Test Result"}, "path": "PARTITION_DIRECTORY/etc/plugin/buildout-slappart0-status.py", "name": "buildout-slappart0-status.py", "execution-time": 0.1, "title": "buildout-slappart0-status"}
   

   The promise execution time should be short, by default promise-timeout in slapgrid is to 20 seconds. If a promise runs in more than the defined promise-timeout, the process is killed and a “promise timed out” message is returned. JSON in the folder PARTITION_DIRECTORY/.slapgrid/promise/result, and promise logs are in PARTITION_DIRECTORY/.slapgrid/promise/log.