Level Up Nginx With Openresty and Lua

Introduction & Background

I honestly love Nginx. It was by chance that, I got to work on Nginx before Apache, but it was by choice that I chose to deploy Nginx as my webserver rather than its elder sibling.
But, recently I felt the need to do more than the usual stuff with Nginx. Apache does provide some of the stuff out of the box. To be able to use Nginx for more sophisiticated usages, I had to take it a notch above than just being able to serve static content at a blazingly fast speed. This blog entry describes the need, solution and approach of taking nginx to the next level.

First, lets draw up a small comparison of Apache v/s Nginx.

Nginx draws its “Blazingly Fast” badge from its Event Driven Architecture rather than Apache’s Per-connection-thread model(though Apache has been made to stand up to the traffic a modern web-app draws by the use of mpm_prefork, mpm_worker, mpm_event request handling mechanisms.)This “shortcoming” is due to Apache precluding Nginx and a more thorough understanding of the C10K problem when Nginx was implemented.
Apache scores over Nginx in the flexibility, ease of configuration, ease of deploying dynamic modules, thriving ecosystem and ability to serve up dynamic content, multi-line comments in the configuration and ability to read out environment variables. This is a testament to the fact that Apache is the Big Brother and a more matured webserver.

So a few questions arise :
- So how does Nginx tilt the balance in its favour again?
- Or is there a balance to be restored?Aren’t Apache and Nginx better together when used in conjunction?
- Is being “Blazingly Fast” good enough to opt for Nginx while trading off the other important aspects of a webserver?

My take on the above is that , though Nginx and Apache work very well together, adding another moving part adds a bit to the devops effort.
Although it is not a face-off, I very much inclined to use Nginx as a standalone Webserver(it’s just a matter of personal taste.)
But it is imperative that Nginx deals well with dynamic content(especially environment variables) for the sake of traffic engineering, canary deployments etc.

Objective To achieve
In this blog post, I will give a brief introduction on how someone who feels at home administering Nginx can have the ability to deal with dynamic configuration and taking baby steps towards extending Nginx into a full blown web application server.

Problem Statement
Lets define the a problem statement for easier understanding.
The problem statement that we will solve is a seamless semiautomated blue/green deployment from a distance,. In one of the previous blog posts, I described how one can have a Correct & Cohesive Blue/Green Environment Interpolation using Nginx

The summary of that post is that using a upstream block having and array of servers to handle the reverse proxying of requests we can manage Blue Green Environment Interpolation with a sliver of downtime. The trick is to use a combination of the backup and down flags against a server entry to manipulate the traffic. While the approach works like a charm, it requires one to ssh into the server,edit the configuration and hot reload nginx using nginx -s reload. This becomes especially clunky when you look at it by the centralized devops perspective.

Probable Solutions

To implement the upstream management as described above from a distance, we would need a way for nginx to pick up values for the down and backup values from a place where they can be changed dynamically. I first thought of using environment vars to set the values and having Nginx read them on the fly.

Gotchas

Inspite of this, Nginx+Lua does profess huge benefits; Manipulating Dynamic content/configuration is not the least of it.

Openresty

Openresty is a platform that builds on top of Nginx by integrating the core Nginx Module, LuaJit and many 3rd party modules effectively allowing you to embed the power of Lua in your Nginx Webserver. The Openresty official site presents a great introduction.Do check it out.
We will use Openresty instead of compiling Nginx from source to enable the Lua modules.

The official site also documents the installation steps lucidly.

Once installed, the installation is found at /usr/local/openresty/.

The installation directory structure is as follows:

/usr/local/openresty/
|--/bin
|--/luajit
|--/lualib
|--/nginx
    |-..
    |-..
    |-..
    |-/sbin
      |-./nginx
    |/conf
      |-nginx.conf
|--/pod
|--/site  

You will notice that in addition to the Lua modules, the directory wraps a installation of nginx. The directory structure of /usr/local/openresty/nginx is similiar to vanilla nginx installations.
To run this installation of nginx ,we need to add the /usr/local/openresty/nginx/sbin/nginx executable to the PATH environment var. Henceforth ,we will be using this nginx script to run our Webserver.
The relevant nginx configuration is sourced from /usr/local/openresty/nginx/conf/nginx.conf file.

Note : The above assumptions are simplifies for the purpose of the blog post.To be able to use it in production, you need to add init scripts, configure users as you would normally do.
For more detailed information, the Getting Started Section is a great read.

Openresty comes with the Lua module enabled by default.

What we will try to achieve?

As mentioned in the problem statement, we will expand on the Blue/Green Environment interpolation from a distance i.e without having to ssh into the server and manually editing the value of down and backup flags.

How?
As mentioned, Openresty comes with the Lua module enabled by default.This means we can embed Lua code in our nginx.conf file and have it interpreted on demand.

Openresty also ships with a nifty module called lua-upstream-nginx-module which can be used for upstream management.
The method of this module that caught my eye was
ok, err = upstream.set_peer_down(upstream_name, is_backup, peer_id, down_value)
Looking at the usage ,it shows that we can set the down flag dynamically by calling this method.
So with a clever use of set_peer_down we can achieve upstream management from a distance.

Configuration
The nginx.conf file that we will use to dynamically handle the upstreams will have the following content.(Some of the location and server configurations are edited out.)

worker_processes 1;

events { worker_connections 1024; }

http {

upstream backend {
    server blueapp.example.com:443 fail_timeout=0;
    server greenapp.example.com:443 fail_timeout=0 backup down;
}

  #Gzip Configs
  ..
  #Content Optimisations

	server {

        #SSL Configs
        ..
        #Security hardening
        ..
        #Static configs
        ..


	  location = /upstream_switch{
	    auth_basic_user_file /usr/local/openresty/nginx/http_auth/.htpasswd;
	    content_by_lua_block{
		local upstream = require "ngx.upstream"
		local args = ngx.req.get_uri_args()



		--custom function to figure out the length of a table

		local function len_of(taab)
			count = 0
			for index,value in pairs(taab) do
				count = count + 1
			end
			return count
		end

    <br>

		--custom function to verify whether a table has a value

		local function has_value (tab, val)
                    for index, value in ipairs (tab) do
                        if value == val then
                                return true
                        end
                    end

                    return false
                end



		--custom function to verify whether a table has a key

                local function has_key (tab, key)
                    for index, value in ipairs (tab) do
                        if value == key then
                                return true
                        end
                    end

                    return false
                end

		--declare the variables.Currently hardcoded as per the upstreams that are defined.

		local valid_primary_peers_index ={"0"}
                local valid_backup_peers_index ={"0"}
                local valid_peer_status = {"true","false"}
                local valid_peer_type = {"primary","backup"}
                local valid_params = {"peer_index","peer_down_status_to_set","peer_type"}
                local upstream_name = "backend"

		--validate the query params length

		local qry_arg_length = len_of(args)
		if (qry_arg_length ~= 3) then
		ngx.say("Wrong number of params passed: ",qry_arg_length)
		return
		end		



		--validate our query params to ensure they contain the correct parameters

		for k,v in pairs(args) do
			if not has_key(valid_params,k) then
				ngx.say("invalid param in query string ",k)
				return
		 	end
		end


		 --validate our query param and their values

		 for k,v in pairs(args) do
                        if not has_value(valid_primary_peers_index,v) and not has_value(valid_peer_status,v) and not has_value(valid_peer_type,v) and not has_value(valid_backup_peers_index,v) then
                                ngx.say("invalid param value "..v.." for "..k)
				return
                        end
                end

		--extract the requested values

	       	local requested_peer_type = args["peer_type"]
              	local requested_peer_status_to_set = args["peer_down_status_to_set"]
             	local requested_peer_index = args["peer_index"]

             	--derieve the status to set

		local bool = requested_peer_status_to_set == "true" and true or false

		--notify the user what is to be done
		ngx.say(requested_peer_type.." peer_type will have the down value as  "..requested_peer_status_to_set.." and its index is  "..requested_peer_index.." if everything is successful")

		--since the nginx lua module needs to pass in the value of the peer_type in form of is_backup ,we need to if/else blocks
		if requested_peer_type == 'primary' then
			ok,err = upstream.set_peer_down(upstream_name,false,requested_peer_index,bool)
			if not ok then
                                ngx.say("Could not set the primary peer with index "..requested_peer_index.." to the status "..bool.."- ",err)
				return
                        end			
		elseif requested_peer_type =='backup' then
			ok,err = upstream.set_peer_down(upstream_name,true,requested_peer_index,bool)
			if not ok then
				ngx.say("Could not set the backup peer with index "..requested_peer_index.." to the status "..bool.."- ",err)
				return
			end

		else
			ngx.say("Incorrect peer type requested")
			return
		end

		ngx.say("The above action has been successfully executed")
		return				

		}
	}

     location ~ ^/ {
        proxy_pass https://backend;
        error_page 502 503 504  = @maintainence;
      }

    #Fallback Location for Handling 502,503,504
    ...
  }

}

Agreed it is not the prettiest Lua code, as the code parses and validates the request params by hand.
For a more abstracted approach, one can try lapis web application development framework which wraps a openresty wrapper around a nginx (whew!).
Please note that to use Lapis ,only the executable file changes.The way to configure nginx remains the same (via nginx.conf)

A word on whats happening here
1. We define the upstream as an array of two servers.One is annotated with the backup keyword which belongs to the backup group and the one not annotated with this keyword belongs to the primary group.Each servers in the backup/primary group are part of a zero-indexed array. So by this logic
- https://blueapp.example.com is the 0 indexed element in the primary group.Initially it is up and serves traffic.
- https://greenapp.example/com is the 0 indexed element in the backup group.Initially it is down and doesnt serve traffic till the above is down.

2.As mentioned ,most of the code in the /upstream_switch location is for validation and request parsing.Lua doesn’t have clean cut iterator functions that one would expect.So need to use the pairs() and ipairs() functions to validate the request params.

3.The content_by_lua_block block allows the Lua code to formulate a response.This is a very powerful feature that enables Nginx to perform dynamic processing rather than just reverse proxying requests.

4.This processing can take many forms.For the problem statement here, we will dynamically manipulate the upstream.

5.The core of Nginx’s abilities is exposed via the ngx object of lua-nginx-module. The functionngx.req.get_uri_args() allows us to get the request params for the location /upstream_switch

6.After (statically) validating the requests, we call the function to mutate the down flag(true for making it down, false for making it up) thus:
- For https://blueapp.example.com (primary group) upstream.set_peer_down(upstream_name,false,requested_peer_index,down) - For https://greenapp.example.com (backup group) upstream.set_peer_down(upstream_name,true,requested_peer_index,down) - Note the difference between the second parameter is_backup which denotes the primary/secondary group that this server belongs to.
- Also the value of the down flag remains the same till then time the server is bounced or is mutated via this API.

7.This location is protected by a .htacess file.For a more sophisiticated authentication, we can have a bearer token which is sourced from a redis data-store or use lapis.

Usage
With the above explanation out of the way, one can use the above setup as :
1.To make the Blue-App down,
https://app.example.com/upstream_switch?peer_type=primary&peer_index=0&peer_down_status_to_set=true

2.To make the Green-App down,
https://app.example.com/upstream_switch?peer_type=backup&peer_index=0&peer_down_status_to_set=true

3.To make the Blue-App up,
https://app.example.com/upstream_switch?peer_type=primary&peer_index=0&peer_down_status_to_set=false

4.To make the Green-App up,
https://app.example.com/upstream_switch?peer_type=backup&peer_index=0&peer_down_status_to_set=false

Objective Achieved
With the above setup, we are able to expose a secure upstream manipulation endpoint which is baked into the frontline server itself.
Since this service is exposed via REST, you can do all upstream management (Blue/Green environment interoperation) from a distance, eliminating the need to ssh into the server.
This is another crucial step to a centralized ops.
Please note that using Lua for will not affect the speed of Nginx while serving the static content or functioning as reverse proxy.

Further steps
We have seen how we can embed the power of Lua into Nginx using Openresty. Upstream manipulation from a distance is just one of the use cases.
Now that our trusted Nginx has the ability to deal with dynamic content, we can extend its abilities to incorporate sophisticated traffic engineering mechanisms like (dynamic proxies, fallbacks, bit rate adjustments request rate throttling etc.)

Do drop in what you achieve using this as a starting point in the comments section below!
Cheers!