Ortus Redis Extension

Ortus Redis Extension is a Lucee Extension that allows your CFML server to connect to a Redis server/cluster and leverage it for built-in caching, session storage, and NoSQL document storage.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. Learn More

Redis Commander Dashboard

Requirements

Lucee 5.1.0 and above
Redis 4.0.X and above

Features In A Nutshell

Add Redis functionality to any Lucee application
Install at server level (Available to all contexts)
Create Cache connections in the Lucee web administrator or via Application.cfc to connect to any network-accessable Redis cluster
Set and get objects from Redis via standard CFML functions and tags (cachePut(), cacheGet(), cfcache action="get|put")
Fully supports all built-in Lucee cache functions including wildcard filters
Seamlessly distribute storage of the following to any Redis Cluster ** Lucee session storage ** Lucee client storage ** Lucee Ram resouce (ram://...)
Seamlessly cache the following to any timeout-sensitive Redis key ** Results of database queries ** Results of deterministic functions ** Complex or simple objects in your application's code ** Cached templates (cfcache action="content|cache|serverCache")
Extremely lightweight and fast

Installation

Redis Setup

You will need to install a Redis Server separately. Your Redis cluster can be a single node, or multiple nodes. The nodes can be installed along side your web servers, or on a server farm of their own-- it's entirely up to you. You can also use docker to spin up a Redis instance. You can use the following docker compose file for quick testing:

docker-compose.yml

version: '2'
services:
  redis:
    image: redis:latest
    ports:
      - "6379:6379"

Then issue a docker-compose up and you will have your very own Redis instance running.

Docker Compose Output

You can learn more about Redis replication here: https://redis.io/topics/replication

Lucee Setup

The Redis extension is installed as a Lucee extension into any Lucee Server version 5.1.0 and above. The extension has to be installed at the server level context.

If installed in the server context, you can create server-level caches tha are available to all web contexts in Lucee. The driver will also be availble to all web contexts to add their own local caches or override caches if they need to.

Add Extension Provider

In your Lucee Administrator under Extension > Providers paste in your Ortus provider URL (http://lucee.ortussolutions.com) and click save.

Adding Extension Provider

One added, the new provider URL should show up in the list as verified.

Ortus Extension Provider Added

Installing the Extension

Now click Extensions > Applications and wait for the list to load. There should be a item in the list called Ortus Redis Cache. You can activate it as a trial or as a full version with a license key after you install.

Extension in the list of not installed items

Click it and then click the install button to begin the installation process.

Extension details

Activating the Extension

Extension Menu

Once the extension installs, you will now see a new menu item in the admin called Ortus with a sub menu called Redis Cache. Click this menu and you'll see the activation screen where you can enter the license key and license email with which you purchased the extension with.

You will also have to determine the type of server you are installing the extension to. Development or non-public facing servers are FREE of charge and by default receive up to 4 activations.

Production servers get only 1 activation, so make sure you choose the right server type. Once you get all your information in the form then click on the activate button to finalize the installation. Choose the trial option if you don't have a license and just want to try out the extension. When the trial expires, the cache provider will stop working! The trial is not for production use.

Extension Activation

Note : Development and staging servers are FREE of charge and each license includes up to 4 activations. Production licenses are on a per Lucee instance and are allowed 1 activation. If you have any activation issues please contact us at support@ortussolutions.com. Also, make sure you havea valid internet connection in order to activate your product.

The Ortus Lucee Redis Extension should now be installed on your server and ready to use. Make sure your Redis cluster is running and proceed on to the next step ---> creating a cache.

Important: Please note that the Redis Extension is licensed on a per-JVM basis or on a per container pack (5-container pack). You will need a license for each separate JVM, regardless of how many contexts (sites) are deployed in your Lucee installation. The typical setup is one JVM per physical/virtual server. Please email us at support@ortussolutions.com if you have licensing questions.

Configuration

If you install the Extension in the Server Administrator, you will be able to create server-level caches as well as site-specific caches in each of the Web Administrators.

To add a new cache, click on Services > Cache and you should see a list of existing caches. If there are no existing caches, you should be taken straight to the create screen. Type a ''name'' for this cache connection and choose Ortus Redis Cache from the type dropdown and click create.

Create a new cache

You will then be taken to a page of options for the Redis Cache implementation. Check the Storage box if you want to be able to use this cache for session or client storage distribution. Enter the Redis server IP addresses or host names in the text area one per line. If you have many servers in your Redis cluster, you only need to list a few of them here. The client library will do auto discovery of the rest of the nodes once it connects. Enter the port for the server and fill out the Cache Key Prefix. The key prefix is really important as it allow you to create virtual buckets in the Redis key-value store according to cache connection. If you create multiple cache connections back into Redis make sure their prefixes are unique.

Important: Make sure you use a prefix for uniqueness and to avoid collisions between apps and cache connections.

The Default dropdown allows you to specify this connection to be used as the default storage mechanism for function, object, template, query, or resource caches in Lucee. If you set this at the server level, it can be overridden at the website level. Click Submit to finish creating the cache connection and save its settings.

Cache settings

Usage

There are a number of ways that the extension integrates Redis into your Lucee server. There is built-in caching, manual get/set of objects, and scope storage.

CFML

The first and easiest way to use your new cache is via the built-in cache functions and tags in CFML. You can specify the cache name to use, but by default Lucee will use the cache you have designated as your default object cache.

Caching Objects

If you have selected your cache as the default Object cache in the admin, you can simply use functions like cachePut() and cacheGet(). Redis can handle arbitrary snippets of text (like HTML snippets) and even complex objects such as queries, arrays, or structs. The cache will automatically serialize the values and store them as text. They will be reconstituted when you retreive them from the cache later.

// Set a simple value into the default "object" cache for 2 days
cachePut( "key", "value", 2 );
// Get that value back
cacheGet( "key" );

// Set a value into a specific cache for 1 day
cachePut( "key", "value", 1, "myCacheName" );
// Get that value back
cacheGet( "key", "myCacheName" );

// Cache this array for 3 hours
cachePut( "myArray", [1,2,3,4,5], createTimeSpan(0,3,0,0) );

Caching Function Output

Once you have selected a cache as the default function cache in the admin, you can use Redis to cache the results of oft-run functions. The cache key will be created for you automatically based on a hash of the method arguments thanks to Lucee. Cached functions should therefore be deterministic-- meaning the output of the function is purely a product of its parameters. Using function caching is easy, just add a cachedwithin attribute to the cffunction tag or function declaration.

<!--- Cache results of this function for 5 minutes --->
<cffunction name="myFunc" cachedwithin="#createTimeSpan(0,0,5,0)#">
	<cfargument name="inputText">

	<cfreturn reverse(inputText)>
</cffunction>

<!--- First execution stores result in cache --->
<cfset result = myFunc("Brad Wood")>

<!--- subsequent call skips execution and pulls results from cache --->
<cfset result = myFunc("Brad Wood")>

<!--- Different parameters will create a new cache entry --->
<cfset result = myFunc("Luis Majano")>

This is what that function would look like in script

function myFunc( inputText ) cachedwithin="#createTimeSpan(0,0,5,0)#" {
	return reverse(inputText);
}

Important: Lucee only supports caching functions that accept simple values as paramters. Therefore, a function that accepts an array would not be cacheable.

Caching Queries

Once you have selected a cache as the default query cache in the admin, you can use Redis to cache the results of oft-run database queries. The cache key will be created for you automatically based on a hash of the SQL statement and its parameters thanks to Lucee. Using query caching is as easy as function caching, just add a cachedwithin attribute to your query.

<!--- Cache results of this query for 15 minutes --->
<cfquery datasource="datasourceName" name="qryTest" cachedwithin="#createTimeSpan(0,0,15,0)#">
	select *
	from [tableName]
</cfquery>

Lucee RAM Resources

Lucee has the concept and feature of virtual file systems. Our extension taps into it by allowing you to leverage the ram:// resource to talk to Redis for storing documents and treating Redis like a big file system. This means that any file or directory related tag/function can work with Redis RAM resource like: fileRead(), fileWrite(), fileDelete(), directoryNew(), directoryList(), include, etc. It even works when defining application mappings. You can define Redis to be the default cache for resources by selecting the default element in the cache details page and pointing it to resource.

`Important`: Please note that Lucee only allows you to determine 1 cache connection to be the default for either object, template, query, resource or function caching. So you might end up creating different cache connections if you want to leverage them as default cache locations.

Once you setup Redis to be your default resource cache you can leverage it in many ways:

// Create a mapping to it
this.mappings[ "/cluster-fs" ] = "ram://";


// Do some file operations
fileWrite( "/cluster-fs/suzi.txt", myContent );
include template="/cluster-fs/aop.cfm";
directoryList( "/cluster-fs/caches" );

Per-application Cache Settings

You can speicify what caches are the default storage mechanism for functions, queries, objects, etc in the Lucee server and web administrator. There is one more programmatic level this can be configured as well.

The following settings are available in your Application.cfc to override at an application level. Remember, the "object" cache is used by default if no cache name is specified to functions such as cachePut() and cacheGet().

You can also define an entire cache in your Application.cfc which makes your configuration completely portable.

component{

	this.cache[ '<cache-name>' ] = {
		class  	= 'ortus.extension.cache.Redis.RedisCache',
		storage = false,
		custom={
			host = 'localhost',
			port = 6379,
			keyprefix = "lucee-prefix"
		}
	};

	this.cache.function="<cache-name>";
	this.cache.query="<cache-name>";
	this.cache.object="<cache-name>";
	this.cache.resource="<cache-name>";
	this.cache.template="<cache-name>";
}

Scope storage

Lucee will allow you to seamlessly defer storage of your session and/or client scopes to Redis. You will set and access the scopes like you normally would in CFML, but behind the scenes Lucee stores a cache entry for each user containing their session/client variables in the Redis cluster. This serves several purposes:

Scopes are persisted across server restarts
Lower memory usage since variables are NOT stored in the JVM's heap
User sessions can be shared across web servers (No more sticky sessions)
Redis cluster can be scaled out to handle whatever amount of users you need in a distributed fashion

This means you could take your single-server application with a heavy dependence on session/client variables, and scale it out to as many web servers as neccessary behind a round-robin load balancer. Users are free to roam to any web server they want, and their session variables will follow them anywhere with no extra ovearhead. Perhaps you have Millions of web site visitors and you're running out of heap space due to all the session storage. Push those sessions off to a distributed Redis layer that can scale out to meet demand, and that is no longer a bottleneck.

For both session and client storage, you will need to check the Storage box when creating or editing the cache connection you want to use in the Lucee admin. Remember you can create multuple buckets in Redis to help manage your documents and segregate RAM.

Set a cache as usable for session or client storage

Session

Once the cache is created, enabling session storage is easy. In your Application.cfc simply confirm sessionManagment is turned on and set the following two settings as well in the this scope:

sessionType : This is the type of sessions to use, valid values are cfml and j2ee. Make sure that you select cfml or the extension will NOT work.
sessionStorage : This is the name of the cache defined in your Lucee administrator that you wish to store sessions in
sessionCluster : This controls how/when session data is sent and retrieved from the cache. ** true: Session data is primarily stored in cache. It is always pulled from the cache at the start of each request, and backed up at the end. Use this setting if you have multuple applications using cached storage and you want the full benifit of distributed sessions. ** false: Session data is primarily stored in memory but backed up on every request to the cache. Session data is only pulled from the cache if it doesn't exist in memory. Set this to false if you only have one application using session cache storage and you simply want to persist session storage across restarts.

component{
	this.name = "myApp";

	this.sessionType = "cfml";
	this.sessionManagement = true;
	this.sessionTimeout = createTimeSpan(0,0,30,0);

	// This is the name of the Redis cache to use
	this.sessionStorage = "SessionStorageCache";
	// Set this to true if you have more than 1 application
	this.sessionCluster = true;
}

Important: Make sure your session type is set to `cfml` or our extension will NOT work.

Client

Storing your client scope in Redis is just as easy as session storagey. In your Application.cfc confirm clientManagement is turned on and set the following two settings in the this scope:

clientStorage: This is the name of the cache defined in your Lucee administrator that you wish to store client data in
clientCluster: This controls how/when client data is sent and retrieved from the cache. ** true: Client data is primarily stored in cache. It is always pulled from the cache at the start of each request, and backed up at the end. Use this setting if you have multuple applications using cached storage and you want the full benifit of distributed client data. ** false: Client data is primarily stored in memory but backed up on every request to the cache. Client data is only pulled from the cache if it doesn't exist in memory. Set this to false if you only have one application using client cache storage and you simply want to persist client storage across restarts.

component{
	this.name = "myApp";

	this.clientManagement = true;
	this.clientTimeout = createTimeSpan(25,0,0,0);

	// This is the name of the Redis cache to use
	this.clientStorage = "CientStorageCache";
	// Set this to true if you have more than 1 application
	this.clientCluster = true;
}

Direct Java API

If you need more power or are familiar with the Redis Java API and want to use some features we haven't implemented yet, first let us know so we can consider that addition in future version, and then head over to the Java Cluster to get direct access to the Redis Java SDK. Behind the scenes our extension leverages the Jedis library for Redis.

We have created two new global CFML functions to interact with Redis

redisGetProvider( [cachename=default] ) - Retrieves the underlying Ortus Cache implmentation
redisGetConnectionPool( [cachename=default] ) - Retrieves the Jedis pool for a specific connection cache

`RedisGetProvider()`

RedisGetProvider() returns an instance to the Java class that Ortus wrote that proxies between the Redis client and Lucee. It implements lucee.commons.io.cache.Cache. For the most part you probably won't need this class, but there are a few methods in the Lucee cache interface that aren't available to you via CFML. You can also dump the class and check out all of its functions.

// Get the Ortus Java Provider for the default object cache
myProvider = RedisGetProvider();

// Get the Ortus Java Provider for the queryStorage cache
myProvider = RedisGetProvider( "queryStorage" );

// Get all keys in the cache as a java.util.List
keys = myProvider.keys();

// Get all the values in the cache as a java.util.List
values = myProvider.values();

`RedisGetConnectionPool()`

RedisGetConnectionPool() Returns access to a cache's Redis Connection Jedis Pool. It implements redis.clients.jedis.JedisPool - https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/jedis/JedisPool.java. This will give you direct access to the Jedis Pool connection for that cache.

pool = redisGetConnectionPool( 'sessions' );

// Get funky....

Debugging/Troubleshooting

Here's a general list of troubleshooting steps if you're getting errors. The log files are VERY important since the error Lucee throws doesn't always contain the information you need. You will have to crack open your Lucee logs to get the actual error out.

Look in the "out" and "err" logs for information. These are usually found in the servlet container's log directory. For instance, if you're using Tomcat, look in Tomcat's log directory. There are often messages logged there that never make it up to the CFML engine.
The Log4J output of the underlying Redis Java SDK will be redirected to your Lucee server's "Application" log. Adjusting the logging level for this log will affect how much information you get from Redis.
Always scroll to the bottom of Java stack traces to look for a "caused by" section which is the original error.
If you are getting connection errors or Null pointer exceptions, check the spelling of your hostnames and ports and make sure Redis is actually running with the bucket name you specified in the config.
Once the library connects, you should see lines in the "out" log showing the servers being connected to successfully.
Open up the Redis web admin on one of your nodes and watch the "Ops per second" graph while you hit your app. If you see spikes in the graph every time you hit the app, then it's working!

Help & Support

If you need any help related to our Redis Extension product, you can use our online help group at http://groups.google.com/a/ortussolutions.com/forum/#!forum/Redis- Lucee -ext. If you need any type of custom consulting or support package hours, please contact us at support@ortussolutions.com or visit us at www.ortussolutions.com.