Redis programmability

• 1: Redis functions
• 2: Scripting with Lua
• 3: Redis Lua API reference
• 4: Debugging Lua scripts in Redis

1 - Redis functions

Redis Functions is an API for managing code to be executed on the server. This feature, which became available in Redis 7, supersedes the use of EVAL in prior versions of Redis.

Prologue (or, what’s wrong with Eval Scripts?)

Prior versions of Redis made scripting available only via the EVAL command, which allows a Lua script to be sent for execution by the server. The core use cases for Eval Scripts is executing part of your application logic inside Redis, efficiently and atomically. Such script can perform conditional updates across multiple keys, possibly combining several different data types.

Using EVAL requires that the application sends the entire script for execution every time. Because this results in network and script compilation overheads, Redis provides an optimization in the form of the EVALSHA command. By first calling SCRIPT LOAD to obtain the script’s SHA1, the application can invoke it repeatedly afterward with its digest alone.

By design, Redis only caches the loaded scripts. That means that the script cache can become lost at any time, such as after calling SCRIPT FLUSH, after restarting the server, or when failing over to a replica. The application is responsible for reloading scripts during runtime if any are missing. The underlying assumption is that scripts are a part of the application and not maintained by the Redis server.

This approach suits many light-weight scripting use cases, but introduces several difficulties once an application becomes complex and relies more heavily on scripting, namely:

1. All client application instances must maintain a copy of all scripts. That means having some mechanism that applies script updates to all of the application’s instances.
2. Calling cached scripts within the context of a transaction increases the probability of the transaction failing because of a missing script. Being more likely to fail makes using cached scripts as building blocks of workflows less attractive.
3. SHA1 digests are meaningless, making debugging the system extremely hard (e.g., in a MONITOR session).
4. When used naively, EVAL promotes an anti-pattern in which scripts the client application renders verbatim scripts instead of responsibly using the !KEYS and ARGV Lua APIs.
5. Because they are ephemeral, a script can’t call another script. This makes sharing and reusing code between scripts nearly impossible, short of client-side preprocessing (see the first point).

To address these needs while avoiding breaking changes to already-established and well-liked ephemeral scripts, Redis v7.0 introduces Redis Functions.

What are Redis Functions?

Redis functions are an evolutionary step from ephemeral scripting.

Functions provide the same core functionality as scripts but are first-class software artifacts of the database. Redis manages functions as an integral part of the database and ensures their availability via data persistence and replication. Because functions are part of the database and therefore declared before use, applications aren’t required to load them during runtime nor risk aborted transactions. An application that uses functions depends only on their APIs rather than on the embedded script logic in the database.

Whereas ephemeral scripts are considered a part of the application’s domain, functions extend the database server itself with user-provided logic. They can be used to expose a richer API composed of core Redis commands, similar to modules, developed once, loaded at startup, and used repeatedly by various applications / clients. Every function has a unique user-defined name, making it much easier to call and trace its execution.

The design of Redis Functions also attempts to demarcate between the programming language used for writing functions and their management by the server. Lua, the only language interpreter that Redis presently support as an embedded execution engine, is meant to be simple and easy to learn. However, the choice of Lua as a language still presents many Redis users with a challenge.

The Redis Functions feature makes no assumptions about the implementation’s language. An execution engine that is part of the definition of the function handles running it. An engine can theoretically execute functions in any language as long as it respects several rules (such as the ability to terminate an executing function).

Presently, as noted above, Redis ships with a single embedded Lua 5.1 engine. There are plans to support additional engines in the future. Redis functions can use all of Lua’s available capabilities to ephemeral scripts, with the only exception being the Redis Lua scripts debugger.

Functions also simplify development by enabling code sharing. Every function belongs to a single library, and any given library can consist of multiple functions. The library’s contents are immutable, and selective updates of its functions aren’t allowed. Instead, libraries are updated as a whole with all of their functions together in one operation. This allows calling functions from other functions within the same library, or sharing code between functions by using a common code in library-internal methods, that can also take language native arguments.

Functions are intended to better support the use case of maintaining a consistent view for data entities through a logical schema, as mentioned above. As such, functions are stored alongside the data itself. Functions are also persisted to the AOF file and replicated from master to replicas, so they are as durable as the data itself. When Redis is used as an ephemeral cache, additional mechanisms (described below) are required to make functions more durable.

Like all other operations in Redis, the execution of a function is atomic. A function’s execution blocks all server activities during its entire time, similarly to the semantics of transactions. These semantics mean that all of the script’s effects either have yet to happen or had already happened. The blocking semantics of an executed function apply to all connected clients at all times. Because running a function blocks the Redis server, functions are meant to finish executing quickly, so you should avoid using long-running functions.

Loading libraries and functions

Let’s explore Redis Functions via some tangible examples and Lua snippets.

At this point, if you’re unfamiliar with Lua in general and specifically in Redis, you may benefit from reviewing some of the examples in Introduction to Eval Scripts and Lua API pages for a better grasp of the language.

Every Redis function belongs to a single library that’s loaded to Redis. Loading a library to the database is done with the FUNCTION LOAD command. Let’s try loading an empty library:

redis> FUNCTION LOAD Lua mylib ""
(error) ERR No functions registered

The error is expected, as there are no functions in the loaded library. Despite the error, we can see that the basic form of invoking FUNCTION LOAD requires three arguments: the engine’s identifier (Lua), the library’s name (mylib), and the library’s source code.

Every library needs to include at least one registered function to load successfully. A registered function is named and acts as an entry point to the library. When the target execution engine handles the FUNCTION LOAD command, it registers the library’s functions.

The Lua engine compiles and evaluates the library source code when loaded, and expects functions to be registered by calling the redis.register_function() API.

The following snippet demonstrates a simple library registering a single function named knockknock, returning a string reply:

redis.register_function(
  'knockknock',
  function() return 'Who\'s there?' end
)

In the example above, we provide two arguments about the function to Lua’s redis.register_function() API: its registered name and a callback.

We can load our library and use FCALL to call the registered function. Because redis-cli doesn’t play nicely with newlines, we’ll just strip these from the code:

redis> FUNCTION LOAD Lua mylib "redis.register_function('knockknock', function() return 'Who\\'s there?' end)"
OK
redis> FCALL knockknock 0
"Who's there?"

Note that we’ve provided FCALL with two arguments: the function’s registered name and the numeric value 0. This numeric value indicates the number of key names that follow it (the same way EVAL and EVALSHA work).

We’ll explain immediately how key names and additional arguments are available to the function. As this simple example doesn’t involve keys, we simply use 0 for now.

Input keys and regular arguments

Before we move to the following example, it is vital to understand the distinction Redis makes between arguments that are names of keys and those that aren’t.

While key names in Redis are just strings, unlike any other string values, these represent keys in the database. The name of a key is a fundamental concept in Redis and is the basis for operating the Redis Cluster.

Important: To ensure the correct execution of Redis Functions, both in standalone and clustered deployments, all names of keys that a function accesses must be explicitly provided as input key arguments.

Any input to the function that isn’t the name of a key is a regular input argument.

Now, let’s pretend that our application stores some of its data in Redis Hashes. We want an HSET-like way to set and update fields in said Hashes and store the last modification time in a new field named _last_modified_. We can implement a function to do all that.

Our function will call TIME to get the server’s clock reading and update the target Hash with the new fields' values and the modification’s timestamp. The function we’ll implement accepts the following input arguments: the Hash’s key name and the field-value pairs to update.

The Lua API for Redis Functions makes these inputs accessible as the first and second arguments to the function’s callback. The callback’s first argument is a Lua table populated with all key names inputs to the function. Similarly, the callback’s second argument consists of all regular arguments.

The following is a possible implementation for our function and its library registration:

local function my_hset(keys, args)
  local hash = keys[1]
  local time = redis.call('TIME')[1]
  return redis.call('HSET', hash, '_last_modified_', time, unpack(args))
end

redis.register_function('my_hset', my_hset)

If we create a new file named mylib.lua that consists of the library’s definition, we can load it like so (without stripping the source code of helpful whitespaces):

$ cat mylib.lua | redis-cli -x FUNCTION LOAD Lua mylib REPLACE

We’ve added the REPLACE modifier to the call to FUNCTION LOAD to tell Redis that we want to overwrite the existing library definition. Otherwise, we would have gotten an error from Redis complaining that the library already exists.

Now that the library’s updated code is loaded to Redis, we can proceed and call our function:

redis> FCALL my_hset 1 myhash myfield "some value" another_field "another value"
(integer) 3
redis> HGETALL myhash
1) "_last_modified_"
2) "1640772721"
3) "myfield"
4) "some value"
5) "another_field"
6) "another value"

In this case, we had invoked FCALL with 1 as the number of key name arguments. That means that the function’s first input argument is a name of a key (and is therefore included in the callback’s keys table). After that first argument, all following input arguments are considered regular arguments and constitute the args table passed to the callback as its second argument.

Expanding the library

We can add more functions to our library to benefit our application. The additional metadata field we’ve added to the Hash shouldn’t be included in responses when accessing the Hash’s data. On the other hand, we do want to provide the means to obtain the modification timestamp for a given Hash key.

We’ll add two new functions to our library to accomplish these objectives:

1. The my_hgetall Redis Function will return all fields and their respective values from a given Hash key name, excluding the metadata (i.e., the _last_modified_ field).
2. The my_hlastmodified Redis Function will return the modification timestamp for a given Hash key name.

The library’s source code could look something like the following:

local function my_hset(keys, args)
  local hash = keys[1]
  local time = redis.call('TIME')[1]
  return redis.call('HSET', hash, '_last_modified_', time, unpack(args))
end

local function my_hgetall(keys, args)
  redis.setresp(3)
  local hash = keys[1]
  local res = redis.call('HGETALL', hash)
  res['map']['_last_modified_'] = nil
  return res
end

local function my_hlastmodified(keys, args)
  local hash = keys[1]
  return redis.call('HGET', keys[1], '_last_modified_')
end

redis.register_function('my_hset', my_hset)
redis.register_function('my_hgetall', my_hgetall)
redis.register_function('my_hlastmodified', my_hlastmodified)

While all of the above should be straightforward, note that the my_hgetall also calls redis.setresp(3). That means that the function expects RESP3 replies after calling redis.call(), which, unlike the default RESP2 protocol, provides dictionary (associative arrays) replies. Doing so allows the function to delete (or set to nil as is the case with Lua tables) specific fields from the reply, and in our case, the _last_modified_ field.

Assuming you’ve saved the library’s implementation in the mylib.lua file, you can replace it with its (optional) description with:

$ cat mylib.lua | redis-cli -x FUNCTION LOAD Lua mylib REPLACE DESCRIPTION "My application's Hash data type enhancements"

Once loaded, you can call the library’s functions with FCALL:

redis> FCALL my_hgetall 1 myhash
1) "myfield"
2) "some value"
3) "another_field"
4) "another value"
redis> FCALL my_hlastmodified 1 myhash
"1640772721"

You can also get the library’s details with the FUNCTION LIST command:

redis> FUNCTION LIST
1) 1) "library_name"
   2) "mylib"
   3) "engine"
   4) "LUA"
   5) "description"
   6) "My application's Hash data type enhancements"
   7) "functions"
   8) 1) 1) "name"
         2) "my_hset"
         3) "description"
         4) (nil)
      2) 1) "name"
         2) "my_hgetall"
         3) "description"
         4) (nil)
      3) 1) "name"
         2) "my_hlastmodified"
         3) "description"
         4) (nil)

You can see that it is easy to update our library with new capabilities.

Reusing code in the library

On top of bundling functions together into database-managed software artifacts, libraries also facilitate code sharing. We can add to our library an error handling helper function called from other functions. The helper function check_keys() verifies that the input keys table has a single key. Upon success it returns nil, otherwise it returns an error reply.

The updated library’s source code would be:

local function check_keys(keys)
  local error = nil
  local nkeys = table.getn(keys)
  if nkeys == 0 then
    error = 'Hash key name not provided'
  elseif nkeys > 1 then
    error = 'Only one key name is allowed'
  end

  if error ~= nil then
    redis.log(redis.LOG_WARNING, error);
    return redis.error_reply(error)
  end
  return nil
end

local function my_hset(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  local hash = keys[1]
  local time = redis.call('TIME')[1]
  return redis.call('HSET', hash, '_last_modified_', time, unpack(args))
end

local function my_hgetall(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  redis.setresp(3)
  local hash = keys[1]
  local res = redis.call('HGETALL', hash)
  res['map']['_last_modified_'] = nil
  return res
end

local function my_hlastmodified(keys, args)
  local error = check_keys(keys)
  if error ~= nil then
    return error
  end

  local hash = keys[1]
  return redis.call('HGET', keys[1], '_last_modified_')
end

redis.register_function('my_hset', my_hset)
redis.register_function('my_hgetall', my_hgetall)
redis.register_function('my_hlastmodified', my_hlastmodified)

After you’ve replaced the library in Redis with the above, you can immediately try out the new error handling mechanism:

127.0.0.1:6379> FCALL my_hset 0 myhash nope nope
(error) Hash key name not provided
127.0.0.1:6379> FCALL my_hgetall 2 myhash anotherone
(error) Only one key name is allowed

And your Redis log file should have lines in it that are similar to:

...
20075:M 1 Jan 2022 16:53:57.688 # Hash key name not provided
20075:M 1 Jan 2022 16:54:01.309 # Only one key name is allowed

Functions in cluster

As noted above, Redis automatically handles propagation of loaded functions to replicas. In a Redis Cluster, it is also necessary to load functions to all cluster nodes. This is not handled automatically by Redis Cluster, and needs to be handled by the cluster administrator (like module loading, configuration setting, etc.).

As one of the goals of functions is to live separately from the client application, this should not be part of the Redis client library responsibilities. Instead, redis-cli --cluster-only-masters --cluster call host:port FUNCTION LOAD ... can be used to execute the load command on all master nodes.

Also, note that redis-cli --cluster add-node automatically takes care to propagate the loaded functions from one of the existing nodes to the new node.

Functions and ephemeral Redis instances

In some cases there may be a need to start a fresh Redis server with a set of functions pre-loaded. Common reasons for that could be:

• Starting Redis in a new environment
• Re-starting an ephemeral (cache-only) Redis, that uses functions

In such cases, we need to make sure that the pre-loaded functions are available before Redis accepts inbound user connections and commands.

To do that, it is possible to use redis-cli --functions-rdb to extract the functions from an existing server. This generates an RDB file that can be loaded by Redis at startup.

Function flags

Redis needs to have some information about how a function is going to behave when executed, in order to properly enforce resource usage policies and maintain data consistency.

For example, Redis needs to know that a certain function is read-only before permitting it to execute using FCALL_RO on a read-only replica.

By default, Redis assumes that all functions may perform arbitrary read or write operations. Function Flags make it possible to declare more specific function behavior at the time of registration. Let’s see how this works.

In our previous example, we defined two functions that only read data. We can try executing them using FCALL_RO against a read-only replica.

redis > FCALL_RO my_hgetall 1 myhash
(error) ERR Can not execute a function with write flag using fcall_ro.

Redis returns this error because a function can, in theory, perform both read and write operations on the database. As a safeguard and by default, Redis assumes that the function does both, so it blocks its execution. The server will reply with this error in the following cases:

1. Executing a function with FCALL against a read-only replica.
2. Using FCALL_RO to execute a function.
3. A disk error was detected (Redis is unable to persist so it rejects writes).

In these cases, you can add the no-writes flag to the function’s registration, disable the safeguard and allow them to run. To register a function with flags use the named arguments variant of redis.register_function.

The updated registration code snippet from the library looks like this:

redis.register_function('my_hset', my_hset)
redis.register_function{
  function_name='my_hgetall',
  callback=my_hgetall,
  flags={ 'no-writes' }
}
redis.register_function{
  function_name='my_hlastmodified',
  callback=my_hlastmodified,
  flags={ 'no-writes' }
}

Once we’ve replaced the library, Redis allows running both my_hgetall and my_hlastmodified with FCALL_RO against a read-only replica:

redis> FCALL_RO my_hgetall 1 myhash
1) "myfield"
2) "some value"
3) "another_field"
4) "another value"
redis> FCALL_RO my_hlastmodified 1 myhash
"1640772721"

For the complete documentation flags, please refer to Script flags.

2 - Scripting with Lua

Redis lets users upload and execute Lua scripts on the server. Scripts can employ programmatic control structures and use most of the commands while executing to access the database. Because scripts execute in the server, reading and writing data from scripts is very efficient.

Redis guarantees the script’s atomic execution. While executing the script, all server activities are blocked during its entire runtime. These semantics mean that all of the script’s effects either have yet to happen or had already happened.

Scripting offers several properties that can be valuable in many cases. These include:

• Providing locality by executing logic where data lives. Data locality reduces overall latency and saves networking resources.
• Blocking semantics that ensure the script’s atomic execution.
• Enabling the composition of simple capabilities that are either missing from Redis or are too niche to a part of it.

Lua lets you run part of your application logic inside Redis. Such scripts can perform conditional updates across multiple keys, possibly combining several different data types atomically.

Scripts are executed in Redis by an embedded execution engine. Presently, Redis supports a single scripting engine, the Lua 5.1 interpreter. Please refer to the Redis Lua API Reference page for complete documentation.

Although the server executes them, Eval scripts are regarded as a part of the client-side application, which is why they’re not named, versioned, or persisted. So all scripts may need to be reloaded by the application at any time if missing (after a server restart, fail-over to a replica, etc.). As of version 7.0, Redis Functions offer an alternative approach to programmability which allow the server itself to be extended with additional programmed logic.

Getting started

We’ll start scripting with Redis by using the EVAL command.

Here’s our first example:

> EVAL "return 'Hello, scripting!'" 0
"Hello, scripting!"

In this example, EVAL takes two arguments. The first argument is a string that consists of the script’s Lua source code. The script doesn’t need to include any definitions of Lua function. It is just a Lua program that will run in the Redis engine’s context.

The second argument is the number of arguments that follow the script’s body, starting from the third argument, representing Redis key names. In this example, we used the value 0 because we didn’t provide the script with any arguments, whether the names of keys or not.

Script parameterization

It is possible, although highly ill-advised, to have the application dynamically generate script source code per its needs. For example, the application could send these two entirely different, but in the same time perfectly identical scripts:

redis> EVAL "return 'Hello'" 0
"Hello"
redis> EVAL "return 'Scripting!'" 0
"Scripting!"

Although this mode of operation isn’t blocked by Redis, it is an anti-pattern due to script cache considerations (more on the topic below). Instead of having your application generate subtle variations of the same scripts, you can parametrize them and pass any arguments needed for to execute them.

The following example demonstrates how to achieve the same effects as above, but via parameterization:

redis> EVAL "return ARGV[1]" 0 Hello
"Hello"
redis> EVAL "return ARGV[1]" 0 Parameterization!
"Parameterization!"

At this point, it is essential to understand the distinction Redis makes between input arguments that are names of keys and those that aren’t.

While key names in Redis are just strings, unlike any other string values, these represent keys in the database. The name of a key is a fundamental concept in Redis and is the basis for operating the Redis Cluster.

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

Any input to the function that isn’t the name of a key is a regular input argument.

In the example above, both Hello and Parameterization! regular input arguments for the script. Because the script doesn’t touch any keys, we use the numerical argument 0 to specify there are no key name arguments. The execution context makes arguments available to the script through KEYS and ARGV global runtime variables. The KEYS table is pre-populated with all key name arguments provided to the script before its execution, whereas the ARGV table serves a similar purpose but for regular arguments.

The following attempts to demonstrate the distribution of input arguments between the scripts KEYS and ARGV runtime global variables:

redis> EVAL "return { KEYS[1], KEYS[2], ARGV[1], ARGV[2], ARGV[3] }" 2 key1 key2 arg1 arg2 arg3
1) "key1"
2) "key2"
3) "arg1"
4) "arg2"
5) "arg3"

Note: as can been seen above, Lua’s table arrays are returned as RESP2 array replies, so it is likely that your client’s library will convert it to the native array data type in your programming language. Please refer to the rules that govern data type conversion for more pertinent information.

Interacting with Redis from a script

It is possible to call Redis commands from a Lua script either via redis.call() or redis.pcall().

The two are nearly identical. Both execute a Redis command along with its provided arguments, if these represent a well-formed command. However, the difference between the two functions lies in the manner in which runtime errors (such as syntax errors, for example) are handled. Errors raised from calling redis.call() function are returned directly to the client that had executed it. Conversely, errors encountered when calling the redis.pcall() function are returned to the script’s execution context instead for possible handling.

For example, consider the following:

> EVAL "return redis.call('SET', KEYS[1], ARGV[1])" 1 foo bar
OK

The above script accepts one key name and one value as its input arguments. When executed, the script calls the SET command to set the input key, foo, with the string value “bar”.

Script cache

Until this point, we’ve used the EVAL command to run our script.

Whenever we call EVAL, we also include the script’s source code with the request. Repeatedly calling EVAL to execute the same set of parameterized scripts, wastes both network bandwidth and also has some overheads in Redis. Naturally, saving on network and compute resources is key, so, instead, Redid provides a caching mechanism for scripts.

Every script you execute with EVAL is stored in a dedicated cache that the server keeps. The cache’s contents are organized by the scripts' SHA1 digest sums, so the SHA1 digest sum of a script uniquely identifies it in the cache. You can verify this behavior by running EVAL and calling INFO afterward. You’ll notice that the used_memory_scripts_eval and number_of_cached_scripts metrics grow with every new script that’s executed.

As mentioned above, dynamically-generated scripts are an anti-pattern. Generating scripts during the applicaiton’s runtime may, and probably will, exhaust the host’s memory resources for caching them. Instead, scripts should be as generic as possible and provide customized execution via their arguments.

A script is loaded to the server’s cache by calling the SCRIPT LOAD command and providing its source code. The server doesn’t executed the script, but instead just compiles and loads it to the server’s cache. Once loaded, you can execute the cached script with the SHA1 digest returned from the server.

Here’s an example of loading and then executing a cached script:

redis> SCRIPT LOAD "return 'Immabe a cached script'"
"c664a3bf70bd1d45c4284ffebb65a6f2299bfc9f"
redis> EVALSHA c664a3bf70bd1d45c4284ffebb65a6f2299bfc9f 0
"Immabe a cached script"

Cache volatility

The Redis script cache is always volatile. It isn’t considered as a part of the database and is not persisted. The cache may be cleared when the server restarts, during fail-over when a replica assumes the master role, or explicitly by SCRIPT FLUSH. That means that cached scripts are ephemeral, and the cache’s contents can be lost at any time.

Applications that use scripts should always call EVALSHA to execute them. The server returns an error if the script’s SHA1 digest is not in the cache. For example:

redis> EVALSHA ffffffffffffffffffffffffffffffffffffffff 0
(error) NOSCRIPT No matching script

In this case, the application should first load it with SCRIPT LOAD and then call EVALSHA once more to run the cached script by its SHA1 sum. Most of Redis' clients already provide utility APIs for doing that automatically. Please consult your client’s documentation regarding the specific details.

EVALSHA in the context of pipelining

Special care should be given executing EVALSHA in the context of a pipelined request. The commands in a pipelined request run in the order they are sent, but other clients' commands may be interleaved for execution between these. Because of that, the NOSCRIPT error can return from a pipelined request but can’t be handled.

Therefore, a client library’s implementation should revert to using plain EVAL of parameterized in the context of a pipeline.

Script cache semantics

During normal operation, an application’s scripts are meant to stay indefintely in the cache (that is, until the server is restarted or the cache being flushed). The underlying reasoning is that the script cache contents of a well-written application are unlikely to grow continuously. Even large applications that use hundereds of cached scripts shouldn’t be and issue in terms of cache memory usage.

The only way to flush the script cache is by explicitly calling the SCRIPT FLUSH command. Running the command will completely flush the scripts cache, removing all the scripts executed so far. Typically, this is only needed when the instance is going to be instantiated for another customer or application in a cloud environment.

Also, as already mentioned, restarting a Redis instance flushes the non-persistent script cache. However, from the point of view of the Redis client, there are only two ways to make sure that a Redis instance was not restarted between two different commands:

• The connection we have with the server is persistent and was never closed so far.
• The client explicitly checks the runid field in the INFO command to ensure the server was not restarted and is still the same process.

Practically speaking, it is much simpler for the client to assume that in the context of a given connection, cached scripts are guaranteed to be there unless the administrator explicitly invoked the SCRIPT FLUSH command. The fact that the user can count on Redis to retain cached scripts is semantically helpful in the context of pipelining.

The SCRIPT command

The Redis SCRIPT provides several ways for controlling the scripting subsystem. These are:

• SCRIPT FLUSH: this command is the only way to force Redis to flush the scripts cache. It is most useful in environments where the same Redis instance is reassigned to different uses. It is also helpful for testing client libraries' implementations of the scripting feature.

• SCRIPT EXISTS: given one or more SHA1 digests as arguments, this command returns an array of 1’s and 0’s. 1 means the specific SHA1 is recognized as a script already present in the scripting cache. 0’s meaning is that a script with this SHA1 wasn’t loaded before (or at least never since the latest call to SCRIPT FLUSH).

• SCRIPT LOAD script: this command registers the specified script in the Redis script cache. It is a useful command in all the contexts where we want to ensure that EVALSHA doesn’t not fail (for instance, in a pipeline or when called from a MULTI/EXEC transaction), without the need to execute the script.

• SCRIPT KILL: this command is the only way to interrupt a long-running script (a.k.a slow script), short of shutting down the server. A script is deemed as slow once its execution’s duration exceeds the configured maximum execution time threshold. The SCRIPT KILL command can be used only with scripts that did not modify the dataset during their execution (since stopping a read-only script does not violate the scripting engine’s guaranteed atomicity).

• SCRIPT DEBUG: controls use of the built-in Redis Lua scripts debugger.

Script replication

In standalone deployments, a single Redis instance called master manages the entire database. A clustered deployment has at least three masters managing the sharded database. Redis uses replication to maintain one or more replicas, or exact copies, for any given master.

Because scripts can modify the data, Redis ensures all write operations performed by a script are also sent to replicas to maintain consistency. There are two conceptual approaches when it comes to script replication:

1. Verbatim replication: the master sends the script’s source code to the replicas. Replicas then execute the script and apply the write effects. This mode can save on replication bandwidth in cases where short scripts generate many commands (for example, a for loop). However, this replication mode means that replicas redo the same work done by the master, which is wasteful. More importantly, it also requires all write scripts to be deterministic.
2. Effects replication: only the script’s data-modifying commands are replicated. Replicas then run the commands without executing any scripts. While potentially more lengthy in terms of network traffic, this replication mode is deterministic by definition and therefore doesn’t require special consideration.

Verbatim script replication was the only mode supported until Redis 3.2, in which effects replication was added. The lua-replicate-commands configuration directive and redis.replicate_commands() Lua API can be used to enable it.

In Redis 5.0, effects replication became the default mode. As of Redis 7.0, verbatim replication is no longer supported.

Replicating commands instead of scripts

Starting with Redis 3.2, it is possible to select an alternative replication method. Instead of replicating whole scripts, we can replicate the write commands generated by the script. We call this script effects replication.

Note: starting with Redis 5.0, script effects replication is the default mode and does not need to be explicitly enabled.

In this replication mode, while Lua scripts are executed, Redis collects all the commands executed by the Lua scripting engine that actually modify the dataset. When the script execution finishes, the sequence of commands that the script generated are wrapped into a MULTI/EXEC transaction and are sent to the replicas and AOF.

This is useful in several ways depending on the use case:

• When the script is slow to compute, but the effects can be summarized by a few write commands, it is a shame to re-compute the script on the replicas or when reloading the AOF. In this case, it is much better to replicate just the effects of the script.
• When script effects replication is enabled, the restrictions on non-deterministic functions are removed. You can, for example, use the TIME or SRANDMEMBER commands inside your scripts freely at any place.
• The Lua PRNG in this mode is seeded randomly on every call.

Unless already enabled by the server’s configuration or defaults (before Redis 7.0), you need to issue the following Lua command before the script performs a write:

redis.replicate_commands()

The redis.replicate_commands() function returns _true) if script effects replication was enabled; otherwise, if the function was called after the script already called a write command, it returns false, and normal whole script replication is used.

This function is deprecated as of Redis 7.0, and while you can still call it, it will always succeed.

Scripts with deterministic writes

Note: Starting with Redis 5.0, script replication is by default effect-based rather than verbatim. In Redis 7.0, verbatim script replication had been removed entirely. The following section only applies to versions lower than Redis 7.0 when not using effect-based script replication.

An important part of scripting is writing scripts that only change the database in a deterministic way. Scripts executed in a Redis instance are, by default until version 5.0, propagated to replicas and to the AOF file by sending the script itself – not the resulting commands. Since the script will be re-run on the remote host (or when reloading the AOF file), its changes to the database must be reproducible.

The reason for sending the script is that it is often much faster than sending the multiple commands that the script generates. If the client is sending many scripts to the master, converting the scripts into individual commands for the replica / AOF would result in too much bandwidth for the replication link or the Append Only File (and also too much CPU since dispatching a command received via the network is a lot more work for Redis compared to dispatching a command invoked by Lua scripts).

Normally replicating scripts instead of the effects of the scripts makes sense, however not in all the cases. So starting with Redis 3.2, the scripting engine is able to, alternatively, replicate the sequence of write commands resulting from the script execution, instead of replication the script itself.

In this section, we’ll assume that scripts are replicated verbatim by sending the whole script. Let’s call this replication mode verbatim scripts replication.

The main drawback with the whole scripts replication approach is that scripts are required to have the following property: the script always must execute the same Redis write commands with the same arguments given the same input data set. Operations performed by the script can’t depend on any hidden (non-explicit) information or state that may change as the script execution proceeds or between different executions of the script. Nor can it depend on any external input from I/O devices.

Acts such as using the system time, calling Redis commands that return random values (e.g., RANDOMKEY), or using Lua’s random number generator, could result in scripts that will not evaluate consistently.

To enforce the deterministic behavior of scripts, Redis does the following:

• Lua does not export commands to access the system time or other external states.
• Redis will block the script with an error if a script calls a Redis command able to alter the data set after a Redis random command like RANDOMKEY, SRANDMEMBER, TIME. That means that read-only scripts that don’t modify the dataset can call those commands. Note that a random command does not necessarily mean a command that uses random numbers: any non-deterministic command is considered as a random command (the best example in this regard is the TIME command).
• In Redis version 4.0, commands that may return elements in random order, such as SMEMBERS (because Redis Sets are unordered), exhibit a different behavior when called from Lua, and undergo a silent lexicographical sorting filter before returning data to Lua scripts. So redis.call("SMEMBERS",KEYS[1]) will always return the Set elements in the same order, while the same command invoked by normal clients may return different results even if the key contains exactly the same elements. However, starting with Redis 5.0, this ordering is no longer performed because replicating effects circumvents this type of non-determinism. In general, even when developing for Redis 4.0, never assume that certain commands in Lua will be ordered, but instead rely on the documentation of the original command you call to see the properties it provides.
• Lua’s pseudo-random number generation function math.random is modified and always uses the same seed for every execution. This means that calling math.random will always generate the same sequence of numbers every time a script is executed (unless math.randomseed is used).

All that said, you can still use commands that write and random behavior with a simple trick. Imagine that you want to write a Redis script that will populate a list with N random integers.

The initial implementation in Ruby could look like this:

require 'rubygems'
require 'redis'

r = Redis.new

RandomPushScript = <<EOF
    local i = tonumber(ARGV[1])
    local res
    while (i > 0) do
        res = redis.call('LPUSH',KEYS[1],math.random())
        i = i-1
    end
    return res
EOF

r.del(:mylist)
puts r.eval(RandomPushScript,[:mylist],[10,rand(2**32)])

Every time this code runs, the resulting list will have exactly the following elements:

redis> LRANGE mylist 0 -1
 1) "0.74509509873814"
 2) "0.87390407681181"
 3) "0.36876626981831"
 4) "0.6921941534114"
 5) "0.7857992587545"
 6) "0.57730350670279"
 7) "0.87046522734243"
 8) "0.09637165539729"
 9) "0.74990198051087"
10) "0.17082803611217"

To make the script both deterministic and still have it produce different random elements, we can add an extra argument to the script that’s the seed to Lua’s pseudo-random number generator. The new script is as follows:

RandomPushScript = <<EOF
    local i = tonumber(ARGV[1])
    local res
    math.randomseed(tonumber(ARGV[2]))
    while (i > 0) do
        res = redis.call('LPUSH',KEYS[1],math.random())
        i = i-1
    end
    return res
EOF

r.del(:mylist)
puts r.eval(RandomPushScript,1,:mylist,10,rand(2**32))

What we are doing here is sending the seed of the PRNG as one of the arguments. The script output will always be the same given the same arguments (our requirement) but we are changing one of the arguments at every invocation, generating the random seed client-side. The seed will be propagated as one of the arguments both in the replication link and in the Append Only File, guaranteeing that the same changes will be generated when the AOF is reloaded or when the replica processes the script.

Note: an important part of this behavior is that the PRNG that Redis implements as math.random and math.randomseed is guaranteed to have the same output regardless of the architecture of the system running Redis. 32-bit, 64-bit, big-endian and little-endian systems will all produce the same output.

Debugging Eval scripts

Starting with Redis 3.2, Redis has support for native Lua debugging. The Redis Lua debugger is a remote debugger consisting of a server, which is Redis itself, and a client, which is by default redis-cli.

The Lua debugger is described in the Lua scripts debugging section of the Redis documentation.

Execution under low memory conditions

When memory usage in Redis exceeds the maxmemory limit, the first write command encountered in the script that uses additional memory will cause the script to abort (unless redis.pcall was used).

However, an exception to the above is when the script’s first write command does not use additional memory, as is the case with (for example, DEL and LREM). In this case, Redis will allow all commands in the script to run to ensure atomicity. If subsequent writes in the script consume additional memory, Redis' memory usage can exceed the threshold set by the maxmemory configuration directive.

Another scenario in which a script can cause memory usage to cross the maxmemory threshold is when the execution begins when Redis is slightly below maxmemory, so the script’s first write command is allowed. As the script executes, subsequent write commands consume more memory leading to the server using more RAM than the configured maxmemory directive.

In those scenarios, you should consider setting the maxmemory-policy configuration directive to any values other than noeviction. In addition, Lua scripts should be as fast as possible so that eviction can kick in between executions.

Note that you can change this behaviour by using flags

Eval flags

Normally, when you run an Eval script, the server does not know how it accesses the database. By default, Redis assumes that all scripts read and write data. However, starting with Redis 7.0, there’s a way to declare flags when creating a script in order to tell Redis how it should behave.

The way to do that us using a Shebang statement on the first line of the script like so:

#!lua flags=no-writes,allow-stale
local x = redis.call('get','x')
return x

Note that as soon as Redis sees the #! comment, it’ll treat the script as if it declares flags, even if no flags are defined, it still has a different set of defaults compared to a script without a #! line.

Please refer to Script flags to learn about the various scripts and the defaults.

3 - Redis Lua API reference

Redis includes an embedded Lua 5.1 interpreter. The interpreter runs user-defined ephemeral scripts and [/topics/functions-intro]. Scripts run in a sandboxed context and can only access specific Lua packages. This page describes the packages and APIs available inside the execution’s context.

Sandbox context

The sandboxed Lua context attempts to prevent accidental misuse and reduce potential threats from the server’s environment.

Scripts should never try to access the Redis server’s underlying host systems. That includes the file system, network, and any other attempt to perform a system call other than those supported by the API.

Scripts should operate solely on data stored in Redis and data provided as arguments to their execution.

Global variables and functions

The sandboxed Lua execution context blocks the declaration of global variables and functions. The blocking of global variables is in place to ensure that scripts and functions don’t attempt to maintain any runtime context other than the data stored in Redis. In the (somewhat uncommon) use case that a context needs to be maintain betweem executions, you should store the context in Redis' keyspace.

Redis will return a “Script attempted to create global variable ‘my_global_variable” error when trying to execute the following snippet:

my_global_variable = 'some value'

And similarly for the following global function declaration:

function my_global_funcion()
  -- Do something amazing
end

You’ll also get a similar error when your script attempts to access any global variables that are undefined in the runtime’s context:

-- The following will surely raise an error
return an_undefined_global_variable

Instead, all variable and function definitions are required to be declared as local. To do so, you’ll need to prepend the local keyword to your declarations. For example, the following snippet will be considered perfectly valid by Redis:

local my_local_variable = 'some value'

local function my_local_function()
  -- Do something else, but equally amazing
end

Note: the sandbox attempts to prevent the use of globals. Using Lua’s debugging functionality or other approaches such as altering the meta table used for implementing the globals’ protection to circumvent the sandbox isn’t hard. However, it is difficult to circumvent the protection by accident. If the user messes with the Lua global state, the consistency of AOF and replication can’t be guaranteed. In other words, just don’t do it.

Imported Lua modules

Using imported Lua modules is not supported inside the sandboxed execution context. The sandboxed execution context prevents the loading modules by disabling Lua’s require function.

The only libraries that Redis ships with and that you can use in scripts are listed under the Runtime libraries section.

Runtime globals

While the sandbox prevents users from declaring globals, the execution context is pre-populated with several of these.

The redis singleton

The redis singleton is an object instance that’s accessible from all scripts. It provides the API to interact with Redis from scripts. Its description follows below.

The KEYS global variable

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: no

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a function accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

The KEYS global variable is available only for ephemeral scripts. It is pre-populated with all key name input arguments.

The ARGV global variable

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: no

The ARGV global variable is available only in ephemeral scripts. It is pre-populated with all regular input arguments.

redis object

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

The Redis Lua execution context always provides a singleton instance of an object named redis. The redis instance enables the script to interact with the Redis server that’s running it. Following is the API provided by the redis object instance.

redis.call(command [,arg...])

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

The redis.call() function calls a given Redis command and returns its reply. Its inputs are the command and arguments, and once called, it executes the command in Redis and returns the reply.

For example, we can call the ECHO command from a script and return its reply like so:

return redis.call('ECHO', 'Echo, echo... eco... o...')

If and when redis.call() triggers a runtime exception, the raw exception is raised back to the user as an error, automatically. Therefore, attempting to execute the following ephemeral script will fail and generate a runtime exception because ECHO accepts exactly zero or one argument:

redis> EVAL "return redis.call('ECHO', 'Echo,', 'echo... ', 'eco... ', 'o...')" 0
(error) ERR Error running script (call to b0345693f4b77517a711221050e76d24ae60b7f7): @user_script:1: @user_script: 1: Wrong number of args calling Redis command from script

Note that the call can fail due to various reasons, see Execution under low memory conditions and Script flags

To handle Redis runtime errors use `redis.pcall() instead.

redis.pcall(command [,arg...])

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

This function enables handling runtime errors raised by the Redis server. The redis.pcall() function behaves exactly like redis.call(), except that it:

• Always returns a reply.
• Never throws a runtime exeption, and returns in its stead a redis.error_reply in case that a runtime exception is thrown by the server.

The following demonstrates how to use redis.pcall() to intercept and handle runtime exceptions from within the context of an ephemeral script.

local reply = redis.pcall('ECHO', unpack(ARGV))
if reply['err'] ~= nil then
  -- Handle the error sometime, but for now just log it
  redis.log(redis.LOG_WARNING, reply['err'])
  reply['err'] = 'Something is wrong, but no worries, everything is under control'
end
return reply

Evaluating this script with more than one argument will return:

redis> EVAL "..." 0 hello world
(error) Something is wrong, but no worries, everything is under control

redis.error_reply(x)

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

This is a helper function that returns an error reply. The helper accepts a single string argument and returns a Lua table with the err field set to that string.

The outcome of the following code is that error1 and error2 are identical for all intents and purposes:

local text = 'My very special error'
local reply1 = { err = text }
local reply2 = redis.error_reply(text)

Therefore, both forms are valid as means for returning an error reply from scripts:

redis> EVAL "return { err = 'My very special table error' }" 0
(error) My very special table error
redis> EVAL "return redis.error_reply('My very special reply error')" 0
(error) My very special reply error

For returing Redis status replies refer to redis.status_reply(). Refer to the Data type conversion for returning other response types.

redis.status_reply(x)

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

This is a helper function that returns a simple string reply. “OK” is an example of a standard Redis status reply. The Lua API represents status replies as tables with a single field, ok, set with a simple status string.

The outcome of the following code is that status1 and status2 are identical for all intents and purposes:

local text = 'Frosty'
local status1 = { ok = text }
local status2 = redis.status_reply(text)

Therefore, both forms are valid as means for returning status replies from scripts:

redis> EVAL "return { ok = 'TICK' }" 0
TICK
redis> EVAL "return redis.status_reply('TOCK')" 0
TOCK

For returing Redis error replies refer to redis.error_reply(). Refer to the Data type conversion for returning other response types.

redis.sha1hex(x)

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

This function returns the SHA1 hexadecimal digest of its single string argument.

You can, for example, obtain the empty string’s SHA1 digest:

redis> EVAL "return redis.sha1hex('')" 0
"da39a3ee5e6b4b0d3255bfef95601890afd80709"

redis.log(level, message)

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

This function writes to the Redis server log.

It expects two input arguments: the log level and a message. The message is a string to write to the log file. Log level can be on of these:

• redis.LOG_DEBUG
• redis.LOG_VERBOSE
• redis.LOG_NOTICE
• redis.LOG_WARNING

These levels map to the server’s log levels. The log only records messages equal or greater in level than the server’s loglevel configuration directive.

The following snippet:

redis.log(redis.LOG_WARNING, 'Something is terribly wrong')

will produce a line similar to the following in your server’s log:

[32343] 22 Mar 15:21:39 # Something is terribly wrong

redis.setresp(x)

• Since version: 6.0.0
• Available in scripts: yes
• Available in functions: yes

This function allows the executing script to switch between Redis Serialization Protocol (RESP) versions for the replies returned by redis.call()](#redis.call) and [redis.pall(). It expects a single numerical argument as the protocol’s version. The default protocol version is 2, but it can be switched to version 3.

Here’s an example of switching to RESP3 replies:

redis.setresp(3)

Please refer to the Data type conversion for more information about type conversions.

redis.set_repl(x)

• Since version: 3.2.0
• Available in scripts: yes
• Available in functions: no

Note: this feature is only available when script effects replication is employed. Calling it when using verbatim script replication will result in an error. As of Redis version 2.6.0, scripts were replicated verbatim, meaning that the scripts' source code was sent for execution by replicas and stored in the AOF. An alternative replication mode added in version 3.2.0 allows replicating only the scripts' effects. As of Redis version 7.0, script replication is no longer supported, and the only replication mode available is script effects replication.

Warning: this is an advanced feature. Misuse can cause damage by violating the contract that binds the Redis master, its replicas, and AOF contents to hold the same logical content.

This function allows a script to assert control over how its effects are propagated to replicas and the AOF afterward. A script’s effects are the Redis write commands that it calls.

By default, all write commands that a script executes are replicated. Sometimes, however, better control over this behavior can be helpful. This can be the case, for example, when storing intermediate values in the master alone.

Consider a script that intersects two sets and stores the result in a temporary key with SUNIONSTORE. It then picks five random elements (SRANDMEMBER) from the intersection and stores (SADD) them in another set. Finally, before returning, it deletes the temporary key that stores the intersection of the two source sets.

In this case, only the new set with its five randomly-chosen elements needs to be replicated. Replicating the SUNIONSTORE command and the `DEL’ition of the temporary key is unnecessary and wasteful.

The redis.set_repl() function instructs the server how to treat subsequent write commands in terms of replication. It accepts a single input argument that only be one of the following:

• redis.REPL_ALL: replicates the effects to the AOF and replicas.
• redis.REPL_AOF: replicates the effects to the AOF alone.
• redis.REPL_REPLICA: replicates the effects to the replicas alone.
• redis.REPL_SLAVE: same as REPL_REPLICA, maintained for backward compatibility.
• redis.REPL_NONE: disables effect replication entirely.

By default, the scripting engine is initialized to the redis.REPL_ALL setting when a script begins its execution. You can call the redis.set_repl() function at any time during the script’s execution to switch between the different replication modes.

A simple example follows:

redis.replicate_commands() -- Enable effects replication in versions lower than Redis v7.0
redis.call('SET', KEYS[1], ARGV[1])
redis.set_repl(redis.REPL_NONE)
redis.call('SET', KEYS[2], ARGV[2])
redis.set_repl(redis.REPL_ALL)
redis.call('SET', KEYS[3], ARGV[3])

If you run this script by calling EVAL "..." 3 A B C 1 2 3, the result will be that only the keys A and C are created on the replicas and AOF.

redis.replicate_commands()

• Since version: 3.2.0
• Until version: 7.0.0
• Available in scripts: yes
• Available in functions: no

This function switches the script’s replication mode from verbatim replication to effects replication. You can use it to override the default verbatim script replication mode used by Redis until version 7.0.

Note: as of Redis v7.0, verbatim script replication is no longer supported. The default, and only script replication mode supported, is script effects' replication. For more information, please refer to Replicating commands instead of scripts

redis.breakpoint()

• Since version: 3.2.0
• Available in scripts: yes
• Available in functions: no

This function triggers a breakpoint when using the Redis Lua debugger](/topics/ldb).

redis.debug(x)

• Since version: 3.2.0
• Available in scripts: yes
• Available in functions: no

This function prints its argument in the Redis Lua debugger console.

redis.acl_check_cmd(command [,arg...])

• Since version: 7.0.0
• Available in scripts: yes
• Available in functions: yes

This function is used for checking if the current user running the script has ACL permissions to execute the given command with the given arguments.

The return value is a boolean true in case the current user has permissions to execute the command (via a call to redis.call or redis.pcall) or false in case they don’t.

The function will raise an error if the passed command or its arguments are invalid.

redis.register_function

• Since version: 7.0.0
• Available in scripts: no
• Available in functions: yes

This function is only available from the context of the FUNCTION LOAD command. When called, it registers a function to the loaded library. The function can be called either with positional or named arguments.

positional arguments: redis.register_function(name, callback)

The first argument to redis.register_function is a Lua string representing the function name. The second argument to redis.register_function is a Lua function.

Usage example:

redis> FUNCTION LOAD Lua mylib "redis.register_function('noop', function() end)"

Named arguments: redis.register_function{function_name=name, callback=callback, flags={flag1, flag2, ..}, description=description}

The named arguments variant accepts the following arguments:

• function_name: the function’s name.
• callback: the function’s callback.
• flags: an array of strings, each a function flag (optional).
• description: function’s description (optional).

Both function_name and callback are mandatory.

Usage example:

redis> FUNCTION LOAD Lua mylib "redis.register_function{function_name='noop', callback=function() end, flags={ 'no-writes' }, description='Does nothing'}"

Script flags

Important: Use script flags with care, which may negatively impact if misused. Note that the default for Eval scripts are different than the default for functions that are mentioned below, see Eval Flags

When you register a function or load an Eval script, the server does not know how it accesses the database. By default, Redis assumes that all scripts read and write data. This results in the following behavior:

1. They can read and write data.
2. They can run in cluster mode.
3. Execution against a stale replica is denied to avoid inconsistent reads.
4. Execution under low memory is denied to avoid exceeding the configured threshold.

You can use the following flags and instruct the server to treat the scripts' execution differently:

• no-writes: this flag indicates that the script only reads data but never writes.

  By default, Redis will deny the execution of scripts against read-only replicas, as they may attempt to perform writes. Similarly, the server will not allow calling scripts with FCALL_RO / EVAL_RO. Lastly, when data persistence is at risk due to a disk error, execution is blocked as well.

  Using this flag allows executing the script:

  1. With FCALL_RO / EVAL_RO against masters and read-only replicas.
  2. Even if there’s a disk error (Redis is unable to persist so it rejects writes).

  However, note that the server will return an error if the script attempts to call a write command.

• allow-oom: use this flag to allow a script to execute when the server is out of memory (OOM).

  Unless used, Redis will deny the execution of scripts when in an OOM state, regardless of the no-write flag and method of calling. Furthermore, when you use this flag, the script can call any Redis command, including commands that aren’t usually allowed in this state.

• allow-stale: a flag that enables running the script against a stale replica.

  By default, Redis prevents data consistency problems from using old data by having stale replicas return a runtime error. In cases where the consistency is a lesser concern, this flag allows stale Redis replicas to run the script.

• no-cluster: the flag causes the script to return an error in Redis cluster mode.

  Redis allows scripts to be executed both in standalone and cluster modes. Setting this flag prevents executing the script against nodes in the cluster.

Please refer to Function Flags and Eval Flags for a detailed example.

redis.REDIS_VERSION

• Since version: 7.0.0
• Available in scripts: yes
• Available in functions: yes

Returns the current Redis server version as a Lua string. The reply’s format is MM.mm.PP, where:

• MM: is the major version.
• mm: is the minor version.
• PP: is the patch level.

redis.REDIS_VERSION_NUM

• Since version: 7.0.0
• Available in scripts: yes
• Available in functions: yes

Returns the current Redis server version as a number. The reply is a hexadecimal value structured as 0x00MMmmPP, where:

• MM: is the major version.
• mm: is the minor version.
• PP: is the patch level.

Data type conversion

Unless a runtime exception is raised, redis.call() and redis.pcall() return the reply from the executed command to the Lua script. Redis' replies from these functions are converted automatically into Lua’s native data types.

Similarly, when a Lua script returns a reply with the return keyword, that reply is automatically converted to Redis' protocol.

Put differently; there’s a one-to-one mapping between Redis' replies and Lua’s data types and a one-to-one mapping between Lua’s data types and the Redis Protocol data types. The underlying design is such that if a Redis type is converted into a Lua type and converted back into a Redis type, the result is the same as the initial value.

Type conversion from Redis protocol replies (i.e., the replies from redis.call() and redis.pcall() to Lua data types depends on the Redis Serialization Protocol version used by the script. The default protocol version during script executions is RESP2. The script may switch the replies' protocol versions by calling the redis.setresp() function.

Type conversion from a script’s returned Lua data type depends on the user’s choice of protocol (see the HELLO command).

The following sections describe the type conversion rules between Lua and Redis per the protocol’s version.

RESP2 to Lua type conversion

The following type conversion rules apply to the execution’s context by default as well as after calling redis.setresp(2):

• RESP2 integer reply -> Lua number
• RESP2 bulk string reply -> Lua string
• RESP2 array reply -> Lua table (may have other Redis data types nested)
• RESP2 status reply -> Lua table with a single ok field containing the status string
• RESP2 error reply -> Lua table with a single err field containing the error string
• RESP2 null bulk reply and null multi bulk reply -> Lua false boolean type

Lua to RESP2 type conversion

The following type conversion rules apply by default as well as after the user had called HELLO 2:

• Lua number -> RESP2 integer reply (the number is converted into an integer)
• Lua string -> RESP bulk string reply
• Lua table (indexed, non-associative array) -> RESP2 array reply (truncated at the first Lua nil value encountered in the table, if any)
• Lua table with a single ok field -> RESP2 status reply
• Lua table with a single err field -> RESP2 error reply
• Lua boolean false -> RESP2 null bulk reply

There is an additional Lua-to-Redis conversion rule that has no corresponding Redis-to-Lua conversion rule:

• Lua Boolean true -> RESP2 integer reply with value of 1.

There are three additional rules to note about converting Lua to Redis data types:

• Lua has a single numerical type, Lua numbers. There is no distinction between integers and floats. So we always convert Lua numbers into integer replies, removing the decimal part of the number, if any. If you want to return a Lua float, it should be returned as a string, exactly like Redis itself does (see, for instance, the ZSCORE command).
• There’s no simple way to have nils inside Lua arrays due to Lua’s table semantics. Therefore, who;e Redis converts a Lua array to RESP, the conversion stops when it encounters a Lua nil value.
• When a Lua table is an associative array that contains keys and their respective values, the converted Redis reply will not include them.

Lua to RESP2 type conversion examples:

redis> EVAL "return 10" 0
(integer) 10

redis> EVAL "return { 1, 2, { 3, 'Hello World!' } }" 0
1) (integer) 1s
2) (integer) 2
3) 1) (integer) 3
   1) "Hello World!"

redis> EVAL "return redis.call('get','foo')" 0
"bar"

The last example demonstrates receiving and returning the exact return value of redis.call() (or redis.pcall()) in Lua as it would be returned if the command had been called directly.

The following example shows how floats and arrays that cont nils and keys are handled:

redis> EVAL "return { 1, 2, 3.3333, somekey = 'somevalue', 'foo', nil , 'bar' }" 0
1) (integer) 1
2) (integer) 2
3) (integer) 3
4) "foo"

As you can see, the float value of 3.333 gets converted to an integer 3, the somekey key and its value are omitted, and the string “bar” isn’t returned as there is a nil value that precedes it.

RESP3 to Lua type conversion

RESP3 is a newer version of the Redis Serialization Protocol. It is available as an opt-in choice as of Redis v6.0.

An executing script may call the redis.setresp function during its execution and switch the protocol version that’s used for returning replies from Redis' commands (that can be invoked via redis.call() or redis.pcall()).

Once Redis' replies are in RESP3 protocol, all of the RESP2 to Lua conversion rules apply, with the following additions:

• RESP3 map reply -> Lua table with a single map field containing a Lua table representing the fields and values of the map.
• RESP set reply -> Lua table with a single set field containing a Lua table representing the elements of the set as fields, each with the Lua Boolean value of true.
• RESP3 null -> Lua nil.
• RESP3 true reply -> Lua true boolean value.
• RESP3 false reply -> Lua false boolean value.
• RESP3 double reply -> Lua table with a single score field containing a Lua number representing the double value.
• RESP3 big number reply -> Lua table with a single big_number field containing a Lua string representing the big number value.
• Redis verbatim string reply -> Lua table with a single verbatim_string field containing a Lua table with two fields, string and format, representing the verbatim string and its format, respectively.

Note: the RESP3 big number and verbatim strings replies are only supported as of Redis v7.0 and greater. Also, presently, RESP3’s attributes, streamed strings and streamed aggregate data types are not supported by the Redis Lua API.

Lua to RESP3 type conversion

Regardless of the script’s choice of protocol version set for replies with the [redis.setresp() function] when it calls redis.call() or redis.pcall(), the user may opt-in to using RESP3 (with the HELLO 3 command) for the connection. Although the default protocol for incoming client connections is RESP2, the script should honor the user’s preference and return adequately-typed RESP3 replies, so the following rules apply on top of those specified in the Lua to RESP2 type conversion section when that is the case.

• Lua Boolean -> RESP3 Boolean reply (note that this is a change compared to the RESP2, in which returning a Boolean Lua true returned the number 1 to the Redis client, and returning a false used to return a null.
• Lua table with a single map field set to an associative Lua table -> RESP3 map reply.
• Lua table with a single _set field set to an associative Lua table -> RESP3 set reply. Values can be set to anything and are discarded anyway.
• Lua table with a single double field to an associative Lua table -> RESP3 double reply.
• Lua nil -> RESP3 null.

However, if the connection is set use the RESP2 protocol, and even if the script replies with RESP3-typed responses, Redis will automatically perform a RESP3 to RESP2 convertion of the reply as is the case for regular commands. That means, for example, that returning the RESP3 map type to a RESP2 connection will result in the repy being converted to a flat RESP2 array that consists of alternating field names and their values, rather than a RESP3 map.

Additional notes about scripting

Using SELECT inside scripts

You can call the SELECT command from your Lua scripts, like you can with any normal client connection. However, one subtle aspect of the behavior changed between Redis versions 2.8.11 and 2.8.12. Prior to Redis version 2.8.12, the database selected by the Lua script was set as the current database for the client connection that had called it. As of Redis version 2.8.12, the database selected by the Lua script only affects the execution context of the script, and does not modify the database that’s selected by the client calling the script. This semantic change between patch level releases was required since the old behavior was inherently incompatible with Redis' replication and introduced bugs.

Runtime libraries

The Redis Lua runtime context always comes with several pre-imported libraries.

The following standard Lua libraries are available to use:

• The String Manipulation (string) library
• The Table Manipulation (table) library
• The Mathematical Functions (math) library

In addition, the following external libraries are loaded and accessible to scripts:

• The struct library
• The cjson library
• The cmsgpack library
• The bitop library

struct library

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

struct is a library for packing and unpacking C-like structures in Lua. It provides the following functions:

• struct.pack()
• struct.unpack()
• struct.size()

All of struct’s functions expect their first argument to be a format string.

struct formats

The following are valid format strings for struct’s functions:

• >: big endian
• <: little endian
• ![num]: alignment
• x: padding
• b/B: signed/unsigned byte
• h/H: signed/unsigned short
• l/L: signed/unsigned long
• T: size_t
• i/In: signed/unsigned integer with size n (defaults to the size of int)
• cn: sequence of n chars (from/to a string); when packing, n == 0 means the whole string; when unpacking, n == 0 means use the previously read number as the string’s length.
• s: zero-terminated string
• f: float
• d: double
• (space): ignored

struct.pack(x)

This function returns a struct-encoded string from values. It accepts a struct format string as its first argument, followed by the values that are to be encoded.

Usage example:

redis> EVAL "return struct.pack('HH', 1, 2)" 0
"\x01\x00\x02\x00"

struct.unpack(x)

This function returns the decoded values from a struct. It accepts a struct format string as its first argument, followed by encoded struct’s string.

Usage example:

redis> EVAL "return { struct.unpack('HH', ARGV[1]) }" 0 "\x01\x00\x02\x00"
1) (integer) 1
2) (integer) 2
3) (integer) 5

struct.size(x)

This function returns the size, in bytes, of a struct. It accepts a struct format string as its only argument.

Usage example:

redis> EVAL "return struct.size('HH')" 0
(integer) 4

cjson library

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

The cjson library provides fast JSON encoding and decoding from Lua. It provides these functions.

cjson.encode(x)

This function returns a JSON-encoded string for the Lua data type provided as its argument.

Usage example:

redis> EVAL "return cjson.encode({ ['foo'] = 'bar' })" 0
"{\"foo\":\"bar\"}"

cjson.decode(x)

This function returns a Lua data type from the JSON-encoded string provided as its argument.

Usage example:

redis> EVAL "return cjson.decode(ARGV[1])['foo']" 0 '{"foo":"bar"}'
"bar"

cmsgpack library

• Since version: 2.6.0
• Available in scripts: yes
• Available in functions: yes

The cmsgpack library provides fast MessagePack encoding and decoding from Lua. It provides these functions.

cmsgpack.pack(x)

This function returns the packed string encoding of the Lua data type it is given as an argument.

Usage example:

redis> EVAL "return cmsgpack.pack({'foo', 'bar', 'baz'})" 0
"\x93\xa3foo\xa3bar\xa3baz"

cmsgpack.unpack(x)

This function returns the unpacked values from decocoding its input string argument.

Usage example:

redis> EVAL "return cmsgpack.unpack(ARGV[1])" 0 "\x93\xa3foo\xa3bar\xa3baz"
1) "foo"
2) "bar"
3) "baz"

bit library

• Since version: 2.8.18
• Available in scripts: yes
• Available in functions: yes

The bit library provides bitwise operations on numbers. Its documentation resides at Lua BitOp documentation It provides the following functions.

bit.tobit(x)

Normalizes a number to the numeric range for bit operations and returns it.

Usage example:

redis> EVAL 'return bit.tobit(1)' 0
(integer) 1

bit.tohex(x [,n])

Converts its first argument to a hex string. The number of hex digits is given by the absolute value of the optional second argument.

Usage example:

redis> EVAL 'return bit.tohex(422342)' 0
"000671c6"

bit.bnot(x)

Returns the bitwise not of its argument.

bit.bnot(x) bit.bor(x1 [,x2...]), bit.band(x1 [,x2...]) and bit.bxor(x1 [,x2...])

Returns either the bitwise or, bitwise and, or bitwise xor of all of its arguments. Note that more than two arguments are allowed.

Usage example:

redis> EVAL 'return bit.bor(1,2,4,8,16,32,64,128)' 0
(integer) 255

bit.lshift(x, n), bit.rshift(x, n) and bit.arshift(x, n)

Returns either the bitwise logical left-shift, bitwise logical right-shift, or bitwise arithmetic right-shift of its first argument by the number of bits given by the second argument.

bit.rol(x, n) and bit.ror(x, n)

Returns either the bitwise left rotation, or bitwise right rotation of its first argument by the number of bits given by the second argument. Bits shifted out on one side are shifted back in on the other side.

bit.bswap(x)

Swaps the bytes of its argument and returns it. This can be used to convert little-endian 32-bit numbers to big-endian 32-bit numbers and vice versa.

4 - Debugging Lua scripts in Redis

Starting with version 3.2 Redis includes a complete Lua debugger, that can be used in order to make the task of writing complex Redis scripts much simpler.

The Redis Lua debugger, codenamed LDB, has the following important features:

• It uses a server-client model, so it’s a remote debugger. The Redis server acts as the debugging server, while the default client is redis-cli. However other clients can be developed by following the simple protocol implemented by the server.
• By default every new debugging session is a forked session. It means that while the Redis Lua script is being debugged, the server does not block and is usable for development or in order to execute multiple debugging sessions in parallel. This also means that changes are rolled back after the script debugging session finished, so that’s possible to restart a new debugging session again, using exactly the same Redis data set as the previous debugging session.
• An alternative synchronous (non forked) debugging model is available on demand, so that changes to the dataset can be retained. In this mode the server blocks for the time the debugging session is active.
• Support for step by step execution.
• Support for static and dynamic breakpoints.
• Support from logging the debugged script into the debugger console.
• Inspection of Lua variables.
• Tracing of Redis commands executed by the script.
• Pretty printing of Redis and Lua values.
• Infinite loops and long execution detection, which simulates a breakpoint.

Quick start

A simple way to get started with the Lua debugger is to watch this video introduction:

Important Note: please make sure to avoid debugging Lua scripts using your Redis production server. Use a development server instead. Also note that using the synchronous debugging mode (which is NOT the default) results in the Redis server blocking for all the time the debugging session lasts.

To start a new debugging session using redis-cli do the following:

1. Create your script in some file with your preferred editor. Let’s assume you are editing your Redis Lua script located at /tmp/script.lua.

2. Start a debugging session with:

   ./redis-cli –ldb –eval /tmp/script.lua

Note that with the --eval option of redis-cli you can pass key names and arguments to the script, separated by a comma, like in the following example:

./redis-cli --ldb --eval /tmp/script.lua mykey somekey , arg1 arg2

You’ll enter a special mode where redis-cli no longer accepts its normal commands, but instead prints a help screen and passes the unmodified debugging commands directly to Redis.

The only commands which are not passed to the Redis debugger are:

• quit – this will terminate the debugging session. It’s like removing all the breakpoints and using the continue debugging command. Moreover the command will exit from redis-cli.
• restart – the debugging session will restart from scratch, reloading the new version of the script from the file. So a normal debugging cycle involves modifying the script after some debugging, and calling restart in order to start debugging again with the new script changes.
• help – this command is passed to the Redis Lua debugger, that will print a list of commands like the following:

lua debugger> help
Redis Lua debugger help:
[h]elp               Show this help.
[s]tep               Run current line and stop again.
[n]ext               Alias for step.
[c]continue          Run till next breakpoint.
[l]list              List source code around current line.
[l]list [line]       List source code around [line].
                     line = 0 means: current position.
[l]list [line] [ctx] In this form [ctx] specifies how many lines
                     to show before/after [line].
[w]hole              List all source code. Alias for 'list 1 1000000'.
[p]rint              Show all the local variables.
[p]rint <var>        Show the value of the specified variable.
                     Can also show global vars KEYS and ARGV.
[b]reak              Show all breakpoints.
[b]reak <line>       Add a breakpoint to the specified line.
[b]reak -<line>      Remove breakpoint from the specified line.
[b]reak 0            Remove all breakpoints.
[t]race              Show a backtrace.
[e]eval <code>       Execute some Lua code (in a different callframe).
[r]edis <cmd>        Execute a Redis command.
[m]axlen [len]       Trim logged Redis replies and Lua var dumps to len.
                     Specifying zero as <len> means unlimited.
[a]abort             Stop the execution of the script. In sync
                     mode dataset changes will be retained.

Debugger functions you can call from Lua scripts:
redis.debug()        Produce logs in the debugger console.
redis.breakpoint()   Stop execution as if there was a breakpoint in the
                     next line of code.

Note that when you start the debugger it will start in stepping mode. It will stop at the first line of the script that actually does something before executing it.

From this point you usually call step in order to execute the line and go to the next line. While you step Redis will show all the commands executed by the server like in the following example:

* Stopped at 1, stop reason = step over
-> 1   redis.call('ping')
lua debugger> step
<redis> ping
<reply> "+PONG"
* Stopped at 2, stop reason = step over

The <redis> and <reply> lines show the command executed by the line just executed, and the reply from the server. Note that this happens only in stepping mode. If you use continue in order to execute the script till the next breakpoint, commands will not be dumped on the screen to prevent too much output.

Termination of the debugging session

When the scripts terminates naturally, the debugging session ends and redis-cli returns in its normal non-debugging mode. You can restart the session using the restart command as usual.

Another way to stop a debugging session is just interrupting redis-cli manually by pressing Ctrl+C. Note that also any event breaking the connection between redis-cli and the redis-server will interrupt the debugging session.

All the forked debugging sessions are terminated when the server is shut down.

Abbreviating debugging commands

Debugging can be a very repetitive task. For this reason every Redis debugger command starts with a different character, and you can use the single initial character in order to refer to the command.

So for example instead of typing step you can just type s.

Breakpoints

Adding and removing breakpoints is trivial as described in the online help. Just use b 1 2 3 4 to add a breakpoint in line 1, 2, 3, 4. The command b 0 removes all the breakpoints. Selected breakpoints can be removed using as argument the line where the breakpoint we want to remove is, but prefixed by a minus sign. So for example b -3 removes the breakpoint from line 3.

Note that adding breakpoints to lines that Lua never executes, like declaration of local variables or comments, will not work. The breakpoint will be added but since this part of the script will never be executed, the program will never stop.

Dynamic breakpoints

Using the breakpoint command it is possible to add breakpoints into specific lines. However sometimes we want to stop the execution of the program only when something special happens. In order to do so, you can use the redis.breakpoint() function inside your Lua script. When called it simulates a breakpoint in the next line that will be executed.

if counter > 10 then redis.breakpoint() end

This feature is extremely useful when debugging, so that we can avoid continuing the script execution manually multiple times until a given condition is encountered.

Synchronous mode

As explained previously, but default LDB uses forked sessions with rollback of all the data changes operated by the script while it has being debugged. Determinism is usually a good thing to have during debugging, so that successive debugging sessions can be started without having to reset the database content to its original state.

However for tracking certain bugs, you may want to retain the changes performed to the key space by each debugging session. When this is a good idea you should start the debugger using a special option, ldb-sync-mode, in redis-cli.

./redis-cli --ldb-sync-mode --eval /tmp/script.lua

Note: Redis server will be unreachable during the debugging session in this mode, so use with care.

In this special mode, the abort command can stop the script half-way taking the changes operated to the dataset. Note that this is different compared to ending the debugging session normally. If you just interrupt redis-cli the script will be fully executed and then the session terminated. Instead with abort you can interrupt the script execution in the middle and start a new debugging session if needed.

Logging from scripts

The redis.debug() command is a powerful debugging facility that can be called inside the Redis Lua script in order to log things into the debug console:

lua debugger> list
-> 1   local a = {1,2,3}
   2   local b = false
   3   redis.debug(a,b)
lua debugger> continue
<debug> line 3: {1; 2; 3}, false

If the script is executed outside of a debugging session, redis.debug() has no effects at all. Note that the function accepts multiple arguments, that are separated by a comma and a space in the output.

Tables and nested tables are displayed correctly in order to make values simple to observe for the programmer debugging the script.

Inspecting the program state with print and eval

While the redis.debug() function can be used in order to print values directly from within the Lua script, often it is useful to observe the local variables of a program while stepping or when stopped into a breakpoint.

The print command does just that, and performs lookup in the call frames starting from the current one back to the previous ones, up to top-level. This means that even if we are into a nested function inside a Lua script, we can still use print foo to look at the value of foo in the context of the calling function. When called without a variable name, print will print all variables and their respective values.

The eval command executes small pieces of Lua scripts outside the context of the current call frame (evaluating inside the context of the current call frame is not possible with the current Lua internals). However you can use this command in order to test Lua functions.

lua debugger> e redis.sha1hex('foo')
<retval> "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"

Debugging clients

LDB uses the client-server model where the Redis server acts as a debugging server that communicates using RESP. While redis-cli is the default debug client, any client can be used for debugging as long as it meets one of the following conditions:

1. The client provides a native interface for setting the debug mode and controlling the debug session.
2. The client provides an interface for sending arbitrary commands over RESP.
3. The client allows sending raw messages to the Redis server.

For example, the Redis plugin for ZeroBrane Studio integrates with LDB using redis-lua. The following Lua code is a simplified example of how the plugin achieves that:

local redis = require 'redis'

-- add LDB's Continue command
redis.commands['ldbcontinue'] = redis.command('C')

-- script to be debugged
local script = [[
  local x, y = tonumber(ARGV[1]), tonumber(ARGV[2])
  local result = x * y
  return result
]]

local client = redis.connect('127.0.0.1', 6379)
client:script("DEBUG", "YES")
print(unpack(client:eval(script, 0, 6, 9)))
client:ldbcontinue()