#####
Redis
#####

.. image:: /_images/image_redis.png
   :width: 150pt

.. raw:: html 

   <br>
   <br>

As an architect on the `Internet of Things Red Hat ITCM project`_ I was a big supporter of heavyweight AMQP message queue clusters across numerous data centers for achieving true 5 9's of uptime while focused on ``no messages getting lost``. Since this was a massive safety system used by all railroads, there was a true business need to prevent message loss and mitigate downtime. With a focus on safety + high-availability, the overhead of implementing a massive message queue (MQ) system like that had direct implications on development speed and budgetary requirements to build the solution. 

After spending years setting up routemaps and implementing publisher-subscriber (pub-sub) messaging patterns, I have found Redis to serve a great dual purpose of not only caching but also as a very simple publisher-subscriber queueing system. Knowing how expensive it is to build and support highly available 99.999% AMQP solutions, I have created, supported and open sourced multiple variations of highly-available Redis clusters that run in Docker containers that can be distributed across multiple nodes just like RabbitMQ_ or MRG-M_ AMQP clusters. I use these Redis clusters for caching and publisher-subsriber messaging every single day. 

Redis is the backbone and first lego block I would choose when laying the foundation for a new distributed real-time stack. For the past two years, my Redis servers are utilizing over 30+ TB of data transfer in a month. In May 2016, my AWS bill recorded the data transfer in at ``47,567.947 GB``. I have tuned and optimized my Redis clusters to never die while achieving a good throughput somewhere around +1 TB/day.

I am indebted to Antirez_ for his Redis_ caching technology and `Todd Lucas`_ for introducing me to Redis as a means to efficiency cache data.

.. _`Internet of Things Red Hat ITCM project`: https://www.redhat.com/en/about/press-releases/red-hat-and-meteorcomm-collaborate-to-support-deployment-of-interoperable-positive-train-control-communication-system
.. _RabbitMQ: https://github.com/GetLevvel/testing-rabbitmq-clustering-with-docker
.. _MRG-M: https://access.redhat.com/products/red-hat-enterprise-mrg-messaging
.. _`Antirez`: https://github.com/antirez
.. _`Redis`: https://github.com/antirez/redis
.. _`Todd Lucas`: https://github.com/toddlucas

-----------------------------------------
Load-Balanced, Self-Healing Redis Cluster
-----------------------------------------

I created this repository for housing my investment platform's self-healing real-time stock and news data pipelines. I use it for caching and messaging. It is a `load-balanced HAProxy for connectivity across a multi-host 3-node Redis Cluster (no sharding)`_

.. _`load-balanced HAProxy for connectivity across a multi-host 3-node Redis Cluster (no sharding)`: https://github.com/jay-johnson/docker-redis-haproxy-cluster

To date the only crash has been due to overrunning memory and filling up the hard drive...I find the docker deployment tools to be a fantastic fit for message queue technologies like Redis and RabbitMQ.

----------------------------
6-Node Sharded Redis Cluster
----------------------------

I built this repository as a means to distribute a `6-node Redis cluster with sharding`_. It is the scaled out HAProxy version mentioned above:

.. _`6-node Redis cluster with sharding`: https://github.com/jay-johnson/docker-redis-cluster

-------------------
Want to learn more?
-------------------

*  .. raw:: html 

        <a href="mailto:jay.p.h.johnson@gmail.com?Subject=Hello">Email me <i class="fa fa-envelope"></i></a>

* `Contact Information`_

.. _Contact Information: https://jaypjohnson.com/contact.html


..  SEO Metadata
    SEO_SET_HEADER_URL="https://jaypjohnson.com/redis.html"
    SEO_SET_HEADER_SITENAME="Redis - A Fundamental Product Component"
    SEO_SET_HEADER_TITLE="Redis - A Fundamental Product Component"
    SEO_SET_HEADER_DESC="A discussion about using Redis clusters as a backbone for caching and simple message queueing after using it in multiple real-time news and data-ingestion products"
    SEO_SET_HEADER_KEYWORDS="software, architecture, python, high availability, redis, clusters, redis ha, healing, haproxy, load balancing, docker"
    SEO_SET_HEADER_TYPE="website"
    SEO_SET_HEADER_TWITTER_CREATOR="@jayphjohnson"
    SEO_SET_HEADER_TWITTER_SITE="@jayphjohnson"
    SEO_SET_HEADER_SEEALSO="https://jaypjohnson.com/"
    SEO_SET_HEADER_FAVICON="/favicon.png"
    SEO_SET_HEADER_IMAGE="/_images/image_redis.png"
    SEO_SET_HEADER_CARD="summary"
    SEO_SET_HEADER_AUTHOR="Jay Johnson"