Trying to keep your data updated to a database while undergoing heavy traffic can lead to a very slow response for your queries. Ever wish you could push your writes to a queue so they can be handled during system time and not user time? It’s pretty easy. This model requires there to be a non-disk key/value data store (Redis, memcache) available that can act as a forward cache. This works better if the system can support lists. If an object does not exist in the forward cache when asked for, it needs to be pulled from the database, assembled and stored in the forward cache. When an object is updated, it needs to be updated in the forward cache as well as being pushed to the write queue. This way your user script only accesses the forward cache and the write cache during standard calls when the cache is already populated with the needed data.

This trick can be enhanced even further by storing a list of unique IDs in the write cache that can be replaced when an objected is updated before one of it’s previous writes was processed. This means that what would have been two database writes is now one! This requires storing a list in cache as well as the objects, which is done very effectively with Redis.

When used correctly and with a properly designed database that reduces the number of unique objects, your writes are faster and fewer! With a few more tricks this system can also be used easily and effectively in a cluster environment. More on that later.

Just today I was challenged with a project that will require value caching in a PHP/MySQL cluster environment. This project will have to deal with loads of up to 500 requests per second, most of which will involve writes. When working on a small scale it is easy to forget the latency issues associated with all engines in MySQL. This project will require a massive read/write caching system that will sit between PHP and MySQL and will require full data persistence. I found a valuable article that shows some of the timing differences of the popular storage systems under load. Redis seems to be the real winner here. With write-only data persistence (write to disk), this storage engine still comes out on top! This is truly the best system if you are caching write calls to MySQL. This way, most of the data will survive in case of a crash. Persistence can be increased even further by using another fully redundant Redis server. This is a must have for high-load write caching.

This video showcases some new features of the deferred rendering system in my Logos Engine including HDR, auto-exposure, edge detect anti-aliasing, shadow mapping and more. Check the page for more info.