TCP/IP programming has been a really ‘fun’ experience so far in my journey to create a JavaScript web server. Starting with sockets all the way to HTTP, I have succeeded in destroying my confidence in my ability to understand computers. In case any of you suffer from this same affliction dealing with sockets, here is a little tip that may help you get to bug #341: Content-Length Means Bytes, Not Characters.

Trying to keep your data updated to a database while undergoing heavy traffic can lead to a very slow response for your queries. Ever wish you could push your writes to a queue so they can be handled during system time and not user time? It’s pretty easy. This model requires there to be a non-disk key/value data store (Redis, memcache) available that can act as a forward cache. This works better if the system can support lists. If an object does not exist in the forward cache when asked for, it needs to be pulled from the database, assembled and stored in the forward cache. When an object is updated, it needs to be updated in the forward cache as well as being pushed to the write queue. This way your user script only accesses the forward cache and the write cache during standard calls when the cache is already populated with the needed data.

This trick can be enhanced even further by storing a list of unique IDs in the write cache that can be replaced when an objected is updated before one of it’s previous writes was processed. This means that what would have been two database writes is now one! This requires storing a list in cache as well as the objects, which is done very effectively with Redis.

When used correctly and with a properly designed database that reduces the number of unique objects, your writes are faster and fewer! With a few more tricks this system can also be used easily and effectively in a cluster environment. More on that later.

This video showcases some new features of the deferred rendering system in my Logos Engine including HDR, auto-exposure, edge detect anti-aliasing, shadow mapping and more. Check the page for more info.