Youtube batch processing – less costly requests

The need: YouTube is a very popular video hosting service as everyone knows. And there are many others like me who use this service to host videos. Then when needed these videos are searched based on user request and different levels of information are displayed. While this seems to be a trivial task and yes YouTube has a very nice API with all the functionality needed right there, and the documentation is really very comprehensive. On a side note I

Continue Reading

The “Shadow Table” trick.

The need: Often there is a requirement where data in a particular table has to be processed, and the data processing might be slow, while the table might be a one that is used by your application extensively. For example, a logging table that logs page hits. Or there might be an archiving operation that has to be performed on a particular table. Archiving / processing / aggregating records, all these operations are slow and can really blog down a

Continue Reading

Performance tuning using vertical partitioning.

Does having small data-sets really help? Of course it does! Are memory lookups faster that disk lookups. Of course ! So many times I have seen people complain about queries taking too long now, while they were not taking that long earlier. There is one big reason for this, earlier the size of data-set was small so it could fit into memory. Now that the data-set has grown large enough that it cannot fit entirely into memory, the disk seeks

Continue Reading