For the purpose of what I said above, and this comment, I’m calling the SQL equivalent of ‘insert/update’ an update, and the SQL equivalent of ‘select’ a query.

I’m not sure if this will work or not, but if you call ‘update’ one dimension, and ‘query’ another dimension, you can probably do a multi-dimensional FFT. It just so happens that there’s a nice module written in C to do just this sort of thing from Ruby.

http://ruby.gfd-dennou.org/products/ruby-fftw3/doc/ruby-fftw3.html

I think in order to treat them like orthogonal components, you need to prove them to be statistically independent. However, if this is possible, it removes the directional correlation problem and reduces caching to a multi-dimensional FFT and a knapsack problem (filling your online buffer).

We can talk more in person – I don’t want to keep filling up your blog :-p

I don’t know if you remember or not, but I think when you were at the AFRL that summer, I built a little tool to export the structure of an Oracle database to a series of HTML files. The tool used metadata queries to grab a current version of the schemas, tables, fields, and constraints. The first time I did it, I think I managed to do it in O(n^2), but I was querying O(n) times. The program completed its task in about 6 minutes.

After talking with Nick Watts (our database guy) a bit, I found a way to get all the data I needed from a single query, but rebuilding the structure from the query operated in O(n^3) time. This completed in under 5 seconds.

I might be off-base here, since I don’t know Ruby or ActiveRecord, but keep in mind that the big thing with this type of code isn’t really making sure that your online processing time is low, it’s minimizing the number of queries performed on your offline structure (in this case, subversion). However, If you can keep your query count low, and have constant time, then you’re golden.

Hopefully my (somewhat) unsolicited advice is useful. If not, feel free to tell me to buzz off

As for the solicited advice, I think the best way to tell which is faster would be to profile them both. In some ways, removing a middle-man is a good thing, but I think your obvious concern is that MySQL might be a beneficial caching mechanism. Unfortunately, the only way to determine what’s faster is empirical analysis.

I (once again) don’t know ActiveRecord at all, but I think the *best* (yet hardest to code) solution would be to write in a layer that identifies frequently queried data, and caches it based on frequency of access and an upper memory bound. If you wanted to get really fancy, you could do an FFT on your data’s update timestamps, an FFT on the query timestamps, and then cache the data that is both at the high-end of the query spectrum, but the low end of the update spectrum. I think there are some ‘directional correlation’ algorithms that will handle this well. Of course, you’d have to implement a smart pedigree mechanism – but if you’re correlating FFTs on subversion records, I think you can handle that.

Wow, I babble a lot… I hope this was useful, and congrats again for graduating.