Besides I’m not talking about Java vs. Ruby. I’m talking about Grails vs. Ruby on Rails. Groovy != Java.

As far as I can tell, Grails is more beautiful and elegant than RoR.

It is amusing to see Ruby fanbois on the attack though. I’ll admit I’ve never seen that before.

I really recommend the ROCR package (in R) as a well-done, reusable package for all the standard confusion plots, e.g. P/R and ROC, expected costs, etc etc. I was thinking of porting my plotting routines into their system, eventually…

I skimmed through the Carterette paper and it’s interesting. My concern with pairwise setup is, in order to get comparability among query-result pairs, you need to get annotators to do an O(N^2) amount of work. (Unless you do something horribly complicated with partial orders.) The absolute judgment task scales linearly, of course. Given the AMT environment and a fixed budget, if I stay in the smaller-volume task, instead of spending a lot on a quadratic taskload, I can simply get a higher number of workers per result and boil out more noise. Of course, if it’s true the pairwise judgment task is easier — as the paper claims — that might make my spending more efficient. But since it’s polynomial, no matter the cost/benefit ratios, there has to be a tipping point where, for a given data set size, you’d always want to switch back to absolute judgments.

Absolute judgments are just so much easier to compute with — both for analysis and to use as machine learning training data. I really don’t want to have fancy utility inference or stopping rule schemes just to know the relative ranking of my data. (And I think real-valued scores will always become a necessity. Theoretical microeconomists have made boatloads of theorems about representing preferences by pairwise comparisons. It turns out that when you add enough rationality assumptions — e.g. the sort that are demanded of search engine ranking tasks anyways — then your fancy ordering can always be mapped back to real-valued utility function.)

I’d be most interested in a paper that compares real-valued scores derived from some sort of pairwise comparison task, versus absolute judgments, and is mindful of the cost tradeoffs in service of an actual goal, like ranking algorithm training.

For the future, I would suggest using pairwise preference judgments as an alternative (Here or There: Preference Judgments for Relevance by Ben Carterette, et al.).

I was just curious how you found Slicehost to work in this situation and if Slicehost had a different solution that they weren’t announcing publicly for situations like yours.

I have personally steered away from the round robin DNS approach in the past. If one of your frontline servers is down, it can take a little time to propagate DNS updates to remove the server from the round-robin. Some DNS servers will cache the old IP for a period of time, thereby preventing access to your app until their cache is flushed or until the slice is operational once again. RR DNS is not the best solution, but the best one with Slicehost and it gets the job done in a pinch. A reverse proxy would be optimal to prevent this as you mentioned.

NAT is the one feature I’d love to see Slicehost add, as it would allow larger customers to add new slices behind their single public IP without depending upon a single slice to do the reverse proxy work or stale DNS situations when using RR DNS. I noticed that Amazon EC2 now has this, though I haven’t investigated it fully.

PJ, Chris: To be fair, Grails is in the Groovy language, not Java. I rather liked Groovy the last time I used it; the biggest issue was that the compiler was hideously slow, though I’m sure it’s faster now. Probably F#/.NET is the most advanced dynamic language platform anyway