Scott,
You have hit the nail on the head. We are not doing fundamental research. The sciences are turning into craft. Lee Smolin first brought this up about physics in the last 5 chapters of his book, "The Trouble with Physics."
In CS, we have this disease in spades and partly for the reasons you outlined below, the pursuit of the dollar. I also think to some degree what I have come to characterize by paraphrasing Arthur C. Clark, 'Any sufficiently advanced craft is indistinguishable from science.' We are so dazzled by the products of Moore's Law that we don't see that what we are doing is craft.
The trouble with craft is that it stagnates.
The classic example is Chinese "science" prior to Western contact. See Needham's "Science and Civilization in China." To some degree, Needham ends up arguing (and most scholars agree) that 'science' in pre-Qing China was more technique or craft. There was no theory, no abstraction, no attempt at a theory that holds it all together. (By their own admission, this problem still plagues China and India. There are the exceptions, but in general it is a recognized problem.)
By late Ming (17thC), it had pretty much stagnated and they were losing knowledge. Needham says that it is because merchants (capitalists) were at the bottom of the heap. The government power structure controlled everything. I also believe it is because there was no Euclid. There was no example of an axiomatic system. The Holy Grail of a scientist is to do to his field what Euclid did to geometry. Interestingly Heilbrun points out in his book on geometry book that the Greeks were the only ones to develop the concept of proof. Other civilizations have mathematics, they have recipes, algorithms; but not proof. Proof is at the root of building theory. Theory gives the ideas cohesion, shows how they relate in ways you didn't expect, and shows you where the gaps in your knowledge are. The quest for theory is more important to avoiding stagnation as the pull of capitalism.
Needham didn't live to see it. But we now have the example of how the entrepreneurial drive leads to stagnation. That drive is fine for exploiting *within* a paradigm, but it won't get you to the next one. And we have seen the example of that as well.
And we are seeing the same stagnation in CS. One sees the same the same papers on about a 5 years cycle. The "time constants" have changed but they are the same papers.
Early CS was much more scientific. We went about things much more methodically, we were more concerned with methodically understanding the fundamentals than just building something that worked. (BTW to your comment: We *did* do a lot of RJE on the early ARPANET. We had many scientific users submitting jobs on particle physics, economics, weather simulation, etc. However, we never saw it as the future. We had much bigger ideas in mind, for distributed computing (ask Dave). It is really depressing that 40 years later, things really haven't moved anywhere. The hardware is 10s of thousands times faster and bigger. You are right. We have re-labeled RJE, cloud computing, and never gotten past the 3270/Mainframe days.)
You are right. We do have to get back to this. And there I am afraid it gets disheartening. We have 30 years of conditioning the field toward everything else but. I don't see many who even when they say we need to do it, know how to do it. We have selected against the ability for decades. I am even finding that CS students (and professors) have trouble with abstraction. For a field that one could say was founded on abstraction, this is really scary.
Take care,
John Day