In case anyone else wants to play about with the same data I’ll be using to prototype with, the site to go to is http://www.ssc.wisc.edu/wlsresearch/ — home of the famous Wisconsin Longitudinal Study, the WLS. Their data is wonderful stuff, believe me. I don’t like the way social surveys are done, but I think this the best of a poor lot, at least. Very good for our purposes. I have downloaded by the Comma Separated Value, CSV, format data and the Stata data, hoping that the R documentation is correct in saying that R can import Stata data files. I don’t actually want to use R, though it is a great package, not to mention free, but from R it is easy enough to export it in a format that I can read and manipulate using Python. I am more concerned about the variable information in the Stata files than the actual user response data, which I could easily read using the CSV files.
All this is for prototyping, doing things with the data that I have mentioned in earlier blog posts. I will only use a subset of the data, since dealing with more than a little of that massive data mountain (37 megabytes in zipped CSV) would be too much for one person, and I am only prototyping anyway. On the other hand, I do want the prototype to be capable of expansion and refinement into something real, so I’ll try to avoid limited the data capacity.
I am still going to be posting requirements analysis and design information from time to time, but I need to get more contact with the data and do a bit of coding in order to — well, to keep from being too bored, frankly. Dealing with data is fun, coding is fun, the paperwork is not. Not fun, but important. Don’t think I am minimizing its importance.
The last time I dealt with data from the WLS was 2002, and the survey had not reached 2000 yet. Now the whole dataset covers the test group from 1957 to 2007, which is a lot better, and more supplementary studies have been added.
Well, this is something to keep me busy, and will take a while. If anyone wants to get involved, please, get involved. — dpw