Software to make plans. See the migration plans website.
With the advent of the Social Systems Project, this site will be more active.
Please visit the old blog at http://softwaredevelopment.socialtechnology.ca/sdwp/
Joomla has proved unsatisfactory for this and other sites, so I am migrating them to WordPress. Content will appear shortly. Please have patience.
Just a quick note: some of the data which wanted and could not get the ICPSR, such as the Educational Longitudinal Study of 2002, is available at http://nces.ed.gov/edat/ — I will continue to look for data sources other than the better known but clearly obstructionist ICPSR. I have managed to find on my old disks some data which the ICPSR used to make available before they cracked down on it. I wish I had updates for it, since for example, my GSS data is years old. If I can find it somewhere, I’ll let you know. — dpw
I was a bit worried about getting the formatting data out of the SAS Catalog or Command files in the Wisconsin Longitudinal Study. These are ASCII files, and necessary to read the CSV data files, but they are designed to be read by SAS, not by some program I might write. The answer to this is surprisingly simple. I used an ordinary text editor to strip off the first few lines and all the format lines at the end, then saved the file as a variable and value file, the whole of which is in one single format, like this:
value SEXRSP /* sex of respondent */
1 = ‘male’
2 = ‘female’ ;
Then I did the opposite, saving only the format lines, which maps variables to possibly new names:
format DEATYR DEATYR.;
format GROUP91 GROUP9A.;
These two new files are easily readable by a program which will be easily writeable.
It seems that the Python package Bitarray will work. It returns only a one dimensional array of bits, but as many as necessary can be put in a list. This will create something like a two dimensional array of bits. I need columns to be in the one dimensional arrays, so I will have to transpose the data. Did you know that you can transpose a two dimensional list of lists in a single line of Python code, using the map and zip functions? I don’t know if it will work for bit arrays yet, and especially don’t know if it will work for something huge, but I’ll try. — dpw