Homework 2 run-time is excessive

Homework 2 run-time is excessive

by Cindy -

I am stuck on the last question of homework 2 on logistic modeling. Even after filtering out the unnecessary variables (including only 4 variables in my modeling), the Modeler has been running the logistic model for the past 2 hour 31 min and still counting....I have plenty of hard drive space and reasonably large RAM (16GB) and fast Processor (2GHz Core i7). Knowing that the NEISS dataset for homework 2 is much larger the Titanic dataset we used in prior assignment, this execution time seems excessive to me. 

For anyone who successfully completed the modeling in HW 2--How long did your logistic modeling take? What is your computer hardware specification?

Thank you.

 

Re: Homework 2 run-time is excessive

by Thomas Newman -

Hi, Everyone --

I haven't tried HW 2 yet, but one tip I used for HW 1 that is useful for analyzing large datasets is to try everything first on a much smaller version of the dataset -- perhaps 10,000 records, so everything runs fast.  Then once you've written the commands or (in modeler created all of the nodes that you want) and everything seems to work, just change the source/input command and run it on the large dataset.  You can just run it overnight, knowing that it's pretty likely to work.

You can make a smaller version of the data set by using the sample command in Stata, which keeps a random subset of the data.  You could also cut rows from Excel, but that's a little riskier if the data are sorted in some meaningful order.

You can still do this even you need a very large N because the outcome is rare.  For example if the outcome is a 0/1 variable (e.g. "cancer") you could read the dataset into Stata and then run the following command:

sample 1 if cancer==0

This will take a 1% sample of all of those who don't have cancer but keep all those who do.

I'm too far behind in this class to offer any more specific suggestions!

Good luck!

Tom

 

 

Re: Homework 2 run-time is excessive

by Anoop Sheshadri -

I'm not certain what's happening with your stream. Mine took maybe 15-20 seconds, and slightly less when running your stream (possibly because your partition parameters are set differently from mine). The only change I made to your stream at all was that I had to connect the source file to the one I have locally on my laptop. Unless there's a problem with the origin file, then I'm not sure I have an answer, but even that should have caused a problem earlier in the stream for you than the logistic command.

On the bright side, you got very similar answers to what I did when performing the analysis. I will try to see what is going on but I don't have an answer yet.