Sounds like PHP might be timing out.
In Sources/shop/items/filedownload2.php (I think that's what it's called), right below the <?php, add:
set_time_limit(0);
See if that fixes it. Alternatively, you might want to store your files in a web-accessible location (perhaps a directory with a random name and indexing turned off), and then just use the Display Message item to show a link to the file? This might work better.