What Drupal / Ubercart processes run every 15 minutes that could kill a file download?

Posts: 25
Joined: 09/24/2008

I may be totally off the mark, but my testing has led me to the hypothesis that something is going on behind the scenes that is killing long downloads from ubercart / drupal.

Here's what I know:

Files are failing at peculiar times: 10:00PM, 10:30PM, 11:30PM, 12:00AM, 2:30AM, 4:AM, etc. This has been consistent. A few have died at other times, like 2:58, but the majority are right on the 15 minute mark. This is for a 1.8 Gigabyte file.

1. I can download the file successfully via FTP.

2. I also copied the file (via SSH) to an unprotected folder, emailed myself a link, and was able to download it via HTTP. Out of three tries, it did die once, but at a much higher percentage of completion than the others.

3. I don't have any Cron jobs running.

4. I am running on MediaTemple Grid Service, and haven't had any other major problems. Download speed fluctuates between ~250 to ~500 kbps on my connection (typical cable internet).

5. I am not finding any leads in the Error logs of either the server or Drupal. There are some peculiar errors in there, but they are not consistent with losing download connection.

These conditions lead to me to believe there might be something going on behind the scenes with either Drupal, UberCart, or MediaTemple Grid Service (THESE ARE HYPOTHESES, and I don't know how to test or fix any of them):

1. The users session / connection is expiring or timing out
2. There could be a problem with the way UberCart is masking the file name
3. Mediatemple is restarting the server every 15 minutes
4. My Database is corrupt
5. Your guess here.

Can anyone help me with this?

Let me know what I can do to help you understand the problem better.

-Mike

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

Offhand, I'm not sure. My first inclination is to look at plugins you might be using in your browser. Are you on Firefox? Are you downloading using a manager, or the browser's built-in downloader?

Since HTTP transfers can be a little wonky, my guess is that it's something either lagging the browser and dropping the file... or something else. I've only heard of this occuring for a couple people, and I think that might be an email we just received on our end as well. You should be able to keep resuming the file until it's fully downloaded. However I'd like to get the bug figured out, if it is a bug. (It might be some kind of PHP issue as well.)

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

Hello,

I'm using firefox w/ built in downloader, but customers are using various browsers and download managers and are unable to retrieve the full file.

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

Hmm.. interesting. Do you have this same issue with any smaller files? Have you noticed any type of "cutoff" where the filesize becomes too big for the transfer? I'm just trying to flesh out some ideas... there is the possibility that it has to do with script timeout or running out of memory.

I know you mentioned there were no Watchdog errors, but have you checked Apache?

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

Most files are cutting off around 200-300MB... but i had some cut off that were in the 600MB range.

I checked the server error log, and there's nothing much here, just some missing favicons and robots.txt

Posts: 25
Joined: 09/24/2008

I've also modified php.ini for the following:

max_execution_time = 600

default_socket_timeout = 600

could this be a problem?

Posts: 25
Joined: 09/24/2008

Other ideas I had were to disable the Drupal Cache (it's time based)

I've also run through other modules to find out which ones worked on a "scheduled" basis, that could be running every 15 minutes, but couldn't find anything.

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

Yeah I'm not really sure at this point... I think it might be specific to your server, since we have files that are 1GB+ and we haven't been told of a lot of issues. Since yours seems to be reproduceable, then there might be something in your config that is causing it. Of course there could be other issues at hand but we'll need to narrow them down.

The cache idea is an interesting one but the file downloads don't actually involve the cache system AFAIK.

One thing you might look into is the file_request variable at the top of uc_file.module. I think this is the root cause of issues that we are seeing. Try editing the module, if you are inclined, to not worry about the number of requests. (I think it's set to 50, and it stores this value in the cache table... there is the possibility that this is your root cause as well.)

My suggestion is to just find another way to cache the requests; since a user could be getting simultaneous connections from their browser or manager, the number of requests will be skewed higher. Even though you are only downloading the file once, your multiple connections are hitting that cache function a bunch of times.

So... to reiterate. Try commenting out the function (or the lines in the function) that increment the cache value and then check to see if they've gone over the limit. Then try again and see if that solves it.

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

I changed UC_FILE_REQUEST_LIMIT from 50 to 500, then ran the database upgrade script.

I'm downloading the file now, and am waiting to see what happens.

The Drupal cache is still "Off" and the minimum cache lifetime is set to "None"

Posts: 25
Joined: 09/24/2008

The File Download stopped at 610MB at 4:45PM.

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

Okay so I doubt that's the issue.. hmm, I'm really at a loss here. Have you tried contacting media temple? They might be able to shed some light on this.

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

I opened a support request with them last time i was working on this problem, and they were unhelpful. They did the same testing that I am doing now, and couldn't find any problems, since they were downloading it straight off of their own system.

Very frustrating experience with their tech support.

Posts: 95
Joined: 08/22/2008
Getting busy with the Ubercode.

Hey, sorry to get in late on this.

In the patest file module, I added a warning about using download accelerators. It can ruin the verification system and prohibit the user from finishing the download.

Time for ascii-art diagrams...

[                                    ] Whole file
[        ][        ][       ][       ]
  Chunk 1   Chunk 2  Chunk 3  Chunk 4

Now, this is usually the scheme that downlaod accelerators use. They will open (in this casE) 4 simultaneous socket connections, and (thanks to torgos's patch) start downloading at each chunk beginning.

The rub between this model and Ubercart's authentication mehcanism is as follows: When a file download is complete, the download count is incremented, and that specific file link is invalidated. How do we know when that happens? We know because the last byte of the file is transferred. Now let's look at our diagram again:

[                                    ] Whole file
[xxxx    ][xxxxxxx ][xxxxxx ][xxxxxxx]
  Chunk 1   Chunk 2  Chunk 3  Chunk 4

You can see, the first 3 chunks aren't quite done yet, but wait... the very last one is. As soon as Ubercart transfers the last piece of file information, any additional connections will be shut out from the link given.

Now, I'm no t exactly sure if this is the problem you're having, it may very well be something host-specific... regardless, when someone emails you for support, make sure to make it abundantly clear that download acceleration is not reliable for authenticated downloads, and they need to download the normal way, one connection per file, to ensure the transfer succeeds.

Hope this helps.

From the look of your failure times, exactly on :00, :30... etc, it looks like this may not be the actual problem. Oh well, worth a shot. Smiling

--

Try FreeBASIC!
My game Lynn's Legacy

Posts: 25
Joined: 09/24/2008

Thanks for the tip, we do have some users who tried a download manager or accelerator, but in my test environment, i'm using just FireFox and it's built in download manager.

Posts: 95
Joined: 08/22/2008
Getting busy with the Ubercode.

Another thing you could try (if you have SSH access) is 'uptime' which will tell you how long the server's been up... if it directly correlates with the failure times, then we have the problem...

--

Try FreeBASIC!
My game Lynn's Legacy

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

IIRC I think Firefox 3's system is the same (or at least similar to) the way a Download Manager works. I know this because when I was testing, I found that my patch worked fine in FF2 but not in FF3, and it turned out that FF3's system also requires ETags (and something else, I believe) or else it wouldn't resume.

However this isn't an issue with resumes, per se, but I'm wondering if these issues are related somehow.

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

cha0s wrote:
Another thing you could try (if you have SSH access) is 'uptime' which will tell you how long the server's been up... if it directly correlates with the failure times, then we have the problem...

This would be great information to have, however I get "permission denied" when trying the uptime command.

Is there another way to do this?

Posts: 25
Joined: 09/24/2008

torgosPizza wrote:
IIRC I think Firefox 3's system is the same (or at least similar to) the way a Download Manager works. I know this because when I was testing, I found that my patch worked fine in FF2 but not in FF3, and it turned out that FF3's system also requires ETags (and something else, I believe) or else it wouldn't resume.

However this isn't an issue with resumes, per se, but I'm wondering if these issues are related somehow.


I have been testing with Firefox, and some of the complaints i had were from FireFox users... I'm testing in Safari now...

Posts: 25
Joined: 09/24/2008

rosswog wrote:
torgosPizza wrote:
IIRC I think Firefox 3's system is the same (or at least similar to) the way a Download Manager works. I know this because when I was testing, I found that my patch worked fine in FF2 but not in FF3, and it turned out that FF3's system also requires ETags (and something else, I believe) or else it wouldn't resume.

However this isn't an issue with resumes, per se, but I'm wondering if these issues are related somehow.


I have been testing with Firefox, and some of the complaints i had were from FireFox users... I'm testing in Safari now...

Ok, I tested in Safari, and the download stopped at exactly 7:30PM.

Posts: 95
Joined: 08/22/2008
Getting busy with the Ubercode.

I'm sorry no, I don't know of any other command. Funny they wouldn't give you permission to execute such a benign command. Maybe you can convince their support to grant you permission. Smiling

--

Try FreeBASIC!
My game Lynn's Legacy

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

The timing does seem to be a clue - perhaps there is some kind of event that happens on their system every 15 minutes. What kind of hosting are you on with them? I'm assuming it's not a dedicated server, but probably a shared environment? I'm fairly sure that if you were on a dedicated box you wouldn't be seeing this happen.

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com

Posts: 25
Joined: 09/24/2008

It's a shared server, the call it "Grid Service"

Any recommendations on a host? I'm really hesitant to switch hosts unless I know this will work for certain on a new system.

Posts: 1377
Joined: 08/14/2007
Bug FinderEarly adopter... addicted to alphas.Getting busy with the Ubercode.

Well I have a dedicated server with SoftLayer - www.softlayer.com - and our users download files that range in size from 10mb to 2gb. So it definitely works on other servers.

--

"Pain don't hurt." - Dalton

Mike Nelson's RiffTrax! www.rifftrax.com