Home » Forums » Ubercart » Support
9 replies [Last post]
Thu, 05/29/2008 - 05:09
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
Was this information Helpful?

Hello,
I'm trying to import really big database (40mb xml file, about 10000 products), but i keep hitting timeout error. Is there any chance someone had encountered this problem and somehow solved it? I know its a big amount of products, but still..

My php settings is following:
max_execution_time = 3000
max_input_time = 60
memory_limit = 1000M

And yea I'm serious, it really took 50 minutes and not even then was the importing complete. Ubercart/import module probably wasn't build for such a big database Sad Although I was kinda surprised, I always thought about drupal as superior over other CMS, but I'm running site on joomla's virtuemart and importing this big database there is possible, but obviously not perfect too.

Top
Thu, 05/29/2008 - 09:06
#1 View page
Lyle
Lyle's picture
Offline
AdministratoreLiTe!
Joined: 08/07/2007
Juice: 6846
Re: Importing big databases

Can you break the file up into smaller pieces? Try doing just the categories and the attributes first, and then a few products at a time.

The importer really needs to be redone. It's probably the least polished part of Ubercart. But I don't want to hold back the next release while I try to make it better.

Top
Thu, 05/29/2008 - 09:46
#2 View page
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
Re: Re: Importing big databases

Well I could do that, but still it seems to me that it take to much time. I mean 50 minutes? are we in the 80's? :-] But main point is that the databased is synchronized every day with other site, so it would take too much time to do that every time.

BTW if I separate the categories the actual products will be stripped from the categories tags and following error will uccur:

warning: array_change_key_case() [function.array-change-key-case]: The argument should be an array in /var/www/vhosts/sc-comp.cz/subdomains/admin/httpdocs/sites/all/modules/ubercart/uc_store/includes/simplexml.php on line 434.

Its a shame that import/export module isn't finished, I think that a lot of shops using this function pretty often.

Edit: BTW i tested the import export api module, but it doesn't even show product content type as option for import so I didnt exactly tested that, but if this worked, it might be good temporary alternative for uc_import module.

Top
Thu, 05/29/2008 - 11:20
#3 View page
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
image module

I'm kind a suspecting the imagecache module of causing some delay with making thumbnails, or maybe it is the fact that the files actually are downloaded along the way. Probably both together. Is there a way to make this module keep just links to images, or starts with download only after the actual nodes are being displayed?

Top
Thu, 05/29/2008 - 11:33
#4 View page
Lyle
Lyle's picture
Offline
AdministratoreLiTe!
Joined: 08/07/2007
Juice: 6846
Re: image module

It's probably just the images being downloaded because Imagecache doesn't do anything until the modified images are requested.

You can leave the <image> tags out of the products. I would then see about modifying the database to attach the files to the new product nodes.

Top
Sun, 06/01/2008 - 11:46
#5 View page
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
Re: Re: image module

Tried some option past few days like replacing tag with tags to add image as text area (which I added by cck), disabling whole imagecache module or in the end deleting tags like you said but it still taking more that 3000seconds. I think thats the script which just cannot take that much information.

Anyway, when do you think you will have time for redoing the importer module? I mean, is it a matter of month, half a year, haven't thought about that yet?

Please dont take this like I'm pushing you to do that, I'm just curious.

Top
Mon, 06/02/2008 - 16:34
#6 View page
Lyle
Lyle's picture
Offline
AdministratoreLiTe!
Joined: 08/07/2007
Juice: 6846
Re: Re: Re: image module

The bulk of the work done on the importer module will probably be done for Ubercart 2.0. This means after the port to Drupal 6, but I'm hoping that gets done by Drupalcon. Since we'll be requiring PHP 5 for Uber Tuber, I'll be able to use the SimpleXML PHP module, which will be a heck of a lot faster than the importer currently is. There's no telling when the 2.0 release will be, but I'm sure the importer code will be available well before then.

Top
Tue, 06/03/2008 - 04:38
#7 View page
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
Re: Re: Re: Re: image module

Ok, thanks for info.

Top
Fri, 06/13/2008 - 08:50
#8 View page
zeezhao
zeezhao's picture
Offline
Joined: 04/23/2008
Juice: 969
Re: Re: Re: Re: Re: image module

Hi. See my comments on: http://drupal.org/node/81505#comment-880881

I am using node_import with the ubercart changes

I tried a couple of suggestions:

1. improve MySql for large imports see: http://drupal.org/node/259580
[meant changes to my.cnf]

2. used the set_time_limit() tip in this thread: http://drupal.org/node/191197
[ one line change to node_import.module]

3. changes to php.ini: e.g. I used large values because I am experimenting with importing a 250MB csv file with over 1 million records...

max_execution_time = 5000 ; Maximum execution time of each script, in seconds
max_input_time = 5000 ; Maximum amount of time each script may spend parsing request data
memory_limit = 508M ; Maximum amount of memory a script may consume (16MB)
post_max_size = 1500M
upload_max_filesize = 1500M
session.cache_expire = 18000

It now works without breaking. But I can't complete it in one shot, as it is going at the rate of about 30,000 rows/hr so will need days to complete 1 million rows....

So for me, the outstanding question is how to speed up node_import.

Top
Sat, 09/13/2008 - 11:56
#9 View page
svihel
svihel's picture
Offline
Joined: 05/29/2008
Juice: 140
Hello, any news on importer

Hello,
any news on importer module?

Top