import/export hooks for other modules

33 replies [Last post]

Thu, 09/13/2007 - 04:54

CpILL

Offline

Joined: 08/08/2007

Juice: 549

I got though the long thread on Import/Export and noticed that mid way there was a comment by catorghans@drupal.org on incorporating the SimpleStock levels module into the import/export routines. I think I suggested a while back that perhaps a hook (or two) could be introduced into the fray that would allow any module to read/write while importing/exporting data. This would seem to follow the general "Drupal way".

I've only looked at the import code briefly but I will make this happen for myself if the UC team hasn't got time and post it here.

I haven't used SimpleXML before and I'm not sure if its an expat or DOM based parser. I hope its the later as I'd want to pass whole chunks of the XML to the module_invoke_all() call i.e. all the children of '/store/products/product' for example.

Any comments on this?

Also I'm going to make an interface for CSV importing as my client will need to use it. I'll start with macgeek script I imagine.

I also noticed that the undocumented image import feature remains undocumented

Uberdevelopment www.tsd.net.au/blog

Thu, 09/13/2007 - 06:43

#1 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Hi,

I already made an CSV import export for the SimpleStock. But I am waiting on the hook system within the importer as well to finish it.
My current version is a hack within the uc_importer.module (current bazaar version).

It uses a separate Xpath:
/store/inventory/product
with
/store/inventory/product/model
/store/inventory/product/quantity
/store/inventory/product/is_tracked

I made an import and export and checked for the existence of module "uc_stocklevels" and for corresponding "adjustments" (the model should exist). Adjustment are allowed in import/export (but are also not documented).

It also works if there is only "/stock/inventory" information in the XML file, without the products.

You obviously have a better understanding of the details of SimpleStock, so change it how you want. But this version works for me for now.

Hans

Attachment	Size
uc_importer.zip	9.55 KB

Thu, 09/13/2007 - 08:51

#2 View page

Ryan

Offline

Joined: 08/07/2007

Juice: 15438

Re: import/export hooks for other modules

Hey, CpILL, if you want to give the hook a shot, that would be great. I'm not sure where the importer stands on the "need-to-revisit" list, but I don't think it's come up in conversation for at least a week or two. We can look into rolling the changes into core once you knock it out. I think you're right about the necessity of a hook there, btw.

Thu, 09/13/2007 - 09:33

#3 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: import/export hooks for other modules

@catorghans - thanks for that. Two heads are always better than one. I'll extract what you have done and try and create a new module for it or perhaps move it to the Simple Stock code so that it can be invoked by a hook call eventually, which it looks like I'll be writing.

The CSV import module will just extent the importer I think providing an interface with some CSV specific options.

Here I go then...

Uberdevelopment www.tsd.net.au/blog

Thu, 09/13/2007 - 13:49

#4 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: Re: Re: import/export hooks for other modules

I also have a script that converts a simple XML export from access to the pretty advanced Import XML (including my Stock XML). It is absolutely undocumented and certainly not generic, needs my specific XML import, handles only 1 sort of attribute (size) but it creates a working import XML, including categories, manufacturers, attributes, products with "adjustments" with the (in my opinion) strange serialization. Maybe it helps with your csv import. I can help with it as well next week.

I included the convert script, the "allsmall.xml" which came from access and the result, which is a working import XML.

Attachment	Size
convert.zip	72.03 KB

Thu, 09/13/2007 - 15:09

#5 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: import/export hooks for other modules

@catorghans - thanks you've been a big help with this so far. I have started to think about going from Excels XML to the UC XML since everything Excel (and Access I imagine) can open it can save as its own XML format. If I used XSLT, and perhaps had a select list with a list of XSLT files to choose from so you could go from any XML format to UC this would be a really handy addition to the project. MySQL, Access, MYOB etc all have XML export functions. what do you think?

Also, how did you handle the adjustments import? Looking at Lyls code it seem to expect a data structure like:

/adjustments/adjustment/combination

which is a serialized PHP array. Obviously not much good when coming from human data entry. This a bit of a show stopper for me for I don't see how I can go from a CSV to the required XML if I have to generate serialized PHP arrays if reference IDs that won't exists on first import? I guess this is why catorghans separated the inventory to a different branch of the XML root(?)

Uberdevelopment www.tsd.net.au/blog

Thu, 09/13/2007 - 15:32

#6 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Thanks,

The combination is part of the /products, also with me. The smart part of the serialization thing in Lyls code is that you can serialize your own attribute/option ID's, the import unserializes it AND matches them to the real ID's and serializes it again. (look at my code, it works)
Don't know how you could do that in XSLT, but in php it is easy

I separated the inventory because I will let an external application update the stock when it needs to, and that application has only knowledge about "model" and has no clue about product type, name, or even sell-price. It also needs to update the stock of the "options", and it could impossible know those "id's".

I know about the access XML and that uses the table name and the table columns in it's default XML export. You'll need some kind of mapping like the "macgeek" script, seems hard (not impossible) in XSLT too, so I would go for an php conversion.

Hans

Thu, 09/13/2007 - 17:58

#7 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Thanks,

Ah, so I would have to generate my own IDs for attributes when importing. I guess that can be done, but how does it do the matching, based in the SKU?

I'll have a sniff tomorrow

Uberdevelopment www.tsd.net.au/blog

Fri, 09/14/2007 - 03:13

#8 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: Re: Thanks,

Yes, your own ID's can be anything you want (including text).
In my case they are "size" and "L" for example.

The importer module looks them up in the system by "name", and adds them or updates them and gets the real "ubercart"-id's, and those are used for the "matching".
You can enter the model(SKU) with the "product" it self and in the "adjustments" for the options. Those have to be unique.

Then in the "stock"-import, the model(SKU) is used for matching.

Hans

Thu, 09/20/2007 - 05:53

#9 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

import XML format question

I am importing a spread sheet that has one or more columns that are taxonomy's, that is, the values in each row are terms in the taxonomy which is the column. I have an interface for selecting which column is for which taxonomy. The question is: how do I represent this in the XML import format as 'manufacturer' seems to be the only privileged vocabulary to be represented there.

there is:
/store/products/product/categories/category/id

but I don't understand which ID this is looking for and is this really for taxonomys?

Uberdevelopment www.tsd.net.au/blog

Thu, 09/20/2007 - 06:40

#10 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: import XML format question

Categories is only for one taxonomy category -> Catalog.

I just made a "test" taxonomy and gave products one of the terms. They did not appear in the export. So it seems there is no taxonomy import/export besides the "catalog" and "manufacturer" taxonomy.
Would be a nice feature

category/id
attribute/id
option/id
product/id

can be: the id's in the system (IF you have them)
OR
the "name" of the item.

The import will look at the id and see if it exists as that ID, if not it will look at if that ID exists as name. Then it will create a matching array for the "import/id" and the "system/id".

Can I help?

I do also have some ideas for the hooks in uc_importer.module

Thu, 09/20/2007 - 06:54

#11 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: Re: import XML format question

I also need an "unpublished" field in the import/export.

Thu, 09/20/2007 - 14:01

#12 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Can I help? YOu sure can! If

Can I help?
YOu sure can!

If you I was gong to suggest perhaps having a go at hooks for the impoter, and since you have ideas for it already, that would be smashing.

What are your ideas? (before you start so we can both get on the same frequency)

Uberdevelopment www.tsd.net.au/blog

Thu, 09/20/2007 - 17:01

#13 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: Can I help? YOu sure can! If

Well, the import could do with one normal invoke_all because every module can do the things it has to do with the xml it receives. The hook will get the complete xml as a parameter.

The export can also have an invoke_all. All modules will add a part of the xml to the invoke_all return array. All elements of the array have to be placed inside the xml.

There might be more hooks needed in the export if we want to allow modules to write within parts of the xml. So one hook at the end of /store/products/product/ and one at the end of /store and maybe even at more places.

That where my ideas until now.

Fri, 09/21/2007 - 12:02

#14 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Can I help? YOu sure can! If

Ya, it might not be very efficient to invoke_all with all the XML twice. And if we're going to do it for some sub-bits of the XML I think we should be systematic in our approach.

It might be good to invoke_all for every path and allow the modules to step in anywhere they like i.e.

for path '/store/products/product' we might automaticly create a hook

<?php
module_invoke_all('importer_'.implode('_', explode('/', $path)), $simple_xml_object);
?>

So someone might define

<?php
function mymodule_importer_store_products_product($simple_xml) {}
?>

Which would get to process every product I guess after the main importer/exporter had a go at it (so all the info was there).

If thats too dynamic perhaps just a hook that asks for an array of XPaths at the end of import/export and then feeds the result to them, but then you might as well just give them the XML at the end...

The way I would do it is to actually try and implement it for some module you need this for and see what is actually required by the module. I think this needs to be done for the 'taxonomy module' so products can be associated with any vocabulary as many times as is desired (and possibly with terms that are hierarchical) so it might be a good test case.

Uberdevelopment www.tsd.net.au/blog

Fri, 09/21/2007 - 16:50

#15 View page

Lyle

Offline

Joined: 08/07/2007

Juice: 6846

Re: Re: Re: Can I help? YOu sure can! If

PHP objects are passed by reference in functions, so sending the whole simpleXML object isn't any more inefficient than part of it. As long as nobody writes a module that changes that object, things should be alright.

Then, too, the burden is on the modules to know which XPath they have to deal with. On the other hand, they would have access to the entire XML structure to get any information they need.

What really needs figuring out is how to handle the id map array. I've got some thoughts about that, but it's Friday afternoon, so they're not very coherent. I'll mull over this over the weekend and see what it looks like on Monday.

Sat, 09/22/2007 - 08:28

#16 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Here here! I suppose passing

Here here!

I suppose passing the whole tree is better as you ill never know how much information is needed by a module. The attributes module would need to access:
/store/attributes/*
/products/product/attributes/*
and possibly
/adjustments/adjustment/*
While processing just one product.

I'm on the reverse side of the equation and have to consider how to build the XML from one adjustment. I'm using SimpleXML to build the tree and render the output, would it be faster to just pass the SimpleXML object over to the importer rather than
SimpleXML -> XML -> SimpleXML?

Uberdevelopment www.tsd.net.au/blog

Sat, 09/22/2007 - 11:24

#17 View page

catorghans@drup...

Offline

Joined: 08/16/2007

Juice: 72

Re: Re: Re: Re: Can I help?

Didn't think about that one: "handle the id map array".
The "import" modules should be able to access them.

The id-map's could be accessed by functions (accessible by the modules) or passed as parameter(s) to the hooks. I am curious about your ideas.

Tue, 09/25/2007 - 06:59

#18 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Can I help?

Have you started anything yet catorghans?

I'm looking at it now and I'm thing that just passing the ID map array as a param to the hook is the way forward.

I was going to experiment with putting a hook call right before

<?php
if (module_exists('uc_attribute')){
?>

And make everything in that block an example of the hook I'm going to create.

ALso, shoudln't everyting inside the block starting with:

<?php
foreach ($store->attributes->attribute as $attribute_data){
?>

be in the module_exists('uc_attribute') block?

Uberdevelopment www.tsd.net.au/blog

Tue, 09/25/2007 - 17:02

#19 View page

Lyle

Offline

Joined: 08/07/2007

Juice: 6846

Re: Re: Re: Re: Re: Re: Can I help?

Yes, module_exists() tests should be made for uc_attribute, uc_manufacturer, and uc_catalog where they are needed.

Wed, 09/26/2007 - 06:34

#20 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Re: Re: Can I help?

Ah yes, what I was trying to suggest was that they be replaced with module_invoke_all() calls. It seems to me that this is the Drupal way of de-coupling modules and keeping everything loose and independent and also extensible.

Uberdevelopment www.tsd.net.au/blog

Fri, 09/28/2007 - 08:22

#21 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

I've hacked away at uc_impoter.module a little (maybe 6-8 lines worth) to add in two hooks:
hook_uc_import_product which is called fro ever product in the XML and,
hook_uc_import which is called at the end of the XML processing look once.

I've also added a function to this file that implement on of the hooks for taxonomy. I've also done this for the uc_stocklevels module which I will post up under contributions if these hooks are accepted. To get the full idea about the taxonomy hook, the CSV importer (XML converter) defines a function which generates the XML that the 'taxonomy_uc_importer_product' function processes:

<?php
/**
 * Implements hook_uc_import_csv_row for the taxonomy module
 */
function taxonomy_uc_import_csv_row($product_xml, $data, $store_xml)
{
    $property_keys = array();
    foreach($data as $marker => $value)
    {
        // Firstly: Is it one of ours? ...if it begins with 'tid_'
        if(!empty($value) and preg_match('/^vid_/', $marker))
        {
            $vid = array_pop(split('_', $marker));
            $values = array_map('trim', split(',', $value));
            $vocabularies = (!isset($product_xml->vocabularies))? $product_xml->addChild('vocabularies'): $product_xml->vocabularies;
            if(!$taxonomy_list = $vocabularies->xpath("taxonomy[@id=$vid]"))
            {
                $taxonomy = $product_xml->vocabularies->addChild('taxonomy');
                $taxonomy->addAttribute('id', $vid);
            }
            else
                $taxonomy = $taxonomy_list[0];
            foreach($values as $term)
                if(!$taxonomy->xpath("term['$term']"))
                    $taxonomy->addChild('term', $term);
        }
    }
}
?>

I would post up th CSV importer but can't take the strain of supporting it right now as its a bit brittle still.

Attachment	Size
uc_importer.module.txt	47.31 KB

Uberdevelopment www.tsd.net.au/blog

Fri, 09/28/2007 - 14:56

#22 View page

druru

Offline

Joined: 08/08/2007

Juice: 228

Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

post it! post it! )csv importer module) purty please

just put a disclaimer on it "i'm not supporting this until i'm good and ready" .

Sat, 09/29/2007 - 14:18

#23 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

I will, I'm still having some issues with the images. i think I might just handle them myself. I should have something up on Monday.

Uberdevelopment www.tsd.net.au/blog

Mon, 10/01/2007 - 14:41

#24 View page

druru

Offline

Joined: 08/08/2007

Juice: 228

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

thanks cp. i'll look for it when it arrives.

Wed, 10/03/2007 - 13:09

#25 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

Here another copy of my CSV importer.

Issues remaining are:
- Adjustments are not importing correctly
- Images are not importing correctly (well, they did once but it was a fluke).

Not supported:
- Calalog ain't working but if you use my adjusted uc_importer module you can use taxonomy's which is the same thing.
- Manufacturers don't work ether but would be a small function o write if your keen (I'm not).

requires PHP 5 (SimpleXML).

Attachment	Size
uc_import_csv.zip	8.09 KB

Uberdevelopment www.tsd.net.au/blog

Wed, 10/03/2007 - 14:46

#26 View page

druru

Offline

Joined: 08/08/2007

Juice: 228

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

Probably can't look at this until next week but thanks for posting.

Image support is kind of key to this though. Too bad it's not working.

Is simplexml a separate php5 download plus install or does it ship in the core php5 pkg/rpm?

thanks cp

Wed, 10/03/2007 - 12:11

#27 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

Re: Re: Re: Re: Re: Re: Re: Re: Re: Can I help?

Heres a version of the uc_importer.module that includes the hook calls that I was hoping to get merger into the main UC trunk so i wouldn't thave to update this every time the importer got changed.

I've just checkout the latest from bazaar (your still using it?)

Attachment	Size
uc_importer.module.txt	47.88 KB

Uberdevelopment www.tsd.net.au/blog

Wed, 10/03/2007 - 14:09

#28 View page

Lyle

Offline

Joined: 08/07/2007

Juice: 6846

Code Review

I like the idea of the hooks, and searching for nodes by title, but the taxonomy_import_product function needs some cleaning up. The Drupal Coding Standards "strongly encourage" curly braces around control structures, but I think they should be mandatory, especially because I don't put the opening braces on the next line. I got confused by the if(!$found) block in your function, whether it was meant to be included in the foreach before it or not. I suspect it isn't, but I don't want to guess wrongly.

Please make these changes and I'll be happy to commit them to the repository.

Wed, 10/03/2007 - 17:07

#29 View page

CpILL

Offline

Joined: 08/08/2007

Juice: 549

chanegs made and the imagefield upload...?

ya ya, no worries. all braced up now.

On a side note, have you managed to get the 'imagefield' images importing correctly? I was thinking of completely rewriting it as a hook function since the NID would be available then which means a record in the {files} table could be created and the file placed where desired instead of letting it move it to file/images or some such place?

Attachment	Size
uc_importer.module.txt	47.93 KB

Uberdevelopment www.tsd.net.au/blog

Thu, 10/04/2007 - 09:33

#30 View page

Lyle

Offline

Joined: 08/07/2007

Juice: 6846

Awesome. Committed. I think

Awesome. Committed.

I think the image issues have been fixed, unless it's one I haven't been aware of. The last one I remember was the problem with URL-encoded file names. Just make sure that the new site has access to the path given in the <image><path> element so it can save it to it's files/ubercart_images directory.

Mon, 10/15/2007 - 17:14

#31 View page

Lyle

Offline

Joined: 08/07/2007

Juice: 6846

Re: Awesome. Committed. I think

Spent a lot of time improving the importer today. Added a lot of <any> elements to the schema and a similar number of hooks into the importer and exporter. All these changes have been committed, but are still untested. I'll be doing that tomorrow, but if anyone wants to go ahead with that, I'd appreciate it.

Documentation will be updated tomorrow as well.

Tue, 03/18/2008 - 00:18

#32 View page

ronjo84

Offline

Joined: 03/18/2008

Juice: 2

import documentation

Where can we find the documentation and final file?

Thu, 09/13/2007 - 16:23

#33 View page

druru

Offline

Joined: 08/08/2007

Juice: 228

Re: Re: Re: Re: import/export hooks for other modules

Anxiously awaiting anybody that knocks this out first and supplies a good documentation page (hopefully with an example data case including product images) to accompany it.

	User login
	Log in using OpenID: What is OpenID? Username or e-mail: * Password: * Remember me Log in using OpenID Cancel OpenID login Create new account Request new password

Ubercart

Navigation

User login