44 replies [Last post]
univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465

Some background first:
I have been involved in running a paid membership site for few years and have used amember (http://www.amember.com/) for this. While this program is not ideal it does work and handles recurring payments and assignment of roles fairly well. We really needed to upgrade to something more flexible (and something I can actually extend and tweak to suit our needs) and since I am already using Drupal it made sense to give ubercart a run for its money. I am using an unsupported payment gateway so I have already written my own module based on the uc_authorizenet module - and it all basically works.

Having played around in the ubercart code a little now I have run into a few thing I would like changed and was wondering whether there are design decisions I should be aware of or if they are possible areas for improvement.

The issues:

1. Why are recurring payments not treated as a new order?
Advantages of treating a recurring payment as a new order:
  • Each payment gets its own invoice
  • Invoice would then get emailed to the person as normal
  • Other modules like uc_roles could automatically handle the extending the expiration of a role, since you can just invoke the hooks to allow other modules to jump in a do what ever they would normally do on a new order

If there is some reason why its not treated as a separate order now then I guess this could be done as an option ie: in the setting have an option: "treat recurring payment as new order [yes|no]" - although I think it should just be done by default for the above reasons.

2. Rather then have the option in the settings of which module should handle recurring payments, uc_recurring should handle everything and just have it call hooks in the payment gateways that process recurring fees
If a payment gateway exposed the right hooks then it is registered as be able to handing recurring payments otherwise its not. Everything about a recurring fee should first go through uc_recurring which calls the payment gateway responsible for handling that fee.

As an example: uc_recurring_cron - it would just check the order if it was charged by say uc_authorizenet then it invoked a hook provided by uc_authorizenet to process anything it wants to do on the renewal.

This allows ubercart to support more then one payment gateway for recurring payment. ie: I want to give my customers the option of paying for subscription via credit card though my current payment gateway as well as paypal subscription (at the moment i have to choose one or the other).

3. Nothing is setup to handle recurring payments failing?
A payment failing should be registered with uc_recurring and then it handles it in what everway is setup in ubercart. What I would like to happen (and this is how amember works) is that the current order is extended for a few days and an email sent to the member notifying them of the error (so they have a chance to updated expired CC detail or fix what ever the problem is before they are locked out). Ususally you would give the subscriber a two extra attempt so to that on the 3rd try it suspends their subscription.

This probably would also need a function/hook added to uc_roles so it can extending role(s) associated with an order for a set number of days - actually uc_roles should support this by default, if they purchase a role for 1month and then purchase another separately (or though some other product) this should give them 2months access by default.

Related to this I would also like to have uc_recurring check details like CC expiry and if it detects ahead of schedule that a payment is going to fail because credit card details will be out of date at the time of renewal, a reminder is sent to that user to prompt them to update their CC details (this is going to be gateway specific).

4. Trial memberships and showing more information on the display price as to what that price is for...
I would like to have the option to add things like a 2 week free trial and I have seen there is a free product module, but this is not what i need, as I still want to collect payment details and setup up the recurring payment schedule, the easiest way to set this up at the moment is to make the product price $0 and then add the recurring fee, but this is still not idea as then it displays the price as being $0 (nothing about how long until the first payment kicks in).

I think the simplest fix is if there is a recurring payment then in general there should be something shown by default as to when the first payment kicks in, something small under the display price.. so it shows $X.xx (X days/weeks/months)

5. Members being able to changing renewal options? (something thats on my wishlist of features)
With the particular site I am working on members have the option to subscribe on different renewal terms e.g. monthly/6monthly/yearly - the longer you renew the cheaper it works out etc...

I have this setup using an attribute to select month/yearly and it works well, but it would be good to give members the option to change this at anytime or for administrators of the site to change it for a member, so as an example upgrading them from month -> yearly - this is going to be gateway specific as I know you can't change Paypal subscription, but if a gateway supports changing the payment terms it would be good if uc_recurring could handle changing the fee for future payments

I have already implemented some of the above (e.g. recurring payments as new orders) as part of my recurring payment gateway module, but it really shouldn't be in that module, I would like to make these changes in uc_recurring which would then allow me to support multiple payment gateway options (e.g. paypal/credit card) on the subscription products.

I guess I am just looking for comments/thoughts/suggestions before I really hack into the uc_recurring code (particularly if things have been done a certain way for a reason)

Also I am only looking forward so I only intending to work on the Drupal 6.x-2.x codebase.

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: uc_recurring thoughts/ideas...

I have posted a patch here: http://drupal.org/node/426466 that implements the first two points.

mrmeech's picture
Offline
Joined: 03/11/2009
Juice: 66
Re: Re: uc_recurring thoughts/ideas...

This is... Brilliant!!!

Maybe an additional suggestion would be to keep in mind shipping quotes/costs. So, for example: You have a subscription product that is snail-mailed each month, and you'd like to have the cart get the current shipping quote when the order is re-generated with each cycle of the recurring subscription. That made sense, right?? Smiling

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: Re: Re: uc_recurring thoughts/ideas...

I haven't really played around with the shipping modules yet, but in theory as long as they work by adding their stuff through hooks in the order process then it should just work without uc_recurring worrying about it. This is exactly the type of advantage of making a recurring fee a new order that I was talking about.

aaronbauman's picture
Offline
Joined: 04/20/2009
Juice: 6
Curious how you set up frequency attribute

I've got an attribute setup for monthly/yearly as well, but i haven't figured out the best way to tie it to a recurring fee.

At the moment i'm using a cck field in combination with FAPI hooks to select a pre-configured product... but this feels dirty.

Are you using Product attributes for this somehow?

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: Curious how you set up frequency attribute

Setting up different recurring fees for a product is quite easy:

  • under a products attributes you can define different sku for each selection
  • then you need to add a recurring fee for each of the sku's, setting up the different payment amount and intervals to match what your attributes is. This means you will have more then one recurring fee attached to a product, but because each recurring fee is linked to only one sku, when a product is purchased only one recurring fee will actually be used.
bloodylag's picture
Offline
Joined: 04/22/2009
Juice: 2
Re: Re: Curious how you set up frequency attribute

this sounds like exactly what I need.

the store I want to run basically ships out a package for christmas.

So basically the main product has a total price, and the recurring fees should be equal total price divided by how often they want to repay (weekly/monthly).

The thing is that someone who buys the product in January will pay a recurring fee less then a person who purchases the product in April.

Do you think the recurring fees module could handle something like that?

mrmeech's picture
Offline
Joined: 03/11/2009
Juice: 66
bloodylag wrote:this sounds
bloodylag wrote:

this sounds like exactly what I need.

the store I want to run basically ships out a package for christmas.

So basically the main product has a total price, and the recurring fees should be equal total price divided by how often they want to repay (weekly/monthly).

The thing is that someone who buys the product in January will pay a recurring fee less then a person who purchases the product in April.

Do you think the recurring fees module could handle something like that?

How about this: (keeping the math simple) say you offer customers to pay for the product over 5 weeks or 10 weeks, and your product price is $100. For people who want to pay over 5 weeks, create a product that has a cost of $20 and will recur 4 times with a one-week interval ($20 on their checkout, and then 4 more $20 payments). For a 10 week payment, have another product that costs $10 with 9 recurrances with one-week intervals.

If you want to offer people a discount for committing early on in January, make a product with a different price. When you don't want to offer the lesser price, hide the product or make it unavailable for checkout. (I'm not super familiar with UC yet, so you'd have to look into the hiding/unavailable part).

Anyway, this way you might be able to get away with not hacking any code.

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: bloodylag wrote:this sounds

These are all interesting points/ideas and is exactly the type of discussion I was wanting to provoke before attempting to really try and re-work the code - so keep them coming. I also just noticed this post here which has another use-case for recurring payments: http://www.ubercart.org/forum/bounties/10421/google_checkout_recurring_m...

It this stage I'm not actually planning to work on the auth.net gateway myself [$$ could change my mind :)], my intention is just to get the underlying recurring handling process in place so the any gateway that is capable of handling recurring payments just needs to provide their specific interfaces and everything just works - at the moment the auth.net module looks like it has a lot of extra code that really shouldn't be in that module, it should just be concerned with the interface with the gateway.

From my research there appears to be two distinct types of payment gateways that support recurring payments:

1. those gateways that you setup the amount and intervals at the beginning and the gateway takes over and charges each interval (e.g. payal subscriptions)
2. those gateways that store the credit card details for you and allow you to trigger a payment at a specific time.

So an *updated* uc_recurring should be able to support both these types of systems transparently, that means uc_recurring will need to assuming that each payment amount and interval between payments are the same (thats how it is now anyway). So from that we can at least conclude a recurring fee only needs to have:
* next charge date
* fee amount
* interval between (days/weeks/months/years)
* number of renewals remaining
So this at least means that database structure shouldn't need any major changes.

Ofcourse what we could provide is a hook so that other modules can jump in and modify the recurring fee before its about to be charged. That should allow you to create the custom feature of having the recurring fee based on the period over which you are paying off the purchase - ofcourse you would have to write the code to handle that part as I can't see that being part of uc_recurring itself.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
Re: Re: bloodylag wrote:this sounds

I've got a fair bit of experience in the past creating a recurring billing solutions which are flexible and generic enough to work for most users. I've also done a fair bit of work dealing with the issues which arise from Auths/Captures/Voids/Refunds/Chargebacks which arise from moving from one recurring payment to the next. I've also created solutions to assist in aiding the recurring payments to continued to be processed for optimal revenue (probably outside the scope of this initial problem). I also understand what needs to be done, to move all your recurring subscriptions from one processor to another (again, probably outside the scope of this problem).

I'm fairly familiar with Ubercart, but have not yet needed to use recurring payments. I've seen some discussion and complaints about how the current implementation is not robust enough to properly deal with things like roles, so I'm happy to see it taken out of core and worked on.

I'd be interested in helping this discussion along and providing guidance on what a good way to tackle this problem, so that it's can be flexible for extra features in the future. I'll have to review the discussion and code as it stands now though, so give me a couple days.

Consider this a subscribing post.

PS. are there any other threads I should be reading about this discussion to get caught up?

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
Re: Re: bloodylag wrote:this sounds

Ok. I've read over this thread and this thread and I'm ready to provide insight on how I accomplished this in a previous project.

Assumptions:

  • We're ignoring PCI compliance for now and we will store credit cards internally in the database ourselves. This can be abstracted later to be PCI compliant.
  • We will never use the Automatic Rebill functionality in any processor. Instead of will opt to rebill subscriptions ourself.
  • We will need to create front ends for Customers (aka Drupal users) to update their subscriptions information such as update credit card details, update shipping/billing address and cancel subscription.
  • Customers (aka Drupal users) will be created before Orders. This means you can have an anonymous purchase for which a user is created, but no order. We can leave them in a special role and prune them later (think login toboggan).
  • New Order will get created for every new recurring purchase.

Customer Schema Overview:

Customer:
Stores the customer information. Name, email.
Think Drupal user table
Customer Address:
Stores customers address information. Customer can have many addresses, but only 1 current shipping and 1 current billing. A timestamp range is added to the address to determine which were active and when.
Customer will need a page where they can maintain their addresses shipping and billing (uc_addresses could work)
Customer Credit Card:
Stores customers credit card information. Customer can have many credit cards, but only 1 active at a single time. Has timestamp range is added to the credit card to determine which were active and when.
Customer will need a page where they can update their current credit card. This is important for continuing to rebill people who's credit cards will have expired.

Recurring Billing Schedule Schema

Schedule:
ScheduleID, Duration till next rebill, Type of Transaction (Auth, Capture, Auth & Capture), Price, next Schedule ID.
With each schedule essentially having a next schedule ID, it becomes a linked list of schedules. If there's no next schedule ID set, it's the end of the chain and we expire the subscription.
Schedule IDs can set themselves as next schedule ID, which will essentially continue to rebill the same amount until an external source stops this flow.
Customers Schedule
Customer ID, Current Schedule ID, attempts, Timestamp of when this schedule was activated, timestamp of when this schedule will get rebilled.
Upon successful Success customers scheduleID is updated to Current Schedule IDs, next Schedule ID if success. Upon fail, we look for an extension, if found extend rebill timestamp, increment attempt. If no extension, cancel the subscription.
Hook should be called to update users "Role" once a new schedule is activated
Extensions
ScheduleID, attempt, days to extend upon failure.
When a rebill attempt fails, we check the Extensions table to see if we should extend this schedule to attempt to be rebilled at another date. If not, it's a hard fail and we cancel the user.

Processors
Each processor will have to be able to deal with these transaction types.

Auth, Capture, Auth + Capture (not required), Void (release of an Auth), Refund (refund of a capture).

Processors will need to store the required additional data which is returned from a transaction so that they will be able to do things like Capture an already Authorized transaction or Refund a captured amount.

Processor's Cancels, Rebills, Chargebacks
Most processors you work allow your customers to cancel their subscriptions directly on their website (and not yours). Refunds work like this occasionally too. Chargebacks are controlled directly from Visa and you have no control over them. Either way, your processor will always provide you with a method to get these.

*Method #1* Your processing company sends you a postback ala Paypal IPN.
*Method #2* Your processor provides you with some kind of method to download these. Web Service, FTP, Web Scrape at the very worst.

Each processor will have to know how to get this data and process it accordingly so that it's properly dealt with.

While method #1 (realtime postback) are real time, they can be lost in transit. So it's best to implement Method #2 (cron batch pull down) always and allow method #1 to be implemented if real time if required.

Free Trials
3 DAY FREE TRAILS, we've always dealth with as an Auth for the full amount, which a capture 3 days later. The Auth to Capture time period differs between processors, but is usually like 14 or so days. One processor this was short (72 hours) and caused some issues.

Roles
Roles should get updated via a hook when a customer is billed successfully and his schedule updates or when it fails and they are provided with an extension or cancelled.

Processor Independence
Since we're storing all the information and providing all the recurring billing operations ourselves, this allows us to create a broad and shared feature set upon multiple processors. This allows us to not only add recurring billing into processors which do not natively support it, but also allows for features like customizable extensions and future innovations.

It would also allow us to switch our rebilling customers from one processor to the next....say Paypal goes out of business.

RSTaylor's picture
Offline
Joined: 04/02/2008
Juice: 99
j0rd wrote: We're ignoring
j0rd wrote:
  • We're ignoring PCI compliance for now and we will store credit cards internally in the database ourselves. This can be abstracted later to be PCI compliant.

Strongly disagree. Security is the top priority in things like this, and PCI compliance is not an optional luxury if this is to be useful.

j0rd wrote:
  • Customers (aka Drupal users) will be created before Orders. This means you can have an anonymous purchase for which a user is created, but no order. We can leave them in a special role and prune them later (think login toboggan).

I don't understand this. Why would you have a purchase without an order? And what would you be pruning?

I have found one circumstance in custom code where I needed to use a temporary 'anonymous customer' role during the course of the checkout process so that after the account was created, a previously inserted custom database row could be updated with the uid. That sort of trick could be used if needed, but I wouldn't throw out the account or the order.

Other than those two things, this sounds pretty good to me so far.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
j0rd wrote: Customers (aka
j0rd wrote:
  • Customers (aka Drupal users) will be created before Orders. This means you can have an anonymous purchase for which a user is created, but no order. We can leave them in a special role and prune them later (think login toboggan).

I don't understand this. Why would you have a purchase without an order? And what would you be pruning?

I have found one circumstance in custom code where I needed to use a temporary 'anonymous customer' role during the course of the checkout process so that after the account was created, a previously inserted custom database row could be updated with the uid. That sort of trick could be used if needed, but I wouldn't throw out the account or the order.

Other than those two things, this sounds pretty good to me so far.[/quote]

As for users getting created before successful sale, perhaps this isn't required....but I remembered reading something from todda which required this. Also in my system this is how it worked. We had a customer object which would get all the customer data, then associate that customer with the credit card info. We kept the failed attempts in the database as well because we needed to rate limit the amount of those attempts...since those cost money. With out some kind of rate limitation, you could waste thousands of dollars processing unnecessary transactions. Some clients also ran a MaxMind fraud checker and checked the score before running actual transactions.

This is up for debate...and perhaps you could not create user objects for non-successful transactions...but I believe for some/most processors, they will need something temporary to hold information while they see if a transaction is successfully processed or not.

Also there are issues with anonymous checkout in UC 1.x as it wasn't hooked in good enough, so that you can get the UID of a successful new transaction easily enough.

Here's todda's quote from http://www.ubercart.org/forum/development/10138/recurring_billing_discus... , which perhaps he could expand on.

todda00@drupal.org wrote:

The only thing that I see as a step backwards from the core ARB is not having the ability for anonymous checkout. This is because the CIM information has to be linked to an account, and in the checkout process, the account isn't created until after the CIM code has to run. I was able to modify the CIM code without touching the core Ubercart to create the order during the CIM process, but the problem with that is that then Ubercart thinks it is an existing account and reports to the user that it found its existing account. It works, but is hackish.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
3rd Party Hosted Payment

3rd Party Hosted Payment Solutions

Paypal and Google checkout, while these are very popular options for most people.... are abnormalities in the "Online Processor World" and we can not use these in the debate. I believe since these are soo different than how the majority of processors work on the internet, they will need to be dealt with as special cases. I believe since any hosted payment solution is so unique, they will need to get dealt with on a case by case basis.

Payment Solutions where you send the card info, instead of forwarding the user to 3rd party website.

This is the majority of other online processors and where you're going to be able to abstract a lot of code. Most of these are small mickey mouse operations with minimal feature sets. They simply provide you with an API which you integrate and toss the credit card numbers too and they process them. Having a processor which you can store credit card numbers on, in my experience, is the exception to the norm. Very few even provide you with a method to rebill your customers. If you expect to have these features for "future supported processors" you will quickly figure out, you will not be able to support most processors, since they do not have these features. In my opinion, you should be going for the lower common denominator feature set for all processors, and then allow ubercart to build up functionality around this common feature set. Any additional features which specific processor can support (storing rebilling information, storing credit cards) should be an extension for that processor...but Ubercart should be able to provide rebilling for processors which do not support it internally, this will require ubercart to store credit cards internally for a processor which does not support storing on the processors servers.

Think of all the functionality Ubercart could provide on top of a processors feature set, by dealing with the majority of processing itself. Also keep in mind that any extension of these features, would then be shared by all processors which Ubercart supports. Believe me, this is a powerful and useful bonus of dealing with things yourself. You'll actually find when implemented it like this, that it simplifies things, rather than makes the problem harder, because you can ignore the differences between processors and instead simply use your time to focus on what they have in common.

Also if you rely on a processors feature set, for say, rebilling. Then you also need to rely on their feature set for cancelling. Say one of your admins deletes a user which has a rebilling subscription on your site. You will need to contact the processor to cancel his recurring subscriptions (which is difficult and messy) instead of simply changing a field or two in your rebilling database. This gets messy really quickly and in my opinion again, is best internally dealt with.

I also if we're dealing with most things ourselves, we will need to down chargeback, refund and cancel information and cancel subscriptions, otherwise you have accounts which will continue to get billed, which shouldn't. This is important and should not be neglected as it will bring chargebacks to your account. I of course believe this can be optional, but should warn the admin if this "hook" is not implemented for his processor. Dealing with his yourself, is a lot simpler than having to deal with all the other issues which arise from having a processor deal with your subscriptions. Also 90% of processor support batch downloading of this information and if they don't processors I've worked with are able to created and provide you with a web service to pull the info down in a couple of days, as it's simple and they understand why it's important to you.

Another good reason to deal with the majority of the functionality yourself is that many processors charge you different amounts of "Levels of Integration", which essentially boils down to, you get charged a different amount for the feature set they provide for you. The lower you can get, the cheaper these solutions become. Some processors will charge you more, say if you want to use their recurring billing and cancel features.

If you rely on your processor to deal with all your credit card information and you run a subscription site...what do you do, once that processor no longer exists. Or what do you do, when say your merchant account gets too many chargebacks (with mastercard, that's 2%) and you are kicked off. Or what do you do, when you find a cheaper processor which will save you thousands of dollars a month in fees. The answer is nothing, if you've relied to heavily on the processor you're currently dealing with. Believe me, try going to a processor and asking for the stored credit card numbers, so that you can move off of them and onto a cheaper processor. Not going to happen.

What's the best way to make everyone happy?
Some processors support some features and some processors don't. In my opinion, you should chose the lowest common denominator of options most processors will support and create all the other functionality you need yourself. This functionality will be the default fall back for processor which do not support internal processor replacements for that functionality.

Ideally, we'd create a "shell module", which you copy and paste and fill in the functionality. Hooks will be used this.

  • Auth
  • Capture
  • Refund
  • Void
  • Store Credit Card as Key (optional, store internally in PCI complaint method)
  • Some kind of hook for rebilling (optional, use your own schedules)
  • Batch Download Cancels, Chargebacks, Refunds (optional, warn admin if not implemented)
  • Real Time Sample Menu Hook for Postback cancels, refunds, chargebacks (optional, warn admin if not implemented)

This is essentially the base feature set. Recurring functionality can get created from this. Extensions can get created from this. A bunch of other features can get created from this and once they are, they are shared between all processors supported by ubercart.

Another bonus is, that while some processors will support a bunch of other features in their API, they can essentially be ignored and only the lowest common denominator base functionality will be all that's required to get a processor working with Ubercart. This will allow to get a processor up and running in ubercart as quickly as possible. It can of course be extended after that to provide all the bells and whistles that processor supports internally (store credit cards on their server, use their rebilling methods).

When a admin installs a processor on their ubercart, they should be provided with a checklist of features that processor supports. Not all processors will support captures, so the admins should be warned about this. Not all processors will allow you do store credit cards on their servers. Not all processors will allow you to store rebill information on their servers. For Processors which do support "Extended Optional Features" we should allow the admin to choose if they want to use them or not. If they chose to opt out of storing their credit card information on Authorize.net's servers, and they can instead store them internally in Ubercart using a PCI complaint method. Perhaps we should default to allowing the processor to deal with as much as possible, or as little as possible. I don't care which one as long as I can choose.

Schedule Linked Lists
Essentially this is the most complicated schedule queue i've seen, but creating the architecture in the way I've discussed allows it to be extended very easily and hooked well into things like extension and discounts.

Scenario #1

      1. 3 Day - Free Trial ($30 Auth) | Next Schedule #2
      2. First Month ($30 Capture) | Next Schedule #3
      3. Second Month ($30 Auth + Capture) | Next Schedule #3

Then you create a product and associate it with Rebilling Schedule #1.

Scenario #2
Say you have a physical product, which gets purchased once then ships a new suply every 30 days. Schedule would look like this.

      1. 30 Day Supply ($30 Capture) | Next schedule #1

Failed Rebill Extensions

A client has a 30 day subscription which failed to bill. You extended him 3 days and tried again, failed. You extended him another 3 days at which point it was successful. Do you give him 30 days from the current date you've been able to bill him? Or do you give him 30 days from the time his subscription expired?

Expired is the most accurate answer, but this also increased chargebacks for our clients. Customers would see perhaps 2 charges in the same month maybe 2 weeks apart. We had to make this feature of the extensions customizable, with most clients choosing the later less correct option to avoid chargebacks.

bearstar's picture
Offline
Joined: 05/08/2009
Juice: 41
Keep going!

I am excited by your discussion. Please keep on with the progress in enhancing uc_recurring module.

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Hi j0rd, I don't feel we can

Hi j0rd,

I don't feel we can just ignore paypal/google or other gateways that have built in features to take care of recurring billing. As you said they are one of the more popular options - which I would guess is considered a lot by those just starting out with a particular business as they are cheap to setup.

The fact is that ubercart is not an enterprise level ecommerce system and in many cases it is installed on shared hosting by people with little technical background (which is a recipe for all sorts of security concerns that I am not sure can all be addresed within the software itself). Ubercart does have the test_gateway that stores card details already - anyone with the technical understanding of security to manage credit card details themselves should be able to implement a solution that stores credit card details and mange everything themselves with little difficulty from that module - and we could put together somewhere in some docs a recipe for those wanting to do this with uc_recurring with some references to PCI compliance issues that need to be considered - ie: make it easy to do for those with the technical skills, but hard for those that don't (a good example of a non-technical person blaming ubercart for a security compromise - imagine the same person haveing a 1000 credit card details stolen and having the merchant providers/banks on their butt).

I think we should be able to solve the problem for both types of gateways with the one module. uc_recurring should implement solutions for a lot of the functionality (that we need) so it can manage everything itself if required (e.g. canceling, refunding etc...) - we can use the test_gateway as the test gateway to implement all these features. But then I would like to see the ability to override the built in functionality with a payment gateway's own implementation or disabling that feature completely if a gateway doesn't support it. The current solution for adding other features in uc_recurring was the "fee_ops()" callback that a gateway could implement. We need something like this that a gateway can expose what feature are handled by uc_recurring(ubercart), what are handled by the gateway and what are disabled.

I guess one issue is that this it would require a lot of internally checking to make sure that enabling/disabling gateways or adding/editing billing schedules is allowed by the user - you don't want to allow a complex billing schedule if the gateway(s) enabled doesn't support that option or allow you to disable one of your gateways that allows all the complex billing options that are setup without warning the user somehow. As for things like cancel subscriptions when deleting an account that should be able to be handled though a hook (if ubercart can't cancel the subscription from ubercart then you can't delete the account and are warned to cancel that first - I think I would want to be warned that a subscription is active before deleting an account anyway)

I am definitely excited by the schema you have described and looking to take on some of those suggestions as that really would expand the possible functionality of uc_recurring a lot, especially the schedules/extension schema's.

The only thing I would add is that for the solution I am suggesting to work well it would be more ideal if the payment gateway architecture was all object oriented code. I have been thinking about a lot of the issues above and I keep thinking that this would be so mush easier if the gateways where all written in OO. I posted the following to see if anyone else is thinking about this also - http://www.ubercart.org/forum/ideas_and_suggestions/10771/object_oriente...

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: Hi j0rd, I don't feel we can

Actually one problem with the your customer schedule schema. If you make a change to a products recurring billing you don't necessarily always want to force all existing customer onto the new schedule.

So you either need the software to create a new schedule everytime you update the recurring fee (if there are exist orders with that schedule) or create a new product.

Ie: you can't easily just create an offer signup before X date and you only pay $X per month after that it is $Y dollars per month.

Am I correct or have I missed something in the schema?

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
Re: Re: Hi j0rd, I don't feel we can

Schedules are independent of products. A Product get associated with an "initial schedule". It's the actual schedule queue (each schedule item, points to the next schedule item), which dictates the interval and amount of billing. So if you change the initial schedule of a product, all new sales will go on that queue. All old customers will continue to get billed on their existing "customer schedule"...since that queue change is not updated.

A schedule which contains no "next schedule id", is an end point schedule and after that we don't attempt to rebill. This is the end point of the queue.

If on the other hand you decide to change a schedule item's amount, duration or next schedule ID...this will essentially change this flow for all customers who's schedule queue will eventually fall on this item.

A every customer is associated with their current schedule, next schedule and when they will get billed on their next schedule. If you change "next billing schedule", this will dictate for that customer the rebilling queue they will attempt to get billed for next.

With this schema you can edit the recurring "flow" in 3 parts.

You can change a products initial schedule, which will put all new sales onto that schedule. Does not update existing sales (unless you wanted it too).

You can modify an item in a schedule queue, in which will essentially change how all customers on that queue will get rebilled. I would on the other hands recommend, creating a new schedule and inserting it into that queue.

You can change a customers current schedule, which will dictate how that customer will get billed. We created "Upgrade Now" or "Discounted Month" entry points, which would disregard a customers current schedule, attempt to bill him on another one and if successful, update that customers current schedule...if not, display error message and leave him on old schedule / queue.

Think of 3 day free (but limited access) trial, in which you provide an entry point to allow that user pay his monthly fee immediately and be provided with full access.

Let me know if I can clarify further.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
univate@drupal.org
univate@drupal.org wrote:

Ie: you can't easily just create an offer signup before X date and you only pay $X per month after that it is $Y dollars per month.

This is essentially two different products. Or a single product which two price points, which Ubercart doesn't allow (with out modules).

Ignore recurring billing. You sell an apple for $5 before april. After april this apple is $10. In Ubercart after april you would have to edit your product and change the price.

You could do this with my schema, by updating the initial schedule of a product. All new successful billings would get this new initial schedule. All existing customers would continue to get billed as dictated by their "customer schedule" entry.

Customer schedule holds that customers current schedule and rebill date.

Each schedule holds a next schedule, interval and amount, which essentially creates a schedule queue.

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: univate@drupal.org

Ok, that all makes sense and is basically how I was reading it.

So another couple of examples, I have seen posted on these forums.

1. Recurring donations:
The problem is that someone would like the person to be able to specify how much they want to donate and how often (probably selecting from weekly or monthly etc...), that would require a new schedule for each purchase, correct?

If this is an ongoing renewal thats only requires one entry in the schedule table (since it links to itself) for each purchase, but if it was weekly for say one year then that would be 52 entries in the schedule table for every purchase.

2. Christmas hampers:
$240 Product to be delivered in December, it is paid off monthly, but depends on what month you purchase as to the price, so a person pays $20 per month if they purchase in Jan, $21.82 in Feb, $24 Mar etc... obviously this requires some custom code to workout the price the person pays at the time of purchase, I calculate about 78 entries in the schedule table for that one product.

Also the extensions table suggests you need to setup extensions for every schedule id. So say I want to have 2 attempts to rebill a failed payment that adds another 150+ entries to the extension table for that one product. I think I would prefer to have a default extension option on any failed recurring payment and then be able to override that if I need to.

Of course there is also the one entry in the customer schedule table for each purchase to track where they are in the schedule.

Finally the only way to work out how many payments are left is to work your way along the linked list and count them.

My opinion:
I think your solution is elegant for certain problems where the amount and intervals change for each payment on a fixed schedule but for (what I would assume are) more common recurring problems where the amount and intervals remain the same but may be different for each order it make the solution rather complex.

I subscribe to the idea that you should come up with the most simple solution for most of the cases in a problem and add extra complexity to handle less common cases.

I feel that the best way to handle your variable amount & interval subscriptions is to provide a hook in uc_recurring at the renewal step to allow another module ('uc_recurring_advanced') to jump in and modify the amount and length for the next subscription fee based on a more complex schedule schema like you have described.

I have also never really thought of SQL being able to handle linked lists, I'm no database guru but is there a simple query that you can pull out an entire schedule or count how many are left. I think you have to create a separate queries for each step along the way, so a weekly subscription for a year would require 52 queries to dump out the whole schedule to view/modify it.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
QUOTE TAGS ARE BROKEN! My

QUOTE TAGS ARE BROKEN!

My schema was designed around recurring schedules for site subscriptions, which continue to be rebilled until canceled. There was no thought around physical goods or donation schemes. You've brought up interesting schemes here, which do poke holes in what I've outlined.

Quote:

1. Recurring donations:
The problem is that someone would like the person to be able to specify how much they want to donate and how often (probably selecting from weekly or monthly etc...), that would require a new schedule for each purchase, correct?

If this is an ongoing renewal thats only requires one entry in the schedule table (since it links to itself) for each purchase, but if it was weekly for say one year then that would be 52 entries in the schedule table for every purchase.

I feel this case could come up quite a bit and it would be easy to extend the current schema to deal with it.

Currently weekly donations for a year, this would require 51 "7 day, bill again in 7 days" and 1 "7 day, do not bill again" schedules. It would also require 1 queue of subscriptions for each price point.

The way I've dealt with more complex transactions, is by adding logic just before the code, which took the customer and billed him an amount. In Drupal we could do this with Hooks.

Just before you're billing a customer his amount, check to see if it's 51 billing. If so, do not bill him, but set his next schedule to "7 day, no rebill".

Something like this could be done to change the price point as well.

Otherwise we should rethink my schema, as this scheme will get hackish quickly and multiple hooks into this area of code could mess with each other if they're not provided in core by default.

Quote:

2. Christmas hampers:
$240 Product to be delivered in December, it is paid off monthly, but depends on what month you purchase as to the price, so a person pays $20 per month if they purchase in Jan, $21.82 in Feb, $24 Mar etc... obviously this requires some custom code to workout the price the person pays at the time of purchase, I calculate about 78 entries in the schedule table for that one product.

Of course this breaks what I've provided, but I feel it's a small case. Could you show me a 3 site which provide this functionality Laughing out loud Or any recuring billing solutions which do. Speaking of which, are there perhaps other open source projects, who's recurring billing solutions we should be looking at?

Quote:

Also the extensions table suggests you need to setup extensions for every schedule id. So say I want to have 2 attempts to rebill a failed payment that adds another 150+ entries to the extension table for that one product. I think I would prefer to have a default extension option on any failed recurring payment and then be able to override that if I need to.

I provided multi-levels of overrides with a method that worked, but I'm not to fond off.

Extensions Schema:
ScheduleID, rebill attempt, days to extend.

In the code:

  1. I took a scheduleID and queried over the extensions table for that scheduleID and rebill attempt.
  2. If it didn't exist...then I checked for that scheduleID with rebills = 0 (default extension for all rebills).
  3. If that didn't exist, then I looked for scheduleID = 0 (default) and rebill attempt = rebill attempt.
  4. If that didn't exist, then I looked for scheduleID = 0 (default) and rebill attempt = 0.
  5. If that didn't exist, we didn't extend

Of course if I did find something, then I used it. You can do it in 1 database query, and then use some code logic if you want to minimize queries.

As mentioned, I wasn't a huge fan of this schema, but I wrote the end user a decent front end, which dealt with all the gross backend for it. Then on each schedule load, I wrote something, so that the proper extension was loaded up. It worked, but perhaps could be made more elegant.

I used this default system (0 in IDs) a couple other places too. Again, not ideal, but worked.

Quote:

Finally the only way to work out how many payments are left is to work your way along the linked list and count them.

Yep.

Quote:

My opinion:
I think your solution is elegant for certain problems where the amount and intervals change for each payment on a fixed schedule but for (what I would assume are) more common recurring problems where the amount and intervals remain the same but may be different for each order it make the solution rather complex.

I subscribe to the idea that you should come up with the most simple solution for most of the cases in a problem and add extra complexity to handle less common cases.

I feel that the best way to handle your variable amount & interval subscriptions is to provide a hook in uc_recurring at the renewal step to allow another module ('uc_recurring_advanced') to jump in and modify the amount and length for the next subscription fee based on a more complex schedule schema like you have described.

I believe I described above what you're getting at here. Much of the more robust features (extension, discounts) in my code were provided this way, by adding more logic. We had the idea of "Handlers" in our code, and there were things like "Rebill Handler", where we could add extra logic to perform more robust features. In drupal we call these hooks.

Quote:

I have also never really thought of SQL being able to handle linked lists, I'm no database guru but is there a simple query that you can pull out an entire schedule or count how many are left. I think you have to create a separate queries for each step along the way, so a weekly subscription for a year would require 52 queries to dump out the whole schedule to view/modify it.

Perhaps there's a better way to design the table to provide this functionality. Since I believe this is a common issue, google (or a guru) should be able to provide us with the optimal solution.

Since this table should be small for most people, I would recommend perhaps pulling down the entire table in 1 query and caching it. Using code logic to build the queues as needed. This would avoid database hits.

Otherwise something more elegant would have to be thought up.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
Re: QUOTE TAGS ARE BROKEN! My

The method I've provided was created in house, to solve the problems our clients had. I don't believe we researched at all.

I would be curious to hear how other open source carts are doing their recurring billing solutions.

Anyone want to provide an overview of what recurring schemes other open source carts provide and then perhaps we can review the pros/cons of the way they're solving those problems. Perhaps getting on contact with some of their devs about what they've come up with and how they would improve them if they could start from scratch.

Edit: Quick search of oscommerce, magenta, zen cart turned up, pretty much nothing on the subject. If it was supported, appears it's on a processor by processor basis with contrib modules. Not supported in an overall "core".

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: Re: QUOTE TAGS ARE BROKEN! My

Yeah I haven't found any good open source recurring billing solutions anywhere either.

My vision for uc_recurring is that is should be be a basis for all types of recurring payments and not just limited to site membership/subscriptions. My previous two example cases were problems I had seen posted on these forums, they may or may not be common problems and I wouldn't expect uc_recurring written to solve just one of these problems at the expense of other more common recurring problems.

The only other product I have used for recurring billing is amember (amember.com). Its not open source, but for simple site subscriptions it actually works reasonably well.

Its schema is based on the assumption that each subscription has a fixed amount (except for a trial amount field) and fixed intervals. Each subscription is closely tied to a product (since it assumes each product is a subscription).

With ubercart the current uc_recurring schema looks like this:

uc_recurring_products - defines a recurring fee schedule (it isn't really a product recurring fee as you can define more then one recurring fee on each product.
- pfid - unique ID, this is essentially the schedule ID
- model - SKU that this recurring fee is for
- fee_amount - what is charged each interval
- initial charge - when the first payment occurs
- regular_intervals - how often to rebill
- number_intervals - how many times to rebill

uc_recurring_users - when a customer places an order a recurring fee is setup for that user so we can rebill them.
- rfid - unique ID
- uid - user ID so we can link this to a drupal user account
- fee_handler - what payment method/gateway is handling this fee on renewals
- next_charge - when should this be charged next
- fee_amount - how much should be charged on next renewal
- remaining_intervals - how many more charges are due (we need add something like use -1 to give us UNLIMITED charges)
- regular_intervals - when this is charged what should we add to the next_charge date field for following renewal
- charged_intervals - how many charges have occurred already
- order_id - what order does this fee belong too
- data - somewhere to chunk data related to the specific payment gateway - e.g. with paypal it might be the subscription id
- created - when was this recurring fee setup

I think this current schema does account for most recurring problems, the only thing that we might want to add is an advanced schedule table

My first attempt at thinking throught what can be added to improve the recurring billing process/features:

uc_recurring_schedule - this will allow you to override any specific recurring fee
- sid - unique schedule ID == pfid
- interval_num - this is the id of the recurring fee interval number that we want to effect
- fee_amount - amount to charge on the specific interval
- next_interval - what should the next_charge date be after we charge this schedule payment
- rfid - either NULL to effect all payments or if specified we can override the schedule for just one specific user

We also would need to add sid field to the uc_recurring_users table. But that table would allow all sorts of recurring billing schedules.

For the extensions, I think your schema should do the job:
uc_recurring_extensions
- sid - schedule ID
- rebill_attempt - not sure using the default 0 would make sense as you could get in a case where you extend forever if you are not careful, although that may be a issue to worry about at code level rather then DB.
- time_to_extend - rather then days to extend, i think we should use time so we can just add to next_charge field which is already in the unix time format (if 0 we cancel instead of extend)

I think a global extension setting should just be stored in the drupal variables table.

Also need to add field to uc_recurring_users table for # attempt, so we can match with the rebill_attempts field on failed payments.

j0rd's picture
Offline
Getting busy with the Ubercode.
Joined: 07/16/2008
Juice: 452
Re: Re: Re: QUOTE TAGS ARE BROKEN! My

I found this discussion on the other Rebilling Discussion Thread.

Quote:

This really depends on the payment gateway you want to use to process payments. But I am looking to provide this functionality with some hooks in uc_recurring that give your own module a chance to jump in during the renewal process so you can modify the new order for that renewal schedule.

But that will only work on payment gateways where you have the ability to modify the amount each payment interval.

I've already voiced my concerns with forcing users to use certain processors extended functionality, since I feel it's important, I will voice this again.

In my opinion it's best to allow Ubercart to deal with everything (store credit cards in a PCI complaint manor, handle rebilling) and optionally (but by default) offload this task to the processors. This will allow any processor which can bill a client when you send their credit card information the ability to provide any rebill schedules / payment schemes Ubercart can create. This will include unified extensions schemes between all processors. Unified failed rebilling attempts between all processors. Unified discount scheme between all processors. List goes on and on.

This also offloads the users dependence on a particular processor and allows them to move off of it, should it make good business sense for them. Find a cheaper processor, kicked off processor because of chargebacks, processor goes down for 3 days.

Not sure if you plan on doing this, but from your last comment it seemed to indicate that if a processor didn't support discounted rates at rebill time, that his processor wouldn't be able to support this feature.

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: Re: Re: Re: QUOTE TAGS ARE BROKEN! My

There is always going to be a "depends on the payment gateway" condition with uc_recurring, as some people don't realize that you can't just take a standard paypal account (or any hosted payment gateway) and bill your users what you want and when - some people need to be told this. Also when I said "payment gateway" in that post, I am really meaning the the "payment gateway module"

There are plenty of payment gateways that provide great functionality to store credit cards details for you and let you bill that card whenever you need to without you needing to worry about the extra headache of the security of those details - so people (including myself) are going to make use of these services where they are suitable and it makes sense to include them in a system that provides recurring billing solutions.

Heres the issues is I see them:

  • Ubercart itself can never guarentee that it is PCI compliant, because PCI compliance also requires conditions on the hosting environment, encrypted communication (e.g SSL) and also depends on the volume of transaction you are performing. So its always going to depend on the person setting it up to ensure that the system is compliant.
  • PCI compliance is not the only requirement on online merchants, at the end of the day its the bank behind your merchant account that has to foot the bill for fraudulent transactions and I have come across certain requirements from banks about what they expect from you using their merchant account (and over the last few years they only seem to be adding to these requirements including requiring audits of system that are storing credit card details).
  • Advertising (by providing all the code) exactly how credit card details are encrypted and stored - is an invitation for attackers on ubercart stores.
  • Ubercart is used mostly by small to medium business, who in the process of saving startup costs will look toward solutions like paypal/google for their first payment gateway option because of the cheap setup costs - so ignoring them doesn't make sense. Many of these users are also non-technical so will potentially overlook basic security requirements of an online store - so for the rest of us that want to buy their products/services our details are always going to be safer if they use gateways like paypal.

Please correct me if I'm mistaken about any of these issues...

What I 100% agree with is that uc_recurring should provide a layer to handle all the common recurring stuff (orders/payments/renewals/extensions etc.) in a standard way and only goes to the specific payment gateway(s) for the bits they need to perform.

If you want to store credit card details yourself then this requires a certain level of technical knowledge to ensure you consider and address all the security concerns that this involved. It only requires the code in test_gateway combined with a simple wrapper function around any current payment gateways charge function to have this solution now.

If you want to use paypal or something like this, you still should be able to since they support recurring billing as well, but the condition is that you need to realise that you may have a subset of features available to you then if you used another gateway.

RSTaylor's picture
Offline
Joined: 04/02/2008
Juice: 99
j0rd wrote:In my opinion
j0rd wrote:

In my opinion it's best to allow Ubercart to deal with everything (store credit cards in a PCI complaint manor, handle rebilling) and optionally (but by default) offload this task to the processors.

Being able to take your data and use it anywhere, in the way that you want regardless of the featureset of the processor, is a very worthy ideal. I don't disagree at all with that, I'd love to see it.

However, when that data is customer credit card info, I think that there's a potential for problems that has to be dealt with.

The fact that some processors are 'write-only' for some of this data is a limitation, but it means that neither a site developer, nor an employee, nor someone who hacks the site can get the card info once it's been submitted. They could potentially run up charges on it, but all such charges would presumably pass through one site, one host, and one processor, making it much easier to catch, halt, and resolve a problem than if the CC info got out into the wild.

(Of course, if someone can get in and catch the data as it's being submitted, that's a different problem. Likewise if a processor gets compromised.)

But for payment info on file for recurring billing, what would be as safe to the site owner and/or customer as write-once to a processor? What method would allow you to move the data to another processor and use it how you like, without allowing access even to insiders? Anything on-par with, for example, Authorize.net's security?

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
Re: j0rd wrote:In my opinion

On the subject of storing credit cards, this person seems to know what they are talking about:
http://www.ubercart.org/comment/39492/PCI_not_difficult

univate@drupal.org's picture
Offline
Getting busy with the Ubercode.
Joined: 03/27/2009
Juice: 465
j0rd wrote: We're ignoring
j0rd wrote:
  • We're ignoring PCI compliance for now and we will store credit cards internally in the database ourselves. This can be abstracted later to be PCI compliant.
  • I have to admit to not analyzing the PCI comliance requirements in their full detail, but my understanding is that you can store credit card details and still be compliant, there is just other things you need to do to ensure you protect those details - using firewalls, stored using strong cryptographic hashes etc... The thing is that easiest way to remain compliant is to not save credit card details yourself and I would think it would be irresponsible for an open source project like ubercart that is often installed by end users who aren't fully aware of all these issues. This is the exact reason alot of gateway provide these rebill features

    j0rd wrote:
  • We will never use the Automatic Rebill functionality in any processor. Instead of will opt to rebill subscriptions ourself.
  • There are services out there that allow you to have full control over the re billing amount and date - the ideas is that ubercart recurring solution should be able to deal with both types of systems.

    j0rd wrote:
  • We will need to create front ends for Customers (aka Drupal users) to update their subscriptions information such as update credit card details, update shipping/billing address and cancel subscription.
  • That is possible to do if the specific gateway supports it, obviously if you choose to use a gateway like Paypals standard subscriptions they don't offer those features. But others can.

    j0rd wrote:
  • Customers (aka Drupal users) will be created before Orders. This means you can have an anonymous purchase for which a user is created, but no order. We can leave them in a special role and prune them later (think login toboggan).
  • New Order will get created for every new recurring purchase.
  • A solution to provide this feature has been committed to uc_recurring 2.x

    j0rd wrote:

    Customer Schema Overview:

    Customer:
    Stores the customer information. Name, email.
    Think Drupal user table
    Customer Address:
    Stores customers address information. Customer can have many addresses, but only 1 current shipping and 1 current billing. A timestamp range is added to the address to determine which were active and when.
    Customer will need a page where they can maintain their addresses shipping and billing (uc_addresses could work)
    Customer Credit Card:
    Stores customers credit card information. Customer can have many credit cards, but only 1 active at a single time. Has timestamp range is added to the credit card to determine which were active and when.
    Customer will need a page where they can update their current credit card. This is important for continuing to rebill people who's credit cards will have expired.

    updating credit cards comes down to the gateway supporting this feature - I have already implemented this with my uc_securepayau gateway - my customers using this gateway will be able to change their CC details.

    j0rd wrote:

    Recurring Billing Schedule Schema

    Schedule:
    ScheduleID, Duration till next rebill, Type of Transaction (Auth, Capture, Auth & Capture), Price, next Schedule ID.
    With each schedule essentially having a next schedule ID, it becomes a linked list of schedules. If there's no next schedule ID set, it's the end of the chain and we expire the subscription.
    Schedule IDs can set themselves as next schedule ID, which will essentially continue to rebill the same amount until an external source stops this flow.

    very interesting schema (allows for all sorts of billing schedules - I like), do people actually need this flexibility though, (I know I don't, my recurring billing are always the same amount over some regular interval). Another problem is that some gateways don't support this and I would like to be able to setup a site with more then one payment option and let the user choose which gateway they want to use, this just restrict what I can provide to the customer. If the only way they will do business with me is by using Paypal where their details from keep from me, I don't want to refuse that business.

    j0rd wrote:
    Customers Schedule
    Customer ID, Current Schedule ID, attempts, Timestamp of when this schedule was activated, timestamp of when this schedule will get rebilled.
    Upon successful Success customers scheduleID is updated to Current Schedule IDs, next Schedule ID if success. Upon fail, we look for an extension, if found extend rebill timestamp, increment attempt. If no extension, cancel the subscription.
    Hook should be called to update users "Role" once a new schedule is activated
    Extensions
    ScheduleID, attempt, days to extend upon failure.
    When a rebill attempt fails, we check the Extensions table to see if we should extend this schedule to attempt to be rebilled at another date. If not, it's a hard fail and we cancel the user.

    This is something that desperately needs to be implemented in uc_recurring and I appreciate the ideas suggested here, extensions are not only useful for failed payments, they can be useful for doing things like adding an extra month to a subscription (to compensate a unhappy member or maybe give a prize for a competition - free month etc...)

    j0rd wrote:

    Processors
    Each processor will have to be able to deal with these transaction types.

    Auth, Capture, Auth + Capture (not required), Void (release of an Auth), Refund (refund of a capture).

    Processors will need to store the required additional data which is returned from a transaction so that they will be able to do things like Capture an already Authorized transaction or Refund a captured amount.

    Processor's Cancels, Rebills, Chargebacks
    Most processors you work allow your customers to cancel their subscriptions directly on their website (and not yours). Refunds work like this occasionally too. Chargebacks are controlled directly from Visa and you have no control over them. Either way, your processor will always provide you with a method to get these.

    *Method #1* Your processing company sends you a postback ala Paypal IPN.
    *Method #2* Your processor provides you with some kind of method to download these. Web Service, FTP, Web Scrape at the very worst.

    Each processor will have to know how to get this data and process it accordingly so that it's properly dealt with.

    While method #1 (realtime postback) are real time, they can be lost in transit. So it's best to implement Method #2 (cron batch pull down) always and allow method #1 to be implemented if real time if required.

    this is more directly related to individual payment gateway modules and getting them to implement all these features.

    j0rd wrote:

    Free Trials
    3 DAY FREE TRAILS, we've always dealth with as an Auth for the full amount, which a capture 3 days later. The Auth to Capture time period differs between processors, but is usually like 14 or so days. One processor this was short (72 hours) and caused some issues.

    It's possible to do this with the current system, if you set the product price to zero and then set the recurring billing to start 3 days later. But I would like to see a better way to setup trials, as the way described is a little clunky and doesn't allow you to work with roles.

    j0rd wrote:

    Roles
    Roles should get updated via a hook when a customer is billed successfully and his schedule updates or when it fails and they are provided with an extension or cancelled.

    this is achieve without any work by just making a recurring billing an order - uc_recurring doesn't need to know that roles need to be created, by just creating a new order for the same product you get everything that is part of the originally product e.g. roles, shipping etc... although I think we are going to be forced to add in some link between a recurring billing and a role, as there are limitations like described above with trials. Actually a better way is what if the uc_roles had an option that allowed it to get the period to renew for from the order

    j0rd wrote:

    Processor Independence
    Since we're storing all the information and providing all the recurring billing operations ourselves, this allows us to create a broad and shared feature set upon multiple processors. This allows us to not only add recurring billing into processors which do not natively support it, but also allows for features like customizable extensions and future innovations.

    It would also allow us to switch our rebilling customers from one processor to the next....say Paypal goes out of business.

    This is definitely an ideal thing to have, but I'm not sure we should be implementing a solution that adds alot of work to ensure it is compliant - does anyone know what the implications of not being PCI compliant are? I am assuming this this compliance process is a way that credit card providers are going to be able to put the blame (and use any legal methods to go after compensation) on merchants whose systems are compromised and did not go to any reasonable effort to ensure they were going to all efforts to protecting sensitive details or can at least trace back to who was accessing the system at the time of compromise.

    j0rd's picture
    Offline
    Getting busy with the Ubercode.
    Joined: 07/16/2008
    Juice: 452
    Re: j0rd wrote: We're ignoring

    Again. I've actually created this solution before and I know my method works.

    Once you start to rely on certain features for processors, which other in the future will not support, you solution becomes messy and broken. It's best to pick the simplest feature set, which you can assume all processors pretty much support and implement all additional features yourself. We essentially created a processor you signup with once and under us, we supported about 15 underlaying processors, merchants and bank interfaces. We also allowed our customers to switch all their clients from 1 processor to another, should say they find a cheaper one with less costs.

    My solution again is only for processors for which you control the payment page. Standard processors which you send over the credit card information and they bill the customer. Other processors which use hosted payment pages like Paypal, Google Payment, will not work with the method described and since I'm not familiar with them, I can provide you with a method to abstract these. I assume they'll be coded one by one and support what ever feature set they support. This will be fine for users I believe. The bulk of processors on the internet, allow you to use hosted payment pages.

    Seems the biggest gripe people are having with my solution is of PCI compliance. I would not recommend being non-PCI complaint, but for the sake of simplicity, I've left this part out from my post. My solution will require you to have access to the credit card numbers and data. You will need this data to create the recurring billing functionality, with out relaying on the processor actually having this feature set. I'm not exactly too sure what is required for PCI compliance, as this is not my area of expertise, but I'm sure we could come up with a solution where the credit card information is still available to us and it's still PCI complaint.

    For one of our clients, we actually created a web service on another machine, which we sent over the credit card information, which was then encrypted on that machine, and they provided us back with a key. Essentially mimicking what some processors support, but providing this functionality ourselves. This separate machine was on a bare-bones web server and only provided that functionality and nothing else.

    As for PCI complaince, this is a visa regulation and not an actual law. Visa has it, so they can fine you, should you lose customer data. I will recommend that we create something that is PCI complaint, but still allows us to be flexible with out having to rely on processors to store of credit cards for us, since some do not support this.

    I'll have to read through everything and provide a more detailed response back in a couple days.

    D4V
    D4V's picture
    Offline
    Joined: 02/13/2009
    Juice: 15
    Re: uc_recurring thoughts/ideas...

    This is awesome! Keep up the good work! Laughing out loud

    mrmeech's picture
    Offline
    Joined: 03/11/2009
    Juice: 66
    Further thoughts... From

    Further thoughts...

    From what i gather: The current Authorize.net CIM checkout in Ubercart will still process the transaction as either a SIM or AIM transaction, and it will set up a CIM profile for later billing to that customer, where Auth.net holds the CC info and the cart has a Auth.net token to use for billing. And i believe this is only accessable via the admin - there are currently no front end customer benefits.

    With not too much alteration of the current work-in-progress patch, is it possible to use the Auth.net CIM token for recurring payments? So, customer purchases a subscription and they are immediately charged, then when the subscription recurs they are charged again using the CIM token. This approach should also help with being PCI compliant, as you are not storing any sensitive CC info in the Ubercart database.

    Also, to add to the wishlist -- To allow the customer to select, during checkout, when they would like their subscription to start, if not immediately.

    Looking forward to hearing back on these things. I'm really excited about this project and will help out where i can. PLEASE PLEASE keep working on this, because it's excellent and UC really needs the whole recurring area reworked (as you are keenly aware). Smiling

    EDIT: Some backup on what i was saying about the current Auth.net CIM features: http://www.ubercart.org/forum/support/8596/what_do_new_cim_setting_authr...

    justageek's picture
    Offline
    Bug Finder
    Joined: 10/29/2008
    Juice: 189
    how will you do this part

    # We're ignoring PCI compliance for now and we will store credit cards internally in the database ourselves. This can be abstracted later to be PCI compliant.

    this seems scary, storing card numbers in your drupal system.

    mrmeech's picture
    Offline
    Joined: 03/11/2009
    Juice: 66
    Re: how will you do this part

    I agree with Justageek that this is not a...wise?... approach. Less knowledgeable users are not going to be able to abstract the CC info to become PCI compliant / less at risk.

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Re: uc_recurring thoughts/ideas...

    I'm working on a membership site and need to meet these requirements:

    a. [Automatic] Suspending (removing the membership role)an account if credit card declines on renew. This needs to happen automatically without admin having to take action
    b. [Automatic] Un-suspending (reassigning the membership role) an account if user resolves the problem themselves, e.g. they update their credit card data, the transaction is resubmitted and is successful
    c. [Manual] Un-suspending an account if payment is run manually or through alternative means (delivering a check for example)
    d. [Automatic] Sending notifications to users with ample time before a credit card expires or is charged for renewal
    e. Allowing users to update credit card info (before card expiration, use a different card for renewal next time, etc.)

    What is the current best solution for me? I have installed uc_recurring, and I can see how that would work to assign the role and to rebill upon expiration, but I dont' think it will do these other things. And I assume I have to use Authorize.net ARB with it? How do I do this otherwise?

    Any help/insight/guidance/wisdom is greatly appreciated.

    TIA
    -L

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    univate@drupal.org's picture
    Offline
    Getting busy with the Ubercode.
    Joined: 03/27/2009
    Juice: 465
    Re: Re: uc_recurring thoughts/ideas...

    withe the current version of uc_roles and the new uc_recurring 2.x-dev everything except for (d) is currently supported.

    Although with (e) it will depend on the payment gateway you are using and if that supports updating credit cards.

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Re: Re: Re: uc_recurring thoughts/ideas...

    Thank you a ton for your prompt reply. It is very helpful.

    I wonder if I could use Workflow_ng for item d.? To detect when the stored expiration date is close at hand and trigger emails to the card owner and the site admin?

    I know that Auth.net's ARB will not rebill until the following cycle if the payment fails for some reason, will UC/uc_recurring be made aware of the failure and notify someone and/or try to rebill?

    I think I probably just need a better payment gateway than Authorize.net. Any suggestions for a small business based in the US?

    -L

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    univate@drupal.org's picture
    Offline
    Getting busy with the Ubercode.
    Joined: 03/27/2009
    Juice: 465
    Re: Re: Re: Re: uc_recurring thoughts/ideas...

    You may want to also take a look at the uc_cim module for authorize.net:
    http://www.ubercart.org/contrib/2537

    I would like to work on a way to add support for notifying users when their credit card is about to expire, we just need to store the expiry date somewhere...

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Re: Re: Re: uc_recurring thoughts/ideas...

    I've been in touch with PayTrace, and they look like they are going to be compatible with our needs. They do support (e), but through the API from Ubercart. So that will need testing on the UC end.

    PayTrace will also not rebill if a payment fails (until the next cycle), but if the gateway sends back the information of the failure to UC, will uc_recurring notify the admin and the member of the failure? And then resend the request to bill for that cycle if the problem is fixed (new cc, for example)?

    Also, with regard to the expiration date, we need to figure out how to store it, as you said. B/c then we can run workflow_ng or other actions based on it.

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Re: Re: Re: uc_recurring thoughts/ideas...

    PayTrace tells me they will support updating the cc info and using that new info in the next billing cycle.

    But how do I send that from Ubercart so that the gateway knows it's an update, not a new order? Does uc_recurring support updating cc info in the gateway?

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Re: Re: Re: Re: uc_recurring thoughts/ideas...

    I'm not even sure I understand how this works...

    I have to have a recurring payment ability with the gateway, right? So the gateway auto charges every x interval?
    But uc_recurring is also set to bill every x interval?

    Does the gateway only store the cc info and waits for a trigger from uc_recurring? Does uc_recurring just check for a validated payment?

    I'm missing something here.

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    univate@drupal.org's picture
    Offline
    Getting busy with the Ubercode.
    Joined: 03/27/2009
    Juice: 465
    Re: Re: Re: Re: Re: uc_recurring thoughts/ideas...

    Every payment gateway is different (which is half the problem of trying to implement uc_recurring). The goal of uc_recurring is to handle everything related to recurring payments that it can (e.g. setting products up and keeping track of orders and payments so it can also work with the rest of ubercart to update roles).

    But when it comes to a specific payment gateway function (e.g. charging the user) it passing control to the module that is responsible for talking to that gateway, So that module needs to have implemented a couple of function to allow this communication to work.

    In the case of updating credit cards this is a payment gateway specific task, so that is completely in the hands of the payment gateway module to handle.

    In the case of working out if a credit card has expired, well normally be the role of the credit card gateway module to handle credit card details, but for this to work we need to get the date information somehow into uc_recurring so it can check of against upcoming charges.

    The one idea I do have is that when you charge an order, usually the payment gateway return hashed out credit card details and the expiry date, so it might be possible to trigger an email from that or queue an email to be sent from that???

    lindsayo's picture
    Offline
    Bug Finder
    Joined: 03/04/2008
    Juice: 86
    Every payment gateway is
    Every payment gateway is different (which is half the problem of trying to implement uc_recurring). The goal of uc_recurring is to handle everything related to recurring payments that it can (e.g. setting products up and keeping track of orders and payments so it can also work with the rest of ubercart to update roles).

    But when it comes to a specific payment gateway function (e.g. charging the user) it passing control to the module that is responsible for talking to that gateway, So that module needs to have implemented a couple of function to allow this communication to work.

    So I'm still confused...does the gateway trigger the rebill or does uc_recurring tell the gateway to bill again with the info the gateway has stored?

    The one idea I do have is that when you charge an order, usually the payment gateway return hashed out credit card details and the expiry date, so it might be possible to trigger an email from that or queue an email to be sent from that???

    So how would that work? Something like
    if [gateway cc details exp date] <30days than [date of next billing cycle] then workflow_ng sends out a notifying email to admin and user?

    Does that make sense?

    -----
    Five Rings Web Design
    www.fiveringswebdesign.com
    Drupal site dev, graphic design, HTML/CSS layouts, site maintenance, ...

    univate@drupal.org's picture
    Offline
    Getting busy with the Ubercode.
    Joined: 03/27/2009
    Juice: 465
    Re: Every payment gateway is

    How rebilling works depends on the payment gateway... so when setting up a recurring billing schedule uc_recurring asks the payment gateway module to setup a recurring payment. When it comes time to renew, uc_recurring ask the payment gateway again "do you want to do something at this point"

    THe payment gateway module needs to be setup to work with uc_recurring.

    mrmeech's picture
    Offline
    Joined: 03/11/2009
    Juice: 66
    Univate - Just wondering

    Univate -

    Just wondering how things are coming along on this as there hasn't been any conversing in here for 5 weeks or so now.

    Thanks! Smiling

    activelyOUT's picture
    Offline
    Joined: 04/20/2009
    Juice: 70
    subscribing

    this is a great feature.