0

Our UAT uses chargeover for payment and other user management features. A couple weeks after we enabled automated testing about 50 test cases running about 2 runs every hour (can peak to 6-7 runs when more people are doing pre-commit validations), we got a call from chargeover saying we are making too many API calls to their service (10times/sec when peaking).

Now, what can we do about this? We are already paying for commercial. Is there a way to set up a fake CO instance for development and testing? (probably not)

PS: This is for integration testing, not API testing. The calls are made by our back-end services behind the scene.

  • There is always a way to set up a fake instance. The question is, typically, do you want to go through the effort of building it. Based on the answer from Keith it sounds like there is underlying issues within your application that should be considered a defect and addressed. One thing I have done historically is set up a MITM that logs all requests and then review over the request/response for the APIs and start building out my mocking service with as little logic as possible. – Paul Muir May 31 '16 at 15:17
5

I can provide a bit more detail here -- I work for ChargeOver.

Firstly -- Please do feel free to reach out with us directly! We haven't spoken with you directly (we've spoken with Alvin about this only) but are more than happy to help in any way we can! Really! We love to help and code and want to make sure things are going well for you, and also don't impact our other customers negatively.

Our concerns revolve around the really unusual profile/patterns of the API requests and the high volume of API requests. Currently we're seeing much higher API usage than just 10 requests/second -- as of this morning:

  • 110,000+ API requests per hour
  • average of 30+ API requests/second
  • peek well over 60+ API requests/second

Some specific counts:

2016-05-31 06am CST     119,206 requests/hour
2016-05-31 05am CST     130,273 requests/hour
2016-05-31 04am CST     106,377 requests/hour
2016-05-31 03am CST      65,089 requests/hour
2016-05-31 02am CST     112,494 requests/hour
2016-05-31 01am CST     117,621 requests/hour
etc. etc. etc. 

What triggered some warnings internally here is that you only have roughly ~120 subscriptions (active customers) you are testing with, and ~5000 customers you are testing with, while doing ~100,000+ GET /customer/X requests/hour.

Why do your tests need to query for customers ~100,000+ times per hour when you have only ~120 active customers?

With regards to your specific questions:

Now, what can we do about this?

  • Caching - Implement some level of caching in your application -- even caching a customer for 2 or 3 seconds is going to make a massive different in the number of API requests, and will offer the additional benefit of improving application performance on your end too!

  • Profile/examine your code - I suspect something is a little wonky within your app, given that you're querying for the same customers over and over and over again thousands of times every hour.

  • Talk to us - If you can reach out to us with more specifics, we'd be glad to look over your code, or discuss other options if you really need this volume of API requests. I don't want to really talk about pricing in public given it's private financial information about your company, but there may be options here if you reach out to us.

Please let us know how else we can help -- we are happy to help in any way we can (but also need to make sure your unusual API usage doesn't impact our other customers).

We are already paying for commercial.

We may have options here if you reach out to us -- without getting into specifics, you are on one of our smallest/lowest pricing tiers.

Is there a way to set up a fake CO instance for development and testing? (probably not)

You are already making these API requests to a copy of your production environment (a staging/test environment).

  • Thanks Keith for your comment. The product under testing is on the early stage, we know the code that makes the amount of calls will be improved but I'm looking for a way to avoid this in the future. We are adding caching to reduce the calls, not sure why it doesn't do much yet. PS: On the other hand, since I have you here, is there a tool for us to monitor the # of calls that reach you from our side? This would solve my problem from a different angle. If this is not the proper way of communication we can chat via other channel. Thanks again. – Randy Lei Jun 1 '16 at 10:43
  • If you send us a support ticket, we can get you a custom report that shows you a count of the # of each type of call hitting each endpoint, by hour. I can probably get you a custom report of the specific API calls being made as well. Maybe some better logging than what's already available in the report center. Today looks much better! – Keith Palmer Jr. Jun 1 '16 at 11:34
2

I don't know anything about chargeover, though looking at their Web site, I don't see any easy way of getting a fake instance. However, this is the sort of thing you'd use either mocking or some sort of other test framework for. For most of your tests, you probably don't actually need to really access the CO service, you just need to make an API call and get some sort of response back, both of which I assume are documented. So you should be able to write either a mock server or local test framework that pretends to be CO.

  • My problem is this is integration testing, not unit or api testing. And the API calls to CO is made by our backend services that actually depend on the functions provided by CO's APIs. So mocking is probably not applicable. I will edit the original question to clarify, this is not testing for APIs. – Randy Lei May 30 '16 at 4:28
  • Have you asked them what they recommend, then? Or spoken to other users of the service? – Kevin McKenzie May 30 '16 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.