Lessons Learned while Integrating with Third-Party Web APIs

How hard can it be?

At some point if you’re running a serious web application you’re going to need to integrate with a third party API. Seriously, it’s just going to happen.

Does your site have users? Then someone is going to want to share their whatsit on Facebook or Tweet about it on Twitter or vomit filters all over it on Instagram.

Do you work with businesses? One of your clients will want you to integrate with Salesforce or send emails via Mailchimp or integrate with your Second Street database (shameless plug).

The good news is that these companies are in the business of making software too, and they’ve spent a huge amount of time and energy coming up with a solution that lets you work with them. In all likelihood, you’ve got access to extensive documentation and -- if you’re extremely lucky -- a large community of other developers to work with.

How hard can it be?

A lot harder than you think. At Second Street, we’ve integrated with every Web API listed above and across the board there’s only been one constant: Integrating with a third-party web API sucks.

Why it sucks

Your standards mean nothing

You’ve spent a ton of time working through how you architect every layer of your system. You’re running the hipster front end framework du jour that communicates to your immaculately architected JSON REST API. The world is in perfect balance.

A threat looms on the horizon. What’s that you say -- this API uses XML? It requires us to use OAuth? It needs us to pass through an API key as a header value instead of a query string parameter?

There’s almost no chance that the standards of a Web API are going to align with the standards you’ve developed. Even if you’re actively trying to follow a standard like JSON API the odds are good that no one else is (yet!).

The first concession you’re going to make is finding you’re going to have to muck up your code by breaking existing conventions.

You are terrible at estimating

Humans are very bad at estimating in general yet it’s somewhat critical in most development environments that you have some vague sense of how long something will take. Usually you can get by when you guess that something will take 2 weeks by saying it will be 1-3 weeks.

Unfortunately, when it comes to third-party integrations you’re estimates are going to be even more garbage than usual.

Every third-party integration we’ve ever done has had at least one major complication that has seriously impacted development speed that no one at all foresaw. We’re all smart people over here -- it’s just the nature of the beast that when you’re working with someone else’s code, you’re going to have some unknown unknowns. In short, your project manager is going to be frustrated with you from start to finish.

You’re supporting more browsers now

As you start working with third-parties more you may find that the audience you’re supporting will begin to shift. This is particularly true if you start working with social media services like Facebook, Twitter, Instagram, and others. In fact, this shift is likely a huge part of the reason you did those integrations in the first place -- you wanted to attract users of those services!

As you bask in your expanded userbase, pride swelling from a job well-done, you’ll be brought back down to earth by bug cases from users that can’t seem to use your software at all.

A fun fact about Social Media sites is that a huge portion of users actually tend to use the app instead of the native web experience. For Android devices, that’s all well and good. iOS, though? Not so much.

iOS apps have the spectacular habit of not opening links clicked in app in your system default browser -- instead, they open directly in the app in a UIWebView. This makes sense from their perspective as they don’t want to force users to leave the app experience. From your perspective, however, this is a tremendous pain in the ass.

UIWebViews tend to be far, far slower than a dedicated browser (in our testing the Twitter app’s browser loaded pages about 10x slower than Safari) so moderate performance issues can quickly become a major performance nightmare.

Further, UIWebViews lack some very basic browser behavior. For example, in general you can’t open a new window. Sure hope your app doesn’t have the complicated, futuristic technology of “opening a new browser window” anywhere inside of it!

None of this is insurmountable, but you’ll find yourself struggling to keep up with the bizarre idiosyncrasies of some of these in-app browsers as time goes on. Around the time you start writing a special case to handle the fact that Facebook authentication in the Twitter in-app browser causes a user to be stranded on a blank page after logging in is around the time you’ll start questioning just how much we really need Twitter anyway...

You’ll never be done

Even when you’ve finally finished the agonizing process of integrating with a third-party, the fun doesn’t stop. Now that you’re tied to the third-party, you’re subject to the whims of their software updates. In most cases, public APIs tend to be mostly stable, so it’s uncommon (but possible) for a breaking change to be deployed that suddenly leaves you high and dry. That being said, it is far more common for policies surrounding API usage to change.

Recently, Facebook made sweeping changes to their policies surrounding under what circumstance a user can be forced to like a page to continue (hint: it’s under no circumstance). As a result, a huge usecase for their API was deemed no longer in accordance with Facebook Terms of Service. This leaves those that have integrated with the API with the burden of at the very least drastically modifying their code and others with the prospect of completely rethinking their services.

Integrating with a third-party API is a lifetime-of-product commitment.

What you can do

Let the API tell you how it wants to work, don’t tell it how to work

Your intuition will be to come up with your desired user flow and then use the third-party API to accomplish that. Your intuition is garbage. With a third-party API, generally speaking the “right” user flow already exists. This is particularly apparent in authentication flows.

As an example, with Facebook apps you are expected to present a user with a dialog to log in before ever presenting them with interactive Facebook elements. If you choose to not follow that path, you will find that there’s a longstanding bug with Like Buttons where only the first click on the first Like Button on the page is registered. Simply adopting the flow of forcing a login before displaying those buttons, as Facebook intends, will prevent that negative behavior.

That’s just one relatively trivial example, but the further you deviate from the “right” way to interact with the API, the more likely it is you’re going to run into bugs.

Don’t reinvent the wheel

Most of the problems you’re facing have already been solved by someone else. Frameworks typically exist that wrap up API calls in easy to use native code within your environment of choice. Before you even get started on your project you should take a look at the resources the API developers link to, as often you’ll be able to cut out more than half of your work just by installing a library.

Similarly, it’s pretty likely that the thing you’re trying to do isn’t quite so novel as you thought. A simple search on Stack Overflow is likely to yield at least a couple of results from users that attempted more or less what you’re trying to and have run into a problem -- make sure you aren’t going to run into that same issue!

Keep it LEAN

This is a suggestion that can apply to all software development but it’s particularly salient here. You should begin your interactions with a third-party API in the most basic way possible so you can better evaluate how useful the integration really is. Working with third-party APIs is a time consuming endeavor, so the more quickly you can learn if what you’re doing is worth your time, the better.

...And it’s still going to suck

Sorry. No way around it. If you can stumble through your integration following the guidelines above, you should make it through relatively unbloodied. Don’t expect to enjoy the process, though.