Join us for this event

Subscribe to newsletter
By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Rutter Odyssey: How to Build Integrations 10x Faster

Watch a webinar where Rutter’s Head of Engineering Danny Keller and Engineering Lead Frank Chan dive into all things APIs, how to build integrations 10x faster and why we built Rutter Odyssey, our proprietary infrastructure that helps us build the highest quality integrations. They also share best practices on structuring teams for building integrations, philosophy for building APIs, and more.

Transcript:

Welcome everyone to the webinar. So the webinar that we're running today is just to focus on building integrations. So just introductions as a matter of course. My name is Danny. I'm the head of engineering here at Rutter. I run our engineering teams, work really closely with our product teams to figure out what to build, what to invest in, and help support phenomenal engineers like Frank. So Frank, you want to give you a quick intro? Hi everybody.

My name is Frank. I work on the integrations platforms team as a software engineer. Danny and I, we work very closely together and very excited to host this webinar. And then the broader introduction is that little logo in the bottom right corner. So we both work at Rutter. Rutter is a universal API for commerce and accounting data. And so that's kind of why we're talking about building integrations.

Both Frank and I have kind of spent the last year plus of our lives dealing with nothing but building integrations. Have some thoughts on them and would love to share some of those thoughts with you guys. So the agenda here is why it's hard to build integrations, best practices we've learned, and how working with Rutter helps you move faster. Kind of like core target audience is engineers who have maybe done some work with integrations previously but have not massively jumped into them before and are not kind of well versed experts on the subject. So that's kind of like the level of data that we're trying to provide. The level of insight is kind of geared towards that novice integration builder to moderate integration builder audience. Why is it actually hard to build integrations? So we're going to do a case study here of building a financing application.

So we're using maybe, let's say Ramp as an example customer. Ramp wants to basically build financing apps that allow them to authenticate a merchant through their ecommerce platform, fetch all of their order and transaction data, store this within their database, and then run queries over this data in order to analyze and underwrite the merchant or borrower that they're working with. And so that's just the particular use case we are focusing on. We definitely also have a lot of other customers like either drop shipping customers or fulfillment providers or even some marketing analytics companies. All of those use cases look different. We're just going to focus on this one use case because it makes this presentation fit within an hour instead of fitting within 6 hours. And we're also focusing on financing because it's a read only use case versus a read write use case.

And once you add in write endpoints, that greatly increases kind of the complexity of the problem that you're dealing with just because the way to think about it is that there are 20 ways that like reading data from an integration can go wrong. There are a thousand ways that writing integration to a writing data to an integration can go wrong. So we're choosing to focus on the read only use case here, just to make this webinar a little more digestible, easy to go through. So let's assume that you're starting to build integrations for the first time at your company. You need to build your first integration and you choose Shopify. Shopify has the most merchants in the ecommerce market. It's a very easy pick.

And so you kind of go through three steps here. And those steps are basically like building authentication and getting a test store connected. Then you build the actual system to ingest data from those stores. Maybe use Shopify's GraphQL API because that's kind of the best in the business, the most modern API that they have built out. And you listen to web hooks for incremental updates. And then step three, you then like plug the data into your own system and you write your queries that allow you to run analysis over the revenue data. So those are just the three steps.

Shopify is a pretty simple platform to work with. The data model is very kind of concise and simple, where everything that you need for a product is kind of underneath the top level product entity. They have this GraphQL API that allows you to conditionally fetch different fields. And most importantly, the documentation is actually very fleshed out, thorough and accurate. And so you might build this first integration, have like a very easy time and think that you're ready to go into the next one. So let's say the next integration you build is Magento. Magento is important because a lot of very large important enterprise customers exist on Magento.

So a lot of our customers really like building out this platform for those larger merchants or borrowers. And so it looks at first glance that Magento is the same, right? Like you need to build authentication, you need to get test or connected, you need to build data ingestion, and then you need to plug that data into your data warehouse and actually run analysis over it. Sounds very similar. Unfortunately, the devil's in the details. So when you actually start building out Magento authentication, you're realizing that setting up a Magento account requires the user to follow a really complex workflow. So specifically with magento, many users choose to make a particular user account that then has certain and specific permissions that they then provide as authentication credentials. And that's like a multi step complex workflow that you need to help your user work through.

And that requires building up a whole onboarding guide and basically finding some way to make this more self serve, which is already a complexity step up from Shopify. And then even once you've done that and the first few customers authenticate, you realize that because Magento is self hosted, some of these customers have two FA set up, particularly with their Magento instance. And this makes authentication a lot harder. And because the documentation is poor. You didn't even realize that this was a thing until the first customer signed up with two FA. And so your engineering team is in a rush to figure out this poorly documented solution while merchants or borrowers are sitting there on the line waiting for you to unblock them. And then the other issue with Magento is just the data model is totally different from Shopify.

So this picture is actually the database diagram for Magento. It is clearly representing more things than just products and orders. And so in that way, it's like far more complex than what you'll actually need to deal with. But just from our own experience building with Magento, we found a lot of gotchas that we weren't necessarily expecting when we were first building. And so great examples of that is like comparing Shopify to Magento. Magento breaks product up like products and orders up into several different entities. So while with Shopify you can just fetch top level orders and top level products, with Magento for orders, you might need to fetch the top level order and then also line items and maybe even fetch some additional tables in order to get the correct order statuses.

It's just like there's a lot more your dependency graph products are actually products and product variants are all represented as the same thing. So for any product variant, that's actually just another row in the products table that is instead marked as a variant. And so that requires writing special bespoke logic for Magento in order to figure out which is a variant and which is a top line product. And so, as you're seeing already, even with the second integration, we're starting to run into a really big increase in complexity. And so by the time of your fifth integration, maybe you're actually feeling a lot better because you've seen a lot of different types of platforms and you get into a groove, you're building more quickly and you're kind of starting to understand what all these different Pitfalls are from previous integrations. But even starting to catch these Pitfalls, there are frequent errors that you run into. So handling rate limits per platform is a massive headache.

Some of them are rather simple, all of them are different. And so finding a way to kind of abstract rate limits across these platforms is going to be like a major headache. Another thing that we commonly run into at Rutter, which is a big pain, is just that dealing with platform specific errors is really hard. Especially because depending on the platform, these errors can have multiple languages or no language at all. And so, for example, one great example of this is that WooCommerce will sometimes just give you 400 errors with absolutely no error message. And it requires a lot of time to actually read through all of the WooCommerce forums and figure out all the different reasons why you might be getting this 400 error. Because WooCommerce is definitely not going to help you.

And so building up that knowledge over time around how to deal with these different errors is like a big pain on other platforms, like both WooCommerce and PrestaShop. Sometimes these error codes come back in different languages, which is doable, because thankfully we have Google Translate, but just makes the process a little more complicated. And then finally, one thing that you might realize as you move along is that some platforms don't provide any form of date filtering. And so even if you're pulling these platforms incrementally, and using date filtering as a way to kind of make sure that you're fetching incremental data, you'll eventually run into that N plus one platform that doesn't have that at all. And for this one platform, you need to build a separate system that allows you to kind of fetch all of the data, but potentially in a smarter way. You might need to hack around with the API to see if there's an unpublished date filter. All of these things that just make building each new integration much harder than they should be.

So this is where I pick up. So you have five integrations in production. And honestly, that's amazing. Great job. Everything works until it doesn't. And so what's going on here? As your number of connected stores grows, you find that more and more of the data ingestions begin to fail. This is because when you're building these integrations, you test on a limited number of stores.

And as new stores connect, you're bound to find outliers. In our experience, outliers like this occur because first and foremost, inconsistent data schemas. A lot of times, platforms don't even have API docs. And so what you have to do is kind of take your best stab at what the data schema you expect is. And when you make those assumptions, of course there will be a new store that comes along the way that breaks those assumptions. The greatest example being that sometimes you always expect fields to be populated with the value, and then suddenly you're hit with a null and then your integration breaks. The second biggest challenge as Danny alluded to before, is that some platforms are self hosted, meaning that the merchant hosts their store themselves.

And this is just a source of like a plethora of different issues. One of the greatest ones being just sporadic outages with the hosting of the stores themselves. We might get 404 SSL errors. Just 500. The greatest challenge with self hosted stores as well is that a lot of times they're hosting a different version of the software that is their ecommerce store. And so with each different version, there are slight changes to schemas. There are different behavioral patterns that you get when hitting the APIs, and these are sometimes documented, sometimes not.

You just kind of have to find out as more stores connect. And the last thing is that different stores have different configurations set up. For example, especially when your cell hosted. Some of them have cloud flare set up, and this just blocks your ability to send requests directly to the customer's API in the first place. And then you need to get your customer to what it lists your API. And the final thing is authentication is breaking. So Danny alluded to this before, but sometimes different stores just have different methods of authenticating, or even in the worst case, the platform may just change their code without warning.

And this breaks the assumptions you made. One of the scary incidents we had earlier this year was that PayPal made a slight change to their authentication method, and that threw us into a loop for a little bit, but we fixed it. And so basically, that's all you can do in this case. This is nothing that can't be fixed. Over time, we become more flexible with the data schema. We handle newfound corner cases and exceptions, and we build a knowledge base of how merchants can fix the issues from their end, for example, the cloudflare issue. And so over time, we harden our API, we expand our coverage, and we support more and more stores.

And this is great until something new comes along. So let's say your product team comes to you with a great idea they have for these integrations that you've built. All they need is this data that you currently don't supply them for every single platform. And so you go through each integration and you add some features so that you're querying more data that we now need and the product team now depends on. But this introduces holes into your API. First of all, you have to go ahead and make all those data schema decisions that you did for all the previous data that you were fetching, and then reconcile those across all the different platforms. And then once you've done this initial work, you have all the same problems as before, where there are holes in your coverage due to all these unforeseen issues across different stores.

And then you need to begin the whole process of API hardening again until you have the full coverage that you need. And this is just a massive, massive maintenance cost that you have to incur over time. Some best practices that we've learned while dealing with this maintenance first and foremost being if we could start over, we would try to start with the three integrations that matter the most and then abstract based on that. So abstractions are everything, right? They make your code manageable, and they allow you to keep maintenance costs low over time. But setting the right scope of abstraction is really important if you try to support the bottom line of everything. Sometimes your abstractions become too general, and you can't rely on them to move fast. Other times, if you have too few integrations and your abstractions are really specific, as soon as you want to add the N plus one integration, your abstractions start to fall apart.

One of the biggest problems that we have to tackle with, and that we've tried to abstract away, is this idea of connection health. So when a merchant connects and the connection ends up causing some sort of error down the way either throughout navigation or data ingestion, what do we do? Well, we need to proactively monitor these errors over time and we need to determine where they're coming from. We need to alert our team about them. We need to root cause and fix the errors that are due to our own code misbehaving. Or we need to handle the errors that come from the platform that we can't do anything about except for maybe like customer communication or having the customer unblock the connection in some way. And what we found is that the way we deal with these really does depend on volume. When you have a lower volume of connections, you might be able to approach it just with something like pager duty where an exception comes along and then you page the on call so that they can hop on, see what's going on through, cause it and fix it.

But as your connection grows, this quickly becomes infeasible. And so we need to do a lot. We need to invest a lot in observability and monitoring through tools like Datadog and Sentry. We found Sentry to be especially helpful with larger volumes of errors because it does cluster similar errors for you. It exposes the root cause of what's causing them and you can configure alerting to alert on these clusters for debugging and observability into what's going on in our system. We found that Datadog is honestly priceless, although it is pretty expensive. So the one learning that we took away is that you need to invest heavily in connection health more than you think initially.

Because even though your initial build does work, over time, you'll find that with maintenance, there are always going to be exceptions and errors that pop up as new stores connect. The next problem that we want to tackle is data quality. We have successfully connected a store and we're doing the data ingestion for these stores. But we need to make sure that the data we're collecting from these stores is actually accurate as it flows through our system. We think about data quality in a couple of different dimensions, the first one being completeness. If we have 1000 orders on the platform and we fetch those, how many do we end up with at the end of the day? And we expect to be 1000. Right.

This is actually a really hard problem to ensure though, because you're going to use your code to query this platform as a source of truth and then you're going to try to validate that code with some more code. And it's just going to be difficult to make sure there are no bugs all across your system. Cleanliness is the second one. So let's say for each of the thousand rows of orders, each has ten fields. How many of those are correct or the expected value that they are supposed to represent? This one's also pretty hard, right? Again, the platform is the source of truth, and you're going to be running these transformations on the fields that you retrieve. So maybe the data is clean when you first fetch it, but by the time you store it in your own database or use it for some sort of analysis, are they still correct? The next one is freshness. How up to date is your data? This one's pretty easy until it gets hard, right? So the naive solution is just configure your data syncs to have a shorter period.

The shorter the period, the fresher your data will be. And that makes sense. But how do you do this in a sustainable way? A single one of these merchant stores can be huge and you could just be downloading millions and millions of rows of data and you don't want to do that every minute or every five minutes, or even every ten minutes. And then you start to get into the world of incremental data syncs. And that can get really complicated really fast, depending on which API you depend on. And then the last one is duplication for every platform order. How many of those do you have? And this one, honestly, is the easiest problem that we've had to tackle.

If we're storing this data in the database, a unique index will solve our problem, so long as we can identify the keys that make each row unique. The largest thing that we've seen is that we have to consider each one of these dimensions not just once, but in different steps of our system. So something that is clean when we first query from the platform might not be clean after we transform it and store it, or might not be clean when we query it and use it for something. And so we do have to consider these every step of the way. And the biggest takeaway here is that these are really hard problems. And because of that, it's really important to think about the data quality from day zero and really try to identify the behavior that flags something as suspicious. I rather we've used a multiple of different types of approaches to try to figure out when data does seem suspicious, from sampling, from running canaries, and all sorts of schema validation.

And the last problem tackle is just platform nuances. If you have more integrations, you have more business logic that you need to keep in your mind, more platform specific nuances that you've learned from, especially over time as you maintain the system. And so you have all this knowledge, and once you do the initial build, that's great. But you will have to revisit this when bugs come up that force you to reconsider whether your current logic is correct. Or you want to build new abstractions or new features on top of your existing integrations, on top of your existing code base. And you need to basically understand what was going through your mind in the first place. Or finally someone on your team leaves and you need to transfer this knowledge, right? A lot of this might just be in somebody's head.

And so the biggest takeaway that we've learned is that we need to start documenting immediately. There is just too much going on with each API and they have public documentation, but it's going to be nothing compared to the internal lessons that you've learned when actually working with the API. So this is kind of how working with Rutter can make you faster, the slightly more sales pitch, but we think also really helpful when it comes to thinking about integrations. So basically when you're sitting there and you're actually thinking about like, hey, we need to build these integrations, this is important to our company strategy. It allows us to get to the market faster, reach more merchants, reach more borrowers, and generally increase the value of our business. There's always going to be this decision that you first need to make around, do we build or do we buy? Right? And so of course, building internally might offer you more control, but might be very expensive versus buying. It's probably going to give you more cost savings and hopefully, if it's actually with a good vendor, provide a lot of clean integrations and support that you wouldn't otherwise get.

So when it comes down to when people are making this decision, the core thing that people are deciding around is basically the time of their engineers, right? So as we've kind of discussed by going through what makes these integrations difficult, there's a lot of things that you need to build. There's both like the actual integrations themselves, there's all of them, the abstractions and systems that you need to build on top of those integrations in order to handle and maintain them well. And that just takes a bunch of time, right? And so when we typically talk to our customers, the thing that they're always trying to figure out is how many engineers does this take and how many engineers do I want to focus on our core product? And so for a lot of people like that desire to have like your core team focus on the core product the most and focus on your customers, and delivering value to those customers in a way that drives an increased value in your business is what people want their engineers to be doing. And oftentimes building and maintaining integrations is completely orthogonal to that. And the better you build out these integrations, it might improve product quality, but it's not going to significantly improve the overall customer experience in a way that's making them more money and therefore making you more money. And so thinking about building Rutter we want to make it as easy as possible to build integrations. And so Rutter kind of helps people ship multiple integrations in weeks instead of months and significantly, ideally decrease your maintenance flows.

Two things I want to call out here. One, we put it as weeks versus days because a lot of these platforms still require app approvals, and those can take weeks. And unfortunately, we can save engineering time, but we can't make Shopify move faster in their approval process. And then additionally, I just wanted to really focus on that. A lot of the times when we talk to people and they're considering a build versus buy solution, they kind of systematically undervalue the cost of maintenance over time. That's due to a lot of reasons bugs are hard to track. It's hard to track how much time your engineers are spending on bugs.

But especially if you're building four, five, six integrations, that's four, five, six external dependencies that are going to be consistently changing on you and not documenting their changes, not explaining what's happening. And that's going to be like a major maintenance headache over time. And ideally, if we build the right product at Rutter, which we're working on every day, those maintenance loads decrease for you if you choose to use an external vendor like Rutter. So when it kind of came to, what do we actually need Rutter to be? In order to solve some of those problems, we realized that we needed a consistent API contract. We needed monitoring, logging, and operability on every integration. We needed really strong connection health management, and we needed to also unify errors so that that problem of very complex platform errors doesn't actually touch our customers. And so ideally, our customers are just seeing kind of a Rutter error that tells them exactly what's wrong, exactly what they need to do, and they don't need to deal with WooCommerce sending them a 400 error with no explanation whatsoever.

And so this led us to build like what we're calling internally, rather Odyssey, which there's also a blog post on if people want to read about it. But it's that platform investment that Frank talked about where we want to make sure that we know that we're going to be building more and more integrations forever. And so we want to make sure that our platform and our foundation is incredibly strong. And so this graphic is pretty complicated, but hopefully shows that there are a lot of different concerns that we're handling at any one time. And so error handling is one concern worth calling out. That's like kind of how we're unifying errors into a single more manageable format and also dealing with the different errors from different platforms because not only do they change messages, but oftentimes the way that Amazon sends you an error is very different from how Shopify will send you an error. And so we also needed to basically abstract how we handle errors across these different integrations.

And I would also want to call out Rutter link, which is just one of the, in my opinion, coolest things that we've built, which is just a JavaScript snippet that then handles authentication for you across all these platforms. And then finally like connection health monitoring. And so the whole thing with Connection health monitoring, we've invested even more heavily in that than we'd recommend you to if you were to choose to build it yourself. Because for us, the most important thing is kind of constant service for our customers. And we know that these external dependencies are going to change their mind constantly and change things without telling us. And so we need to know when something is broken as soon as possible so that we can fix it, and so that therefore our customers aren't negatively affected by external dependencies being terrible. So that previous graphic covers a lot of the different abstractions and things that we needed to build, that you might need to build if you chose to build your own integrations in house.

But that's just initial build time. I also wanted to talk a little bit about the actual maintenance cost. If you kind of think about this as a before and after Rutter, or before and after any universal API, some of the benefits here are just that before you're using something that actually consolidates these APIs, you're kind of consistently running into corner cases. As Frank mentioned, as new customers come in, new things will break because there are stores that are configured in different ways or there are products that look different from other products. And then also over time, abstractions will change because as you add that N plus one platform, there's no guarantee that that N plus one platform isn't going to break all the assumptions that you've made, which will force you to kind of scrap all of the abstractions that you've already worked on building. Meanwhile, using something like Rutter allows you to kind of first get benefit from the fact that our API is being hardened by just like a lot more customers. So, like WooCommerce, when we first started building on top of it, had 20 different ways in which it's broken that we just weren't expecting because it was a self hosted platform.

You use something like Rutter, we've already solved 20 of those problems, whereas if you build it internally, you need to kind of run headfirst into that wall 20 times, which leads to a lot of customer pain. And then finally, because we have this like promise of a unified data model and a consistent API contract, you can kind of abstract and implement only once. And so you never need to deal with that M plus one platform problem where that next platform or integration is going to break all of your abstractions, because Rutter is going to hold a consistent abstraction that allows you to make consistent assumptions as you build everything. And this is. Just a little fun graphic which shows that like Frank especially helping to lead the team, we've been investing a lot into our platform and infrastructure to make it easier for us to build these integrations. So this little graphic is basically how many new integrations we've launched every quarter. And it hopefully shows that the number of integrations that we've been releasing has kind of been steadily increasing as our investment in platform has kind of allowed us to better position ourselves and better build these integrations in a more maintainable way.

Hopefully seeing that. You can either see a the power of using something like Rutter, or b the power of kind of building your own internal version of Rutter, though wouldn't recommend it because it's taken us a good chunk of engineers and a good chunk of time to get where we are now, and that would be an expensive investment. So that kind of concludes the section on how Rutter can help you build these integrations faster to kind of close it up on the customer perspective. Right? This is just representing a lot of the work that we've done to make it easier for our customers. The outcome of that is that actually then when our customers use us, ideally they're able to access these platforms much faster and their maintenance costs are also much lower moving on. And so in that way, you only need one engineer to maintain Rutter instead of needing three or four engineers or infinitely scaling as you try to increase the number of integrations that you need to deal with. So, Q and A, is there a sandbox to test integrations? So yeah, there is a sandbox to test integrations for Rutter specifically.

I think the sandbox does not cover all of the integrations, but I believe that we cover anywhere between two to three integrations per type of platform that we have. And we've just kind of stopped at two or three because we've seen that that is typically what helps our customers see the benefit of the unified data model across several different platforms at once. Any plans to add CRM integrations? Eg, HubSpot, Zoho, CRM, et cetera. So not on our near term roadmap, so we're not thinking about it for this quarter. There's a world in which we maybe potentially next year build it or the year after that. Over the long term, we know that we kind of want to build a lot of integrations, especially ones that tie directly into revenue data because that's where a lot of customers get value from us or in some of the kind of operational standpoints of businesses, especially when it comes to things like commerce and commerce enablement. So CRMs are a thing that we are considering, it's just a question of when and because we want to do commerce and accounting very well first.

CRMs are probably pushed out later, so I don't know exactly when they might be, whether it's six months, a year, a year and a half, but they are definitely something that, in the longer term, we definitely want to tackle. How do you test the integrations? Yeah, a lot of different things. So we have a manual QA process. We have unit tests set up, and then we're consistently kind of improving our testing environment so that things kind of go through as part of the manual. QA for every single integration that we build, one of the things that we need to do, we set up a test store for every single platform that we work with or every single integration that we work with. And kind of over time, we've kind of consistently improved the data set that we view as an expectation for each of those test stores. And so we actually will then lean on parts of the Ops team.

So whether it's like, people on the Customer Success team or people who are directly on our operations team to kind of help actually make sure that those test stores or workspaces are filled with as realistic data as we can get. And then oftentimes, especially if we're building a brand new integration, we'll also just, like, I'm sure that anyone here who's an engineer is well aware there is no such thing that is as useful as, like, live production data. And so typically, when we're building an integration, we'll find a customer who really wants that integration and is willing to kind of partner with us in order to deal with those initial rough edges once live data actually starts flowing through it. And so typically, that's the process that needs to be in place before we go on to release something to the more general public. Anonymous attendee. Did that help with the test workspaces? So the answer there is just we'll set up a test store, and then we have Ops teams that will then fill in those test stores with what we kind of want to see when it comes to realistic data. How many engineered days does it typically take to build an integration? Yeah, so it really depends on the integration.

Shopify is one of the easiest ones, and it also depends on whether you need to read or write. So Shopify is one of the easiest ones. If you're trying to build a read only integration on Shopify, it could probably take a good engineer that isn't blocked by corporate meetings and things like that, maybe like a week to build and to test a moderate degree. Of course, if they wanted to really bomb proof it, that would be something that maybe take two weeks. But then when you start looking at things like Magento or Amazon, some of these more complicated platforms, amazon would probably take you anywhere between two to three months. It's really poorly documented. You need to, like, fetch data from multiple places just because some APIs have, like, only give you the last 90 days of data.

But then you still need to fetch that data. There's a lot of complexities. The API itself uses a lot of reports and other things which requires like a polling mechanism to know when the port is ready. There's basically just like seven different concerns that you need to handle that are totally different from just operating on a standard rest API. So it depends on platform. It's kind of what I would cover. And then if you need to build right endpoints, that's going to take a lot longer because right endpoints, as previously mentioned, break in a lot more ways.

And so even on Shopify, if you want to build out right endpoints, I would then put that from one week to three to four weeks and the maintenance cost is just going to be significantly higher. Right, so building that initial integration is maybe going to take a little less time, but it's going to break in a lot more ways as you actually get hit with live traffic. Frank, any other comments on time to build? Yeah, I think that Shellfire is a good case of the happy path towards building integration. Danny said it will take a week to build that. That's optimistic for sure. That's Danny as a manager speaking, but from an engineering perspective, I do think that it's definitely possible within a week. We've seen that happen here at Rutter where we've cranked out these integrations, but that's after we invested very heavily into the platform on which these integrations are built.

On top of that's, after we've already hammered out all the best practices, it's after we have tons of helper methods and all the scaffolding in place to really support a new platform for your initial build. If you're just starting with Shopify, that's a great start. I do think it will take longer than a week. Are there any other new data types like CRM listed above? Coming soon. So one new data type that we recently actually released is just subscription data. So a lot of our customers were basically saying that they wanted to support merchants or borrowers both, or customers that both operated on something like Shopify, but also had some form of recurring data, recurring revenue, or they wanted to be able to actually create subscriptions in the same way that they were able to create products. And so we have at least initially launched out our read only subscription platforms.

So those don't have read support at the moment, but they have read support for, I think Recurly is one of them. Stripe's subscriptions or square subscriptions is one of them. But those are publicly listed on our website and that's like a kind of new platform type that we have recently moved into. And then what back end programming languages have you found to be ideal for low effort maintenance and speed of development? So we use TypeScript here at Rutter. We found that for speed of development, TypeScript is really great for the type safety part TypeScript is pretty good. We use the schema validation. We get free type script types from that as well.

So we both handle type safety and validation all in one. We found that it helps us move fast, and the node runtime environment is generally just very friendly for ease of development. But the caveat to that is that you really need to know the programming language you're going to do these things in very well. And so I would still say that the right answer is just the programming language. Do you know the best have API specs such as Swagger Open API been useful in building the integration clients? Yes and no. We've definitely tried to use open API specs provided by external platforms before. They are helpful in knowing what the Schema might look like.

They're not always a great source of truth. Even though a platform might provide a spec like that, it doesn't mean that they'll adhere to it themselves, is what we've learned. Yeah, one of the biggest learnings from working with all these external dependencies is that the phrase trust no one really bears a lot of truth. And so even like best in class APIs like Shopify, we've consistently run into things where Shopify changes an API without putting it as part of their change log, and that might break some of the assumptions that we've had. Or even with their great documentation, there still might be things where they say something isn't nullable, and then suddenly we're getting a null from that field. And so you can't really trust Shopify, you can't really trust anyone. And it's especially true that some platforms you really just cannot trust at all.

And their documentation is sparse and basically nonexistent. And to the degree that there is documentation, it's oftentimes more actively misleading than anything else. A follow up question. Do you use any client SDKs in TypeScript for integration, such as Shopify? They typically come with good error handling built in. This is a great question, actually. Like something that we thought about internally is, I believe, when we first created our Shopify integration, we did use the SDK. It does come with error handling built in.

But I think that over time we realize that as we build more integrations, it's more reliable to just have a common client, in our case like axios, just for making Http arrest calls, rather than using the pre built SDKs that are provided. Because with something like axios, we can build internal tooling across the board across different platform integrations. And that's kind of exactly what we did. We come into some sort of Http error with the Axios Library. We immediately know how to parse that regardless of what platform is during the error, because we have that common library across different platforms. So over time, we've realized that it's better to stray away from these client specific SDKs and just towards something that we kind of internally have built support around. Yeah, I'd also add on to there that especially when you're working with external dependencies, you kind of want to decrease your dependencies as much as you possibly can.

And so when we've used these client SDKs that are provided, especially if they're open sourced and not fully managed by the company themselves, we've just found a lot of problems and issues with those SDKs in particular. And so that has taught us over time to try to decrease our reliance on even more dependencies. Right, so you assume the integration itself is untrustworthy and then you're going to put an SDK on top of it that you also can't totally trust. You're just increasing the chances that something goes wrong and disrupts your customer service. How we handle rate limits, it really depends on the platform. The best platforms will have a very specific header in the response that says like, wait time until you can make your next request. When you hit a 429 rate limit exception.

Other platforms aren't so generous with that kind of knowledge and you have to dig through the docs and evaluate what even like, the rate limiting algorithm is based on what's generally available for different tiers of the merchant stores. And so some larger stores have more generous rate limits just because they pay more money to host their ecommerce shop. Other platforms, like Ebay, for example, they have a daily rate limit. So per endpoint, if you hit, I don't know, 5000 requests a day, you're absolutely done until the next day. You cannot make another successful request until the clock turns like, I don't know, midnight or something like that. And so how we've dealt with these rate limits really has depended, as you see, enough integrations. You can aggregate the different methods that platforms try to rate limit you into certain categories and build shared tooling around those.

But it is something that goes up in the loop every now and then just because it's such a different problem every time. Yeah. Just to finish a quick thought on the rate limits, one of the things that we found is like, it requires first a strong foundation and like good abstractions around error handling. Right? So you need like a common place in where you're handling all of the errors in order for you to find rate limit errors if those do come up. And then once we have kind of like a common class to handle those rate limit errors, it's just that we then have different implementations for different platforms because so many platforms have different implementations and so some people use the same thing, most of them don't, unfortunately. So we need to build around that. Some platforms don't even have any clear form of rate limiting whatsoever.

And so in those situations, we just kind of try to rely on exponential back off and kind of use that as like a best way to handle a bottleneck resource. How does the flow of data work? Do you dump the API responses in some intermediate storage or directly transform them plus update the final destination? Yeah, definitely dump the raw API responses in some intermediate storage before you do anything with it. If you were to transform the data and then store it in some final destination without knowing the original raw format, you're in for a bad time if you made any sort of mistake at all with the data transformations or anything. If you do store your transform data anywhere in storage, you will end up eventually having to run a ton of backfills where you're fixing a small bug or you're just updating some code due to some sort of platform change. And now you have all this raw data that you now need to run through your pipeline again to transform it and then restore. So we find that we extract the data that we need to in order to run operations like filters, in order to index data, et cetera. But we like to keep the data in its raw format otherwise and then do the transformations at the time they're necessary in order to just continue moving fast when there is any change in platform logic and data needs to be backfilled.

Okay? Yeah. So some of this comes down to some of the kind of more top level things that you need to deal with. We do put in like, the platform updated that time for any platform that supports it, and we do move that outside of the raw data dump and put it at the kind of like top level table. And that allows us to kind of query effectively against times and changes. But typically when we found things like changes in our own platform logic where we might need to backfill, it really depends on the number of connections that we're dealing with it. So if it's like a few connections, oftentimes you can actually get away. A lot of connections will fetch data within three or 4 hours.

And so it's not actually that much of a disruption of service. If you tell your customer like, hey, for a period of time we're going to backfill, we're going to remove this data, it's not going to be accessible, and then everyone and you just kind of delete the old data and refresh from the platform. Right, because one of the benefits of working with external dependencies is that the real source of truth is always the platform. And so it's very easy to actually throw out a bunch of data and re backfill if you need to, because it's always going to be the same or be accurately an accurate representation of what our customers want to know or what your customer teams will want to know. In the situation where there are a lot more connections, it's a little more complicated. And like that, we handle more on a case by case basis. But we might also just decide to change things for only a certain set of customers who really need those changes, set a certain date at which we need these changes to be accurate in our database, and move forward.

But whenever we can, we try to basically just regrow over the data. Using backfills, though, Frank knows a lot more about this. So, Frank, please fill in. Yeah. backfills is one of the times when a good abstraction from the start really helps. And so we've designed our system so that we can be flexible about the data that we're fetching at any given time. So we can fetch the last three months of data for a store, we can fetch the last six months, we can fetch between three months ago and six months ago, or we can fetch everything before a certain point, after a certain point, etc.

Or it was really important to us that we build an interface that we can work with so that we are flexible with what kind of operations we need to do because we run into every sort of situation. How do you structure your engineering team? Does everyone work on integrations? This is really going to be dependent on every single company. And this has changed around a few times for us as our team has grown and as our business objectives have changed. But at least at the moment, we have three teams. There's the Integrations platform team, the Integrations coverage team, and then also a kind of customer services team. And so integrations platform team is responsible for solving a lot of these core abstractions and core problems that affect every single platform, right? And so the fitness function that they're looking at is like keeping the number of bugs down and keeping it so that we're finding issues proactively and we have the tools and the systems so that we can find an issue, solve it an hour, 2 hours, so that our customers have the best service possible. The Integration coverage team is responsible for further building out coverage across new integrations or integrations we already have.

So their fitness function is on supported endpoints. There's a lot of conversation and it's a much more complicated issue as to what you count as a supported endpoint. But that team is much more focused on building out new integrations or expanding the integrations that we already have. So, like, for example, that team right now, one of the big things that they're focused on is building out, like, right, commerce and right, accounting for a lot of our customers that are interested in that functionality. And then finally the customer services team is responsible for kind of like, a lot of the things that sit on top of our integrations that our customers might interact with directly, such as our dashboard and other things like that, to basically make it an easier experience to build on top of Rutter and then also operate your business on top of Rutter. And so a lot of the problems that they're dealing with are like, how do we make it easier to see what the connection health status of what a particular connection is? And then they're also working on some more directly customer facing work around, helping to support our sales team with some important customers and their integrations. Last question when hiring for software engineers to build integrations, what are the most important skill sets you look for? Attention to detail would be the largest thing.

These platforms really test your attention to detail, I think with the Gotchas and nuances in both their documentation and in the responses they actually returned, whether they match the documentation or not. And so when building an integration, the most important thing you can do is really look at all the documentation and just inspect the actual data you're getting back, and then just a ton of manual testing, even. And so I think that attention to detail and just thoroughness are honestly the largest and most important skill sets. But it does depend on how much support an integration engineer has, right? If the platform is already built out, then I would say those two are the most important skills because the rest of the development can kind of go through these, like, cookie cutter motions that already have a lot of tools that are meant to help them build fast. If there's no foundation in place, then you definitely want an engineer who can really think on multiple levels of abstraction and help you kind of set up that foundation for success moving forward. I would definitely say that if you don't have any foundation, like having depending on the overall project size that you want to take on, whether it's like five integrations or 20 integrations, having people who are really strong when it comes to system design abstraction and like, building that foundation from the ground up is really important, especially in a way that can be future proofed as much as possible against N plus one. That N plus one problem of like, you start building Mercado Libre and then it breaks every assumption you ever had for me as well.

Another thing that I tag on to Frank's response is just that problem solving is really important, right? Because one of the biggest problems with integrations, especially when you're maintaining them, is just that things break all the time. And a lot of these integrations will have very minimal documentation, very minimal support, not super popular like developer support forums. And so you really, actually need to be pretty creative and structured about your problem solving process in order to figure out why something is going wrong when it starts going wrong, because things will not a matter of if, it's a matter of when. And there will be things that break in these integrations over time. And so you want someone who can solve those instances very well. Thanks, everyone.

Subscribe to newsletter
By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Stay up to date with the Rutter blog

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.