Prerequisite readings:
Middleware is one of the most confusing and misunderstood topics in all of technology, yet it is the most important for business stakeholders to understand.
Most companies today have a variety of different software applications working behind the scenes to perform critical functions. Businesses in any industry rely on a large variety of software applications from shipping platforms, to email service providers, to HR platforms. However, due to companies needing such a diverse tech stack with a variety of features, transferring data from one platform to another becomes a common technical business problem.
As a Sales Engineer at Klaviyo, I’ve run into this more times than I can count. Figuring out how to connect platforms and move data from one to another is a priority for companies who want to implement another tool or application. To help demonstrate this concept, I’ll give an example through a client experience.
My client sold solid colored curtains for nurses. The curtains went behind nurses when they were on Zoom to help provide a more professional background while working remotely. The goal was to help this company sell more orders online and figure out a better way to communicate these orders to the manufacturer.
The process of a customer purchasing an item and the order being sent to the manufacturer required two different software applications to work together. One was the eCommerce website that took the orders, and the other was the technology the manufacturer used to register the orders. Unfortunately, when we started this project, we found out that these functions were being performed manually. This meant that every day, someone on the team would export all of the orders from the past day into an Excel file, and then would send that data over to the manufacturer.
This process takes a lot of time and manual energy for everyone involved. It’s an unsustainable use of resources for the manufacturer to rely on manual import and for the marketing team to manually export the file. It also provides a poor customer experience, because now a customer isn’t able to receive an order confirmation or follow up email on the day of purchase.
As a result, I was looped in to help automate and improve this process. I started an email chain with both the manufacturer and the eCommerce store and quickly learned of the restrictions involved and reasoning behind manual input. The reason the process wasn’t automated was because the manufacturer’s software only accepted JSON formatted files whereas the eCommerce store only allowed data to be exported into XML.
Note: JSON and XML are just two different file types, learn about them in my API article.
This can be a frustrating problem as a vendor. XML and JSON are two different file types that perform the same function, so the fact that neither software could support the other posed a challenge.
As a consultant, I immediately thought of remediation options. Upon a quick brainstorm, I realized the three ways of fixing this were:
The manufacturer’s software developing a way to handle XML files.
The eCommerce platform developing a way to export JSON files.
Creating middleware that’s hosted on a server with a script that performs a daily automated ETL process to get the data from the eCommerce store to the manufacturer’s software. (I know this wording is confusing… I’ll break this down below).
The first two options are immediately out. Even though making an improvement to be more flexible and accommodate this use case would be beneficial for either company, I couldn’t rely on these applications to change the way they did things just for one customer’s use case. Due to this, I realized number three was the best option.
The first term of the third option, “middleware” refers to the project as a whole that connects one platform to another. The script, server, and automation that are required for this project to work are all wrapped into the term “middleware”.
Creating the middleware for this project required three steps.
Spin up* a server to run a script.
This process is necessary, because once a script is created it will need to live somewhere that’s always available at the time it needs to run.
Remember a server is just a computer hosted by someone else, so this step is actually optional, but very important. One reason why this process should be on a server instead of my computer is that if I were to set up this script to run at noon every day on my computer, then my computer would need to be on at noon everyday for it to run.
Imagine I go on vacation one day and close my computer by accident before noon. This would prevent the script from running, and no orders that came in that day would be sent to the manufacturer.
In order to prevent that, having a script hosted on a server is critical to ensure reliability.Create a script to perform an ETL process, or more specifically Extract data from the eCommerce store, Transform it from XML to JSON, and Load it into the manufacturers software.
A script is the term used for a document consisting of code. To write any sort of program a script is necessary. My language of choice is Python but whether this specific script is in Python, Ruby, Javascript, Go, PHP, etc.. doesn’t really matter.
The “ETL Process” is a common process for programmers, where they “extract” or take data from one system, “transfer” or structure it in a way the other system is expecting, and “load” or send it into the receiving system.
Since JSON and XML are common technologies, pre-existing scripts and open source work can most likely be used here, meaning that the development for the script in question will be a lighter amount of work.
Automate the script on the server.
Every server has a time based scheduler called CRON installed. It’s very easy to use and implement, meaning that once the server and script is set up, the programmer just has to turn on and configure a “CRON job” (or “CRON tab”) to run the script at a specific time of day.
A CRON job can be configured to run as many times as needed in a day, anywhere from twice a minute to once a day. The only caveat is the more times a day that a script needs to be run, the more expensive the server costs.
*A server is just a computer living somewhere else. A commonplace technical term for turning one on and owning it is framed as “spinning up a server”.
Now that you have an idea of what the work looks like, it’s time to create a statement of work. This is the important part of the article for any business decision maker. It is crucial to have the ability to read a statement of work, understand the job at hand, and get a gauge on whether the technical consultant is giving both parties a fair estimate with the best possible solution. It allows the business decision maker to understand if the solution is actually solving their problem at the appropriate price.
For this project, my statement of work would consist of the above three items with time estimates. Laid out simply looks like:
If my hourly rate was $200, I would charge $2,000 for this job. Any less wouldn’t be planning for potential contingencies and anymore would be a robbery for a script of this size.
Some things to look out for:
If the actual automation of the script or creation of the server are taking up more time than listed above, make sure to push back and get an understanding of what the developer is doing.
If the project isn’t well scoped out, and has a more general “20 hours” component, rather than mentioning specific aspects, make sure to scope out with the developer more of what they’re actually doing.
If scripts that do simple one-off tasks take longer than a 5-10 hours, then make sure to give a bit of push back and get a good understanding of why they’ll be more complicated.
The above doesn’t include more enterprise related concerns like security, scalability, or reliability. That’s where these cases can get very complicated and expensive. It’s always best to have an in-house developer or trusted technical consultant if a script of that complexity is needed.
If I were the purchaser of this software and this was the only job I had with the technical consultant, then I’d also ask for the following details:
Server credentials. Without these, you’re solely dependent on the developer who made the script for edits.
Documentation of the script. This will make it easier for future developers to edit any aspect of it.
In the event that the script ever breaks, needs edits, or requires maintenance, any developer should be able to pick up this job when supplied with these two pieces of information.
This article discusses what middleware is, why it’s needed, what developers take into consideration when starting a project, and an outline of a statement of work one can expect when contracting a developer. Having a good grasp of this knowledge can help streamline development jobs and can help you make sure that you’re getting a job for the right price.