Modern single-page-apps are bad for developer fatigue

26 August 2020

To me there's no simpler way of creating a web app than a traditional server rendered web app. The user requests a page from your server, your code does some computation, then produces some output which is sent back to the user. We used to write web apps like this in the 2000's. There are frameworks written in PHP, ASP.NET, Ruby and Python and they all work this way. Times were good and life was easy.

At some point down the line we wanted to give our users a much better experience. The Single-Page-App was born - SPA. This meant we didn't wipe the webpage slate clean every time a user clicked a button, we keep some portions of the page and load other portions with new data. For our users this made our apps more intuitive, more understandable and it opened the web to a lot more people. However in exchange for doing this, we made our application's architecture incredibly complicated. Now we have to manually update pieces of the DOM - a canvas of mutable state. And now we had to juggle multiple pieces of asynchronous data.

What made React popular for writing UIs was the developer experience. With React you write your UI in terms of a single point in time, instead of amending your UI as time goes on. A page view is effectively a function of local state and request parameters. React meant we could have a lot more quality user interactions without full page reloads, and simultaneously being able to write them in a much more natural way that other frameworks didn't have.

Except somehow things are still not as simple as things were in Web 1.0.

Sure our React components may be defined as simple pure functions but we still need to provide those functions with data. Getting that data into the components is still pretty complicated. For a given page we often have to orchestrate multiple asynchronous requests, global state models and caches. We then have to ensure that the right React components update. React doesn't help us with this part so many different solutions have been invented in order to help.

These frontend frameworks have actually added extra cost for us and haven't fully brought back the ease-of-use. As much as I enjoy writing React apps, I feel there are a few problems that haven't been solved that are true for all modern SPAs.

There is a lot of extra busy-work involved in creating an SPA. Every new app you produce requires you to write a whole boat-load of custom code to patch together different APIs, libraries and state management. It's impossible to create a generalised solution.
A lot of work goes into replicating what our backend frameworks already did for us. Think about routing, validation and form submission.
If you don't plan enough upfront, it can also mean your business logic is duplicated across different technologies because of business logic affecting UI. This means extra work when responding to changing requirements and keeping things in sync.
Additionally, there's a lot more unnecessary decisions and choices a developer has to make. From what library to use, to how to marshal data around. These often don't have an obvious answer and getting some of these wrong may even cost a lot of technical debt further down the line.

All of this extra work and unnecessary complexity manifest in developer fatigue, costing more developer time. It also creates a much higher barrier for entry for newcomers. Think about how hard it is for a backend developer who wants to dip their toes into frontend.

I want to go into details of some of the complexity we added to our applications, complexity that I believe is completely unnecessary. I'll look at how much additional work we made for ourselves and see if we can reduce that work and complexity, while keeping the UX benefits.

API design is hard

If we choose to make a Single-Page-Application we have to decide how we fetch and receive data from our frontend. To do that we have to make one or more HTTP APIs. These APIs are actually pretty difficult to design and hard to change. Importantly, we can't find an off-the-shelf solution for this kind of thing because every application is different. I believe we waste a lot of time by dealing with these web APIs, and for 90% of web apps we could create a more general solution or even remove them entirely.

HTTP APIs are likely the most inflexible part of our codebase - we want them to change as little as possible. It's not as clear who calls HTTP APIs as it is with regular functions. There's also an overhead that each model must be serialised and deserialised at each respective end, tending to produce large amounts of code that does nothing except transform data into other data. It is often littered with bespoke details and can be a source of bugs (ever had problems with datetime fields?). We also may have to take care that the endpoints are backwards compatible, maybe even version them. You've also got to validate, authenticate and handle errors on your endpoints. And then fully test them.

The next problem in my mind is when business logic intersects with the UI. Business logic often changes so we often prefer these details to be isolated, so that changing these rules doesn't mean making large code modifications across your entire app. We certainly don't want business rules spread across the frontend/backend stack - this increases the risk of them becoming inconsistent and makes the code difficult to change. But if our business logic intersects our UI we must cross this inflexible API boundary for each feature we write.

I want to choose an example that may appear to be a bit unrealistic for the real-world. But the example will highlight the fact that often we receive requirements that we never anticipated in our current code base. Often the requirements cross a bunch of concerns that we never thought of. From a technological point of view it's certainly not uncommon to receive a requirement like this that combines UI presentation and several separate resources.

So in this example let's say we are running some sort of message board app, so users, comments and upvotes are part of our business logic. Let's say Tuesdays are special promotion days for our business and we want to highlight certain users on this day. Given the following requirement from our product manager, what fields might we add to our API and what different tradeoffs do we have to make?

Next to each user display a green icon if today is a Tuesday and the user has a comment with more than 20 upvotes.

As you can see there are a number of different possibilities with no perfect answer. Perhaps the best option here is isUserHighlighted - it reduces the need for the frontend to consider any business logic while still not having that much of a direct UI concern. However it's still not perfect - what if a requirement comes along that needs to change the icon yellow under some conditions? Do you change it to isUserHighlightedGreen/Yellow? isUserHighlightedPrimary/Secondary? iconHighlightedLevel? What type does it return? These changes will all need to be made across the whole stack because our business logic affects our UI.

Look at how many choices we have to make and code to modify to add a simple icon. For every feature we add to our UI there are lots of choices we have to make. Everything has a tradeoff depending on the rest of your codebase. Requiring a HTTP API to cross two separate codebases forces you to make this decision. Remember, in a traditional Django Views app we have everything we need directly in the view layer. We just... check the condition... and write the HTML.

def my_view(request):
  icon_src = 'no-icon.png'
  if current_day() == TUESDAY and user.comments.filter(upvotes__gt=20):
    icon_src = 'green-icon.png'
  
  return HttpResponse(f"""
    ...
    <img src="{icon_src}" />
    ...
  """)

Granted, in a production app you might add some abstraction here, perhaps an HTML template layer and maybe a controller layer, but the point is: it's technically possible to have this much simplicity if you want it, and any additional indirection has minimal overhead.

What if we could skip the API like Web 1.0, and delay any decisions of abstraction until our app needed it? If I'm just prototyping an idea or I'm at the early stage where things are changing often, I certainly don't want the overhead of these APIs. Of course if your app needs an API then so be it, but if the API is only there to send data to the frontend then must we need it?

Frontend complexity

The code in the previous section is simple - no serialisation of data and no APIs to create. But more importantly there is no frontend technology we have to mess with. There's no JavaScript to recompile, no Redux stores to juggle, no cache to invalidate. Just write the code.

When moving to a SPA we introduced more complexity. Not only do we have to write some more code, but at every step of the way we also have an overwhelming amount of decisions to make. Here is a diagram of a traditional React app.

In red is what I mentioned before - the new APIs that the front end needs to communicate with.

In blue are the modules that you have to write on a per-app basis.

Firstly, the page specific components. These are React components that hold the content of each of your pages and is the only actual meat of your SPA.
The API integration is the code that has to orchestrate different APIs, caching and prefetching. Maybe you want to try "one query per page" like GraphQL to minimise this orchestration? If you're writing composable components then your request needs to know what data every component on the page needs. GraphQL Relay has a concept of 'fragments' which forces you to handle all this data, but at the cost of another abstraction and another build step.
You have your routing module which interfaces with a routing library such as react-router. This may be simple or it may be complex.
Frontend model state. A lot of the data that your application needs will need to be stored across different pages. You'll have to design a data model for this data. Some of that data will be cached responses from API's but you'll need to figure out how to do that. Some data will be derived from that state and you'll have to figure out how to do that, and when to invalidate it. How you do this will depend on the state management library you use.
There is also glue code that just sets up communication with the other blue modules. That means more choices must be made. Does your routing layer call your API's before your page component is loaded? Or does it render the page and the page is responsible for fetching data? If there is interactivity in your pages then it may make additional calls to your API throughout the lifetime of the page. If you have routes that depend on something server-side some like authentication and permissions, your routing layer must coordinate with a new API endpoint for it.

And finally in green are libraries that have already been created and are reusable across multiple projects. For each of these boxes you may have to choose from a large number of libraries with varying feature sets and popularity. Note here that there are always new alternatives of each of these libraries appearing. It's exceedingly difficult to know which ones have the right features and will be the best for your app down the line. Whatever choice you make here may have large reaching ramifications of your application and may be hard to change later. There are also a set of build-tools you have to use, some of which are only necessary because of the web's shortcomings.

More decisions to make. More work to do.

If you want to read about some more specifics then Tom MacWright has a blog post that resinates strongly with me, diving deeper into some specifics. There is also the fact that splitting frontend and backend too much may create problems from a management point-of-view, which Rufus Raghunath from Thoughtworks looks at in his article.

All of these decisions to make and code to write for every single app we make. All of this busy-work when responding to changing requirements. What if the frontend was simply just done for us? We wouldn't have to pull in all these different libraries. We wouldn't write any glue code. We wouldn't even need to recompile any JavaScript.

Writing SPAs for the future

So we have apps where the business logic is spread out, we have to write APIs that can be a bottleneck of productivity, we have all these additional tooling and libraries to choose from that don't always fit our purposes. And to top it all off every single app we create requires loads of custom code. What can we do to mitigate these points? What if we reduce the amount of frontend code? We need to return to using the most of our backend frameworks.

You can do this already, in-fact. There are a few projects that bring the UI closer to the backend. These can use traditional server rendered architecture while also retaining parts of the UI across page loads. Projects like Turbolinks, Intercooler, Unpoly and PJAX do this by effectively replacing a subsection of the page's HTML. This I think, is the kind of idea we should be thinking about. The backend produces the page UI, the frontend handles individual components. This idea is extremely simple and it can be language agnostic. There are also some ideas out there that enable code sharing between server and client using NodeJS - Marko, and NextJS to some extent.

However those solutions either have very simplistic frontend involvement meaning they just swap out HTML directly in the page, or they require NodeJS on the backend. For many purposes one of these might work well, but as someone who does most of their backend work in Python I wanted a solution that works for Django (and Flask to some degree) and that interoperates with React to take advantage of the existing React ecosystem.

A bit about Django - developers using Django know that a lot is included which makes bringing up apps insanely fast. Templating, views, routing, authentication, permissions and internationalisation are all integrated extremely well and lots of third party libraries exist to supplement it all. Forms, input parsing and validation are all declaratively defined and in-fact, having the ORM tightly integrated into Django means you can even automatically derive them. Even somewhat controversially we can derive full pages from our database models. Django Admin didn't always get this right but you can't argue with the developer productivity boost when it works. Many businesses have full Django backends already set-up and many Django developers know these features inside-out.

Almost all of these things are completely thrown away when moving to a Single-Page-Application.

To solve this I'm working on a product that lets write your apps exactly as you are used to. It attempts to address almost all that's written in this article. I call it Spartan UI.

Spartan UI is Django+React library that allows you to take full advantage of the backend framework which means you write your routing, views, templates, forms, authentication and validation exactly like always have done. It allows you to create the user experience of a polished React SPA, while simultaneously requiring no frontend tooling and greatly reduces the need of any JavaScript experience. Developers need to create zero HTTP APIs or serialisation layers. The best part is that it will come with many beautifully created, customisable out-of-the-box templates and components that you can simply hook into your existing Django app.

I believe this is huge. This solution will be amazing for a certain class of applications, that is shopping cart flows, checkout flows, onboarding flows, CRUD apps, admin/dashboards, form heavy apps. Basically anywhere where you have a heavy backend involvement but also want those UX features of SPAs and quality React components such as dropdown menus, modals, advanced form inputs, autocomplete, multi-page flows, drag-and-drop, filtering, etc.

The templates included aren't simply frontend HTML - since the library integrates the frontend and backend together, we can provide complete flows that hook in directly to your database ORM. Think a dashboard where searching and filtering data is all set-up for you, or an onboarding flow that is generated automatically from the data you need from a user. This is simply impossible with today's current architecture. This means the time it takes to get your SPA up and running will be hours rather than weeks/months.

Ongoing developer time is heavily reduced as well because changing requirements doesn't affect the whole stack, allowing you to focus on whats important. Developers that have adversity to JavaScript and the tooling that comes with it can still create great user experiences. The unnecessary work and frontend complexity that I've outlined in this article is almost completely eradicated. On the other hand if you do need bespoke React components or want to integrate with existing libraries you can optionally do so exactly as you're used to.

I'm extremely excited to share this product, however, I need something from you readers. In order to provide a polished set of out-of-the-box templates and examples that cover the common use cases I need feedback. I want to get to understand the feature set of what should be included. I also want to get in touch with those of you who resinate with what I've written here and who might be interested in my product. I'd like to start a discussion with those who are interested in this technology and get some ongoing feedback.

If this is for you please get in touch below and sign up for future updates.