What is technical debt?

The term technical debt gets thrown around at any company that maintains any sort of code base, often used to explain to stakeholders the need to dedicate resources to refactoring. Generally I hear it summarized as expediting delivery by intentionally writing imperfect code, thus saving time. Eventually this debt has to be paid back eventually - either in development time to address it proactively or in the time to recover when things go wrong. However, financial debt has always felt like a poor comparison and at best miscommunicates the problem and at worst misleads the audience.

Financial debt is a bad analogy

Financial debt has little in common with what we call technical debt. Financial debt has to be paid back eventually, usually in a very defined schedule with specific consequences for not doing so. Code is not like that all; there is no “due date” for technical debt and it may never come. The internet is held together with scripts written in a weekend that have been running for years. You also usually don’t know the consequences of not refactoring. Sometimes the market for the product just ends and the code gets thrown away, never needing to be refactored.

There is also an element of intentionality when it comes to financial debt. You take out a loan with the goal of accomplishing something that has a higher rate of return than the interest on the loan (well, in theory, but that’s another blog post). With code, there are occasions where you intentionally make that decision. For example, I often work on an MVP and see an opportunity to use a design pattern to make a feature more scalable, but opt not to go that path because solving for scale isn’t needed at the time. If that need for scale comes suddenly, it will be harder to implement on that tighter timeline in the future, but we don’t handle it now because if we don’t deliver, scale won’t be a problem. This is what I think most of people consider to be technical debt. However, most technical debt is not intentional and as such we shouldn’t call it debt. We should call it “rot.”

Code Rot

The second code is delivered it has an expiration which could come due to known future scenarios and unknown external influences. For example:

• A security vulnerability could be discovered, requiring a rewrite several pieces of functionality
• Version upgrades could deprecate functionality that you depended on, requiring a work around
• Your product could start to be used in unexpected ways, creating bottlenecks in unexpected places
• Even lack of documentation causes code rot, expiring immediately the day the one person that understands everything decides to leave. (There’s no such thing as legacy code, just code that you didn’t write and now have to maintain!)
• You could have just plain not anticipated something. It happens all the time and should be expected

And kind of like food, there’s no defined expiration date. Sure, you know that the milk will go bad after a few weeks, but how is the flour after a year? And honey doesn’t go bad, right? If you aren’t careful, one day you will open the fridge and it will reek and you’ll have no idea where it’s coming from.

Keep your code fresh

If you want to keep your code fresh, you need to look through the fridge regularly. A really good practice is to get rid of functionality you don’t need. Code that doesn’t exist doesn’t go bad! The idea of “what’s the harm in keeping it?” is a poor one because all code must be maintained.

Log everything. You should have technical notes for each of your features, either in comments or in separate documentation. Call out places that are potential technical debt so that future people (including yourself) have an idea why something is the way it is.

Speaking of technical documentation, make sure it exists! I am a huge fan of code comments on methods explaining what’s going on and the intent. Runbooks are great too, i.e. documentation of how a feature is expected to be used. Keep your solution docs archived and make sure you have a system where your commits can be linked to a change request. Historical context is key to making sure you are addressing rot correctly.

Code rot is everyone’s problem

I often see technical debt used as a way to deflect on a problem - with technical stakeholders saying they don’t get enough time and resources to address issues and non-technical stakeholders saying they don’t have enough metrics to measure success. Cleaning out the code fridge is everyone’s problem.

If you are technical contributer, it is on you to track where the rot is, what needs to be done, and take a swing at some predications on the consequences of not doing so. If you’re a non-technical contributer, you need to consider that this time is table stakes for successfully delivering a product and be ruthless when it comes to prioritizing new work. If you cannot tell the technical stakeholders what the consequences of not adding a feature is, then the feature doesn’t need to be added.

This is all easier said than done - this is both a technical problem and a cultural one. I’m hoping if that if we start using a term that better fits the reality, then maybe we’ll be more successful at communicating it’s importance to the code we work on.

Why Host Your Own Server

With Mastodon becoming more popular, I wanted to see what was invovled in creating my own server. I figured if it didn’t work out I could join one of the bigger servers later, either by creating a new profile, or by migrating the profile from my own server. There are a few reasons that I think make hosting your own server a good fit for you.

• You want to have more control over your own content on the internet. While you may be able to move Mastodon servers, only your profile and followers go with you, NOT your posts
• You want to experiment with the technology itself
• You already have a bunch of people you want to follow and that want to follow you. Without this, it may be hard for your content to reach other federated timelines if people aren’t able to find you easily.
• You want to host a community and are committed to maintaining and moderating your server.

If you are going to open up registrations, that last piece is important. While people can migrate out of your server should you decide not to maintain the server anymore, I think it’s important to be upfront with your intentions so your users aren’t suddenly scrambling to find somewhere else to go.

Still Want to Host a Server?

There are several options for hosting. There are managed products such as masto.host that you can just sign up and they manage the infrastruture for you. Something like cloudron.io gives you a little more control in that you own the machine itself (even if just in the cloud), but they manage installing the product on your machine. I don’t know much else about these services, but they are available if you are not interested in diving in the command line.

I’m interested in the code itself, but was also intimidated by the idea of installing from source. Luckily Digital Ocean had something to meet my needs.

My setup experience.

I opted to host on Digital Ocean after I found their 1 click app. If you’d like, you can use my referal code to get some Digital Ocean credits for yourself and for me if you don’t have an account yet. Once you have an account, go to the 1 click app page and you can create a droplet. When you do, make sure to click a droplet size that has at least 2GB of RAM. I was running into a lot of issues in with a 1GB droplet and am currently running at 80% in the 2GB droplet.

During that setup, make sure to choose to upload an SSH key. Here’s instructions on how to generate an SSH key. I find it’s more secure than using a password.

While that is setting up, we’ll need a few other things.

First I needed a domain name. I used name.com to register mine, but feel free to use whatever provider you want. After getting my domain, I needed to configure the domain to in Digital Ocean. Here’s the link to the official documentation from Digital Ocean on how to setup your domain and DNS records.

I also needed an email provider. I tried to set up my your own email server on the droplet with the sendmail command, but my email were not being delivered to gmail. Whether it was an issue with my setup or gmail just blocking the emails, I’m not sure, but I eventually gave up. SendGrid has a free tier, which I think should be enough for a single user server. If you’re planning to open registrations you might want to shop around to find a provider that fits your budget. If you use SendGrid, follow the instructions in their documentation to set up an API Key.

With that all set up, let’s ssh into the droplet and install Mastodon. Here are instructions on connecting with SSH. Once you are connected, you’ll be greeted by the setup wizard and some prompts. Here’s what I put to configure my server. When things went wrong I was able to use disconnect from the droplet and reconnect to restart the wizard. The wizard appears to keep running until you succesfully complete it:

Welcome to the Mastodon first-time setup!
Domain name: forcedconversation.com
Do you want to store user-uploaded files on the cloud? No
SMTP server: smtp.sendgrid.net
SMTP port: 587
SMTP username: apikey
SMTP password: <my_send_grid_api_key>
SMTP authentication: plain
SMTP OpenSSL verify mode: none
E-mail address to send e-mails "from": Sean <[email protected]>
Send a test e-mail with this configuration right now? no

I didn’t want to configure S3 or some other cloud host, so all user uploaded files will live on the droplet. It’s just me so I figured it’d be ok. The only other thing of note is that the SMTP username is literally apikey.

After completing those prompts I was able to create my admin account:

Great! Saving this configuration...
Booting up Mastodon...
It is time to create an admin account that you'll be able to use from the browser!
Username: squizzleflip
E-mail: [email protected]

Next I was given my initial password, and prompted to setup Let’s Encrypt for SSL

You can login with the password: YOUR-PASSWORD-WILL-BE-HERE
The web interface should be momentarily accessible via https://forcedconversation.com/
Launching Let's Encrypt utility to obtain SSL certificate...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): [email protected]

This failed a BUNCH of times - turns out when I set up the DNS recodrs I had a typo in my domain name. A frustrating error, but at least not a complicated one.

After that I had to update certificates:

sudo apt install ca-certificates

Lastly I needed to add 127.0.0.1 localhost forcedconversation.com to the file /etc/hosts. And that was it! I was up and running!

Immediate Upgrades

I started posting on my instance and one of the first messages I got was “How did you manage upgrades?” to which I thought “Upgrades? I just installed it, what would I need to upgrade?” Turns out the Digital Ocean 1-click-app is set to Mastodon version 3.1.3, while the latest release is 3.5.3. COOL.

I connected to my droplet again and in another monitor and opened up the upgrade documentation. Fortunately you can update directly from 3.1.3 to the latest version, but the documentation does warn about making sure to follow any intermediary upgrade steps. For example, if version 3.2.1 has a unique upgrade setup, you’ll probably need to do that as well. I opened up all the releases and found I needed to do the following:

• Run apt install shared-mime-info as root
• Update ruby to 3.0.3 with RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.0.3
• Run bundle install
• Run yarn install
• Run SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate
• Run RAILS_ENV=production bundle exec rails assets:clobber assets:precompile
  • The docs don’t include the assets:clobber part, which caused me to run into this issue
• Run systemctl restart mastodon-sidekiq
• Run systemctl restart mastodon-web
• Run RAILS_ENV=production bin/tootctl cache clear
• Run RAILS_ENV=production bundle exec rails db:migrate
• Run systemctl restart mastodon-sidekiq
• Run systemctl restart mastodon-web

Once that was complete I was finally upgraded the latest version. Hopefully following those steps will make this a little more straightforward if you try this.

Lastly I set the server to single user mode by adding SINGLE_USER_MODE=true in the file /home/mastodon/live/.env.production. This disable registrations and points the domain directly to my profile.

What’s Next?

Things seem fairly stable right now, but I’m actually not sure what more work will be required. After a few months and some upgrades, I may decide to open up some limited registrations. If you want to try this yourself and have any questions, please feel free to tag in a post on @[email protected]! Good luck and happy tooting!

The Fediverse

I’ll start with I kind of hate the term “fediverse”, but I do like the concept! If you’re not familiar with a federated network, Mastodon, or how this “bird site replacement” works, I’ll try to explain in as few words possible. A federated network is distributed amongst separate servers that communicate with each other using a common protocol or “language” (called ActivityPub if you want to dig into the details). So while your account lives in one server, because of the common language across servers, you are able to follow and reply to people on different servers.

Mastodon is one type of software that uses a shared language to create a Twitter-like replacement, and because of the federated network, all of these Mastodon servers are able to interact with each other. You could join any server and you’d be able to follow/reply to people on other servers fairly seemlessly. There are also other types of software like Pleroma or Pixelfed (which is like instagram), that also implement the same protocol, so you can even follow people using those platforms and vice versa!

You can join any server you want, and if you’re unsure I’d say just start with one of the bigs ones like mastodon.social. The server you join doesn’t matter in the sense that you can communicate with people on other servers if you want, but it does matter in that it is where your posts will live and more importantly how the server is moderated.

Whoever is hosting a Mastodon server has complete control over that server. The admins can read any post on the server, including DMs. This is the case with something like Twitter, but the perception of trust can be different when it’s just a few hobbyists running a server as opposed to a giant corporation. Those admins also have control over which servers can interact with their server. For example, an admin on a Star Wars Mastodon server could block that servers interactions with a Star Trek server. So even though the Star Trek users are following the Star Wars users (and vice versa), the admin can decide that no Star Trek server posts will appear on their timeline.

Dunbar’s Number and a Better Online Experience

Having a distributed network means smaller user bases for each server. Instead of having to moderate the interactions and posts of millions of users on a single server, moderators only have to monitor the thousands or even just hundreds on their own server, while also having the ability to block communities in mass that don’t align with the values of theirs. Additionally, if an asshole suddenly becomes the owner of the instance you’re on, the fediverse’s portability allows you to leave a server and bring your followers with you to a server that you more align with.

The concept of many, small servers really appeals to me because I think it’s the best way to scale social networking that prioritizes safety because of the theory behind Dunbar’s Number. For those not familiar, Dunbar’s Number is the concept that an individual has a cognitive limit on the number of people they can maintain a stable social relationship, that number suggested to be around 150. To me, that means that I could potentially host a server of about 100 or so users and successfully priotize the safety of their interactions in my server. And because of the federated network, those users are not isolated to my server, they can freely interact with others in other servers. Again, if some asshat decides to buy his way into owning another server and it becomes a toxic hellscape, I can easily block that server’s posts from showing up in my server’s timeline.

If you’re familiar with Discord, which while not federated, you’ll see that it has a very similar feel. My online interactions on Discord are by far the least toxic and most meaningful in large part because of the small and separated nature of the servers.

Hosting My Own Instance

Currently I host a single user instance on forcedconversation.com. I’m not sure if I will open registrations; I’m not sure what is going to be required of me to keep this thing running, what the costs will end up looking like, what moderating posts will be like, or even if there is interest to join. However, I am intrigued by the idea, especially if there were others that would be interested in helping admin and moderate the server. For now feel free to come follow me @[email protected] and if you would be interested please send me a message! I’m really excited about this technology and I hope more people start to jump on board. This feels like what the future of social networking should be!