ayush sharma (づ｡◕‿‿◕｡)づ

Consuming APIs responsibly

Nov 21, 2022

Building the Fediverse.to requires hitting the API endpoints of thousands of servers. Some of these servers are huge, some are tiny, and almost all of them are managed by just 1 person or a small team. Being a sysadmin myself I know how painful it is for a server to crash due to mysteriously large amounts traffic from seemingly nowhere, so I put together a small list of dos and donts to make sure my hobby project doesn’t ruin someone’s night. What best-practices and guidelines should I follow when polling a diverse mix of other people’s servers? How can I make sure that my work doesn’t cause unintended consequences?

First do no harm

The first part of the etiquette is ensuring that you don’t thrash the API server. The best way of doing this is to assume it has a small amount of compute/storage/network and serves other clients more important than you. Be kind. The best course of action is taxing the server as little possible to allow it to serve other requests.

Network performance

Make requests in series rather than in parallel which will let the API server save its threads/cores for more important things. This is a judgement call since it will slow down your app/client considerably but that’s not the worst thing. Your work may be important but not mission-critical.

The API might also support fetching multiple records in a single request. For example, there may be a provision to supply multiple record IDs and get the array of results in one response (like GraphQL). This ensures two things: first, the API might be able to, for example, fetch the data in a single SELECT with multiple WHERE conditions lowering database transaction costs. Second, it saves multiple network calls. Use the network conservatively and batch requests wherever possible although your mileage will vary.

The fediverse.to backend uses the requests library which supports gzip compression by default, meaning if the server supports it, there should be fewer bytes-on-the-wire. The backend also caches the responses for re-use leading to fewer requests to the API server.

Let the UX drive performance

If your application is still good with 24 hour old data, don’t fetch it every 5 minutes just because you can. Think about the product you’re building and what your data freshness requirements are. This can provide huge performance gains to the target API server. The API requests you don’t make are as important as the ones you do.

Backoff and retry

If the server is unable to comply with your request for data, wait for a generous amount of time before trying again. It’s possible the server is using a small amount of resources and immediately retrying a deadlocked request isn’t helpful. Use exponential backoffs and retries to give the server time to recoup and recover before requesting content again.

When all else fails…

Stay reachable

When (not if) you end up breaking someone else’s baby, make sure you’re reachable. Your website/domain should contain your contact information so you’re reachable in an emergency.

Your user-agent string should let the server admin know what technology you’re using to hit their API. The format User-Agent: <product> / <product-version> <comment> is simple and informative.

The FROM header field should contain an email address where the server admin can reach you.

Use a reputable hosting provider

Most hosting providers publicly disclose the IP address range that they use. So if your code starts misbehaving, your IP address can be traced back to your hosting provider. And if your hosting provider has a good support team, they can reach you and help you fix things. I was recently contacted by the DigitalOcean support team about some issues one of my scripts caused. Apparently my Python client was hitting too many non-existent URLs on an API server and the server admins got in touch with DigitalOcean. I identified the problem, fixed it, and everyone lived happily ever after.

Create a series of posts with navigation in Jekyll

Feb 18, 2022

Blogging about individual self-contained ideas is great. But some ideas require a more structured approach. Chaining together simple ideas into one big whole is a great journey for both the writer and the reader. Which is why I wanted to add a Series feature to this Jekyll blog. As you may have guessed already, Jekyll’s high degree of customsation makes this a breeze.

Goal

What I want to achieve is:

Each article should list the other articles which are part of the same series.
To simplify content discovery, the home page should display all series in a category.
Moving articles into different series should be easy since they may evolve over time.

Step 1: Adding series metadata to posts

Given Jekyll’s high customisability there are several ways to handle a series. Either we can leverage Jekyll variables in the config to keep a list of series. Or we can use collections. Or define a Liquid list somewhere in a global template and iterate over it.

The cleanest way is to keep the series along with the posts contained in that series. For example, for all the posts in the Jekyll series, I’ve added the following 2 variables in the post front matter:

is_series: true
series_title: "Jekyll"

The first variable, is_series, is a simple boolean which says whether this post is part of a series or not. Booleans work great with Liquid filters and will allow me to filter only those posts which are part of a series. This will come in handy later on when we’re trying to list all the series in one go.

The second variable, series_title, is the title of this series, in this case Jekyll. It’s important that posts in the same series contain the exact same title. We’ll use this title to match posts to a series. If it contains extra spaces or special characters it won’t match the series.

You can view the source-code for this here.

Step 2: Adding links to posts

With our series defined, we now need to show other articles in the series. Meaning if I’m seeing a post in the Jekyll series there should be a list of other articles in the Jekyll series. A series won’t make sense without this essential navigation.

This blog uses the posts layout to display posts. To display other posts in the same series as the currently viewed post, I’m using the code below:

<!-- Series links -->
{% if page.is_series == true %}
<h3 class="text-success p-3 pb-0">{{ page.series_title | upcase }} series</h3>
{% assign posts = site.posts | where: "is_series", true | where: "series_title", page.series_title | sort: 'date' %}
 
{% for post in posts %}
        {% if post.title == page.title %}
 <p class="nav-link bullet-pointer mb-0">{{ post.title }}</p>
        {% else %}
 <a class="nav-link bullet-hash" href="{{ post.url }}">{{ post.title }}</a>
        {% endif %}
{% endfor %}

{% endif %}

The logic above goes as follows:

Check if the is_series boolean of the current page is true, meaning the post is part of a series.
Fetch posts where is_series is true and series_title is the current series_title. Sort these in ascending order of date.
Display links to other posts in the series. Or show a non-clickable span if the list item is the current post.

I’ve stripped some of the HTML for clarity but you can view the full source-code here.

Step 3: Adding links to all series to home page

We now have the post pages showing links to other posts in the same series. Next, I want to add a navigation to all series under a category on my home page.

For example. On the home page, the Technology section should show all series in Technology. The same for Life Stuff, Video Games, and META categories as well. This will make it easier for users to browse and read a complete series.

<!-- Series posts -->
{% assign series = "" | split: "," %}
{% assign series_post = "" | split: "," %}
{% assign posts = site.posts | where:"Category", cat.title | where: "is_series",true | sort: 'date' %}

{% for post in posts %}
{% unless series contains post.series_title %}
{% assign series = series | push: post.series_title %}
{% assign series_post = series_post | push: post %}
{% endunless %}
{% endfor %}

{% if series.size > 0 %}
<div class="row m-1 row-cols-1 row-cols-md-4 g-3 align-items-center">
 <div class="col">
 <span class="h3 text-success">Article series →</span>
 </div>
    {% for post in series_post %}
        {% include card-link.html url=post.url title=post.series_title %}
    {% endfor %}
</div>
{% endif %}
{% endfor %}

To identify all series for a particular category, I’m using the code above, which:

Initalizes two variables: one for series names and another for the first post of each series.
Fetches all posts which have is_series set to true and which belong to the current category.
Adds the series_title to the series names array and the first post to the series post array.
Finally, it displays the name of the series which links to the first post in that series.

You can find the full source code here.

Conclusion

Jekyll’s high degree of customisation is why I enjoy working with it so much. And why this blog’s underlying Jekyll engine has survived redesigns and refactors. Jekyll makes it easy to add dynamic logic to your otherwise static website. And while my website remains static, the logic that renders it doesn’t have to be.

There are many improvements you can make to what I’ve shown you today.

One improvement I’m thinking of is handling series post ordering. For example, right now the posts in a series show in ascending order of their publish date. But I’ve published several posts belonging to a series at different times. So I can add a series_order key and use it to order articles rather than their publish date. This is one of the many ways you can build your own series feature.

Happy coding :)

A practical guide to light and dark mode in Bootstrap 5 and Jekyll

Jan 25, 2022

Adding a light and dark mode to my side project www.fediverse.to was a fun journey. I especially loved how intuitive the entire process was. The prefers-color-scheme CSS property contains the user’s color scheme - light or dark. We then define SASS or CSS styles for both modes, and the browser applies the style the user wants. That’s it! The seamless flow from operating system to browser to website is a huge win for users and developers.

After tinkering with www.fediverse.to (and the resulting post blowing past 50K+ views!) I decided to add light and dark modes to this website as well. I began with some internet research on how to best approach this. This GitHub thread shows the current progress of the feature. And this in-depth POC demonstrates how challenging the process can be.

The challenge

The biggest challenge is that sometimes SASS and CSS don’t play well with each other.

Let me explain.

From my earlier post on light and dark themes, to create both styles we need to define CSS like this:

/* Light mode */
:root {
   --body-bg: #FFFFFF;
   --body-color: #000000;
}

/* Dark mode */
@media (prefers-color-scheme: dark) {
   :root {
       --body-bg: #000000;
       --body-color: #FFFFFF;
   }
}

This is simple enough. With the styles defined, we use var(--body-bg) and var(--body-color) in our CSS. This causes the colors to switch based on the value of prefers-color-scheme.

Bootstrap 5 uses Sass to define color values. My website’s color scheme in _variables.scss looks like this:

// User-defined colors
$my-link-color: #FFCCBB !default;
$my-text-color: #E2E8E4 !default;
$my-bg-color: #303C6C;

The solution seems obvious now, right? We can combine prefers-color-scheme with the variables above, and boom!

/* User-defined colors */
:root {
    --my-link-color: #FFCCBB;
    --my-text-color: #E2E8E4;
    --my-bg-color: #303C6C;
}

/* Dark mode */
@media (prefers-color-scheme: dark) {
    :root {
        --my-link-color: #FF0000;
        --my-text-color: #FFFFFF;
        --my-bg-color: #000000;
    }
}

Additionally, we need to replace the $ values with their -- variants in _variables.scss. After making the change and running jekyll build, we get the following:

Conversion error: Jekyll::Converters::Scss encountered an error while converting 'css/main.scss':
                    Error: argument `$color2` of `mix($color1, $color2, $weight: 50%)` must be a color on line 161:11 of _sass/_functions.scss, in function `mix` from line 161:11 of _sass/_functions.scss, in function `shade-color` from line 166:27 of _sass/_functions.scss, in function `if` from line 166:11 of _sass/_functions.scss, in function `shift-color` from line 309:43 of _sass/_variables.scss from line 11:9 of _sass/bootstrap.scss from line 1:9 of stdin >> @return mix(black, $color, $weight); ----------^ 
             Error: Error: argument `$color2` of `mix($color1, $color2, $weight: 50%)` must be a color on line 161:11 of _sass/_functions.scss, in function `mix` from line 161:11 of _sass/_functions.scss, in function `shade-color` from line 166:27 of _sass/_functions.scss, in function `if` from line 166:11 of _sass/_functions.scss, in function `shift-color` from line 309:43 of _sass/_variables.scss from line 11:9 of _sass/bootstrap.scss from line 1:9 of stdin >> @return mix(black, $color, $weight); ----------^ 
             Error: Run jekyll build --trace for more information.

The error means that the Bootstrap mixins expect color values to be, well, color values. And not CSS variables. From here we can dig down into the Bootstrap code to rewrite the mixin. But we’ll end up rewriting most of Bootstrap to get this to work. This page describes most of the options available to us at this point. But I was able to make-do with a simpler approach.

Since I don’t use the entire suite of Bootstrap features, I was able to add light and dark mode with a combination of prefers-color-scheme, some CSS overrides, and a little bit of code duplication.

Step 1: Separate presentation from structure

Before applying the new styles to handle light and dark mode there was some clean up to perform on the HTML and CSS.

The first step is ensuring that all the presentation layer stuff is in the CSS and not the HTML. The presentation markup (CSS) should always stay separate from the page structure (HTML). But a website’s source code can get messy with time. If your color classes are already separated into the CSS you can skip this step.

I found my HTML code peppered with Bootstrap color classes. Certain divs and footers were using text-light, text-dark, bg-light, and bg-dark within the HTML. Since handling the light and dark theme relies on CSS, the color classes had to go. So I moved them all from the HTML into my custom SASS file.

I left the contextual color classes (bg-primary, bg-warning, text-muted, etc.) as-is. The colors I’ve picked for my light and dark themes would not interfere with them. Make sure your theme colors work well with contextual colors. Otherwise you should move them into the CSS as well.

So far I’ve written 100+ articles on this site. So I had to scan all my posts under the _posts/ directory hunting down color classes. Like the step above, make sure to move all color classes into the CSS. Don’t forget to check the Jekyll collections and pages as well.

Step 2: Consolidate styles wherever possible

Consolidating and reusing styling elements will ensure we have less to worry about. My Projects and Featured Writing sections on the home page were displaying card-like layouts. These were using custom CSS styling of their own. I restyled them to match the article links and now I have less to worry about.

There were several other elements using styles of their own. Instead of restyling them, I chose to remove them.

The footer, for example, was using its own background color. This would have required two different colors for light and dark theme. I chose to remove the background from the footer to make the migration easier. The footer now takes the color of the background.

If your website uses too many styles it might be prudent to remove them for the migration. After the move to light/dark themes is complete you can add them back.

The goal is that we want to keep the migration simple and add new styles later if required.

Step 3: Add the light and dark color schemes

With the clean up complete we can now focus on adding the styling elements for light and dark themes. We define the new color styles and apply them to the HTML elements. I chose to start with the following:

--body-bg for the background color.
--body-color for the main body/text color.
--body-link-color for the links.
--card-bg for the Bootstrap Card background colors.

/* Light mode */
:root {
    --body-bg: #EEE2DC;
    --body-color: #AC3B61;
    --body-link-color: #AC3B61;
    --card-bg: #EDC7B7;
}

/* Dark mode */
@media (prefers-color-scheme: dark) {
    :root {
        --body-bg: #303C6C;
        --body-color: #E2E8E4;
        --body-link-color: #FFCCBB;
        --card-bg: #212529;
    }
}

With the colors defined I changed the CSS to use the new colors. For example, the body element now looks like this:

body {
    background-color: var(--body-bg);
    color: var(--body-color) !important;
}

You can view the rest of the CSS changes on GitLab.

You can override Bootstrap 5 defaults if it’s compiled with your Jekyll source and not from the CDN. This might make sense to simplify the custom styling you need to handle. For example, turning off link decoration made life a little easier for me.

$link-hover-decoration: none !default;

Last but not the least: the navbar toggler. In Bootstrap 5, navbar-light and navbar-dark control the color of the toggler. These are defined in the main nav element along with .navbar. Since we’re not hard-coding color classes in the HTML anymore, we’ll need to duplicate the CSS in our own. So I extended the default Sass and added my theme colors.

.navbar-toggler {
    @extend .navbar-toggler;
    color: var(--text-color);
    border-color: var(--text-color);
}

.navbar-toggler-icon {
    @extend .navbar-toggler-icon;
    background-image: escape-svg(url("data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 30 30'><path stroke='#000000' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/></svg>"));
}

The code above is the default Bootstrap 5 toggler CSS code, with some minor changes. One thing to note here. For the toggler icon I’m hardcoding stroke='#000000' since black works with my theme colors. You may need to be more creative about picking colors schemes that work well across the board.

Enlarge — Image showing www.ayushsharma.in in dark mode on the left and in light mode on the right.

And that’s about it! The light and dark modes now work as expected!

Enlarge — Image showing www.ayushsharma.in in dark mode in desktop view.

Enlarge — Image showing www.ayushsharma.in in light mode in desktop view.

Enlarge — Image showing www.ayushsharma.in in dark mode in mobile view.

Enlarge — Image showing www.ayushsharma.in in light mode in mobile view.

Conclusion

Bootstrap 5 is complex to say the least. There is lots to think about when overriding it with your custom styling. Providing a light and dark variant for every Bootstrap 5 component is difficult. But it’s possible if you don’t have too many components to deal with.

By ensuring that the markup stays in Sass/CSS, reusing styles, and overriding some Bootstrap 5 defaults, it’s possible to achieve light and dark modes. It’s not a comprehensive approach but it is practical. And serviceable until Bootstrap 5 decides to provide this feature out of the box.

I hope this gives you more practical ideas on how to add light and dark themes for your own website. If you find a better way using your own CSS magic, don’t forget to share it with the community.

Happy coding :)

Nemo - The Ubuntu file manager you didn’t know you needed

Jan 25, 2022

So… I finally got bored with Nautilus.

Ubuntu has been my daily driver for the last several years. I remember when Linux used to be a second-class citizen in the desktop space, only used by nerds and devs who could unravel its arcane and obscure mysteries. But it has come a long way. Not only is it a lot more user-friendly with a plethora of customisation options, but Steam, and especially Proton, has added first-class gaming support for Linux. I haven’t had a dual boot Ubuntu/Windows system in quite a while and that’s a huge win.

Nautilus seems out of place in this brave new world, like it’s still stuck in the old way of doing things. It’s become more stable, sure, but it doesn’t match up to Windows’s Explorer or Mac’s Finder. So a few days ago I just searched for “best ubuntu file manager 2022”… and that’s how I found Nemo.

Nemo (wiki, GitHub) is the file manager for the Cinnamon desktop environment. It’s beautiful, snappy, and useful. It’s beautiful, snappy, and, erm, useful. Compared to Nautilus, browsing remote networks is faster and the feature-set more comprehensive.

Enlarge — Image showing Nautilus file manager on the left and Nemo file manager on the right. Using Ubuntu 20.04 LTS.

Feature list

These are some of the features of Nemo that I found most useful:

Type-ahead search.
Menu bar! Plus customizable toolbar, customizable tooltip, and customizable date format.
You can open a separate panel to browse files, besides new tabs and windows.
The sidebar has a tree view! Also shows filesystem free space graphically.
Copying, moving, or extracting files shows progress in a separate mini-window (like Mac). A queue manages all file operations.
Advanced bookmarks editor - like the one in Firefox.
Separate view settings for separate folders.
The status bar shows number of files and available disk space.
File renames are in-place (like Mac) rather than in a popup.
Trash customisation - bypass Trash on delete or add a separate button to bypass Trash.
Built-in bulk-rename (though I haven’t used this one yet).
Extensions!

Installing Nemo

Installing Nemo is about as easy as you would expect. Do this:

sudo apt install nemo

And voila! You should now have another “Files” app in your Applications Menu. Keep Nautilus around till you’re comfortable with Nemo, and then you can nuke Nautilus for good.

Set Nemo as the default file manager

To set Nemo as the default file manager, do this:

xdg-mime default nemo.desktop inode/directory application/x-gnome-saved-search
gsettings set org.gnome.desktop.background show-desktop-icons false
gsettings set org.nemo.desktop show-desktop-icons true

Screenshots

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with View menu open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Bookmarks menu open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with View preferences open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Behavior preferences open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Display preferences open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Toolbar preferences open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Context Menu preferences open.

Enlarge — Image showing Nemo file manager on Ubuntu 20.04 LTS with Plugins preferences open.

Enjoy your new Ubuntu :)

Announcing Fediverse.to!

Jan 1, 2022

Over the last few weeks, I’ve been working on a search engine for Fediverse instances!

The idea was to help new users, Facebook refugees, and Twitter rage-quitters find their ideal home on the fediverse. Fediverse instances are either for a wide audience or centered around a theme, a language, a hobby, or all of the above. The fediverse ecosystem has been growing like crazy - there are now close to 5000 instances with 4 million users. And finding an instance can be quite daunting.

While instances.social already has a great API and filtering mechanism, I wanted to build something more visual. I began by polling the original API to build a visual search feature using Lunr.js, but it has now become something bigger, thanks to feedback and support of the community.

Fediverse.to!

If you visit fediverse.to, you’ll see the new face of what was originally findmymastodon.com.

The branding has changed - the site uses the fediverse logo (link to proposal).
The home page now has a larger selection of instances for new users to choose from, with dedicated sections for some languages and categories.
Every instance now has its own dedicated page which, at the moment, shows basic statistics about that instance.
The home page shows newest instances at the top to support new members of the federation.

Apart from the above, the original features are still there:

Support for light and dark theme.
Display total users, total status updates, and weekly active users for all instances.
Display description, language, categories, and prohibited content settings for all instances.
The search features allows searching for specific text in instance names and descriptions.
Filtering by language, category, content blocks, minimum users, and maximum users is supported.
The website visitor stats are public thanks to Plausible.
The website is still an open-source static website deployed using Netlify.

A new engine under the hood

Under the hood, though, is where the real changes have happened.

Originally, I was using a small Python script to scrape the main API and generate a Lunr search index. The front-end website would then use JavaScript, Bootstrap, and Lunr to display visual cards for search results. There was also a basic filtering mechanism to mirror the original instances.social API which still exists.

Now fediverse.to uses a Django + PostgreSQL back-end to dynamically generate Jekyll pages for each instance.

First, the API is used to fetch a list of instances. Then Django visits each instance’s metadata URL and pulls its information and the list of domains it peers with. The process is then repeated for all the peers. This way I can be certain I’m not missing any instance in the network. Once this process completes, Django generates a Jekyll page for each instance.

For example, if you browse Mastodon.social’s Jekyll page, you’ll see the following:

layout: instance
page: /
name: "mastodon.social"
instance_url: "mastodon.social"
short_description: "Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!..."
description: "Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!"
admin: "staff@mastodon.social"
admin_email: "staff@mastodon.social"
statuses: 32950135
active_users: 599995
users: 599995
statuses_display: "33M"
active_users_display: "600k"
users_display: "600k"
languages: en
categories: 
thumbnail: "files.mastodon.social/site_uploads/files/000/000/001/original/vlcsnap-2018-08-27-16h43m11s127.png"
registrations: True
approval_required: False
version: "3.4.3"
created_at: 1638080432.0

Future plans

The idea behind generating a separate page for each instance is simple - it allows the site to grow! I’m planning to add the following features soon:

Use D3 to show the entire fediverse network using network/peer visualizations.
Add more information for each instance, such as the fediverse software its running (Mastodon/Misskey/etc) and the country its hosted in.
Parse the instance domain for country information, such as .de and .fr, and categories, such as .tech and .art.
Parse the instance descriptions for country and category information.
Build an API for other users and admins to scrape.
Host everything on a remote server and automate content updates using cronjob or django-crons.

Eventually I’m hoping fediverse.to can become its own data source and the de-facto directory for the new internet - but that’s still a long way off :)

Work to be done

Aside from new features, there’s still work to be done to make fediverse.to fully functional.

The search feature currently has some bugs to iron out. I’m still working on the new back-end to generate the Lunr search index. I debated whether or not to release the site in its current state, but the earlier I get feedback the better the site will be. I’m hoping all of you will bear with me just a little while longer.

Stay tuned!

Anyhoo. That’s the general plan. fediverse.to is my first open-source project so I’m learning as I go along. But you’ve all been very supportive with ideas and feedback and I’m already excited about version 2.0!

So wish me luck… and stay tuned for more! I’ll post future updates here, so follow me using the link on the, erm, top right?

And before I forget… spread the word!

Thanks, Ayush.

Easy pretty URL redirects with Jekyll and Netlify

Nov 16, 2021

Pretty URLs, aka vanity URLs, aka tiny URLs, are great for improving SEO and the UX. They’re shorter versions of a domain, making them useful for copy-pasting and printing. There are already plenty of domain shortening services out there. But with Jekyll and Netlify I can configure pretty URLs all by my lonesome!

What I want to achieve is as follows.

Every article on this website has a number in its Front Matter. I want to use these sequence numbers to redirect to the main post. Using numbers as pretty URLs will make it easier for users to type it into the address bar.

The plan

My article on Getting started with ArgoCD is article 94. So I want https://<my-pretty-domain>/94 to redirect to the full article. Any other link for https://<my-pretty-domain> which doesn’t contain a valid article number should redirect to the primary domain, ayushsharma.in.

By breaking down the plan into individual tasks, I get the following list:

Register pretty domain and point it to Netlify.
Redirect pretty URLs for articles to full URL.
Redirect all other pretty URLs to primary domain, ayushsharma.in.

Register pretty domain and point it to Netlify

The first tasks is to get a pretty domain.

I’ve registered ayush.to as my pretty URL and pointed it to Netlify. There are several options on how to point a domain to Netlify’s load balancers. I’m pointing ayush.to’s A record to Netlify which you can verify with nslookup.

> nslookup ayush.to

Server:		10.1.0.1
Address:	10.1.0.1#53

Non-authoritative answer:
Name:	ayush.to
Address: 75.2.60.5

Additionally, I’ve also added ayush.to as a domain alias so Netlify can issue the SSL certs.

Redirect pretty URLs for articles to correct URL.

Netlify offers two ways of handling redirections for static websites: _redirects file and Netlify.toml. Both options use a different syntax for the redirection and provide different features.

Both options also have a specific order of processing. Netlify processes the _redirects file before Netlify.toml. So my plan is to use _redirects for the actual redirects and Netlify.toml for the catch-all.

To create the _redirects file I’ll use the following code:

---
---
{% assign posts = site.posts | sort: 'number' %}
{% for post in posts %}{{ post.number | prepend: "https://ayush.to/" }}    {{ post.url | prepend: site.baseurl | prepend: site.url }}
{% endfor %}

Jekyll ignores files beginning with an underscore (_) by default. So we’ll need to include it in the _config.yml.

include:
  - _redirects

After the build process completes, the _redirects file in the destination folder looks like this:

https://ayush.to/0    https://www.ayushsharma.in/2016/08/hello-world
https://ayush.to/1    https://www.ayushsharma.in/2016/08/douglas-noel-adams
https://ayush.to/2    https://www.ayushsharma.in/2021/08/introduction-to-ansible
https://ayush.to/3    https://www.ayushsharma.in/2016/08/json-web-tokens
https://ayush.to/4    https://www.ayushsharma.in/2016/08/nano-keyboard-shortcuts
https://ayush.to/5    https://www.ayushsharma.in/2016/08/introduction-to-fluentd
...

If there were no syntax errors in the _redirects file, Netlify should confirm this by telling you the number of redirect rules it processed.

Enlarge — Netlify logs showing the total number of redirect rules processed.

I’m almost there! I’ve generated the _redirects file. Now to handle the catch-all.

Redirect all other pretty URLs to primary domain

My _redirects file can handle specific redirections. But I also need to set a catch-all.

This is where Netlify.toml comes in. In Netlify.toml, I’m going to tell Netlify to redirect everything to my primary domain. Netlify.toml comes after _redirects in the order of processing. Meaning my catch-all only kicks in if no specific redirection rules matched. Nice!

My Netlify.toml now contains the following code:

## ayush.to
[[redirects]]
  from = "https://ayush.to/*"
  to = "https://www.ayushsharma.in"
  status = 301
  force = true
  [redirects.headers]
    X-From = "Netlify"

[[redirects]]
  from = "https://www.ayush.to/*"
  to = "https://www.ayushsharma.in"
  status = 301
  force = true
  [redirects.headers]
    X-From = "Netlify"

The test drive!

Now let’s see if this whole thing even works.

I’ve pushed the above code to Netlify and the robots have completed their task.

So. If I go to ayush.to/94 I should see the article on “Getting started with ArgoCD”.

And…

> curl https://ayush.to/94

Redirecting to https://www.ayushsharma.in/2021/07/getting-started-with-argocd

… it works!

Conclusion

Pretty URLs are a powerful feature in a tiny package. And I was able to set it up neatly and quickly thanks to Jekyll and Netlify.

The ability to configure Netlify’s robots using simple scripts is why I love the service so much. There are of course plenty of improvements we can make. We could improve handling of www and non-www versions of domains. Instead of using numbers as in this example, we could also use smaller slugs or post tags and categories. We can set one-off redirects for special occassions like sales or book launches. The list goes on and on.

I hope the wheels in your head have begun turning. Let me know if you come up with a crazy new use for the Netlify redirection robots. Good luck, and happy coding :)

Linking Jekyll pages back to their Git source code

Nov 16, 2021

I’m a huge fan of open-source! One of the little ways I’ve supported the cause is by keeping this blog open since the very beginning. You can always browse the full source-code here. The reason is partly to let people see the history of changes behind each page. But also because I didn’t find many open-source Jekyll blogs to learn from when I started down this road. So I keep this website open in the hopes that my trials and errors will save someone else a lot of time.

I was recently asked by someone (shout-out to Seth!) to send them the Markdown source of one of my articles. This gave me the idea to add a permanent “View source” button to the footer of every article page. I’ll walk you through the process of adding this to your own Jekyll website.

Jekyll’s helpful variables have exactly what I was looking for: the page.path. This variable contains the raw filesystem path of each page. The official description even highlights its usefulness for linking back to the source!

I’ll use my article on creating Linux apps for webapps as an example. Printing {{ page.path }} within the article’s Markdown file outputs the following:

_posts/2021-10-10-make-linux-apps-for-notion-mastodon-webapps-using-nativefier.md

The above article’s source code is at the following path:

https://gitlab.com/ayush-sharma/ayushsharma-in/-/blob/master/_posts/2021-10-10-make-linux-apps-for-notion-mastodon-webapps-using-nativefier.md

This means that prepending https://gitlab.com/ayush-sharma/ayushsharma-in/-/blob/master/ to any article’s page.path will generate a link back to its source code!

In Jekyll, generating this full link will look like this:

<a href="{{ page.path | prepend: site.content.blog_source_prefix }}" target="_blank">View source</a>

And it’s that’s easy :)

Conclusion

If you scroll down to the footer of this page, you should see a new “View source” link. Go ahead and try it out!

Optimising JPG and PNG images for a Jekyll website

Nov 6, 2021

This website has been text-oriented for a long time now. The decision to add article banners and thumbnails is a recent one. I stayed away from images for a long time since handling and optimising images can be a PITA. I was glad to find that optipng, jpegoptim, and imagemagick already exist to solve this problem for me.

My plan is as follows.

I already have all article images in my static/images folder. From there, I want to generate two copies of all PNG and JPG images. The first would be a cropped thumbnail version measuring 422-by-316. The second would be a larger banner version measuring 1024x768.

Both copies, the thumbnail and the banner, will be in folders of their own. I’ll then leverage Jekyll’s custom variables for the folder paths.

Installing the binaries

On my Mac OS X, installing the binaries required a single brew command.

brew install optipng jpegoptim imagemagick

Creating folders for thumbnails and banners

Next, I’ll create new folders under static/images. Thumbnails will go in img-thumbs and banners will go in img-normal.

cd static/images
mkdir -p img-thumbs img-normal

With the folders created, I’ll first copy all GIF, SVG, JPG, and PNG files to both folders. I’ll use the GIFs and SVGs as-is for thumbnails and banner images.

cp content/*.gif img-thumbs/; cp content/*.gif img-normal/
cp content/*.svg img-thumbs/; cp content/*.svg img-normal/
cp content/*.jpg img-thumbs/; cp content/*.jpg img-normal/
cp content/*.png img-thumbs/; cp content/*.png img-normal/

Processing thumbnails

First let’s resize and optimise the thumbnails. As mentioned earlier, I want the thumbnails to be 422-by-316. I’ll use the mogrify command from ImageMagick to resize the JPGs and PNGs.

cd img-thumbs
mogrify -resize 422x316 *.png
mogrify -format jpg -resize 422x316 *.jpg

Now let’s optimise the PNGs using optipng and the JPGs using jpegoptim.

for i in *.png; do optipng -o5 -quiet "$i"; done
jpegoptim -sq *.jpg

In the above command:

For optipng, -o5 swtich sets the level of optimisation, with zero being the lowest.
For jpegoptim, -s strips all image metadata, and -q sets quiet mode.

Processing banners

I’ll process the banner images like I processed the thumbnails above. Other than the file dimensions everything else will stay the same.

cd ..
cd img-normal
mogrify -resize 1024x768 *.png
mogrify -format jpg -resize 1024x768 *.jpg
for i in *.png; do optipng -o5 -quiet "$i"; done
jpegoptim -sq *.jpg

Configuring the paths in Jekyll

img-thumbs now contains my thumbnails and img-normal contains the banners. To make my life easier, I’ll set both of them to Jekyll variables in the _config.yml.

content-images-path: /static/images/img-normal/
content-thumbs-images-path: /static/images/img-thumbs/

Using these is simple. When I want to display the thumbnail I’ll prepend content-thumbs-images-path to the image. When I want to display the full banner I’ll prepend content-images-path.

For example, articles on this site now have a banner_img property in the front matter. The value of this property is the name of the image file. To display the thumbnail for the article, I use the following code:

{% if page.banner_img %}
 <img src="{{ page.banner_img | prepend: site.content-images-path | prepend: site.baseurl | prepend: site.url }}" alt="Banner image for {{ page.title }}" />
{% endif %}

Conclusion

There are several improvements we can make to the commands above. The most obvious might be to use rsync to only copy changed files to img-thumbs and img-normal. That way we’re not re-processing files over and over again. Another improvement might be to add those commands to Git pre-commit hooks or a CI pipeline.

Resizing and optimising images to reduce their size is a win for the user and the web as a whole. Fewer bytes transmitted over the wire means a lower carbon footprint, but that’s another article. The UX victory is good enough for now :)

Happy coding :)

Quick and easy client-side JavaScript search with Lunr.js

Nov 1, 2021

Search is a must-have for any website or application. A simple search widget can allow users to comb through your entire blog. Or allow customers to browse your inventory. Building a custom photo gallery? Add a search box. Website search functionality is available from a variety of third-party vendors. Or you can take the DIY approach and build the entire backend to answer search API calls.

Lunr.js works on the client-side via JavaScript. Instead of sending calls to a backend, Lunr looks up search terms in an index built on the client-side itself. This avoids expensive back-and-forth network calls between the browser and your server. There are plenty of tutorials online to showcase Lunr’s website search functionality. But you can actually use Lunr.js to search any array of JavaScript objects.

In this how-to, I’ll build a search index for the top 100 books of all time. After that, I’ll show you how to pre-build the index for faster indexing. I’ll also show you how to make the most of Lunr’s search options. And finally, I’ll show off findmymastodon.com - a real-world implementation of Lunr.

Getting started with Lunr.js

Create a new HTML page called lunr.html. We’ll use this file throughout this guide. At the top of lunr.html, call the main Lunr JS library.

<script src="https://unpkg.com/lunr/lunr.js"></script>

Note: You can find the complete code here

Loading the dataset

Next, create a variable called my_big_json. This variable will contain the JSON-ified string of our main dataset. Define the variable in lunr.html within <script> tags.

var my_big_json = '[{"author":"Chinua Achebe","country":"Nigeria","imageLink":"images/things-fall-apart.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Things_Fall_Apart","pages":209,"title":"Things Fall Apart","year":1958},{"author":"Hans Christian Andersen","country":"Denmark","imageLink":"images/fairy-tales.jpg","language":"Danish","link":"https://en.wikipedia.org/wiki/Fairy_Tales_Told_for_Children._First_Collection.","pages":784,"title":"Fairy tales","year":1836},{"author":"Dante Alighieri","country":"Italy","imageLink":"images/the-divine-comedy.jpg","language":"Italian","link":"https://en.wikipedia.org/wiki/Divine_Comedy","pages":928,"title":"The Divine Comedy","year":1315},{"author":"Unknown","country":"Sumer and Akkadian Empire","imageLink":"images/the-epic-of-gilgamesh.jpg","language":"Akkadian","link":"https://en.wikipedia.org/wiki/Epic_of_Gilgamesh","pages":160,"title":"The Epic Of Gilgamesh","year":-1700},{"author":"Unknown","country":"Achaemenid Empire","imageLink":"images/the-book-of-job.jpg","language":"Hebrew","link":"https://en.wikipedia.org/wiki/Book_of_Job","pages":176,"title":"The Book Of Job","year":-600},{"author":"Unknown","country":"India/Iran/Iraq/Egypt/Tajikistan","imageLink":"images/one-thousand-and-one-nights.jpg","language":"Arabic","link":"https://en.wikipedia.org/wiki/One_Thousand_and_One_Nights","pages":288,"title":"One Thousand and One Nights","year":1200},{"author":"Unknown","country":"Iceland","imageLink":"images/njals-saga.jpg","language":"Old Norse","link":"https://en.wikipedia.org/wiki/Nj%C3%A1ls_saga","pages":384,"title":"Njál\u0027s Saga","year":1350},{"author":"Jane Austen","country":"United Kingdom","imageLink":"images/pride-and-prejudice.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Pride_and_Prejudice","pages":226,"title":"Pride and Prejudice","year":1813},{"author":"Honoré de Balzac","country":"France","imageLink":"images/le-pere-goriot.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Le_P%C3%A8re_Goriot","pages":443,"title":"Le Père Goriot","year":1835},{"author":"Samuel Beckett","country":"Republic of Ireland","imageLink":"images/molloy-malone-dies-the-unnamable.jpg","language":"French, English","link":"https://en.wikipedia.org/wiki/Molloy_(novel)","pages":256,"title":"Molloy, Malone Dies, The Unnamable, the trilogy","year":1952},{"author":"Giovanni Boccaccio","country":"Italy","imageLink":"images/the-decameron.jpg","language":"Italian","link":"https://en.wikipedia.org/wiki/The_Decameron","pages":1024,"title":"The Decameron","year":1351},{"author":"Jorge Luis Borges","country":"Argentina","imageLink":"images/ficciones.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/Ficciones","pages":224,"title":"Ficciones","year":1965},{"author":"Emily Brontë","country":"United Kingdom","imageLink":"images/wuthering-heights.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Wuthering_Heights","pages":342,"title":"Wuthering Heights","year":1847},{"author":"Albert Camus","country":"Algeria, French Empire","imageLink":"images/l-etranger.jpg","language":"French","link":"https://en.wikipedia.org/wiki/The_Stranger_(novel)","pages":185,"title":"The Stranger","year":1942},{"author":"Paul Celan","country":"Romania, France","imageLink":"images/poems-paul-celan.jpg","language":"German","link":"","pages":320,"title":"Poems","year":1952},{"author":"Louis-Ferdinand Céline","country":"France","imageLink":"images/voyage-au-bout-de-la-nuit.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Journey_to_the_End_of_the_Night","pages":505,"title":"Journey to the End of the Night","year":1932},{"author":"Miguel de Cervantes","country":"Spain","imageLink":"images/don-quijote-de-la-mancha.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/Don_Quixote","pages":1056,"title":"Don Quijote De La Mancha","year":1610},{"author":"Geoffrey Chaucer","country":"England","imageLink":"images/the-canterbury-tales.jpg","language":"English","link":"https://en.wikipedia.org/wiki/The_Canterbury_Tales","pages":544,"title":"The Canterbury Tales","year":1450},{"author":"Anton Chekhov","country":"Russia","imageLink":"images/stories-of-anton-chekhov.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/List_of_short_stories_by_Anton_Chekhov","pages":194,"title":"Stories","year":1886},{"author":"Joseph Conrad","country":"United Kingdom","imageLink":"images/nostromo.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Nostromo","pages":320,"title":"Nostromo","year":1904},{"author":"Charles Dickens","country":"United Kingdom","imageLink":"images/great-expectations.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Great_Expectations","pages":194,"title":"Great Expectations","year":1861},{"author":"Denis Diderot","country":"France","imageLink":"images/jacques-the-fatalist.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Jacques_the_Fatalist","pages":596,"title":"Jacques the Fatalist","year":1796},{"author":"Alfred Döblin","country":"Germany","imageLink":"images/berlin-alexanderplatz.jpg","language":"German","link":"https://en.wikipedia.org/wiki/Berlin_Alexanderplatz","pages":600,"title":"Berlin Alexanderplatz","year":1929},{"author":"Fyodor Dostoevsky","country":"Russia","imageLink":"images/crime-and-punishment.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/Crime_and_Punishment","pages":551,"title":"Crime and Punishment","year":1866},{"author":"Fyodor Dostoevsky","country":"Russia","imageLink":"images/the-idiot.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/The_Idiot","pages":656,"title":"The Idiot","year":1869},{"author":"Fyodor Dostoevsky","country":"Russia","imageLink":"images/the-possessed.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/Demons_(Dostoyevsky_novel)","pages":768,"title":"The Possessed","year":1872},{"author":"Fyodor Dostoevsky","country":"Russia","imageLink":"images/the-brothers-karamazov.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/The_Brothers_Karamazov","pages":824,"title":"The Brothers Karamazov","year":1880},{"author":"George Eliot","country":"United Kingdom","imageLink":"images/middlemarch.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Middlemarch","pages":800,"title":"Middlemarch","year":1871},{"author":"Ralph Ellison","country":"United States","imageLink":"images/invisible-man.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Invisible_Man","pages":581,"title":"Invisible Man","year":1952},{"author":"Euripides","country":"Greece","imageLink":"images/medea.jpg","language":"Greek","link":"https://en.wikipedia.org/wiki/Medea_(play)","pages":104,"title":"Medea","year":-431},{"author":"William Faulkner","country":"United States","imageLink":"images/absalom-absalom.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Absalom,_Absalom!","pages":313,"title":"Absalom, Absalom!","year":1936},{"author":"William Faulkner","country":"United States","imageLink":"images/the-sound-and-the-fury.jpg","language":"English","link":"https://en.wikipedia.org/wiki/The_Sound_and_the_Fury","pages":326,"title":"The Sound and the Fury","year":1929},{"author":"Gustave Flaubert","country":"France","imageLink":"images/madame-bovary.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Madame_Bovary","pages":528,"title":"Madame Bovary","year":1857},{"author":"Gustave Flaubert","country":"France","imageLink":"images/l-education-sentimentale.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Sentimental_Education","pages":606,"title":"Sentimental Education","year":1869},{"author":"Federico García Lorca","country":"Spain","imageLink":"images/gypsy-ballads.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/Gypsy_Ballads","pages":218,"title":"Gypsy Ballads","year":1928},{"author":"Gabriel García Márquez","country":"Colombia","imageLink":"images/one-hundred-years-of-solitude.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/One_Hundred_Years_of_Solitude","pages":417,"title":"One Hundred Years of Solitude","year":1967},{"author":"Gabriel García Márquez","country":"Colombia","imageLink":"images/love-in-the-time-of-cholera.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/Love_in_the_Time_of_Cholera","pages":368,"title":"Love in the Time of Cholera","year":1985},{"author":"Johann Wolfgang von Goethe","country":"Saxe-Weimar","imageLink":"images/faust.jpg","language":"German","link":"https://en.wikipedia.org/wiki/Goethe%27s_Faust","pages":158,"title":"Faust","year":1832},{"author":"Nikolai Gogol","country":"Russia","imageLink":"images/dead-souls.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/Dead_Souls","pages":432,"title":"Dead Souls","year":1842},{"author":"Günter Grass","country":"Germany","imageLink":"images/the-tin-drum.jpg","language":"German","link":"https://en.wikipedia.org/wiki/The_Tin_Drum","pages":600,"title":"The Tin Drum","year":1959},{"author":"João Guimarães Rosa","country":"Brazil","imageLink":"images/the-devil-to-pay-in-the-backlands.jpg","language":"Portuguese","link":"https://en.wikipedia.org/wiki/The_Devil_to_Pay_in_the_Backlands","pages":494,"title":"The Devil to Pay in the Backlands","year":1956},{"author":"Knut Hamsun","country":"Norway","imageLink":"images/hunger.jpg","language":"Norwegian","link":"https://en.wikipedia.org/wiki/Hunger_(Hamsun_novel)","pages":176,"title":"Hunger","year":1890},{"author":"Ernest Hemingway","country":"United States","imageLink":"images/the-old-man-and-the-sea.jpg","language":"English","link":"https://en.wikipedia.org/wiki/The_Old_Man_and_the_Sea","pages":128,"title":"The Old Man and the Sea","year":1952},{"author":"Homer","country":"Greece","imageLink":"images/the-iliad-of-homer.jpg","language":"Greek","link":"https://en.wikipedia.org/wiki/Iliad","pages":608,"title":"Iliad","year":-735},{"author":"Homer","country":"Greece","imageLink":"images/the-odyssey-of-homer.jpg","language":"Greek","link":"https://en.wikipedia.org/wiki/Odyssey","pages":374,"title":"Odyssey","year":-800},{"author":"Henrik Ibsen","country":"Norway","imageLink":"images/a-Dolls-house.jpg","language":"Norwegian","link":"https://en.wikipedia.org/wiki/A_Doll%27s_House","pages":68,"title":"A Doll\u0027s House","year":1879},{"author":"James Joyce","country":"Irish Free State","imageLink":"images/ulysses.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Ulysses_(novel)","pages":228,"title":"Ulysses","year":1922},{"author":"Franz Kafka","country":"Czechoslovakia","imageLink":"images/stories-of-franz-kafka.jpg","language":"German","link":"https://en.wikipedia.org/wiki/Franz_Kafka_bibliography#Short_stories","pages":488,"title":"Stories","year":1924},{"author":"Franz Kafka","country":"Czechoslovakia","imageLink":"images/the-trial.jpg","language":"German","link":"https://en.wikipedia.org/wiki/The_Trial","pages":160,"title":"The Trial","year":1925},{"author":"Franz Kafka","country":"Czechoslovakia","imageLink":"images/the-castle.jpg","language":"German","link":"https://en.wikipedia.org/wiki/The_Castle_(novel)","pages":352,"title":"The Castle","year":1926},{"author":"Kālidāsa","country":"India","imageLink":"images/the-recognition-of-shakuntala.jpg","language":"Sanskrit","link":"https://en.wikipedia.org/wiki/Abhij%C3%B1%C4%81na%C5%9B%C4%81kuntalam","pages":147,"title":"The recognition of Shakuntala","year":150},{"author":"Yasunari Kawabata","country":"Japan","imageLink":"images/the-sound-of-the-mountain.jpg","language":"Japanese","link":"https://en.wikipedia.org/wiki/The_Sound_of_the_Mountain","pages":288,"title":"The Sound of the Mountain","year":1954},{"author":"Nikos Kazantzakis","country":"Greece","imageLink":"images/zorba-the-greek.jpg","language":"Greek","link":"https://en.wikipedia.org/wiki/Zorba_the_Greek","pages":368,"title":"Zorba the Greek","year":1946},{"author":"D. H. Lawrence","country":"United Kingdom","imageLink":"images/sons-and-lovers.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Sons_and_Lovers","pages":432,"title":"Sons and Lovers","year":1913},{"author":"Halldór Laxness","country":"Iceland","imageLink":"images/independent-people.jpg","language":"Icelandic","link":"https://en.wikipedia.org/wiki/Independent_People","pages":470,"title":"Independent People","year":1934},{"author":"Giacomo Leopardi","country":"Italy","imageLink":"images/poems-giacomo-leopardi.jpg","language":"Italian","link":"","pages":184,"title":"Poems","year":1818},{"author":"Doris Lessing","country":"United Kingdom","imageLink":"images/the-golden-notebook.jpg","language":"English","link":"https://en.wikipedia.org/wiki/The_Golden_Notebook","pages":688,"title":"The Golden Notebook","year":1962},{"author":"Astrid Lindgren","country":"Sweden","imageLink":"images/pippi-longstocking.jpg","language":"Swedish","link":"https://en.wikipedia.org/wiki/Pippi_Longstocking","pages":160,"title":"Pippi Longstocking","year":1945},{"author":"Lu Xun","country":"China","imageLink":"images/diary-of-a-madman.jpg","language":"Chinese","link":"https://en.wikipedia.org/wiki/A_Madman%27s_Diary","pages":389,"title":"Diary of a Madman","year":1918},{"author":"Naguib Mahfouz","country":"Egypt","imageLink":"images/children-of-gebelawi.jpg","language":"Arabic","link":"https://en.wikipedia.org/wiki/Children_of_Gebelawi","pages":355,"title":"Children of Gebelawi","year":1959},{"author":"Thomas Mann","country":"Germany","imageLink":"images/buddenbrooks.jpg","language":"German","link":"https://en.wikipedia.org/wiki/Buddenbrooks","pages":736,"title":"Buddenbrooks","year":1901},{"author":"Thomas Mann","country":"Germany","imageLink":"images/the-magic-mountain.jpg","language":"German","link":"https://en.wikipedia.org/wiki/The_Magic_Mountain","pages":720,"title":"The Magic Mountain","year":1924},{"author":"Herman Melville","country":"United States","imageLink":"images/moby-dick.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Moby-Dick","pages":378,"title":"Moby Dick","year":1851},{"author":"Michel de Montaigne","country":"France","imageLink":"images/essais.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Essays_(Montaigne)","pages":404,"title":"Essays","year":1595},{"author":"Elsa Morante","country":"Italy","imageLink":"images/history.jpg","language":"Italian","link":"https://en.wikipedia.org/wiki/History_(novel)","pages":600,"title":"History","year":1974},{"author":"Toni Morrison","country":"United States","imageLink":"images/beloved.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Beloved_(novel)","pages":321,"title":"Beloved","year":1987},{"author":"Murasaki Shikibu","country":"Japan","imageLink":"images/the-tale-of-genji.jpg","language":"Japanese","link":"https://en.wikipedia.org/wiki/The_Tale_of_Genji","pages":1360,"title":"The Tale of Genji","year":1006},{"author":"Robert Musil","country":"Austria","imageLink":"images/the-man-without-qualities.jpg","language":"German","link":"https://en.wikipedia.org/wiki/The_Man_Without_Qualities","pages":365,"title":"The Man Without Qualities","year":1931},{"author":"Vladimir Nabokov","country":"Russia/United States","imageLink":"images/lolita.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Lolita","pages":317,"title":"Lolita","year":1955},{"author":"George Orwell","country":"United Kingdom","imageLink":"images/nineteen-eighty-four.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Nineteen_Eighty-Four","pages":272,"title":"Nineteen Eighty-Four","year":1949},{"author":"Ovid","country":"Roman Empire","imageLink":"images/the-metamorphoses-of-ovid.jpg","language":"Classical Latin","link":"https://en.wikipedia.org/wiki/Metamorphoses","pages":576,"title":"Metamorphoses","year":100},{"author":"Fernando Pessoa","country":"Portugal","imageLink":"images/the-book-of-disquiet.jpg","language":"Portuguese","link":"https://en.wikipedia.org/wiki/The_Book_of_Disquiet","pages":272,"title":"The Book of Disquiet","year":1928},{"author":"Edgar Allan Poe","country":"United States","imageLink":"images/tales-and-poems-of-edgar-allan-poe.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Edgar_Allan_Poe_bibliography#Tales","pages":842,"title":"Tales","year":1950},{"author":"Marcel Proust","country":"France","imageLink":"images/a-la-recherche-du-temps-perdu.jpg","language":"French","link":"https://en.wikipedia.org/wiki/In_Search_of_Lost_Time","pages":2408,"title":"In Search of Lost Time","year":1920},{"author":"François Rabelais","country":"France","imageLink":"images/gargantua-and-pantagruel.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Gargantua_and_Pantagruel","pages":623,"title":"Gargantua and Pantagruel","year":1533},{"author":"Juan Rulfo","country":"Mexico","imageLink":"images/pedro-paramo.jpg","language":"Spanish","link":"https://en.wikipedia.org/wiki/Pedro_P%C3%A1ramo","pages":124,"title":"Pedro Páramo","year":1955},{"author":"Rumi","country":"Sultanate of Rum","imageLink":"images/the-masnavi.jpg","language":"Persian","link":"https://en.wikipedia.org/wiki/Masnavi","pages":438,"title":"The Masnavi","year":1236},{"author":"Salman Rushdie","country":"United Kingdom, India","imageLink":"images/midnights-children.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Midnight%27s_Children","pages":536,"title":"Midnight\u0027s Children","year":1981},{"author":"Saadi","country":"Persia, Persian Empire","imageLink":"images/bostan.jpg","language":"Persian","link":"https://en.wikipedia.org/wiki/Bustan_(book)","pages":298,"title":"Bostan","year":1257},{"author":"Tayeb Salih","country":"Sudan","imageLink":"images/season-of-migration-to-the-north.jpg","language":"Arabic","link":"https://en.wikipedia.org/wiki/Season_of_Migration_to_the_North","pages":139,"title":"Season of Migration to the North","year":1966},{"author":"José Saramago","country":"Portugal","imageLink":"images/blindness.jpg","language":"Portuguese","link":"https://en.wikipedia.org/wiki/Blindness_(novel)","pages":352,"title":"Blindness","year":1995},{"author":"William Shakespeare","country":"England","imageLink":"images/hamlet.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Hamlet","pages":432,"title":"Hamlet","year":1603},{"author":"William Shakespeare","country":"England","imageLink":"images/king-lear.jpg","language":"English","link":"https://en.wikipedia.org/wiki/King_Lear","pages":384,"title":"King Lear","year":1608},{"author":"William Shakespeare","country":"England","imageLink":"images/othello.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Othello","pages":314,"title":"Othello","year":1609},{"author":"Sophocles","country":"Greece","imageLink":"images/oedipus-the-king.jpg","language":"Greek","link":"https://en.wikipedia.org/wiki/Oedipus_the_King","pages":88,"title":"Oedipus the King","year":-430},{"author":"Stendhal","country":"France","imageLink":"images/le-rouge-et-le-noir.jpg","language":"French","link":"https://en.wikipedia.org/wiki/The_Red_and_the_Black","pages":576,"title":"The Red and the Black","year":1830},{"author":"Laurence Sterne","country":"England","imageLink":"images/the-life-and-opinions-of-tristram-shandy.jpg","language":"English","link":"https://en.wikipedia.org/wiki/The_Life_and_Opinions_of_Tristram_Shandy,_Gentleman","pages":640,"title":"The Life And Opinions of Tristram Shandy","year":1760},{"author":"Italo Svevo","country":"Italy","imageLink":"images/confessions-of-zeno.jpg","language":"Italian","link":"https://en.wikipedia.org/wiki/Zeno%27s_Conscience","pages":412,"title":"Confessions of Zeno","year":1923},{"author":"Jonathan Swift","country":"Ireland","imageLink":"images/gullivers-travels.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Gulliver%27s_Travels","pages":178,"title":"Gulliver\u0027s Travels","year":1726},{"author":"Leo Tolstoy","country":"Russia","imageLink":"images/war-and-peace.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/War_and_Peace","pages":1296,"title":"War and Peace","year":1867},{"author":"Leo Tolstoy","country":"Russia","imageLink":"images/anna-karenina.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/Anna_Karenina","pages":864,"title":"Anna Karenina","year":1877},{"author":"Leo Tolstoy","country":"Russia","imageLink":"images/the-death-of-ivan-ilyich.jpg","language":"Russian","link":"https://en.wikipedia.org/wiki/The_Death_of_Ivan_Ilyich","pages":92,"title":"The Death of Ivan Ilyich","year":1886},{"author":"Mark Twain","country":"United States","imageLink":"images/the-adventures-of-huckleberry-finn.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Adventures_of_Huckleberry_Finn","pages":224,"title":"The Adventures of Huckleberry Finn","year":1884},{"author":"Valmiki","country":"India","imageLink":"images/ramayana.jpg","language":"Sanskrit","link":"https://en.wikipedia.org/wiki/Ramayana","pages":152,"title":"Ramayana","year":-450},{"author":"Virgil","country":"Roman Empire","imageLink":"images/the-aeneid.jpg","language":"Classical Latin","link":"https://en.wikipedia.org/wiki/Aeneid","pages":442,"title":"The Aeneid","year":-23},{"author":"Vyasa","country":"India","imageLink":"images/the-mahab-harata.jpg","language":"Sanskrit","link":"https://en.wikipedia.org/wiki/Mahabharata","pages":276,"title":"Mahabharata","year":-700},{"author":"Walt Whitman","country":"United States","imageLink":"images/leaves-of-grass.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Leaves_of_Grass","pages":152,"title":"Leaves of Grass","year":1855},{"author":"Virginia Woolf","country":"United Kingdom","imageLink":"images/mrs-dalloway.jpg","language":"English","link":"https://en.wikipedia.org/wiki/Mrs_Dalloway","pages":216,"title":"Mrs Dalloway","year":1925},{"author":"Virginia Woolf","country":"United Kingdom","imageLink":"images/to-the-lighthouse.jpg","language":"English","link":"https://en.wikipedia.org/wiki/To_the_Lighthouse","pages":209,"title":"To the Lighthouse","year":1927},{"author":"Marguerite Yourcenar","country":"France/Belgium","imageLink":"images/memoirs-of-hadrian.jpg","language":"French","link":"https://en.wikipedia.org/wiki/Memoirs_of_Hadrian","pages":408,"title":"Memoirs of Hadrian","year":1951}]';

We need to parse this data as JSON so JavaScript can process it.

my_big_json = JSON.parse(my_big_json);

for (var i = 0; i < my_big_json.length; i++) {

    console.log(i + 1, " -> ", my_big_json[i]);
}

To ensure that the loading process was successful, I’m iterating over the JSON dataset and printing each entry. You should see the following in your console log.

Enlarge — Loading Lunr search dataset into JSON in JavaScript.

Building the search index

Now we will build the search index in a variable called idx using the lunr() function. This step requires 3 things:

Lunr returns a document reference for every document that matches a search query. And we need to tell Lunr which field in our dataset should be the reference. Usually this reference is a numeric ID unique to every document. Since our dataset does not contain such a field I’ll use the link field as the reference field.
Lunr also requires the list of fields which should be part of the search index. For this example, I want to search on author, title, and country.
Lastly, Lunr requires our dataset… the my_big_json variable.

We can now call the lunr() function to build our search index idx.

var idx = lunr(function () {
 
    this.ref('link')
    this.field('author')
    this.field('title')
    this.field('country')

    my_big_json.forEach(function (doc) {
        this.add(doc)
    }, this)
});

If there were no errors in your browser’s console log then our index built was successful. Let’s take it for a test drive.

Looking for books in our dataset with Lunr

Let’s first see if there are any documents containing the word adventures:

results = idx.search("adventures");
console.log('Results: ', results.length);

We should see the following in our console log:

Results:  1

Alright. So there is 1 document that matches the search query adventures. But how do we display it?

As I mentioned before, Lunr returns the reference of the matching document. But not the document itself. Meaning if we print results we’ll see the following.

Enlarge — Searching for the best books of all time using Loading Lunr search.

Here results contains an array of objects with a single element. And that element’s ref field contains a value. This is because we used the link field as the reference. To show the full document, we’ll need to work a bit harder.

results = idx.search("adventures");
console.log('Results: ', results.length);

var results_full = results.map(function (item) {
    return my_big_json.filter(function (value, index, arr) {
        return value.link == item.ref;
    })[0];
});
console.log(results_full);

In the above code, we’re iterating through the results using the map() function. We then find documents in my_big_json for all the references in the result set. So results_full contains the complete search results.

0: Object { author: "Mark Twain", country: "United States", imageLink: "images/the-adventures-of-huckleberry-finn.jpg", … }

Let’s repeat the search for the keyword india. The code is as follows.

results = idx.search("india");
console.log('Results: ', results.length);

var results_full = results.map(function (item) {
    return my_big_json.filter(function (value, index, arr) {
        return value.link == item.ref;
    })[0];
});
console.log(results_full);

And the results are:

Results:  4
Object { author: "Kālidāsa", country: "India", imageLink: "images/the-recognition-of-shakuntala.jpg", … }
Object { author: "Valmiki", country: "India", imageLink: "images/ramayana.jpg", … }
Object { author: "Vyasa", country: "India", imageLink: "images/the-mahab-harata.jpg", … }
Object { author: "Salman Rushdie", country: "United Kingdom, India", imageLink: "images/midnights-children.jpg", … }

And it’s that simple! Adding search for any array of JSON objects only requires 5 easy steps:

Call Lunr.js.
Identify the reference fields and the search fields.
Build the search index by iterating over the dataset.
Call the search() method to search the index and return the matching references.
And finally, retrieve the documents for the matching references.

Prebuilding Lunr.js’s search index

You may have noticed that the search index takes some time to build on every page refresh. The time may be imperceptible right now. But it won’t be when lunr.html is on a remote server. Lunr allows pre-building the search index to make the search more responsive.

There are 2 ways of pre-building the index. The first method is to serialise the index after building it. Since we’ve already created the index in our tutorial we’ll use this method.

var serializedIdx = JSON.stringify(idx);
console.log(serializedIdx);

We would then replace my_big_json with the serialised string. Our index load command would also change.

var my_big_json = lunr.Index.load(JSON.parse(serializedIdx));

The second method involves calling the above commands in CLI. This method is great for running in a CI/CD pipeline. I’ll talk about this method more later on.

Lunr.js search tips and tricks

Web search engines like DuckDuckGo provide a way to prioritise search terms. For example, a + sign will include a search term and - will exclude it.

Lunr provides similar features to prioritise search terms or search fields. + will include a search term and - will exclude it. The ~ symbol allows fuzzy matching. The ^ symbol allows weighting search terms. And the : symbol allows search for specific fields.

For example, to find the best books published in England but not by Shakespeare, our Lunr query will be:

results = idx.search("country:England -author:Shakespeare");

This gives me 2 results (truncated):

0: {author: 'Geoffrey Chaucer', country: 'England', imageLink: 'images/the-canterbury-tales.jpg', language: 'English', link: 'https://en.wikipedia.org/wiki/The_Canterbury_Tales'}
1: {author: 'Laurence Sterne', country: 'England', imageLink: 'images/the-life-and-opinions-of-tristram-shandy.jpg', language: 'English', link: 'https://en.wikipedia.org/wiki/The_Life_and_Opinions_of_Tristram_Shandy,_Gentleman'}

Lunr in action: findmymastodon.com!

findmymastodon.com helps users find, well, Mastodon instances. These instances often cater to specific languages, themes, and interests. And so the website needed a search engine.

I started by first building the dataset using Python. The dataset contains metadata for thousands of Mastodon instances currently live. I then used Node to create a serialised Lunr index from the original JSON. This Lunr index is then loaded as a static asset. Search executes against this index and the Mastodon instances, if found, display on the search page. You can browse the JavaScript source code here.

Conclusion

Lunr can prove to be a useful ally in webdev. It’s especially great for static websites which don’t rely on an active backend. Support for pre-built indices using CLI is a huge performance bonus. This allows using CI/CD to fetch data from a remote backend and create the index. Meaning the backend no longer needs to remain active to serve search requests.

Client-side search is a new thing for me. I’ve been a DevOps/cloud guy for a long time and I’m eager to find out how this can improve security and optimise costs. Web browsers are becoming more sophisticated every day. Letting them handle the heavy-lifting for search might grant performance improvements, improved user experiences, and lower cloud costs.

Thanks for reading this how-to, and I hope it sparked some creative ideas in your head. Happy coding :)

Creating light and dark themes for websites the right way using prefers-color-scheme

Oct 10, 2021

You’re likely already familiar with media queries. They’re in widespread use for making websites responsive. The width and height properties contain the viewport’s dimensions. We then use CSS to render different layouts at different dimensions.

The prefers-color-scheme media query works the same way. The user can configure their operating system to use a light or dark theme. prefers-color-scheme contains that value. The value is either light or dark, though the W3C spec states that it might support future values like sepia. We specify different values of CSS variables for both modes and let the user’s OS decide.

The prefers-color-scheme media queries

The two variations of the prefers-color-scheme media query are:

/* Light mode */
@media (prefers-color-scheme: light) {
    :root {
        --body-bg: #FFFFFF;
        --body-color: #000000;
    }
}

/* Dark mode */
@media (prefers-color-scheme: dark) {
    :root {
        --body-bg: #000000;
        --body-color: #FFFFFF;
    }
}

In the above CSS, --body-bg and --body-color are CSS variables. As you can see, they contain different values for both modes. In the light theme, we’re setting a white background with black text. In dark theme, we’re setting black background with white text.

Since the spec says that W3C might introduce future values, it makes sense to convert our CSS into a boolean.

/* Light mode */
:root {
    --body-bg: #FFFFFF;
    --body-color: #000000;
}

/* Dark mode */
@media (prefers-color-scheme: dark) {
    :root {
        --body-bg: #000000;
        --body-color: #FFFFFF;
    }
}

In the above code, we’re defining a light theme by default, and converting to the dark theme if the media query is dark. This way any future values added to the media query will set the light theme by default.

Using the CSS variables

Now that we have different values for different themes, we need to actually use it to style our page.

body {
    background: var(--body-bg);
    color: var(--body-color);
}

The var() syntax is how CSS uses variables. In the above code, we’re saying that background should be set to the value of --body-bg. And color should be set to the value of --body-color. Note that the values of these variables are coming from our media query. Meaning that our background and foreground colour changes based on the OS’s setting! This is the real power of the media query: providing a consistent user experience from OS to the web page.

If you go to findmymastodon.com and switch you OS’s theme you’ll see the transition from one theme to another. The video below shows the smooth transition from light mode to dark mode on Mac OS X.

Your browser does not support this video.

The CSS WG website also uses the same media queries. Change your OS theme and the website will switch themes to adjust.

Conclusion

Notice that using prefers-color-scheme is no different from using a regular programming language. We define variables whose values change based on some logic. And those variables are then used for further operations.

The ability to let your website adjust to the user’s theme of choice is a great accessibility feature. And it further blurs the line between desktop and web for the benefit of the user. Latest browser versions support prefers-color-scheme so you can begin experimenting today.

Happy coding :)

ayush sharma (づ｡◕‿‿◕｡)づ

Consuming APIs responsibly

First do no harm

Network performance

Let the UX drive performance

Backoff and retry

When all else fails…

Stay reachable

Use a reputable hosting provider

Create a series of posts with navigation in Jekyll

Goal

Step 1: Adding series metadata to posts

Step 2: Adding links to posts

Step 3: Adding links to all series to home page

Conclusion

A practical guide to light and dark mode in Bootstrap 5 and Jekyll

The challenge

Step 1: Separate presentation from structure

Step 2: Consolidate styles wherever possible

Step 3: Add the light and dark color schemes

Step 4: The Navbar Toggler

Conclusion

Nemo - The Ubuntu file manager you didn’t know you needed

Feature list

Installing Nemo

Set Nemo as the default file manager

Screenshots

Announcing Fediverse.to!

Fediverse.to!

A new engine under the hood

Future plans

Work to be done

Stay tuned!

Easy pretty URL redirects with Jekyll and Netlify

The plan

Register pretty domain and point it to Netlify

Redirect pretty URLs for articles to correct URL.

Redirect all other pretty URLs to primary domain

The test drive!

Conclusion

Linking Jekyll pages back to their Git source code

Conclusion

Optimising JPG and PNG images for a Jekyll website

Installing the binaries

Creating folders for thumbnails and banners

Processing thumbnails

Processing banners

Configuring the paths in Jekyll

Conclusion

Quick and easy client-side JavaScript search with Lunr.js

Getting started with Lunr.js

Loading the dataset

Building the search index

Looking for books in our dataset with Lunr

Prebuilding Lunr.js’s search index

Lunr.js search tips and tricks

Lunr in action: findmymastodon.com!

Conclusion

Creating light and dark themes for websites the right way using prefers-color-scheme

The prefers-color-scheme media queries

Using the CSS variables

Conclusion