tekkie

Getting indexed without waiting for the crawler

Thu, 30 Jul 2026 00:00:00 +0000

Publishing is only half of the job. A post that nobody can find might as well not exist, and for a static site the gap between deploying a page and a search engine noticing it is entirely out of your hands. Both Google and Bing offer APIs to close that gap by pushing URLs at them directly — and both are fiddly enough that hardly anyone bothers. indeks is a small Rust CLI that does the bothering for you.

Waiting versus telling

The conventional route is passive. You publish a sitemap, you reference it in robots.txt, you register it in Search Console, and then you wait for a crawler to come round. On a site that changes a few times a year that is perfectly adequate. On one that changes weekly, the lag is irritating: you have fixed a typo in a heading, or corrected something factually wrong, and the version the world sees is the old one for days.

Crawl budget is the underlying constraint. Search engines allocate a finite amount of attention per host and spend it according to how often a site has historically changed. A small site that publishes rarely gets visited rarely, which is precisely the site whose occasional new page most needs announcing.

The active route is to tell the engine yourself. Two protocols cover most of the ground:

IndexNow — an open protocol, originally from Microsoft, now with several participants sharing submissions among them. You POST a list of URLs with a key.
Google Indexing API — Google’s own endpoint, authenticated with a service account tied to a verified property.

What indeks does

indeks is a single command that takes URLs, sitemaps, or both, and submits them to one of those two engines.

Installation became easier as of version v0.1.1 that shipped the installer:

curl -fsSL https://raw.githubusercontent.com/ain/indeks/main/install.sh | sh

For developers that have Rust installed, using cargo directly is of course the easiest:

cargo build --release

The shape of an invocation is:

indeks <google|bing> [--url <URL>]... [--sitemap <SITEMAP>]... --credentials <TOKEN_OR_PATH> [--dry-run] [--verbose]

The first positional argument picks the engine — google for the Indexing API, bing for IndexNow — and one is required. Everything else is a flag.

Feeding it URLs

--url takes a single absolute http or https URL. --sitemap takes a sitemap, which may be a remote URL, a local file path, or a file:// URL — useful when you want to submit a sitemap your build just generated but has not deployed yet.

Both are repeatable and can be mixed freely, so long as at least one is present:

indeks bing --url https://example.com/a \
            --sitemap tests/fixtures/sitemap.xml \
            --sitemap https://example.com/sitemap.xml \
            --credentials abcdef0123456789

A URL that turns up more than once — passed twice, or passed directly and also present in a sitemap — is submitted once. That matters more than it sounds, because Google’s daily quota counts requests, not distinct URLs.

Sitemap parsing is deliberately strict. A sitemap must be valid XML in the sitemaps.org format with at least one urlset > url > loc, and every loc an absolute http or https URL.

Failing before the network

The feature I find myself using most is --dry-run, which validates everything and contacts nothing:

$ indeks bing --url https://example.com/a \
              --sitemap tests/fixtures/sitemap.xml \
              --credentials abcdef0123456789 \
              --dry-run
Dry run: no external system will be contacted.
Engine: Bing IndexNow
Credentials: token

URLs (1):
  https://example.com/a

Sitemaps (1):
  tests/fixtures/sitemap.xml — local file, 3 URLs
    https://example.com/
    https://example.com/about
    https://example.com/contact

Input is valid. Remove --dry-run to submit.

Local sitemaps are read and fully parsed during a dry run; remote ones are not fetched, so only the URL itself is checked. You see the expansion before committing to it.

Validation reports every problem at once rather than one per run, which is the difference between one round trip and four:

$ indeks google --url /relative --sitemap ftp://x.example --credentials ./nope.json
error: --url /relative: not an absolute URL
error: --sitemap ftp://x.example: scheme `ftp` is not supported; use http, https or a file path
error: --credentials ./nope.json: no such file

--credentials is required even with --dry-run, which looks pedantic until you consider that a pre-flight check passing on input that a real run would reject is worth nothing. A credentials file is read and its JSON parsed during a dry run, since neither step touches the network.

Credentials, per engine

For Google, the credential is either a service-account JSON file or a bare OAuth access token. The service account must be an owner of the property in Search Console and the Indexing API must be enabled for its project — a 403 tells you which.

For Bing, the credential is an IndexNow key: 8–128 characters of a-z, A-Z, 0-9 or -, passed directly or held in a JSON file as a key field:

{ "key": "abcdef0123456789" }

IndexNow also requires that key to be readable at https://<host>/<key>.txt for every host you submit. indeks cannot do that for you, but a 403 names the exact URL the file has to appear at — the single most common way to get this wrong. URLs are grouped by host, since one request may only carry one, and each host’s batch is sent separately, so one host failing does not stop the others.

Quotas, and what a 429 actually means

This is where the tool earns its keep. Google applies two limits per Cloud project:

Quota	Default
Publish requests per day	200, resetting at midnight Pacific
Requests per minute, all endpoints	380

A 429 names which one ran out, and indeks reads that instead of guessing. If it is the daily limit, waiting cannot help before midnight Pacific, so the run stops immediately and the remaining URLs are reported as not attempted. If it is the per-minute limit — or a 429 naming no metric at all — the limit clears on its own, so the request is retried with a doubling backoff of 10s, 20s and 40s, honouring Retry-After when the server sends one. Only if it still fails does the run stop.

Treating those two cases identically is the obvious implementation and the wrong one: it either burns forty seconds waiting for a limit that resets tomorrow, or gives up on one that would have cleared by itself.

Output you can paste into a bug report

By default, success logs a response code and a URL, and failure logs the code and the error:

$ indeks bing --url https://example.com/a --credentials abcdef0123456789
[403] https://example.com/a: the key was not accepted; it must be readable at https://example.com/abcdef0123456789.txt
1 of 1 URLs were not accepted

Please consider using `--verbose` to find out more

--verbose logs connection setup, TLS negotiation, headers, bodies and timings.

Verbose output is designed to be pasteable as-is. The Authorization header is always redacted. So if you do find a bug, feel free to report a new issue!

Exit codes are conventional:

0 when everything was submitted or a dry run passed
1 when input was fine but submission failed
2 for invalid arguments or an unreadable sitemap

Enough to branch on in a deploy script.

Getting it

indeks is on crates.io and the source is on GitHub, released under GPL-3.0-only. It is roughly 2,400 lines of Rust on the 2024 edition, and the test suite reaches nothing outside the machine it runs on.

Linting Fastly VCL with Falco GitHub Action

Thu, 23 Jan 2025 00:00:00 +0000

When pushing VCL files to Fastly, a missing semicolon or comma can lead to a failure. The configuration will fail to activate, you have to check why and what errored. Typos happen. It’s linter’s job to catch them and let us fail fast.

Fastly VCL

Fastly is one of the world’s leading CDNs and Edge Computing platforms that helps to accelerate a fairly big chunk of today’s internet. From eCommerce to streaming solutions, if they’re performing well, they’re likely to have been built around Fastly¹.

What makes Fastly great is its flexibility powered by the possibility to use custom Varnish configuration files (VCLs) to route traffic, alter and handle requests, or even compute at the edge. All close to the end user.

Falco VCL parser

Falco is an Open Source VCL parser that is built against Fastly’s implementation of Varnish.

It has a comprehensive set of features one of which is dedicated to linting Fastly VCL.

Linting VCL with Falco GitHub Action

In order to automate the linting of Fastly VCL, I’ve built the Falco GitHub Action.

Everyone who has worked with CI/CD pipelines on GitHub before, the configuration of the Falco GitHub Action is trivial and self-explanatory. You define the includes used by the VCL file in options and the VCL file to lint in target:

- name: Lint VCL
  uses: ain/falco-github-action@v1
  with:
    subcommand: lint
    options: "-v -I test/vcl/includes"
    target: test/vcl/valid_with_include.vcl

ATTN: you can also lint ACL files! Linting has a lot of value against a missing comma or invalid CIDR notation in those big dictionaries with thousands of IP ranges.

Fastly Customer Stories ↩

MacBook Pro M1 looks like a developer machine

Tue, 10 Nov 2020 00:00:00 +0000

Apple’s new MacBook Pro delivers more than the new M1 chip. Something good is back for the developers.

MacBook Pro’s keyboard

Having tested the first MacBook Pro with Touch Bar back in 2017, it felt like MacBook was dropping developers. Some essential ergonomics got disrupted esp. for the Vim-based devs.

Today, not only has Apple made a prominent switch to their own M1 chips and tripled the performance¹² by doing so, they’ve also brought back the Esc key. Did Apple’s designers listen to the critique? Perhaps.

Docker Desktop

Docker and Apple Silicon

Official support for M1 chip is there in Docker Desktop 3.3.1³.

Unfortunately no improvement in Docker performance on macOS has been reported.

Docker Desktop, Big Sur and Intel

Docker Desktop runs fine on old Intel-based MacBook Pros with macOS Big Sur. There’s no performance drop.

Since the performance bottleneck for Docker on macOS is the file system, it is important to note that all versions since 2.4.2.0 feature gRPC FUSE for file sharing and are considerably faster than the legacy osxfs, also on Big Sur.

MacBook Pro M1 for developers

With Docker Desktop supporting MacBook Pro M1, it surely serves as a very performant platform for developers who need the mesh of microservices set up locally.

Avoid DevOps disruption by Docker Hub changes

Tue, 20 Oct 2020 00:00:00 +0000

Docker will start rate-limiting image pulls from Docker Hub on November 1st, potentially halting a fair number of CI/CD pipelines.

Docker dealing with network egress

Roughly 30% of all downloads on Docker Hub come from only 1% of anonymous users

Not only has it a price tag, but it surely affects the overall performance of the world’s largest container registry¹ as well.

New subscription plans

To control the unfair share of network egress, the structure of the subscription plans has been announced as follows:

Free plan – anonymous users: 100 pulls per 6 hours
Free plan – authenticated users: 200 pulls per 6 hours
Pro plan – unlimited
Team plan – unlimited

Docker Hub pull authentication

In most development teams 100 pulls per 6 hours is very likely to be insufficient, especially in teams that rely on Continuous Deployment.

Nevertheless, many teams would also find 200 pulls in 6 hours sufficient, meaning that authenticating the pulls would do.

Docker auth on CircleCI

Job setup on CircleCI² is straightforward:

jobs:
  build:
    docker:
      - image: acme-private/private-image:321
        auth:
          username: my-docker-hub-user
          password: $DOCKER_HUB_PASSWORD

Pull rate limit for CircleCI images

Many developers use pre-built CircleCI Docker images³ (e.g. circleci/ruby) to leverage CircleCI’s caching. Pulls are therefore much faster.

It’s relevant to keep in mind that all CircleCI images count towards pull rate limit as well.

Docker auth on GitHub Actions

Workflow step configuration on GitHub Actions leverages docker/login-action:

steps:
  - name: Login to Docker Hub
    uses: docker/login-action@v1
    with:
      username: ${{ secrets.DOCKER_HUB_USERNAME }}
      password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

Caching Docker images on GitLab CI

GitLab team advises developers to start the registry mirror⁴ that would avoid reaching pull rate limits. Mirror would cache the pulls and not turn to Docker Hub when the cache is there.

First major CDN launches its data centre in Estonia

Thu, 29 Mar 2018 21:03:07 +0000

Big news from Cloudflare: it is expanding its CDN network to Baltics.

Bringing world-class web performance¹ to Baltic region, Cloudflare announced² its data centres in Tallinn (Estonia), Riga (Latvia), and Vilnius (Lithuania) today:

Riga, Tallinn and Vilnius: Launching three new European Cloudflare data centers - #MarchOfCloudflare - https://t.co/LX6Ga1tOvR pic.twitter.com/cO4DHnHoJA
— Cloudflare (@Cloudflare) March 29, 2018

BALT-IX, LIXP, RTIX and Tallinn Internet Exchange (TLLIX)³ will all be used to interconnect with local networks.

Jekyll HTTP response handling with Cloudflare Workers

Thu, 29 Mar 2018 00:00:00 +0000

Jekyll is great at speed, which is why it’s used for GitHub Pages, but it’s not as flexible to maintain.

Hacked Wordpress

Some years ago this very blog got hacked. It was on Wordpress. Code was injected to all PHP and HTML headers, hundreds of spam posts were created. These posts made it to sitemap and got indexed on Google. The blog was dead. Epic failure.

Introducing DevOps

After some try ‘n cry development the issue got fixed by a migration from Wordpress to Jekyll. The blog entered DevOps era with automatic publishing on GitHub Pages.

The remaining issue was the indexed spam content on Google.

Configuring Jekyll

GitHub Pages ships with a sensible set of Jekyll plugins¹. One such example is jekyll-redirect-from², that can be used to resolve Google’s 404 Crawl Errors by redirecting content.

Even so, forwarding 404s raised by missing spam content does not make sense. 410 Gone would be a more appropriate response, but there’s no obvious way in Jekyll.

Enter Cloudflare

Cloudflare is great. Its free option can be used with GitHub Pages for TLS termination, caching, page rules. The latter can do redirects, but not 410.

Cloudflare Workers

Cloudflare Workers³ is a relatively new feature that allows running code at the edge. It’s running V8 for JavaScript instead of the standard VCL. Anyone with some basic JavaScript knowledge can create a custom response condition.

JavaScript for 410 response

Since all the spam URLs had a common pattern, e.g. /eoti4/<dashed keywords> or /nhuenh/<dashed keywords>, it took just 17 lines of JavaScript on Cloudflare Workers Editor to respond with 410 on all the spammy endpoints:

addEventListener('fetch', event => {
  event.respondWith(fetchAndLog(event.request));
})

async function fetchAndLog(request) {
  if (/\/(eoti4|nhuenh|nhueto|nhuegp|wish9y|nhsbsb|nhuekz)/g.test(request.url)) {
    return new Response("Page is gone.", {
      status: 410,
      statusText: "Gone"
    });
  }
  return fetch(request);
}

Default Jekyll plugins on GitHub Pages ↩
jekyll-redirect-from - Seamlessly specify multiple redirections URLs for your pages and posts ↩
Cloudflare Workers - Run code at the edge, deliver powerful web extensibility ↩

Stable Docker image to run JIRA

Thu, 15 Mar 2018 00:00:00 +0000

JIRA is one of the best Agile issue tracking tools out there, but choosing the right Docker image to run it is tricky.

Docker Hub lists more than 1 000 JIRA Docker images. Fortunately only a few have considerable number of stars helping to narrow it down to two:

When running JIRA behind HAProxy that terminates SSL, e.g. @LetsEncrypt, cptactionhank/atlassian-jira-software turned out to be painful. Since JIRA does a lot of AJAX, considerable number of XHR requests broke over SSL termination rendering this Atlassian JIRA Docker image unusable.

In comparison, blacklabelops/jira ran out of the box with the following Docker Compose configuration:

services:
  web:
    image: 'blacklabelops/jira:7.5.0'
    hostname: '<jira host>'
    restart: always
    ports:
      - '8080:8080'
    links:
      - db
    volumes:
      - ./jira:/var/atlassian/jira
    environment:
      JIRA_DATABASE_URL: 'mysql://<database user>@db/<database name>'
      JIRA_DB_PASSWORD: '<database password>'
      JIRA_PROXY_NAME: '<jira host>'
      JIRA_PROXY_PORT: '443'
      JIRA_PROXY_SCHEME: 'https'
      DOCKER_WAIT_HOST: 'db'
      DOCKER_WAIT_PORT: '3306'
    logging:
      driver: json-file
      options:
        max-size: 10m
  db:
    image: mariadb:10.1
    environment:
      MYSQL_DATABASE: '<database name>'
      MYSQL_USER: '<database user>'
      MYSQL_PASSWORD: '<database password>'
      MYSQL_ROOT_PASSWORD: '<database root password>'
    volumes:
      - ./data:/var/lib/mysql
    logging:
      driver: json-file
      options:
        max-size: 10m
    restart: always
    command: mysqld --character-set-server=utf8 --collation-server=utf8_bin

Facebook detox experiment

Thu, 11 Jan 2018 00:00:00 +0000

Facebook is toxic. It farms people for data and feeds off reactionary content igniting polarised discussions. It delivers very little value for the time spent.

The background

Fast everything

H&M does fast fashion, McDonald’s does fast food and IKEA does fast furniture. Facebook is about fast friendships. Unfriending works the same way as does throwing away a tee shirt that only cost 5€ to buy. Quantity over quality.

One of Hollywood’s greatest, Denzel Washington, put it nicely:

We all want to be liked, but now we all want to be liked by 16 million

Privacy

Facebook’s success builds on privacy violations. You’re in a private chat telling your friend about a football team you’ve never mentioned before and 2 minutes later you have a fan gear ad of the team in your Newsfeed. All your data, however private, is digested for advertising purposes.

In 2014 I joined @maxschrems’ class action against Facebook exactly because of it. The lawsuit has gone all the way up to Court of Justice of the European Union, awaiting verdict.

Negativity

Facebook provides an excellent platform for flat earthers, anti-vaxxers, trumpists. Hate speech and alternative facts distribute with little control. The more people like this type of content, the more likely it is to appear on their Newsfeeds. Facebook’s algorithm takes care of that for obvious reasons: getting people hooked for higher advertising conversions.

Fast friends are the usual suspects of corrupt Newsfeed and whilst confronting them is not mandatory, it’s sad to witness. Not merely a presumption, Facebook use has been directly linked to depressioni.¹

The experiment

Since I removed Facebook from all my mobile devices a year ago already, next steps should go further:

Deactivate Facebook for a quarter as of Jan 15th
Monitor
- overall productivity
- spare time for activities like sports, hobbies, contributions to Open Source Software
- communication with friends
- mood

Addressing all friends until further notice: text or call. Our keystrokes are limited.

Study Links Heavy Facebook And Social Media Usage To Depression ↩

How to run AngularJS end-to-end tests on GitLab CI

Wed, 13 Dec 2017 00:00:00 +0000

AngularJS-based applications are meant to be developed in test-driven manner, but how to run e2e tests in browser on GitLab CI?

End-to-end tests for AngularJS run TypeScript specs on Protractor in real browser as if user would interact with software.

The problem

How to launch an actual browser on Continuous Integration platform with just command line at our disposal? There is no obvious out-of-the-box solution for CI/CD pipelines.

The plan

Run Xvfb (X virtual framebuffer)
Launch all AngularJS tests on Xvfb display with headless Chrome (Selenium testing)

The solution

Protractor configuration

// protractor.conf.js

const { SpecReporter } = require('jasmine-spec-reporter');

exports.config = {
  allScriptsTimeout: 11000,
  specs: [
    // spec paths
    './e2e/**/*.e2e-spec.ts'
  ],
  capabilities: {
    browserName: 'chrome',
    chromeOptions: {
      args: ['--headless', '--no-sandbox']
    }
  },
  directConnect: true,
  baseUrl: 'http://localhost:4200/',
  framework: 'jasmine',
  jasmineNodeOpts: {
    showColors: true,
    defaultTimeoutInterval: 30000,
    print: function() {}
  },
  beforeLaunch: function() {
    require('ts-node').register({
      project: 'e2e/tsconfig.e2e.json'
    });
  },
  onPrepare: function() {
    jasmine.getEnv().addReporter(new SpecReporter({ spec: { displayStacktrace: true } }));
  }
};

GitLab CI configuration

# .gitlab-ci.yml

image: node:8

stages:
  - test
  - build

test:
  stage: test
  before_script:
    # Add Google Chrome to aptitude's (package manager) sources
    - echo "deb http://dl.google.com/linux/chrome/deb/ stable main" | tee -a /etc/apt/sources.list
    # Fetch Chrome's PGP keys for secure installation
    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
    # Update aptitude's package sources
    - apt-get -qq update -y
    # Install latest Chrome stable, Xvfb packages
    - apt-get -qq install -y google-chrome-stable xvfb gtk2-engines-pixbuf xfonts-cyrillic xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable imagemagick x11-apps default-jre
    # Launch Xvfb
    - Xvfb :0 -ac -screen 0 1024x768x24 &
    # Export display for Chrome
    - export DISPLAY=:99
    # Install AngularJS CLI exclusively
    # Add --unsafe-perm to resolve problems with node-gyp infinite loop on Docker
    - npm install --silent --unsafe-perm -g @angular/cli@1.1.2
    # Install remaining project dependencies
    - npm install --silent
    # Download Selenium server JAR, drivers for Chrome
    - node ./node_modules/.bin/webdriver-manager update
  script:
    - ng test --single-run --progress false
    - ng e2e --progress false

build:
  stage: build
  before_script:
    - npm install --silent --unsafe-perm -g @angular/cli@1.1.2
    - npm install --silent
  script:
    - ng build --prod --progress false
  artifacts:
    paths:
      - dist/
  only:
    - master

Open Source should encourage contributors

Wed, 07 Jun 2017 00:00:00 +0000

As a maintainer, you can not afford to lose contributors, esp. if your day job is not FOSS and you’re not Linus Torvalds.

Open Source documentation

There’s a serious problem with documentation in Open Source. 60% of contributors rarely or never contribute documentation¹. It kills adoption. Negative interactions over dismissive responses, conflicts, unexplained rejections of Pull Requests etc. amount to around 50% of experiences. It kills contributions.

Communication quality of Open Source maintenance

Every project has and will receive flawed Pull Requests. Things that can not be merged over its quality, missing specs or bloat. All fair reasons. What matters is how that Code Review reveals these shortcomings. Interaction quality matters. Instead of closing the Pull Request right off, you can change its course. Explain the shortcomings politely, propose, and, in case of something being overlooked, lead a contributor to work on documentation instead. Explaining the situation may take maintainer 5 minutes more than the actual implementation would, but it will engage the contributor. People will see that Pull Request merged and continue on a positive note.

To illustrate the scenario, smartbanner.js received #51 recently where the contributor was not aware that his implementation duplicates already existing functionality. The easiest would have been to close the PR with one-liner tip that would have explained it. Instead, I requested changes in Code Review, explaining the situation and referring to the need of documentation update. That PR was merged today with a valuable documentation update.

Open Source Survey 2017 ↩