Announcing Our RESTful API — Contributions Welcome

In addition to our next gen Web interface, we have been building a new RESTful API.  Until today, this repo has been private, but it is now ready for community feedback and help in maturing it for production.

View the API on GitHub

One reason for making this public now is that we believe that the API is ready for you to look at and experiment with. The API is at the same point in its development as FTL was when we first shared that project with the world, and it benefited from the feedback it got in those early stages.

Another reason for opening up the project now is that many of us are back in school or our jobs have been busier. We’re hoping by make the work in progress code available now, some community members would help contribute to it and provide feedback.

Goals for the API

As part of replacing the current PHP web interface, the work that was done by the backend PHP code must be moved to a new subsystem. The new web interface only runs in the browser (unlike PHP), so it can not perform the backend duties that the PHP code performed (such as enabling/disabling blocking). To perform these backend jobs, the new API was created.

The API handles the server-side work necessary to support the web interface. This includes communicating with FTL and retrieving the stats it generates, as well as managing the white/black/regex lists and settings. PHP managed to do these tasks, but often issues arose due to the dynamic nature of PHP and the organic way in which the codebase evolved. The new API needed to be able to handle these tasks without introducing bugs and while providing an extendable codebase which could be easily tested.

In addition to PHP, lighttpd is often mentioned in issues, including how it recently broke the config option we use to provide an optional external config. Because the new Web interface only requires a static file server, the API will host the Web interface. This will give us more control over the web server, make changing the Web interface’s port simple, and make Pi-hole easier to package into a Docker container.

Technical Details

The API is written in Rust, a language new to the Pi-hole project. Rust is a safe and fast language which matches well with our goals for the API. It is statically typed and prevents whole categories of errors while being productive and extendable.

The API exposes a RESTful HTTP API which is used by the web interface, but also is general enough for other applications. We have seen some users interested in using Pi-hole’s (PHP) API, which often required them to read the API’s code before understanding how to use it. With this new API, we are making that experience better by following RESTful practices and providing a more intuitive interface.

The API (and the new web interface) were started with testing in mind, and both boast a suite of fine-grained tests. As Pi-hole organically grew, we sometimes ran into issues (in production), which could have been easily found by testing. There are some small test suites in some subsystems, but the system which often has the most bugs (the PHP web interface and API) is very hard to test and must rely on manual testing. This does not scale, and so we strove to find a system which would make testing both a breeze and would be less likely to break. This led us to Rust and the architecture we use in the API’s codebase.

The main way the API communicates with FTL is via shared memory. This is still a work in progress, but it is proving to be very effective (API branch, FTL branch). The idea is that FTL will store its data in a shared-memory space so that the API can come in and read from that memory to fulfill requests. This makes their communication very fast and easy to code. Also, because they share memory instead of sending messages over sockets, the communication is statically typed. This means that instead of assuming the type of the data coming from FTL, the API knows exactly what type the data is in. The PHP API had issues which arose out of how it serialized the data returned by FTL. It had to read the data as a string and assume how it was structured. This shared memory connection will either eliminate or greatly reduce those issues.

This shared memory system means FTLDNS will be even lighter as it no longer needs to do the work of preparing the data for the API (as it did for PHP). Instead, it will concentrate on generating the statistics, and the API can read FTL’s data directly.

So take a look at the code for our new API and feel free to send us a pull request.

https://github.com/pi-hole/api

Thanks again for all of your continued support.

Pi-hole v4.0 Released With FTLDNS, Improved Blocking Modes, Regex, Docker, and More

We’re very pleased to release Pi-hole v4.0 today, which includes fixes, tweaks, and lots of new stuff, including FTLDNS (special thanks to our beta testers!)  In a sentence, FTLDNS is dnsmasq with Pi-hole’s special sauce baked in.

FTLDNS does everything dnsmasq does because it is dnsmasq–just our fork of it.  So all of your existing config files will still work with it.  We intentionally modified the original dnsmasq source code as little as possible so that we can easily integrate any upstream changes as they are released.

Read on to find out everything included in this release or read the technical details in the changelogs.

Continue reading “Pi-hole v4.0 Released With FTLDNS, Improved Blocking Modes, Regex, Docker, and More”

Blocking via regex now available in FTLDNS

We have implemented GNU Extended Regular Expressions for blocking domains into FTLDNS (as used by popular tools such as egrep (or grep -E ...), awk, and emacs).

To try it, you need to be participating in the FTLDNS beta test (see here for more details).  This is a new feature and we invite you to test it out but you should expect some rough edges.   We would also appreciate if you could help us find any bugs or issues you run into.

Reach out to us on Discourse or Reddit with any issues you run into.

Once you’re on the beta testing branch you can configure the regex of your choice in /etc/pihole/pihole-FTL.conf.  In contrast to our already existing wildcard blocking implementation, you can now configure arbitrarily complex blocking filters with Pi-hole FTLDNS.  The following regex:

BLOCKINGREGEX=^ab.+\.com$

will block all domains that start with “ab” (^ab), have at least one further character (.+) and end in “.com” (\.com$).

Examples for what would be blocked by this rule:

  • abc.com
  • abtest.com
  • ab.test.com
  • abr-------.whatever.com

Examples for what would not be blocked by this rule:

  • testab.com (the domain doesn’t start with “ab”)
  • tab.test.com (the domain doesn’t start with “ab”)
  • ab.com (there is no character in between “ab” and “.com”)
  • test.com.something (the domain doesn’t end in “.com”)

Hopefully this illustrates how powerful the new blocking method of FTLDNS is but also why testing its mandatory to ensure it is working correctly in all possible situations.  The potential of this new blocking is huge and may even help with things like this.

Help Us Beta Test FTLDNS

In case you missed it, FTLDNS is Pi-hole’s Faster Than Light (FTL) daemon combined with a DNS/DHCP server (our fork of dnsmasq).  So instead of installing dnsmasq as a dependency, we will be distributing all services in a single binary.  This will allow us to hook into dnsmasq‘s functionality at a code- level, as well as ensuring that you are using the latest stable version of the resolver and receive, e.g., important bug fixes early on.

Today, we’re looking for some users to help us beta test the software.  If you are comfortable with troubleshooting issues, and if you’re interested, run these commands (after reading the details of this beta): Continue reading “Help Us Beta Test FTLDNS”

FTLDNS: Pi-hole’s Own DNS/DHCP server

What Is FTLDNS™?

In a sentence, FTLDNS™ is dnsmasq blended with Pi-hole’s special sauce.  We bring the two pieces of software closer together while maintaining maximum compatibility with any updates Simon adds to dnsmasq.

The Problems

We have used dnsmasq as our DNS/DHCP server since Pi-hole began.  However, as the project has grown and evolved, we reached a point where we started looking into providing our own powerful resolver for a few reasons:

  • With FTL, we generate a variety of statistics by interpreting dnsmasq‘s log file. While this works, doing it through a middleman (a log file) is obviously quite inefficient
  • If a user chooses to disable logging, we’re unable to compute any statistics at all
  • There are some details of the DNS server we do not have access to (like cache usage) through the log file alone
  • Each OS distribution provides a different version of dnsmasq and this can prevent users for getting a flawless Pi-hole experience

The Solution

All these problems lead us to the idea of actually integrating a DNS resolver directly into FTL. The obvious choice for this is, of course, to base it on dnsmasq. We have a proof-of-concept implementation of FTL already developed and running.  So we’re officially announcing FTLDNS™ and you can find it right here.  

One of the many ways to support us is purchasing from our Swag Store.  Get your hands on this sweet mug to help spread the word.

The Non-technical Details

FTLDNS has many benefits for you as user:

  • No more dnsmasq version compatibility issues since we develop and provide the DNS resolver and FTL together
  • Upstream updates to dnsmasq can easily be integrated into our code since we don’t modify its code heavily
  • Processing will be much faster since we interact directly with the resolver instead of periodically parsing a log file
  • We can provide stats on cache usage and probably much more, which has been previously unavailable
  • Due to the direct integration into the resolver (using “FTL hooks”), we can compute statistics without any need for a log file. Instead of a “must have”, the log file will become an “opt-in” feature!

Continue reading “FTLDNS: Pi-hole’s Own DNS/DHCP server”

Pi-hole v3.3 Released: It’s “Extra” Special

Update 2018-02-20 18:05

Hi All, After a few days of pulling out our hair and troubleshooting this whitelisting issue that some of you have reported, we’re finally getting to the bottom of it.

The good news is, whitelisting is not completely broken. You can still whitelist domains from the cli with no issues by calling pihole -w [domain-to-whitelist]. The issue only affects whitelisting from the admin page (whitelist page, query log, and block page).

Take a look over this pull request where /u/promofaux has attempted to explain what is going on. Though, we’re a bit confused ourselves, and any insight from the community would be greatly appreciated!

There are a couple of options, we can either revert the change that broke it, or use the fix in the above pull request. Whichever way we go, rest assured that we are working hard internally to make sure that we have the bug well and truly squashed, and will try to get a fix out as soon as we can (and really, take that soon™ in the Blizzard sense of the word).

In the mean time, do not attempt to whitelist from the web admin, it wont work… apologies for any inconvenience this causes.

In other news, we have updated the to include instructions on how you may possibly be able to update your version of dnsmasq to be able to update to Pi-hole 3.3

Update 2018-02-18 06:12

If you’re running Raspbian Jessie and you updated Pi-hole to v3.3, you likely ran into issues.  This is because the version of dnsmasq that ships with it does not support the log-queries=extra option, which we use in v3.3.

You have two options to resolve this: revert Pi-hole to a previous version or upgrade dnsmasq manually.

Option one: downgrade Pi-hole to the previous version

Instructions for this can be found here.

Option two:  install the version of dnsmasq that supports the extra flag (v2.76)

Please note, you should only try this on Rasbpian Jessie and do so at your own risk (but in our opinion the risk is low)

First step: Download more recent version of dnsmasq compiled for Raspbian Jessie from the official sources

wget https://archive.raspberrypi.org/debian/pool/main/d/dnsmasq/dnsmasq-base_2.76-5+rpi1_armhf.deb
wget https://archive.raspberrypi.org/debian/pool/main/d/dnsmasq/dnsmasq_2.76-5+rpi1_all.deb

Second step: Ensure requirements are fulfilled

sudo apt-get install libnetfilter-conntrack3 libmnl0

Third step: Install downloaded packages

sudo dpkg -i dnsmasq-base_2.76-5+rpi1_armhf.deb
sudo dpkg -i dnsmasq_2.76-5+rpi1_all.deb

Fourth step: Verify it worked:

dnsmasq -v

should return:

Dnsmasq version 2.76  Copyright (c) 2000-2016 Simon Kelley
Compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify

You should now be able to use Pi-hole v3.3 on Raspbian Jessie.

Update 2018-02-14 18:43 (version issues and not working after update)

If you’re running Rasbian Jessie, your version of dnsmasq will not work with this release, so you’ll need to revert to the previous versions:

cd /etc/.pihole
sudo git fetch --tags
sudo git checkout v3.2.1
cd /var/www/html/admin
sudo git fetch --tags
sudo git checkout v3.2.1
pihole -r
pihole checkout ftl v2.13.2

The Release


This release takes full advantage of dnsmasq‘s extra logging feature, which means you’ll get 100% accurate log analysis.  This release also includes full DNSSEC support, Teleporter enhancements, several important security fixes, as well as some other tweaks. This blog post will focus on the main features of this release, but if you want a detailed breakdown, the full changelogs can always be found at changes.pi-hole.net. Continue reading “Pi-hole v3.3 Released: It’s “Extra” Special”

Pi-hole Web Interface: The Next Generation

We have been working on a new Web interface for Pi-hole (referenced internally as Next Gen Admin or NGAdmin).  The existing interface built off of AdminLTE has served us well, but we have grown beyond the capabilities of an existing template.  We’re also looking to implement an HTTP API.

This new interface is open source and we welcome your contributions as we have just made the repo public.  Read on to learn more or check out a demo of the new interface here. Continue reading “Pi-hole Web Interface: The Next Generation”

Pi-hole v3.2.1 Released With Lots Of Fixes

v3.2 was one of our biggest releases but many of you found bugs and issues we didn’t.  We have fixed the DNS resolution is currently unavailable issue, the settings page not working on some systems, the FTL version not showing correctly, as well as many more.  Just run pihole -up to install the update and get the fixes in place.  Read on for more details or checkout changes.pi-hole.net for the full changelogs. Continue reading “Pi-hole v3.2.1 Released With Lots Of Fixes”

Pi-hole v3.2 Introduces Long-term Statistics, An Audit Log, Colours, and More!

We are very pleased to release a new version of Pi-hole–version 3.2.  This release comes with a long-awaited request: long-term statistics.  You can now store and view more than 24 hours worth of Pi-hole data to help see your historical queries and performance.

We also have a new audit log for keeping track of domains you may want to white or blacklist, a new tabular settings page for easier navigation, and an improved debugger.  Not only that; you’ll also notice colorized output on the command line as well as a slew of fixes and improvements.  This is a big release so read on to find out more. Continue reading “Pi-hole v3.2 Introduces Long-term Statistics, An Audit Log, Colours, and More!”