Spotting fake Googlebots in Laravel with forward-confirmed reverse DNS

Look through the access log of any public Laravel app and you'll find requests for /.env, /wp-login.php and /.git/config from user agents claiming to be Googlebot. They aren't Google. Scanners borrow the name because plenty of sites and firewalls wave search crawlers through, and nobody wants to be the person who blocked Google.

The user agent is just a header the client chooses. To know whether a request really comes from Google you need to check the IP address. There are two ways to do it, and you can use both.

Option 1: forward-confirmed reverse DNS

This is the check Google documents for verifying its crawlers. Bing and Yandex document the same check, and Apple says Applebot's addresses generally resolve under applebot.apple.com.

1. Look up the hostname for the IP (a reverse DNS lookup). A real Googlebot address resolves to a name ending in googlebot.com. Google's other crawlers and fetchers use google.com and googleusercontent.com, but those aren't Googlebot, so we don't accept them for a request that says it is.
2. Look up that hostname's IP addresses (a forward lookup) and check the original IP is among them.

The second step matters. Whoever controls an IP range controls its reverse DNS, so a scanner can make its own address resolve to crawl-1-2-3-4.googlebot.com. It can't make Google's DNS point that name back at the scanner's IP.

namespace App\Support;

use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\RateLimiter;
use Illuminate\Support\Str;

class CrawlerVerifier
{
    /** Hostname suffixes each crawler's real addresses resolve to. */
    private const DOMAINS = [
        'googlebot' => ['.googlebot.com'],
        'bingbot' => ['.search.msn.com'],
        'applebot' => ['.applebot.apple.com'],
    ];

    /** Which crawler the user agent claims to be, if any. */
    public function claimed(?string $userAgent): ?string
    {
        $userAgent = Str::lower((string) $userAgent);

        foreach (array_keys(self::DOMAINS) as $crawler) {
            if (str_contains($userAgent, $crawler)) {
                return $crawler;
            }
        }

        return null;
    }

    public function isGenuine(string $crawler, string $ip): bool
    {
        $key = "crawler:{$crawler}:{$ip}";

        if (($cached = Cache::get($key)) !== null) {
            return $cached;
        }

        // Anyone can claim to be Googlebot, so cap the lookups. Out of
        // lookups this minute: give the benefit of the doubt, uncached.
        if (RateLimiter::tooManyAttempts('crawler-dns', 60)) {
            return true;
        }

        RateLimiter::hit('crawler-dns');
        $genuine = $this->lookup($crawler, $ip);

        // Trust a pass for a day. Recheck a failure soon: it may have
        // been a DNS hiccup rather than a fake.
        Cache::put($key, $genuine, $genuine ? now()->addDay() : now()->addMinutes(10));

        return $genuine;
    }

    private function lookup(string $crawler, string $ip): bool
    {
        if (! filter_var($ip, FILTER_VALIDATE_IP)) {
            return false;
        }

        $host = gethostbyaddr($ip);

        // gethostbyaddr returns the IP unchanged when there's no PTR record.
        if ($host === false || $host === $ip) {
            return false;
        }

        if (! Str::endsWith(Str::lower($host), self::DOMAINS[$crawler])) {
            return false;
        }

        // Compare as binary: one IPv6 address has many text forms.
        $needle = inet_pton($ip);

        foreach ($this->addressesFor($host) as $address) {
            if (inet_pton($address) === $needle) {
                return true;
            }
        }

        return false;
    }

    /** @return list<string> */
    private function addressesFor(string $host): array
    {
        $records = dns_get_record($host, DNS_A | DNS_AAAA) ?: [];

        return array_values(array_filter(array_map(
            fn (array $record) => $record['ip'] ?? $record['ipv6'] ?? null,
            $records,
        )));
    }
}

A few things worth pointing out:

• Cache passes longer than failures. Real crawlers come back constantly, so a day's cache for a pass saves a lot of lookups. A failure is cached for ten minutes, so a resolver blip doesn't brand the real Googlebot a fake for a whole day.
• Cap the lookups. The user agent is attacker-controlled, so a scanner can send "Googlebot" from thousands of addresses and make each request wait on DNS. The rate limiter allows 60 lookups a minute across the app; past that, requests aren't flagged rather than slowed down. These calls also block the PHP worker until the resolver answers, so a slow resolver hurts. If that matters to you, use the published ranges below on the request path instead.
• Use dns_get_record, not gethostbynamel. gethostbynamel only returns IPv4 addresses, and Googlebot crawls over IPv6 too.
• Compare addresses as binary. The same IPv6 address can be written compressed, expanded or in different case, so compare inet_pton() results rather than strings.
• Get the real client IP first. Behind Cloudflare or a load balancer, $request->ip() is the proxy's address until you configure trusted proxies. Trust only the proxies that actually sit in front of the app, and make sure the server can't be reached directly. Trusting every proxy ('*') lets anyone set X-Forwarded-For and choose which IP you verify.

Option 2: Google's published IP ranges

Google publishes the ranges its crawlers use as JSON. Googlebot's are in common-crawlers.json (it used to be googlebot.json, which now redirects). Bing and Apple publish theirs too. Download it on the scheduler, store it, and check the IP against the CIDR ranges with Symfony\Component\HttpFoundation\IpUtils::checkIp(). No DNS lookups at request time at all.

use Symfony\Component\HttpFoundation\IpUtils;

$ranges = collect($json['prefixes'])
    ->map(fn (array $prefix) => $prefix['ipv4Prefix'] ?? $prefix['ipv6Prefix'])
    ->all();

IpUtils::checkIp($ip, $ranges);

Ranges cost nothing at request time and can't be slowed down by a busy resolver, which makes them the better choice on the request path. DNS needs no file to keep up to date and gives you a hostname to show, which helps when you're explaining why an address was flagged. Porro's connector uses DNS for its IP details page and shows the hostname next to the verdict.

Using it

The natural place is middleware that runs before anything that might trust crawlers, such as a rate limiter with a crawler exception:

namespace App\Http\Middleware;

use App\Support\CrawlerVerifier;
use Closure;
use Illuminate\Http\Request;
use Symfony\Component\HttpFoundation\Response;

class FlagFakeCrawlers
{
    public function __construct(private CrawlerVerifier $verifier) {}

    public function handle(Request $request, Closure $next): Response
    {
        $crawler = $this->verifier->claimed($request->userAgent());

        if ($crawler && ! $this->verifier->isGenuine($crawler, $request->ip())) {
            $request->attributes->set('fake_crawler', $crawler);
        }

        return $next($request);
    }
}

Flag rather than block. Mark the request and decide later: log it, skip your crawler exemptions, or count it towards a ban along with other behaviour. Blocking outright on one failed lookup means a DNS hiccup could turn away the real Googlebot, which is the one mistake that costs you search traffic.

Only do the lookup when the user agent claims to be a crawler. Running reverse DNS on every visitor would be slow and pointless.

What you'll find

Leave it running for a week, then look at which "Googlebots" asked for /.env, /.git/config or wp-login.php. A fake crawler asking for those is a strong scanner signal. Block it once the check has failed more than once, or alongside other behaviour like repeated probes, rather than on a single lookup.

Porro's free Laravel connector uses this check on its IP details page.