Log File Analysis: See Full How Googlebot Crawls Your Site
Google Search Console'un “Crawl Stats” raporuna hiç baktıysanız ve hikayenin yarısını kaçırdığınız gibi hissettim, yalnız değilsiniz. Bu rapor size anlatıyorkaç tane kaç taneGooglebot'in sunucunuza yaptığı istekler, ancak hangi belirli URL'lerin tarandığını, ne sıklıkta veya hangi statü kodlarının geri döndüğünü söylemez. İşte burasılog file analiziiçeri girin.
On yıldan fazla teknik SEO yapıyorum ve sizi tereddüt etmeden anlatabilirim: sunucu log dosyaları Googlebot tarama davranışı hakkında en zengin gerçek kaynağı. Size sunucunuzun aldığı gerçek istekleri gösterir, ikincisine aşağı. Kendi sitenizi, bir aletle sadece size ne anlattığınızı söylercould could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could couldtaranır -log dosyaları size ne olduğunu söylerwas was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was was wasAslında tarandı.
Bu kılavuzda, günlük dosya analizinin tam olarak ne olduğunu, neden Googlebot davranışını anlamak için önemli olduğunu, doğru araçlarla nasıl performans göstereceğimi ve içgörüler ile ne yapacağımı. Sonunda, tarama bütçesini nasıl teşhis edeceğini bileceksiniz, indeksleme sorunlarını tanımlayacak ve Googlebot en önemli sayfalarınıza odaklandınız.
Log File Analysis Matters for Googlebot Crawl Davranışı
Her zaman Googlebot sitenizi ziyaret eder, sunucu loglarınızda bir ayak izi bırakır. Bu loglar URL, zamantamp, kullanıcı ajanı (Googlebot vs. Bingbot vs. gerçek bir kullanıcı), HTTP durumu kodu, transfer edilenler ve bazen referans veri. Bu logları analiz ederek, soruları cevaplayabilirsiniz:
- Hangi sayfalar en sık Googlebot taranıyor?
- Googlebot ince içerik, paginasyon döngüleri veya sonsuz kaydırma sayfaları üzerinde zaman harcıyor mu?
- 404'leri veya 500'leri geri döndürecek herhangi bir sayfa var mı?
- Googlebot, güncel içeriği nasıl sıklıkla ziyaret eder?
- Büyük siteler üzerinde bütçe sorunları var (100k+ sayfalar)?
Log file analiziKesin cevaplar elde etmenin tek yolu. Bu bir SEO "nice to have" değildir - özellikle işletme siteleri için temel bir teknik SEO uygulamasıdır. Yüz binlerce sayfa ile bir siteyi denetim ettiğimde, her zaman sayfa optimizasyonundan önce loglarla başlarım.
Googlebot Crawl Davranış vs. Crawl Budget
Google'ın size verdiği bir şey “kahkadar bütçesi”nin size verdiği yaygın bir yanlış var. Aslında, sunucunuzun Googlebot'e nasıl tepki verdiğinin sonucu. Eğer sunucunuz hızlıysa ve 200 Tamam sürekli döndürürse, Googlebot daha sık geri gelecektir. Yavaş sayfalarınız varsa, yönlendirme zincirleri veya hatalarınız varsa, Googlebot yavaşlayacak veya taramayı bırakacaktır.
Yemin olsunlog file analiziAncak, tarama bütçesinin nerede harcandığını tam olarak görebilirsiniz. Örneğin, bir zamanlar Googlebot'in günde 80.000 URL'yi taradığı bir e-ticaret müşterisiyle çalıştım - ancak bunların% 40'ı 301 yönlendirmeye yol açtı. Bu boşanmış kaynaklar. Yönlendirmeleri temizlemek ve robotları eklemekten sonra.txt yönergeleri, taramayı yarı yarıya kestik ve üç hafta içinde organik trafik artışı gördük.
Nasıl oynanır Log File Analysis: Tools and Steps
Giriş dosyalarınıza erişmek için bir sistem yöneticisi olmanız gerekmez. Çoğu web barındırma, cPanel, FTP veya doğrudan indirme yoluyla erişim sağlar. Büyük siteler için, DevOps ekibinizle Apache Kombine, Nginx veya AWS CloudFront gibi ortak formatlarda ham giriş yapmanız gerekebilir.
Adım 1: Loglerinizi toplayın
Standart yaklaşım en az 7-14 gün sunucu loglarını indirmektir. Doğru Googlebot tarama davranışı için, tam bir ay tavsiye ederim. Girişlerin kullanıcı ajan dizesini içerdiğinden emin olun, böylece Googlebot (Mozilla/5.0 uyumlu; Googlebot/2.1; +http://www.google.com/bot.html).
Adım 2: Bir Özel Log Dosya Analizini Kullanın
Python veya grep ile oturum açabilirsiniz, ancak uzmanlaşmış araçlar muazzam zaman tasarrufu sağlar. İşte benim üst picks:
- Screaming Frog Log File Analyser- 1.000 satıra kadar ücretsiz; ücretli sürüm milyonlarca kişiyi idare eder. Analiz ve görselleştirme için mükemmel.
- Botify Botify- Gerçek zamanlı günlük işleme, tarama bütçe raporları ve Google Analytics ile entegrasyon.
- Lumar (formerly DeepCrawl)– Log file import ve detaylı tarama sağlık panoları ile başka bir işletme aracı.
Adım 3: Googlebot için Filtre Sadece
Araç otomatik olarak Googlebot kullanıcı ajanlarını tanımlayacaktır. Diğer tüm botları (Bing, Yandex, Facebook) Googlebot tarama davranışına odaklanmak.
Adım 4: Analyze Key Metriks
Bak:
- Durum kodu dağıtım- Googlebot için 200, 301, 404, 500%.
- Crawl frekansı URL tarafından- Hangi sayfalar en sık vuruldu.
- Crawl derinlik- Kaç seviye derin Googlebot gidiyor?
- Crawl timestamps– Peak tarama süreleri, bazen recrawl kaence işaret ediyor.
İşte geçen yıl denetim ettiğim gerçek bir orta doğu e-ticaret sitesinden bir örnek:
| Metrik | Optimizasyondan Önce Optimizasyon | Optimizasyondan Sonra Optimizasyon |
|---|---|---|
| Günlük Toplam Googlebot talepleri günde | 12,000 | 8,500 |
| İsteklerin% 200'ü geri dönüyor | 60% | 80% |
| % of requests to non-canonical URLs | 25% | 5% |
| Crawl bütçesi boşa harcıyor (%) | 40% | 15% |
Bu rakamlar 50.000 ürün sayfası ile bir müşteriden gelir. Log file analizinden sonra Googlebot'in permutations ve seans tabanlı URL'leri taradığını tespit ettik. Bunu robotlarda engelledik.txt ve kanonik sayfalara içsel bağlantı geliştirdik. Sonuç: daha organik trafik ve daha hızlı indeksleme.
Log File Analizi
Verileriniz olduğunda, gerçek çalışma başlar. İşte tekrar gördüğüm desenler:
1. İnce veya Duplicate Content
Googlebot organik bir değeri olmayan sayfaları vuruyorsa – görünüşe göre, paginated arşivler, yazıcı-dost versiyonları – sen tarama bütçesini yapıyordun. Use Use Use Use Use Uselog file analiziBu URL'leri tanımlamak ve eklemeknoindexveyadisallowTalimatlar.
2. Hata Pages Hala Crawlediliyor
Eski sayfaları sildikten sonra bile, Googlebot, indeksden çıkarılana kadar 404'leri taramaya devam edebilir. Logs size URL'lerin 404'leri geri döndüğünü ve ne sıklıkta döndüğünü gösterecektir. Bir 404'ün günlük vurduğunu görürseniz, ilgili bir sayfaya 301 yönlendirme yapın veya özel 404 işleme uygulayın.
3. Crawl Frekans Disparities
Sık sık güncelleme yapan sayfalar (örneğin haber makaleleri veya ürün fiyatları gibi) daha sık taranmalıdır. En önemli içerikiniz sadece bir ay boyunca taranırsa, eski blog yazıları günlük ziyaretler alır, bu kırmızı bayrak. Log dosyaları bu dengesizlikleri ortaya çıkarır.
GSC
Google Search Console, bir tarama oranı limitini belirlemenize olanak sağlar, ancak bir blunt cihazıdır. Log dosyaları size bu ayarların gerçek zamanlı etkisini gösterir ve taze içeriğin daha hızlı indekslenmesi için sınırı arttırmanız gerekir.
5. Bot Mimik'in Tespiti
Bazen kötü niyetli botlar Googlebot gibi davranıyor. DNS doğrulama ile Server girişleri, engelleyicileri tanımlayabilir. Legitimate Googlebot talepleri her zaman googlebot.com hostnames'den gelir. Sahte bir kullanıcı ajanı kaynakların tüketilmesini görürseniz, bunu güvenlik duvarında engelleyebilirsiniz.
Log File Analiz Araçları Karşılaştırma
Doğru aracı seçmenize yardımcı olmak için, burada ellerim-on deneyimime dayanan bir karşılaştırma masası:
| Tool Tool Tool | Fiyatlandırma | Anahtar Özellikler | En iyisi için |
|---|---|---|---|
| [Screaming Frog Log File Analyser](https://www.screamingfrog.co.uk/log-file-analyser/) | Ücretsiz (limited) / £149 / yıl | Gerçek zamanlı giriş ithalat, kullanıcı ajanı filtreleme, yanıt kodu kesinti, ihracat edilebilir raporlar | Bireyseller ve orta sitelere küçük |
| [Botify] (https://www.botify.com) | Enterprise özel fiyatlandırma | Sürekli giriş akışı, tarama bütçe panjuru, gelir tahsisi, API | Büyük e-ticaret ve işletme |
| [Lumar (DeepCrawl)](https://www.lumar.io) | Enterprise özel fiyatlandırma | Log file yükleme, tarama yolu analizi, GSC ve GA ile entegrasyon | In-house SEO takımları ölçeğinde |
| Özel senaryolar (Python/AWK) | Ücretsiz Ücretsiz | Tam kontrol, ölçeklenebilir | Zaman ve bütçe ile geliştiriciler |
Bütün dört kullandım. Çoğu ajans ve ev takımlarında, Screaming Frog Log File Analyser en iyi değerdir. Büyük siteler için (1M+ sayfalar), Botify'ın otomatik işlemesi yenmek zordur.
Crawl Data üzerinde Nasıl Yasası
Verilerin toplanması eylem olmadan işe yaramaz. İşte tam olarak senin peşinden ne yapacağımlog file analizi:
Fix Redirect Chains and Broken Links
Eğer Googlebot 404'lere yol açan 301'leri vuruyorsa, yönlendirme zincirini güncelle. En taramalı kırık URL'lerin listesini derlemek için günlük dosya verilerini kullanın ve ilgili sayfalara uygun 301'leri uygulayın.
Robotlarla Crawl Bütçesini optimize edin.txt
Tüm bağımsız bölümlere blok: yönetici, yüzt parametreler, seans kimlikleri, yorum sayfaları. Kullanıncrawl-delaySadece gerekli olursa – çoğu site buna ihtiyaç duymaz.
Key Pages'te İç Bağlantıyı Geliştirmek
Eğer loglar yüksek değerli sayfaların (en çok satan ürünleriniz gibi) nadiren tarandığını gösterirse, yüksek hacimli sayfalardan daha fazla iç bağlantı ekleyin ( ana veya kategori sayfaları gibi). Bu sinyaller Googlebot için önemlidir.
Hız Up Server Response Time
Googlebot, sunucu yanıt süresi 200 m'yi aştığında tarama oranını azaltır. Yavaş uç noktaları tanımlamak ve onları optimize etmek için logları kullanın. Tekrarlanan URL'ler için CDN veya kalibrasyon düşünün.
GSC aracılığıyla kritik Sayfaları gönderin
Hemen indekslemeye ihtiyaç duyan bir haber makalesi veya yeni bir ürün lansmanınız varsa, Googlebot zaten sayfayı logları kontrol ederek vurdu. Değilse, URL Muayene aracını indeksleme talep etmek için kullanın.
Crawl Data'dan Rankings
Bunu nasıl kontrol ettik (Eylül 2026): 2026 Eylül’de müşteri özellikleri üzerinde tekrar günlük kontroller yapıyoruz; araç isimleri ve ücretsiz limitleri bu tarih olarak doğrulayabileceğimizi yansıtıyor.
Crawl verileri size neler olduğunu söyler. Bu kılavuzlar onu düzeltmelere dönüştürür:
- Web sitesi SEO analizör yürüyüş- tam denetim iş akışı günlük bulgularınızın etrafında
- Next.js SEO: SSR vs ISR vs SSG- Form sorunları genellikle yüzeyleri düzeltin
- AEO vs SEO- Taramalı sayfalar AI alıntıları da kazanıyor
Sık Sorulan Sorular
1. Giriş dosyası, bir siteyi tarayarak aynı şeyi analiz eder mi?
Hayır. Crawling a site (kırık veya benzer) bir botun ne olduğunu simüle edercould could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could could couldtarama. Log file analizi, GooglebotAslında aslında aslında aslında aslında aslındataramalı. İkisi birbirini tamamlar.
2. Paylaşılan barındırmadan log dosyaları alabilir miyim?
Evet, en çok paylaşılan evler, “Metrics” bölümünde ham erişim girişleri sağlar. Değilse, onları sağlamak için destek isteyin.
3. Ne kadar giriş verileri analiz etmem gerekiyor?
En az 7 gün; 14-30 gün haftalık kalıpları yakalamak için daha iyidir. Bu sizin odakınız olmadıkça tatil trafiği gibi zirvelerden kaçının.
4. Kayıt dosyaları JavaScript tarama verilerini içeriyor mu?
Access logs, XHR/fetch JS-rendered content için çağrıları da dahil olmak üzere tüm HTTP taleplerini yakalar. Ancak, bu genellikle tarayıcıdan gelir, Googlebot değil. Kullanıcı ajanı tarafından dikkatle filtreye ihtiyacınız var.
5. What if Googlebot's IP range changes?
Google IP aralıklarını yayınlarGooglebot FAQÇoğu log analizörleri otomatik olarak IP'leri bu listeye doğrular.
Final Düşünceleri&Sonraki Adımlar
Log file analiziGoogle'ın gözleriyle görmek için alabileceğiniz en yakın şey. Size tam olarak hangi sayfaların tarandığını, ne sıklıkta ve Googlebot'in düşük değerli URL'ler döngüsünde sıkışıp kaldığını söyler. Kombine ile birlikteGooglebot tarama davranışıSearch Console'dan gelen veriler, tarama bütçesini kurtarmak ve indeksleme verimliliğini artırmak için kesin bir teknik SEO stratejisi oluşturabilirsiniz.
DG10 Agency'te, her işletme denetiminde günlük dosya analizi kullanıyoruz. Seçmeli değil - temelsel. indeksleme sorunları ile mücadele ediyorsanız, gizemli trafik damlaları veya asla tam indekslenmiş gibi görünen büyük bir site, loglara bakmak zamanı.
Sitenizi nasıl Googlebot taradığını tam olarak görmeye hazır mısın?İletişim DG10 AgencyTeknik bir SEO denetim için, derinlemesine günlük dosya analizi ve eylem edilebilir tarama optimizasyonu içerir. Sunucu loglarınızı daha iyi organik performans için bir yol haritasına dönüştüreceğiz.
Bu makale, DG10 Agency'teki teknik SEO ekibi tarafından yazılmıştır. Tüm veriler ve vaka çalışmaları gerçek müşteri projelerinden, gizlilik için anonimleşmiştir. Teknik SEO temelleri hakkında daha fazla bilgi için, okuyunKurulum teknik SEO.