Crawlens

A Go-based web crawler that audits websites for SEO and accessibility issues including broken links, missing alt tags and improper headings.

Features • Installation • Usage • Documentation • Contributing

🚀 Features

Link Analysis

🔍 Broken Link Detection - Find all broken links (404s, 500s, timeouts) with source page tracking
⚡ High-Performance Crawling - Concurrent crawling with configurable workers (default: 5)
🔄 Redirect Chain Detection - Identify and report multi-hop redirects (301/302)
👻 Orphaned Page Detection - Find pages with no internal links pointing to them
🔗 Global Link Deduplication - Check each unique link only once for maximum efficiency

SEO Analysis

📊 SEO Score (0-100) - Comprehensive scoring for each page
✅ Title Tag Optimization - Check presence and length (30-60 chars optimal)
📝 Meta Description Analysis - Validate presence and length (50-160 chars)
🏷️ Heading Structure - H1 and H2 tag validation
📄 Content Quality - Word count analysis and thin content detection
🖼️ Image Accessibility - Find images missing alt text
🌐 Social Media Tags - Validate Open Graph and Twitter Card tags
🔗 Technical SEO - Canonical URLs, language attributes, robots meta tags
📈 Duplicate Detection - Find duplicate titles and descriptions across pages

Performance Metrics

⏱️ Page Load Time Tracking - Monitor and report slow-loading pages
📊 Comprehensive Statistics - Crawl duration, link counts, data transfer
🎯 Smart URL Normalization - Treats /page and /page/ as same URL
🚫 Build Artifact Exclusion - Automatically skips _next/static/, webpack chunks, etc.

Export & Reporting

📄 JSON Export - Complete structured data export
📋 CSV Export - Excel-friendly tabular format
🗂️ XML Export - Standards-compliant XML format
🎨 Color-Coded Console Output - Easy-to-read terminal output
📍 Real-Time Progress - Live worker count and progress indicators

Politeness & Compliance

🤖 robots.txt Support - Respects website crawling rules
⏲️ Rate Limiting - Configurable delays between requests
🔒 User-Agent Identification - Properly identifies itself

📦 Installation

Download Pre-Built Binary (Recommended)

Download the latest release for your platform:

Linux

# AMD64
wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/
wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/

# ARM64
wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-arm64.tar.gz
tar -xzf crawlens-1.0.0-linux-arm64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/
wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-arm64.tar.gz
tar -xzf crawlens-1.0.0-linux-arm64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/

macOS

# Intel
curl -LO https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-darwin-amd64.tar.gz
tar -xzf crawlens-1.0.0-darwin-amd64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/
curl -LO https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-darwin-amd64.tar.gz
tar -xzf crawlens-1.0.0-darwin-amd64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/

# Apple Silicon (M1/M2/M3)
curl -LO https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-darwin-arm64.tar.gz
tar -xzf crawlens-1.0.0-darwin-arm64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/
curl -LO https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-darwin-arm64.tar.gz
tar -xzf crawlens-1.0.0-darwin-arm64.tar.gz
chmod +x crawlens
sudo mv crawlens /usr/local/bin/

Windows

Download crawlens-1.0.0-windows-amd64.zip
Extract the ZIP file
Add the folder to your PATH or run from the extracted location

Build from Source

# Clone the repository
git clone https://github.com/qacancode/crawlens.git
cd crawlens
git clone https://github.com/qacancode/crawlens.git
cd crawlens

# Download dependencies
go mod download

# Build
go build -o crawlens
go build -o crawlens

# Run
./crawlens -u https://example.com
./crawlens -u https://example.com

🎯 Usage

Quick Start

# Basic crawl
crawlens -u https://example.com
crawlens -u https://example.com

# With SEO analysis and JSON export
crawlens -u https://example.com -f report.json
crawlens -u https://example.com -f report.json

# Verbose mode with CSV export
crawlens -u https://example.com -v -f results.csv
crawlens -u https://example.com -v -f results.csv

Command-Line Options

Short	Long	Description	Default
`-u`	`--url`	Starting URL to crawl (required)	-
`-d`	`--depth`	Maximum crawl depth	`3`
`-p`	`--pages`	Maximum pages to crawl	`100`
`-c`	`--concurrency`	Number of concurrent requests	`5`
`-e`	`--external`	Check external links	`false`
`-s`	`--slow`	Slow page threshold in milliseconds	`3000`
`-f`	`--file`	Export report to file (.json/.xml/.csv)	-
`-v`	`--verbose`	Enable verbose output	`false`

Advanced Examples

# Deep crawl with high concurrency
crawlens -u https://example.com -d 5 -p 500 -c 20
crawlens -u https://example.com -d 5 -p 500 -c 20

# Check external links too
crawlens -u https://example.com -e
crawlens -u https://example.com -e

# Find slow pages (>5 seconds)
crawlens -u https://example.com -s 5000
crawlens -u https://example.com -s 5000

# Full SEO audit with verbose output
crawlens -u https://example.com -d 10 -p 1000 -c 15 -e -f audit.json -v
crawlens -u https://example.com -d 10 -p 1000 -c 15 -e -f audit.json -v

# Quick scan with CSV export
crawlens -u https://example.com -d 2 -p 50 -f quick-scan.csv
crawlens -u https://example.com -d 2 -p 50 -f quick-scan.csv

📊 What Gets Analyzed

Link Analysis

✅ Broken internal links (404s, 500s, timeouts)
✅ Broken external links (optional with -e flag)
✅ Redirect chains (301/302 paths)
✅ Orphaned pages (no internal links)
✅ Internal reference counts per page

SEO Checks (Per Page)

✅ Title Tags - Presence, length (30-60 chars optimal)
✅ Meta Descriptions - Presence, length (50-160 chars optimal)
✅ H1 Tags - Exactly 1 per page recommended
✅ H2 Tags - Content structure validation
✅ Content Length - Minimum 300 words recommended
✅ Image Alt Text - Accessibility compliance
✅ Canonical URLs - Duplicate content prevention
✅ Language Attributes - Proper internationalization
✅ Open Graph Tags - Social media optimization (og:title, og:description, og:image)
✅ Twitter Cards - Twitter sharing optimization
✅ Robots Meta Tags - Indexing directives

Performance Metrics

✅ Page load times (with configurable slow threshold)
✅ Total bytes transferred
✅ Average page load time
✅ Crawl duration tracking

Statistics Provided

Total links found
Unique vs duplicate links
Internal vs external link breakdown
Pages crawled count
Average SEO score
Common SEO issues
Duplicate title/description detection

📈 Sample Output

Console Output

🚀 Starting crawl from: https://example.com
Max depth: 3 | Max pages: 100 | Concurrency: 5
Checking external links: false | Slow threshold: 3s
Start time: 2025-01-17 14:30:45

✓ Loaded robots.txt

[1/100 | Workers: 1/5] Crawling: https://example.com
[2/100 | Workers: 3/5] Crawling: https://example.com/about
[3/100 | Workers: 5/5] Crawling: https://example.com/products
...

⏳ Waiting for link checks to complete...
✓ Crawl complete!
End time: 2025-01-17 14:32:18
Duration: 1m33s

======================================================================
📊 CRAWL REPORT
======================================================================

⏱️  TIMING:
Start time: 2025-01-17 14:30:45
End time: 2025-01-17 14:32:18
Duration: 1m33s
Avg page load time: 456ms

📈 STATISTICS:
Pages crawled: 87
Total links found: 1,245
Unique links: 342
Duplicate links: 903
Internal links: 298
External links: 44
Data transferred: 2.45 MB

🔍 ISSUES FOUND:
Broken links: 5
Redirect chains: 3
Slow pages: 8
Orphaned pages: 2

🎯 SEO SUMMARY:
Average SEO Score: 78/100
Pages with SEO issues: 15
Missing titles: 2
Missing descriptions: 5
Duplicate titles: 3
Thin content pages (<300 words): 8
Images without alt text: 45

Most Common Issues:
  1. Meta description too short (< 50 characters) (12 pages)
  2. Some images missing alt text (10 pages)
  3. Content could be longer (< 500 words) (8 pages)

JSON Export

{
  "start_url": "https://example.com",
  "crawl_start_time": "2025-01-17T14:30:45Z",
  "crawl_end_time": "2025-01-17T14:32:18Z",
  "crawl_duration": "1m33s",
  "pages_crawled": 87,
  "total_links_found": 1245,
  "unique_links": 342,
  "duplicate_links": 903,
  "internal_links": 298,
  "external_links": 44,
  "avg_page_load_time": "456ms",
  "total_bytes_transferred": 2568421,
  "broken_links_count": 5,
  "seo_summary": {
    "average_seo_score": 78,
    "pages_with_issues": 15,
    "missing_titles": 2,
    "missing_descriptions": 5,
    "duplicate_titles": 3,
    "thin_content_pages": 8
  },
  "seo_data": [
    {
      "url": "https://example.com",
      "title": "Example Domain",
      "title_length": 14,
      "description": "Example description text",
      "description_length": 24,
      "h1_count": 1,
      "word_count": 450,
      "seo_score": 85,
      "issues": ["Title too short (< 30 characters)"]
    }
  ]
}

🏆 Why Crawlens?

Feature	Crawlens	Screaming Frog	Sitebulb	Other Crawlers
Free & Open Source	✅	❌ (Paid)	❌ (Paid)	Varies
Broken Link Detection	✅	✅	✅	✅
SEO Scoring (0-100)	✅	❌	✅	❌
JSON/XML/CSV Export	✅	✅	✅	Limited
CLI Interface	✅	❌	❌	✅
Cross-Platform	✅	✅	Windows Only	Varies
Concurrent Crawling	✅	✅	✅	Varies
Lightweight	✅ (Single binary)	❌	❌	Varies
CI/CD Ready	✅	❌	❌	Limited

🎓 Use Cases

For Developers

🔧 CI/CD Integration - Automate link checking in deployment pipelines
🐛 Pre-deployment Testing - Catch broken links before going live
📊 Performance Monitoring - Track page load times over time

For SEO Professionals

🎯 SEO Audits - Comprehensive site analysis with scoring
🔍 Content Quality - Find thin content and missing meta tags
📈 Competitive Analysis - Compare site quality metrics

For Content Managers

✅ Quality Assurance - Ensure all links work before publishing
🖼️ Accessibility - Find images missing alt text
📝 Content Gaps - Identify orphaned pages

For DevOps Teams

🤖 Automated Monitoring - Schedule regular crawls
📊 Reporting - Generate compliance reports
🔔 Alerting - Detect issues early

🛠️ Advanced Usage

CI/CD Integration

GitHub Actions

name: Check Links

on:
  push:
    branches: [ main ]
  schedule:
    - cron: '0 0 * * 0'  # Weekly

jobs:
  crawl:
    runs-on: ubuntu-latest
    steps:
      - name: Download Crawlens
        run: |
          wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
          tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
          chmod +x crawlens
          wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
          tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
          chmod +x crawlens
      
      - name: Crawl Site
        run: ./crawlens -u https://yoursite.com -f report.json
        run: ./crawlens -u https://yoursite.com -f report.json
      
      - name: Upload Report
        uses: actions/upload-artifact@v3
        with:
          name: crawl-report
          path: report.json

GitLab CI

crawl:
  stage: test
  script:
    - wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
    - tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
    - chmod +x crawlens
    - ./crawlens -u https://yoursite.com -f report.json
    - wget https://github.com/qacancode/crawlens/releases/download/v1.0.0/crawlens-1.0.0-linux-amd64.tar.gz
    - tar -xzf crawlens-1.0.0-linux-amd64.tar.gz
    - chmod +x crawlens
    - ./crawlens -u https://yoursite.com -f report.json
  artifacts:
    paths:
      - report.json

Scheduled Monitoring (Linux/macOS)

# Add to crontab (daily at 2 AM)
0 2 * * * /usr/local/bin/crawlens -u https://yoursite.com -f /var/reports/daily-$(date +\%Y\%m\%d).json
0 2 * * * /usr/local/bin/crawlens -u https://yoursite.com -f /var/reports/daily-$(date +\%Y\%m\%d).json

Windows Task Scheduler

# Create scheduled task
$action = New-ScheduledTaskAction -Execute "C:\tools\crawlens.exe" -Argument "-u https://yoursite.com -f C:\reports\report.json"
$action = New-ScheduledTaskAction -Execute "C:\tools\crawlens.exe" -Argument "-u https://yoursite.com -f C:\reports\report.json"
$trigger = New-ScheduledTaskTrigger -Daily -At 2am
Register-ScheduledTask -Action $action -Trigger $trigger -TaskName "DailyCrawl"

📚 Documentation

Understanding SEO Scores

Crawlens assigns each page a score from 0-100 based on SEO best practices:

90-100 🟢 Excellent - Follows all SEO best practices
70-89 🟡 Good - Minor improvements recommended
50-69 🟠 Fair - Several issues need attention
0-49 🔴 Poor - Critical SEO problems

Score Breakdown

Title Tag (20 points) - Must be 30-60 characters
Meta Description (15 points) - Must be 50-160 characters
H1 Tag (15 points) - Exactly 1 per page
Content Length (15 points) - Minimum 300 words
Image Alt Text (10 points) - All images should have alt text
Open Graph (10 points) - Social media tags present
Canonical URL (5 points) - Prevents duplicate content
Language (5 points) - HTML lang attribute set
H2 Tags (5 points) - Proper content structure

Common Issues & Fixes

Issue	Impact	Fix
Missing title tag	High	Add `<title>` tag to `<head>`
Title too short	Medium	Expand to 30-60 characters
Missing meta description	High	Add meta description (50-160 chars)
Multiple H1 tags	Medium	Use only one H1 per page
Thin content	Medium	Add more valuable content (300+ words)
Missing alt text	Medium	Add descriptive alt text to all images
No canonical URL	Low	Add canonical link tag
Duplicate titles	High	Make each page title unique

🔧 Configuration

Environment Variables

# Set custom user agent
export CRAWLENS_USER_AGENT="MyBot/1.0"

# Set default concurrency
export CRAWLENS_CONCURRENCY=10

# Set default timeout
export CRAWLENS_TIMEOUT=30s

Exclude Patterns

Crawlens automatically excludes common build artifacts:

/_next/static/ (Next.js)
/_next/data/ (Next.js data)
/static/chunks/ (Webpack)
/node_modules/
/.git/
/dist/
/build/

🤝 Contributing

Contributions are welcome! Here's how you can help:

Report Bugs - Open an issue with details and reproduction steps
Suggest Features - Share your ideas in the issues section
Submit Pull Requests - Fix bugs or add features
Improve Documentation - Help make the docs better
Share Crawlens - Star the repo and tell others!

Development Setup

# Clone the repository
git clone https://github.com/qacancode/crawlens.git
cd crawlens
git clone https://github.com/qacancode/crawlens.git
cd crawlens

# Install dependencies
go mod download

# Run tests
go test ./...

# Build
go build -o crawlens
go build -o crawlens

# Run locally
./crawlens -u https://example.com
./crawlens -u https://example.com

Running Tests

# Run all tests
go test ./...

# Run with coverage
go test -cover ./...

# Run specific package
go test ./internal/crawler

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Credits

Built with ❤️ using:

Go - Programming language
goquery - HTML parsing
robotstxt - robots.txt parsing

📞 Support

📖 Documentation: Wiki
🐛 Bug Reports: Issues
💬 Discussions: Discussions

🗺️ Roadmap

v1.1.0 (Planned)

v1.2.0 (Planned)

JavaScript rendering support (headless browser)
Authentication support (login forms, OAuth)
API mode (REST API server)
Screenshot capture for broken pages

v2.0.0 (Future)

Distributed crawling
Plugin system
Web UI dashboard
Historical trend analysis

⭐ Star History

If you find Crawlens useful, please consider giving it a star! It helps the project grow.

Made with ❤️ by Gopalakrishnan

⬆ Back to Top

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
internal		internal
.gitignore		.gitignore
README.md		README.md
build.ps1		build.ps1
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Folders and files

Latest commit

History

Repository files navigation

Crawlens

🚀 Features

Link Analysis

SEO Analysis

Performance Metrics

Export & Reporting

Politeness & Compliance

📦 Installation

Download Pre-Built Binary (Recommended)

Linux

macOS

Windows

Build from Source

🎯 Usage

Quick Start

Command-Line Options

Advanced Examples

📊 What Gets Analyzed

Link Analysis

SEO Checks (Per Page)

Performance Metrics

Statistics Provided

📈 Sample Output

Console Output

JSON Export

🏆 Why Crawlens?

🎓 Use Cases

For Developers

For SEO Professionals

For Content Managers

For DevOps Teams

🛠️ Advanced Usage

CI/CD Integration

GitHub Actions

GitLab CI

Scheduled Monitoring (Linux/macOS)

Windows Task Scheduler

📚 Documentation

Understanding SEO Scores

Score Breakdown

Common Issues & Fixes

🔧 Configuration

Environment Variables

Exclude Patterns

🤝 Contributing

Development Setup

Running Tests

📝 License

🙏 Credits

📞 Support

🗺️ Roadmap

v1.1.0 (Planned)

v1.2.0 (Planned)

v2.0.0 (Future)

⭐ Star History

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages