fix: ensure dispatcher cleanup on crawler shutdown

[rozin-not-found]

Summary

Fixes #2083.

This PR resolves a task/memory leak that occurs when AsyncWebCrawler.arun_many() breaks early during a stream. Because Python delays finalization of async generators, relying on the generator's finally block caused race conditions (RuntimeError: aclose(): asynchronous generator is already running). This PR implements a crawler-level dispatcher tracking system that explicitly forces a native cleanup during the crawler's teardown phase.

List of files changed and why

• crawl4ai/async_webcrawler.py: Added dispatcher tracking (self._dispatchers) to AsyncWebCrawler and updated the teardown phase to enforce cleanup on shutdown.
• crawl4ai/async_dispatcher.py: Implemented a native cleanup() method in MemoryAdaptiveDispatcher to explicitly cancel active tasks and stop the memory monitor.

How Has This Been Tested?

• Created a local reproduction script measuring len(asyncio.all_tasks()) before starting the crawler and after the async with context manager exits.
• Verified that breaking an async for loop over arun_many(stream=True) correctly clears all background tasks.
• Verified the ResourceWarning is completely eliminated.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes