Reading a big file into a list can kill your program. A generator handles one line at a time. See the memory difference measured on a real file, learn what yield does, and chain small generators into a pipeline.
asyncio.gather runs many waiting tasks at once in one thread and returns results in order. Avoid await in a loop and blocking calls, keep failures with return_exceptions=True, and limit concurrency with a Semaphore.
Don't guess which part is slow. Run cProfile, read cumtime and tottime the right way, fix the one line that matters, and measure again. In the example, the obvious suspect isn't the problem.
Downloads spend most of their time waiting. ThreadPoolExecutor lets many waits overlap: 20 half-second downloads in 2 seconds instead of 10. Handle failures with as_completed, pick a sensible worker count and respect the server.
We value your privacy
We use cookies to keep this website working and, with your permission, enable additional features, measurement and advertising. Accept all, reject all optional cookies, or learn more to customize your choices. Necessary cookies stay active.
