Recommended reading - Webbots, Spiders, and Screen Scrapers

The Internet is bigger and better than what a mere browser allows. Webbots, Spiders, and Screen Scrapers is for programmers and businesspeople who want to take full advantage of the vast resources available on the Web. There's no reason to let browsers limit your online experience--especially when you can easily automate online tasks to suit your individual needs.

Learn how to write webbots and spiders that do all this and more:

Programmatically download entire websites

Effectively parse data from web pages

Manage cookies

Decode encrypted files

Automate form submissions

Send and receive email

Send SMS alerts to your cell phone

Unlock password-protected websites

Automatically bid in online auctions

Exchange data with FTP and NNTP servers

Sample projects using standard code libraries reinforce these new skills. You'll learn how to create your own webbots and spiders that track online prices, aggregate different data sources into a single web page, and archive the online data you just can't live without. You'll learn inside information from an experienced webbot developer on how and when to write stealthy webbots that mimic human behavior, tips for developing fault-tolerant designs, and various methods for launching and scheduling webbots. You'll also get advice on how to write webbots and spiders that respect website owner property rights, plus techniques for shielding websites from unwanted robots.

Some tasks are just too tedious--or too important!-- to leave to humans. Once you've automated your online life, you'll never let a browser limit the way you use the Internet again.

Reviews

"Webbots, Spiders, adn Screen Scrapers" is a solid book for building basic scripts to do web scraping. Michael Schrenk goes covers the "should you do this" aspect very well, and devotes much of the book to these kinds of topics. On that reason alone I give him major kudos, "just because you CAN do a thing, doesn't mean you SHOULD." Technically the book and examples are very basic and beginner level. All code is procedural and has absolutely no references to object oriented programming at all. This is great for a simple project, but building anything larger than a targetted webbot or two is beyond the scope of this book. I was very dismayed at Mr. Schrenk's opinion of regular expressions: "The use of regular expressions is a parsing language in itself, and most modern programming languages support aspects of regular expressions. In the right hands, regular expressions are also useful for parsing and substituting text; however, they are famous for thier sharp learning curve and cryptic syntax. I avoid regular expressions whenever possible." This disregard for regular expressions effectively wipes out a powerful toolset for budding developers. Regular expressions are no harder to learn than PHP. The reasons for his disdain for them is also flawed: "The regular expression engine used by PHP is not as efficient as engines used in other languages, and is certainly less efficient than PHP's built-in functions for parsing HTML." PHP uses the same regular expression engine used (very effectively) in PERL with the use of the preg_* functions. There has been many studies that show preg_* style expressions outperform basic text matching in PHP. In this assesment the author is terribly wrong. The book does a great job of explaining how to make single use scripts for scraping, but never how to create a larger infrastructure. There is no focus on creating multi process engines with pcntl_fork(), or proc_open(), these are critical for scaling web scraping applications. A single script scraping a few hundred websites on a single thread would take ages over a multi-threaded engine. If you are looking to break into web scraping and not sure where to start, this is likely the best (and possibly only) book on the market. If you are intermediate or advanced you will quickly question the author's logic and see that scaling will become the number one issue you have to over come.
I picked up this book full of enthusiasm, spiders are just plain cool, they go out and start downloading data for you, reading webpages, and even understanding them a little. My enthusiasm was dashed a little however on page four: You may use any of the scripts in this book for your own personal use, as long as you agree not to redistribute them... and agree not to sell or create derivative products under any circumstances.. I develop in PHP professionally, and a lot of the code I write ends up getting used somewhere with some sort of a for-profit basis, which pretty effectively prevents me from using any code between the covers (at its strictest reading, Im not sure I can even change the code). The book does a great job of introducing different sorts of web agents that you can create programatically (more than just spiders) and introduces all sorts of interesting projects along those lines. Throughout the book a series of libraries written by the author are leveraged to make the retrieval and parsing of the various pages much easier. While newer developers will enjoy being able to concentrate on the big picture I found myself itching for more information on the nitty gritty. Some of the projects explored include: price monitoring, image capturing (want to be your own google image search? :) ), link verification, spiders, and snipers. Each of the different projects received its own chapter, and effectively covered a lot of the topics covered within. Overall, I would recommend this book to beginner to intermediate PHP developers looking to tackle the world of web agents, its a good primer on the related topics, and at the very least will give you some ideas on the complexities involved. As their skill grows they will probably find them-self either moving past the libraries included with the book, or modifying them greatly. My biggest complaint is the lack of coverage on the robots.txt file, some talk is given to it in terms of blocking robots from your own site, but I didnt see any code that actually dealt with parsing it for your own robot.
I waited months for this book to come out and the wait was worth it. This is a great introduction to webbots , spiders and scrapers. The writing is easy and never boring. Lots of code examples and resources to tap into. I couldn't put it down. When was the last time you got a computer book that made you run to the keyboard to try something out? I'm sure there will be some a#$h@#e that will say it's too rudimentory. It's an intro and it takes you up to intermediate and explains stuff about PHP that I didn't even know existed. Definitely worth the money. I can't wait for the sequel.
This book covers every aspect I could ever hope a book on web bots would cover. It goes into great detail and provides lots of background information about things such as why you should use web bots, security issues, how to authenticate a bot with password protected sites, writing search engine crawlers, parsing HTML, how to handle cookies, HTTP headers, dealing with forms and a lot more. I was very pleased with how this book covered concepts. The book uses PHP and the cURL library as a teaching tool instead of trying to give a lesson in how to use PHP as a crawler language. The way the code is explained makes it very easy to translate into whatever language you are most comfortable coding in. The book uses fundamental functional programming concepts which make it easy to pick up the general idea without actually knowing PHP. My boss bought this book to help my group us with a project we were working on, and even my co-workers who had no background with PHP were able to use this book to write a web bot in C# (using the cURL library) very easily. The concepts from this book easily transfered over to object-oriented concepts.
This book is simply awesome. You will need to come armed with at least a basic knowledge of php, but everything is pretty straight forward. The projects are well explained and applicable to a wide range of projects that you might be getting yourself into.
The power of this book is not so much in it's code examples but rather in it's ability to change your perspective. We are all aware that the Internet is a client-server topology, but what does that really mean? Reading the first few chapters gave me a whole new viewpoint of the Internet and what I could do with it, or to it. In the year since I first read it, I have stopped developing websites and now code web agents exclusively. It's amazing the number of uses they fulfill. The code in the book is basic, not fit for production (the author tells you this) but it is invaluable to teach the theory and fundamentals of CURL. If you use the code and the provided website to practice with, you will soon be able to develop your own code library. Scale is also left to you to figure out. The obvious first step is a database and a NAS. Start small and use this book for the invaluable reference it is. I really have to rate this book as one of my most influential reads of the last few years.
This is a great introduction on the subject. The supplied PHP library does all the work.
This book is not like very algorithmic, but you can know the basic of webbots writing and some techniques involved. curl is good for starters, but it is the ideas instead of the codes that help us understand the concept. What you need to do is not copying the code, but to study what it does and why things are implemented. Good book. 5/5
Need to learn how to browse the web with your own software instead of manually browsing? The is the best book on the subject. Written for people new to writing webbots, the example code is straightforward. A basic understanding of php is sufficient for understanding the examples. Michael Schrenk takes you directly to the point of the book with fully explained examples. They are specific-use scripts, which makes them easy to learn from. With an understanding of the basics, you can combine and extend the sample projects to build larger multi-purpose webbots on your own. The example scripts can be tested against the authors' web site to ensure consistent results. Most of the material naturally deals with browser emulation. In addition, there are chapters on POP3 mail servers interfaces, FTP webbots, and NNTP newsgroup interfaces. This is a great basic book that will take you from curiosity to a working knowledge of webbot authoring in a short time period.
I won't re-iterate the excellent reviews already posted on this book, other than to say this is probably my favorite all-time programming book: excellently written, highly informative, and full of smarts. Plus, if you're not too aware of how powerful and useful web spiders can be, this is the book for you.