BookAdder: Search TuningBut First!Please be sure you have already installed and done a trial first run with BookAdder, as set forth in the preceding BookAdder docfile Installing BookAdder. Please: RTFM! This docfile describes what is normally a part of the 10-step initial package installation, so that's why it will begin with a heading referring to "Step 7". Some SEO NumerologyBut first, it's worth our while to take a moment to review what we're doing and why and how. First, know this about Amazon: no matter how many items Amazon carries that match a given search, their data feed will never return more than 4,000--so that's as many titles as one can get from any book search, no matter the topic. Moreover, there is always some "leakage", mainly of two sorts: "books" that aren't really (they're calendars or greeting cards or some such), typically about 3%; and duplications (which ought not to happen but do, roughly another 3% loss). Also--but, so far, only in the US division--searches return a lot of results classed "digital", usually signifying downloadable items; BookAdder takes the position that such items are not "books", and rejects them, so there is a currently non-trivial loss owing to that issue. (There seems no way whatever to narrow searches to "true books"; considering how hard Amazon is pushing its "Kindle" electronic book reader, that is scarcely surprising.) (In principle, one could get a few more title listings by not excluding trashy non-book items, but--in my opinion--the gain is very small compared to the loss of credibility in your visitors' eyes that your shop would experience if it included such gewgaws.) If we choose our search phrase so that at each division it will give us the full 4,000 titles, we will in actuality end up with an average of perhaps 3,765 titles per division from that 4,000. That's 22,590 titles--site pages--in all. But wait! There's more! (As they say on TV.) Each title, at each division, actually produces five site pages for you: one for the new-book-at-Amazon page, one for the used-copies-at-Abebooks page for that ISBN, one for the change-Abe-parameters page for that ISBN, one for the Abebooks search for that title/author combination, and one for the Abebooks-parameters page for that title/author combination. So the 22,500 titles become nearly 113 thousand distinct new pages on your site. (In strictest exactness, you will usually not get precisely 5.00 pages per title, in that Amazon has a few specialty items Abebooks doesn't have, but in practical reality the value will range from about 4.90 to 4.99 pages per title, plus you get another 35 fixed pages per division no matter the titles count.) In times gone by, there could have been some question as to whether all those pages were "useful" in an SEO sense, in that searchbots were widely rumored to read no more than the first 101 kilobytes of any web-page file. Whether that is or ever was true is, however, no longer relevant, owing to the wonderful advent of sitemaps. This is not a tutorial on sitemaps--there already is a satisfactory one at sitemaps.org. It is merely the observation that we no longer need worry about whether we have "saturated" a given page of links by either sheer byte length or by number of links. We can now tell the search engines directly about all our pages. We still need to have those links out there on our own pages, but the sitemaps now carry the real load. If you're wondering if all those pages really will end up indexed, all I can say is that from my own experience using BookAdder on about a dozen quite diverse web sites (some of which are quite high in SE rankings for their topics), Google reports site-page counts that, while varying in individual cases from a low of 19% to a high of 83%, average over half (53%) of all bookshop pages reported. Whether the page report is inaccurate (no one knows if Google, or any engine, reports full site page counts, whether by error or on purpose, but what they do report is necessarily a minimum) or whether some are deemed close variants of others, who knows? But it is absolutely, positively certain that 1) at least tens of thousands of pages are recognized by Google, and 2) that based on site placement, they can only be helping, not hurting. I emphasize again: these are not "junk" pages or scrapings or search results or any of the other garbage the search engines so rightly complain about. These are honest, functional, relevant, dynamic, content-rich pages that are a true augmentation of your site in the fullest sense, not just by the numbers. Constructing PhrasesThe mechanics of the phrase are important. The search done is an Amazon "Power Search" of books, and the field used is "keywords". The syntax is Boolean, which you need to know. If you need a full tutorial on how Boolean phrase construction works, find one by using a search engine. Here is the bare outline (which really ought to be enough). Note that a simple series of words is interpreted as if there were an and between each word; that is, flower gardens is effectively the same as flower and gardens as a search term. Boolean CombinationsBoolean phrase-construction uses the logical operators and, or, and not to mix and match various ordinary words. You can also use parentheses to group "sub-phrases" into a larger whole. It's probably easier to show than to try to explain at length. Note that in Amazon book searches, the or is always exclusive: A or B but not both.
For those needing to construct some fairly complex expression to either get enough relevant titles or to best focus the titles, the lack of an "inclusive OR" term is a very serious handicap (but the sort of folly we have come to expect from Amazon). If, for an example, one wanted to find books related to either "home repair" or "house repair", the normal Boolean approach would be to use (house or home) and repair; but with Amazon's "or" being exclusive, that would omit books whose subject description used both of the words "house" and "home". It would return only books whose description used only the word "house" or only the word "home", but not those using both. Nuts, counter-intuitive, tedious, difficult, and typically Amazon. When you need to combine words into a fixed phrase, you can group them inside double-quote marks. A search for horror movies will return books whose descriptions include the words "horror" and "movies", but that is not quite the same thing as descriptions containing the phrase "horror movies". There is a further serious complication: Boolean search-term combinations are not available in all divisions. The "power search" that BookAdder uses as described above only works in the three Anglophone divisions (US, UK, CA); it does not work in the three non-Anglophone divisions (DE, FR, JP), even though those divisions carry about as many English-language books as the others (sometimes more, because they can carry both US and UK versions of the same book, which copyright problems may bar the others from doing). Searches for "Foreign Books" in the non-Anglophone divisions are constrained to "keyword searches", which are not Boolean. That stupid (Amazon, Amazon) restriction means that BookAdder has to use a half-clever scheme to try to allow complex (Boolean) searches used for the Anglophone divisions to somehow be "translated" to something that will at least sort of work in the non-Anglophone divisions. The scheme for now, till Amazon can be persuaded to amend its ways (don't hold your breath waiting) is this, and I know it's feeble: if you do need Boolean "complex" searches for the Anglophone divisions, you must start the search term with some term in parentheses, such that the first character of the total search is an open parenthesis ( character; BookAdder will use the full Boolean search term in the three Anglophone divisions, but will use whatever is in that opening set of parentheses as the "keywords" search term in the non-Anglophone divisions. An example might make this clearer. A site that wants books about home repair might use this complex Boolean search term: (home repair) or ((house or home) and (repair or maintenance)) That illustrates both Amazon problems. In the Anglophone divisions, it will (or may--it's hard to be sure) miss books described with both "house" and "home" in their description, but it will capture more total books than just, say, plain "home repair". Meanwhile, in the non-Anglophone divisions, the "keywords" search term used will be that in the first parentheses set, home repair, which at least catches a fair share of the wanted books, and which we can do no better than with any other term. Principles of BuildingFor many sites, you will be able to get the wanted minimum of 4,000 titles per division with some quite simple search phrase--very possibly even a single word. For example, the simple search word cats produces--at some one moment, remember, these things change almost by the minute--these results:
By the way, the "note" referred to in the listings above is simply that--as we already know--anything over 4,000 might as well be 4,000. How we get those data you'll see in a moment; meanwhile, that is not perfect results--the UK division is not saturated at 4,000--but it's pretty good for a single word. And one could experiment with other possibilites. But often, if your site's topic is rather narrowly focussed, you have to get creative. Here's an example of how to go about it. The general way to proceed is to start with the narrowest phrase reasonably likely to produce some non-trivial number of titles. The narrower the scope, the more strongly relevant the list will be to your site's theme. For example, if you have a site about the history museums of Ritzville, Washington, you don't even have to try it to realize that you're just not going to get anywhere with Ritzville and history as your phrase (and sure enough, it produces zero results everywhere). But you can start by generalizing just a bit: try "washington state" and history; now we're getting somewhere--1,238 titles in Amazon U.S. and at least several hundred each in the non-Anglophone divisions. But we should be able to do better; and the usual way of "doing better" is to expand the scope. Here is a summary of a possible evolution of search ideas for this case: For the search keyword phrase (history and "washington state"): Note that the sum of Amazon U.S.A. for the first two is 1,960 titles, but or'ing them only produces 1,847 titles; clearly (because of Amazon's bizarre choice of how to interpret or, we lose the 113 titles in whose descriptions both phrases appear. Still, it's an improvement over either alone. But if we widen out concepts a bit more, we should be able to do better yet. For the search keyword phrase (history and "american west"): That last is about as good as it's going to get: we have saturated the Anglophone divisions and done the best we could in the non-Anglophone divisions (in which, remember again, we cannot use any Boolean terms at all save the implied "and" between each word or quote-delimited phrase). While there is some question of the degree of closeness of "American west" to the history museums of Ritzville, Washington--but I for one would reckon that the "history" part makes for reasonable relevance. Anyway, you now get the idea (I assume). Actual TrialsStep 7: Develop Your Actual Search PhraseSo now you need to know how to quickly and easily test your various possible search phrases. Simple: just run the BookAdder script bookcount.php. At first run, it will use the phrase built into your customize.php file--right now, presumably (unless you fiddled with it already) "spokane". The page would look something like this:
The process couldn't be easier: type your phrase into the text box, click the button, wait a few seconds (6 to 12) for the results, and there you are. (Note that the display above is not functional, as it has no way to know your site's URL.) Note! Phrases containing any apostrophes ( ' ) will come back showing an escape backslash in front of each apostrophe ( \' ); the backslash is required, and will be supplied automatically if you forget it; just don't be amazed or upset when it appears. Here is an illustration of what you'd get (at one particular moment, anyway) using the search phrase cats:
You can play around with possible phrases till the cows come home using bookcount and the instructions above. Just remember that your ideal would be to have each of the six divisions report a saturated 4,000 titles, with an actual not much higher; but if the search is "complex"--Boolean--then you will have to settle for lower counts in the non-Anglophone divisions. (That "not much higher" point is scarcely critical; it would just suggest that your phrase is as narrowly focussed as reasonably possible while still getting the wanted minimum of titles.) Step 8: Apply Your Chosen Search Phrase to the CustomizationWhen you have finally settled on what you are sure you want to use as your real search phrase, do these things:
You have now completed all basic customization. (There is much more you can customize, described in a separate docfile described farther below in this file, but you are now ready to begin.) Step 9: Do Your First Real, Full Run (and some other things)You do this very simply: you run the script cronf.php. What cronf will do is simply start monitor.php as a "free-running" (daemon) process; once it starts monitor, cronf waits five seconds (pure paranoia), then exits, its job done. cronf will deliver an extremely brief message--its PID--and nothing else, like this:
Wait 5 seconds (or till your browser's "loading" symbol stops), then just exit the page. The run has started: monitor is running and functioning just as it did when you ran it by hand, except that having been started by cronf, it simply does its job silently in the background. The only ways for you to keep track of what is going on--if you feel a need--is to use status.php. (Well, you could manually inspect the underlying data and log files, but that buys you nothing except needless extra complexity.) My advice is to not be a nervous Nellie: check the Status Report every ten minutes or so if you really must, otherwise just give it an hour, then check Status. (If you have made--and properly applied--your new search-term entries, you can start cronf.php right from this link.) Meanwhile, there are a couple of useful ways you can fill that hour or so while your first real search is running. Update Your robots.txt FileNote! if you are using, or ever used, any earlier version of this software, you should edit your robots.txt file to remove all exclusions specific to that package; you can spot them by the directory name in their path, that being whatever directory you were previously using for the software; if the BookAdder directory name remains the same, the installer will help out there--otherwise you need to do it manually. We will deal with new additions to your robots.txt in a few minutes. When you ran the installer, finstall.php, one of the things it did was locate and scan your existing robots.txt file (if you had one), and to combine its contents (if any) with new entries appropriate for BookAdder, presenting the results in a new file named robots.new stored in your main BookAdder directory. If it found lines in your existing robots.txt file that refer to your bookshop (BookAdder) directory, it edited those out (but only in robots.new--BookAdder does not itself ever do anything to your real robots.txt file). What you want to do now is to eyeball-review robots.new to make sure it is correct, then--if it is--manually substitute it for your existing robots.txt file. So that you can correctly evaluate the new file, here is some education about robots.txt files. General robots.txt ConsiderationsAs you should and doubtless do know, your robots.txt, which must reside in your site's root directory, exists to tell searchbots certain things--in particular, to tell them which directories and files to not try to list in their indexes. There can be many reasons why you don't want each and every single file on your site to be listed in some search engine: some may be php scripts no one else has any business running, some may be test or archived files, some may be private, and so on. This file allows you to exclude those files from the bots' probing. (That's assuming the bots are "well behaved"--many scummy outfits and individuals ignore robots.txt directives, but BookAdder contains other ways of slowing such swine down, more about which we'll discuss later.) Though most webmasters know about robots.txt files, and use them, a surprising number don't know their actual syntax, and so make basic errors--especially since the syntax rules have been substantially changed quite recently (in June, 2008). While there are on-line guides to the new syntax available, let's take a moment to review the highlights pertinent here to make sure you have them right. (But this is not an exhaustive tutorial on robots.txt files--just a touch on the highlights.) Originally, and to a large extent still, robots.txt files were meant as a way to tell "well-behaved" searchbots what areas of your site to not explore (such as private or temporary working directories); the "well-behaved" part is because honoring the contents of a robots.txt file is purely voluntary on the part of the searchbot, and many black-hat outfits (like email "harvesters") largely or wholly ignore such files. Newer developments now allow robots.txt files to also convey a growing amount of other information to searchbots. Be aware, though, that not all searchbots necessarily "know about" the newer protocols (though all of the Big Three--Google, Yahoo, and MSN--do). A robots.txt file is made up of "directives": a directive is a particular instruction to searchbots, and is one line in a robots.txt file. Directives appear in "blocks", each block constituting a set of directives that apply to a particular searchbot or set of searchbots; which 'bots a block is meant for must be specified at the head of the block. Also (what many overlook), blocks (if you have more than one) must be separated by a blank line in the robots.txt file. Most everyday webmasters will use only a single directive block, aimed at all searchbots; the need to have separate directives for different 'bots is unusual. A block starts with a User-agent declaration, which specifies which 'bots it is meant for. There's no a priori way to know the correct User-agent value for a given engine: you just have to look it up somewhere. If, though--like most webmasters--you are content to use the same directives for all searchbots, the block-head declaration line is very simple: User-agent: * The following directives used to be just "Disallow:" followed by a filespec that covered what you wanted blocked; now, however, you can also have "Allow: declarations. But be sure you understand the basic rule of User-agent files: Whatever is not expressly blocked is allowed. The general strategy in designing directive blocks is to use a sweeping form of Disallows followed by a few particular Allows that constitute exceptions to the Disallow blocking. As an example, a sample BookAdder directive block (sticking just to the Disallow/Allow directives) might look like this: User-agent: * Disallow: /red-cats-books/ Allow: /red-cats-books/allbooks/allbooks. Allow: /red-cats-books/red-cats-books.php Allow: /red-cats-books/abechange.php Allow: /red-cats-books/abes.php Allow: /red-cats-books/book-search.php Allow: /red-cats-books/free.php Allow: /red-cats-books/holder.php Allow: /red-cats-books/search.php Allow: /red-cats-books/used-books.php Allow: /red-cats-books/Bookshop_index.xml Allow: /red-cats-books/red-cats-books-sitemap The Disallow directive initially blocks the whole package; then the Allow directives give access to the particular files intended for visitors to see, which is all we need or want the searchbots to know about. The rule when a particular page is covered by more than one directive (as are the Allow'ed pages above, which are covered in both the sweeping Disallow and their individual Allow's) is that the longer--that is, the more exact--directive applies. The order of directives is said to be unimportant within a block. Although explicit wildcards are now supposedly allowed in filespecs, I feel it is better to continue with the older method of "implied wildcards", because some 'bots may still have problems with ? and * wildcards--plus wildcarded specs can be difficult to construct so as to do what you think they will. "Implied wildcards" means that the searchbots only seek to match a given page's full spec with as much of a spec as a directive line gives; that is equivalent to having each spec that is not an exact, particular file be followed by a * wildcard. For example, the directive in the sample above, Allow: /red-cats-books/red-cats-books-sitemap would match any filespec that starts with that spec; it thus matches (among others) /red-cats-books/red-cats-books-sitemap-uk.xml as well as /red-cats-books/red-cats-books-sitemap-us.xml.gz. All specs you use must start at the root of your site, which means that even for files in the root directory itself, that leading slash / must be present. Think of the spec as what you would have to tag onto your domain URL to reach that file or directory. The sole exception to the rule that robots.txt directives apply only to such search engines as are specified at the head of a directives block is one very important to us: it is the Sitemap: directive. That directive is "stand-alone" and is always considered available to all searchbots (that know how to use it, which includes the Big Three and, one assumes, all significant players). There can be more than one Sitemap: directive in a robots.txt file; normally, one points such directives at index maps rather than at individual maps, but that's up to each webmaster. The question of where to place such directive lines is, to my mind, not clearly answered; the sitemaps.org official site says: This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. I take that to mean that you can place it within any block or outside of a block, but I have not tested; and I still think it better to expressly inform the major engines of your sitemap locations--especially since that means that you can then notify them every time the maps are updated, whereas relying on robots.txt directives leaves it up to the 'bots to notice when and if your maps have updated. There is one other useful directive, though not all of even the major search engines recognize it (Google does not, Yahoo and MSN do): the Crawl-delay: directive. It specifies, for those 'bots that recognize and honor it, the minimum interval, in seconds, between searchbot indexing "hits" on your site, and is intended to stop 'bots from dragging down a site's performance by rapid hitting. I recommend its use, but BookAdder does not rely on it, because evil 'bots just ignore it; instead, BookAdder has a special built-in throttler for misbehaving 'bots, more about which later. (Note, though, that even the "best" searchbots will sometimes try to hammer a site.) Finally, note that robots.txt files are case-sensitive: you have to enter directive leaders (such as User-agent:) in the exact case shown in these examples, and you must include the colon after each. Updating robots.txt For BookAdderAs part of its install process, BookAdder reads your current robots.txt file (if there is one) and generates a suggested replacement, which includes the original contents plus add-ons for BookAdder. That suggested replacement is stored in your main BookAdder directory under the name robots.new. The installer tries to locate and omit any lines in your extant robots.txt that look like older BookAdder-related (or even "Freebie"-related) directives, but it cannot deal with all possibilities; it looks for, and omits, lines that contain either the term /freebie/ or the current name of your new BookAdder directory. But if you had an older package in a directory whose name differs from the new one you are working in, the installer cannot and will not know that, so you'll have to weed out any of those by hand from robots.new before using it. So, note carefully: It is up to you to fastidiously check robots.new before using it! If, after such careful eyeball checking, the suggested robots.new looks correct, substitute it for your current robots.txt (in your site's root directory). (Mind, there's no harm in having outdated exclusion directives in your robots.txt--you're just telling the 'bots not to index directories that don't exist anyway--but it's a good idea to keep that file neat and easily comprehensible.) Updating your robots.txt file was the first useful way of passing time while waiting for your first real search run to finish; now for the second way. Set Up SitemappingIt is important that you have and use sitemaps, not just for this project but for your site as a whole. As I write this, both Google and Yahoo make use of sitemaps, and MSN is expected on board more or less any day now (they may already be by the time you read this). A sitemap is just what its name says: a map of your site. It is a list of all the pages you would like to have in a search engine's index, but it's more than just a flat list: it's a list with other helpful guidelines for the searchbots, such as how often a given page is likely to change, how important you think it is relative to the other pages on your site, and perhaps more. And it is formatted in a certain exact special way, as an XML file. For further details on sitemaps, go to the horse's mouth, the collaborative sitemaps.org site. For right now, what we are concerned with is setting your site up initially to make use of the sitemaps that BookAdder automagically makes for you at every run, because you need to get your sitemaps registered with Google and Yahoo (and maybe MSN) before they will use them. (Well, they may use them anyway, but by registering you are enabled to send the engines a notification whenever your maps change, instead of relying on their finding your updates at their own pace.) BookAdder makes no files outside its own directory, not even sitemaps or sitemap indexes. That means that even if you already are using sitemapping, you will need to separately set up the master map-index file for BookAdder at each engine's site. That way, there is no entanglement between the BookAdder sitemaps and any maps you may provide on your own for the rest of your site. (Mind, you could include the BookAdder index map in your site's main sitemap index--if you have one--but I think it wiser to keep it completely separate for future convenience accomodating any changes.) What Maps BookAdder MakesSitemaps actually come in two sorts: true maps, and "index" files, which are--in essence--maps of maps. Index maps exist because it is often much more reasonable to map a site into several distinct maps rather than one grand, all-inclusive map. (In fact, sometimes you have to make multiple maps, because the sitemap protocol imposes upper limits on number of pages per sitemap--50,000--and sitemap byte length, 10 MB). Index sitemaps are just that, indices: they are maps of the existing actual sitemaps. BookAdder sitemaps itself, that is, all the search-engine-indexable pages it contains and makes. It does that as six distinct map files, one for the "pages" you have for each Amazon division. The files are gzipped (which the protocol allows and the search engines, and common sense, recommend) to save searchbot time and your bandwidth use. The titles of the mapfiles BookAdder makes are of this form:
There, the red-cats-books is whatever is the actual name of your main BookAdder directory. If any map were to be so large it would need to be chunked into more than one file (which will not be the case till Amazon lifts or increases that insane 4,000-item limitation, if it ever does), the files would be, say for the U.K. division: red-cats-books-sitemap-uk.xml.gz You do not, however, want to register those maps with the search engines; instead, you will be registering your site's master bookshop sitemap index file. That file is named Bookshop_index.xml, and it was automatically made for you when you ran the finstall installer script; it now resides in your main BookAdder directory, where it belongs. Every time you do a search run (except in Test Mode), the run concludes by re-making the six divisional maps listed above, then modifying the index file to reflect the date and time of the newest maps. Not only that, but when it finishes mapping and updating the index, BookAdder will then automatically notify the search engines that an updated set of maps is ready--you don't have to do anything. (If, for any reason you ever want or need to re-create the Bookshop_index.xml file afresh, you can run the ancillary BookAdder script makeindex.php. It will not over-write an existing index file; if it finds such a file, it will instead make a new version under the name Bookshop_index.xml.NEW.) Registering With the EnginesIf you're not using sitemaps now, you should. Here's how you go about registering for sitemap use with the search engines that use them. The general process is this: you tell the search engine, on a special page of their site, that you want to register your site with them; usually the "registration" will include not only sitemapping but other useful services from them as well. They, in response, will give you some "magic" file, typically with a long alphanumeric scramble as its name, that you then need to post in the root directory of your site; when you have posted it, you go back to the search engine's site and tell them it's there. (That is so they can be assured that it was indeed the site owner who registered the site.) Note! For Google, since your bookshop sitemap-index file will be in the BookAdder main directory, you have to list that directory as a separate "site" before you can register the map itself. That's just the way it works. You add your actual site for its sitemapping, then you add the BookAdder directory so you can list its map. (Yahoo seems to accept the bookshop index file with no special problem, so you don't need to add the directory as a separate "site" for them.) As to sitemaps, they will ask you for the name or names of your sitemap files (or what Yahoo chooses to call "feeds"). Give them the full URL to your BookAdder master index file, which ought to be something of this form: http://www.friends-of-red-cats.com/red-cats-books/Bookshop_index.xml It would be wise, though, to wait till your first real run has finished, and the maps have been made and the index updated, before registering the file with the engines. To force sitemap-making after a completed run (if, for example, it was made in Test Mode, with no mapping done), just run the script named makemaps.php. Mapping the Rest of Your SiteIf you already have the rest of your site mapped, well and good, and you need nothing further here. If you do not have any non-BookAdder sitemaps, however, BookAdder includes a "bonus" tool that will help you make one--a script called makestatic.php, which makes an output file named staticmap.xml. There are various points about that script that you should clearly understand. Note this carefully: I provide BookAdder, and I provide assured mapping of the files that BookAdder makes. The staticmap-file script is a bonus, and I expressly do not guarantee it. In reality, I think it works OK, but it is your responsibility to check every file listing in the static map. Even if you need to hand-modify a lot, it's easier than building the whole thing from scratch by hand. (There are also many other tools for building sitemaps, and they probably work as well or better than mine; but here it is, it's free, and you might get some good out of it.) The makestatic script first looks into your robots.txt file, and will not list any file excluded there. Otherwise, it goes through your site's directories and subdirectories looking for all files with any of a few particular extensions--.html, .htm, .shtml, and .php--and adds any not excluded by robots.txt to its list (but it does not map any files in or below your BookAdder directory, because BookAdder maps those itself). When done, it converts that list to mapfile format; it then checks to see if you already have a staticmap.xml file; if you do, it further checks each entry in the tentative new file to see if there is a corresponding entry in the existing file, and if there is it uses the ancillary data (update frequency, relative priority) in the new listing. Then it makes a new staticmap.xml in your BookAdder directory--it is up to you to eyeball-validate that file, then move it to your root directory, then register it with the search engines. Here ar the points to consider:
Let me elaborate on the special-characters issue. If all of your directory and file names are "simple", meaning that they are composed wholly of alphanumeric characters-- a - z A - Z 0 - 9 --there is no dificulty. Potential problems aris if you have used any "exotic" characters in your directory or file names. I will not rehearse here the various complications associated with "URL-encoding", "HTML-encoding", "escaping", and so, but will simply spell out exactly how the static-map maker works. The bottom line is that you must eyeball all of the generated file listings to assure yourself that they conform to the standards of sitemap specifications. My advice, always and ever, is to avoid using non-alphanumeric characters in directory or file names. Your site's URL is always used as PHP reports it--typically of the form: http://wwww.mywonderfulsite.com After that, makestatic separates the rest of the filespec, as PHP has reported it (which means as your server's filesystem reports it to PHP), into its components, using your local separator slash (which will be a plain slash / under Unix and a backslash \ under Windows) as the component divider. Thus, a spec like-- /index.html --will have just one part, index.html. But a spec like-- /français/puré/good&bad/some good & some bad.html will be separated into the components: français (The character between the "n" and the "a" of "français" is a "c-cedilla"; and the character at the tail of "puré" is an "e-acute"; both are non-ASCII, accented characters.) The script then applies "URL encoding", as required by the sitemap specifications), to each of the separated parts in turn (it works this way so that the separator slashes do not themselves get encoded). That encoding process, to quote the PHP Manual: "Returns a string in which all non-alphanumeric characters except -_. have been replaced with a percent (%) sign followed by two hex digits. This is the encoding described in RFC 1738 for protecting literal characters from being interpreted as special URL delimiters, and for protecting URL's from being mangled by transmission media with character conversions..." Thus, the subdirectory-name and filespec examples above would be returned as: fran%87ais The individual pieces are then put back together with appropriate separator slashes re-inserted, and the whole appended to the domain URL. For the given example, we would end up with: http://wwww.mywonderfulsite.com/fran%87ais/pur%82/good%26bad/some%20good%20%26%20some%20bad.html Also: the encoding listed in the sitemap is "UTF-8"; the sitemap specifications like the sitemap file in "UTF-8" encoding, but also carefully insist that the actual filespecs be encoded so that your server will read them correctly. Presumably the form in which your server's filesystem reported them to php is the way in which it would be able to read them, but the easy test is to copy, exactly, any questionable spec into your brower's URL bar and see if it indeed takes you to the file in question; if it does, then the encoding is correct in that respect. Note carefully that this initial encoding is probably incomplete--in the sense that if you have any php script files, or other files that take parameter strings, those of course are not provided. You must code them in by hand. For example, makestatic.php might have an entry like: http://www.mywonderfulsite.com/divisions/division-info.php That might well represent, in effect, many different pages, depending on the parameters, or "query string", following. You would have to hand-change that to something like, to make up an example: http://www.mywonderfulsite.com/divisions/division-info.php?nat=usa&div=nyc Note there the following out of the protocol's requirement that certain special characters, which include the ampersand, be translated to entities. That can be done automatically, but not when a script has no way to know what parameters you need in your URL calls. You have to do it as you enter the strings. Also take note of the "change frequency" listed for each file. The makestatic script has a primitive intelligence: it looks at how long ago the file was last modified (or made) and uses that datum to select a tentative "change frequency" for the file. For example, a file last modified 45 days ago would be assigned a change frequency of yearly, since it last changed over a month ago, so that monthly seems inappropriate. That is better than giving everything some one arbitrary frequency, but it is nonetheless crude; a file last modified two days ago might be one that typically changes only rarely; a file last changed 11 months ago might be one that will soon start changing daily. Only you know; only you can check that those assignments make sense, and change them by hand where they don't. Again: I provide BookAdder, and I provide assured mapping of the files that BookAdder makes. The staticmap-file script is a bonus, and I expressly do not guarantee it. As I say, I think it works OK, but it is your responsibility to check every file listing in the static map. Even if you need to hand-modify a lot, that's still easier than building the whole thing from scratch by hand. Actually Registering Your SitemapsOf course, if you are already a registered sitemap user at all the search engines, all you may have to do--depending on your case, as described above--is point the engines at the new Bookshop_index.xml master index file. Otherwise, you need to do the initial registration first, then specify that file (as well as any other non-BookAdder ones you may now have). Here's where to go to set up registration:
It looks as if all three (and perhaps others) will accept HTTP notification (commonly if incorrectly called "Pings"), and it may not be necessary to register with them (with MSN you just cannot), but it at least makes available diagnostic information on your site from that search engine. You should, of course, wait till you are satisfied that your first real search run has concluded, and satisfactorily, before registering your new sitemap-index file. (And remember the earlier tip: you don't need to re-run the entire search just to make sitemaps of its results--simply call the script makemaps.php, which will map whatever the current search results are.) Examining Your ResultsWhen sufficient time has passed since you started the search run--give it at least an hour--load status.php (the Status Report script) and see if it's done yet. If the job is done, the Status Report page should look something like this (this sample is slightly shrunken, to fit here)--
--with the various processes (monitor.php, findbooks.php, makemaps.php) all shown as not currently running. (See the docfile Installing BookAdder for more explanation of the Status Report page.) If the job is not complete, review the whole Status page to see if it at least seems to be progressing: several divisions ought to be complete, as you can tell from the date and time shown for their results. It may be that your host server has poor connections, or Amazon may just be having one of its slow days. Don't give it up as lost till 1½ or even 2 full hours have gone by without your getting to that "finished" line in the logfile. (If you have a problem here, and can't figure it out on your own, don't hesitate to e-mail me and we'll work it out for you.) When the run is fully completed, return to your bookshop front-page script (whatever you named it in customize.php at Step 3) and make sure you're happy with everything. I do hope you are. Step 10: Open Up Your New ShopStep 10a: Provide Site LinksIf the shop indeed looks ready for business, you need to provide links to its front page. And be absolutely, positively sure to remember that you need to provide six separate links, one for each divisionally based shop. The actual links will be of this form, with--of course--your own correct data plugged in: US: http://www.friends-of-red-cats.com/red-cats-books/red-cats-books.php?in=us You can put those links anywhere on your current site, but why not put them somewhere on your overall front page? And if you use a templated include/dropin site directory on your pages, be sure all six are there, too. This isn't just added pages, this is money. Meanwhile, let me repeat this: the bookshops for the divisions where English is not the native tongue will list only books in English available for sale through those divisions. If your visitor is a servicewoman stationed in Okinawa, she can order English-language books from nearby Amazon Japan, with your site--and the books you offer her--all in English. (If you want or need to have a bookshop that searches for books in a division's native tongue, e-mail me about it and I'll work it out for you.) Step 10b: Set Up Auto-UpdatingIt is very important that you set your new bookshop up so that it will be updated on average once every 24 hours. (For one thing, Amazon's Terms of Service rightly don't allow displaying prices over 24 hours old). Besides, you want your wonderful new stock of pages to change as often as possible for SEO purposes as well. Fortunately, setting this up is very simple to do, and once it is done, the whole thing takes place, day after day, invisible to you with no further effort on your part. At bottom, you rely on whatever task scheduler your host makes available to you. I know nothing about Windows-based servers, but all Unix-based servers include the well-known facility called cron; if you are on a Windows-based server, you'll have to ask your host to help you out here--but what you want to accomplish is very simple. Access to and use of a cron scheduler varies somewhat from host to host. Here is some general guidance, but your host is the final source of valid information on using cron on your server. Your first task will be to find where the cron interface is. Go to whatever sort of "control center" your host provides for your account and poke about--you're looking for references to either "cron" or a "scheduler" (some hosts hide this under something called "Advanced Options" or the like). Regardless of the exact appearance of the cron interface on your host, it should be simple and intuitive to use; what you basically need to tell it is how often to run the task (in this case, daily), when to run it (you can pick a particular time, or you can be generous and let cron pick a time based on when the server's overall load is low), and what command (with what command parameters, if any) it is to run. If you pick the run time, try to select a time block, figuring about an hour for the total run, when your site is likely to be least busy. As to the command to run, while there are two ways to go about this, both using as the ultimate target command the BookAdder script cronf.php, one is much preferred. That is, you could try calling cronf.php as a local filesystem file; but calling a php script as a local-filesystem command can get hairy (for example, if you are cgi-wrapped, you probably need to instead call the wrapper with the command as a parameter). The easier way by far is to accesses cronf.php through the internet by HTTP. That is the only method I will discuss here. This method assumes that your host makes the wonderful WGET utility available--all good hosts do--so you can use HTTP to start the file. What you need to know to do this is the full path on your host's server to the WGET utility; you'll have to ask about that, but typically the full command you specify to cron might look something like this: /usr/local/bin/wget -q -O - http://friends-of-red-cats.com/red-cats-books/cronf.php > /dev/null There, /usr/local/bin/ is the path, on your host server, to the directory in which the WGET utility is; -q and -O are flags being passed to WGET (don't worry what they do); and > /dev/null tells WGET to just dump its output--any text it reads from the called script--into "the bit bucket" (oblivion, the null device). Again: the exact form you will need you must get from your ISP, but it should look something like that (excepting, of course, for the substitution of your true site URL and the path to your actual bookshop directory. (The CURL utility works in much the same way as WGET, and could be used instead if you know its parameter syntax.) So, again, in summary:
It's easier to do than to tell about. You should check your logfile, and the shop itself, the first two or three days to make sure everything looks right; after that, try to remember to check every month or so to make sure nothing has gone astray for any weird reason. Und dot's dot. But . . . there are some other important things you should also customize, or at least look at. The next docfile along will explain them all to you. Moving OnBookAdder Documentation Files AvailableThey are:
What to Read NextYou are now ready to further personalize BookAdder (which includes setting it up for Google AdSense ads if you use them). Click on to proceed to the Further Customizing BookAdder docfile. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||