Google Shopping Feed

Table of Contents

  1. Description
  2. Features
  3. Installation
  4. Quick Start Videos
  5. Configuration
  6. Shell
  7. Best Practices
  8. Troubleshooting

Description

Version 1.5.20

Extension page: http://www.rocketweb.com/google-base-feed-generator

Supply Google with a properly formatted product text file that will be fetched on a schedule.

Features

Installation

Backup your data

Backup the database and your store's web directory.

Disable Compilation

This step is for Magento 1.4+ versions. If you are running an older version, this step can be skipped. In the Magento admin panel, go to System->Tools->Compilation. In case "Compiler Status" is "Enabled", click on the "Disable" button (in case the status is disabled you can skip this step):

This screen should now look like this:

Download and Extract

Download and extract (unzip) the extension's contents on you computer. Navigate inside the extracted folder.

Upload files

Using a FTP client, upload the content of the extension directory to the store's document root, so that the app directory in the extension folder merges with the app directory on the server. If asked to replace any files, select "Yes".

Clear the cache

In the Magento admin panel, go to System->Cache Management and press the "Flush Magento Cache" button:

Quick Start Videos

Checkout the videos on the Google Shopping Feed Extension page. In these videos we take you through different, basic aspects of a configuration. After watching the videos, explore the documentation below for all the different possibilities for customizing your feed. These are best watched in full screen mode.

Configuration

After the extension is installed, you can visit System->Configuration->Google Shopping Feed Generator in the Magento admin panel to see the main configuration options. This extension has many features but also can be quite complex. We highly recommend configuring a few settings at a time and generating a test feed as you are configuring to see how your product data is being generated. We have done our best to make sure this extension will have an answer for almost any Magento catalog. Often there are several different ways to produce a valid feed. The best solution for your catalog may require a little experimentation and patience.

Configuration / General Settings

Enabled:

Admin option to enable or disable the generator. If you would like to disable this extension on a specific store then switch to store view and save this option as "No".

Feed Localization:

Changing the language of your feed affects how Apparel products are matched using Google taxonomies. Your products should also be in the same language. Refer to Google Category of the Item attribute. Note, this setting does not affect price formatting. Price formatting depends on what is set for your store under System->Configuration->Locale Options->Locale. Also be sure to review the setting "Add Tax to Price" to specify if you want tax included in the display of all your feed prices.

Feed path:

"Feed path" is relative to the web root like: my_dir Allows you to to set an alternate save path. By default the feeds are stored in this directory: [path to magento]/media/feeds/ If you’d like to use the default path than leave this option blank. Make sure the feed path has enough write permissions for both web server’s user and cron user if they are different. Ask your web hosting company for help if needed. Feed files can be created for every store. Files will have following file name format: google_base_[store code].txt Magento comes by default in single store mode with the store code default. If unchanged, this feed file will be in: [path to magento]/media/feeds/google_base_default.txt

Generate Feed Now button:

This will generate the feed on demand. This button is meant to help you quickly test your configuration. It’s not recommended to use this button on catalogs with more than 1000 products. If Magento cron is setup for your system, your feed will be automatically generated each day at 1am server time. This button has a limit of 1000 products. If you know your server can handle more products at one time you can change the limit. Edit app/code/community/RocketWeb/GoogleBaseFeedGenerator/config.xml, tag <button_max_products>1000</button_max_products>

Test Feed Now button:

We highly recommend using this button to see if data fetched from a selection of products is well formed. There are 3 additional fields which specify which product(s) to test. You can load a specific sku, or a range of products with the offset and limit. To view the first product you entered in your Magento catalog set the offset to 0. The offset can be from 0 to the total number of products – 1. Limit can be maximum of 100 and will instruct the test to return that number of products. Using this button won’t write into the main file. It will write in an alternate file with the format: test_google_base_[store code].txt in the feed save path.

Include products only from these categories

This feature enables you to select which categories will be included in the feed. By default, products from all categories are included. If you want to only include products from some categories, please un-check "Include all categories" and select the categories you wish to include by checking the boxes.

Anchors

Note that this feature also respects anchor categories.
In Magento, Anchor categories work like this: if you set a category as Is Anchor, then all products from its children categories will also appear on the parent category page. This is useful in your website: for example to create a parent category page to include all Apparel products - if the visitors want to filter results, they can use the layered navigation to go to Men or Women.

The benefit of using anchors is that we'll include all products of all sub-categories automatically.
This means that:

e.g.: in the screenshot above, we only want to include Furniture and Apparel. Since Furniture is not an anchor, we select its children too. Later, if I want a Bathroom subcategory, I'll have to come back here and include it in the feed. On the other hand, Apparel is an anchor, so there's no need to select its children. What's more, if I later want to add Dresses to Apparel, I don't need to come back here.
Note: to set a category as Anchor, edit the category in Catalog > Categories > Manage Categories (Display Settings tab)

Submit only products of these types:

You can specify by store what types of products should be included in the feed. Adding configurable or bundle types will actually pull also associated simple products if it's specified in the 'Configurable products - non apparel' or 'Apparel' sections. Custom product types like AheadWorks subscription types will do fine.

Submit only products that have these attribute sets:

You can select which attribute sets to include in the feed. If you select only "All attribute sets", then the feed will contain all products. If you select other attribute sets, then the feed will only include products that have the selected attribute sets. Please note that if you add a new attribute set, you'll have to come back here and select it in the list.

Add Out of Stock Products:

This rule is applied to standalone simple products or configurable products. Associated simple products have other rules defined in "Configurable products - non apparel" and "Grouped products - non apparel"

Quantity mode:

This setting is been used only in the "Inventory Count" directive to determine how qty is been computed for grouped products or configurable.

Use Batch Segmentation:

This feature is meant to help with large catalogs or to spread out the server load over time. A concern may be memory consumption when loading large collection of products. The script is optimized to load the minimum data in the collection of products by Magento's native model, but memory leaks will keep consuming more memory as the script runs longer.

Force Logging:

If Magento logging is disabled and this option is Yes it will log anyway. The logs are located in var/log/google_base_[store code].log

Auto Skip Products:

By default this feature is set to "Yes", meaning products that do not meet Google feed data requirements will be skipped. Setting it to "No" will force products to be added to the feed, but will still skip products matching the "Skip by Category" setting or if the product is specifically set to be skipped in the product attributes. Setting this to "No" and opening the feed output in a program like Microsoft Excel or Google Docs can help you identify missing info in your product data.

Log skip products messages:

Will add various messages with brief explanation why a product wasn’t added to the feed. These messages will be located in var/log/google_base_[store code].log

Find And Replace:

This feature allows definition of a rule base matrix that would apply at the output of columns after all mapping has been done. This feature should not be abused for large catalogs.

Configurable and Grouped Products / Non-Apparel

This section applies to configurable and grouped products that are not matched as apparel. Note: apparel matching is done based on the taxonomy used in the 'google_product_category' column.

How to add associated products:

Defines the way the product entity is been presented in the feed, as one or multiple lines.

Add Out of Stock Products - Associated:

Specify whether the stock is being managed per associated product or not.

Price Type:

This setting only applies for grouped products and defines the strategy of how the price is being computed for the parent product.

Associated products - Fetch description from:

Defines the criteria for how to populate description filled. Some stores will not have a description defined for the associated simple products. Others may have the most meaningful description set in the parent product. Pick the option that provides the most useful information. This option does not apply to apparel products with variants.

Associated products - Fetch link from:

Defines the criteria for how to populate URL filled. The link of the associated product item can be fetched from the parent configurable so all associated products line items in the feed will point to configurable product page. This option coupled with Associated products Form unique URLs for not visible associated products will generate unique URLs like: http://example.com/configuable.html?color=123&size=99 Second option 'Associated if is visible in catalog, otherwise from configurable' will try to grab a url from the visible associated product first. Option does not apply to apparel products with variants.

Associated products - Form unique urls for not visible associated products:

When this option is turned ON, The URL in the feed is targeting the specific product configuration that it refers to, using sub-items information like: product id, color, size, etc. EX: http://example.com/configuable.html?color=123&size=99.

Associated products - Fetch main image from:

Defines the criteria on how to populate the image URL. You can grab the image link from configurable product, associated product or either one. There is no support for additional images of associated products. Option does not apply to apparel products with variants.

Configuration / Feed's Map

Your Google Shopping Product Feed depends on your store configuration, types of products and available data stored in your products. Please carefully read Google Products Feed Specification https://support.google.com/merchants/answer/188494?hl=en to configure this extension properly.

Feed Columns Map allows you to associate any column of the feed with product attributes or hard-coded directives. Generally it is recommended to associate directives (the first grouping in the dropdowns) to corresponding columns. Column names must be exactly as Google specifies. English column names are acceptable for any locale but you may also specify local column names that Google approves. By default the extension comes configured with a minimal set of column associations. Most countries now require a valid mpn or gtin for products along with the brand. You should at least add one of these 2 columns before submitting your feed. To configure apparel products you should add the additional columns: color, age_group, gender, size, item_group_id and optionally material and/or pattern. If Static Value is filled, it will be used instead of the Product Attribute unless otherwise specified in the extension config notes below. Order will sort the columns in the feed. Google Shopping does not need Feed Columns to be in any order, use this mostly for your own readability as you review the feed output.

Feed's Columns:

Mapping Directives

Other Mapping Settings

Here are links to text files that list the product taxonomies for supported countries and languages. Note the date at the the top of these files. Google does modify their product taxonomies regularly. It is a good idea to review this a few times per year to see if you can better classify your items.

In case of character encoding problems, use one of these Microsoft Excel files instead. Be sure to put > between any categories like in the text files.

Configuration / Shipping

This section configures output of the "Shipping" directive. If the shipping is disabled here, no output will be generated for the directive.
The configuration options here are meant to construct the output required by Google Shopping Specification docs. Only table rates and static shipping methods are available for use here. Setting up real time shipping methods must be done within Google Merchant Center. Simple shipping rates by price and weight can be defined in Google Merchant center as well so you may wish to check those options and skip this section. Note: because computing shipping rates for each product can bring significant server overhead, the values are being cached throughout feed generation. The time which shipping options are cached is configurable below.

To use this add the Google Shopping Feed Column "shipping" and use the Directive "Shipping" for Product Attribute under the Feed Columns configuration tab.

Configuration / Apparels

This section applies to clothing, apparel and items that fall within the broad categorization of Apparel & Accessories, e.g. Shoes, Handbags, Jewelry, Accessories, etc. Please review the requirements for apparel products: http://support.google.com/merchants/bin/answer.py?hl=en&answer=1347943. To enable apparel functionality, the Google Product Category of an item or Magento category of items should be set to Apparel & Accessories > Clothing or the equivalent for your locale.

When using Apparel, Apparel columns used must be manually added to Feed Columns Map.

Possible columns for apparel: color, size, age_group, gender, material, pattern, item_group_id, size_type, size_system. item_group_id is mandatory for apparel variants as it offers a way of grouping them back with the parent product. Its value is the configurable product id. All apparel columns should be associated with the corresponding directives. Assigning attributes instead of directives will often result in not well formed feed data. Apparel columns like age_group and gender may use a static value if all your products are apparel.

Apparel products are identified by the Google Category of the Item: Apparel & Accessories (including all subcategories). Clothing products are identified by the Google Category of the Item: Apparel & Accessories > Clothing (including all subcategories). Shoe products are identified by Google Category of the Item: Apparel & Accessories > Shoes (including all subcategories).

Variants columns (color, size, pattern, material) can be associated with configurable attributes in this extension. If a color or a size attribute assigned in this configuration section is present in the configurable product it will use that value. Otherwise it will try to fetch the value from attributes set in the Apparel non-variants section. If no value is set it will further try to fetch a value from the configurable (non variant attributes). There may be some product cases when the color is not among the configurable attributes (options of configurable product).

Pattern and Material attribute values will be placed only if there are at least 2 unique values for variants with same group_item_id (if variants belong to same configurable). If there are no variations, an empty value will be placed instead.

NOTE: Apparels are recommended to be used with Use Batch Segmentation: ‘Yes’ to avoid high memory usage if you have more than 2000 skus.

Shell


Important Note

Whenever the shell or cron scripts are been run, they should be run under the same user UUID which your web server (apache), this will allow proper file permissions for the feed and lock files.

su www-data -c 'php gen_gbase_feed.php --verbose --store_code [STORE_CODE]'
Where 'www-data' is the username under running the web apache service. This should be applied also to your cron job / cron.php.



The feed text file is generated on the server via cron, usually this is done daily if any product details commonly change each day.
When you log into Google Merchant Center you can add the url for your datafeed and tell Google to fetch it from your server either daily, weekly or monthly. It's common to just have Google fetch it daily after your feed is scheduled to build.
Is Magento Cron setup so that you're running cron.php or cron.sh every 5 minutes or so? If it is and you don't have batch segmentation setup, the feed is set to run each morning at 1am server time.

Sometimes Magento cron can be a bit finicky, for more control you can run the feed generation script directly on the command line to test or via cron.

cd [MAGENTO_ROOT]/shell && php gen_gbase_feed.php --help
Usage:  php gen_gbase_feed.php -- [options]
store_code <string>       Store Code (e.g. [STORE_CODE] or default). Store must exist and should be enabled. By default uses 'default' value.
batch_mode <int>          Segment the feed generation. Values accepted: 0 or 1. Default is 0.
test_mode <int>           Enable test mode or not. Default is 0.
test_sku <string>         Generate the feed only for a product sku. To be used for tests and debuging.
test_limit <int>          Sql limit parameter in test mode. Is applied to the select of the collection of products.
test_offset <int>         Sql offset parameter in test mode. Is applied to the select of the collection of products.
verbose                   Outputs skus and memory during processing
help                      This help
e.g. php gen_gbase_feed.php --store_code [STORE_CODE] --batch_mode 1 --verbose

If you are using batch segmentation it's best to run this script directly. Turn on batch segmentation and set the number of products to include at each time. If you have 15,000 products and you build the file in batches of 1,000 try running gen_gbase_feed.php on the command line once to see how long it takes to run. If it takes 2 minutes for example, then setup cron to run at least 15 times every 5 minutes for a couple hours like: 

*/5 2,3 * * * php [MAGENTO_ROOT]/shell/gen_gbase_feed.php --store_code [STORE_CODE] --batch_mode 1 > /dev/null 2>&1
Remember to set your cron to run under www-data user.

Something like the above command will run the script 24 times and should work for up to 24,000 skus. When the script runs for the 16th time in this example, it will see it has found all the products and will immediately quit. So it's OK to run the script several more times than needed each night.

Best Practices

Add attribute values to Magento products

If you look in the Feed Columns section of the Google Shopping Feed Generator you will see where you can set default, static values for attributes. This is the easiest way to set values if all or most rows in a column will be the same. Set some defaults as appropriate. Set the product_type and google_base_product_category. Go here to review Google's Product Taxonomy. If you do not have a logical, meaningful category hierarchy on your own site, just set product type to the same thing as product category.

If you scroll down in the feed settings, you will see Google Product Category by Category and Product Type by Category. Here you can easily specify new product types and Google product categories depending on your specific category.

You can also set product attribute values in bulk by going to Catalog -> Manage Products.

The Extended Product Grid with Editor is a great extension for giving you more control over what you select and edit in the Manage Products grid.

Increase the limit for Generate Feed Now button

The Generate Feed button has a limit of 5000 products set by default to protect the script from crashing if it tries to process to many products at once during your regular business hours.

To increase this limit you can edit:

[MAGENTO_ROOT]/app/code/community/RocketWeb/GoogleBaseFeedGenerator/etc/config.xml

Look for the button_max_products setting and adjust this to a limit you're comfortable with. Save the file and you can use the button for product catalogs up to the new limit.

Localization / Data Encoding

For stores that offer languages other than english or have any special characters in product names or descriptions, it is best to use a UTF-8 encoded source of text when pasting into magento products forms to avoid bad character encoding at output. Magento and this extension do not control your data encoding.

Assuming your data has been entered properly, a good practice is to also to configure Apache to append UTF-8 in response headers. It has been proved that in many cases this solution worked where special characters were previously not displaying properly in the feed file.
Add or edit the .htaccess file to you feed output folder and add the following lines to the .htaccess file:

        AddDefaultCharset UTF-8
        AddType 'text/css; charset=UTF-8' txt

Batch Mode / Generate Feeds for Multiple Stores

To generate a feed for multiple stores first get your Magento store codes in the admin.
In the admin go to System -> Manage Stores. Click on the store view name for each store and make a note of the value in the code field. If you did not change the code for your first store, this value would be "default". The store code is used when the feed generation script is called and when your feed file is built. So if your first store has the code "default" the feed file would be called google_base_default.txt. If your second store has the code of store2, the file for that store would be called google_base_store2.txt.

To generate these files, it's best to disable the Magento cron entry for building a single feed file.
Open the file: 

[MAGENTO_ROOT]/app/code/community/RocketWeb/GoogleBaseFeedGenerator/etc/config.xml
And remove this entire block of code from the opening
<crontab> to the closing </crontab> 
<crontab>
<jobs>
    <googlebasefeedgenerator_generate>
        <!-- In batch mode (use_batch_segmentation==1) must be set after midnight otherwise will not finish the queue. -->
        <!-- each 1 am -->
        <schedule><cron_expr>0 1 * * *</cron_expr></schedule>
        <run><model>googlebasefeedgenerator/observer::generateFeed</model></run>
        <!--<store>default</store>-->
    </googlebasefeedgenerator_generate>
    <!--
    Setting generator to run for other stores in a multistore configuration.
    <googlebasefeedgenerator_generate_store_2>
        <schedule><cron_expr>0 2 * * *</cron_expr></schedule>
        <run><model>googlebasefeedgenerator/observer::generateFeed</model></run>
        <store>store_code_2</store>
    </googlebasefeedgenerator_generate_store_2>-->
</jobs>
</crontab>

Now you should start using the shell script which comes with extension. See section "Shell" for more more details. 

su www-data -c 'php [MAGENTO_ROOT]/shell/gen_gbase_feed.php --store_code [STORE_CODE] --verbose'
Using verbose here will give you more detail output, so do not use it when setting up the cron.

If you have ssh access to your server, you should try running the script on the command line to make sure it completes successfully for each store.
It's a good idea to make sure you have logging turned on so you can review the log file after the script runs. Go back into the admin to System -> Configuration -> Google Base Feed Generator and in the Settings tab set Force Logging to Yes and Log skip products messages to Yes.
Now a text log file will be created at 

[MAGENTO_ROOT]/var/log/google_base_feed_[STORE_CODE].log

If you are not using ssh or your ftp client does not support viewing files, you can download this file to your computer and open it in any text editor.
This file will tell you how many products are added and how many are skipped: 

2013-06-24T08:55:04+00:00 INFO (6): (mem 42.33 Mb) products (items added, skipped)(1913,   1) to file /home/user/public_html/media/google_base_default.txt
When products are skipped, you'll see notices explaining why: 
2013-06-24T08:53:02+00:00 INFO (6): (mem 20.05 Mb) product id 111 product sku ABC123, skipped - product has price 0 = '0'.

Depending on how many products you have and how much memory you allow php to use, you may run out of memory trying to build the file.
If this happens, you'll simple need to enable batch segmentation by running the script with this option. 
su www-data -c 'php [MAGENTO_ROOT]/shell/gen_gbase_feed.php --store_code [STORE_CODE] --batch_mode 1 --verbose'
Go in System -> Configuration -> Google Base Feed Generator Settings and set the batch limit. Look back in your log file and see how many products were processed before the script ran out of memory. You may see a final line like: 
2013-06-24T08:53:51+00:00 INFO (6): (mem 129.01 Mb) (1100, 15000) products (processed, max)
This tells us that the feed generated 1100 out of a total of 15,000 products. So set your batch limit to 1000 to be safe and allow for a little more buffer.
You should also look at how long it takes to run the script for one store. If it takes 2 minutes for example, then setup cron to run at least 15 times every 5 minutes for a couple hours like: 
*/5 2,3 * * * [PATH_TO_PHP] [MAGENTO_ROOT]/shell/gen_gbase_feed.php --store_code [STORE_CODE] --batch_mode 1 > /dev/null 2>&1
Something like the above command will run the script 24 times from 2am to 3am and should work for up to 24,000 skus. When the script runs for the 16th time in this example, it will see it has found all the products and will immediately quit. So it's OK to run the script several more times than needed each night.
With multiple stores, it's common to just build one file completely first and then run the second script to build the next one. A cron job for the second store following the first one would look like: 
*/5 4,5 * * * [PATH_TO_PHP] [MAGENTO_ROOT]/shell/gen_gbase_feed.php --store_code [STORE_CODE] --batch_mode 1 > /dev/null 2>&1

Submitting a Feed to Google Merchant Center

The extension generates a text file of your product data in a format Google defines for feeds. Then you can instruct Google to grab that text file on a schedule to keep your product listings up to date.

The easiest way to submit your feed to Google Merchant Center is to make sure your feed is accessible on the web. In the extension configuration look in the settings tab and put "media" without quotes in the Feed Path field.

Now generate your feed.
This will put your feed in http://www.domain.com/media/google_base_default.txt ;... or http://www.domain.com/media/google_base_[store_code].txt if you use a store code other than "default". You should be able to visit this url for your stores hostname in your browser and see the contents of your feed.

Now login to Google Merchant Center: http://www.google.com/merchants/merchantdashboard

Google will grab the feed file from your server and then fetch it regularly according to the schedule you picked.
NOTE: Google will also accept ftp or sftp urls if you'd like to give Google your ftp password and keep the feed in a non web accessible directory.

Troubleshooting

When I try to access your extension's configuration at System->Configuration->Google Shopping Feed, I get a 404 error

Logout & Login back into the Magento admin panel.

My site is broken.

Please verify that you have turned off compilation see this link for help if you cannot log into your admin, http://kb.magenting.com/content/24/81/en/disable-magento-compiler.html. If compilation is off, make sure you have uploaded all the code. If you have uploaded all the code, try reverting from a backup and contact us for support though http://help.rocketweb.com if you do not have a backup, open your installer file and review the files included so that you can delete these files from your installation. Our extension is significantly modularized and we have never yet seen it break any site when it's fully uploaded and compilation is off.