User Stories / Scope
The main purpose of this extension is to generate the Catalog Products Feed in a CSV format (or XML nice to have) using Google Shopping specifications for the output and following google’s policies. Additional feeds like Promotions Feed or Local Inventory Feed for Goole Merchant Center can also be generated.
The second objective is to have the flexibility to generate feeds outside Google’s specifications as there are many other integrations which require similar kind of product data to be feed.
List & manage the feeds
As an admin I need to view a list of feeds so I can manage them, view their status and access the feed file contents.
user-guide: Feeds management
The list of feeds should display:
checkbox used for mass actions
ID - unique identifier
Store View - scope in relation to the store’s catalog
Name - an arbitrary name for easier identification.
Use for microdata - a flag, maybe listed under the feed’s name to show which feed configuration is used by store view when rendering related microdata.
Feed Type - one of (generic, google shopping, google local inventory) it is used later for templating the feed’s configuration and access control in relation to mapping and processing functionalities
Run schedule - a comma separated list of times when feed will be generated
Feed file - URL / link to the file where it can be downloaded or see the content
Status- one of (Disabled, Scheduled, Completed, % of the processing wile generating)
Actions - buttons main individual operations:
run now - will queue the feed to be generated with the next cron run
configure - will edit the feed configuration
test feed - will open the testing form where individual products can be sampled to see output
view log - lists the log content of the feed processing
Mass Actions:
Enable / Disable - when feeds are disabled, the processing queue will ignore any of their processing schedule, so they will not generate files or update the file
Clone - Will duplicate the feed configuration as a new feed, creating a new ID (and a new filename), the name of the new feed is sufixed with (clone) which can later be edited.
Delete - Removes the feeds along with their configurations, feed files and logs, but also cleans up the processing cache/tables and the queue for any related schedule.
Export/Import - exports the feed configurations in a json format which can be then imported into a new store. Useful for migration from dev/staging to production and so on. Import would recreate the feeds using the configuration json exactly the way it was when exporting, but less the feed file content and the processing logs.
Create / Edit feeds
As an admin I need to create a feed and configure the output based by the feed type and available catalog information.
user-guide: Adding a feed Optimizing feed output
When creating a new feed, the admin chooses the type of feed, and with minimum input (ex: filling the feed’s name) it should be able to have a feed ready for processing. There is a set of configuration defaults which are used for each feed type, to fill in the configuration forms.
To edit a feed configuration the admin will use the configuration button from feeds list. It uses the same configuration form as for feed creation, but having the saved configuration filled in the forms.
Saving the feed, it checks for other feeds of the same store view, and disables the “use for microdata“ if it is enabled in the current feed configuration. Only one feed per store view can have this flag.
Also saving the feed during feed generation does not update the running processing configuration even in batch mode, and saving feed status from processing cronjobs should not revert configuration updates made during processing.
Supported Feed Types
Releasing new feed types should be as much as possible plug-and-play, specifications can be defined via code configuration files. The configuration files should define:
default columns map and feed configuration as a template for new feeds.
access control to Directives and their parameters, along with feed specific implementation of those directives.
Generic Feed
It holds all the basic functionalities and configurations to define a feed.
General Configuration
user-guide: General Configuration
input name | type | description |
|---|---|---|
Name | text | arbitrary feed name to identify it easier |
Store View | dropdown | select the the context of which catalog data is been processed. |
Feed Currency | dropdown | conditioned for allowed currencies of the store view. Product prices will be computed using the specified currency, regardless of the currency displayed on product pages of the store. |
Feed Path | text combo | where the feed file should be saved on the server. Specified the file name which is prefilled with a naming convention feed_<id>.txt |
Delimiter | dropdown | tab, comma, other. if other delimiter is selected a new input should allow to specify a custom delimiter. |
Apply Catalog Price Rules | yes/no | will use promotion rules on top of sale price definitions |
Use default Stock Statuses | yes/no | set to yes will compute avalability using the store’s core functions, but when set to no, a new filed will allow to select a product attribute to be used for pulling availability data. Availability must be in a list of accepted values: (in_stock, out_of_stock, preorder) and it is important information used across catalog processing including product filtering, this is why it needs to be specified outside the Columns Mapping. The Availability directive will use this information for output as well. |
Use QTY Increments | yes/no | Similar to availability, this needs to be specified outside the columns map as it is used along with availability information and qty computation along feed processing. Also used in “Inventory Count” directive. |
Use stock reservations | yes/no | similar to availability, stock reservations are either considered when computing availability or ignored. |
Columns Map
user-guide: Columns Map
Defines the map of feed properties / columns onto product data.
The configuration widget/form should:
be filled in with a predefined set of columns and mappings based on feed type when creating a new feed.
allow add or remove columns
specify columns order in the feed
edit column names and their mapping attribute or directive
change the mapping arguments by column or directive
The basic mapping is a 1 to 1 relation to a product attribute, but certain columns like link, price, availability, etc, need specific functionality which we call them “Directives“ and they can accept arguments. For instance the Price directive can be configured to compute the values including tax or excluding them.
As nice to have is for the feed allow output of XML instead of CSV in which this mapping should also allow configuration of the xml template to be used with the properties (columns).
Available Directives for mapping:
directive name | arguments | description |
|---|---|---|
Static Value | value - text | Outputs the specified in the arguments |
Product Id | Add source code - yes/no | For multi-store ids must be unique across all feeds. Set argument to 'Yes' will concatenate the product id and the store code. |
Product URL | Add URL Suffix - text | Common usage is to send extra GET parameters for analytics tracking. |
Product Image URL | Type - dropdown (base, thumbnail, small) | Which product image URL should be listed. |
Product Additional Images URLs | Exclude Type - dropdown (base, thumbnail, small) | Output is a comma separated list of image urls, excluding the one selected here. |
Product Category Image URL | First category found with an image is been used. | |
Price | Add tax - yes/no | Aims to match the price displayed on the product page, so it takes into account currency conversions, tax, promotions, MSRP and eco tax. Configurable, bundle and grouped items will have the minimal price of an item. |
Tire Price | Customer Group - dropdown | Outputs lowest tier price for selected customer group. |
Sale Price | Add tax - yes/no | Output the lowest value for which the product can be purchased at. It computes the Special Price of the product, and any catalog promotion rules when enabled. |
Sale Price Date Range | Computes the start and end dates of special price. If end date is not specified, defaults to one year. | |
Availability | Computes the product availability | |
Inventory Count | Count Mode - (Item's qty, sum of associated items qty) | Pulls the stock inventory qty information. The Count Mode defines how the qty count should be performed in case of complex products. For stores with MSI enabled it would sum QTYs from all inventory sources. |
Variant Attributes | configurable attributes dropdown | Select all possible attributes building this value. |
Product Review Average | Output product's average rating in the interval 1 - 5. Returns 0 if no reviews have been approved on your product | |
Product Review Count | Outputs the count of product reviews. Returns 0 if no reviews have been approved on your product. | |
Concatenate Attributes | Value - text template | Concatenate text and product data using placeholder in this format: {{attribute_code}} or {{column_name}}. The {{variants}} placeholder is designed to pull values of all configurable attributes for an associated product in the format: "Option 1, Option 2, Option 3 ...". The separator can be changed under ‘Configurable Products > Associated Product Attribute Value Separator'. Any columns selected in 'Value inheritance by column' from Configurable and Grouped section will inherit values from parent. Placeholder which don’t pull values can be removed using 'Product Filters > Find and replace' |
Is Bundle | Outputs TRUE for bundle items and FALSE for the rest. | |
Product Option | Options - dropdown | Product options will be considered as a variant in the feed, creating a product row for each of the option’s value assigned to the product |
Shipping | Shipping Methods - dropdown Countries - dropdown Only min price- yes/no Only Free Shipping - yes/no Add Tax - yes/no | Shipping rates in google’s format. |
Shipping Weight | Unit of measure - dropdown (gram, kilogram, pound, ounce) | Outputs the product shipping_weight attribute concatenated with the unit defined as argument here |
Type by Magento Category Path | Paths limit - number (default = 3) | Uses product categories to create a comma separated list of category paths. Categories are matched using the priority values set under Category Map. Limit paths, defines how many comma separated values to output. |
Type by Magento Category | The value is been matched against the type value definition according to the category assigned to products and using the Priority set at each category under Categories Map. | |
Product Expiration in Feed | No. of Days - number | Computes an expiration date based on product creation and the "No. of Days" option provided. |
Categories Map
user-guide: Categories Map
It defines a mapping of catalog categories onto a set of taxonomies like ones used by Google or other integrating service. Each category can also be assigned with an extra free-text field call type which is used by “Type by Magento Category“ directive.
Because a product can belong to multiple categories a refinement system like priority numbers is needed to pick the most relevant one when computing taxonomy or type of the product. The priority (numbers) may be used in two modes:
across category tree
between categories at the same level in the tree
If no priorities are defined, the system should prioritize a more granular matching of the categories, meaning deeper leaf in the category tree should take priority when matching. To differentiate between categories at the same level, the priorities can be set accordingly and the matching should be set to mode2.
The same category mapping widget is also used as a products filter to include or exclude products by category, this using a checkbox for each category in the tree. It is an important filter most admins look for, but at the same time, it is ineffective to configure.
The challenges around it:
Products belong to multiple categories.
If a product is assigned to two different categories, and one of them is set to not include products, would you include the products in the feed or not? Priority numbers may help decide this case, but what if too many categories are used.
Categories can be active or inactive in different store views, so they may not be shown in the widget depending on the store view selected for the feed.
the system will have to intersect the active categories in store view with active products in the same store.
New categories will be left unconfigured.
either through priorities or tree hierarchy, new categories should inherit parent filters and setup.
Some product may not be assigned to categories.
A setting to exclude products with no categories is needed.
Catalogs with large number of categories
I have seen categories in the numbers of thousands which tend to break the Category Map form and the effort to configure it is significant. The current solution is to limit the depth of category tree shown in the widget, but this means some of the catalog will be left uncovered.
I propose a new kind of Category Map widget for both taxonomies/type mapping and the filte, to have a more flexible configuration, based on adding rules (category ID - filter, category - feed property) with inheritance and priority flags.
Complex Products
user-guide: Configurable Products Grouped Products Bundle Products Columns Map
Complex products is an important topic the feed should deal with, and it brings the appropriate complexity with it. There seems to be a need for dedicated configuration for each complex product type to decide onto the following:
Output one or multiple rows per product?
In general configurable products would be set to output all variants, as for grouped products, only parent item may be listed, but price and availability needs to consider it’s sub-items. So there’s a need to configure how each complex product woudl output in the feed:
no parent / only associated products
only parent/ no associated products
both, parent and it’s associated products (unlikely needed, but good for testing)
In case of multiple rows, each will have to vary the attributes (color, size, etc). The URL in general pints to the parent product page, where the appropriate options need to be auto-selected (color, size, etc.) This can be done by appending the product URL for each variant all the variations selected ex: http://mystore.com/parent-product.html#color=12,size=53?aid=234 The aid parameter is needed later for rendering the appropriate microdata in the product page, see automatic item updates topic.
in case of one row, variable attributes will output in a comma separated fashion, but the delimiter needs to be configurable, or template like defined.
Since availability is an important property, and it’s function changed from one version to another of magento, there was a need to configure whether associated product will inherit out of stock status from the parent, defaults to yes.
Value inheritance
An importnat rule-based configuration is to set value inheritance for a limited list of columns, to define where the values should be pulled:
associated if exists / otherwise from patent
associated only
parent if exists / otherwise from associated
parent only
Usually associated products which are not visible on their own will lack importnat data like description, proper name, images, etc. Value inheritance should come handy in those cases, but it should exclude key columns like: id, product_url, availability, pice, sale_price for obvious reasons.
One common use case is the need to pull product name from parent product for configurable products, but at the same time because their names are not SEO optimized, we need to use the Concatenate attributes to glue together something like: (parent product name and brand + associated color and size). Value inheritance rules should also define by each column-rule which attributes should consider when Concatenation directive is used for that column.
Grouped products will have to compute the price considering all sub-items, there needs to be setting for computation type:
minimal price (between sub-items)
sum of all sub-item prices. Default quantity of each associated product multiplied with the price of the associated product and then summed together. If no default quantity is defined, it will output minimal price.
For bundle products the computation of product weight can be defined as Dynamic or Fixed, which is used further in the feed output and shipping rate calculations. There was the need to overwrite this on some stores to either combine weights for parent output or not.
Custom Options
user-guide: Custom Options
Product custom options may have similar output effect in the feed to a complex product like configurable, basically a product varies by one or more attributes (options in this case) creating multiple feed rows having the variant column (size, color, etc) different with each row. Or may output one row having all the variant values comma separated or in a specific order or template.
The feed configure which column maps to specific custom option values using the Product Option directive in the columns map. And a feed level setting should define if the output will be multiple or single row in the feed, and for which categories would it apply.
Shipping
use-guide Shipping
Admins need to specify which shipping methods to consider in the feed. Because most shipping methods require both origin and destination for computing the correct rates, there needs to be a way to specify them in order to support those methods.
Origin is mostly configured by the store, and destination(s) should be configured in the feed as a selection of countries-zipcodes. The more refined destination is define the more values at output and more computation, so consider a system to minimize this process and output.
Current system is not doing great, and most of the times we end up recommending the rates setup in the Merchnat Account. That usually means the feed should include columns for shipping dimensions like: shipping_height, shipping_length, shipping_width beised shipping_weight, those are needed to allow consuming system (Google) to calculate carrier rates.
Computation of shipping rates is usually slow and resource consuming, and since it is not changing that often, a caching system needs to be considered.
The format of shipping output seems to be standard for most integration services, following the minimum google’s format, ex: US:900*-901*:15.99 USD
The whole shipping setup is done in a dedicated section of the feed configuration and than mapped in the feed using Shipping directive, by default there should be no output / no methods selected to be used.
Google Shopping Feed
It has all the Generic Feed functionalities and configuration plus some Google specific features:
General Configuration
input name | type | description |
|---|---|---|
| yes/no | When set to yes, store's microdata uses mapping of this feed. Only one feed per store should have this turned on, which is restricted at the save action. |
Columns Map
directive name | arguments | description |
|---|---|---|
Item Group ID | Base on parent attribute - dropdown (id, sku) | Will output the same value for all associated items, and empty for the complex product. The value is computed using the parent product attribute specified, and sufix of variant attributes not mapped in the columns. |
Adwords Price Buckets | widget to defined price intervals of prices and correspondent labels | Outputs the configured label if price is in the correspondent interval |
Taxonomy By Magento Category | Uses a set of rules (category - priority) defined under the Categories Map to output a matching taxonomy. | |
Identifier Exists | Required identification columns - csv text, defaults to "gtin,mpn,brand". | Returns FALSE if brand or any of other two (mpn, gtin) is missing, otherwise retunrs nothing. When customized, output is FALSE if any of the spcified columns is empty. |
Identifier Attribute | attribute - dropdown | Select an attribute like UPC, EAN or ISBN. When the attribute's value is missing for a configurable or grouped item, will fetch the value from lowest priced associated item, and if empty for a bundle, will go for the higherst priced sub-item. |
Promotions ID | This will output the generated ID of promotions feed, if configured such feed is configured. |
Default mapping:
column | map to (Directive / attribute) | options |
|---|---|---|
id | Product Id | Add store code = No |
item_group_id | Item Group Id | Base on parent attribute = id |
title | name | |
description | description | |
link | Product URL | suffix = ?utm_source=google_shopping |
image_link | Product Image URL | type = base |
additional_image_link | Product Additional Image URLs | exclude type = base |
availability | Availability | |
price | Price | add tax = yes |
sale_price | Sale Price | add tax = yes |
sale_price_effective_date | Sale price Date Range | |
brand | manufacturer | |
mpn | sku | |
condition | Static Value | value = new |
product_type | Type by Magento Category | paths limit = 3 |
google_product_category | Taxonomy by Magento Category | |
identifier_exists | Identifier Exists | required columns = “gtin,mpn,brand“ |
is_bundle | Is Bundle | |
shipping | Shipping | |
shipping_weight | Shipping Weight | unit = gram |
promotions_id | Promotions ID |
Categories Map
Taxonomies field includes an auto-suggest function which would present options of Google’s predefined taxonomies as you start typing.
Google Local Inventory
user guide: Set up Google Local Inventory
Local inventory feeds are designed to work with Multi-Source-Inventory system of the store, so that the products availability and qty is listed by location (source).
This means the feed would have multiple rows for each product to list availability and qty for each inventory source.
Default columns map:
column name | map to (Directive) | |
|---|---|---|
source_code | Inventory Source | |
id | Product Id | Add Store Code = No |
availability | Availability | |
price | Price | Add tax = Yes |
sale_price | Sale Price | Add tax = Yes |
sale_price_effective_date | Sale Price Date Range | |
quantity | Inventory Count | count mode: item’s qty |
Most feed configurations from Generic feeds type also apply here, but since the the list of columns needed for this feed is limited, also some of the configurations and directives are not available. For example there are no categories or taxonomies in this feed, so the related directives are not available and Category Map works strictly for filtering purpose. Same goes for Product Options, Shipping setup or anything around product Urls.
Additional directive functionality:
directive name | parameters | decription |
|---|---|---|
Inventory Source | Outputs the inventory source code if MSI module is active. | |
Inventory Count | count mode = (item’s qty / Sum of associated items qty) | This directive is also available in other feed types but for this feed it works a bit differently. Instead of listing the sum of QTY's across inventory sources, it will list the qty of individual source. If MSI is not available it will fallback to regular inventory qty information. |
Google Promotions
Filtering Products
As an admin I need to filter the products added to the feed so that only products with a matching criteria would be added. Besides data maping in the feed, this is the next major feature of the extension, most advertisers need to track or run campaigns on a specific set of products.
Dealing with poor catalog data
As an admin I need to configure the feed in such way so that I am able to fill in required and most important Google feed columns, making the most out of the catalog data, ideally without having to fill in new data on each product.
Service integration (GMC)
As an advertiser or admin I need to upload the feed file in a service like Google Merchant Center, Bing, Facebook, etc.
user-guide: Set up Google Shopping
The feed can be consumed in two ways: fetched by URL or uploaded via SFTP/FTP. The feed configuration has a section “Uploads“ where one or multiple upload accounts can be configured.
After a successful feed generation, after the physical feed file is been replaced with the temporary file where lines are written during processing, the cron process will attempt to upload this file to all the FTP accounts configured for the feed.
The log should mention each account where file was uploaded and if it was successful or not.
When configuring Google Merchnat Center, admin’s may generate the FTP account need for uploads which can then be configured in the store’s feed configuration. In GMC the feed needs to be defined and specify the mode it will be ingested. by scheduled fetches where the feed URL is provided, or by FTP uploads where the filename is provided, the filename is listed and configured in the store’s feed management.
Feed Generation
Dedicated cron-jobs
user-guide: Detaching feed generation from magento cron
Testing Output
As an admin I need to see the output of the feed by individual product so I can asses feed configuration updates or catalog data quality, without having to wait for the large feed file to be generated and uploaded.
user-guide:Testing and Troubleshooting
Troubleshooting
Review file generation log
No comments to display
No comments to display