Screenscraping
I recently had to look into ‘screenscraping’ information from one website to another. In this
Of course should be very straightforward you would think? There are loads of products out there that say eBay to Opencart. There are some issues:
- Opencart v3 is not supported very well…
- eBay structures change
- eBay layout out information differently on a regular basis
- Integration tools do not map the information correctly (this is a biggy!)
- eBay to other shopping cart software solutions are VERY slow and most (if not all …) do not support HTML descriptions that are now available in Opencart v3 (more of this later!)
- eBay’s long description is actually added to the page via an iFrame and not that easy to get at – also it allows the user to add all sorts of HTML that would not look consistent on a shopping site.
- Plus a lot more!
So having gone through various ‘trials’ of integration tools, including:
- Expandly (not very good support for OC v3) – does not map across HTML at all, misses MPU and other stock numbers or at least miss manages the field mapping – plus too many other things. Literally spent days on this one!
- Linnworks
- About 5 others…
Eventually I came up with the following list of tools.
We will go into a little detail with each one
SixBit management tool to integrate eBay with Other tools
Ok, for what we wanted, this software is superb! Yes, you have to pay a monthly fee (which is very reasonable BTW). BUT, you get a full client management tool that offers the following:
- Integration with Amazon, eBay and ETSY
- Excel and CSV import and export (this aspect is very very good!)
- Pick list management
- Packing list management
- Shipping management
- eMail management
- Full reporting
- Dropshipper management
- … and a lot more
So, don’t WASTE your time like WE did with trying to scrape products and other information from eBay and the like. Use THIS TOOL to get the information, management your products and if you want to export so that you can import into OpenCart or any other shopping platform.
BRILLIANT… if you want to get products in and out of other systems from eBay and Amazon you do not need to read on – get this!!
SixBit software just does the job thanks!
Scraper (chrome plugin)
Chropath
(chrome plugin)
Web Scraper by Martin Balodis (chrome plugin)
Advanced Scraper by Agenty ($Commercial and trial version available)
Data Miner ($Commercial and free version **)
** At the moment the free plan gives you 500 pages/month. The count resets monthly if you don’t exceed the 500 page limit in any given month. If you do exceed the 500 page scrapes in a given month your account will be automatically locked indefinitely. Then you can unlock your account by upgrading to any of our paid plans.
Other products to consider would be:
- Greasemonkey scripts
- Excel spreadsheet formulas
The process
I decided the best approach was to work out either the ‘xpath’ or ‘css’ structure of the eBay listing page. This was actually quite easy using the free chrome plugins.
