By default both Advanced ETL Processor and Visual Importer ETL count number of records in the source table (or select statement) before importing the data. That might take a lot of time and most of the time not realy necessary. Disable "Count Number of Records" option to improve performance

CountNumberOfRecords

In the latest versions of Advanced ETL Processor and Visual Importer ETL  processing Date filelds works up to 10 percent faster.

Old Version

date performace 1

New Version

date performace 1

Performance tips for processing Date Fields:

  • Formats with single character are the slowest, for example YYYY-M-D  has sigle M and D
  • Three characters are slow as well, for example YYYY-MMM-DD
  • Use relevant format for example if the data looks like this 2001-02-01 format YYYY-MM-DD HH:NN:SS.FFF will work but YYYY-MM-DD will be faster
  • Special tip for Advanced ETL Processor if the data is already in YYYY-MM-DD HH:NN:SS.FFF format there is no need to convert it
Visit Support Forum Learn More Download Buy It

Extracting Data from Shopify

Background

Shopify is a Canadian e-commerce company headquartered in Ottawa, Ontario, that develops computer software for online stores and retail point-of-sale systems.
Source Wikipedia

In this article we will demonstrate how easy it is to extract and manipulate Shopify data.

Preparation

1)  Create Shopify account at www.shopify.com

2 ) Log in

Shopify1

3 ) Click apps

Shopify1

4) Click Private Apps

Shopify1

5) Create new Private App

Shopify1

6) Note API key and passport we will need it later

Python

1)  Download and install python 2.7  version 32 bit for 32 bit version or 64 bit for 64 bit version of etl software .

https://www.python.org/downloads/release/python-2712/

2) Once installed run the following command from the command line

python.exe pip install requests

That will install requests python package

Extracting Data

We now ready to extract the data.

Run Advanced ETl Processor Enterprise

Create new package, add Script Action and make sure that language used is set Python

Shopify1

Enter this code, do not forget to replace SHOPNAME, KEY and PASSWORD

 import requests
response = requests.get( "https://SHOPNAME.myshopify.com/admin/orders.xml",
                               auth=("KEY","PASSWORD"))
file = open("orders.xml", "w")
file.write(response.text)
file.close()

Shopify1

Data can be extracted as XML or as JSON, just replace file extension XML with JSON. Once data is saved on the local drive it can be loaded into the database

Shopify1

More information about Shopify API can be found here

https://help.shopify.com/api/reference/order

Visit Support Forum Learn More Download Buy It

IP Location is very useful function when we want determine the geographical location of a given IP address.

We use www.freegeoip.net web site to get the location and it only allows to make maixumun 10000 cals per hour.

 Important Information:

  • There is no garantie that location is 100% accurate
  • IP can be behind the firewall so it might be not possible to get exact location
  • The more aggregated (country level) the more accurate.
  • The less aggregated (exact street number or post code) the less accurate.
  • IP Location Requires direct connection to the internet
  • IP Location is very heavy operation, therefore it can be slow

Basic usage example:

IP Geo Location Example3

IP Geo Location Example1

IP Location Transformation Function

IP Geo Location Example2

The Result

IP Geo Location Result

Visit Support Forum Learn More Download Buy It

We have 297 guests and no members online