QVX

  • Problem:

    We have a scenario where we have a large data source that is currently used to produce a single QVW. What we would like to do is produce a number of smaller QVWs using e.g. Publisher - however, we would like to define the reload schedule and destination folders that these smaller QVWs end up in from an external MySQL table (or config file or similar), rather than through the Publisher GUI. Has anyone done anything similar, or got any recommendations on how to approach this?

    Original Post on LinkedIn

    Solution: Advanced ETL Processor QlikView automation package

    The package is executed from the left to the right starting from "Init Variables" object

    Package

    First of all, we've created the following table to store parameters, it can be extended if necessary

    table

    Table data:

    TableDetails

    Step 1: Init variables.

    Variables are used to replace one string with another, for example anywhere in the package where <RoorDirectory> is found it will be replaced with c:\Customers.

    Variables1

  • Advanced ETL Processor is a perfect solution for automating QVX file generation in a complex enterprise environment

    • QVX files can be generated in HQ or Regional offices
    • Data can be transferred via FTP, HTTP, EMAIL, file servers or downloaded from the WEB
    • The entire database can be saved into QVX files with just a few clicks
    • Complex data transformation and validation can be performed on the fly
    • In case of failure, the solution provider support team can be notified via email
    Download Free Trial Learn More About Our ETL Software
  • The latest version of Advanced ETL Processor can create 14000 QVX records per second, approximately 1 million records per minute.

    This is 10x increase comparing to the previous version!

    Creating QVX

    There are two ways of creating QVX files using packages or from the transformation. It is possible to save all tables in the database into QVX files with several clicks. And it is also possible to perform complex data transformations.

    Download Free Trial Learn More About Our ETL Software
  • Great news: Q-Eye version 2 can now edit QlikView QVD and QVX files

       

    QVD Files Viewer

     

    Download Q-Eye 32 Bit Download Q-Eye 64 Bit Download Q-Eye Portable Version

  • The release of Q-Eye allows QlikView users to view and even edit residential tables.

    Here are some examples:

    1. Viewing Data

    SUB INSPECT (T)
     LET LocalAppDataPath = GetRegistryString('HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion', 'ProgramFilesDir');
     LET QViewerPath = '$(LocalAppDataPath)\DB Software Laboratory\Q-Eye\QEye.exe';
     STORE $(T) into [$(QvWorkPath)\~$(T).qvd] (qvd);
     EXECUTE "$(QViewerPath)" "$(QvWorkPath)\~$(T).qvd";
     EXECUTE cmd /c del "$(QvWorkPath)\~$(T).qvd";
    ENDSUB

    2. Editing Data

    SUB EDIT (T)
     LET LocalAppDataPath = GetRegistryString('HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion', 'ProgramFilesDir');
     LET QViewerPath = '$(LocalAppDataPath)\DB Software Laboratory\Q-Eye\QEye.exe';
     STORE $(T) into [$(QvWorkPath)\~$(T).qvd] (qvd);
     EXECUTE "$(QViewerPath)" "$(QvWorkPath)\~$(T).qvd";
     DROP TABLE $(T);
     LOAD * FROM [$(QvWorkPath)\~$(T).qvd] (qvd);
     EXECUTE cmd /c del "$(QvWorkPath)\~$(T).qvd";
    ENDSUB


    If we want to edit the table we just

    CALL EDIT('Log');

    Or if we want to view the table we use this code

    CALL INSPECT('Log');

    Note: By default Q-Eye uses partial load, to avoid problems load entire QVD file before editing it 


    This example is based on:
    http://bi-review.blogspot.co.uk/2012/12/how-to-look-inside-resident-tables-at.html

  • We have just released version 3.5 of Q-Eye.

    This version loads and saves QVD files much faster. (Between 2 and 5 times on our test computers)

    Q-Eye3.5

    Performance is not guaranteed and depends on number factors such as: Hardware configuration, Size of files, Number of distinct fields values

    Since QVD is a closed format it is not possible to test it completely. Therefore we would like to ask QlikView users to assist us with testing.  

    Approximate test plan:
    Performance Test  

    Find the largest possible qvd file and load it completely into Q-Eye (Record the time took)   
    Find the largest possible qvd file and load it completely into QlikView (Record the time took)

    Data Quality test

    Open a number of qvd files and load them completely into Q-Eye.   
    Save the files.   
    Load the files into QlikView and check for data corruption.

    Data format test  

    Create qvd file manually by editing the data in Q-Eye.   
    Make sure that you are using all possible formats (text, numeric, date) and languages   
    (We are especially interested in testing non-English languages)   
    Save the file.   
    Load the file into QlikView and compare the data.

    Post your findings or suggestions on our support forum:

    Visit Support Forum

    Please support the development of Q-Eye by donating

    Donate

  • New version Advanced ETL Processor is available for download. In this version, we have enhanced support for QVX QlikView files.

    • Load data from QVX files
    • Generate QVX files
    • UsesQVX_QV_DUAL field type (the fastest way of loading data into QlikView) 
    • Can process files generated by QlikView itself

    Transforming QVX Files

    We would like to thank our customers for providing useful feedback

    This task was particularly challenging for our development team because there is no documentation for QVX_QV_DUAL field types

    Download Free Trial Learn More About Our ETL Software
  • Here how our customers use Advanced ETL Processor for data migration

    tableau partner
    Converting QVD files to TDE files

    The largest QVD file has 22 million records and it is 5.9 gigabytes in size

    Files

    Using Advanced ETL Processor 64 bit and latest tableau SDK (9.2)

    QVD To TDE 64

    It took 36 minutes to convert the data with an average speed of 10k per second.

    Notes:

    1. Performance depends on a number of factors: Hardware, hard disk performance, number of fields and the complexity of transformations
    2. 32-bit version slightly slower for the file's creation
    3. Always use the 64-bit version with large QVD's
    4. Please use our support forum to provide us with feedback
    Visit Support Forum Learn More Download Buy It
  • Q-Eye version 6 is available for download

    In this version we have made a lot of changes to file metadata dialogue:

    • Incremental search
    • Fields sorting
    • Data export
    • Print/preview

    Q Eye1 V5

    Q Eye1 V6

    Q Eye2 V5

    Q Eye2 V6

    Q Eye3 V5

    Q Eye3 V6

    32 Bit 64 Bit Portable Donate
  • Version 4.1 of  Q-Eye is available for download.

    Changes are:

    Added File Meta-data Dialogue
    Added Auto Format Option
    Added Single copy option
    Addressed issues with Single copy option
    Added Load Status to the status bar
    It is now possible to rearrange tabs
    Added donate button
    Added About button
    Added Support button
    Added List of Recent files
    Removed menu

    Q-Eye4 Metadata

     Q-Eye4.1

    Questions?: Post your findings or suggestions on our support forum:

    Visit Support Forum

    Please support the development of Q-Eye by donating

    Donate

  • The day has come: Q-Eye is available from Microsoft App Store

    It took a lot of time and effort but finally, we are there. We would like to thank Microsoft staff for their assistance.

    Store

    32 Bit 64 Bit Portable Donate

    Note: Q-Eye is not on Qlik Market despite the fact that the number of people and companies promised us to help with this matter. We believe that eventually, it will happen. Meantime we will continue making it better. 

  • Today we have not only great news but the one we are proud of, As you can see from the screenshots below our QVD Editor Q-Eye is incredibly fast now:

    it loads 31 million of records in 4 seconds, That is 7.3 million of records per second

    Q-Eye

     We still have some room for improvement for memory usage:

    MemoryUsage

     The same file is loaded into QlikView:

    QlikView

    The computer used for testing: Intel 7i, 8 Gigs of memory, 512 MB SSD disk
  • We have great news for all QlikView users today.

    Our freeware QVD Editor Q-Eye(3.3) Loads data faster than QlikView now.

    Q-Eye Performance Boost

    Same File is loaded using QlikView

    We have managed to achieve such performance increase by carefully optimizing code, removing bottlenecks and checking memory usage

    Notes:

    It took Q-Eye 3-7 seconds to load the file and QlikView between 10 and 17 seconds.
    Q-Eye was always faster than QlikView
    The actual result depends on the hardware and memory allocation by windows. Loading same file second time usually faster.
    Users can also use Advanced ETL Processor to transform and create QVD files 

    Download Free Trial Learn More About Our ETL Software
  • Version 5.0 of  Q-Eye is available for download.

    Changes are:

    Updated user interface
    Extended support for QlikSense QVD files
    Better support for Timestamp and Date fields

    qeyev2

    Visit Support Forum

    Please support development of Q-Eye by donating

    Donate

    Download Q-Eye 32 Bit Download Q-Eye 64 Bit Download Q-Eye Portable Version
  • Version 3.4.4.0 of ETL tools QlikView connector is available for download

    Changes are:

    + Third-party components update
     + Support for the new functions introduced in the Excel 2016 January Update
     + Improved compatibility with third-party xlsx files
     + Improved compatibility with invalid xls and xlsx files
     + Improved: Excel 2010 equations are now preserved in xlsx
     + Improved: Log and rejected records files are no longer using executable location
     + Improved: JDBC Errors Handling
     + New : Support SSL v2, v3, v1_1 and v1_2 was introduced
     + New: Support for opening xls versions from 2 to 4
     + Improved support for RSS feeds
     + Minor transformation designer improvements
     + Calculation transformation function support Python now
     + Added Write to File transformation function
     + SSL Libraries update
     + SalesForce connection uses TLS 1.1 now
     + Add refresh databases button to source and target connections
     + Added support for Unicode in the server, database and table names
     + ODBC error messages support Unicode now
     + Added ability to extract 'List of groups the user(s) belongs to' from Active Directory
     + Added ability to extract 'List of group(s) members' from Active Directory
     + Better support for QVD's Timestamp and Date fields
     + Added Ignore Data Format option to Excel Data reader
     + Added Lookup Replace Transformation function
     - Fixed: Problems with reading data from Unicode files
     - Fixed order of records specific for Excel 2010 to workaround a bug in Excel 2010.
     - Some very complex files could raise an error when opened in Excel 2010, even when they were correct by the xlsx spec.
     - Fixed : Xlsx files with 0 bytes xml files inside could throw an "Invalid XML file" error.
     - Fixed: Data validations entered manually in xls files could fail to work when opened in Excel
     - Address issues with Change Date Static, Change Date Dynamic and If WeekDay functions
     - Addressed Salesforce data update issues
     - Addressed issues with Running Total transformation
     - Log Object bugs fixes
     - Date fields processing bugs fixes
     - Joiner Object bugs fixes
     - Addressed DTD is prohibited issues
     - Addressed Date validation and conversion problems
     - Addressed issues with the print preview screen
     - Addressed issues flicking
     - Addressed issues with loading data from JDBC data sources
     - Addressed issues with loading data from Windows Event Log
     - Fixed issues with syntax highlighting in Mapping editor
     - Addressed issues with clone row object
     - Addressed issues with union and sorter objects
     - Addressed issues with the sub-transformation function
     - Addressed issues with split function
     - Addressed issues with joining empty files

    QlikViewConnectorV3.4.4.0

    Special offer

    Make a suggestion or ask a question on our support forum and get 10 percent discount

    32 Bit BUY
  • Version 4.1.0.2 of  Q-Eye is available for download.

    Changes are:Updated Metadata dialogue

    QVD Metadata1

     QVD Metadata2

    Questions?: Post your findings or suggestions on our support forum:

    Visit Support Forum

    Please support the development of Q-Eye by donating

    Donate

  • Comparison of data loading strategy into QlikView

    Using QVD files

    Working With QVD files

    There some advantages and disadvantages of using QVD files
    • Same data for all users
    • Loader QVS pulls data into memory and saves it into the file
    • Additional processing on the QlikView server
    • QVD’s can be generated only by QlikView
    Using QVX files

    Working With QVX files

    Benefits of using QVX files
    • Same data for all users
    • Advanced ETL Processor data saves the data into the file without loading everything into the memory
    • AETL can perform data transformation and validation on the fly
    • Can automate file transfers via FTP/Emails/HTTP/File operations.
    • Can work in a distributed environment
    Download Free Trial Learn More About Our ETL Software
  • Q-Eye version 6.5 is available for download

    The most important change is Metadata dialogue refreshment with a lot of useful information

    Q EyeMetaData

    Next important enhancement is introduction of Field Metadata Dialogue:

    Q EyeFieldMetaData

    The rest of the changes are:
    • Restructured Ribbon toolbar
    • Double click on the header renames field
    • Metadata dialogue's xml and file tabs are only shown to not yet modified documents
    • Generate LOAD Statement button was added to Metadata dialogue.
    • Ability to edit "generated load statement" was introduced
    • Added option to set number of rows
    • Added option to set number of columns
    • Added shortcut to insert row
    • Added shortcut to delete row
    • Added shortcut to insert column
    • Added shortcut to delete column
    • Improved column drag and drop functionality
    • Format button was renamed to fit the content
    • The software automatically adds rows when necessary
    • Lots of small but useful bugs fixes

    Q EyeMarked

    32 Bit 64 Bit Portable Donate

    Note: Q-Eye is not on Qlik Market despite the fact that the number of people and companies promised us to help with this matter. We believe that eventually, it will happen. Meantime we will continue making it better. 

  • A new version of Advanced ETL Processor is available for download

    Changes are:

     + Up to 2 times faster data extraction from SQL Server
     + Up to 2 times faster data extraction from ODBC sources
     + Up to 40 percent faster loading data into SQL Server
     + Up to 40 percent faster loading data into ODBC
     + Up to 10 percent faster QVX files creation
     + Up to 10 percent faster loading data into PostgreSQL
     - Various bugs fixes and improvements

    Here are our test results: pooling data from SQL server

    VersionRecords per secondTime Taken
    5.8.0.19 41,100 1min 12sec
    5.8.0.6 21,436 2min 19sec

    Extract Data From SQL Server NEW Version

     Extract Data From SQL Server OLD Version

    Notes:

    1. The performance also depends on the hardware configuration
    2. Please use our support forum to provide us with feedback
    Visit Support Forum Learn More Download Buy It
  • Those people who work with QlikView know that it requires a direct connection to the database to pull the data from, which is not possible to implement in some situations. One of the solutions is to use QVD files but again there is no way to create them without using QlikView.

    In version 10 QlikViewhas introduced support for QVX files.

    QVX (QlikView Data eXchange) is a new file/stream format for high-performance data input into QlikView. A QVX formatted file contains metadata describing a table of data and the actual data. In contrast to the QVD format, which is proprietary and optimized for minimum transformations inside QlikView, the QVX format is public and requires a few transformations when exporting data from traditional database formats.

    Advanced ETL processor and Visual Importer ETL allow the user to export entire data warehouse into QVX files just within a few clicks and no programming

     

    Create QVX form any datasource

    Data can be generated from virtually any data source.

    Note

    All data is exported as UTF8 text using row delimiter

    Here is the link to file format specification.

    http://community.qlikview.com/servlet/JiveServlet/previewBody/2677-102-1-2852/QlikView%20QVX%20File%20Format.pdf

    Online tutorial.

    How to save entire data warehouse into QVX files with just s few clicks

This site uses cookies. By continuing to browse the site, you are agreeing to our use of cookies