Integration for Hadoop using Robot Framework

Question

Is there any library that I can use to integrate Hive/Impala/Spark scripts using Robot Framework? I tried using pyodbc but it forces to connect to SQL Server:

*** Settings ***
Library     DatabaseLibrary
Library     OperatingSystem

Suite Setup         Connect To Database       pyodbc  ${DBName}   ${DBUser}   ${DBPass}   ${DBHost}   ${DBPort}
Suite Teardown      Disconnect From Database

*** Variables ***
${DBName}   xxxx
${DBUser}   xxxx
${DBPass}   xxxx
${DBHost}   xxxx
${DBPort}   10000

*** Test Cases ***
Create person table
    ${output}=  Execute SQL String  SELECT COUNT(*) FROM my_table
    log to console  ${output}
    should be equal as strings  ${output}   None

The output:

Parent suite setup failed:
OperationalError: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect); [08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (Connect()). (53
)')

Anyone have an example how I can connect Robot Framework to Hive/Impala?

Thanks!

Answer 1

DatabaseLibrary is based on DB API 2.0 which supports these database systems: Generic Database Interfaces and APIs. So you won't be able to use this library in your case.

I think you best option would be to use a Python module and write your own library that you import into Robot. I don't really know these technologies (hive etc.) much, but it seems you can use e.g. PyHive, you can find more on StackOverflow

In terms of Robot library, a template could look like this:

from robot.api import logger
from pyhive import hive

class HiveConnector(object):
    """
    Hive connector.    

    = Table of content =
    - Keywords    
    """

    ROBOT_LIBRARY_VERSION = 1.0
    ROBOT_LIBRARY_SCOPE = 'GLOBAL'

    def __init__(self, host="", port="", username=""):
        self._conn = hive.Connection(host=host, port=port, username=username)

    # you add methods that will become keywords in Robot

Then in your keywords file(s) and/or test suites, you can import your newly created library:

*** Settings ***
Library    ../Libraries/HiveConnector.py    ${DBHost}   ${DBPort}   ${DBUser} 

You can find out more details in the documentation for Robot Framework: Creating test libraries. This shouldn't even take that long and I'm sure Python has already modules for connecting to Hive/Impala/Spark. So you just need to wrap them in a custom library you create.