WOHOO 1st Python Script - Upload Files to Gdrive

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago - 2 years 1 month ago #24040 by prashant

WOHOO 1st Python Script - Upload Files to Gdrive was created by prashant

Ofcourse the included package works , but I had like to share my first success . Using ETL Tool package to run python
Sailent Features

Uses Service Workers , so no need to authenticate individual user
Impersonation of any user in domain (use at own peril)
Ability to delete uploaded files after upload
Get file_ID of the newly uploaded file -> This was critical for us , as we need to pass this to new workflows
Use Set Variable package to pass all relevant variables

MISSING

Instructions to Create JSON file
Set Variables photo
No proper logging of error

Code:

import os
import mimetypes
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
import etltools

# Function to get credentials
def get_credentials(service_account_file, impersonated_user_email, scopes):
    credentials = service_account.Credentials.from_service_account_file(
        service_account_file, scopes=scopes
    )
    return credentials.with_subject(impersonated_user_email)

# Function to create Google Drive service
def create_drive_service(credentials):
    return build('drive', 'v3', credentials=credentials, cache_discovery=False)

# Function to upload file
def upload_file(service, file_path, parent_folder_id=None):
    file_metadata = {'name': os.path.basename(file_path)}
    if parent_folder_id:
        file_metadata['parents'] = [parent_folder_id]
    mime_type = guess_mime_type(file_path)
    media = MediaFileUpload(file_path, mimetype=mime_type)

    file = service.files().create(
        body=file_metadata,
        media_body=media,
        fields='id'
    ).execute()

    file_id = file.get('id')
    etltools.SetVariable('gdrive_value', file_id)
    return file_id

# Function to guess the MIME type
def guess_mime_type(file_path):
    mime_type, _ = mimetypes.guess_type(file_path)
    return mime_type or 'application/octet-stream'

# Main function
def main():
    try:
        service_account_file = etltools.GetVariable('service_account_file')
        impersonated_user_email = etltools.GetVariable('impersonated_user_email')
        file_path = etltools.GetVariable('file_path')
        parent_folder_id = etltools.GetVariable('parent_folder_id')
        delete_after_upload = etltools.GetVariable('delete_after_upload')

        scopes = [
            'https://www.googleapis.com/auth/drive',
            'https://www.googleapis.com/auth/drive.file'
        ]
        
        credentials = get_credentials(service_account_file, impersonated_user_email, scopes)
        service = create_drive_service(credentials)
        file_id = upload_file(service, file_path, parent_folder_id)

        if delete_after_upload == 'TRUE':
            os.remove(file_path)

        Result.Value = 1
    except Exception as e:
        etltools.SetVariable('error_message', str(e))
        Result.Value = 0

if __name__ == '__main__':
    main()

Last edit: 2 years 1 month ago by prashant.

The following user(s) said Thank You: admin

Please Log in or Create an account to join the conversation.

admin
Offline
Administrator
Deeds not Words

More

2 years 1 month ago #24041 by admin

Replied by admin on topic WOHOO 1st Python Script - Upload Files to Gdrive

Looks great

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #24042 by prashant

Replied by prashant on topic WOHOO 1st Python Script - Upload Files to Gdrive

Special instructions for chatgpt to convert your regular script to ETL

1. **Import ETL Tools**:
import etltools
2. **Replace Argument Parsing**:
- Remove traditional argument parsing (e.g., `argparse`) and replace with `etltools.GetVariable` to retrieve variables:
variable_name = etltools.GetVariable('variable_name')
3. **Remove Traditional Logging**:
- Eliminate any logging setup and log statements, replacing them with ETL error logging
etltools.SetVariable('error_message', str(e))
4. **Set Result Values**:
- Use `Result.Value` to indicate success or failure:
Result.Value = 1 # Success
Result.Value = 0 # Failure
6. **Main Function**:
- Encapsulate the main script logic within a `main()` function and handle exceptions to log errors:
def main():
try:
# ETL variable handling
# Main logic
Result.Value = 1
except Exception as e:
etltools.SetVariable('error_message', str(e))
Result.Value = 0

Please Log in or Create an account to join the conversation.

prashant
Topic Author
Offline
Platinum Member

More

2 years 1 month ago #24071 by prashant

Replied by prashant on topic WOHOO 1st Python Script - Upload Files to Gdrive

Learning Lesson. Variables passed are converted to string , in my case , JSON passed from one script to another has to re-converted back

xjson = etltools.GetVariable('<json_variable>')
xjson_value = json.loads(xjson)

--
1. import etltools
2. Replace Argument Parsing
- Remove traditional argument parsing (e.g., `argparse`) and replace with `etltools.GetVariable` to retrieve variables:
variable_name = etltools.GetVariable('variable_name')
3. Remove Traditional Logging: Eliminate any logging setup and log statements, replacing them with ETL error logging - etltools.SetVariable('error_message', str(e))
4. Use `Result.Value` to indicate success or failure:
Result.Value = 1 # Success
Result.Value = 0 # Failure
6. Encapsulate the main script logic within a `main()` function and handle exceptions to log errors:
def main():
try:
# ETL variable handling
# Main logic
Result.Value = 1
except Exception as e:
etltools.SetVariable('error_message', str(e))
Result.Value = 0
7. Variables passed are converted to string , in my case , JSON passed from one script to another has to re-converted back
xjson = etltools.GetVariable('<json_variable>')
xjson_value = json.loads(xjson)

The following user(s) said Thank You: Peter.Jonson

Please Log in or Create an account to join the conversation.

ETL Forum