sherlock.imports.marshall (class)

class sherlock.imports.marshall(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source]

Bases: sherlock.imports._base_importer._base_importer

Import the ePESSTO Marshall transient streams (includes multiple on-going transient survey streams) into the Sherlock-catalogues database

Key Arguments:
  • log – logger
  • settings – the settings dictionary

Usage:

To import/update the marshall catalogue streams in the sherlock-catalogues database, run the following:

from sherlock.imports import marshall
stream = marshall(
    log=log,
    settings=settings
)
stream.ingest()

Todo

  • abstract this module out into its own stand alone script
  • check sublime snippet exists

Methods

add_data_to_database_table(dictList[, …]) Import data in the list of dictionaries in the requested database table
ingest() Ingest the ePESSTO Marshall transient stream into the catalogues database
_add_htmids_to_database_table() Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)
_create_dictionary_of_marshall(…) create a list of dictionaries containing all the rows in the marshall stream
_update_database_helper_table() Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated
ingest()[source]

Ingest the ePESSTO Marshall transient stream into the catalogues database

The method first creates the tables for the various marshall feeder surveys in the sherlock-catalogues database (if they do not yet exist). Then the marshall database is queried for each transient survey and the results imported into the sherlock-catalogues tables,

See the class docstring for usage

Todo

  • convert the directory_script_runner to ‘load in file’
_create_dictionary_of_marshall(marshallQuery, marshallTable)[source]

create a list of dictionaries containing all the rows in the marshall stream

Key Arguments:
  • marshallQuery – the query used to lift the required data from the marshall database.
  • marshallTable – the name of the marshall table we are lifting the data from.
Return:
  • dictList - a list of dictionaries containing all the rows in the marshall stream
_add_htmids_to_database_table()

Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)

Usage:

self._add_htmids_to_database_table()
_update_database_helper_table()

Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated

Usage:

self._update_database_helper_table()
add_data_to_database_table(dictList, createStatement=False)

Import data in the list of dictionaries in the requested database table

Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated

Key Arguments:
  • dictList - a list of dictionaries containing all the rows in the catalogue to be imported
  • createStatement - the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False

Usage:

self.add_data_to_database_table(
    dictList=dictList,
    createStatement=createStatement
)

Todo

  • Write a checklist for creating a new sherlock database importer