sherlock.imports.ned_d (class)

class sherlock.imports.ned_d(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source]

Bases: sherlock.imports._base_importer._base_importer

Import the * `NED-D <https://ned.ipac.caltech.edu/Library/Distances/>`_ *galaxy catalogue in to the sherlock-catalogues database

Key Arguments:
  • log – logger
  • settings – the settings dictionary
  • pathToDataFile – path to the ned_d data file
  • version – version of the ned_d catalogue
  • catalogueName – the name of the catalogue

Usage:

To import the ned_d catalogue catalogue, run the following:

from sherlock.imports import ned_d
catalogue = ned_d(
    log=log,
    settings=settings,
    pathToDataFile="/path/to/ned_d.txt",
    version="1.0",
    catalogueName="ned_d"
)
catalogue.ingest()

Todo

  • abstract this module out into its own stand alone script

Methods

add_data_to_database_table(dictList[, …]) Import data in the list of dictionaries in the requested database table
ingest() Import the ned_d catalogue into the catalogues database
_add_htmids_to_database_table() Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)
_clean_up_columns() clean up columns of the NED table
_count_galaxies_requiring_metadata() count galaxies requiring metadata
_create_dictionary_of_ned_d() create a list of dictionaries containing all the rows in the ned_d catalogue
_get_3000_galaxies_needing_metadata() get 3000 galaxies needing metadata
_get_metadata_for_galaxies() get metadata for galaxies
_query_ned_and_add_results_to_database(…) query ned and add results to database
_update_database_helper_table() Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated
_update_sdss_coverage() update sdss coverage
ingest()[source]

Import the ned_d catalogue into the catalogues database

The method first generates a list of python dictionaries from the ned_d datafile, imports this list of dictionaries into a database table and then generates the HTMIDs for that table.

Usage:

See class docstring for usage

Todo

  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_create_dictionary_of_ned_d()[source]

create a list of dictionaries containing all the rows in the ned_d catalogue

Return:
  • dictList - a list of dictionaries containing all the rows in the ned_d catalogue

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_clean_up_columns()[source]

clean up columns of the NED table

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_get_metadata_for_galaxies()[source]

get metadata for galaxies

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_count_galaxies_requiring_metadata()[source]

count galaxies requiring metadata

Return:
  • self.total, self.batches – total number of galaxies needing metadata & the number of batches required to be sent to NED

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_get_3000_galaxies_needing_metadata()[source]

get 3000 galaxies needing metadata

Return:
  • len(self.theseIds) – the number of NED IDs returned

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_query_ned_and_add_results_to_database(batchCount)[source]

query ned and add results to database

Key Arguments:
  • batchCount - the index number of the batch sent to NED

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_update_sdss_coverage()[source]

update sdss coverage

Todo

  • update key arguments values and definitions with defaults
  • update return values and definitions
  • update usage examples and text
  • update docstring text
  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_add_htmids_to_database_table()

Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)

Usage:

self._add_htmids_to_database_table()
_update_database_helper_table()

Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated

Usage:

self._update_database_helper_table()
add_data_to_database_table(dictList, createStatement=False)

Import data in the list of dictionaries in the requested database table

Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated

Key Arguments:
  • dictList - a list of dictionaries containing all the rows in the catalogue to be imported
  • createStatement - the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False

Usage:

self.add_data_to_database_table(
    dictList=dictList,
    createStatement=createStatement
)

Todo

  • Write a checklist for creating a new sherlock database importer