sherlock.imports.ned_d module

Import ned_d catalogue into sherlock-catalogues database

Author

David Young

class ned_d(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source]

Bases: sherlock.imports._base_importer._base_importer

Import the * `NED-D <https://ned.ipac.caltech.edu/Library/Distances/>`_ *galaxy catalogue in to the sherlock-catalogues database

Key Arguments

  • log – logger

  • settings – the settings dictionary

  • pathToDataFile – path to the ned_d data file

  • version – version of the ned_d catalogue

  • catalogueName – the name of the catalogue

Usage

To import the ned_d catalogue catalogue, run the following:

from sherlock.imports import ned_d
catalogue = ned_d(
    log=log,
    settings=settings,
    pathToDataFile="/path/to/ned_d.txt",
    version="1.0",
    catalogueName="ned_d"
)
catalogue.ingest()
- abstract this module out into its own stand alone script
ingest()[source]

Import the ned_d catalogue into the catalogues database

The method first generates a list of python dictionaries from the ned_d datafile, imports this list of dictionaries into a database table and then generates the HTMIDs for that table.

Usage

See class docstring for usage

- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_create_dictionary_of_ned_d()[source]

create a list of dictionaries containing all the rows in the ned_d catalogue

Return

  • dictList - a list of dictionaries containing all the rows in the ned_d catalogue

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_clean_up_columns()[source]

clean up columns of the NED table

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_get_metadata_for_galaxies()[source]

get metadata for galaxies

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_count_galaxies_requiring_metadata()[source]

count galaxies requiring metadata

Return

  • self.total, self.batches – total number of galaxies needing metadata & the number of batches required to be sent to NED

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_get_3000_galaxies_needing_metadata()[source]

get 3000 galaxies needing metadata

Return

  • len(self.theseIds) – the number of NED IDs returned

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_query_ned_and_add_results_to_database(batchCount)[source]

query ned and add results to database

Key Arguments

  • batchCount - the index number of the batch sent to NED

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_update_sdss_coverage()[source]

update sdss coverage

- update key arguments values and definitions with defaults
- update return values and definitions
- update usage examples and text
- update docstring text
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_add_htmids_to_database_table()[source]

Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)

Usage

self._add_htmids_to_database_table()
_update_database_helper_table()[source]

Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated

Usage

self._update_database_helper_table()
add_data_to_database_table(dictList, createStatement=False)[source]

Import data in the list of dictionaries in the requested database table

Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated

Key Arguments

  • dictList - a list of dictionaries containing all the rows in the catalogue to be imported

  • createStatement - the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False

Usage

self.add_data_to_database_table(
    dictList=dictList,
    createStatement=createStatement
)
- Write a checklist for creating a new sherlock database importer