sherlock.imports.ned_d module ∞
Import ned_d catalogue into sherlock-catalogues database
- Author
David Young
-
class
ned_d
(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source] ∞ Bases:
sherlock.imports._base_importer._base_importer
Import the * `NED-D <https://ned.ipac.caltech.edu/Library/Distances/>`_ *galaxy catalogue in to the sherlock-catalogues database
Key Arguments
log
– loggersettings
– the settings dictionarypathToDataFile
– path to the ned_d data fileversion
– version of the ned_d cataloguecatalogueName
– the name of the catalogue
Usage
To import the ned_d catalogue catalogue, run the following:
from sherlock.imports import ned_d catalogue = ned_d( log=log, settings=settings, pathToDataFile="/path/to/ned_d.txt", version="1.0", catalogueName="ned_d" ) catalogue.ingest()
- abstract this module out into its own stand alone script
-
ingest
()[source] ∞ Import the ned_d catalogue into the catalogues database
The method first generates a list of python dictionaries from the ned_d datafile, imports this list of dictionaries into a database table and then generates the HTMIDs for that table.
Usage
See class docstring for usage
- update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_create_dictionary_of_ned_d
()[source] ∞ create a list of dictionaries containing all the rows in the ned_d catalogue
Return
dictList
- a list of dictionaries containing all the rows in the ned_d catalogue
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_clean_up_columns
()[source] ∞ clean up columns of the NED table
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_get_metadata_for_galaxies
()[source] ∞ get metadata for galaxies
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_count_galaxies_requiring_metadata
()[source] ∞ count galaxies requiring metadata
Return
self.total
,self.batches
– total number of galaxies needing metadata & the number of batches required to be sent to NED
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_get_3000_galaxies_needing_metadata
()[source] ∞ get 3000 galaxies needing metadata
Return
len(self.theseIds)
– the number of NED IDs returned
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_query_ned_and_add_results_to_database
(batchCount)[source] ∞ query ned and add results to database
Key Arguments
batchCount
- the index number of the batch sent to NED
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_update_sdss_coverage
()[source] ∞ update sdss coverage
- update key arguments values and definitions with defaults - update return values and definitions - update usage examples and text - update docstring text - check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_add_htmids_to_database_table
()[source] ∞ Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)
Usage
self._add_htmids_to_database_table()
-
_update_database_helper_table
()[source] ∞ Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated
Usage
self._update_database_helper_table()
-
add_data_to_database_table
(dictList, createStatement=False)[source] ∞ Import data in the list of dictionaries in the requested database table
Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated
Key Arguments
dictList
- a list of dictionaries containing all the rows in the catalogue to be importedcreateStatement
- the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False
Usage
self.add_data_to_database_table( dictList=dictList, createStatement=createStatement )
- Write a checklist for creating a new sherlock database importer