sherlock.imports.ned (class)

class sherlock.imports.ned(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source]

Bases: sherlock.imports._base_importer._base_importer

Using a list of coordinates, query the online NED database and import sources found within a given search radius of each of the loctions into the sherlock-catalogues database

The code:

  1. Uses the list of transient coordinates and queries NED (conesearch) for the results within the given search radius
  2. Creates the tcs_cat_ned_stream table if it doesn’t exist
  3. Adds the resulting matched NED IDs/Names to the tcs_cat_ned_stream table
  4. Updates the NED query history table
  5. Queris NED via NED IDs (object search) for the remaining source metadata to be added to the tcs_cat_ned_stream table

Note it’s up to the user to filter the input coordinate list by checking whether or not the same area of the sky has been imported into the tcs_cat_ned_stream table recently (by checking the tcs_helper_ned_query_history table)

Key Arguments:
  • dbConn – mysql database connection
  • log – logger
  • settings – the settings dictionary
  • coordinateList – list of coordinates (a list of strings with RA and DEC space separated)
  • radiusArcsec - - the radius in arcsec with which to perform the initial NED conesearch. Default * False*

Usage:

To import the ned catalogue stream, run the following:

from sherlock.imports import ned
stream = ned(
    log=log,
    settings=settings,
    coordinateList=["23.12323 -12.34343","345.43234 45.26789"],
    radiusArcsec=180
)
stream.ingest()

Todo

  • test this code is still working after changes
  • add option to filter coordinate list via the tcs_helper_ned_query_history table
  • check sublime snippet exists
  • clip any useful text to docs mindmap

Methods

add_data_to_database_table(dictList[, …]) Import data in the list of dictionaries in the requested database table
ingest() Perform conesearches of the online NED database and import the results into a the sherlock-database
_add_htmids_to_database_table() Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)
_count_ned_sources_in_database_requiring_metadata() Count the sources in the NED table requiring metadata
_create_dictionary_of_ned() Create a list of dictionaries containing all the object ids (NED names) in the ned stream
_do_ned_namesearch_queries_and_add_resulting_metadata_to_database(…) Query NED via name searcha and add result metadata to database
_download_ned_source_metadata() Query NED using the names of the NED sources in our local database to retrieve extra metadata
_get_ned_sources_needing_metadata() Get the names of 50000 or less NED sources that still require metabase in the database
_update_database_helper_table() Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated
_update_ned_query_history() Update the database helper table to give details of the ned cone searches performed
ingest()[source]

Perform conesearches of the online NED database and import the results into a the sherlock-database

The code:

  1. uses the list of transient coordinates and queries NED for the results within the given search radius
  2. Creates the tcs_cat_ned_stream table if it doesn’t exist
  3. Adds the resulting NED IDs/Names to the tcs_cat_ned_stream table
  4. Updates the NED query history table
  5. Queris NED via NED IDs for the remaining source metadata to be added to the tcs_cat_ned_stream table

Usage:

Having setup the NED object with a coordinate list and cone-search radius, run the ingest() method

stream.ingest()

Todo

  • check sublime snippet exists
  • clip any useful text to docs mindmap
  • regenerate the docs and check redendering of this docstring
_create_dictionary_of_ned()[source]

Create a list of dictionaries containing all the object ids (NED names) in the ned stream

Return:
  • dictList - a list of dictionaries containing all the object ids (NED names) in the ned stream

Usage:

dictList = stream._create_dictionary_of_ned()
_update_ned_query_history()[source]

Update the database helper table to give details of the ned cone searches performed

Usage:

stream._update_ned_query_history()
_download_ned_source_metadata()[source]

Query NED using the names of the NED sources in our local database to retrieve extra metadata

Usage:

stream._download_ned_source_metadata()
_get_ned_sources_needing_metadata()[source]

Get the names of 50000 or less NED sources that still require metabase in the database

Return:
  • len(self.theseIds) – the number of NED IDs returned

Usage:

numberSources = stream._get_ned_sources_needing_metadata()
_do_ned_namesearch_queries_and_add_resulting_metadata_to_database(batchCount)[source]

Query NED via name searcha and add result metadata to database

Key Arguments:
  • batchCount - the index number of the batch sent to NED (only needed for printing to STDOUT to give user idea of progress)

Usage:

numberSources = stream._do_ned_namesearch_queries_and_add_resulting_metadata_to_database(batchCount=10)
_count_ned_sources_in_database_requiring_metadata()[source]

Count the sources in the NED table requiring metadata

Return:
  • self.total, self.batches – total number of galaxies needing metadata & the number of batches required to be sent to NED

Usage:

totalRemaining, numberOfBatches = stream._count_ned_sources_in_database_requiring_metadata()
_add_htmids_to_database_table()

Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)

Usage:

self._add_htmids_to_database_table()
_update_database_helper_table()

Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated

Usage:

self._update_database_helper_table()
add_data_to_database_table(dictList, createStatement=False)

Import data in the list of dictionaries in the requested database table

Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated

Key Arguments:
  • dictList - a list of dictionaries containing all the rows in the catalogue to be imported
  • createStatement - the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False

Usage:

self.add_data_to_database_table(
    dictList=dictList,
    createStatement=createStatement
)

Todo

  • Write a checklist for creating a new sherlock database importer