sherlock.imports.ned module ∞
Import ned stream into sherlock-catalogues database
- Author
David Young
-
class
ned
(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source] ∞ Bases:
sherlock.imports._base_importer._base_importer
Using a list of coordinates, query the online NED database and import sources found within a given search radius of each of the loctions into the sherlock-catalogues database
The code:
1. Uses the list of transient coordinates and queries NED (conesearch) for the results within the given search radius 2. Creates the `tcs_cat_ned_stream` table if it doesn't exist 3. Adds the resulting matched NED IDs/Names to the `tcs_cat_ned_stream` table 4. Updates the NED query history table 5. Queris NED via NED IDs (object search) for the remaining source metadata to be added to the `tcs_cat_ned_stream` table
Note it’s up to the user to filter the input coordinate list by checking whether or not the same area of the sky has been imported into the
tcs_cat_ned_stream
table recently (by checking thetcs_helper_ned_query_history
table)Key Arguments
dbConn
– mysql database connectionlog
– loggersettings
– the settings dictionarycoordinateList
– list of coordinates (a list of strings with RA and DEC space separated)radiusArcsec
- - the radius in arcsec with which to perform the initial NED conesearch. Default * False*
Usage
To import the ned catalogue stream, run the following:
from sherlock.imports import ned
stream = ned( log=log, settings=settings, coordinateList=["23.12323 -12.34343","345.43234 45.26789"], radiusArcsec=180 ) stream.ingest()
- test this code is still working after changes - add option to filter coordinate list via the `tcs_helper_ned_query_history` table - check sublime snippet exists - clip any useful text to docs mindmap
-
ingest
()[source] ∞ Perform conesearches of the online NED database and import the results into a the sherlock-database
The code:
1. uses the list of transient coordinates and queries NED for the results within the given search radius 2. Creates the `tcs_cat_ned_stream` table if it doesn't exist 3. Adds the resulting NED IDs/Names to the `tcs_cat_ned_stream` table 4. Updates the NED query history table 5. Queris NED via NED IDs for the remaining source metadata to be added to the `tcs_cat_ned_stream` table
Usage
Having setup the NED object with a coordinate list and cone-search radius, run the
ingest()
methodstream.ingest()
- check sublime snippet exists - clip any useful text to docs mindmap - regenerate the docs and check redendering of this docstring
-
_create_dictionary_of_ned
()[source] ∞ Create a list of dictionaries containing all the object ids (NED names) in the ned stream
Return
dictList
- a list of dictionaries containing all the object ids (NED names) in the ned stream
Usage
dictList = stream._create_dictionary_of_ned()
-
_update_ned_query_history
()[source] ∞ Update the database helper table to give details of the ned cone searches performed
Usage:
```python stream._update_ned_query_history() ```
-
_download_ned_source_metadata
()[source] ∞ Query NED using the names of the NED sources in our local database to retrieve extra metadata
Usage:
```python stream._download_ned_source_metadata() ```
-
_get_ned_sources_needing_metadata
()[source] ∞ Get the names of 50000 or less NED sources that still require metabase in the database
Return
len(self.theseIds)
– the number of NED IDs returned
Usage:
```python numberSources = stream._get_ned_sources_needing_metadata() ```
-
_do_ned_namesearch_queries_and_add_resulting_metadata_to_database
(batchCount)[source] ∞ Query NED via name searcha and add result metadata to database
Key Arguments
batchCount
- the index number of the batch sent to NED (only needed for printing to STDOUT to give user idea of progress)
Usage:
```python numberSources = stream._do_ned_namesearch_queries_and_add_resulting_metadata_to_database(batchCount=10) ```
-
_count_ned_sources_in_database_requiring_metadata
()[source] ∞ Count the sources in the NED table requiring metadata
Return
self.total
,self.batches
– total number of galaxies needing metadata & the number of batches required to be sent to NED
Usage:
```python totalRemaining, numberOfBatches = stream._count_ned_sources_in_database_requiring_metadata() ```
-
_add_htmids_to_database_table
()[source] ∞ Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)
Usage
self._add_htmids_to_database_table()
-
_update_database_helper_table
()[source] ∞ Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated
Usage
self._update_database_helper_table()
-
add_data_to_database_table
(dictList, createStatement=False)[source] ∞ Import data in the list of dictionaries in the requested database table
Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated
Key Arguments
dictList
- a list of dictionaries containing all the rows in the catalogue to be importedcreateStatement
- the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False
Usage
self.add_data_to_database_table( dictList=dictList, createStatement=createStatement )
- Write a checklist for creating a new sherlock database importer